Perception: CP (coarse perception), FP-S (single-instance fine-grained perception), FP-C (cross-instance fine-grained perception), HL (Hallucination)
Reasoning: LR (logic reasoning), AR (attribute reasoning), RR (relation reasoning), CSR (commonsense reasoning), TR (temporal reasoning)
The best results are highlighted in bold and underlined.
Model | Perception | Reasoning | Overall Mean | Model Type | |||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|
CP | FP-S | FP-C | HL | Mean | LR | AR | RR | CSR | TR | Mean | |||
GPT-4o-[8f] 🥇 | 1.82 | 1.59 | 1.43 | 1.95 | 1.63 | 1.33 | 1.89 | 1.60 | 1.60 | 1.44 | 1.57 | 1.62 | Proprietary LVLMs for Images |
GPT-4v-[8f] 🥈 | 1.68 | 1.45 | 1.43 | 1.79 | 1.51 | 1.14 | 1.81 | 1.70 | 1.59 | 1.39 | 1.52 | 1.53 | Proprietary LVLMs for Images |
Gemini-Pro-v1.0-[8f] 🥉 | 1.72 | 1.50 | 1.28 | 0.79 | 1.49 | 1.02 | 1.66 | 1.58 | 1.59 | 1.40 | 1.45 | 1.49 | Proprietary LVLMs for Images |
Gemini-Pro-v1.5-[8f] | 1.51 | 1.30 | 0.98 | 2.03 | 1.32 | 1.06 | 1.62 | 1.36 | 1.25 | 0.94 | 1.22 | 1.30 | Proprietary LVLMs for Images |
InternVL-Chat-v1.5-[8f] | 1.51 | 1.22 | 1.01 | 1.21 | 1.25 | 0.88 | 1.40 | 1.48 | 1.28 | 1.09 | 1.22 | 1.26 | Open-Source LVLM for Images |
Claude-3v-Opus-[4f] | 1.37 | 1.11 | 1.00 | 1.56 | 1.16 | 1.12 | 1.35 | 1.36 | 1.17 | 1.05 | 1.20 | 1.19 | Proprietary LVLMs for Images |
mPLUG-Owl2-[8f] | 1.34 | 1.18 | 0.99 | 0.27 | 1.15 | 0.63 | 1.33 | 1.30 | 1.03 | 1.11 | 1.11 | 1.15 | Open-Source LVLM for Images |
VideoStreaming-[64f+] | 1.38 | 1.13 | 0.80 | 0.32 | 1.13 | 0.77 | 1.27 | 1.11 | 1.01 | 1.10 | 1.09 | 1.12 | Open-Source Video LLM |
idefics2-8B-[8f] | 1.23 | 1.07 | 0.89 | 0.77 | 1.06 | 0.77 | 1.27 | 1.41 | 1.11 | 1.14 | 1.16 | 1.10 | Open-Source LVLM for Images |
ShareGPT4Video-8B-[16f*] | 1.20 | 1.05 | 1.00 | 0.32 | 1.04 | 0.89 | 1.06 | 1.19 | 1.01 | 0.99 | 1.03 | 1.05 | Open-Source Video LLM |
Video-LLaVA-[8f] | 1.14 | 1.08 | 0.88 | 0.50 | 1.04 | 0.72 | 1.23 | 1.03 | 0.89 | 0.97 | 0.99 | 1.05 | Open-Source Video LLM |
PLLaVA-7B-[16f] | 1.08 | 1.06 | 0.86 | 0.52 | 1.02 | 0.64 | 1.25 | 1.17 | 0.98 | 1.01 | 1.03 | 1.03 | Open-Source Video LLM |
Chat-UniVi-[64f] | 1.07 | 1.00 | 0.93 | 0.39 | 0.98 | 0.59 | 1.18 | 1.14 | 0.75 | 0.98 | 0.97 | 0.99 | Open-Source Video LLM |
VideoChat2-[16f] | 1.18 | 0.94 | 0.98 | 0.66 | 0.98 | 0.42 | 1.13 | 1.24 | 0.86 | 0.94 | 0.95 | 0.99 | Open-Source Video LLM |
Video-ChatGPT-[100f] | 0.91 | 0.94 | 0.81 | 0.39 | 0.90 | 0.70 | 1.15 | 1.12 | 0.84 | 0.94 | 0.97 | 0.93 | Open-Source Video LLM |
Qwen-8B-[8f] | 0.44 | 0.62 | 0.33 | 0.15 | 0.53 | 0.45 | 0.59 | 0.50 | 0.36 | 0.37 | 0.45 | 0.52 | Open-Source LVLM for Images |