Results and analysis

Model performance, rank correlations, ablations, and error analysis from the AgentVQA paper.

Model performance Rank correlations Ablations Error modes

Model performance

The best overall accuracy is 58.5%. No single model leads across all five domains.

Paper-reported accuracy (%). Select a column heading to sort.

Rank correlations

We compare model rankings on AgentVQA with general reasoning benchmarks and an online agent benchmark.

A different ordering from general benchmarks

The paper evaluates matched subsets with standardized prompts. Model rankings shift across MMMU-Pro, GPQA-Diamond, and ChartQA-Pro, with moderate to weak rank correlations.

MMMU-Pro0.67

GPQA-Diamond0.59

ChartQA-Pro0.44

Spearman’s ρ · 10-model comparison

Rank comparisons for ten models between AgentVQA and MMMU-Pro, GPQA-Diamond, and ChartQA-Pro. — Rank shifts in the paper’s general-benchmark comparison. Open the figure to inspect model labels.

Strong agreement in a matched online domain

AgentVQA Spatial Planning and the SpatialWorld Physical track agree strongly across their 12 shared models. This is a separate overlapping-model cohort from the main 22-configuration evaluation.

ρ = 0.89512 shared models · p = 8 × 10⁻⁵

Correlations between AgentVQA domains and SpatialWorld Physical and Digital tracks, with model ranks for the matched spatial-planning comparison. — The paper’s online comparison. The figure rounds the primary coefficient to 0.90; the text reports 0.895. Lower correlations in other domains emphasize that the capabilities being measured matter.

Ablations

Action history improves web-agent decisions

Across five models, including previous actions improves performance on AitW and MONDAY. The figure shows average accuracy with and without action history.

Action-history ablation: AitW rises from 38 to 42 percent, and MONDAY from 45 to 50 percent, as rounded in the paper figure. — Rounded values shown in the paper’s action-history figure.

Reasoning gains depend on the model

Extended reasoning improves both GPT-5.6 variants, but GLM-4.1V-9B-Thinking trails its base counterpart. Accuracy alone does not tell the full story of how models use extra computation.

Overall accuracy (%) from the main results
Model	None / Base	High / Thinking	Δ (pp)
GPT-5.6-sol	38.2	55.0	+16.8
GPT-5.6-luna	32.8	47.9	+15.1
GLM-4.1V-9B	36.3	35.4	−0.9

Δ is the difference in percentage points. The paper also documents “thinking loops” in GLM-4.1V-Thinking.

Error analysis

The paper analyzes incorrect Gemini 3.1 Pro predictions sampled approximately evenly across datasets. Error categories are model-assisted and manually verified.

Domain	Most common error	Share of analyzed errors
Web Agents	Grounding error	78%
Egocentric Videos	Motion misperception	39%
Robotics	Spatial confusion	65%
Games	Visual miscomprehension	71%
Spatial Planning	Viewpoint gap	62%

Percentages are shares of the analyzed errors within each domain—not error rates across all questions.

Paper distribution of error categories across Web Agents, Egocentric Videos, Robotics, Games, and Spatial Planning. — Full error-mode distribution from the paper. The dominant failures concern perception, grounding, and state construction.