When VLM Judges Miss the Big Picture

Local accuracy ≠ long-horizon understanding

A VLM judge does not observe a video; it observes a lossy encoding of one. Agreement with humans is bounded by the sampling operator applied before the judge reasons at all, and evidence on that operator is poor: T*: Re-thinking Temporal Search for Long-Form Video Understanding (Ye et al., CVPR 2025) reports that prior state-of-the-art keyframe selection methods reach only a 2.1% temporal F1 score on the LVBench subset of LV-Haystack, and the strongest learned search policy since, TimeSearch-R (2025), lifts the state of the art to 8.1 - a roughly threefold gain that still leaves the deciding frames unretrieved most of the time.

Temporal failure is also not reducible to context length. Vinoground documents dense temporal reasoning failures on short videos - and its 2025 update finds even o3, the strongest model tested, near 50% on the group metric against a human baseline near 90% - while Temporal Reasoning Transfer from Text to Video localizes much of the bottleneck in the language backbone rather than the visual encoder. Validation sets can themselves be frame-solvable: the Neptune authors observe that many long-video benchmark questions yield to strong image models applied per frame, often to very few frames.

Judge errors decompose into two stages usually measured as one. Detection correctness asks whether a mismatch was correctly identified; decision correctness asks whether it should have moved the score. In production caption evaluation we observe that most rubric-induced false positives are decision errors sitting on correct perception - an operational pattern rather than a controlled result - so the fix is architectural rather than a better prompt: multi-resolution evaluation, structured evidence extraction, explicit salience and severity weights, confidence gating, and human adjudication at the decision boundary.

The hidden mismatch in multimodal evaluation

Write the judge as J(x, y, r) → s: visual evidence x, candidate output y, rubric r, score s. The signature makes visible which arguments the judge is actually given.

The substitution most reports leave implicit sits in x: for video, x is never the video v. It is S(v), the output of a sampling or retrieval operator: uniform sampling, keyframe selection, segment retrieval, or learned compression into a token budget. The judge computes J(S(v), y, r) while the human it is benchmarked against approximates H(v, y). Evidence that S destroys is unrecoverable downstream, capping agreement independently of judge perceptual accuracy. Disagreement therefore has two causes with different fixes: perceptual error calls for a better judge, evidence absence for better retrieval.

A second substitution sits in r. A rubric is not a neutral instruction but a weighting function over error types, and rubrics written as detail checklists weight enumerated items roughly uniformly. Human judgment does not: it is salience-weighted and organized around communicative adequacy. Decomposing the target makes the gap explicit:

Human-aligned quality ≈ local correctness + temporal consistency + semantic coverage + salience weighting − material errors

Local correctness is close to what pretraining optimizes; temporal consistency depends on the receptive field of S and the reasoning backbone. Salience weighting and materiality are pragmatic judgments about what a reader needed, not perceptual facts about what a frame contained, and are least well supplied by either pretraining or a typical rubric. Teams building multimodal AI training and evaluation data meet this asymmetry early: annotating what is visible is tractable, annotating what mattered requires explicit policy.

Why image judges become detail-maximizing critics

Evaluation granularity is an axis along which judges and humans separate. Six levels matter: object, attribute, relationship, event, narrative, and task-level usefulness. Judges are pretrained and benchmarked overwhelmingly on the first three; rubrics are routinely written as though competence there generalizes to the last three.

Structured extraction helps at those levels: the VisCE² method of Maeda and colleagues substitutes structured visual context for human-written references and improves correlation with human judgment relative to CIDEr, CLIPScore, and BERTScore. The failure mode is extrapolating past them - a judge that enumerates well, deployed under a rubric that treats enumeration as the score. Improved object recall then increases detectable omissions without changing adequacy, so recall gains need not improve correlation with human preference.

Reference construction compounds this. If r anchors scoring to a single reference rather than scoring reference-free, the judge inherits one annotator's coverage decisions as ground truth. The FIOVA benchmark refuses that shortcut, using 3,002 videos independently annotated by five annotators each - 15,010 annotations in total - and reports a “complexity compression effect” in which human descriptions diversify as semantic complexity rises while model outputs converge toward uniform templates. If human references form a distribution rather than a point, measuring omission against one sample is a biased estimator of adequacy.

Detection correctness is not decision correctness

The pattern is a two-stage error model. Detection correctness: the judge correctly identifies that y omits, contradicts, or misstates something present in x. Decision correctness: the judge correctly determines whether that mismatch should change s. They are separately measurable and separately fixable: a judge can pair excellent detection precision with poor decision precision, and the resulting false positives read as perception failures when they are rubric failures.

Operationally, the distinction is between detecting that a detail is absent and determining whether its absence is a material quality failure. The pattern is visible in current caption evaluation with an image-based quality judge, where the characteristic false-positive mode is not fuzziness about language but the opposite: the judge behaves as a stickler, treating the presence of an individual detail as disproportionately important, and human reviewers override it to credit the big picture. This is an operational observation rather than a controlled result, but its mechanisms are identifiable: uniform weighting of details, rubric over-specification promoting optional content to required, entailment-style framing in which any unentailed reference content registers as a defect, single-annotator reference anchoring, and no representation of communicative intent. It is also the image-domain instance of the video pattern this article concerns - judge-flagged errors concentrating in very short spans while human evaluation forms over a longer horizon. One failure at two temporal scales: a correct local detection reversed by a salience-weighted human decision.

Over-penalization fed into reward modeling or model selection becomes directional pressure toward enumeration: the trained policy learns to list rather than communicate. Judge false positives inside a training loop are not symmetric noise.

The remedy is explicit weighting, because salience is not visibility: primary objects, causal details, and narrative events matter differently from incidental and transient ones. Formally, replace an unweighted mismatch count with a salience-weighted error function: if E is the set of detected mismatches and each e carries weight w(e) reflecting its effect on task meaning, the penalty is Σ w(e) rather than |E|. Weights should be sparse and task-dependent: the same omission can be immaterial in scene description and disqualifying in safety inspection. w cannot be inferred from the image; it encodes what the output was for, which lives in the task specification and calibration set. That is why LLM-as-a-judge rubric design is better understood as specifying a weighting function than writing instructions.

From isolated frames to temporal evidence

Perception asks what is visible. Temporal reasoning asks what changed, in what order, why, and which changes matter.

The mechanisms are separable and fail asymmetrically. Sparse frame sampling drops events outright, detectable as missing evidence; temporal aliasing keeps the frames and inverts the apparent dynamics, which is not detectable, because the judge reports high confidence over a coherent but wrong sequence. State transitions, object permanence, and identity across occlusion fail on persistence rather than perception. Event segmentation fixes the unit of evaluation before scoring begins; causal-chain reasoning must link evidence minutes apart inside a token budget standing in for thousands of frames.

Retrieval precedes reasoning. T* formulates temporal search as a Long Video Haystack problem: finding the minimal set of relevant frames, typically one to five, among tens of thousands, and introduces LV-Haystack, a benchmark of 3,874 human-annotated instances with fine-grained metrics for keyframe search quality and efficiency. The 2.1% temporal F1 figure - what prior state-of-the-art selection methods achieved on its LVBench subset - is a statement about S, not about judge reasoning. The gap is not static: TimeSearch-R trains the search policy end to end with self-verification reinforcement learning and more than triples the previous best temporal F1 on LV-Haystack, from 2.5 to 8.1. The direction of travel is right and the absolute number is still low. If the deciding frames are never retrieved, the judge performs well-formed reasoning over the wrong evidence.

Towards Neuro-Symbolic Video Understanding (ECCV 2024) argues existing approaches fail because they “intertwine per-frame perception and temporal reasoning into a single deep network,” and instead pairs vision-language models for per-frame semantics with state machines and temporal logic for long-term event evolution, reporting F1 gains of 9–15% on complex event identification over GPT-4-based baselines. TemporalVLM and TimeSuite pursue the same decoupling through timestamp-aware clip encoding and grounded tuning. The direction of travel is multi-scale temporal grounding, which aligns with what video action and intent recognition annotation must supply: event boundaries, state transitions, and timestamps, not only frame labels.

Why short-span competence does not equal long-horizon understanding

Two results complicate treating temporal failure as a context-length problem.

Vinoground evaluates 1,000 natural video-caption pairs designed as temporal counterfactuals and finds that existing large multimodal models “severely struggle to distinguish temporal differences between different actions and object transformations.” Roughly 50% on text and video scores for the strongest model originally tested, GPT-4o, against a human baseline near 90% - on short videos, so a larger context window does not address this failure class. The benchmark's 2025 update re-runs the suite against newer systems and reports the strongest reasoning model tested, o3, still near 50% on the group metric. Two model generations did not close the gap.

Temporal Reasoning Transfer from Text to Video locates the constraint. Its diagnostic study finds video representations already contain enough information for small probing classifiers to reach near-perfect accuracy, concluding that the primary bottleneck sits in the language model's handling of temporal concepts rather than in the visual features. Its method improves TempCompass accuracy by 5.3 points for a 7B model using only synthesized text data. Seeing the Arrow of Time in Large Multimodal Models (NeurIPS 2025) points the same way on temporal directionality, and Tracing the Arrow of Time (2026) diagnoses where directional information is lost inside the video-LLM stack.

If some long-video benchmarks are solvable from very few frames, as the Neptune authors observe, validating a judge on them yields an agreement estimate that will not hold on genuinely temporal content. TempCompass attacks this from the construction side, building conflicting videos that share static content but differ in one temporal aspect, blocking single-frame and language-prior shortcuts. Video-MME-v2 (2026) pushes construction further, grouping related queries and scoring non-linearly so that fragmented or guess-based correctness earns no credit. Judge validation sets need these properties, or measured temporal competence is partly an artifact of frame-solvability; benchmarks built for long interleaved-context referring reasoning, such as LongVideoBench, validate better. The same requirement now applies to purpose-built judge-validation suites: VL-RewardBench and Multimodal RewardBench measure multimodal judge agreement with human preference, and recent work trains self-improving VLM judges against them without human annotation, but neither suite stratifies items by temporal horizon, so agreement measured there does not certify temporal competence.

Local errors, global meaning, and human override

Two examples, constructed for exposition, not drawn from data.

Example A. A person repairs a bicycle beside a red toolbox; the caption reads “A person repairs a bicycle outdoors.” The judge flags the omitted toolbox correctly; a human still rates the caption adequate. Nothing perceptual is in dispute - the disagreement resolves on whether r specifies exhaustive coverage or primary-scene description. Where r is silent, the judge defaults to coverage, the only term it can compute.

Example B. A long instructional video contains a brief moment where a tool is set on the wrong surface; the candidate description gets procedure, order, and outcome right but omits it. A clip-level judge surfaces it with high confidence and propagates it into a low score, while a human scoring procedural adequacy end to end discards it. The inversion is the dangerous case: if the misplaced tool contaminates a later step, the identical omission becomes material. Only sequence-level reasoning distinguishes them, so a judge tuned to pass Example B by discounting local detail fails its inverse.

The human role follows - not scoring every item, but constructing rubrics and typing fields as required or optional; building calibration sets that include deliberately frame-unsolvable cases; adjudicating local-global conflicts; analyzing false positives; assigning severity weights and thresholds; and updating policy when content drifts. Humans have their own limits - selective attention, fatigue over long sequences - so the design question is which combination of retrieval, architecture, rubric, and escalation is reliable. Elicitation format matters: the ICML 2024 MLLM-as-a-Judge study finds multimodal judges align with human preference substantially better in pairwise comparison than in absolute scoring or batch ranking - an argument for pairwise preference at the decision boundary, and absolute scores only where criteria are crisply verifiable. Model evaluation and integrity work operationalizes these steps.

Designing a multi-resolution evaluation stack

A defensible pipeline separates evidence gathering from judgment, and judgment from aggregation.

• Segment and sample at multiple temporal resolutions rather than one fixed stride.
• Run localized perceptual checks on frames and short clips.
• Extract structured entities, attributes, actions, state changes, and timestamps.
• Retrieve evidence relevant to the specific criterion being scored.
• Build a global sequence representation or event graph over that structure.
• Score separate dimensions rather than emitting one holistic score.
• Apply salience and severity weights to detected mismatches.
• Gate on confidence, and detect disagreement across resolutions or judge samples.
• Route ambiguous, high-impact, and local-global-conflict cases to human adjudication.
• Measure judge-human agreement continuously by dimension and failure category.

Frame-level checks catch attribute and object errors; clip-level, short-duration action errors; event-level, boundary, ordering, and state-transition errors; full-sequence, narrative and causal errors. A single-resolution judge is structurally blind to every class above its receptive field.

Aggregation must not be a mean: a mean makes one material error and twenty immaterial ones interchangeable, in the direction that hides failures worth catching. Severity-weighted aggregation with a hard floor for material errors is the minimum. Cross-resolution disagreement is itself signal: frame-level checks passing while sequence-level judgment fails, or the reverse, is a reliable escalation trigger. Sustaining this as content drifts is where a multilingual LLM-as-a-judge service applies.

Metrics for diagnosing judge-human disagreement

Aggregate agreement is uninformative; useful measurement is decomposed and conditional. Report precision and recall for judge-identified errors separately, then false-positive rate by error taxonomy category, since overall precision hides miscalibration in one. Track human override rate with direction: overrides that raise a score indicate over-penalization and point at rubric weighting; overrides that lower one indicate missed material errors, so retrieval or perception. Add severity-weighted agreement and local-global disagreement rate. Then stratify by video duration, state transition count, temporal distance between the events an item depends on, and rubric dimension - the axes along which a receptive field binds and aggregates hide degradation.

The statistics differ in how they fail, which is the only basis for choosing among them. Cohen's kappa inherits prevalence and marginal sensitivity, so skewed severity distributions produce the kappa paradox - high raw agreement, near-zero kappa - and it discards ordinal distance, scoring one-level and three-level severity disagreements identically. Krippendorff's alpha admits multiple raters, missing data, and ordinal scales, but forces an explicit distance function - where severity weighting stops being implicit and becomes auditable. Spearman's rho is invariant to calibration offsets and therefore blind to a judge that ranks correctly while scoring uniformly low; under long-tailed severity it is dominated by easy items. Kendall's tau operates on concordant and discordant pairs, matching pairwise elicitation, though tie treatment must be chosen deliberately where judges tie often. Calibration curves and expected calibration error are prerequisites for confidence gating and say nothing about which errors matter. Confusion matrices are what separate a uniform threshold shift, which recalibration fixes, from category confusion, which it does not.

One discipline underlies this: benchmark judge-human agreement against human-human agreement on the same items, not a notional ceiling of 1.0. If inter-annotator agreement is 0.62, a judge at 0.60 is near the achievable ceiling and further tuning is the wrong investment - the rubric is the ambiguous artifact. Reporting both makes the other numbers interpretable, a standing requirement for LLM evaluation benchmarks informing model selection.

Implications for VLM training and evaluation teams

The highest-leverage work is usually the evidence stage and the rubric, not the judge model: retrieval and weighting failures are invisible to judge-side scaling, and a stronger judge over the same S and r reproduces the same disagreements more confidently. The tolerable false-positive rate is a function of downstream use, not a property of the judge. Salience and severity are labeling targets in their own right, which makes materiality a data problem before a modeling one: a calibration set recording only what is visible cannot supply what mattered.

Conclusion

The tension is not that machines see while humans understand. It is that J and H take different arguments: J receives a compressed encoding, a rubric approximating uniform weighting, and no representation of what the output was for, while H receives the full sequence and a model of communicative purpose. Agreement between them is not one capability to improve but a composition of a retrieval operator, a temporal receptive field, a weighting function, and a threshold, each failing distinguishably and admitting a different fix.

The published evidence supports specific claims, not a general one: temporal reasoning fails on short videos as well as long, and held across two model generations; the bottleneck is partly in the language backbone; keyframe retrieval scores remain low even after a threefold improvement; some long-video benchmarks are frame-solvable and therefore poor validators, and current judge-validation suites do not stratify by temporal horizon. Each implies a different intervention, none addressed by a larger judge model alone. What remains open, and answerable with a team's own data, is where local machine sensitivity improves quality and where it distorts global judgment - an empirical, task-specific, measurable boundary.

Research references

• Ye, J., Wang, Z., Sun, H., Chandrasegaran, K., Durante, Z., Eyzaguirre, C., Bisk, Y., Niebles, J. C., Adeli, E., Fei-Fei, L., Wu, J., & Li, M. T*: Re-thinking Temporal Search for Long-Form Video Understanding. CVPR 2025. arXiv:2504.02259
• TimeSearch-R: Adaptive Temporal Search for Long-Form Video Understanding via Self-Verification Reinforcement Learning. 2025. arXiv:2511.05489
• Zhang, J., Cai, M., & Lee, Y. J. Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos.OpenReview · arXiv:2410.02763 · 2025 updated version
• Nagrani, A., Zhang, M., et al. Neptune: The Long Orbit to Benchmarking Long Video Understanding.OpenReview · arXiv:2412.09582
• Temporal Reasoning Transfer from Text to Video (T3). ICLR 2025. OpenReview · arXiv:2410.06166
• Xue, Z., Luo, M., & Grauman, K. Seeing the Arrow of Time in Large Multimodal Models. NeurIPS 2025. OpenReview · arXiv:2506.03340
• Tracing the Arrow of Time: Diagnosing Temporal Information Flow in Video-LLMs. 2026. arXiv:2605.07568
• Choi, M., Goel, H., Omama, M., Yang, Y., Shah, S., & Chinchali, S. Towards Neuro-Symbolic Video Understanding. ECCV 2024. arXiv:2403.11021
• TemporalVLM: Video LLMs for Temporal Reasoning in Long Videos.arXiv:2412.02930
• Maeda, K., Kurita, S., Miyanishi, T., & Okazaki, N. Vision Language Model-based Caption Evaluation Method Leveraging Visual Context Extraction (VisCE²).arXiv:2402.17969
• TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning. ICLR 2025. OpenReview · arXiv:2410.19702
• FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning (Five-in-One Video Annotations). OpenReview · arXiv:2410.15270
• Liu, Y., Li, S., et al. TempCompass: Do Video LLMs Really Understand Videos? Findings of ACL 2024. ACL Anthology · arXiv:2403.00476
• Wu, H., et al. LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding. NeurIPS 2024 Datasets & Benchmarks. arXiv:2407.15754
• Chen, D., Chen, R., et al. MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark. ICML 2024. arXiv:2402.04788
• Video-MME-v2: Towards the Next Stage in Benchmarks for Comprehensive Video Understanding. 2026. arXiv:2604.05015
• Self-Improving VLM Judges Without Human Annotations. 2025. arXiv:2512.05145 (evaluates on VL-RewardBench and Multimodal RewardBench)

‍