01
The failure surface
A multimodal model can produce a fluent answer even when the source is difficult to read. Recognition, extraction and reasoning should be measured as separate tasks with explicit truth and scoring rules.
02
What to measure
Freeze prompts, model versions, decoding settings and output schemas. Compare clean and degraded cohorts while retaining raw responses, parse failures and unsupported answers for review.
03
A scoped evaluation
Use deterministic document variants and a source-group holdout. Agree task-specific labels for extraction or question answering before execution; an OCR token score alone is insufficient for those tasks.
04
What the evidence supports
A VLM engagement requires its own task definition, baseline and acceptance criteria.