01
Clean recognition is a starting point
In the historical v0.1 run, Google Document AI reached 98.29% clean token F1 and Amazon OCR reached 98.05%. Those values describe the mean performance of the clean cohort.
The corresponding severe synthetic cohort means were 69.24% and 72.73%. The cohorts are unpaired and uneven, so these figures are a controlled stress signal rather than a matched per-document causal estimate.
02
Aggregation hides the failure surface
A strong average can coexist with failures concentrated in a small set of capture conditions. Report cohort size, empty responses and condition membership so a reader can distinguish a broad weakness from a small sparse subgroup.
A model that loses a few amounts may remain strong on ordinary words. Numeric-token and date-text measures add useful task-adjacent views, but do not replace a field-level extraction evaluation.
03
Use the benchmark to choose the next question
Investigate the weakest conditions, inspect failed inputs and check whether the synthetic effect resembles the failure of interest. Then design a scoped holdout that can answer the production question.
Do not map the synthetic severity distribution directly onto expected customer incident rates. That requires an independently measured production distribution.