STRESSBENCHAIMenu
Research / DocTorture-1K benchmark v0.1
Clean accuracy hides
production failures.
1,000 identical synthetic documents. Recognition and reading-order metrics reported separately, with durable evidence behind every accepted sample.
A baseline is only
part of the picture.
Google Document AI moves from 98.29% clean token F1 to 69.24% in the severe synthetic cohort. Amazon OCR moves from 98.05% to 72.73%.
These are unpaired cohort means. They reveal a synthetic stress response; they do not estimate how often a production system will fail.
Measured on DocTorture-1K v0.1 (1,000 documents), separate from the 10K v1.0 dataset. Unpaired synthetic cohort means. Clean n=250, subtle n=182, moderate n=147, severe n=421.
01 / Recognition quality · higher is better
How much text was recovered?
Arithmetic means of per-document metrics across all 1,000 samples. Token F1 is order-independent. Numeric and date metrics use the retained exact normalized token-recall definitions, not field-level extraction accuracy.
| Engine | Token F1 | Numeric-token accuracy | Date-text accuracy | Empty responses |
|---|---|---|---|---|
| Google Document AI | 82.54% | 78.51% | 85.60% | 27 / 1,000 |
| Amazon OCR | 84.69% | 82.45% | 87.10% | 10 / 1,000 |
02 / Serialization-sensitive quality · lower is better
How closely did sequence order match?
CER and WER compare the provider serialization with the frozen truth sequence. They remain useful diagnostics, but are not combined with recognition into one score.
| Engine | CER | WER | Exact tokens, sequence errors* |
|---|---|---|---|
| Google Document AI | 41.54% | 43.72% | 165 |
| Amazon OCR | 13.91% | 18.77% | 1 |
*Samples with exact token recognition but sequence errors. CER/WER alone do not block an otherwise strong recognition result.
03 / Severity cohorts
Keep the curve and its sample counts together.
| Engine / cohort | Samples | Token F1 | Numeric | Date text | CER | WER |
|---|---|---|---|---|---|---|
| Google Document AI / clean | 250 | 98.29% | 95.97% | 100.00% | 31.13% | 29.51% |
| Google Document AI / subtle | 182 | 90.97% | 88.35% | 93.68% | 34.94% | 36.32% |
| Google Document AI / moderate | 147 | 83.42% | 78.17% | 86.39% | 41.07% | 44.37% |
| Google Document AI / severe | 421 | 69.24% | 64.00% | 73.28% | 50.73% | 55.12% |
| Amazon OCR / clean | 250 | 98.05% | 96.02% | 100.00% | 1.85% | 3.22% |
| Amazon OCR / subtle | 182 | 92.74% | 90.69% | 96.15% | 5.64% | 9.41% |
| Amazon OCR / moderate | 147 | 86.25% | 83.43% | 88.44% | 11.72% | 17.87% |
| Amazon OCR / severe | 421 | 72.73% | 70.48% | 75.06% | 25.42% | 32.37% |
04 / Descriptive robustness retention
Severe recognition relative to clean.
Severe mean token F1 divided by clean mean token F1. This ratio compares unpaired synthetic cohort means; it is not training uplift.
| Engine | Severe / clean retention |
|---|---|
| Google Document AI | 70.44% |
| Amazon OCR | 74.18% |
05 / Condition-level diagnostics
Locate the failure surface.
Selected historical condition memberships. Mixed-condition documents appear in multiple rows; means do not isolate the causal effect of a single condition. Historical recipes are not claims of physical calibration.
| Historical condition | Samples | Google token F1 | Amazon token F1 | Google below 85% | Amazon below 85% |
|---|---|---|---|---|---|
| motion blur | 42 | 2.98% | 7.23% | 42 | 42 |
| blur | 12 | 64.81% | 69.60% | 5 | 4 |
| jpeg | 75 | 52.38% | 55.31% | 46 | 44 |
| perspective | 100 | 84.14% | 86.13% | 32 | 27 |
| rotation skew | 45 | 65.75% | 78.26% | 33 | 18 |
| page cropping | 70 | 84.65% | 85.81% | 41 | 34 |
| contrast loss | 11 | 98.32% | 98.10% | 0 | 0 |
| color cast | 17 | 96.42% | 96.63% | 1 | 1 |
85% token F1 is a diagnostic reference to the clean admission floor, not a rejection gate for degraded samples.
Definitions and limitations.
Numeric-token and date-text accuracy are exact normalized token recall. Extra predicted tokens are not penalized by these two fields; an empty truth-token set scores 1. They must not be interpreted as extraction precision.
The retained normalization is ASCII-oriented. This is not a multilingual evaluation. Source documents are synthetic invoices, not an estimate of any customer’s document distribution.
Open-source reference
Tesseract is retained as an open-source/legacy reference: overall token F1 50.31%, numeric-token accuracy 45.16%, date-text accuracy 63.55%, CER 40.11%, WER 71.76%, 1,000 samples.
Trace the source.
- Source leaderboard SHA256
- 550d0b712e556170832c40ab17c9a34a00cecdb2d7e57e029f1093b52dff8527
- Immutable v0.1 checksum-index digest
- 1f22aa5800ef2eb7327dcdaa9036b43481e7d27436de66d1225cbc90442e6ad1
- Frozen truth sidecar SHA256
- c794787c0f7fdee894eb96bc5fe630e0928da8fcc92b8badf409af0934df9996
The public JSON contains the displayed aggregates and their source identifiers. It does not expose private raw provider responses or licensed dataset archives.
Download aggregate evidence JSON ↓Build with evidence
See what breaks
before production does.
Run StressBench against your document model and get a condition-level robustness readout.