Engineering notebook

Failure-conditioned dataset generation

Turn a failure map into an explicit, reproducible generation specification.

01

Diagnose before choosing weights

A low condition mean is a starting point for investigation. Inspect the failed input and raw response, check sample count and mixture membership, and distinguish recognition loss from parsing or serialization behavior.

Avoid over-weighting a condition simply because it appears frequently in a synthetic evaluation recipe. Its relevance must come from the deployment question and observed failure examples.

02

Specify the recipe

A generation specification records document classes, source groups, conditions, severity parameters, mixture order, random seed and renderer version. Labels and image checksums travel with the resulting corpus.

Mixture order matters: cropping then perspective is not necessarily equivalent to perspective then cropping. Treat the ordered recipe as part of the sample identity.

03

Close the loop with a holdout

Generate the proposed training cohort separately from the frozen test sources. After training, rerun the evaluation with the declared model version and scoring rules.

If the evaluation set or policy changes, publish a new version and explain the change.

Build with evidence

See what breaks
before production does.

Run StressBench against your document model and get a condition-level robustness readout.