Non-Determinism and Calculation Validation: The Short Version

How the Document Substrate keeps the AI model away from the numbers. The full article has the detail, the measured results and a glossary.

---

Every number in a report, workpaper or answer is one of two things:

• a figure stated in a source document, or
• a figure computed by ordinary code from those source figures.

There is no third option. The model never produces a number anyone relies on.

What the Model Does and What Code Does

The model reads and suggests. Code checks and decides. A person signs.

Token-Prediction Drift: Three Problems

1. Context accumulation. Long conversations dilute early instructions. The part of the system that produces numbers has no conversation: a workflow is a single pass over stored fields. Chat is bounded, but only the excerpts have a hard size limit.
2. Error cascade. An early mistake becomes context for everything after it. The model does no arithmetic, each document is read in its own call, and completeness checks read stored fields, never the model's prose.
3. Surface-form sensitivity. Small input differences, such as spaces or line breaks, change the tokens and the output. This is measured, not prevented: extraction varies between runs even at temperature zero. A parse-fidelity test checks that figures and table structure survive in both the digital and scanned versions of the same page.

Drift is kept to places where it is visible and can be checked, and out of the arithmetic.

1. Separating Arithmetic From Reasoning

The assistant is told: "Report figures as the documents state them. Do not calculate."

A prompt alone is weak. Three controls enforce it:

• A calculation tool written in code. When a total or average is needed, the model calls a tool. The tool does the arithmetic and returns every row that contributed. Currencies are never added together. A missing currency is treated as the home currency and the answer says so. An unreadable value cancels the total instead of being skipped.
• A completeness engine with no model in it. Whether a client file is complete is decided in code over stored fields.
• A check on every answer for figures that appear in none of the source excerpts. Deterministic failures can block. Probabilistic signals only flag.

Rules are declared as data (which fields they read, what kind of check they perform), not generated code. A failure reports its own inputs, and renaming a field breaks the build instead of silently breaking a check.

2. What Goes In Decides What Comes Out

1. Classification with a confidence gate. A document type is applied automatically only above 0.9 confidence and only if no one has set it. A wrong type is the most common cause of a wrong report.
2. A declared field list per type, with a description of each field. Shape is validated automatically.
3. Arithmetic checks inside each document, run at extraction: for example, transactions must foot to the closing balance. New checks start as warnings until their false-positive rate on real documents is known.
4. Tracing. Code searches the page for every extracted value. Statuses: Grounded, Check, Derived, Reformatted, Not checkable. A single, plain figure that says Check is the one to look at first.
5. Corrections stored beside the extraction, never over it. Quality measurement keeps measuring the original extraction.

Workbooks are generated by code, not by a model writing formulas into a spreadsheet. A second pass then checks:

• that the workbook agrees with itself: lines foot, GST adds up;
• that it agrees with the source documents: every figure is re-derived from the extracted fields.

A figure that comes from neither a source document nor arithmetic over source figures is reported as a finding.

3. Personal Details

Redact, retrieve, prompt, then restore the real values only for an authorised person.

• Pattern rules for identifiers with a fixed format: IRD numbers, bank accounts, NZBN.
• A model pass for names, organisations and addresses.
• A check afterwards for anything missed.

Real values are held in an encrypted vault, and everything downstream sees only placeholders. Real values are restored after the model has answered, limited to the documents in play and recorded in an audit log. A placeholder that cannot be matched cleanly becomes a neutral label instead of a guess.

Zero-data-retention contracts do not replace redaction. Redaction stops personal details reaching the model. A contract only governs what happens after they arrive.

4. Verification and Audit

• Schema validation is necessary but not sufficient. It cannot catch a wrong number in the right field.
• A check blocks only after its false-positive rate has been measured on real documents.
• Findings are stored records with two states. Accept: a known exception, carried forward. Resolve: fixed, not carried forward.
• Run history stores a snapshot and is never re-run.
• Every workflow states on its own definition what a pass does and does not establish.

Quality measurement keeps two groups apart:

• Gates: deterministic, must stay at 100 per cent.
• Trends: scored by a model, read for direction only, never used to gate a release.

What Has Gone Wrong

• A bank statement failed footing by exactly 230.00 because the model took an opening balance from a filtered export that did not state one.
• A chat total left out two receipts that printed no currency.
• A test suite scored empty files for ten days.
• The test harness and production processed answers separately and had diverged.
• Rules written against invented sample documents failed on the first real ones.

What This Does Not Claim

• The model is not deterministic. It is removed from the steps that produce numbers.
• Extraction still varies between runs.
• Nothing decides whether a figure is correct, only that it is stated, consistent and traceable.
• Everything depends on correct document typing.

The Pattern

1. Let the model read, classify and draft.
2. Put all arithmetic, completeness decisions and numeric checks in code, over fields traced to the page.
3. Handle personal details with redaction, a vault and restoring on the server, plus a processing agreement for what remains.
4. Treat prompts as guidance. Enforce constraints with schemas, code, measured thresholds and workflows that state what a pass means.