The Document Substrate: Solving Garbage In, Garbage Out for AI on Business Documents
Most document AI fails the same way. Someone connects an LLM to a folder of PDFs and asks it a question about a client's bank statement. It either hallucinates the answer or refuses to answer, because the OCR produced garbled text. The fault is in the input, not the model. Garbage in, garbage out, and most document pipelines produce garbage.
Update, September 2026. The argument here still holds, and the details that no longer matched the system have been corrected or removed. The Document Substrate, Revisited describes how it works now, what changed, and why.
Businesses still store and share most of their information in unstructured documents. Contracts, invoices and reports mix layout, language and context, and no other system can read them directly.
Document intelligence is the discipline of turning those documents into structured, machine-readable data. It is what makes the rest of the AI stack useful to a business. Without reliable parsing, classification, splitting and extraction, every retrieval pipeline, agent and analytics dashboard works from unreliable data.
Building useful AI on real business documents requires solving four problems that are easy to overlook. This post covers those problems, the pipeline that solves them, and what can be built once they are solved.
Why Putting the PDF Into a Prompt Fails
The naive approach is to extract text from a PDF and put it into a prompt. It works in demos, on clean documents, on documents where the answer is the first thing the model finds. In production, it fails in ways that quickly lose users' trust:
OCR quality is very uneven. A scanned bank statement that looks clean to a person might have columns misaligned, decimal points dropped, or handwritten annotations merged into the surrounding text. The model reads what the parser gives it. If the parse is wrong, the answer is based on wrong text.
Tables are lost in plain text extraction. A statement with five columns and two hundred rows becomes a run of numbers with nothing to show which column each belongs to. The model can't tell a withdrawal from a deposit. The structure that was clear on the page is gone.
Most business documents contain personal details, and they cannot be sent to any LLM. Invoices, tax returns, bank statements and legal records are the documents most worth building AI on, and they contain the most personal information. Sending raw client documents to a third-party model is a privacy and compliance problem, not just a technical one. For most professional practices, that rules it out.
Unstructured text retrieval is unreliable for precise questions. Semantic search finds roughly relevant chunks. But "what was the total GST on supplier invoices in Q2?" is not a semantic question. It is an aggregation over structured data that was never extracted. The model reads prose when it should query a table.
These four problems compound. A bad parse produces bad text. Bad text produces bad embeddings. Bad embeddings produce irrelevant retrieval. The model, however capable, can only work with what retrieval gives it.
The Document Substrate solves all four before the model is involved.
Four Steps and One Rule
Every uploaded document is parsed, redacted, indexed for search, and read into structured fields. Those four pieces of work are what the rest of this article is about. The order they run in, and which of them share a pipeline, has changed since: extraction became a flow of its own, and a classification step was added.
One rule applies to all of it: no model reasoning over a client's documents is given real personal details, and every restore of a real value is scoped to particular documents and written to the audit log. Page text after redaction, chunks, embeddings, structured extracts, prompts and model responses contain only opaque placeholders. Real values are decrypted server-side, briefly, for an authorised user.
This rule is what makes the rest safe.
Parse
The first problem is getting accurate text out of the document. This is not flat text extraction. It is structured, page-aware OCR that handles layouts, tables, multi-column designs, and mixed text and figures.
A dedicated parsing service handles this, with quality tiers configurable per document type, so you only pay for the precision a document needs. Which tier suits which document is a question to settle by measuring the output against real files rather than by assuming the most expensive tier is the best one.
The output is stored per page, so every citation can name the page a figure came from. Everything downstream depends on parse quality.
Redact
This step determines what you can build.
Redaction has three stages:
-
Pattern rules catch values with a fixed shape or a checksum. These are the values that matter most, and pattern matching finds them reliably without a model.
-
A model pass catches what patterns cannot: people's names, organisations, physical addresses. It runs on every page rather than as a fallback, because context is the only thing that identifies a name. Its output is a list of entities to redact, not free-form text that could introduce new risk.
-
A final check runs an independent PII detector over the final redacted text. It is treated as a warning, not a blocker, because its false-positive rate is too high to fail on. It reports anything the first two stages missed.
Each detected value gets a stable placeholder, scoped to the document it came from, so one placeholder can never stand for two different people. The same entity mentioned twice in one document gets the same placeholder, so the model can reason about relationships between occurrences without knowing who or what they are. The real value is encrypted at rest into the entity vault. Nothing downstream ever sees it.
The query path has a matching check: before any model call, the prompt is scanned for personal details, and what it finds is removed and recorded. Redaction is measured against a test suite. It is a mechanism with limits, not a guarantee.
Extract
Parse and redact give you clean, safe text. Extract turns that text into structured data.
When a document's type is known (bank_statement, utility_bill, invoice, tax_return), the Document Substrate runs the redacted text against a schema for that type. The schema defines what to pull out, what to call it, and how to handle fields that can't be found (return null, never guess). The result is stored as typed JSON. A classification step now proposes the type on upload, and a document with no confirmed type is still read against a generic schema.
Extraction has since moved out of the pipeline into a flow of its own, so a failed extraction cannot leave a document unsearchable. Every extracted value is also traced back to the page it was read from.
There are two extraction engines:
Text extraction works from the redacted text directly, with the schema enforced on the output. Best for flat, single-value data such as invoice totals, utility bill amounts and policy numbers. It is fast, cheap and accurate on fields that are intact in the text.
Layout-aware extraction is better for layout-heavy documents: bank statement transaction tables, multi-column forms, anything where the column boundaries and row relationships matter as much as the values. Where it needs the original document layout, the extracted output is post-redacted against the vault before it's stored, so no personal details are kept in it.
The schema is what makes extraction reliable. Without it, you're asking a model to guess what matters. With it, you get the same fields, in the same format, every time, validated before storage.
Chunk and Embed
The final stage turns the redacted text into a searchable index. Page-aware chunking splits the text into overlapping chunks, each recording the page it came from, so a citation can open the document at the page a figure was read from. Each chunk is embedded and stored with its vector.
Embeddings are computed from the redacted text, never the original. The retrieval layer only ever returns placeholder-form content. The rule has no exception for retrieval.
Why Redaction Makes More Possible
Redaction sounds like a constraint. In practice it is what allows a firm to use a capable hosted model.
Without it, you can't send sensitive client documents to a modern cloud LLM at all. The alternatives are a smaller model run on-premises, which is less capable, or data processing agreements that most firms don't have the resources to maintain.
With it, the model only ever sees opaque placeholders. It doesn't know what those values are. It can still reason about relationships between them, such as "these two mentions are the same person" or "this account number is referenced three times", but it never handles the real values. You can use any capable model, hosted anywhere, without the compliance exposure of sending real client data offsite.
Those relationships hold within a document. Recognising the same person across two documents is a separate step, proposed by the system and confirmed by a person, because placeholders are scoped to the document they came from and a wrong match would show one person's data under another person's name.
Real values are restored exactly once: in the hydration step, server-side, after the model has finished reasoning, just before the response reaches the authenticated user. The model does the reasoning, and hydration puts the real names and numbers back for the reader. The audit log records every hydration: who, when, which documents, which placeholders were substituted.
The result can be explained to a client in one sentence: the AI never saw your name, your account number, or your tax ID.
Why Schemas Are Needed
It is tempting to treat document AI as a pure retrieval problem: chunk the text, embed it, retrieve the most relevant chunks, let the model answer. This works for some questions. It fails consistently for others.
"What were the total withdrawals from this account in March?" is not a retrieval question. It's an aggregation over structured data. If the bank statement's transactions were extracted into typed JSON at ingest time, that question is answered from the extracted fields, with code adding the numbers. It's fast, repeatable, and correct. If the transactions weren't extracted, the model has to infer totals from retrieved chunks. It will do so convincingly and sometimes incorrectly.
The same applies to every document with repeating structured content: invoice line items, contract clauses, tax return schedules. Schemas defined up front determine whether the Document Substrate can answer precise questions reliably or only approximately.
Schemas are also the mechanism for adding document types to the Document Substrate. A schema bundles four things: the document type name that triggers it, the output shape, an instruction to the extractor on what to pull and how to handle missing fields, and an explicit choice of extraction engine. Adding support for a new document type was originally one file and one registry entry, which meant a deploy. That still works, and types and rules can now also be published as versioned data, without one.
What You Can Build On Top
The Document Substrate is not an application. It is a foundation: the document-handling layer that project-specific AI code builds on. A firm's written substrate, its captured expertise, needs software to apply it to real work, and this is that software. The API exposes search, hydrate, extract retrieval, and audit; project-specific logic handles the chat UI, reporting, agents, and domain workflows.
A few concrete applications for professional practices:
Accounting and reconciliation. Upload a client's bank statements and supplier invoices. Extraction produces structured transaction data (dates, amounts, references, categorisations), ready to feed reconciliation without manual rekeying. Query answers questions like "what was the total GST on supplier invoices in Q2?" from the extracted data, not from retrieved text.
Financial advice and planning. A financial planner working across a client's tax returns, KiwiSaver statements, and investment reports can ask cross-document questions and get answers with citations, without any of the client's PII leaving the application boundary or reaching a model that hasn't been explicitly cleared to receive it.
Legal document review. For contracts, lease agreements and regulatory submissions, the Document Substrate parses, redacts, extracts key clauses and dates into schema-defined fields, and makes the document corpus queryable. The lawyer asks, the model answers from redacted content, and real values are restored before the response reaches the screen.
Regulatory compliance. A compliance team managing policy documents, audit reports, and regulatory correspondence can use the Document Substrate to track obligations across a document corpus. Schemas extract due dates, obligation owners, and regulatory references into structured fields; queries show what's covered, what's pending, and what's changed since the last review. Because every query and hydration is written to an immutable audit log, the system keeps its own record of evidence, which is useful when a regulator asks how a decision was reached.
Multi-client separation. Each client has their own document group, with queries scoped to that group by default and cross-client queries gated explicitly. An accounting practice with fifty clients needs one install with fifty groups, not fifty installs. The groups are isolated from each other in the database, not by application code.
Onboarding and triage. Upload a batch of mixed documents from a new client. Parse and extraction classify and structure them automatically. A query over the collection shows what's present, what's missing, and what needs follow-up. This is faster than manual review, and it leaves a searchable record of what was found.
Workflows Fit the Document Substrate to Your Firm
Domain workflows are the main part built on top, and they are how the Document Substrate is fitted to a firm. A workflow defines a job or engagement. It has three parts.
- Slots. The documents the job needs, each named. For a rental property: the bank statement, rent summary, loan statement, rates bill and invoices.
- Rules. Checks that run in code over the figures read from those documents: do they agree, is a month missing, is anything counted twice. No model takes part, so the same documents always get the same verdict.
- Outputs. A report on screen and a workpaper in Excel or Word, stamped DRAFT until every slot is filled and every rule passes.
The Document Substrate is the same for every firm. The workflows, and the practice manual that explains each rule, are what change from one firm or industry to the next. The Document Substrate, Revisited describes how workflows are defined, run and reviewed.
What Makes Document AI Dependable
The difference between a demo and a system a professional would rely on for signed work is almost always in the input. The model is capable. The documents are the problem.
Four things make document AI dependable: accurate parsing, PII redaction that lets you use any LLM safely, schema-driven extraction that turns documents into structured data, and a search index built on the redacted text.
The Document Substrate does all four. The project-specific work (the chat UI, the reporting dashboards, the domain workflows) is built on top of it.
The Document Substrate, Revisited · Why "substrate"? · Worked examples across six professions · Map → Architect → Build, in practice