The Document Substrate, Revisited
The first article described a four-step pipeline — parse, redact, extract, embed — and one hard rule: only the original file and the encrypted vault ever hold real personal details. The rule held. The pipeline did not. Extraction is no longer a step in it. Every extracted figure is now traced back to the page it came from. Retrieval is no longer a single vector search. And the most useful thing built on top turned out not to be chat at all, but a completeness check that no model takes part in.
How It Works
Two flows run over a document, and a third path answers questions about it.
The ingest flow runs when a document arrives. The file is read — pages, tables and columns reconstructed — and then redacted: personal details are replaced with placeholders, and the real values are encrypted into a vault. From there the document is made searchable. A type is suggested, the text is indexed, and a short summary is written.
The extract flow runs after that, and separately. It reads the document's figures into typed fields, traces each one to the page it came from, and checks the arithmetic.
The query path runs on every question. The names in the question become placeholders, the passages that matter are found, the model answers from redacted excerpts, and real values are restored on the server at the end.
Redaction is the second step of the first flow, and the design depends on that order. Everything after it — the search index, the extracted fields, the prompts and the answers — holds placeholders rather than real values.
What Real Documents Changed
| The first article said | What is true now |
|---|---|
| Extract is step 3 of one pipeline | Extraction is a separate flow with its own status. A failed extraction no longer leaves a document unsearchable |
| A person sets the document type before extraction | A classification step proposes a type on upload and applies it on its own when it is confident enough |
| A placeholder identifies an entity | Placeholders are scoped to a single document, so one string can never stand for two different people |
| Extracted values are validated against a schema | They are also traced to the page they came from, checked arithmetically, and correctable by a reviewer |
| Retrieval is a vector search | Several complementary searches run over the redacted text and are combined by rank |
| A pre-flight check fails the query if it finds personal details in a prompt | The check removes what it finds, records it, and the turn proceeds |
| A router decides between retrieval and structured data | The router is gone. The model asks for what it needs, and totals are added by code |
| Document types and rules live in code | They can also be published as versioned data with an expiry, on top of the code definitions |
| Citations deep-link by a chunk's position on the page | Citations link by page, and highlight by matching the text |
Most of these are corrections rather than additions. Each one is a place where a reasonable design failed on a real document.
The Rule, Stated More Precisely
The first article gave the rule as: only the original file and the encrypted vault hold raw personal details. That is true of the data at rest that models read, but it is imprecise in two ways. Here is the version the system is built to:
Personal details never enter the context of a model that is reasoning about a client's documents, and every reveal of a real value is scoped to particular documents and written to the audit log.
"Reasoning about a client's documents" covers chat, classification, summarising, extraction from text and letter drafting. All of them read redacted text.
The rule has an exception. A document has to be read before it can be redacted, and a small number of layout-heavy financial forms lose the column structure that carries their meaning if placeholders are substituted before the figures are read. Those components are identified individually, handled on the server, bounded by what they are allowed to return, and reviewed as a group. None of them is a model reasoning over a corpus. Everything else works on placeholders.
This is not only a compliance question. It is the reason a practice can use a capable hosted model at all. The alternative is a smaller model running on your own hardware, with the capability gap that implies, or a set of data processing agreements most firms do not have the capacity to maintain.
Making a Document Searchable
Uploading a document starts a durable job with retries at each step. If one step fails, the document stops there, says which step failed, and can be restarted from that step without redoing the rest.
Parse. Structured, page-aware reading that reconstructs tables and multi-column layouts, rather than flat text extraction. Quality tiers are chosen per document type, and the choice was measured rather than assumed: the most expensive tier dropped figures that the default tier kept, so nothing uses it. Spreadsheets take the cheapest tier, because every tier produces the same output for a grid.
Redact. Deterministic rules handle values with a provable shape. A model pass handles the ones only context identifies — people, organisations, addresses. Each detected value is replaced by a placeholder, and the real value is encrypted into the vault. The placeholder is stable within a document, so a model can tell that two mentions are the same person without knowing who that person is. From this step on, the redacted text is the only text the system uses.
A change since the first article: placeholders are now scoped to one document. The old form restarted its numbering in every file, so the same string meant different people in different documents while the rest of the system treated it as a unique key. Matching a person across documents is now a deliberate step, proposed by the system and confirmed by a person. There is no automatic merge. A wrong merge would show one person's data under another person's name, and relatives share names often enough that this is not a theoretical risk.
Classify. One call reads the opening pages of the redacted text and proposes a document type, with a confidence and a line of reasoning, which are always recorded. Above a configured confidence the type is applied automatically and marked as automatic, so a person can see where it came from; below it, the type stays a suggestion. An earlier version applied every suggestion without a threshold. It was removed, because a confident wrong guess costs an extraction against the wrong schema.
Chunk and embed. The redacted text is split into overlapping, page-aware chunks, so every chunk carries a real page number and every citation can open the document at that page. Embeddings are computed from the redacted text. There is no unredacted copy in the search layer.
Summarise. A short summary and a set of key terms per document, written from the redacted chunks and redacted again before it is saved. The summary answers "which document is this in?" before chunk search answers "where in the document?". Asking which document first is what lets a question across fifty files find the few that matter.
Extraction Became Its Own Flow
In the first article, extraction was step 3 of the pipeline. That was the design mistake that cost the most.
A failed extraction stopped the run before the searchable index was built. A document that had parsed and redacted perfectly was then unsearchable — because of a structured extract that nothing in search depended on. Chat and search read the chunks and summaries. Only the reporting side reads the extracts.
So extraction moved out. It has its own trigger, its own status and its own limits, and it never writes the document's status. The document stays answerable in chat while extraction runs, re-runs, or fails. Changing a document's type and extracting again costs one extraction and nothing else, because it re-reads text that was already saved rather than parsing the file a second time.
The schema decides what extraction can get right. A schema names the fields, says what to call them, and instructs the extractor to return nothing rather than guess. Every field can be null. That rule is what stops the model inventing a figure. A schema that required an opening balance made the model invent one for a transaction export that states none — and the invented figure was plausible. A schema is a claim about what a document contains, and extraction cannot be more correct than that claim.
There are two extraction engines, and the schema names one. The first reads the redacted text and suits flat fields such as totals, dates and references. The second is better at tables, forms and line items, where the row and column structure matters as much as the values.
Three things were added, and they are the reason the extracts can be trusted.
Checks. Schema validation confirms shape — the right fields with the right types. It cannot tell that a statement's transactions do not add up to its closing balance, which is what a dropped row in a sixty-row table looks like. So a type can declare arithmetic and continuity checks over its own values. A failure raises a finding for a person rather than quietly correcting itself.
Grounding. After an extract is saved, every value is located in the page text and recorded with the page it was found on. A value that cannot be located is marked for review rather than hidden — sometimes legitimately, as with a subtotal the document never states. Without this, a number that was never on the page looks exactly like one that was.
Corrections. A reviewer can correct a value. Corrections are stored separately and append-only, with the author, the time, the original value and an optional reason, and they are merged when the extract is read. The model's original output is never overwritten, so the record of what the system produced stays intact and can still be measured.
Asking a Question
There is one path. An earlier one-shot endpoint was deleted, because nothing called it and it no longer matched the path people used.
Retrieval is hybrid now. A single vector search is good at meaning and bad at reference numbers, misspelt names and exhaustiveness. Several searches run in parallel — meaning, exact terms, named entities, and a fuzzy pass for scanning errors — and their results are combined by rank position rather than by score, because the scores are not on the same scale. The number of excerpts scales with how many documents matched, and selection takes the best passage from every matching document before filling the remainder by relevance. A question across eleven workpapers should cite all eleven, not the four longest.
Totals are added by code. The first article described structured questions as a query chosen by an intent router. The router was removed after measurement: it changed no evaluation score and added noticeable latency to every turn. The model now asks for an aggregation when it needs one, and code does the arithmetic, per currency, including any reviewer corrections. The principle is unchanged and worth restating — "what was the total GST on supplier invoices in Q2?" is an aggregation, and a model should not be doing the addition.
Narrowing a question is not permission. A user can scope a question to a client, a category, a period or a set of files. None of that grants access to anything. Narrowing a question changes what is searched, not what may be read; access is enforced independently of what the user asked for.
Citations point at pages. The first article promised deep links by a chunk's physical position on the page. That was dropped. Positions changed between parses and produced highlights in the wrong place, which is worse than none. Citations now open the document at the cited page and highlight by matching the text, and the panel says so plainly when it cannot find a match. A highlight that silently failed looks identical to one that was never attempted.
Workflows: Is This File Complete?
This is the part the first article did not anticipate. Chat is useful. What a practice runs every week is less interesting and more valuable: for this client and this period, are the required documents present, in period, and consistent with each other?
The verdict is computed by code over the extracted fields. There is no model call in the engine.
A definition names the slots a file needs — the document types it must contain — the rules that must hold, and the shape of the output. A job is that definition applied to one client and one period, and can be re-run. A run is one execution, stored as a complete snapshot, so a run from April reads later exactly as it read in April.
Rules are data, not functions. There are a handful of declared kinds: two figures agree within a tolerance; statements chain from one period's closing balance to the next period's opening balance; nothing is filed twice; and row-level forms of the same ideas for tables inside a single document. Because a rule is data rather than code, its formula can be shown to the person reading the report, and a failure names the exact values it read from each document rather than saying that something is wrong.
Severity is set from measurement. Before a rule is allowed to fail a file rather than merely warn, it is run across the real documents of its type already in the system, and every failure is examined. A rule that fires on documents that are fine teaches people to ignore findings.
Two statuses, kept separate. Whether the run completed is one thing. What is true of the client's file is another. A job that runs perfectly against a file with no bank statements is a successful run with an incomplete verdict — and in the middle of a season, that is the most common result.
Findings are reviewed, and the decision carries forward. A reviewer either accepts a finding as known and acceptable, which carries it forward to later runs, or resolves it as fixed, which does not. If a resolved finding comes back, it was not fixed.
Outputs. The workpaper is rendered by code from the extracted fields and stays stamped DRAFT until every slot is filled and every check passes. A second pass re-adds the rendered output and re-derives each stated figure from the fields behind it, so a schedule that does not add up becomes a named finding rather than a silent correction. A chase list drafts the request to the client for whatever is missing. Where a client letter is produced, its figures come from the same computed model, a language model writes only the connecting sentences, and any paragraph that states a number the model data does not contain is replaced by fixed text.
Runs can be unattended. A person can queue a season's clients, close the browser, and come back to a verdict for each one.
Definitions as Data
In the first article, adding a document type meant adding a file and a registry entry, which meant a deploy. That still works, and the code definitions remain the fallback. Types and rules can now also be published as versioned data:
- Every published version carries a mandatory expiry. When one lapses or is withdrawn, the next live version applies, or the code definition if there is none. Nothing is overwritten, so an old run can always show what it was checked against.
- A new check starts as a warning. A reviewer states it in a sentence; a model proposes it in one of the declared kinds, using only fields that exist; a validator confirms it; and a dry run shows where it would have fired on real past runs before anyone relies on it. If the idea cannot be expressed in the declared kinds, the builder says so rather than approximating.
- A new document type is drafted from examples. An author picks a few representative documents and lists the fields that matter. The proposed fields come from the examples, and the examples then serve as the test: publishing requires a reviewer to confirm them and a re-extraction that reproduces what they confirmed.
- Tolerances can be relaxed for one client, with a reason and an expiry. The rule itself does not change.
Some things stay in code, deliberately: the shape of a new output document, and how redaction recognises identifiers.
The Practice Manual
Rules and schemas cannot hold everything a firm knows. When to chase a missing document. Which differences are routinely accepted, and why. What a senior checks before signing. That knowledge is held as versioned text, with a set of chapters per engagement type, and each chapter declares which workflow and which rules it explains.
It is read in three places: when someone asks the assistant a "how do we…" question, where the firm's own written procedure is preferred over general advice; in the assistant's own instructions; and beside each finding in a workflow report, as a note on what the manual says about that check. When reviewers keep accepting the same finding, the system drafts a paragraph proposing why, and a person decides whether it becomes part of the manual.
The manual explains. It is never an input to the rules engine. The definition states what is checked and to what tolerance; the manual states why the check exists and what to do when it fires.
What Gets Checked, and When
Four sets of checks, and they run whether or not anyone is watching.
On every question: limits on length, rate and spend; a redaction pass over everything about to be sent to a model; and an inspection of the answer before it reaches the screen, including a check on whether the figures in it are supported by the excerpts it was given.
On every extracted value: the shape, the arithmetic, and whether the value can be found on the page it claims to come from.
On every workflow run: the declared rules, and a reconciliation of the rendered output against the figures behind it.
On every change to the system: evaluation suites over fixed sets of real, hard documents — extraction accuracy, fabrication, grounding, classification, redaction and answer quality — plus the ordinary typecheck, lint, tests and build. Changes to model behaviour are covered by evaluations, not unit tests, because a unit test cannot detect that answers got worse.
Corrections
- "The pre-flight check fails closed." It no longer refuses the turn. It removes what it finds, records what it found and where it came from, and proceeds. Refusing blocked users on documents that were fine, and the record turned out to be more useful than the refusal.
- "Citations deep-link by bounding box." Dropped, for the reason given above. Page plus text match is less precise in principle and far more reliable in practice.
- "Extract is step 3." It is a separate flow, for the reason that took longest to learn: a step that nothing downstream depends on should never be able to stop the steps that everything depends on.
- "Adding a document type is adding one file and one registry entry." Still true, and no longer the only way.
- "The AI never saw your name." Still the right one-sentence summary, but it is a claim about models reasoning over a corpus, not a claim that no component ever touches an original document. The precise version is at the top of this article, and the difference is worth stating.
What It Is For
The substrate is not an application. It is the document-handling layer underneath one: parsing, redaction, classification, search, extraction, verification and audit, with the project-specific work — the chat surface, the reports, the workflows — built on top.
Four months of real documents changed what mattered most. The most useful capability was never the chat window. It is that a figure on a client deliverable can be traced back to the page it came from, that a completeness check gives the same verdict on the same file every time, and that when something does not add up, the system says which two numbers disagree and where each one came from.
And the limit: a passing run means the file is complete and internally consistent. It does not mean the accounts are right. That judgement stays with the person who signs.
The Document Substrate: Solving Garbage In, Garbage Out · How Document Substrate works, explained simply · FYI and the Document Substrate · How Orbital thinks · Why "substrate"?