Could, Should, Might, Don't: A Second Look at When AI Operates the Software

When AI Operates the Software took Elena Verna's argument that the software now operates itself and set out what to build for it. Nick Foster's book Could Should Might Don't is about how people think about the future rather than what it holds. It sorts that thinking into four mindsets, each with a typical failure. Applied to the earlier article, the four mindsets give a short list of changes: one premise was accepted too quickly, one section already describes the transition that forecasts usually skip, and the section on what goes wrong once delegation works is missing.

Four ways of thinking about the future

Foster does not recommend one mindset over the others. He describes what each does well and how each fails:

Could is the optimistic vision of a changed world. It tends to show the end state and skip the years of partial change between here and there.
Should is extrapolation from data. A trend line is evidence up to today. The part drawn beyond today is a forecast, and it is easy to read it as if it were evidence.
Might is scenario planning. It can produce many possibilities and few decisions, and it is weak at taking unlikely events seriously.
Don't looks at risks and consequences, including the side effects a technology has once it is part of ordinary work, and how it ages.

The book gives a set of questions to ask of any claim about the future, not a method for building one. That suits the earlier article, which is an argument about what to build next.

Verna's premise is a trend line

"We're changing who actually operates the software" is a Should claim. It is drawn from what Verna sees at Lovable, a product where a person describes what they want and the agent builds it. The earlier article took it as a starting fact.

For firms that produce signed work, it holds for some tasks and not others. It holds where the agent picks a capability and a person checks the result: finding a document, comparing two years, drafting a chase list. It does not hold for the decision to sign, which stays with a person by design, or for deciding what "complete" means for a job, which a person writes down.

The earlier article already has the test that turns the forecast into evidence, one capability at a time. The Document Substrate's skill-selection eval runs real requests through the production prompt and checks which skill the model loads. Where that eval passes, the agent does operate the software. Where a capability has no cases, the claim is still a forecast.

The delegation ladder is the part a forecast skips

Could thinking shows the end state. The delegation ladder in the earlier article describes the steps in between: answer a question, run a skill, run a workflow, apply results without a person, and be called by another agent. Each step needs evidence before it is handed over.

The fourth step is measured in the Document Substrate. A document classification is applied without a person only when the classifier's confidence is at least 0.9 and its lead over the next most likely type is at least 0.3. An eval scores that decision, not only the label, because a wrong type applied automatically sends a document to the wrong set of fields without raising an error. The admin feedback page shows how many classifications were applied automatically and how many a person later overturned.

So the earlier article's argument is better stated this way: the change is not that agents operate software. It is how a firm moves from one step to the next, and what evidence it asks for at each.

What goes wrong once delegation works

The earlier article lists what to build. It does not say what goes wrong after it is built and working. The Don't mindset asks for that section, and three risks follow from the earlier article's own design.

The acceptance rate cannot tell trust from fatigue. One of the proposed measures was how often a reviewer accepts an item unchanged. That rate rises when the system improves. It also rises when reviewers are tired or busy and stop reading closely, and the number is the same in both cases. The Document Substrate already avoids one cause of this: once a person has confirmed that an organisation is a supplier, later documents that name it are linked without asking again, because a confirmation repeated when nothing new is being decided stops being read. The acceptance rate still needs a second measure beside it. Two options: add items with a known error to the queue from time to time and record whether they are caught, or have a second person re-review a small sample of accepted items each month. Neither is built.

Reading skills fade as delegation grows. A reviewer catches what the model misses because they know what the document should say, and that knowledge comes from reading documents. If staff stop doing the first reading, the reviewer gets worse at the one job left to them, which in signed work is the job that matters. The six stages for accounting staff move people from preparer to reviewer. A practice also has to decide how new staff learn to read a file when the system reads it first. One answer is to have trainees prepare some files by hand and compare their reading with the system's. That is a practice decision, not something the Document Substrate does.

The system ages when the models under it change. Foster asks how a technology behaves years after it is installed. For the Document Substrate the question is what happens when a model or a vendor service is updated. It has already happened twice:

An extraction release arrived unmeasured. A new release of the extraction vendor's engine reached production for thirteen document types without being measured, because the default eval tested a different engine. An eval for that engine now exists.
A dearer parsing tier lost a figure. A measurement found that a more expensive parsing tier dropped a figure from a tax return that the standard tier kept. The tier is now pinned, with that evidence recorded beside it.

The models are set in one place in the code, and the classifier records which model answered. The defence is the set of evals that score skill selection, classification, extraction and redaction without a model judging the result. They were written to block a merge, but since automatic runs were paused in September 2026 they are run locally, by hand. The rule to state is that they run on every model or vendor change, not only on prompt changes. Generating the workflow manifest from the code prevents one kind of drift, between the description and the code. This is the other kind, between the code and the models it calls.

An ordinary day in the queue

Foster prefers futures described in specific, ordinary detail to futures described as transformations. This is an illustrative day for a preparer using the Document Substrate, not a client case.

The queue for a rental property job holds three uncertain items, each with a reason:

A balance that could not be found. A bank statement's closing balance is marked as not found on the page it cites. The preparer opens the page, sees the figure is split across two lines, and confirms it.
A figure read wrongly. The interest total is wrong. The preparer corrects it, and the correction is stored beside the model's reading with a reason. That field has now been corrected more than ten times on that document type, so the next review of the feedback drafts a test case for it, which a person reviews and adds to the suite.
A rule that failed. One expense category is outside the tolerance the practice set. The preparer adds a note for the reviewer.

While checking the file, the preparer also notices a rates invoice that was classified automatically as an insurance document. It was not in the queue, because the classifier was confident. The preparer re-types it, and it shows on the feedback page as overturned. The workpaper stays DRAFT until the reviewer signs.

None of this is dramatic. The confident misclassification that the queue did not show is the case the Don't section is about: delegation works most of the time, and the exceptions are found only by someone who is still reading.

Where to start, with a fourth question

The earlier article ended with three questions: would the agent choose each capability, can a person see what to ask, and can a reviewer check the result without asking the model. That is Might thinking without the diagrams: three tests a team can run, instead of a set of scenarios. The Don't mindset adds a fourth:

What would show that a measure has stopped meaning what you think? For each measure of delegation, name the signal that would show it moving for the wrong reason, and decide how you will collect it.

When AI operates the software · Six stages for accounting staff · The Document Substrate in detail · Principles