How an AI chat really works

Five ideas explain almost everything: messages, tokens, the context window, tools, and agents. Learn these and the AI stops being magic. It becomes a strange but powerful tool that works best when you know its limits.

The whole trip in one picture

Five stations left to right: you type words, they are chopped into numbered tokens, the model reads its one fixed-size desk, it may pass a note to your code for help, and the answer comes back out as words.
1

Messages, and the stage whisper

A chat is a stack of messages: yours, then the AI’s, then yours again. But before the show starts, someone slips the actor a note the audience never sees — the system prompt. It sets the tone, the rules, and the job, and when your message argues with that note, the note usually wins. Some models also mutter to themselves first: a little thinking… before the answer.

A conversation column: a system note set before the show, a user message, a faint thinking line, then the assistant's reply. Beside it, a card explains the system prompt outranks the user when they clash, and that one message can carry text, files and tool calls.
Under the hood

Conversations are structured as user and assistant messages, with a system message at the start acting as a highly influential meta-instruction — it steers behaviour even over conflicting user instructions, though jailbreaks can sometimes override it. Reasoning models emit intermediate “thinking” tokens as part of the assistant message, and a single message can hold multiple parts: text, files, tool calls and tool results.

2

Tokens: the words are really numbers

The AI never sees your words. A tokenizer chops the text into chunks and gives each chunk a number. Common chunks get one number each; a rare word shatters into many little pieces. You pay for every chunk — the ones you send in and the ones that come back out.

The chop: a plain sentence becomes three numbered tiles, while a rare word shatters into five. The bill: your text is encoded into input tokens with a price tag, the model produces output tokens with a second price tag, and they decode back to words.
Under the hood

Tokens are a compression of text into sub-word chunks: encode on the way in, decode on the way out, billed both directions. Vocabulary size is the trade-off — one token per character would compress nothing, one per string would be unlookupable, so tokenizers keep common character groupings plus single characters as a fallback. That fallback is why unusual words fragment into many tokens, and why the cost of a call is genuinely hard to estimate up front: you cannot know exactly how your text will be chopped.

3

The context window: one desk, fixed size

The model works at a desk of a fixed size, and everything must fit on it at once: the stage whisper, the whole conversation so far, the tool menu, and the space for its answer. The size is a hard limit. Overfill the desk and the call fails, or the answer gets cut off mid-sentence.

A fixed-size desk holding the system prompt, the conversation so far and the tool menu, with a dashed area reserved for the answer and a ruler marking the hard limit. A side note shows what overflow does: an error, or an answer cut off mid-sentence.
Under the hood

The context window is the combined input and output token budget, hard-coded per model. Input includes the system prompt, the full message history and tool definitions; output is whatever the assistant generates. Exceeding the limit causes API errors or truncated generations, so long conversations creep toward the ceiling turn by turn.

4

Lost in the middle

Here is the strange part: a bigger pile on the desk makes the AI worse, not better. It pays most attention to the start and the end of what it reads; the middle goes blurry. So the winning move is not stuffing the desk — it is the smallest set of things that really matter. And when a chat gets long, start a fresh one.

A long shelf of conversation pieces with an attention curve above it: high at the start, sagging in the middle where the pieces fade to grey, high again at the end. Beside it, the fix: start fresh threads, and choose fewer, better tokens.
Under the hood

“Lost in the middle” is the measured tendency of LLMs to weight early and late context more heavily than the middle, which is why output quality degrades as the window fills. The craft of fighting it has a name: context engineering — curating the optimal set of tokens at inference time. Given a finite attention budget, good context engineering means finding the smallest possible set of high-signal tokens that maximise the likelihood of the outcome you want.

5

Tools: the AI writes a note, your code does the job

The AI cannot touch the world. What it can do is write a very tidy request note: “please run this tool with these details”. Your code reads the note, does the actual job, and passes the result back. That little loop is the whole trick — and it is enough to build really powerful things. One warning: hand the AI a menu of too many tools and it gets slower and dumber, like anything else crowding the desk.

A four-step loop: the model is handed a small tool menu, writes a request note asking for the weather in Paris, your machine actually runs the job, and the result is passed back so the model can answer. A note warns that nothing runs until your code says so, and that too many tools crowd the desk.
Under the hood

Tool definitions (name, description, JSON-schema parameters) are given to the model up front. The model responds with a tool call message carrying an id and arguments — nothing has executed yet. Your application runs the tool locally and returns a result message (success or error, tied to the same id), and the model continues with the full history. Error handling is essential. And because every definition sits in the context window, tool count is a real cost: trouble can start around 12 tools, and many frameworks recommend a maximum of about 6.

6

Agents and workflows: who decides when to stop

Both are ways of chaining many AI calls together, and one question tells them apart. A workflow is a recipe: your code runs fixed steps and your code decides when it is done. An agent is the AI with tools in a loop, deciding for itself when the job is finished — more flexible, less predictable. And a single call? That is just an API call. Not everything needs to be an agent.

Two lanes. The workflow lane: fixed numbered steps with the code holding the stop flag, and a person able to check between steps. The agent lane: the model loops with its tools and holds the stop flag itself. A footnote: one call is just an API call, neither an agent nor a workflow.
Under the hood

The split comes from Anthropic’s “Building Effective Agents”: workflows are predetermined, code-defined sequences of LLM calls where program logic controls termination; agents are given tools and choose their own next actions and stopping point. Workflows shine at decomposing hard tasks into smaller ones LLMs are better at, at repeatable and parallelisable processes, and at inserting human review (HITL) between steps. Agents suit ambiguous tasks that need improvisation.