step 00 / 19 Start here

S0 AI Systems Overview

What this chapter covers

An AI product combines a model with other components. This chapter explains what those components do, how a user’s message becomes an answer, and when to use a workflow or an agent. You’ll also read a complete agent in under 200 lines, mini-SWE-agent. The next nineteen chapters each cover part of this system, so start here to choose your route.

After this chapter you can

Name the steps a request goes through from user input to final answer, and say which steps are the model computing and which are your code running. (lecture and self-check)
Explain in your own words how a minimal agent loop works and when it stops. (lecture and self-check)
Draw the full path of one notes-assistant request and mark which later chapter covers each step. (core)
Given a requirement, decide whether it calls for a single call, a fixed workflow or an agent, and justify the choice. (core)
Walk through the mini-SWE-agent source and explain what each part does and when it stops. (optional, requires basic Python)

Lecture

Three terms first. You’ll see them throughout the course:

Large language model (LLM): a trained program. Give it some text and it predicts the text most likely to come next, writing a little at a time. Every chat assistant you’ve used runs on an LLM.
Token: the unit a model reads and writes text in. In English a token is often a short word or part of a longer one; in Chinese it’s roughly one or two characters. Pricing, speed and length limits are all counted in tokens (→ Model Crash Course; deeper: S2 Transformers and Decoding).
Context: the full input sent to the model on each call. The model doesn’t remember previous calls; it only sees what’s in this call’s context. There’s a limit on how many tokens the context can hold, called the context window.

The running example is a notes assistant that stores Markdown notes, answers questions about them, searches them and creates new ones. Part II through Part VI (S6–S19) develop this prototype into a complete system.

★ Why not just use a model?

Useful AI products are rarely “one model answering directly.” They’re systems made of a model plus other components.

An LLM on its own knows only what it saw during training, so it can’t read your notes. It can output text but can’t search or save files. Each call also starts from zero, with no memory of what you said last week. Ask it what you decided in last week’s meeting, and it can say it doesn’t know or invent a plausible answer. The latter is called hallucination.

Code and other components can fill these gaps. Berkeley’s BAIR calls this a compound AI system and argues that quality gains increasingly come from system design, not just from switching to a bigger model. Anthropic calls a model with retrieval, tools and memory an augmented LLM, a basic building block for these systems.

Here are the notes assistant’s main components (in the “Covered in” column, S plus a number is a chapter code; see How to Use This Roadmap for how to read the codes):

Component	What it does in the notes assistant	Covered in
Model	Understands the question, decides what to do, writes the answer	S6 LLM APIs and Application Basics (how to call it); model internals are Part I Models (S1–S5)
Prompt and context	Tells the model its role and rules, and supplies the data for this call	S7 Prompts, Structured Output and Workflows, S11 Context Engineering
Retrieval	Finds the few notes relevant to the question among about 200	S9 Retrieval and RAG
Tools	Actions the model can ask to run, such as `notes_search` and `notes_create`	S7 Prompts, Structured Output and Workflows, S10 Tools and MCP
Memory	Remembers user preferences like “answer in bullet points, three at most”	S13 Memory and Personalization
Loop control code	Calls the model, runs tools, decides when to stop	S12 The Agent Loop and Harness
Interface	Streams the answer, lets the user confirm a draft before an email goes out	S6 LLM APIs and Application Basics, S17 Product and AI UX
Evals, security and shipping	Checks that answers are correct, defends against attacks, records cost and latency	S8 Evals Methodology, S14 Agent Evals, Data and Optimization, S16 Security and Governance, S18 Deployment and Architecture

Each component adds a possible failure, latency and cost. Add one to address a specific need. If the notes assistant only rewrites a pasted paragraph as bullet points, one model call is enough; retrieval and tools add nothing to that task.

★ How does one message become an answer?

After the user hits send, a request goes through a fixed set of steps. When the model uses a tool, some of the middle steps repeat in a loop.

The table shows one round. Each step has a chapter dedicated to it:

Step	What happens	Covered in
1. Assemble context	The system prompt, conversation history, retrieval results and tool descriptions are combined into one input	S7 Prompts, Structured Output and Workflows, S9 Retrieval and RAG, S11 Context Engineering
2. Tokenize	The text is split into tokens and becomes a sequence of integers	S2 Transformers and Decoding
3. Prefill	The model reads the whole input at once and stores intermediate results for the tokens it has read (the KV cache) so it doesn’t have to recompute them	S2 Transformers and Decoding, S5 Inference and Serving
4. Decode	The model generates one token at a time, picking each next token according to the sampling settings	S2 Transformers and Decoding, S5 Inference and Serving
5. Tool call	The output is a structured tool call, and your code runs it	S7 Prompts, Structured Output and Workflows, S10 Tools and MCP
6. Feed results back	The tool result goes back into the context and the loop returns to step 3, until the model decides to answer	S11 Context Engineering, S12 The Agent Loop and Harness
7. Respond and log	Stream the answer to the user while recording the trace, cost and latency	S6 LLM APIs and Application Basics, S17 Product and AI UX, S18 Deployment and Architecture

A few terms from the table: the system prompt is the fixed instruction the developer writes for the model (“You are a notes assistant. Answer only from the user’s notes.”); streaming means showing the output as it’s generated, so the user doesn’t wait for the whole answer; a trace is the record of every step in a request, which you go back to when something goes wrong.

Steps 3 and 4 are where the model actually computes, and they behave differently. Prefill processes input that’s already known, so all tokens can be computed in parallel; even an input of several thousand tokens goes fast. Decode produces output that doesn’t exist yet: token 5 can’t be computed until token 4 exists, so tokens come out one by one, and the longer the output, the longer you wait. For each token, the model produces a probability for every candidate token, and the sampling settings (such as temperature) decide how to pick one (→ Model Crash Course; deeper: S2 Transformers and Decoding, S5 Inference and Serving).

Steps 5 and 6 let the model “do things.” The developer lists the tools’ names, purposes and parameters in the context ahead of time. When the model wants a tool, it doesn’t output a sentence for a human; it outputs text in a fixed format that amounts to notes_search(query="meeting", range="last week"). Your code recognizes that format, actually runs it, and puts the result back into the context so the model sees it on the next round.

Let’s run the notes assistant through it once. The user says: “Find my meeting notes from last week, summarize them in three bullet points, and save that as a new note.” The token counts below are illustrative:

Round	Input sent to the model (prefill)	Model output (decode)	What your code does
1	System prompt 800 + tool descriptions 400 + user message 30 = 1,230 tokens	Calls `notes_search`, 25 tokens	Searches and finds 3 notes, 2,400 tokens in total
2	1,230 + 25 + 2,400 = 3,655 tokens	Calls `notes_create` with a three-point summary, 200 tokens	Saves the note and returns “created”, 20 tokens
3	3,655 + 200 + 20 = 3,875 tokens	Answers “Done. Saved as ‘Last week’s meeting summary’”, 60 tokens	Streams to the user and logs the request

The table shows two things. First, a one-sentence request called the model 3 times. Second, every round resends everything before it, so the input adds up to 1,230 + 3,655 + 3,875 = 8,760 tokens while the output is only 25 + 200 + 60 = 285 tokens. Controlling what goes into each round is the subject of S11 Context Engineering.

Going deeper: what this request costs

APIs charge by token, and output tokens usually cost several times more than input tokens (because, as above, decode has to go one token at a time; → S5 Inference and Serving). Assume $3 per million input tokens and $15 per million output tokens (illustrative, not any vendor’s price):

Input: 8,760 × 3 ÷ 1,000,000 ≈ $0.0263
Output: 285 × 15 ÷ 1,000,000 ≈ $0.0043
Total: about $0.031

Output costs 5 times as much per token, yet most of the spend is input, because the agent resends the whole context every round. Two common fixes: keep the start of each round (system prompt, tool descriptions) unchanged so the API’s prompt caching can reuse that already-computed prefix at a lower price (→ S11 Context Engineering); and don’t paste whole tool results back in verbatim (→ S11 Context Engineering, S12 The Agent Loop and Harness). Note that the KV cache avoids recomputing already-read tokens within one call; prompt caching reuses an identical prefix across calls. They’re related but not the same.

Why the model itself can do steps 3–4 is the subject of S1 Neural Network Basics through S4 Post-training and Reasoning. How to evaluate, secure and ship the whole loop is covered in later chapters (S8 Evals Methodology, S14 Agent Evals, Data and Optimization, S16 Security and Governance, S18 Deployment and Architecture). Three more chapters sit outside the table:

S13 Memory and Personalization: reads the user’s long-term memory when assembling context in step 1, and writes back anything worth remembering after step 7.
S15 Multi-agent Systems: the “tool” called in step 5 can be another agent, which runs its own full path and reports back.
S19 Real Agent Systems and the Capstone: assembles the whole path into a real, long-running system; it doesn’t map to a single step.

Your code searches, saves files and sends email; the model outputs text. The tools you provide determine what it can do, and your code must enforce the security checks (→ S10 Tools and MCP, S16 Security and Governance).

★ Who decides the next step?

There are two ways to build a system from a model and tools. The difference is who decides the next step.

In a workflow, the steps are written into your code ahead of time; the model only fills in the content of each step. In an agent, the model decides the next step; your code only runs it and feeds back the result (see the Glossary). Step 6 in the previous section, “return to step 3”, is the agent loop: how many rounds, and which tool each round, are up to the model.

Three requests to the notes assistant as examples:

Request	Approach	Who decides the steps
“Rewrite this note as bullet points”	Single call: one prompt, one answer	Only one step; nothing to decide
“Every morning at 8 a.m., summarize the notes added yesterday”	Workflow: code fetches yesterday’s notes → calls the model to summarize → saves the result; the same three steps every day	Code
“Help me prepare for tomorrow’s meeting with the design team”	Agent: the model might first search “design team”, see that the budget came up last time, then search “budget”, and finally put together a list. How many searches and what to search for can’t be fixed in advance	Model

Both have costs. A workflow is predictable, easy to test and has a fixed cost, but it can’t handle anything the flow didn’t anticipate. An agent can take paths nobody planned for, at the price of higher latency and cost, less predictability, and compounding errors: if the first search uses the wrong keyword, the conclusion is wrong no matter how well the rest is summarized.

Choose between a workflow and an agent based on how much control the model needs. If a workflow already completes a task reliably, an agent usually adds latency, cost and debugging work. You can also combine them, assigning one step of a workflow to a small agent.

Going deeper: Anthropic's five workflow patterns

Anthropic’s “Building effective AI agents” groups common workflows into five patterns. For now, just remember the names; S7 Prompts, Structured Output and Workflows covers them in detail:

Prompt chaining: one step’s output is the next step’s input, e.g. write an outline, then write the full text from it.
Routing: classify first, then hand off to different flows, e.g. split user messages into “asking a question” and “adding a note”.
Parallelization: run several calls at once and merge the results.
Orchestrator-workers: one model breaks the work down and hands the pieces to other calls.
Evaluator-optimizer: one call produces a result, another grades it and asks for changes, for several rounds.

★ When should you build an agent?

Agent capability can be graded into levels. The point of the levels is to remind you that most needs are met at a low level.

Google/Kaggle’s “Introduction to Agents” whitepaper sorts agent systems into five levels. Mapped to the notes assistant:

Level	Whitepaper description	What the notes assistant looks like
Level 0	A reasoning system with a model and no tools	Can only answer from training data; can’t see the user’s notes
Level 1	Connected to tools, can look up live information	Can call `notes_search` to look up notes before answering
Level 2	Plans multi-step tasks and actively manages context	Can handle multi-step tasks like “turn last week’s meetings into a weekly report”, choosing which notes go into the context
Level 3	Multiple agents dividing the work	One searches, one drafts, one checks citations (→ S15 Multi-agent Systems)
Level 4	Can create missing tools or agents on its own	Notices there’s no “export to PDF” tool and builds one itself

When an agent is worth it. OpenAI’s “A Practical Guide to Building Agents” lists three situations:

Complex decisions that need judgment rather than rules.
Rules so numerous they’re hard to maintain.
Heavy reliance on unstructured data (documents, conversations).

Anthropic’s “Building effective AI agents” is more direct: look for the simplest solution first, and if a single call plus retrieval is enough, don’t build an agent. Agents trade latency and cost for flexibility, and their errors compound.

Put the two together and you get this order of questions:

Can a single call (plus retrieval if needed) do it? If yes, stop there.
Can the steps be fixed in advance? If yes, use a workflow.
Only if the steps really can’t be decided in advance, and one of the three situations above applies, consider an agent.

Applied to the notes assistant: “summarize yesterday’s notes every day” has fixed steps and matches none of the three situations, so it’s a workflow. “Prepare for tomorrow’s meeting” requires judging what’s relevant across a pile of unstructured notes and deciding what to search next. It matches situations 1 and 3, so it’s a good fit for an agent.

Ask whether you can decide the next step in advance. A flow with 10 clearly specified steps is still a workflow. When the rules are few and clear, plain code is the most reliable choice.

★ How small can an agent be?

A complete agent that can fix bugs in code can have a core of just over a hundred lines. It’s the loop described above.

Why read it. Once you’ve read it, you’ll see how small an agent’s skeleton really is. The harness components you’ll learn later (the harness is the code that wraps the model and makes the agent reliable), such as permissions, checkpoints and compaction (→ S12 The Agent Loop and Harness), all add things around this loop.

What it looks like. mini-SWE-agent is a minimal coding agent from the SWE-bench team (SWE-bench is a benchmark built from real GitHub issues, the bug reports filed on open-source projects, that tests whether a model can fix the code). The README says the agent class is about 100 lines; the current default.py is under 200, with the extra lines going to configuration and serialization. Its design has three notable features:

Only one tool: run a bash command (bash is the shell you use in a terminal to work with files by typing commands). Each action runs in its own subprocess.run.
Fully linear history: each step just appends new messages; nothing old is edited or removed.
A very simple main loop: send the history to the model → (in the model layer) extract the command from the response → run it in the environment → append the output to the history, and repeat.

It stops when: the model sends the agreed-upon completion command; a step or cost limit is hit; a wall-clock time limit is hit; or the model fails to output a properly formatted command too many times in a row.

Here’s a walk-through of an illustrative task: “The test_parse_date test is failing. Fix it.” Don’t worry if you can’t read the commands; it’s enough to know they find a file, read it, edit it and run the tests, in that order.

The model outputs grep -rn "def parse_date" src/; the result, src/utils/date.py:12, is appended to the history.
The model outputs sed -n '1,40p' src/utils/date.py, reads the function, and that’s appended to the history.
The model outputs a one-line command that edits the file; the result is appended to the history.
The model outputs python -m pytest tests/test_date.py; the result, “1 passed”, is appended to the history.
The model sends the completion command and the loop ends.

Each round maps to steps 1–6 in the path table: the history is the context, the bash command is the tool call, and the command output is the result fed back.

Why bash alone is enough. Bash can read files, search, edit and run tests on its own, so it works as a general-purpose tool. It costs two things: there’s no persistent shell state (cd and environment variables don’t carry over to the next step), and there are none of the view and edit commands SWE-agent designed specifically for models (→ ACI in S10 Tools and MCP).

The loop is small. Product complexity comes from deciding which commands need approval, how to resume after a crash, and how to compress a long history. S12 The Agent Loop and Harness covers these requirements.

Before you build: what to know before reading default.py for the optional exercise

How the files divide the work: agents/default.py is the agent class, handling the loop and stop checks. The code that parses the model’s response and extracts the command lives in the model layer. The environment is what actually runs the command. When you see message["extra"]["actions"], just know the model layer parsed it and put it there.
What running each step separately means: since cd doesn’t persist, the model has to change directory and run the command in the same line every time. The upside is that steps don’t interfere with each other and are easy to reproduce.
What linear history means: the whole history is a complete record, easy to save, replay and debug. But every step makes the context longer, and on a long task you hit the cost problem worked out in the previous section. That’s exactly what compaction in S12 The Agent Loop and Harness is for.

Where to go next

After this chapter, pick a route based on your goal:

Your goal	Route	Notes
Put LLMs into a product and build AI applications or agents (most readers)	S0 AI Systems Overview → Model Crash Course → S6 LLM APIs and Application Basics → … → S19 Real Agent Systems and the Capstone	The Model Crash Course covers the minimum model knowledge for building applications in ten short topics. The 14 application chapters (S6–S19) don’t require Part I Models (S1–S5); when they use a model concept, they link back to the Model Crash Course first and then mark “Deeper: Sn chapter title”
Understand how models work inside, or do research, train models, or build inference infrastructure	S0 AI Systems Overview → Part I Models (S1–S5) → the 14 application chapters (S6–S19)	Part I Models (S1–S5) builds up from the bottom: neural networks, Transformers, pretraining, post-training, inference. It’s lower level and uses more math; it’s the optional deep route
Understand what AI systems do without writing code for now	Concepts route: same order as the first route	In each chapter, read the lecture, do “Try it first” and the self-check, and skip the exercises that need code; see How to Use This Roadmap

If you’re not sure, take the first route. When you want to know “why” partway through, go back and read the matching chapter of Part I Models; the application route page explains when each chapter is worth reading.

Each chapter has two kinds of exercises. Core exercises are the main line and can be done with the lecture alone. Advanced exercises are optional and are collapsed in Simple mode. Small tasks marked “optional” (like this chapter’s source reading) can be skipped too. The advanced exercises form two projects: the system project (A6–A19, what the codes mean) starts in S6 LLM APIs and Application Basics and builds your own AI application chapter by chapter; the model project (M1–M5) has you train small models by hand in Part I Models (S1–S5).

Exercises

Concepts route (no code): after “Try it first”, you can skip to the self-check.

Try it first (about 10 minutes)

Go back to the notes assistant’s three-round table above. If the round 1 search found only 1 note, 600 tokens in total, how many input tokens would the model get in round 2?

Check your answer

1,230 + 25 + 600 = 1,855 tokens. Each round’s input is “the previous round’s input + the model’s previous output + the tool result”, so the longer the tool result, the longer every later round gets.
“Every time the user adds a note, automatically add 3 tags”: single call, workflow or agent?

Check your answer

Workflow. The steps are fixed (note added → call the model to generate tags → save) and decided by code. The model only supplies the content of one step, “what the tags are”; it never has to decide what to do next.

Core: Map a request’s path and decide whether it needs an agent

Goal: draw the request path through an AI system and decide whether an agent is appropriate in three scenarios. You do not need to read or write code.

Steps

Pick a notes-assistant request (for example, “Turn this week’s reading notes into a to-do list and save it as a new note”) and draw its full path. Include at least the 7 steps from the table above, and label each box with its chapter code.
In a different color, mark which steps are decided by code and which by the model. That’s the line between workflow and agent.
Write three requirement scenarios (for example, “Email me a summary of my inbox every day”, “Help me fix this bug”, “Answer the company FAQ”). For each, decide whether it calls for a single call, a workflow or an agent, with two or three sentences of reasoning.

Acceptance criteria

The path diagram has at least 7 steps, each labeled with its chapter code from the table. Next to the diagram, note which part of the path each of these chapters acts on (based on the “Three more chapters sit outside the table” paragraph after the table in How does one message become an answer?):
- S1 Neural Network Basics through S4 Post-training and Reasoning
- S8 Evals Methodology
- S13 Memory and Personalization through S16 Security and Governance
- S19 Real Agent Systems and the Capstone
Each of the three scenario decisions cites one of the “when to build an agent” conditions.

Hints

Draw it on paper, in Excalidraw or in Mermaid. It’s the same for TypeScript and Python readers.
If you get stuck:
- Reread: How does one message become an answer?, When should you build an agent?
- Further: the “Building blocks, workflows, and agents” section of Anthropic’s “Building effective AI agents”

Optional: Read the mini-SWE-agent source (requires basic Python)

If you don’t know Python, skip this. It doesn’t affect any later chapter.