step 19 / 19Part VI Capstone

S19 Real Agent Systems and the Capstone

What this chapter covers

The previous eighteen chapters examined AI systems layer by layer. Here you will study how coding agents, deep research systems and always-on personal agents combine those layers. Compare their use of fixed workflows, model decisions, multiple agents and human approval to understand the design tradeoffs.

The capstone extends your system into an always-on agent. Its five pieces of engineering evidence show that the system can ship (shippable), its performance can be measured (evaluable), and its design decisions can be justified (defensible).

If you are doing only core exercises, analyzing an open-source coding agent completes the route. The A19 capstone is advanced and requires system project steps A6–A18. On the concepts route, finish the lecture, Try it first and self-check.

After this chapter you can

Read a trajectory and say which of the six parts (loop, tools, context, memory, evals, security) each step falls into and which chapter covers it. (lecture and self-check)
Explain the orchestrator-worker and citation design of deep research systems, and what they cost. (lecture and self-check)
Take any open-source agent, map out its six parts, and find each one in the code. (core)
Explain why Agentless and SWE-agent can both be competitive on the same benchmark, and what that means for “when to use an agent.” (core)
Design an always-on agent’s schedules, notification channels and proactive behavior, while keeping its cost and security risk under control. (advanced)
Deliver an always-on agent and five pieces of engineering evidence: a regression suite, an infra variance report, a trifecta audit, harness ablation notes and an architecture document. (advanced)

Lecture

Use the six-part map to examine three real systems. The notes assistant provides a familiar comparison: fixing its own code calls for coding-agent design, researching a question calls for deep research, and working while its interface is closed calls for an always-on agent. The final section covers the capstone’s five pieces of evidence and two statistical ideas for interpreting them.

★ How do you take apart any agent?

Analyze an agent in six parts, map each to earlier chapters, and compare the choices different products make.

Using the same six parts makes systems easier to compare than feature lists alone. See how to read the codes for the chapter references:

Part	Questions to ask	In the notes assistant	Covered in
Loop	Does code or the model decide the next step? When does it stop?	The model decides whether to search again; 15 steps max	S7 Prompts, Structured Output and Workflows (workflows), S12 The Agent Loop and Harness
Tools	What actions can the model take? How is the interface designed?	`notes_search`, `notes_create`, `web_clip`	S10 Tools and MCP
Context	What does the model see at each step? What if it gets too long?	System prompt, the question, 5 retrieved notes; compaction when the conversation gets too long	S9 Retrieval and RAG, S11 Context Engineering, S12 The Agent Loop and Harness
Memory	What does it remember across sessions?	“User prefers bullet points,” time zone UTC+9	S13 Memory and Personalization
Evals	How do you know it got better or worse?	A 50-case regression suite, 3 runs per case	S8 Evals Methodology, S14 Agent Evals, Data and Optimization
Security	What data can it touch, what can it do externally, who approves?	Email drafts need a human to click send; clipped web pages are always untrusted input	S12 The Agent Loop and Harness (permissions), S16 Security and Governance

The fastest way to take an agent apart is to read a trajectory: the complete record of one task from start to finish, every model response, every tool call and its result. Each step falls into one of the parts. Take the notes assistant answering “What did I note down about RAG last month?”:

Load the system prompt, plus the memory “prefers bullet points.” → context, memory
The model calls notes_search(query="RAG", month="2026-10"). → loop, tools
5 notes come back; 1 is a clipped web page, and a hook marks it as untrusted content. → security
The model decides it has enough, answers in bullet points with links to the notes, and makes no more tool calls. → the loop’s stop condition
This trace gets sampled and may later become a case in the regression suite. → evals

These parts need not be separate code modules. Compaction both manages context and chooses what to keep, so one row can map to two chapters. Absence is also a design choice: mini-SWE-agent has no memory or specialized tools.

★ How are coding agents designed?

Coding agents are the most mature kind of agent today. Four representative systems each push one of the six parts to the extreme; comparing them side by side shows best how much to leave to the model.

They’re mature because the task conditions are unusually good: the goal is clear (make the tests pass), feedback is immediate (the compiler and tests point out errors right away), and the environment can be put in a sandbox and thrown away if something goes wrong. What the four systems each demonstrate:

SWE-agent (Yang et al. 2024) pushes the tools part. It introduced the agent-computer interface (ACI) (← S10 Tools and MCP): commands for a model should be designed like a UI for people. For example, the file viewer shows only 100 lines at a time with line numbers, search results are capped, and every edit gets a syntax check right away, rejecting the edit if it fails. This works better than handing the model a raw shell. mini-SWE-agent, which you read in S0 AI Systems Overview, is the other extreme: about 100 lines, bash only.
Agentless (Xia et al. 2024) pushes the loop part, by removing it: the model never decides the flow. Instead there’s a fixed three-phase workflow: localize the problem, generate patches, validate and pick a patch. The model only works within each phase, and it reaches competitive results at very low cost.
OpenHands (Wang et al. 2024) is an open platform focused on the execution environment and security: the agent acts by writing code inside a sandbox, and every action and result is recorded as an event stream for replay and debugging.
Claude Code turns harness components (← S12 The Agent Loop and Harness) into user-configurable interfaces: CLAUDE.md (project memory), hooks (code that runs before and after tool calls), skills, sub-agents and permission modes.

Compare SWE-agent and Agentless on the same issue. Suppose the notes assistant’s repo has a bug: “When a tag is immediately followed by a comma (#reading-notes,), searching for #reading-notes doesn’t find the note.”

	SWE-agent style (model decides the flow)	Agentless style (code decides the flow)
Find the location	The model searches for `tag` itself, opens `parser.py`, scrolls to line 40	Phase 1: give the model the repo structure and narrow down to file, function, line
Change the code	The model edits lines 40–42; the syntax check passes	Phase 2: generate 4 candidate patches at once
Validate	The model decides to run the tests, reads the results, edits again if needed	Phase 3: code runs the existing tests and a reproduction test; 4 become 2, and one is picked
Steps	Depends; could be 6 steps or 30	Always 3 phases

Both approaches are competitive on benchmarks like SWE-bench because of the shape of the task: every case is “one issue, one repo, tests decide pass or fail.” When the structure is known in advance, a fixed flow covers it, and it’s cheaper, more predictable, and when it breaks it’s easy to see which phase went wrong. This is the best real-world example of workflow vs agent (← S0 AI Systems Overview, S7 Prompts, Structured Output and Workflows): start with a workflow, and only hand decisions to the model when the task really needs it.

A fixed workflow falls short when you can’t know the next step until you see the result of the previous one: you have to run the program before you can reproduce the bug, you have to read the logs to know which module to change, a feature touches five files in an order you can’t see in advance, or the issue itself is unclear and you have to ask someone first.

Similar benchmark scores can hide differences in cost and consistency. Use pass^k, the share of tasks completed successfully in all k runs (← S8 Evals Methodology), and test your own workload. The core exercise measures these properties on 5 tasks you choose.

Going deeper: spec-driven development and Claude Code's configuration points

Spec-driven development is one way of working with a coding agent: first write down clearly what to build and why (the spec), turn it into a technical plan, break that into tasks, and only then implement and verify. GitHub’s spec-kit turns this process into a tool. Its value is making human intent a document you can check, instead of something scattered through a conversation. On the six-part map, a spec is context loaded into every session, and also the basis for evaluation at acceptance.
Which part each Claude Code configuration point maps to:
- CLAUDE.md is loaded at session start (context, memory).
- Hooks can block or rewrite an action before a tool call, and check or log after it (security, evals).
- Skills normally expose only a name and description, loading the full text only when needed (progressive disclosure, ← S11 Context Engineering, S12 The Agent Loop and Harness).
- Sub-agents work in their own context and return only the result (context isolation, ← S12 The Agent Loop and Harness).
- Permission modes decide which actions ask a human first (← S12 The Agent Loop and Harness).
The official Best practices is a guide to working with the agent, and you can also infer its harness design from it.
OpenHands’ “act by writing code”: rather than defining dozens of tools, let the agent write some Python or bash to run in the sandbox. The action space gets bigger, which makes sandbox and egress limits (← S10 Tools and MCP, S16 Security and Governance) all the more important.

★ How do research agents divide the work?

Deep research systems answer questions that need a lot of searching and synthesis: a lead agent breaks down the question, several sub-agents search in parallel, and finally a dedicated citation agent ties conclusions back to sources. It can cover more ground, at the cost of using many more tokens.

Research questions have many facets, and each facet needs many searches. Put it all into one agent and the context quickly fills up with search results (← S11 Context Engineering), and facets can only be worked on one after another. Anthropic’s “How we built our multi-agent research system” (2025-06) is the most complete public description of this architecture so far. Mapped onto the six parts:

The loop has two levels: the lead agent plans research directions, decides how many sub-agents to dispatch, and reads their reports to decide whether it has enough; each sub-agent runs its own search loop. This is the orchestrator-worker pattern from S15 Multi-agent Systems.
Contexts are isolated from each other: each sub-agent reads dozens of pages in its own context and returns only a condensed conclusion to the lead agent (← sub-agent isolation in S12 The Agent Loop and Harness).
A dedicated citation agent: after the lead agent writes the report, a separate agent maps each claim to a source. Having one role do only this is more reliable than asking the agent writing the report to add sources along the way.
Security: every web page is untrusted input (← S9 Retrieval and RAG, S16 Security and Governance).

In the notes assistant: Sam asks, “Summarize the main approaches to RAG chunking since 2025 and compare them with the experiment results in my notes.” The lead agent splits this into three facets for three sub-agents: papers, engineering blogs, and Sam’s own notes. The three sub-agents search at the same time, and each returns a summary of about 200 words plus a source list; the lead agent synthesizes them into a report, and the citation agent ties each sentence back to a note or URL.

Do the cost math. Anthropic’s numbers: agents use on average about 4× the tokens of a regular chat, and multi-agent systems about 15×. Those are averages on their system; measure your own.

To decide whether multi-agent is worth it, ask four questions: Is the answer worth more than the extra cost? Can the problem be split into independent facets that can run in parallel? Is there more data than one context can hold? How costly is a wrong or incomplete answer? If the answer to the first two is no, a single agent is usually enough.

Evaluating these systems is also hard, because research reports often have no single correct answer. BrowseComp (Wei et al. 2025) gets around this with questions that are “hard to find but easy to verify once found”: the answer is a short string you can check at a glance; the hard part is the process of finding it. You can borrow this idea when designing your own evals. For example, a notes assistant question like “On what day did I first note down GraphRAG?” requires digging through lots of notes, but the answer is a date, which can be graded automatically.

The 4× and 15× multipliers both use regular chat as the baseline. Compare multi-agent with a single agent when choosing the architecture: about 15 ÷ 4 ≈ 3.75× (← S15 Multi-agent Systems).

Going deeper: how orchestrator instructions go wrong, and how to fix them

The same article notes that an early version of the lead agent spawned up to 50 sub-agents for simple questions, and when task descriptions weren’t specific enough, sub-agents duplicated each other’s work or left gaps. The fix was to spell out rules in the prompt for “scale effort to query complexity,” and to require each sub-agent’s task description to include the objective, output format, available tools and scope boundaries (← S15 Multi-agent Systems).
Research can run a long time and the context can exceed its limit. The lead agent first writes its research plan into memory so it can recover it if the context is truncated; this is the same idea as the structured notes in S12 The Agent Loop and Harness.
The quality improvement reported in the same article (multi-agent outperforming single-agent by 90.2% on their internal eval) was measured on research tasks especially suited to parallelism, using their own eval (← S15 Multi-agent Systems). It doesn’t transfer directly to other tasks.

★ What does an always-on agent add?

An always-on agent doesn’t wait for the user to open a web page before it works. It runs in the background and interacts through channels the user already uses. There’s no new magic: it just adds new “trigger sources” in front of the loop, and that amplifies the hard problems from earlier chapters.

Taking the open-source OpenClaw as an example, these systems usually have a few components:

Gateway: a long-running process that connects the various messaging channels to the agent and manages sessions and routing.
Channels: where users reach the agent, such as a chat app, email, webhooks or desktop notifications. Each channel is a new input source, and potentially a new source of untrusted input (← S16 Security and Governance).
Cron: scheduled jobs that fire at exact times, such as an 8 a.m. daily summary.
Heartbeat: wakes the agent periodically so it can check whether anything needs handling; if not, it quietly exits.
Goal loop: give the agent a goal and let it keep working across many turns until an evaluator judges the goal met, the budget runs out, or a human needs to step in (← stop conditions in S12 The Agent Loop and Harness).

On the six-part map: cron and heartbeat are new trigger sources for the loop, channels change the security part, and the goal loop is a loop with stop conditions and a budget added. In the notes assistant: every morning it organizes the previous day’s new notes and emails a summary (cron); every 30 minutes it checks whether any note mentions something coming due, and only sends a reminder if so (heartbeat); Sam says in a chat app, “Turn this month’s reading notes into an outline,” and the agent works across many turns until it’s done (goal loop).

Do the cost math before choosing the interval. A heartbeat every 30 minutes is 48 a day, 1,440 in 30 days. Assume each carries 5,000 input tokens at $3 per million input tokens (an illustrative price; output ignored): a month is 1,440 × 5,000 = 7.2 million tokens, about $21.60. If the context each run carries grows to 20,000 tokens over time, it becomes 28.8 million tokens, about $86.40, 4× as much. A common way to save is a cheap filtering layer: first use rules or a small local model to check “are there new notes with due dates?” and only call the cloud model if there are. If only 6 checks a day actually need it, a month drops from 1,440 calls to 180.

Always-on amplifies three old problems:

Long stretches with nobody watching: process restarts and provider errors will happen, so state has to be recoverable. That’s durable execution and monitoring (← S18 Deployment and Architecture).
Sending messages proactively: every notification spends the user’s attention, so you need a notification budget and a way to snooze or turn off a category of notifications (← S17 Product and AI UX).
Hitting all three: the lethal trifecta is an agent that can read untrusted content, access private data and send data out (← S16 Security and Governance). Once the notes assistant is connected to a chat app, a message from anyone is untrusted input; it can also read private notes and send messages, so all three come together at once.

Each heartbeat is a full agent turn and incurs a model-call cost even if it does nothing. Shorter intervals cost more. Costs can also grow through expanding context (← S11 Context Engineering, the example above), stacked retries (← S6 LLM APIs and Application Basics), or a heartbeat starting a goal loop or chain of tool calls.

Going deeper: the split between heartbeat and cron is still shifting

Products split heartbeat and cron differently, and they’re still adjusting. OpenClaw’s docs show it migrating heartbeats into ordinary scheduled jobs. When designing your own system, the names don’t matter. What matters is distinguishing two kinds of triggers, “do a fixed thing at a fixed time” and “check periodically whether there’s anything to do,” and giving each its own budget.
A goal loop needs at least three stop conditions: an evaluator judges the goal met; the budget (steps, tokens, money) runs out; consecutive failures hit a threshold and it escalates to a human (← S12 The Agent Loop and Harness). With only the first, a broken run keeps burning money.
In the background, nobody is in front of the screen to approve actions, so high-risk actions should be “queued for approval, confirmed next time the user is online” rather than automatically allowed (← permission modes in S12 The Agent Loop and Harness, S16 Security and Governance).

★ How do you prove it’s ready to ship?

A demo only proves it ran once. The system has to prove it can ship, can be measured, and has a reason behind every decision. An always-on agent runs for long stretches with nobody watching, and the only thing you can rely on is evidence built in advance.

The capstone delivers five pieces of evidence: a regression suite (changes don’t break what already works), an infra variance report (score changes are real, not environment noise), a trifecta audit (new channels and schedules don’t open new attack paths), harness ablation notes (every harness component has a reason to exist), and an architecture document (it can ship, and you know what to do when something goes wrong). The first three and the fifth are mostly things you did in earlier chapters plus the always-on components. What’s genuinely new is how you read the scores, which takes two ideas.

How to start writing: the minimum format for each piece of evidence

Start every piece of evidence with these five lines. If you can fill them in, you have a first draft:

Question: what does this evidence need to answer?
Method: what was run, how many cases × how many runs, which two versions or which factor were compared
Numbers: the core numbers, with 95% confidence intervals or distributions
Conclusion: keep / remove / ready to ship, and which number it's based on
Limits: what wasn't tested, what to do next

Evidence	The “Numbers” line needs at least
Regression suite	Number of cases, pass^3, the paired-difference interval against the version at the end of A18 Multi-user Deployment, and δ set in advance
Infra variance report	The spread of scores over 5 runs of the same commit, and the gap after changing one factor
Trifecta audit	The number of all-three combinations and their countermeasures, and successes in the injection test on the new channel before and after the fix
Harness ablation notes	Tasks × runs, the 95% confidence interval of the paired difference
Architecture document	Runs, failures and recoveries over 7 days, daily cost, and the result of the 30-minute walkthrough test

First, measure how big the noise is. Rerunning the same commit (one saved version of the code) can change the score, and the cause may be nothing but the environment: tool timeouts, concurrency, container resources. As S14 Agent Evals, Data and Optimization mentioned, Anthropic found that changing only the container’s resource settings moved the success rate by about 6 percentage points between the most lenient and strictest settings, and recommends being skeptical of gaps under 3 points. So the infra variance report starts by running the same commit 5 times to see how wide the score distribution is; after that, any “it got better” gap has to be larger than that width to count.

Second, compare paired on the same set of cases. When comparing two versions, run them on the same cases, compute “new minus old” for each case, then compute the 95% confidence interval of the mean difference (← S8 Evals Methodology). Pairing cancels out the difficulty of individual cases, so the interval is much narrower than computing a separate interval for each version. Only cases both versions can run can be paired: the version at the end of A18 Multi-user Deployment has no schedules or notifications, so the new always-on behavior cases can’t be paired and are reported on their own as pass^3.

To show the new version “isn’t worse,” write down the allowed gap δ before running, say 5 percentage points; it passes only if the lower bound of the paired difference’s 95% confidence interval is no lower than −δ. δ must be set in advance; setting it after seeing the results makes it meaningless.

Example: an ablation removes compaction, 20 tasks × 5 runs. With it, 82 of 100 runs succeed; without it, 79, a difference of −3 percentage points. Suppose bootstrapping the differences over the 20 tasks gives a 95% confidence interval of [−7, +1] for the paired difference. The interval includes 0, so you can’t say compaction definitely helps; but the lower bound −7 is below −δ = −5, so you also can’t say removing it has no effect. The reasonable decision here is to keep it for now and narrow the interval with more tasks or more runs, or to look at cost: if removing compaction saves a lot of money per task, it’s worth another round of evals to settle the question.

Overlapping confidence intervals for the individual versions do not establish that the difference includes 0 or is smaller than δ. When both versions use the same cases, inspect the paired-difference interval.

○ What other agent domains are there?

Science agents, theorem proving and embodied agents apply agents to scientific discovery, formal proofs and physical robots. Each has its own specialized environments and evaluation methods, outside the scope of this route. If you’re interested, start from the evals, harness and security ideas in this course and then read a survey of that field.

Exercises

Concepts route (no coding): after “Try it first,” you can skip to the self-check.

Try it first (about 10 minutes)

One step in a trajectory reads: “The model calls web_clip to clip a web page, and a hook marks the returned content as untrusted.” Which two of the six parts does this step fall into? Why can one step fall into two?

Check your answer

Tools (the model takes an action) and security (the harness decides whether this content can be trusted). The six parts are questions to ask, not six modules in the code, so one step can answer two questions at once.
A heartbeat runs every 60 minutes, each carrying 4,000 input tokens, at $3 per million input tokens (an illustrative price; output ignored). How much does a month (30 days) cost?

Check your answer

24 × 30 = 720 runs; 720 × 4,000 = 2.88 million tokens; 2.88 × 3 ≈ $8.64. Even if every run finds “nothing to do,” you still pay.

Core: take apart an open-source coding agent

Goal: actually run an open-source coding agent, map each of its components to concepts you’ve learned, and measure its consistency and cost.

Steps

Pick an open-source coding agent. mini-SWE-agent is recommended (the code is short enough to read in full); if you have time, choose SWE-agent or OpenHands.
Prepare 5 tasks, each with tests that can decide success or failure automatically. A reproducible way to do it (the same way SWE-bench builds its cases):
- Pick a small open-source repo with tests that you can follow (for example a Python or TypeScript library under 2,000 lines).
- Find 5 commits in its git history that “fix a bug or add a small feature, and add tests at the same time.”
- For each task: check out the version just before that commit, add back only the new test files (the tests should now fail), and rewrite the commit message or the corresponding issue as the task description. The agent succeeds if it makes those tests pass without breaking the existing tests.
- If you can’t find 5 in the history, make up the rest in the same format: write the test first, then remove the implementation.
Run each task 3 times, recording success, number of steps, tokens, cost and time.
Read its code and one complete trajectory, and fill in a breakdown table: component (loop, tools and ACI, prompt, context management, stop conditions, permissions / sandbox, error handling, how it’s evaluated, and so on), where it is in the code, and which chapter it maps to.
Write down 3 design tradeoffs it makes, for example “it gives the model no specialized tools, only bash,” and what you think each costs. Compare at least one against Agentless’s fixed three-phase workflow: on your 5 tasks, which approach fits better?

Acceptance criteria

A results table of 5 tasks × 3 runs, with success rate, pass^3 (the share of tasks that succeed all 3 times, ← S8 Evals Methodology), and average cost and time per task.
A breakdown table with ≥ 10 rows, each with a code location (file and line number, or function name) and the chapter it maps to.
3 design tradeoffs, each with a concrete example you saw in the trajectory; at least one includes a comparison with Agentless.

Hints

TypeScript: mini-SWE-agent is a Python project, but your task repo can be TypeScript (for example a small project with Vitest tests); compare the breakdown table against the harness you wrote yourself (← S12 The Agent Loop and Harness).
Python: pip install mini-SWE-agent directly and use pytest for the task repo.
Cost: 15 runs on a mid-size model is a few dollars; you can also hook up a local model, but the success rate will drop noticeably, so note that in your report. The agent’s prompt plus command output easily exceeds Ollama’s default context, so if you use Ollama, increase the context length first (see S6 LLM APIs and Application Basics).
If you get stuck:
- Reread: How do you take apart any agent?, How are coding agents designed?
- Further: the ACI design section of the SWE-agent paper, Claude Code’s official best practices

Advanced: System project A19 (capstone): Turn your system into an always-on agent

Goal: extend your system into an always-on agent, and deliver five pieces of engineering evidence ready for review.

What this project is: take the system you’ve built from S6 LLM APIs and Application Basics all the way through S18 Deployment and Architecture, and extend it into an always-on agent that works without anyone opening it. It runs tasks on a schedule, proactively notices things that need handling, notifies the user through at least one channel, and pursues goals within limited permissions and budget. If your topic genuinely doesn’t suit always-on operation (for example it’s inherently a one-off analysis tool), you can do another kind of “real system extension” instead: make it a coding agent that maintains its own repo, add a deep research mode, or connect it to a real external workflow. The requirements for the five deliverables stay the same.

Why these five deliverables: what defines an always-on agent is that it runs for long stretches with nobody watching it. The only thing you can rely on then is evidence built in advance.

Deliverable	What it proves	Chapters it draws on
Regression suite	Changes don’t break what already works (evaluable)	S8 Evals Methodology, S14 Agent Evals, Data and Optimization
Infra variance report	Score changes are real, not environment noise (evaluable)	S14 Agent Evals, Data and Optimization
Trifecta audit	New channels and schedules don’t open new attack paths (defensible)	S16 Security and Governance
Harness ablation notes	Every harness component has a reason to exist (defensible)	S12 The Agent Loop and Harness, S14 Agent Evals, Data and Optimization
Architecture document	It can ship, it can scale, and you know what to do when something goes wrong (shippable)	S17 Product and AI UX, S18 Deployment and Architecture

Steps

Always-on components: at least one cron job, one heartbeat or event-triggered proactive check, one notification channel (pick one: email, chat app, webhook or desktop notification), and one goal loop with stop conditions and a budget cap. State must be recoverable after a process restart (← S12 The Agent Loop and Harness, S18 Deployment and Architecture).
Proactive behavior design (← S17 Product and AI UX): define a notification budget, what’s worth interrupting the user for, and how users snooze or turn off a category of notifications.
Regression suite: at least 50 cases covering the original features and the new always-on behavior (for example “no notification when there are no new events”), each run 3 times, wired into the CI/CD pipeline from S18 Deployment and Architecture.
Infra variance report: run the whole suite 5 times on the same commit and report each run’s score and the variance; then change one infrastructure factor (for example tool timeout, concurrency, or the model’s region or version) and see whether the score change exceeds that variance range.
Trifecta audit: redo the audit from S16 Security and Governance, including the new channels, scheduled triggers and background execution, and fix any new all-three combinations.
Harness ablation: remove or replace one harness component (for example compaction, a hook, the planning step, a skill), compare on ≥ 20 tasks × 5 runs, and include the 95% confidence interval of the paired difference.
Continuous operation: let it actually run for at least 7 days, recording the number of runs, failures, recoveries, notifications sent, the share of notifications users acted on, and total cost.
Architecture document: build on the architecture document from A18 Multi-user Deployment, adding the always-on components, the 7-day operating numbers, known limits and next steps.

Acceptance criteria

A regression suite of ≥ 50 cases running in the CI/CD pipeline, reporting pass^3 and a 95% confidence interval. Paired comparison uses only cases both versions can run: on the original-feature cases, the final version is no worse than the version at the end of A18 Multi-user Deployment (write down the allowed gap δ before running, say 5 percentage points; the lower bound of the paired difference’s 95% confidence interval is no lower than −δ). That version has no schedules or notifications, so the new always-on behavior cases can’t be paired and are reported on their own as pass^3.
Infra variance report: the score distribution over 5 runs of the same commit, and the effect of at least one infrastructure factor; the conclusion says how big a score difference has to be to count as “really better or worse.”
The trifecta audit covers all new channels and schedules; every all-three combination has a countermeasure; at least 1 injection test targets a new channel, with 0 successes in 5 runs after the fix.
Ablation notes: at least 1 component, 20 tasks × 5 runs, with the 95% confidence interval of the paired difference in the conclusion and a keep / remove decision.
7-day operating log: the number of automatic recoveries without human intervention, notifications sent and the share acted on, and daily cost; if any incident needed manual repair along the way, include a short postmortem.
The architecture document lets an engineer who has never seen the code explain the system’s main components, data flow, trust boundaries and rollback method within 30 minutes (actually try it with someone).

The notes assistant as an example: every morning it organizes the previous day’s new notes and emails a summary (cron); every 30 minutes it checks whether any note mentions something coming due, and only sends a reminder if so (heartbeat); Sam says in a chat app, “Turn this month’s reading notes into an outline for me,” and the agent works across many turns until it’s done (goal loop). The new channel means any message sent to that chat app by anyone is untrusted input, so the trifecta audit has to be redone.

Hints

TypeScript: use node-cron or the system’s launchd / systemd timers for scheduling; the simplest notification channels are the Telegram Bot API, a Slack incoming webhook or email; durable state can reuse the database or checkpointing from S18 Deployment and Architecture.
Python: APScheduler or the system scheduler; Temporal’s Python SDK suits long-running tasks; notifications as above.
Cost (evals): one round of the regression suite is 50 × 3 = 150 agent runs; the infra variance report runs 5 rounds on the same commit, so 4 rounds beyond that first one (4 × 150 = 600 runs), plus 1–5 rounds after changing a factor (150–750 runs), for 750–1,350 runs; the ablation is 2 × 20 × 5 = 200 runs. In total about 150 + 750–1,350 + 200 = 1,100–1,700 agent runs, not counting the suite on every PR in CI. Use the cost accounting from S8 Evals Methodology to estimate the average cost per run and multiply; run the pipeline with a small or local model first, and only use the production model at the end; if you use Ollama, increase the context length first (see S6 LLM APIs and Application Basics).
Cost (always-on): every heartbeat is a full model call. Work out “runs per day × cost per run” before deciding the interval; the first check layer can use rules or a small local model, calling the cloud model only when needed.
If you get stuck:
- Reread: What does an always-on agent add?, How do you prove it’s ready to ship?; the documents you wrote yourself in S16 Security and Governance and S18 Deployment and Architecture
- Further: the s11 (background tasks), s12 (scheduling) and s17 (goal loop) lessons in learn-claude-code; Anthropic’s “Infrastructure noise in agentic coding evals”

Note: during the 7 days of continuous operation, you only need to spend a little time each day checking on it.

Self-check

In your own words: Agentless gets competitive results with a fixed workflow. Does that mean agents are unnecessary? What kinds of tasks make a fixed workflow insufficient?

Hint

Key point: no; the benchmark’s task shape is fixed, which is why a fixed flow can cover it. When you can’t know the next step until you see the previous step’s result (you have to reproduce the bug first, read logs, the issue is unclear), a workflow falls short. Common mistake: assuming a high benchmark score means real tasks also suit a fixed flow. Reread: How are coding agents designed?
Deep research systems use about 15× the tokens of a regular chat, and about 3.75× those of a single agent (← S15 Multi-agent Systems). Given a new research-style requirement, what questions would you use to decide whether a multi-agent architecture is worth it?

Hint

Key point: whether the answer is worth more than the cost, whether it splits into independent parallel facets, whether there’s more data than one context holds, and how costly a wrong or incomplete answer is; if the first two are no, a single agent is usually enough. Common mistake: assuming multiple agents are always stronger, without counting the cost. Reread: How do research agents divide the work?
Scenario: your always-on agent runs a heartbeat every 30 minutes, and the monthly bill is 4× what you expected. List three possible causes and how you’d investigate each.

Hint

Key point: the context each run carries grows over time, failed retries pile up, and a heartbeat triggers a goal loop or a chain of tool calls; compare token counts, retry counts and tool call counts per run from the traces. Common mistake: just blaming a higher model price without looking at tokens per run. Reread: What does an always-on agent add?
Your ablation shows that removing the planning step drops the success rate on 20 tasks × 5 runs from 88% to 84% (−4 percentage points), and the two versions’ 95% confidence intervals overlap. Do you keep or remove the component? What else do you need to know? (Hint: does overlap tell you anything about the difference? See the paired comparison in “How do you prove it’s ready to ship?”)

Hint

Key point: overlap between the two versions’ intervals says nothing about the difference; look at whether the paired difference’s interval includes 0 and whether its lower bound is below −δ. Common mistake: seeing overlap and saying “no difference, remove it.” Reread: How do you prove it’s ready to ship?
Scenario: someone says “My agent demo runs great, it’s ready to ship.” Based on the capstone’s five deliverables, what five questions would you ask them?

Hint

Key point: each question maps to one piece of evidence: will changes break old features, is a score change just noise, does it hit all three, does every harness component have a reason, how do you roll back when something goes wrong. Common mistake: offering one successful demo as the evidence. Reread: How do you prove it’s ready to ship?
Pick one feature of the notes assistant (or your own system), for example “email a summary every morning,” and use the six-part map to say what its loop, tools, context, memory, evals and security are, and which chapter each maps to.

Hint

Key point: fill in all six; “none” counts as a choice too. For example, the daily summary’s loop is triggered by cron, and for security you check whether sending the email needs human approval. Common mistake: leaving the parts it doesn’t have blank. Reread: How do you take apart any agent?
Without looking at the lecture, draw the flow of a research question through a deep research system from the lead agent to the citation agent, marking which steps run in parallel and which contexts are isolated from each other.

Hint

Key point: the lead agent breaks down the question → several sub-agents search in parallel, each with an isolated context, returning only a summary → the lead agent synthesizes → the citation agent ties claims back to sources. Common mistake: having sub-agents stuff their full search results back into the lead agent. Reread: How do research agents divide the work?

S19 Real Agent Systems and the Capstone

What this chapter covers

After this chapter you can

Lecture

★ How do you take apart any agent?

★ How are coding agents designed?

★ How do research agents divide the work?

★ What does an always-on agent add?

★ How do you prove it’s ready to ship?

○ What other agent domains are there?

Exercises

Try it first (about 10 minutes)

Core: take apart an open-source coding agent

Advanced: System project A19 (capstone): Turn your system into an always-on agent

Self-check

Further reading

Official docs and engineering articles

Papers

Tools

Courses

After the route