step 11 / 19Part III Knowledge & Tools

S11 Context Engineering

What this chapter covers

A model call may include system instructions, tool definitions, retrieved passages, tool output and conversation history. Context engineering decides which material to include, how much space to give it and how to order it. This chapter covers token budgets, the effects of long context, truncation and summaries, and ordering for prompt caching.

This chapter covers context assembly for one call. S12 The Agent Loop and Harness handles growth across agent turns with compaction, resets and sub-agent isolation.

After this chapter you can

List the parts of one call’s context and the order they go in, and say which part to cut first when over budget, and why. (lecture and self-check)
Recognize the type of context failure from a trace and propose a matching fix. (lecture and self-check; in the core exercise you’ll see the effects of length and position firsthand)
Order the context so prompt caching works, and measure the cache hit rate. (core)
Plot a “context tokens vs quality” curve. (core; using it to choose a budget is in the advanced exercise)
Measure each part’s token count and set budgets. (advanced)

Lecture

Before each call, a context builder collects available material, measures its token count, truncates or summarizes parts that exceed their budgets, and assembles the request in a fixed order. It must decide:

Which parts there are and how much budget each gets.
Why you can’t put everything in (degradation in long contexts).
How detailed the system prompt should be, and whether to load data up front or let the model fetch it.
What to cut when over budget.
How to order the context so prompt caching kicks in.

The running example follows one request to the notes assistant.

★ How do you budget the context?

Budget each part of the context separately. Every part adds cost and latency (← S6 LLM APIs and Application Basics) and uses some of the model’s attention.

One call’s context usually includes: tool definitions (each tool’s name, description and parameter schema, ← S10 Tools and MCP), the system prompt, retrieval results (← S9 Retrieval and RAG), tool results (what came back from the model’s previous tool call), conversation history, memory (user preferences remembered across conversations, → S13 Memory and Personalization), and the user’s input for this turn. The context window limit counts the output too, so you also have to leave room for the answer.

Anthropic describes this limit as an attention budget. Each extra token slightly dilutes the influence of the others. Aim to give the model as few tokens as possible while retaining the information most likely to produce the desired result.

Sam asks the notes assistant: “What budget did we settle on in the last meeting with the design team? Create a to-do reminding me to follow up next week.” The request proceeds as follows:

The context builder first retrieves notes using the question (← S9 Retrieval and RAG) and gets the 8 most relevant chunks, about 700 tokens each.
First call: after reading them, the model decides it also needs the quotes, so it calls notes_search("quote"). The tool returns 20 results, about 400 tokens each.
Second call: the tool results go back into the context, and the model answers and calls notes_create to create the to-do.

The table below is the context for the second call. The total budget is 16K input tokens (the part budgets add up to 16,000; the user’s input for this turn has no cap), well below the model’s context window, for reasons the next section explains.

Part	Content	Budget	Raw size	Actually included
Tool definitions	`notes_search`, `notes_create`, `web_clip`, `email_draft`	2,000	1,600	1,600
System prompt	Role, answer rules, citation format	1,500	1,400	1,400
User profile	Name, time zone, today’s date	500	300	300
Conversation history	The previous 15 turns	4,000	9,000	2,900
Retrieved notes	8 chunks × 700	6,000	5,600	5,600
Tool results	20 `notes_search` results × 400	2,000	8,000	1,750
This turn’s input	The user’s question	—	100	100
Total			26,000	13,650

Two parts are over budget, and each is cut down to fit (how to cut is in What do you cut when over budget?):

History: keep the last 4 turns verbatim (4 × 600 = 2,400) and replace the earlier 11 turns with a 500-token summary, 2,900 in total.
Tool results: keep the first 3 results in full (3 × 400 = 1,200), keep only the titles of the other 17 (17 × 30 = 510), plus a truncation note of about 40 tokens, 1,750 in total.

Result: each call’s input drops from 26,000 to 13,650 tokens, roughly half, and everything the answer needs is still there. Separately, max_tokens reserves about 1,000 tokens for the answer.

A total budget alone lets growing tool results and history crowd out useful retrieval results. Log each part’s token count in the trace, then assign budgets. Many systems find that tool definitions or long history occupy more than half the context, while the useful retrieved passages take only a small portion.

Before you build: how to measure each part's token count

The usage field is the source of truth: the input token count in the API response is the real billed number, but it’s only a total, not broken down by part.
Per-part counts are on you: calibrate by measuring each part separately with your provider’s token counting API, then use an approximation for real-time counting (for example, a tokens-per-word ratio measured on your own data). The ratio varies by tokenizer and by language (Chinese text, for instance, comes out very differently from English), so don’t borrow someone else’s number.
Trace fields: for every call, log each part’s token count, how many items were truncated, how many tokens were dropped, and cache read/write token counts (see the ordering section). The advanced exercise A11 Context Builder (what the codes mean) requires these fields on 100% of calls.
Write the builder as a pure function: material in, messages out, no side effects, so you can use snapshot tests (← S6 LLM APIs and Application Basics) to confirm ordering and truncation don’t change unexpectedly.

★ Why does more context make things worse?

Fitting in the context window doesn’t mean the model uses it well. The longer the input, the more likely the model misses things, gets distracted, or gets led astray by wrong information, usually without raising any error.

Chroma tested 18 models and found less stable performance as input grew, even on simple retrieval and copying tasks. It called this context rot. Incorrect distractor passages related to the answer reduced accuracy further, as did less overlap between the question’s wording and the answer passage.

Two studies introduced in S9 Retrieval and RAG found related limits. RULER showed that usable context can be shorter than advertised. Lost in the Middle (Liu et al. 2023) found that the same information is more likely to be missed in the middle of a long context than at either end.

Drew Breunig groups context failures into four types beyond input length. Here is how they can affect the notes assistant:

Type	What it is	Notes-assistant example	Common fix
Distraction	The history is so long that the model repeats what the history did instead of judging the next step itself	The history has 10 `notes_search` calls, and the model searches again with an almost identical query	Summarize the older history
Poisoning	Wrong content (an answer the model made up, a tool result that came back wrong) stays in the history and keeps being cited as fact	In turn 3 the model wrongly says “the budget is $50,000,” and every later answer reuses that number	Remove or correct the wrong content; don’t let it stay in the history
Confusion	Content irrelevant to this task is included and throws off the model’s judgment	40 unneeded tools are connected, and the model picks the wrong one	Include only the tools this task might use (tool search, ← S10 Tools and MCP)
Clash	The history contains statements that contradict each other, and the model doesn’t know which to trust	Early on the user says “My time zone is Chicago,” later “I moved to Tokyo,” and both are in the history	Overwrite with the newest information instead of keeping both

The 13,650-token version may answer more accurately than the 26,000-token version, because the 17 full quotes and 11 old turns removed from it are mostly distraction. Removing them also reduces cost.

A 200K context window does not establish that 100K will work well. Measure usable length on your own data, as in the core exercise, and test information at different positions, especially in the middle.

Going deeper: Chroma's experimental design and the source's limits

Chroma’s experiments treat input length as the only variable, holding task difficulty fixed, and watch how performance changes with length. Some of the manipulations: adding distractor passages similar to the answer but wrong, and varying the semantic similarity between the question and the answer passage. The core exercise’s design (needles, same-domain distractor passages, questions worded differently from the passages) is a scaled-down version of this approach.
Chroma is a vector database vendor, and the conclusion “long context is unreliable” favors its product. It published its methodology, and the core exercise has you rerun a scaled-down version of it yourself.
The name “context rot” comes from Chroma; the four types distraction, poisoning, confusion and clash come from Breunig. Classifications in different articles overlap; what matters is being able to recognize them in a trace.

★ How detailed should the system prompt be?

Start with a concise system prompt that states the principles and required format. Add instructions when you observe failures. Anthropic calls this level of detail the right “altitude.”

Hard-coded if-else rules make a prompt brittle and hard to maintain: “If the user asks about the budget, first search notes containing ‘budget’; if nothing turns up, search for ‘quote’ instead; if there’s still nothing…” At the other extreme, “You are a helpful notes assistant” gives little guidance. Use principles the model can follow: “Check the retrieved notes before answering. Cite the note’s title and date. If you can’t find the information, say so rather than guess. Confirm the content before creating or editing a note.”

Use the error analysis from S8 Evals Methodology to decide what to add. Each instruction should address an observed failure category.

Rules can conflict, as in the previous section’s clash, and dilute each other’s influence. A prompt with 60 rules can usually be reduced to a dozen or so principles and a few examples.

Custom context formats. HumanLayer’s 12-factor-agents lists 12 agent engineering principles plus an appendix. Factor 3, “Own your context window,” suggests serializing events, tool results and state into your own compact text format within one message. Serialization turns data into text that can be stored or sent.

This gives you control over what the model sees and can reduce tokens, but sacrifices provider support for standard messages, such as native tool-call parsing. Compare the formats with evals before adopting it.

Before you build: what a custom format looks like

Write what happened in this notes-assistant turn as a single message, for example:

<history_summary>The user is organizing the design team's project; prefers answers as bullet points.</history_summary>
<event type="search" query="quote" results="20" shown="3" />
<note title="2026-10-14 Design team meeting" date="2026-10-14">...</note>
<question>What budget did we settle on?</question>

This format removes each message’s role markers and JSON wrapper, and puts the search query, result count and displayed count on one line. Compare quality and token use with an eval before choosing it (← S8 Evals Methodology).

★ Load data up front or fetch it when needed?

You can load data before a call or give the model references and tools to fetch it as needed. Systems often combine the two approaches based on usage and latency.

Pre-loading puts potentially relevant data into context before the call, as in S9 Retrieval and RAG. Just-in-time loading gives the model references such as file names, queries and links, plus tools to retrieve the content. Claude Code uses grep to search file contents and glob to find files by name pattern this way. Pre-loading is fast and predictable. Just-in-time loading saves context and handles unanticipated needs, but adds latency and requires well-designed tools and guidance.

Run the numbers for the notes assistant. Pre-loading 8 note chunks every time adds 5,600 tokens to every call, even when the user just says “create a to-do for me” and needs no notes at all. Switch everything to just-in-time, and asking about the budget takes an extra round: the model calls notes_search first, and the model is called again once the results come back, adding a round trip of latency, and the second call resends the whole context. So the common compromise: retrieve up front when the question clearly needs notes, and otherwise only provide the tools. Factor 13 in the 12-factor-agents appendix also points out: if you know the model will need some data, fetch it up front instead of spending a round letting the model get it.

Progressive disclosure is a just-in-time design: give a summary or index first, and expand the details when the model needs them. For example, the notes assistant first provides a list of 200 note titles (about 30 tokens each, 6,000 in total), and the model reads the full text of the ones it picks; including every full note would be 200 × 1,500 = 300,000 tokens. Tool lists work the same way: give a one-line description first and load the full schema only once a tool is picked (tool search, ← S10 Tools and MCP). The skills in S12 The Agent Loop and Harness are built on this idea.

Every extra tool round resends the preceding context. If data is needed on almost every request, pre-loading uses fewer tokens and is faster than fetching it just in time.

★ What do you cut when over budget?

Tool results and conversation history are the two fastest-growing parts of the context. Cap every kind of tool result, replace older content with summaries, and always tell the model that something was cut.

Tool results. One search that returns 50 full results, or one file read that returns a whole file, fills the context fast. Common approaches:

Paginate and truncate by default: searches return only the top few results unless asked for the next page.
Return only the fields needed: a note’s title, date and summary, not all its metadata.
For long content, include only a summary and a reference: the model reads the full text with a tool when it needs it.
Clear old tool results in later calls, or replace them with one line of “what was called, how many results came back.” This is the lightest form of compression.

Conversation history. Keep the last few turns verbatim and replace earlier ones with a summary. In the table’s example, 9,000 tokens of history become “the last 4 turns verbatim + a summary of 11 turns,” 2,900 tokens in total. The summary should keep facts that will matter later (the user’s decisions, preferences, to-dos) and drop small talk and steps that are already done.

Example: the truncation note. The table’s tool results include only the first 3 in full. If you include just those 3, the model will think there are only 3 quotes. So add one line: “20 results in total. Below are the first 3 in full and the titles of the other 17; to see the full text of one, call notes_search with its title.”

The model can only see the context you provide. State that results were truncated, give the total count, and explain how to fetch more. Otherwise it may treat a partial result as complete and answer confidently from incomplete information.

Before you build: setting truncation rules

Fix the drop order: for example, cut tool results first, then compress history, and touch retrieval results last; never cut the system prompt or the user’s input for this turn. A fixed order makes behavior predictable and testable.
Don’t rewrite the summary every time: the summary sits at the start of the history, and if it’s rewritten every turn, nothing after it can hit the cache (see the next section). A common approach is to fold the oldest batch into the summary only after a set number of turns (say every 10).
Who writes the summary: a cheap small model is fine, but spot-check it with the S8 Evals Methodology approach for dropped decisions or numbers; a fact the summary drops no longer exists as far as the model is concerned.
The truncation note costs tokens too: leave room for that line in the budget.

★ How do you order the context to hit the cache?

Put stable content before changing content so repeated prefixes can be cached, reducing cost and latency. Claude requires an explicit cache marker.

Why there’s a discount. As the model processes input, it stores each token’s intermediate results as the KV cache (← “KV cache” in the Model Crash Course; deeper: S5 Inference and Serving). Prompt caching reuses the KV cache of an identical prefix on the server side: as long as the beginning of this request is exactly the same as some recent request’s, that portion doesn’t have to be recomputed.

Pricing. As of when this was checked (2026-11), cache reads commonly cost a tenth (0.1×) of the normal input price, and less for some models. Cache writes may cost extra; for example, Anthropic’s 5-minute cache writes cost 1.25× and its 1-hour cache 2×. Caches have a lifetime and prefixes have a minimum length, which differ by vendor and model; go by the official docs (details in the expandable section below).

A worked example. Suppose the notes assistant adds more tools and attaches note templates to the system prompt, so the fixed prefix (tool definitions + system prompt + shared documents) grows to 20,000 tokens and is used 50 times while the cache is live:

Without caching: 50 × 20,000 = 1,000,000 input tokens.
With caching: 1 write (20,000 × 1.25 = 25,000) plus 49 reads (49 × 20,000 × 0.1 = 98,000), the price of about 123,000 tokens.

That saves about 88% on the prefix, and latency drops too.

How to order it. The cache only recognizes the part that is “identical from the very start”; everything after the first differing token has to be recomputed. So the notes assistant’s context is ordered by how often each part changes:

Tool definitions (in a fixed order)
System prompt
User profile (name, time zone, date: different per user and per day)
Conversation history (append only)
This turn’s input, with the retrieved notes attached
Tool calls and tool results (in the standard messages format these follow the model’s tool call)

Parts 1 and 2 are shared by all users; part 4 grows turn by turn within a conversation while its beginning stays the same. The user profile goes in a separate block after the system prompt, not at the start, so it doesn’t invalidate the cache for every user, every day.

With Claude, you turn caching on yourself. Claude’s prompt caching is opt-in: you mark the last block of the prefix you want cached with cache_control, and without it there are no cache reads at all. In the ordering above, mark the last block of part 2 (the system prompt) (another way to mark it is in the expandable section below). OpenAI and Gemini cache sufficiently long prefixes automatically, no setup needed. Whichever vendor you use, confirm hits with the usage field in the response (e.g. Claude’s cache_read_input_tokens).

Timestamps at the start of the system prompt, tool lists in changing order, and rewritten history invalidate all cache entries after the change. Loading tools from multiple MCP servers without a fixed order, or rewriting a summary each turn, can cause this.

When moving code from OpenAI to Claude, add cache_control; otherwise cache reads remain at 0 and there is no discount. Check the cache setup and usage-field names whenever you change providers.

Before you build: caching details by vendor

Pricing (as of 2026-11). Cache reads: 0.1× for most Anthropic models, lower for the newest few (0.05× or 0.025×); mostly 0.1× for newer OpenAI models and 0.05× for a few, while older models may be 0.25× to 0.5×, so check each model’s pricing. Writes: Anthropic’s 5-minute cache is 1.25×, the 1-hour cache 2×.

Claude.

The cached prefix is computed in the order “tool definitions → system prompt → messages,” so the ordering above matches it exactly.
One request can mark up to 4 cache_control breakpoints, for example one after the system prompt and one on the last message of the history.
You can also put cache_control at the top level of the request: the breakpoint is placed automatically on the last block and moves forward as the conversation goes on, which is the simplest option for multi-turn conversations. But if the last block differs every time (e.g. the user’s new question) and no earlier request wrote the same prefix, it rarely hits (step 5 of the core exercise is exactly this case), so mark the fixed prefix’s last block yourself.
Usage fields: cache_creation_input_tokens is the number of tokens written to the cache this time, and cache_read_input_tokens the number read from it.
A cache entry’s lifetime resets whenever it’s read within that lifetime.

OpenAI. Automatic, best-effort caching; you can set prompt_cache_key to make requests sharing a prefix more likely to hit the same cache.

Local models. llama.cpp and Ollama also reuse prefixes, but there are no billing fields; look at the timing information the server returns (whether prefill time drops noticeably).

Minimum lengths, lifetimes and prices all change; check the official docs before you start.

○ Compress the prompt, or move knowledge into the weights?

LongLLMLingua (Jiang et al. 2023) uses a small model to delete the less important tokens in a prompt, compressing the input for long contexts while maintaining or even improving performance; the cost is that compressed text is hard for humans to read, and compression that comes out differently each time can conflict with caching. Another question worth thinking about: should a piece of knowledge go into the context every time, or into the weights through fine-tuning (← S4 Post-training and Reasoning, and the decision in S9 Retrieval and RAG)?

Exercises

Concepts route (no code): after “Try it first”, you can skip to the self-check.

Try it first (about 10 minutes)

1. The notes assistant’s total budget is 16K input tokens. This call’s material: tool definitions 1,600, system prompt 1,400, user profile 300, conversation history 2,800, retrieved notes 5,600, tool results 9,000 (30 full results), the user’s question 100, totaling 20,800. You need to cut about 5,000 tokens. Which part do you cut first, and why?

Check your answer

Cut the tool results first: they’re the biggest, most of the 30 full results are distraction, and anything cut can be fetched again with the tool. Don’t cut the system prompt or the user’s question; the retrieval results are the basis for the answer, so touch them last.

2. A 10,000-token fixed prefix is used 30 times while the cache is live. With writes at 1.25× and reads at 0.1×, how much does caching save on the prefix compared with no caching?

Check your answer

Without caching: 30 × 10,000 = 300,000. With caching: 1 write at 10,000 × 1.25 = 12,500, plus 29 reads at 29 × 10,000 × 0.1 = 29,000, for 41,500. That saves (300,000 − 41,500) ÷ 300,000 ≈ 86%. The common mistake is counting all 30 as reads and forgetting the first one is a write.

Core: Length, position and caching experiments

Goal: see for yourself, in your own experiment, how context length and the position of information affect accuracy, and measure how much prompt caching saves.

Steps

Prepare 20 needles: have an LLM generate 20 short reports about fictional companies (about 60–110 words each, each with one unique, askable fact, such as a number or a person’s name), then have it write a question for each report that doesn’t reuse the report’s wording. Fictional content guarantees the model can’t already know the answers. Check by hand that each of the 20 questions has exactly one correct answer.
Prepare distractor passages: use the same prompt to generate 250 more fictional company reports (same domain as the needles but without those 20 facts, and with no repeated company names), about 30K tokens in total. You can also mix in chunks from your S9 Retrieval and RAG document set.
Build contexts at 3 lengths (about 2K, 8K and 32K tokens) × 3 positions (start, middle, end), 9 conditions in all, running the 20 questions in each. Don’t set temperature (Claude won’t accept it; see S6 LLM APIs and Application Basics); use the model’s defaults, run each condition 3 times, and average accuracy over the 3 runs.
Record each condition’s accuracy (an answer counts as correct only if it matches the needle’s fact), average input tokens, average latency and cost.
Caching experiment: with a fixed prefix of about 10K tokens, send 30 different questions in sequence (not in parallel) while the cache is live, and record each request’s cache reads and writes from the usage field. With Claude, put the fixed prefix in the system prompt (or a separate first block) and mark its last block with cache_control; don’t use top-level cache_control on the request (see How do you order the context to hit the cache? for why). OpenAI and Gemini cache automatically.
Run it again, this time adding a line at the very start of the prefix with a timestamp that differs on every request (to the millisecond, or a per-request UUID instead; a timestamp only down to the minute can still hit the cache), and compare.

Acceptance criteria

A 3 × 3 accuracy table, each cell with a 95% confidence interval (with n = 20 the intervals will be wide, which is itself an observation worth noting).
A plot of “average input tokens vs accuracy.”
In the caching experiment, at least 28 of requests 2–30 have cache reads. That’s what Claude does with cache_control added per step 5 and requests sent in sequence; OpenAI’s and Gemini’s automatic caching is best-effort (with OpenAI you can set prompt_cache_key), and the hit rate may be a bit lower, so just record the actual numbers.
In the caching experiment, the actual cost computed from the usage fields, side by side with the cost without caching.
The version with the timestamp drops to 0 (or nearly 0) cache reads, with the reason written down.

Hints

TypeScript: the call layer from S6 LLM APIs and Application Basics is enough; always read token counts from the usage field; generate needles and distractor passages with structured output (← S7 Prompts, Structured Output and Workflows) and store them as JSONL.
Python: same as above, using the S6 LLM APIs and Application Basics call layer and structured output (← S7 Prompts, Structured Output and Workflows).
Use a cheap small model: 540 calls (9 conditions × 20 questions × 3 runs) averaging about 14K input tokens is about 7.5 million tokens in total, usually under $10 with a cheap model. The model’s context window must be larger than 32K (say 64K or more), because the 32K condition plus the question and output goes over 32K. Local models are free, but a 32K context can be very slow on a laptop, so you can do just 2K and 8K. With Ollama, raise the context length first (see S6 LLM APIs and Application Basics).
Local models (llama.cpp, Ollama) also have prefix caching, but you measure it differently: look at the timing information the server returns.
If you get stuck:
- Reread: Why does more context make things worse?, How do you order the context to hit the cache?
- Further: the experimental design section of Chroma’s “Context Rot”; the usage-field explanation in the official prompt caching docs

Note: about half the work is preparing the needles and distractor passages.

Advanced: System project A11: Context builder

Goal: consolidate the scattered context-assembly logic in your system into one context builder that has budgets, a fixed order and is cache-friendly, and choose the budget from experimental results.