step 06 / 19Part II Applications

S6 LLM APIs and Application Basics

What this chapter covers

This chapter covers reliable model calls: request structure and cost, streaming and cancellation, errors and retries, SDK differences, local models, reasoning models, and logging and testing variable outputs. Later chapters depend on these features. S8 Evals Methodology uses traces for error analysis, S12 The Agent Loop and Harness needs reliable retries and cancellation, and S18 Deployment and Architecture uses per-call token counts to track costs.

After this chapter you can

Explain why multi-turn conversations get more expensive as they go, which errors should be retried, and how an idempotency key prevents a side effect from running twice. (lecture and self-check)
Describe the messages structure of an API call and work out what a call cost from the response’s usage field. (lecture and self-check; the core exercise has you do it for real)
Write a call layer that streams, can be canceled, and retries correctly on 429 / 5xx. (core)
Switch between a cloud API and a local model with the same code, and name the tradeoffs. (core)
Log the token counts, cost and latency of every call. (core uses a JSONL log; the OpenTelemetry trace is in the advanced exercise)
Use record / replay to write tests that never hit the real API and give the same result every run. (core)

Lecture

The running example is the notes assistant. Sam (user ID u_017) has about 200 Markdown notes and asks, “What did I write about Project X last week?” The first version puts the full text of the 20 most recent notes in the system prompt, about 30,000 input tokens per request. S9 Retrieval and RAG will reduce that.

Your code must assemble the request, stream the answer, retry on errors, and record tokens, cost and elapsed time. The sections below cover those steps, along with switching providers, running local models and using reasoning models.

★ What does one API call look like?

A call sends “a system prompt + a list of messages with roles” and gets back some output plus a token usage report. The cost is computed from that usage.

To run this chapter’s examples yourself, you need:

Basic Python (or TypeScript): writing functions, reading dicts / objects.
Installing packages: for example pip install anthropic or npm install @anthropic-ai/sdk.
Setting environment variables: the API key goes in an environment variable (such as ANTHROPIC_API_KEY), never in the code.
Knowing what HTTP and JSON are: an API call is an HTTP request, and what you send and get back is JSON.

(If any of these are unfamiliar, start with the Glossary.)

If you’re not writing code, you can skip the Python / TypeScript snippets; do read the prompt, note and JSON examples.

Using Anthropic’s format, one notes-assistant call looks roughly like this (to keep the numbers easy, this example includes only 2 notes):

{
  "model": "…",
  "system": "You are a notes assistant. Answer only from the notes below. <notes>…2 notes…</notes>",
  "messages": [
    { "role": "user", "content": "What did I write about Project X last week?" }
  ],
  "max_tokens": 500
}

Messages have a role: user is what the user said, assistant is the model’s earlier replies. Where the system prompt (the global instructions) goes differs by vendor; here it’s the top-level system parameter. A message’s content can be several blocks: text, images, tool calls, tool results. max_tokens is the most tokens this call may output.

With the official Python SDK, the smallest possible call is:

import anthropic

client = anthropic.Anthropic()  # reads the key from the ANTHROPIC_API_KEY environment variable
MODEL = "……"                     # use a model name listed in the official docs

response = client.messages.create(
    model=MODEL,
    max_tokens=1024,
    system="You are Sam's notes assistant. Answer only from the notes below; if the notes don't say, say you don't know.\n<notes>……</notes>",
    messages=[{"role": "user", "content": "What's the budget cap from the last design team meeting?"}],
)

print(response.content[0].text)                   # the model's answer
print(response.stop_reason)                       # e.g. "end_turn"
print(response.usage.input_tokens, response.usage.output_tokens)

The TypeScript SDK (@anthropic-ai/sdk) has the same shape; OpenAI’s SDK uses different field names (see How do you switch providers without a rewrite? below). This is only the minimal call; the core exercise adds streaming, retries and logging.

Besides the text, the response has a usage field, for example {"input_tokens": 2000, "output_tokens": 300}. Pricing is per token (for what a token is, see Model Crash Course, “Tokens and tokenization”; deeper: S2 Transformers and Decoding), and input and output are priced separately, with output usually several times more expensive:

cost = input tokens × input price + output tokens × output price (+ cache reads/writes, → S11)

With illustrative prices ($3 per million input tokens, $15 per million output tokens), this call costs 2,000 × 3 / 1,000,000 + 300 × 15 / 1,000,000 = 0.006 + 0.0045 = $0.0105. For the real first version (20 notes, about 30,000 input tokens), the input alone is 30,000 × 3 / 1,000,000 = $0.09. Don’t estimate token counts yourself; read usage. Tokenizers differ by vendor, and the same text can produce very different token counts on two vendors’ models, especially for non-English text.

The model does not remember the previous turn. LLM APIs are essentially stateless: every call has to resend the whole conversation, including the system prompt, the earlier user and assistant messages, and tool results. So the cost of a multi-turn conversation adds up. Suppose (again with only 2 notes) the notes plus system prompt are 1,800 tokens, each question 200 tokens and each answer 300 tokens. Turn 1 has 2,000 input tokens. Turn 2 has to carry turn 1’s question and answer, so it’s 2,500, and each turn after that adds 500, so turn 10 is 6,500. The input over 10 turns totals 42,500 tokens, more than twice the “2,000 every turn” you might expect. There are two kinds of fixes: keep the unchanging start of the request so it hits prompt caching (→ S11 Context Engineering), and truncate or summarize old turns (→ the tool-result budget in S11, compaction in S12 The Agent Loop and Harness).

A Batch API lets you submit many requests at once and collect the results asynchronously (usually within 24 hours) at roughly half the price of real-time calls. It suits work that isn’t urgent, such as generating summaries for a thousand notes, or running evals (→ S8 Evals Methodology).

Going deeper: message format differences and server-side conversation state

Where the system prompt goes: Anthropic uses the top-level system parameter; OpenAI treats it as a kind of message (the system or developer role), and the Responses API also accepts an instructions parameter.
Server-side conversation state: some APIs offer optional server-side state, such as previous_response_id in OpenAI’s Responses API, so you don’t have to resend the history yourself. But the model still reads the whole history on every call and you’re still billed for those input tokens, so the cost buildup above doesn’t go away.

★ How do you stream the answer?

Streaming lets the user see the first words while the model is still generating. When the user presses “Stop”, you have to actually close the connection.

Say the notes assistant’s answer is 300 tokens and takes 6 seconds to generate in full. Without streaming, the screen stays blank for 6 seconds. With streaming, the first word shows up after about 0.8 seconds and the rest keeps coming. Perceived latency depends on TTFT (time to first token, the time before the first token appears; see Model Crash Course, “Cost and latency basics”, deeper: S5 Inference and Serving), not total latency.

Most APIs use SSE (Server-Sent Events): an HTTP response that stays open and sends lines of event: and data:, each event carrying a small piece of text:

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Last week"}}

Your code reads line by line and appends each piece of text to the display. Just before the last event there’s usually one carrying the final usage; compute the cost from that.

Two common pitfalls:

Errors mid-stream: once the HTTP response has returned 200, an error can still arrive as an error event in the middle of the stream. Your code can’t rely on the status code alone.
Partial JSON: tool call arguments (→ S7 Prompts, Structured Output and Workflows, S10 Tools and MCP) are a piece of JSON that arrives in fragments while streaming, and the fragments in between aren’t valid JSON. Either wait until the block ends before parsing, or use a parser that handles incomplete JSON for progressive display.

Stopping screen updates leaves the model generating on the server, with billing continuing. Close the connection to cancel: use AbortController in JS, or cancel the task or close the response in Python. Generated tokens are usually still billed, so cancel as early as possible.

Before you build: event order and browser limits

Anthropic’s event order is roughly message_start → for each content block content_block_start / content_block_delta / content_block_stop → message_delta (with the final usage) → message_stop. Tool argument fragments come as input_json_delta.
In the browser: the native EventSource can only send GET, while LLM requests are usually POST, so frontends mostly parse the stream themselves with fetch plus ReadableStream, or leave it to the SDK. Python clients (httpx, the official SDK) don’t have this limit; just read the response line by line.
Measuring TTFT: start the clock when you send the request and stop it at the first text chunk, not the first event of any kind (message_start carries no text).

★ Should you retry on an error?

Some errors go away if you wait; others fail the same way a hundred times. Classify first, then decide whether to retry, how long to wait and when to give up.

Retryable: 429 (rate limit, too many requests), 500, 502, 503, 504, 529 (Anthropic’s overloaded), network drops, timeouts. Not retryable: 400 (malformed request), 401 / 403 (key or permissions), 413 (request too large).

Wait before retrying, and wait longer each time: exponential backoff plus random jitter. Jitter keeps all the clients from retrying at the same instant and knocking over a service that just recovered. If the response has a retry-after header, wait that long. Cap the number of retries.

The notes assistant’s call layer allows at most 3 retries with a base wait of 1 second:

Situation	Attempts	Result
429 with `retry-after: 2`	Attempt 1 gets 429 → wait 2 seconds → attempt 2 succeeds	2 attempts total
Repeated 500s	Attempt 1 → wait (cap about 1 second) → attempt 2 → wait (cap about 2 seconds) → attempt 3 → wait (cap about 4 seconds) → attempt 4	Reports an error only after all 4 fail
400	Attempt 1	No retry; reports the error right away

The actual wait is random, between 0 and the cap. This is jitter; the expandable section below gives the formula. Official SDKs usually retry automatically. Disable their retries before adding your own so the two layers do not multiply attempts.

There are at least two kinds of timeout: the connection timeout, and the idle timeout “between two chunks” while streaming. A short total timeout on a streaming request (say 30 seconds) also kills answers that are healthy but long; an idle timeout (say 10 seconds without a new chunk) is what catches a connection that’s really stuck.

Retrying an LLM call may just cost another call. Retrying an action can repeat a side effect. For example, the create-note tool introduced in S10 Tools and MCP may save a note before its response is lost on the network. A retry would create a duplicate.

Give each action an idempotency key, such as the conversation ID plus tool call ID. Check whether it has already run before executing, and return the previous result if it has. Agents in S12 The Agent Loop and Harness also need this protection.

Before you build: the backoff formula, SDK defaults, and 429s that won't recover

Backoff formula: the wait cap for retry n (counting from 0) is min(cap, base × 2^n); the common full jitter picks a random value between 0 and that cap. If there’s a retry-after, use it instead.
SDK defaults: the Anthropic SDK retries 2 times by default. Turn that off (for example maxRetries: 0) before testing your own retry logic, or you’ll be testing the SDK’s behavior.
429s that won’t recover on their own: a 429 caused by an Anthropic usage tier’s monthly limit has no retry-after and keeps failing until the quota resets; a spend limit you set yourself on the organization or workspace returns 400. So when a 429 has no retry-after and keeps failing, stop retrying and raise an alert.

★ How do you switch providers without a rewrite?

Vendors’ APIs are similar but not the same. Put a thin interface in your own code and keep the differences inside it.

There are three kinds of tools to choose from. Official SDKs (Anthropic, OpenAI) give you types, built-in retries and streaming helpers, and support each vendor’s new features first. The Vercel AI SDK (TypeScript) talks to many providers through one interface and comes with frontend chat hooks; it’s the most natural fit for frontend engineers. LiteLLM (Python) calls a hundred-plus models using OpenAI’s format and can also be self-hosted as a gateway that centralizes retries, fallback and usage logging.

The notes assistant’s approach: define a callLLM(messages) → { text, toolCalls, usage }, and read the provider, model name, base URL and key from a config file. Switching providers or moving to a local model only changes the config; the rest of the code stays the same.

“OpenAI-compatible” does not mean identical. Ollama, the llama.cpp server and vLLM all offer OpenAI-compatible APIs, but the details can differ:

Where the system prompt goes, and the tool call format
Stop reason names: Anthropic uses end_turn for a normal finish and max_tokens for hitting the limit; OpenAI Chat Completions uses stop and length
Usage field names: Anthropic uses input_tokens / output_tokens; OpenAI Chat Completions uses prompt_tokens / completion_tokens
Reasoning-related parameters, and whether certain parameters are supported at all

So that interface is responsible for converting all of this into your own format, and for keeping the raw response in the log or trace so you can investigate when something goes wrong.

★ When should you use a local model?

A local model runs on your own machine: it’s free and the data never leaves, but quality, context length and tool-calling reliability usually fall short of frontier models.

Three common ways to run one: Ollama is the easiest to start with; one command downloads and runs a model, and it offers an OpenAI-compatible API. llama.cpp is the underlying engine; it runs quantized GGUF models (GGUF is the model file format llama.cpp uses), ships an HTTP server, and has the most knobs. MLX is Apple’s framework for Apple Silicon; mlx-lm runs and fine-tunes models on a Mac.

When to use one: the data can’t leave the machine, you need to work offline, you don’t want to pay for every call during development and testing, or your volume is high enough that API costs exceed hardware costs. The price: quality is usually lower, the usable context window shorter, tool calling less reliable, and the hardware and operations are on you. During development, the notes assistant can start on a small Ollama model (such as qwen3:0.6b) to confirm the whole pipeline works, then switch back to a cloud model to compare quality. Choosing a model works the same way: first confirm the task is doable with the strongest model, then use evals to decide whether you can move to a smaller, cheaper or local one (→ S8 Evals Methodology, S18 Deployment and Architecture).

Quantization stores each parameter in fewer bits, for example 4 bits instead of 16. It saves a lot of memory at a small cost in quality. Local models are almost always run quantized.

Rough memory estimate: parameter count × bytes per parameter. An 8B (8 billion parameter) model quantized to 4-bit uses 0.5 byte per parameter, so the weights are about 4 GB; with runtime overhead you need about 4–5 GB. Add the KV cache on top, which grows with context length (see Model Crash Course, “KV cache”; for quantization details, deeper: S2 Transformers and Decoding, S5 Inference and Serving).

Going deeper: where models come from and how to choose

Closed vs. open-weight: you can’t get the weights of closed models (such as Claude, OpenAI’s flagship GPT models, Gemini); you can only use them through an API (the vendor’s own, a cloud platform, or an aggregator). Open-weight models (such as Llama, Qwen, Gemma, DeepSeek, Mistral’s open-weight series, OpenAI’s gpt-oss) can be downloaded and run on your own machine or server, and fine-tuned. The former usually have the best quality and the least operations work; the latter keep data in-house and give you more control, but the hardware and operations are your responsibility.
Where to get them: call the vendor’s API directly; use an aggregator (such as OpenRouter) that puts many vendors’ models behind one OpenAI-compatible API, which is handy for comparisons but adds a middleman; Hugging Face Hub is the central place for open-weight models, where you can download models or use its inference services; to run locally, use Ollama, llama.cpp, or LM Studio, which has a GUI.
How to choose: list the task’s hard requirements first (can the data leave, context length, whether you need tool calling and structured output, latency and budget), cross off the models that don’t meet them, then compare the rest with your own evals. Public leaderboards are only good for a first cut; see the benchmark section of S8 Evals Methodology.

Context length in Ollama

When a prompt exceeds Ollama’s default context, it may silently drop the earlier content without an error. Before running long prompts on Ollama, set num_ctx and check with ollama ps.

Going deeper: the defaults, and what truncation looks like

Ollama’s default context depends on VRAM (GPU memory): 4K at 24 GiB or less. When a prompt exceeds it, by default the earlier content is dropped (with only a warning in the server log), and the API doesn’t necessarily return an error. Set num_ctx or OLLAMA_CONTEXT_LENGTH, and check with ollama ps. Every later exercise that runs long prompts on Ollama needs this step first.

A request can silently lose part of its prompt without raising an error. The 20 recent notes in this example total about 30,000 tokens, well beyond 4K. Without adjusting the limit, the beginning, often the system prompt and most notes, is silently dropped. The model sees only the final portion, which can look like poor answer quality.

★ What’s different about reasoning models?

A reasoning model generates a stretch of thinking before it answers. That thinking costs money, counts against the output limit, and makes the first visible word show up much later.

For what a reasoning model is, see Model Crash Course, “Training stages” (deeper: S4 Post-training and Reasoning). From the application side, you need to know these things:

Thinking tokens are billed as output tokens and count against the output limit.
Controls differ by vendor and keep changing. As of this writing (2026-11): OpenAI uses reasoning.effort; Anthropic’s newer models use adaptive thinking plus effort, and the old budget_tokens is deprecated and rejected by newer models. Always check the current docs before writing code.
The raw thinking is usually hidden; you only get a summary or encrypted content. In multi-turn tool calling you must pass the previous turn’s thinking blocks (or reasoning items) back unchanged, or you’ll get errors or worse quality.
Interleaved thinking: the model also thinks between two tool calls, which matters a lot for agents (→ S12 The Agent Loop and Harness).
UX: the interface should let the user know “thinking…” is happening, and allow canceling.

With an output limit of 1,000 tokens, spending 900 on thinking leaves only 100 for the answer. It may stop mid-sentence with a limit stop reason. Include thinking tokens when choosing the output limit for a reasoning model, and log the stop reason.

★ What should you log for every call?

Leave one structured record per call: which model, how many tokens, how much money, how long, and whether it succeeded. Start on day one, because the error analysis and cost control that come later all read this data.

The simplest approach is one line of JSONL (one JSON object per line) per call. The record for the notes-assistant call above:

{"ts":"2026-11-07T10:00:00Z","model":"…","input_tokens":2000,"output_tokens":300,"cost_usd":0.0105,"ttft_ms":800,"latency_ms":6000,"retries":0,"status":"ok"}

With this, “how much did we spend this week” and “which kinds of questions are especially slow” are each a single query.

Once the system gets more complex (retrieval, several LLM calls and tool executions inside one request), switch to a trace: the full record of one request, made of spans. Each LLM call and each tool execution is a span; spans have parent-child relationships, so you can see where the time went. OpenTelemetry is the open standard for recording traces, and its GenAI semantic conventions define field names for LLM calls, such as gen_ai.request.model and gen_ai.usage.input_tokens. Follow the standard names and you can view the traces in different backends.

The smallest version of feedback collection starts here too: put thumbs up / thumbs down next to each answer, and when you store a vote, link it to that answer’s trace ID. Later, when you see a thumbs down, you can open the full prompt and response from that moment directly.

Start logging now so S8 Evals Methodology can use the earliest real traces for error analysis. Full prompts and responses may contain personal data; decide whether to store them and how long to retain them (→ S16 Security and Governance).

Before you build: common GenAI convention fields and backends

Common fields: gen_ai.operation.name, gen_ai.provider.name, gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.finish_reasons. Use custom attributes for cost, TTFT and the user ID.
Two caveats: as of this writing (2026-11) the conventions are still in Development status, so fields may change; and there’s no standard “cost” field, so compute the cost yourself and store it as a custom attribute.
Where the spec lives: it has moved to its own repo (see Further reading); go by the client inference span docs there.
Backends: Jaeger, Langfuse, Phoenix and others all accept OTLP, and all of them run locally in Docker.

★ How do you test output that changes?

The model’s output can differ every time, so tests don’t compare its exact text. Instead, test in layers: your own logic against a fake model, and real output only for properties.

Even with temperature set to 0, output isn’t guaranteed to be identical, and some APIs no longer let you set temperature at all (see Limits on sampling parameters below). The notes assistant’s tests have four layers:

Mock: when testing your own logic (retries, parsing, error handling), replace the model with a fake that returns fixed content or fixed errors. For example, have the fake return a 429 and then a normal response, and check that the call layer makes exactly 2 attempts.
Record / replay: the first run hits the real API and saves the HTTP exchange to a file (a fixture); later tests replay it, with no network, the same result every time, and no cost. When you change the prompt or model, re-record and review the diff.
Snapshot: snapshot what you send, such as the assembled system prompt and tool definitions. That’s far more stable than snapshotting model output, and it’s the best way to catch an accidentally broken prompt.
Assert properties, don’t compare full text: for real output, check only properties, such as valid JSON, cited note titles that actually exist, length within range. Full quality evaluation is the job of S8 Evals Methodology.

Rerunning a failing test until it passes can hide a real problem. Require deterministic mock and replay tests to pass in CI (continuous integration). Run real-model tests separately N times, record every result, and apply a pass-rate threshold, such as at least 9 out of 10.

Limits on sampling parameters

On the current Anthropic API version, Claude models don’t accept temperature (nor top_p or top_k; sending any of them returns 400). For experiments that need to control sampling, use a local model or a provider whose docs explicitly support it.

Going deeper: API versions, SDKs and other providers

As of this writing (2026-11), under Anthropic API version 2026-10-21, any Messages request that includes temperature, top_p or top_k returns 400 for every Claude model. The latest major version of the Anthropic SDK in each language sends this API version; older major versions still send 2023-06-01. Even so, models after Claude 4.7 only accept the default values under the old API version too. OpenAI’s reasoning models also restrict these parameters. So “set temperature to 0 to reduce noise” doesn’t work on these models: when comparing, use the model’s default sampling and average several runs per condition. For experiments that need controlled sampling, use a local model (Ollama, llama.cpp) or another provider / model whose docs explicitly say these parameters are supported, and note it in your records. Rules like these change quickly, so check the current docs before you start.

○ Can models run in the browser?

A model can run directly in the user’s browser (WebGPU / WASM), with no server cost and no data leaving the device. The limits: the model has to be downloaded first (usually hundreds of MB at least), only small models work, and device support varies. It suits small tasks such as classification, summarization and embeddings. S17 Product and AI UX comes back to where it fits in a product.

Exercises

Concepts route (no code): after “Try it first”, you can skip to the self-check.

Try it first (about 10 minutes)

1. Should the call layer retry in each of these five cases? (a) 429 (too many requests) with retry-after: 5; (b) 401 (not authenticated); (c) 503 (service temporarily unavailable); (d) 400 (bad request); (e) the network drops mid-stream.

Check your answer

Retry: (a) wait 5 seconds, then retry; (c) exponential backoff with jitter; (e) is a network drop, so it can be retried, but if the call is followed by side effects, an idempotency key is needed to prevent duplicates. Don’t retry: (b) is a key or permissions problem and (d) a malformed request; retrying gives the same result every time.

2. Do the notes assistant’s math yourself: system prompt plus notes is 3,000 tokens, each question 100 tokens, each answer 400 tokens. Using the lecture’s illustrative prices, what do turn 1 and turn 5 each cost?

Check your answer

Turn 1: input 3,100 × 3 / 1,000,000 = 0.0093, output 400 × 15 / 1,000,000 = 0.006, total $0.0153. Turn 5 has to carry the previous 4 turns’ questions and answers: input 3,000 + 4 × 500 + 100 = 5,100 tokens, so 0.0153 + 0.006 = $0.0213. Output stays the same; input grows every turn.

Core: A reliable LLM call layer

Goal: write a small module that streams, can be canceled, retries correctly and logs cost, and prove with offline tests that it really does.

Steps

Write a callLLM function: send some messages, get back the text and usage. Prepare a price config file (input / output prices, copied from the provider’s pricing page with the date noted) and compute each call’s cost from usage. Fix the test inputs as 20 short prompts, such as “Explain what HTTP status code N means in one sentence”, with N set in turn to 200, 201, 204, 301, 302, 304, 400, 401, 403, 404, 409, 413, 418, 422, 429, 500, 502, 503, 504, 529.
Add streaming: print tokens as they arrive, and measure TTFT (from sending to the first text chunk) and total latency.
Add cancellation: a timeout or keypress triggers it; confirm the connection is actually closed. Log the timestamp of the cancellation and the timestamp of the last chunk received.
Write a local mock server that can be configured to return: a 429 with retry-after: 2 (429 the first time, a normal response the second), repeated 500s, a 400, an error event mid-stream, and a 10-second pause between two chunks.
Implement retries: retry only retryable errors, with exponential backoff and jitter, honoring retry-after, at most 3 retries. Turn off the SDK’s built-in retries first (for example maxRetries: 0), or you’ll be testing the SDK’s behavior.
Write one JSONL log line per call: timestamp, model, input tokens, output tokens, cost, TTFT, total latency, retry count, final status.
Record one real call with record / replay and write a test that replays it.
Change only the config (environment variables or config file) to point callLLM at a local model (for example, Ollama’s OpenAI-compatible API with qwen3:0.6b) and run 5 of the prompts.

Acceptance criteria

Call the real model 20 times; all 20 log entries have the 9 fields above. Pick any 3 and your hand-computed cost matches the log exactly.
Against the mock server: the 429 case succeeds on attempt 2, with ≥ 2 seconds between the two attempts; repeated 500s make 4 attempts in total (1 + 3 retries) before reporting an error; the 400 makes only 1 attempt.
A stream that goes idle for 10 seconds is cut off by your between-chunk timeout and logged as a timeout, not a success.
No new chunk is printed more than 500 ms after canceling: in the log, “last chunk timestamp − cancellation timestamp” ≤ 500 ms (it can be negative), and the connection is closed.
With the network unplugged, the whole test suite passes 10 runs in a row.
The 5 log entries from the local model also have all 9 fields (cost recorded as 0), and nothing but the config changed.

Hints

TypeScript: the official SDK (@anthropic-ai/sdk or openai), or plain fetch + ReadableStream; AbortController for cancellation; node:http or MSW for the mock server; Vitest for tests.
Python: the official SDK or httpx + asyncio; respx for mocking, pytest-recording (built on VCR.py) for record / replay; FastAPI or the standard library’s http.server for the mock server.
Cost: 20 small calls usually come to under $1. If you don’t want to pay, use Ollama throughout and let the mock server simulate the 429s and 500s.
Short on time? Do the three mocks (429 / 500 / 400) and their acceptance criteria first, and add the mid-stream error and between-chunk timeout later.
If you get stuck:
- Reread: Should you retry on an error? (formula in the collapsible section), streaming, testing
- Further: the HTTP errors section of Anthropic’s “Errors” page, the “Jitter” section of the AWS article

Advanced: System project A6: Start your system

This is the first step of the system project (for the step codes, see how to read the codes): you pick a topic, and the next 13 chapters (S7–S19) all add capabilities to the same system.

Choosing a topic: a good topic has your own documents (not something freely available online), some actions that do things (create, edit, send), room for 2–3 users, and is something you’ll actually use. That way you get to practice RAG, tools, security and multi-user deployment later on. Four sample topics are in System Project, “Pick a topic”; if you’re unsure, pick the notes assistant.

Goal: build a minimal system that streams chat, has a trace and cost for every call, collects feedback, and can switch between cloud and local models.