Prompt caching: cheaper and faster LLM calls

Every time an agent calls a model, it sends the whole conversation again: the system prompt, the tool definitions, and everything said so far. Most of that is identical to the previous call. Prompt caching exists so that providers don't redo the same work each time, and so you don't pay full price for it.

For agents and any app with a long, stable prompt, this is often the single biggest cost and speed win available, and it takes a few lines to turn on. It also has rules that are easy to trip over, which is why it's worth understanding before you switch it on.

What actually gets cached#

When a model reads your prompt, it does a chunk of computation over every input token before it can start generating. That work depends only on the tokens before it. So if two requests begin with exactly the same tokens, the provider can save the intermediate results from the first and reuse them for the second, instead of recomputing.

That's a prefix cache. The word prefix is doing real work. The reuse only applies to the beginning of the prompt, up to the first place the two requests differ. Everything after that point gets processed normally.

Two consequences follow immediately:

1. Order your prompt from most stable to least stable. Static instructions and tool definitions first, then reference material, then the changing conversation, then the new question.
2. Any change early in the prompt invalidates everything after it. Edit one word of the system prompt, and the cached work for the rest is gone.

Keep those two in your head and most of the rest is detail.

Why it matters for agents#

Look at what happens in a twenty-step agent run. Say the system prompt and tool definitions add up to 10,000 tokens, and they're identical on every step. Without caching you pay to process those 10,000 tokens twenty times, plus the growing conversation.

Prices differ by provider and model and change over time, so treat the following as an illustration of the shape, not a quote. Anthropic's pricing works roughly like this: writing a prefix into the cache costs somewhat more than normal input, reading it back costs a small fraction of normal input (on the order of a tenth), and cached prefixes live for a few minutes by default, refreshed each time they're used. Check the current pricing page for exact numbers.

Using those ratios, with normal input price as 1 unit per token:

• Without caching: 20 calls × 10,000 tokens = 200,000 units for the static prefix.
• With caching: one write at 1.25 × 10,000 = 12,500 units, plus 19 reads at 0.1 × 10,000 = 19,000 units. Total: 31,500 units.

That's about an 84 percent reduction on the prefix, before counting the latency improvement. The savings grow with prefix length and the number of calls, which is exactly the profile of a long-running agent.

Latency improves too. Skipping recomputation shortens the time before the first token arrives, which users feel as responsiveness.

Turning it on with the Claude API#

With Anthropic's API you mark cache breakpoints by adding a cache_control field to a content block. Everything up to and including that block becomes cacheable as a prefix.

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-sonnet-5",   # use a current model name from the docs
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT_AND_REFERENCE_DOCS,
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": "What is our refund policy for annual plans?"}],
)

u = response.usage
print("uncached input:", u.input_tokens)
print("written to cache:", u.cache_creation_input_tokens)
print("read from cache:", u.cache_read_input_tokens)

On the first call, cache_creation_input_tokens is large: you've written the prefix. On later calls within the cache lifetime, cache_read_input_tokens is large and input_tokens covers only the new part at the end. Those usage numbers are how you confirm caching works. If cache_read_input_tokens stays at zero, something is preventing a hit.

Some details to know:

• The cached prefix is built in a fixed order: tool definitions, then the system prompt, then messages. A breakpoint applies to everything before it in that order.
• You can place several breakpoints, up to a small limit. A common pattern is one after the tools and system prompt, and another near the end of the conversation.
• There's a minimum length below which caching won't apply, roughly a thousand tokens for many models and more for some. Short prompts simply aren't cached.
• The default lifetime is short, on the order of minutes, and a longer option exists at a higher write price. Pick based on how often calls arrive.

Caching a growing conversation#

For agents, the interesting trick is caching the conversation too. Each step appends a little to a long history. If you move a breakpoint to the last message on each call, the whole history so far is a cached prefix, and only the newest turn is processed fresh.

def with_history_breakpoint(messages):
    """Mark the final content block of the last message as a cache breakpoint."""
    marked = [dict(m) for m in messages]
    last = marked[-1]
    blocks = last["content"]
    if isinstance(blocks, str):
        blocks = [{"type": "text", "text": blocks}]
    blocks = [dict(b) for b in blocks]
    blocks[-1]["cache_control"] = {"type": "ephemeral"}
    last["content"] = blocks
    return marked

Call that before each request in the loop from the first-agent tutorial, and step twenty no longer re-reads steps one through nineteen at full price. Some SDKs and agent tools handle this for you, so check what yours does before writing your own.

What quietly breaks caching#

Cache misses are silent. Your code works, the answers are right, and the bill is higher than it should be. These are the usual causes.

A timestamp or random value near the top. "The current time is 14:03:22" at the start of the system prompt makes every request unique. Move volatile values to the end, or drop them.

Per-user content before shared content. If the first thing in the prompt is the user's name, every user gets a different prefix. Put shared material first and personal details after.

Reordered or changing tool lists. Tool definitions come first in the prefix. If you send them in a different order each time, or add and remove tools per request, you invalidate everything below them.

Non-deterministic serialization. Building a JSON blob with keys in arbitrary order, or with varying whitespace, produces different text for identical data. Sort keys and format consistently.

Editing earlier turns. Rewriting or trimming older messages changes the prefix. This includes compaction, where you replace a long history with a summary. That's still worth doing when the window fills, but expect a cache rewrite when you do it. See context engineering for when to compact.

Waiting too long. If your calls are spaced further apart than the cache lifetime, the entry expires. A batch job that fires once an hour won't benefit from a five-minute cache.

Parallel first requests. If you launch twenty requests at once with a fresh prefix, they all arrive before the cache has been written, and each pays full price. Send one first, wait for it to start returning, then fan out.

How to check it's working#

Build the measurement in from the start.

1. Log the three token counts on every call: uncached input, cache writes, cache reads.
2. Compute a hit rate: cache read tokens divided by total prompt tokens. For a steady agent workload with a stable prefix, you want this high, often well above 80 percent after the first call.
3. Alert on drops. A sudden fall in hit rate usually means somebody changed the prompt header, the tool list, or how something is serialized.
4. Track cost per task, not just per call. That's the number that shows whether it helped.

Reading the numbers turns caching from a hope into an engineered property.

Designing prompts for caching#

Once you know how it works, a few design habits follow.

• Treat the top of the prompt as read-only. Version it, review changes to it, and don't let application code splice values into it at runtime.
• Put big, stable reference material in the cached region. A style guide, a policy manual, a codebase summary. Retrieval can still supply the rest per request.
• Keep the tool set fixed within a session. If you need different tools for different tasks, choose a set at the start and stay with it.
• Batch related work under the same prefix. If you'll run fifty analyses against one long document, run them back to back, not spread across the day.
• Use the fresh tail for what changes: the question, the newest tool results, the user's message.

That last point suits agents naturally. The instructions and tools are stable, and only the tail of the conversation moves.

Caching versus other cost levers#

Caching is one tool among several, and it combines with the others.

Model choice. Using a smaller, cheaper model for routine steps can matter more than caching, and you can do both. See choosing a model tier.

Shorter context. Trimming tool results and dropping stale history reduces the tokens you send at all, cached or not. Caching a bloated prompt is still paying for bloat, just at a lower rate.

Batch APIs. For work that doesn't need an immediate answer, providers often offer discounted asynchronous processing. It stacks with caching in some cases; check the docs.

Output limits. Output tokens usually cost more than input tokens, so a max_tokens cap and clear instructions about length help.

For a fuller picture of where the money goes, tokens, context windows, and LLM costs lays out the arithmetic.

When it isn't worth it#

Skip caching when prompts are short, when each call has a unique prefix, or when calls are too infrequent to hit the cache before it expires. The write premium means a cache that's never read back costs slightly more than no cache at all. A one-off call with a 300-token prompt gets nothing from it.

Conversely, if you run an agent, a chat assistant with a long system prompt, a document Q&A tool, or a coding tool with a large repository summary, it's usually one of the first optimizations to try.

Other providers#

Most major providers now offer some form of prompt caching, and the details vary: some apply it automatically to long prompts, some require explicit markers, and pricing and lifetimes differ. The principle carries across all of them. Keep the front of the prompt identical, put changing content at the back, and measure hits. If you use more than one provider, read each one's documentation instead of assuming the behavior matches.

A short checklist#

• Static content first, volatile content last
• No timestamps or per-user data in the shared prefix
• Tool list fixed and consistently ordered
• Deterministic serialization
• A breakpoint on the stable prefix, and one on the conversation tail for agents
• Usage logged, hit rate tracked, alerts on drops
• Parallel calls started after the first response

Turn it on, watch the numbers for a day, and fix whatever keeps the hit rate low. If the rate is still poor after that, print the exact prefix from two consecutive requests and compare them character by character. A diff tool finds the stray timestamp in seconds. It's rare to find an optimization that's this cheap to try and this easy to verify.