Prompt caching: the order of your prompt is a cost decision

An agent remembers nothing. On every turn it rereads everything: its tool list, its instructions, the reference documents, the full conversation history, then the new message. By the tenth turn it has read the same opening ten times, and you have paid for it ten times.

Prompt caching avoids that repetition. The provider keeps the processing of a prompt prefix it has already seen and charges much less for it the next time. At Anthropic, reading from the cache costs about a tenth of the normal price, even less on some recent models. Writing to the cache, on the other hand, costs 25% more than the normal price.

That last number is the one people forget. It means a badly used cache does not just fail to help: it costs more than no cache at all.

Bytes, not meaning

The cache does not understand your prompt, it compares bytes. It finds the longest prefix identical to a recent request, and everything after the first differing byte is recomputed, then rewritten. The render order is fixed: tools first, then the system prompt, then the messages.

The direct consequence: what never changes goes first, what changes often goes last. A single moving value at the top of the prompt invalidates everything after it. The simulation below plays ten turns of a conversation with 17,000 tokens of fixed context, and lets you make the three most common mistakes.

With no mistakes, the conversation costs about a quarter of the uncached price. Every turn reads back everything before it and only writes the latest exchange. On an agent that chains dozens of tool calls, like the one running a small online business, it is the easiest saving there is, and it costs nothing in quality: the model reads exactly the same text.

Three mistakes that break everything

The date in the system prompt

This is the most common one. You want the agent to know the date, so you write “Today is October 19, it is 2:32 pm” at the top of the instructions. A minute later the prefix changes right after the tools, and everything else is rewritten every turn. In the simulation, the bill climbs to 111% of the uncached price.

The date goes at the end, in the latest message, and to the day if the time is not needed. Same for the user’s name, their plan, a request id: anything that varies from one conversation or call to the next.

Tools in a shifting order

Tools are rendered first. If their list is built from a plugin registry, a set or a database query, their order can change from one call to the next without anyone noticing. The prefix breaks at the very first byte and everything is rewritten every turn: 125% of the price. The fix is one line, sorting the tools by name. Same for a JSON object serialised without a stable key order: its bytes change while its meaning doesn’t.

For the same reason, avoid adding or removing tools mid-conversation, and switching models: the cache belongs to one model.

The sliding window

To keep the context short, you drop the oldest messages past a certain size. Reasonable on paper. But every drop shifts the whole history: the identical prefix ends right before the first message, and the entire history is rewritten every turn. The cost stays contained in the simulation, 34%, because most of the prompt sits in front. In a long conversation, the history quickly becomes the largest part.

Summarise in steps, rarely, rather than sliding on every turn. Some providers also offer server-side compaction, which summarises the history without breaking the prefix in front of it.

Time matters too

A cache entry lives five minutes by default at Anthropic, and every read resets that timer. An active conversation therefore stays cached indefinitely. But a user who replies after ten minutes finds an empty cache, and their next turn pays the full write. There is a one-hour lifetime, whose write costs twice the normal price: worth it only if pauses between five minutes and an hour are common.

Another surprise: there is a minimum size. A prefix that is too short is not cached, with no error and no warning. The threshold depends on the model, from a few hundred to a few thousand tokens.

Check instead of assuming

The API response tells you how many tokens were read from the cache and how many were written to it. Both numbers are worth logging on every call. If reads stay at zero when the start of the prompt should be stable, something is changing behind your back. Compare two consecutive requests byte by byte: the first difference names the culprit.

response = client.messages.create(
    model=MODEL,
    max_tokens=2048,
    # sorted, so the very first bytes of the prompt never move
    tools=sorted(tools, key=lambda t: t["name"]),
    system=[{
        "type": "text",
        "text": INSTRUCTIONS,  # frozen: no date, no user name, no request id
        "cache_control": {"type": "ephemeral"},
    }],
    # whatever varies goes last, with the new message
    messages=history + [{"role": "user", "content": f"{question}\n\nToday: {today}"}],
    cache_control={"type": "ephemeral"},  # also cache the growing conversation
)

usage = response.usage
log.info(
    "cache read=%s written=%s uncached=%s",
    usage.cache_read_input_tokens,
    usage.cache_creation_input_tokens,
    usage.input_tokens,
)

The order I use

1. Tools, sorted by name, stable for the whole conversation.
2. System instructions, frozen: no date, no name, no id.
3. Reference documents that do not change.
4. The history, append-only.
5. The new message, with everything that varies: date, user context, passages retrieved for this question.

It looks like layout. On the bill, it is a factor of four.