Prompt caching

If you keep sending the same long prefix on every chat completions request — a multi-thousand-token system prompt, a growing conversation history, the same reference document — you don’t have to keep paying for it. With Claude (Anthropic) routes, you can explicitly mark the stable portion to cache. Subsequent calls reuse the cached prefix at roughly 10× cheaper, and you only pay the full rate for what’s new. (Some other providers — OpenAI, Gemini — do their own caching with different mechanics; see the Provider support table below.)

You see the savings in the usage object: cache_read_tokens goes up, and the cached portion drops to the cache-read price.

Think of it like a hotel keycard: the first time you check in, there’s a small activation cost. After that, the same card opens your door instantly until the lease expires. If you change rooms (modify the cached content), you need a new card.

Where the metaphor breaks down: the cache has a short TTL (5 minutes by default), and the cached portion must match exactly — Anthropic requires “100% identical prompt segments” across requests, so even a single-character change in the cached prefix invalidates the cache.

When to use it

• Long, stable system prompts sent on every request (persona, instructions, tool schemas)
• Multi-turn conversations where prior turns don’t change between calls
• Document Q&A where the same long document is repeatedly referenced

If your prefix is short or changes every call, caching costs more than it saves — there’s a small surcharge on the first call to write the cache.

How to send a cache marker

Set cache_control on a message to cache everything up through and including that message. The next request that arrives within the TTL with the same prefix reuses the cache.

curl -X POST https://your-tie-host/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "anthropic/claude-sonnet-4-5",
    "messages": [
      {
        "role": "system",
        "content": "...long stable instructions...",
        "cache_control": {"type": "ephemeral"}
      },
      {
        "role": "user",
        "content": "Earlier question"
      },
      {
        "role": "assistant",
        "content": "Earlier answer",
        "cache_control": {"type": "ephemeral"}
      },
      {
        "role": "user",
        "content": "New question — not cached, charged at full rate"
      }
    ]
  }'

This example uses two markers: one after the system prompt (caches a long stable prefix), one after the last assistant reply (extends the cache through the prior conversation). Anthropic accepts up to 4 cache breakpoints per request.

Per-content-part caching

For messages with mixed content (text + images), put cache_control on individual content parts. Useful when you want to cache one specific block — say, a big document — while keeping the surrounding text uncached.

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Use this 500-page manual as context...", "cache_control": {"type": "ephemeral"} },
    { "type": "text", "text": "Now my question is..." }
  ]
}

Automatic server-side breakpoints (Anthropic)

Even when you don’t set cache_control yourself, Truememory marks the parts of an Anthropic request that are most likely to be stable across turns — the system prompt, the tool schemas, and recent user turns — so caching works out of the box. Any markers you add on top compose with these.

Think of Truememory as drawing a dotted line between “the part that’s the same every call” and “the part that’s new this call.” You can redraw the line yourself with cache_control, but you don’t have to.

• Stable system prompt — Truememory’s built-in instructions (plus persona and skill instructions when present) sit inside the cached prefix.
• Tool schemas — When tools are bound, the last tool definition carries a cache marker so the whole tool schema stays inside the reusable prefix. Anthropic orders tools before the system prompt, so without this marker the schema would invalidate the cache on every change.
• Rolling human turns — The two most recent user turns are marked, extending the cache through the prior conversation. Only the volatile suffix of the current turn — a live listener transcript appended at request time, for example — lands outside the cache.
• Dynamic prompt split — When a system or developer override is sent as content parts with a cache_control marker, Truememory keeps the boundary you drew: text up to and including the marker is cached; anything after stays dynamic. If the override is a plain string, Truememory auto-detects known volatile headings (date/time, page context, live listener context, retrieved graph context) and moves those below the boundary so per-request data doesn’t poison the cache key.

Provider support