---
title: "Prompt caching"
description: "Mark the stable portion of a prompt so repeat requests reuse it at up to 10× lower cost."
---

> Documentation Index
> Fetch the complete documentation index at: https://truememory-docs.mvstg.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Prompt caching

If you keep sending the same long prefix on every [chat completions](/ai-gateway/chat-completions/) request — a multi-thousand-token system prompt, a growing conversation history, the same reference document — you don't have to keep paying for it. With Claude (Anthropic) routes, you can **explicitly mark** the stable portion to cache. Subsequent calls reuse the cached prefix at roughly 10× cheaper, and you only pay the full rate for what's new. (Some other providers — OpenAI, Gemini — do their own caching with different mechanics; see the [Provider support](#provider-support) table below.)

You see the savings in the [`usage` object](/ai-gateway/chat-completions/#understanding-the-usage-object): `cache_read_tokens` goes up, and the cached portion drops to the cache-read price.

**Think of it like a hotel keycard:** the first time you check in, there's a small activation cost. After that, the same card opens your door instantly until the lease expires. If you change rooms (modify the cached content), you need a new card.

**Where the metaphor breaks down:** the cache has a short TTL (5 minutes by default), and the cached portion must match exactly — Anthropic requires "100% identical prompt segments" across requests, so even a single-character change in the cached prefix invalidates the cache.

## When to use it

- **Long, stable system prompts** sent on every request (persona, instructions, tool schemas)
- **Multi-turn conversations** where prior turns don't change between calls
- **Document Q&A** where the same long document is repeatedly referenced

If your prefix is short or changes every call, caching costs more than it saves — there's a small surcharge on the first call to write the cache.

## How to send a cache marker

Set `cache_control` on a message to cache everything up through and including that message. The next request that arrives within the TTL with the same prefix reuses the cache.

```bash
curl -X POST https://your-tie-host/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -d '{
    "model": "anthropic/claude-sonnet-4-5",
    "messages": [
      {
        "role": "system",
        "content": "...long stable instructions...",
        "cache_control": {"type": "ephemeral"}
      },
      {
        "role": "user",
        "content": "Earlier question"
      },
      {
        "role": "assistant",
        "content": "Earlier answer",
        "cache_control": {"type": "ephemeral"}
      },
      {
        "role": "user",
        "content": "New question — not cached, charged at full rate"
      }
    ]
  }'
```

This example uses two markers: one after the system prompt (caches a long stable prefix), one after the last assistant reply (extends the cache through the prior conversation). Anthropic accepts up to **4 cache breakpoints per request**.

## Per-content-part caching

For messages with mixed content (text + images), put `cache_control` on individual content parts. Useful when you want to cache one specific block — say, a big document — while keeping the surrounding text uncached.

```json
{
  "role": "user",
  "content": [
    { "type": "text", "text": "Use this 500-page manual as context...", "cache_control": {"type": "ephemeral"} },
    { "type": "text", "text": "Now my question is..." }
  ]
}
```

## Automatic server-side breakpoints (Anthropic)

Even when you don't set `cache_control` yourself, Truememory marks the parts of an Anthropic request that are most likely to be stable across turns — the system prompt, the tool schemas, and recent user turns — so caching works out of the box. Any markers you add on top compose with these.

Think of Truememory as drawing a dotted line between "the part that's the same every call" and "the part that's new this call." You can redraw the line yourself with `cache_control`, but you don't have to.

- **Stable system prompt** — Truememory's built-in instructions (plus persona and skill instructions when present) sit inside the cached prefix.
- **Tool schemas** — When tools are bound, the last tool definition carries a cache marker so the whole tool schema stays inside the reusable prefix. Anthropic orders tools before the system prompt, so without this marker the schema would invalidate the cache on every change.
- **Rolling human turns** — The two most recent user turns are marked, extending the cache through the prior conversation. Only the volatile suffix of the current turn — a live listener transcript appended at request time, for example — lands outside the cache.
- **Dynamic prompt split** — When a `system` or `developer` override is sent as content parts with a `cache_control` marker, Truememory keeps the boundary you drew: text up to and including the marker is cached; anything after stays dynamic. If the override is a plain string, Truememory auto-detects known volatile headings (date/time, page context, live listener context, retrieved graph context) and moves those below the boundary so per-request data doesn't poison the cache key.

## Provider support

| Provider | How Truememory handles `cache_control` |
|----------|--------------------------------|
| `anthropic/...` (routed via Vertex AI on this instance) | Forwarded to Anthropic. Cache hits show up as `cache_read_tokens` in the response. Truememory also applies the automatic server-side breakpoints listed above. |
| OpenAI | The field is passed through to the OpenAI client. OpenAI does its own **automatic** caching on long stable prefixes and reports it via `cache_read_tokens` regardless of whether you set the marker — you don't need `cache_control` to benefit from OpenAI caching. |
| Gemini, Groq, Mistral, others | The field is passed through but ignored by these providers via the chat completions API. `cache_read_tokens` stays `0`. (Gemini does support caching, but through a separate `cachedContent` API not exposed via this endpoint.) |

:::tip[Longer TTL]
The default 5-minute cache window is fine for active conversations. For longer sessions, Anthropic offers an opt-in `ttl` field: `{"type": "ephemeral", "ttl": "1h"}`. See the [Anthropic prompt-caching docs](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) for current pricing and constraints.
:::

:::note[Verifying cache hits]
Set `stream_options.include_usage: true` (or use the non-streaming endpoint) to see `cache_read_tokens` and `cache_creation_tokens` per call. If `cache_read_tokens` stays `0` after several requests with the same prefix, double-check that the cached content is byte-identical across calls and that requests arrived within the TTL window. See [Understanding the `usage` object](/ai-gateway/chat-completions/#understanding-the-usage-object) for what each field means.
:::

Source: https://truememory-docs.mvstg.com/ai-gateway/features/prompt-caching/index.mdx