---
title: "Thread compaction"
description: "Fold older turns into a rolling summary checkpoint so long-lived threads stop growing in cost — opt in per request with context_management."
---

> Documentation Index
> Fetch the complete documentation index at: https://truememory-docs.mvstg.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Thread compaction

Every turn on a [thread](/ai-gateway/threads/) replays the conversation history to the model, so input tokens — and cost — grow with every message. On long-lived threads this compounds: a companion-style conversation can reach 40k+ input tokens for a one-word user message, and nothing ever shrinks it.

Thread compaction bounds that growth. Before each model call, Truememory checks the upcoming request against a budget; when it's over, older turns fold into a **rolling summary checkpoint** first, and the model call runs on the compacted window. Subsequent requests replay the checkpoint plus the messages that came after it — not the full transcript — so per-turn cost stays flat no matter how long the thread lives.

Compaction never deletes anything: the full transcript stays in the thread, [`/history`](/ai-gateway/threads/#history) keeps returning every message (checkpoints are internal and never shown), and [memory extraction](/memory/overview/) still sees the complete conversation.

## Enabling compaction

Add a `compaction` entry to `context_management` on any `/v1/chat/completions` request (naming follows the OpenAI compaction API):

```bash
curl -X POST https://your-tie-host/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Agent-Id: chatbot" \
  -d '{
    "model": "anthropic/claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello again!"}],
    "thread_id": "my-long-thread",
    "context_management": [
      {
        "type": "compaction",
        "compact_threshold": 6000,
        "keep_tokens": 3000,
        "model": "anthropic/claude-haiku-4-5"
      }
    ]
  }'
```

The entry's **presence enables compaction for that request**; omit it to disable. There is no server-side setting — the caller decides on every request.

### Options

| Field | Required | Default | What it does |
|-------|----------|---------|--------------|
| `type` | yes | — | Must be `"compaction"`. |
| `compact_threshold` | no | overflow mode | Compact when the replayed window exceeds this many tokens — explicit cost control. When omitted, compaction is **overflow protection**: it fires when the estimated request (history + system prompt + tools) approaches the model's context window minus a 20k buffer (or minus `max_tokens`, whichever is larger). On large-context models that means it fires rarely — set a threshold if you want cost-driven compaction. |
| `keep_tokens` | no | `8000` | Budget of recent turns carried **verbatim** inside the checkpoint. |
| `model` | no | request model | Model for the summarization call. Pass a small model — the request model is usually oversized (and overpriced) for summarization. |

## How it works

1. Before every model call — including each iteration of a tool loop — Truememory estimates the upcoming request. Under budget, the check is a no-op costing nothing. Over budget, the summarizer runs first (adding its latency to that turn) so the expensive model call proceeds on the compacted window.
2. Over the threshold, older turns are summarized into an **anchored rolling summary**: each compaction feeds the previous checkpoint's summary back to the summarizer to update ("preserve still-true details, remove stale ones, merge new facts"), so facts converge instead of being re-derived every time. The most recent `keep_tokens` of conversation ride along verbatim.
3. The checkpoint is stored as a message in the thread. The next request replays `checkpoint + everything after it`.
4. If the summarization call fails, the turn is unaffected — the thread simply stays uncompacted until the next attempt.

Compaction requires a `thread_id` in practice: without one there is no replayed history to compact.

### Turning it off — and back on

Compaction is a per-request contract, so the toggle is fully reversible:

- **Omit the entry** on a request and the thread replays its complete raw history, exactly as if compaction had never run. Existing checkpoints stay stored but never reach the model.
- **Re-enable it later** and replay resumes from the newest existing checkpoint.

## When to use it

Compaction is built for **long-lived conversational threads** — companions, assistants, support sessions — where history growth is the dominant marginal cost. Measured on a production companion thread, late-thread messages dropped from $0.068 to $0.036 with a 6000/3000 configuration, and stayed there instead of growing without bound.

It is the wrong tool when:

- **Threads are short-lived.** A thread that never crosses the threshold never compacts; the option is harmless but pointless.
- **You need verbatim recall of arbitrary old details.** A summary is lossy by design. Durable facts (names, dates, decisions) are the [memory graph](/memory/overview/)'s job — it extracts them from the full transcript regardless of compaction. Keep `keep_tokens` at 3000 or above for conversational products so recent nuance survives verbatim.
- **You want request-scoped compression without persistence.** [Message compression](/ai-gateway/features/message-compression/) compresses the outgoing prompt for a single request and stores nothing; compaction writes a durable checkpoint that changes how all subsequent requests replay.

## Supported agents

The agent handling the request must have compaction wired into its graph — currently `chatbot` (the default) and `eliza`. Requesting compaction on an agent without support fails loudly rather than silently ignoring the option:

```json
{
  "code": "AGENT_COMPACTION_UNSUPPORTED",
  "message": "Agent 'research-assistant' does not support thread compaction.",
  "hint": "Omit context_management, or use an agent with compaction wired in (e.g. 'chatbot')."
}
```

## Compaction events

When a checkpoint is written during a turn, the response says so — mirroring how the OpenAI compaction API surfaces a compaction item:

- **Non-streaming** responses carry a top-level field:

  ```json
  "context_management": [{ "type": "compaction", "status": "completed" }]
  ```

- **Streaming** responses emit one extra chunk with the same field, early in the stream before the reply text (compaction runs pre-flight). The summary itself is never streamed.

The event is purely informational — threads are server-managed, so unlike OpenAI there is nothing the client needs to store or send back. Use it to drive debug panels or usage dashboards.

## Observability and billing

Each summarization call appears in tracing as a `thread_compaction` observation (tagged `compaction`) inside the request's trace, so its spend is filterable per feature. Summarization tokens are **not** added to the response's `usage` — they are platform-side cost, typically fractions of a cent per compaction with a small summarizer model.

:::tip[Choosing thresholds]
`compact_threshold` caps your worst-case history spend per turn; `keep_tokens` sets how much recent conversation survives verbatim. Lower values save more but summarize more aggressively. A `6000` / `3000` split roughly halves late-thread marginal cost for a chat product while keeping the last ~10–15 exchanges word-for-word.
:::

Source: https://truememory-docs.mvstg.com/ai-gateway/features/thread-compaction/index.mdx