Thread compaction | truememory docs

Fold older turns into a rolling summary checkpoint so long-lived threads stop growing in cost — opt in per request with context_management.

Every turn on a thread replays the conversation history to the model, so input tokens — and cost — grow with every message. On long-lived threads this compounds: a companion-style conversation can reach 40k+ input tokens for a one-word user message, and nothing ever shrinks it.

Thread compaction bounds that growth. Before each model call, Truememory checks the upcoming request against a budget; when it’s over, older turns fold into a rolling summary checkpoint first, and the model call runs on the compacted window. Subsequent requests replay the checkpoint plus the messages that came after it — not the full transcript — so per-turn cost stays flat no matter how long the thread lives.

Compaction never deletes anything: the full transcript stays in the thread, /history keeps returning every message (checkpoints are internal and never shown), and memory extraction still sees the complete conversation.

Enabling compaction

Add a compaction entry to context_management on any /v1/chat/completions request (naming follows the OpenAI compaction API):

curl -X POST https://your-tie-host/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Agent-Id: chatbot" \
  -d '{
    "model": "anthropic/claude-sonnet-4-6",
    "messages": [{"role": "user", "content": "Hello again!"}],
    "thread_id": "my-long-thread",
    "context_management": [
      {
        "type": "compaction",
        "compact_threshold": 6000,
        "keep_tokens": 3000,
        "model": "anthropic/claude-haiku-4-5"
      }
    ]
  }'

The entry’s presence enables compaction for that request; omit it to disable. There is no server-side setting — the caller decides on every request.

Options

Field	Required	Default	What it does
`type`	yes	—	Must be `"compaction"`.
`compact_threshold`	no	overflow mode	Compact when the replayed window exceeds this many tokens — explicit cost control. When omitted, compaction is overflow protection: it fires when the estimated request (history + system prompt + tools) approaches the model’s context window minus a 20k buffer (or minus `max_tokens`, whichever is larger). On large-context models that means it fires rarely — set a threshold if you want cost-driven compaction.
`keep_tokens`	no	`8000`	Budget of recent turns carried verbatim inside the checkpoint.
`model`	no	request model	Model for the summarization call. Pass a small model — the request model is usually oversized (and overpriced) for summarization.

How it works

Before every model call — including each iteration of a tool loop — Truememory estimates the upcoming request. Under budget, the check is a no-op costing nothing. Over budget, the summarizer runs first (adding its latency to that turn) so the expensive model call proceeds on the compacted window.
Over the threshold, older turns are summarized into an anchored rolling summary: each compaction feeds the previous checkpoint’s summary back to the summarizer to update (“preserve still-true details, remove stale ones, merge new facts”), so facts converge instead of being re-derived every time. The most recent keep_tokens of conversation ride along verbatim.
The checkpoint is stored as a message in the thread. The next request replays checkpoint + everything after it.
If the summarization call fails, the turn is unaffected — the thread simply stays uncompacted until the next attempt.

Compaction requires a thread_id in practice: without one there is no replayed history to compact.

Turning it off — and back on

Compaction is a per-request contract, so the toggle is fully reversible:

Omit the entry on a request and the thread replays its complete raw history, exactly as if compaction had never run. Existing checkpoints stay stored but never reach the model.
Re-enable it later and replay resumes from the newest existing checkpoint.

When to use it

Compaction is built for long-lived conversational threads — companions, assistants, support sessions — where history growth is the dominant marginal cost. Measured on a production companion thread, late-thread messages dropped from $0.068 to $0.036 with a 6000/3000 configuration, and stayed there instead of growing without bound.

It is the wrong tool when:

Threads are short-lived. A thread that never crosses the threshold never compacts; the option is harmless but pointless.
You need verbatim recall of arbitrary old details. A summary is lossy by design. Durable facts (names, dates, decisions) are the memory graph’s job — it extracts them from the full transcript regardless of compaction. Keep keep_tokens at 3000 or above for conversational products so recent nuance survives verbatim.
You want request-scoped compression without persistence. Message compression compresses the outgoing prompt for a single request and stores nothing; compaction writes a durable checkpoint that changes how all subsequent requests replay.

Supported agents

The agent handling the request must have compaction wired into its graph — currently chatbot (the default) and eliza. Requesting compaction on an agent without support fails loudly rather than silently ignoring the option:

{
  "code": "AGENT_COMPACTION_UNSUPPORTED",
  "message": "Agent 'research-assistant' does not support thread compaction.",
  "hint": "Omit context_management, or use an agent with compaction wired in (e.g. 'chatbot')."
}

Compaction events

When a checkpoint is written during a turn, the response says so — mirroring how the OpenAI compaction API surfaces a compaction item:

Non-streaming responses carry a top-level field:

"context_management": [{ "type": "compaction", "status": "completed" }]

Streaming responses emit one extra chunk with the same field, early in the stream before the reply text (compaction runs pre-flight). The summary itself is never streamed.

The event is purely informational — threads are server-managed, so unlike OpenAI there is nothing the client needs to store or send back. Use it to drive debug panels or usage dashboards.

Observability and billing

Each summarization call appears in tracing as a thread_compaction observation (tagged compaction) inside the request’s trace, so its spend is filterable per feature. Summarization tokens are not added to the response’s usage — they are platform-side cost, typically fractions of a cent per compaction with a small summarizer model.