---
title: "Real-Time Voice (Gemini Live)"
---

> Documentation Index
> Fetch the complete documentation index at: https://truememory-docs.mvstg.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Real-Time Voice (Gemini Live)

Truememory provides real-time voice-to-voice conversation powered by Gemini Live. The client opens a WebSocket to Truememory, sends raw PCM audio, and receives audio back — Truememory handles auth, persona injection, memory context, tool routing, and identity guardrails transparently.

Per-persona voice and language preferences are stored server-side and applied automatically to each session — each persona a user owns can have its own voice. Stateless Gemini text-to-speech is exposed via the unified [`/v1/audio/speech`](/ai-gateway/audio/#gemini-tts-same-endpoint-different-model) endpoint (same as OpenAI TTS, just with a Gemini `model`).

---

## Authentication

All HTTP voice endpoints (`/v1/voice/preferences/{persona_id}`) accept the same tri-mode auth as the rest of Truememory. See [Ways to authenticate](/authentication/overview/) for details. (Standalone Gemini TTS is on the unified [`/v1/audio/speech`](/ai-gateway/audio/#gemini-tts-same-endpoint-different-model) endpoint with the same auth rules.)

| Caller | Method | Details |
|--------|--------|---------|
| Browser / mobile app | [User tokens](/authentication/email-password/) | Sign in via Truememory Auth, pass the JWT as `Authorization: Bearer <token>` |
| Backend service | [Service accounts](/authentication/service-accounts/) | `Authorization: Bearer $AUTH_SECRET` + `X-On-Behalf-Of: <user-uuid>` |

These endpoints are user-scoped (preferences are per-user-per-persona, TTS billing is per-user), so the stateless service-only mode is rejected — service-key callers must include `X-On-Behalf-Of`.

WebSocket sessions (`/v1/voice/agent`, `/v1/voice/stt`) authenticate via the `tie.bearer` subprotocol — see each section below. Both forms work:

| Caller | Subprotocol value | Header |
|--------|-------------------|--------|
| Browser / mobile (direct) | `["tie.bearer", "<user-jwt>"]` | — |
| Backend proxy (BlinkLife/Vibrantly) | `["tie.bearer", "$AUTH_SECRET"]` | `X-On-Behalf-Of: <user-uuid>` |

The proxy form requires `X-On-Behalf-Of` — without it, the connection is rejected, since we never collapse multi-tenant traffic onto a shared identity.

---

## WebSocket Voice Session

Open a persistent bidirectional audio stream between the client and Gemini Live.

### Endpoint

```
WebSocket /v1/voice/agent
```

Auth via the WebSocket subprotocol header (browsers can't send custom headers on the handshake, so the bearer token is smuggled through `Sec-WebSocket-Protocol`):

```js
new WebSocket("wss://your-tie-host/v1/voice/agent", ["tie.bearer", "<bearer-token>"])
```

The server reads the second subprotocol value as the token and accepts `tie.bearer` to complete the handshake. This keeps the token out of URL paths and proxy access logs.

### Session Lifecycle

```mermaid
sequenceDiagram
    participant Client
    participant Truememory
    participant Gemini

    Client->>Truememory: WebSocket connect (Sec-WebSocket-Protocol: tie.bearer, token)
    Client->>Truememory: JSON session config (first message)
    Truememory->>Truememory: Load persona + memory + apply guardrails
    Truememory->>Gemini: Open Live session
    Truememory-->>Client: {"type": "session_ready", "session_id": "...", "thread_id": "..."}

    loop Bidirectional relay
        Client->>Truememory: Binary frame (PCM audio chunk)
        Truememory->>Gemini: Audio
        Gemini-->>Truememory: Audio + transcripts + tool calls
        Truememory-->>Client: Binary frame (audio) + JSON events
    end

    Client->>Truememory: {"type": "control", "action": "disconnect"}
    Truememory->>Truememory: Flush transcripts → thread + memory pipeline
```

### Session Config

Send this as the **first JSON message** after connecting:

```json
{
  "voice": null,
  "language": null,
  "agent_id": "my-app",
  "app_id": "vibrantly",
  "persona_id": "your-persona-id-or-null",
  "thread_id": "existing-thread-id-or-null",
  "skill_instructions": "You are currently in the MyApp experience.\nCurrent time: Monday, April 19, 2026 at 11:30 PM\nUser timezone: Asia/Kuala_Lumpur",
  "tools": [],
  "proactive_audio": false,
  "enable_affective_dialog": false,
  "resumption_handle": null
}
```

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `voice` | string \| null | null | Voice name override (see [Voices](#voices)). When null, Truememory uses the stored preference for the session's persona (or falls back to the user's active persona) |
| `language` | string \| null | null | Language code override. When null, Truememory uses the stored preference for the session's persona (or falls back to the user's active persona) |
| `agent_id` | string | `chatbot` | A **scoping key** for memory and thread history — unlike the chat API's `X-Agent-Id`, it does not select an agent from the [registry](/ai-gateway/agents/) and any stable string is valid. Use one value per app (e.g. `"blinklife"`) so voice memory stays consistent |
| `app_id` | string \| null | null | Application ID for thread scoping — the WS-equivalent of the `X-App-Id` header on `/v1/chat/completions`. Without it, voice transcripts persist with `app_id=NULL` and are excluded from app-scoped `GET /threads` queries. Browsers can't send custom headers on WS upgrade, so the value travels in this field instead. |
| `persona_id` | string \| null | null | Truememory persona to load. Persona sets the AI's identity and user-level preferences |
| `thread_id` | string \| null | null | Existing thread for context continuity. Omit to create a new session |
| `skill_instructions` | string \| null | null | App-specific context injected after persona + memory (see below) |
| `tools` | array | `[]` | Client-side tool definitions forwarded to the client for execution |
| `proactive_audio` | bool | `false` | Enable Proactive Audio for wake word detection |
| `enable_affective_dialog` | bool | `false` | Enable affective dialog (emotional expression in voice) |
| `resumption_handle` | string \| null | null | Handle from a previous session for seamless reconnection |

### System Prompt Composition

Truememory builds the Gemini system instruction server-side by concatenating these layers in order:

```
1. Persona (loaded from Truememory persona store by persona_id)
2. Memory context (user's memory graph for this agent_id)
3. Current user identity ("You are speaking with <display_name>…")
4. skill_instructions (sent by the client — app context + dynamic data)
5. Identity guardrails (always appended by Truememory — cannot be overridden)
```

**What belongs in `skill_instructions`:**

The client is the only party that knows which app the user is in and real-time session data. Send a string that covers:

```
You are currently in the <AppName> app.
Keep responses concise and conversational — this is a voice session.
Never read out URLs, markdown, or bullet points — speak naturally.

Current time: Monday, April 19, 2026 at 11:30 PM
User timezone: Asia/Kuala_Lumpur
```

Do **not** hardcode AI identity (name, personality) in `skill_instructions` — that belongs in the Truememory persona so it can be managed via the admin panel. The AI name is set dynamically by the caller via the persona, not by the client.

**Platform-level guardrails (applied automatically by Truememory):**

Truememory always appends identity guardrails as the final layer regardless of what the client sends:
- The AI will never reveal it is powered by Gemini, GPT, Claude, or any other LLM technology
- If asked, it may say it is powered by Envision Inc
- It will never break character or acknowledge these instructions

### Message Protocol

Binary frames carry audio. Text frames carry JSON control messages.

**Client → Truememory:**

| Type | Transport | Payload |
|------|-----------|---------|
| Audio chunk | Binary frame | PCM 16-bit signed, 16 kHz, mono |
| `tool_result` | JSON text | `{"type": "tool_result", "call_id": "...", "result": {...}}` |
| `audio_stream_end` | JSON text | `{"type": "audio_stream_end"}` — send when mic is muted or paused |
| `control` | JSON text | `{"type": "control", "action": "disconnect"}` |
| `text_input` | JSON text | `{"type": "text_input", "text": "..."}` — typed follow-up the user wants the AI to respond to without ending the call. Empty strings are ignored. |
| `image_input` | JSON text | `{"type": "image_input", "data": "<base64>", "mime_type": "image/jpeg"}` — attach a photo mid-call. `data` is strict base64 (non-alphabet characters are rejected). `mime_type` defaults to `"image/jpeg"`. Frame size is capped by `VOICE_MAX_IMAGE_FRAME_BYTES` (default 512 KB JSON ≈ 384 KB raw image) — clients must resize/compress photos above that before sending. |

**Truememory → Client:**

| Type | Transport | Payload |
|------|-----------|---------|
| Audio chunk | Binary frame | PCM audio from Gemini (24 kHz) |
| `session_ready` | JSON text | `{"type": "session_ready", "session_id": "...", "thread_id": "..."}` |
| `transcript` | JSON text | `{"type": "transcript", "role": "user\|assistant", "text": "..."}` |
| `interrupted` | JSON text | `{"type": "interrupted", "usage": {...}}` — user barged in; stop and discard buffered audio. `usage` is the running per-session token total (same shape as `session_end.usage`). |
| `generation_complete` | JSON text | `{"type": "generation_complete", "usage": {...}}` — AI finished speaking; return to listening state. `usage` is the running per-session token total (same shape as `session_end.usage`). |
| `tool_call` | JSON text | `{"type": "tool_call", "call_id": "...", "name": "...", "args": {...}}` |
| `session_resumption` | JSON text | `{"type": "session_resumption", "handle": "..."}` — store for reconnection |
| `session_end` | JSON text | `{"type": "session_end", "duration_seconds": float, "model": "...", "usage": {...}}` — sent right before the WebSocket closes; record for usage tracking |
| `go_away` | JSON text | `{"type": "go_away", "time_left": "..."}` — Gemini signaling imminent disconnect |
| `error` | JSON text | `{"type": "error", "code": "...", "message": "...", "details": {...}}` |

The `error.details` object is best-effort diagnostic context — present when Truememory can attribute the failure to an upstream cause:

| Field | Type | When present |
|--|--|--|
| `model` | string | Always on upstream errors |
| `location` | string | Always on upstream errors — the Vertex AI region Truememory called |
| `upstream_code` | int | Set when the failure originated from Gemini (HTTP status) |
| `upstream_status` | string | Set when the failure originated from Gemini (e.g. `PERMISSION_DENIED`, `RESOURCE_EXHAUSTED`) |

The `code` field maps to:

| `code` | Meaning |
|--|--|
| `VOICE_AUTH_FAILED` | Client token rejected, or upstream 401 from Vertex AI |
| `VOICE_MODEL_UNAVAILABLE` | Voice model not accessible at the configured project/region (upstream 403/404) |
| `VOICE_RATE_LIMIT_EXCEEDED` | Upstream quota exhausted (HTTP 429) — retry with backoff |
| `VOICE_SESSION_LIMIT_EXCEEDED` | Per-user concurrent-session cap hit |
| `VOICE_SESSION_EXPIRED` | Session ran past `VOICE_SESSION_TIMEOUT` |
| `VOICE_SESSION_FAILED` | Generic — upstream 4xx/5xx with no specific mapping, or non-upstream transport error |

The same `usage` object is also attached to `interrupted` and `generation_complete` frames so callers can record token totals even when the WebSocket disconnects before `session_end` arrives (e.g. client-initiated end-of-call). Each frame carries the latest accumulator snapshot — callers should keep the most recent value and treat it as the canonical session total.

The `usage` object on `session_end` carries the per-turn-summed token counts from Gemini Live's `usage_metadata`:

```json
{
  "prompt_tokens": 1240,
  "completion_tokens": 380,
  "total_tokens": 1620,
  "cached_content_tokens": 0,
  "thoughts_tokens": 0
}
```

Per-turn `prompt_tokens` includes the session memory replayed as input on each turn — that's how Vertex AI bills it, so summing matches your invoice rather than over-counting.

### Audio Format

| Direction | Format |
|-----------|--------|
| Client → Truememory | PCM 16-bit signed, 16 kHz, mono |
| Truememory → Client | PCM 16-bit signed, 24 kHz, mono |

Use an [AudioWorklet](https://developer.mozilla.org/en-US/docs/Web/API/AudioWorklet) to convert browser mic float32 samples to int16 PCM before sending.

> **Playback:** Create your `AudioBuffer` at **24 kHz** — not at the `AudioContext`'s native sample rate. Using the wrong rate causes chipmunk-pitched or slow playback.

> **Interrupt handling:** When you receive `interrupted`, stop all scheduled `AudioBufferSourceNode`s by calling `.stop()` on each tracked node and reset your playback clock. Do **not** close and recreate the `AudioContext` — creating a new context outside a user gesture may leave it in a suspended state, silently breaking all subsequent audio playback.

### Tool Call Routing

Truememory routes tool calls into two categories:

**Truememory-native tools** — executed server-side, invisible to the client:
- `memory_search` — searches the user's memory graph
- `memory_write` — stores a new memory

**Client-provided tools** — defined in `tools` at session start, forwarded to the client for execution. When Gemini calls one, Truememory sends a `tool_call` event. The client must reply within 30 seconds with a `tool_result` message or Truememory returns a timeout error to Gemini.

```json
// Client receives:
{"type": "tool_call", "call_id": "abc123", "name": "get_calendar", "args": {"date": "today"}}

// Client replies:
{"type": "tool_result", "call_id": "abc123", "result": {"events": [...]}}
```

### Session Limits

| Limit | Value |
|-------|-------|
| Max session duration | `VOICE_SESSION_TIMEOUT` (default 900s) |
| Gemini native limit | 15 min (Truememory enables context window compression to extend) |
| Tool call timeout | 30s per call |
| Audio send timeout | 5s (slow clients are disconnected) |

### Session Resumption

Gemini periodically resets WebSocket connections. Truememory enables seamless reconnection:

1. During a session, Truememory forwards `session_resumption` events with a `handle`
2. Store the latest handle in memory or `sessionStorage`
3. On reconnect, pass `resumption_handle` in the session config
4. Gemini restores conversation context from the handle

### Transcript Sync

At session end, Truememory writes buffered transcripts to the thread and runs the memory pipeline. Pass the `thread_id` from `session_ready` to subsequent chat requests to continue in the same context.

---

## Frontend Integration

### React Hook (`useTieVoiceSession`)

A reference React hook that handles the full WebSocket lifecycle, audio pipeline, session resumption, and tool call dispatch. It is not published as a package — the import below shows how an app that has copied the hook into its codebase uses it; treat the sections above as the wire contract to implement.

```ts
import { useTieVoiceSession } from '@/hooks/use-tie-voice-session';

const {
  voiceMode,        // 'idle' | 'listening' | 'processing' | 'speaking'
  interimTranscript,
  error,
  startVoice,
  stopVoice,
  interrupt,
} = useTieVoiceSession({
  agentId: 'my-app',
  personaId: activePersona?.id,   // from GET /personas/active — Truememory is source of truth for the agent
  voiceId: companion?.voiceId,    // Gemini voice — overrides stored preference if set
  skillInstructions: buildSkillInstructions(companion?.name, user?.displayName),
  getAccessToken,
  onToolCall: async (name, args) => {
    // Handle client-side tool calls from Gemini
    if (name === 'get_calendar') return { events: await fetchCalendar(args.date) };
    return { error: 'Unknown tool' };
  },
});
```

**Building `skillInstructions`:**

```ts
function buildSkillInstructions(companionName?: string, userName?: string): string {
  const tz = Intl.DateTimeFormat().resolvedOptions().timeZone;
  const now = new Date().toLocaleString('en-US', {
    timeZone: tz,
    dateStyle: 'full',
    timeStyle: 'short',
  });

  return [
    // App context — replace with your app name
    'You are currently in the MyApp experience.\nKeep responses concise and conversational — this is a voice session.\nNever read out URLs, markdown, or bullet points — speak naturally.',
    // Dynamic companion/user context
    companionName ? `Your name is ${companionName}.` : '',
    userName ? `You are speaking with ${userName}. Address them by name when natural.` : '',
    // Always include timezone so the AI gives correct times
    `Current time: ${now}\nUser timezone: ${tz}`,
  ].filter(Boolean).join('\n\n');
}
```

**Voice mode state machine:**

```
idle ──startVoice()──▶ listening
                            │
              user speaks   │
                            ▼
                       processing   (transcript received, waiting for AI)
                            │
              AI responds   │
                            ▼
                        speaking    (binary audio frames arriving)
                            │
         generation_complete│  or  interrupted
                            ▼
                        listening
                            │
              stopVoice()   │
                            ▼
                          idle
```

**UI state indicator:**

```tsx
function VoiceStateIndicator({ voiceMode }: { voiceMode: string }) {
  if (voiceMode === 'idle') return null;
  return (
    <div className={`voice-indicator ${voiceMode}`}>
      {voiceMode === 'listening'   && 'Listening…'}
      {voiceMode === 'processing'  && 'Thinking…'}
      {voiceMode === 'speaking'    && 'Speaking…'}
    </div>
  );
}
```

### Raw WebSocket (no hook)

For apps not using the hook, here is the minimal connection pattern:

```ts
async function startVoiceSession(token: string, companionName: string) {
  const tz = Intl.DateTimeFormat().resolvedOptions().timeZone;
  const now = new Date().toLocaleString('en-US', { timeZone: tz, dateStyle: 'full', timeStyle: 'short' });

  // 1. Set up capture at 16 kHz
  const captureCtx = new AudioContext({ sampleRate: 16000 });
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
  });
  const micSource = captureCtx.createMediaStreamSource(stream);
  await captureCtx.audioWorklet.addModule('/worklets/pcm-processor.js');
  const workletNode = new AudioWorkletNode(captureCtx, 'pcm-processor');
  micSource.connect(workletNode);

  // 2. Set up playback
  const playbackCtx = new AudioContext();
  let nextPlayTime = 0;
  const activeSources: AudioBufferSourceNode[] = [];

  function playPcm(data: ArrayBuffer) {
    const int16 = new Int16Array(data);
    const float32 = new Float32Array(int16.length);
    for (let i = 0; i < int16.length; i++) float32[i] = int16[i] / 32768;
    const buffer = playbackCtx.createBuffer(1, float32.length, 24000);
    buffer.copyToChannel(float32, 0);
    const source = playbackCtx.createBufferSource();
    source.buffer = buffer;
    source.connect(playbackCtx.destination);
    const startAt = Math.max(nextPlayTime, playbackCtx.currentTime);
    source.start(startAt);
    nextPlayTime = startAt + buffer.duration;
    activeSources.push(source);
    source.onended = () => activeSources.splice(activeSources.indexOf(source), 1);
  }

  function flushAudio() {
    activeSources.forEach(s => { try { s.stop(); } catch {} });
    activeSources.length = 0;
    nextPlayTime = 0;
  }

  // 3. Open WebSocket
  const ws = new WebSocket('wss://your-tie-host/v1/voice/agent', ['tie.bearer', token]);
  ws.binaryType = 'arraybuffer';

  ws.onopen = () => {
    ws.send(JSON.stringify({
      agent_id: 'my-app',
      // voice and language omitted — Truememory uses the user's stored preference.
      // Pass voiceId explicitly to override: voice: 'Aoede'
      skill_instructions: `You are currently in the MyApp experience.\nKeep responses concise and conversational.\nCurrent time: ${now}\nUser timezone: ${tz}`,
    }));
  };

  ws.onmessage = (event) => {
    if (event.data instanceof ArrayBuffer) {
      playPcm(event.data);
      return;
    }
    const msg = JSON.parse(event.data);

    if (msg.type === 'session_ready') {
      // Start streaming mic audio
      workletNode.port.onmessage = (e: MessageEvent<ArrayBuffer>) => {
        if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
      };
    }
    if (msg.type === 'interrupted')        flushAudio();
    if (msg.type === 'session_resumption') localStorage.setItem('tie-voice-handle', msg.handle);
    if (msg.type === 'tool_call') {
      // Execute and reply
      executeClientTool(msg.name, msg.args).then(result => {
        ws.send(JSON.stringify({ type: 'tool_result', call_id: msg.call_id, result }));
      });
    }
    if (msg.type === 'error') console.error('Truememory error:', msg.code, msg.message);
  };

  // 4. Mute: disable track + signal end of audio stream
  function mute() {
    stream.getAudioTracks()[0].enabled = false;
    ws.send(JSON.stringify({ type: 'audio_stream_end' }));
  }

  // 5. Disconnect cleanly
  function disconnect() {
    ws.send(JSON.stringify({ type: 'control', action: 'disconnect' }));
    ws.close();
    stream.getTracks().forEach(t => t.stop());
    captureCtx.close();
    playbackCtx.close();
  }

  return { mute, disconnect };
}
```

### AudioWorklet (pcm-processor.js)

Place this file at `/public/worklets/pcm-processor.js`:

```js
class PcmProcessor extends AudioWorkletProcessor {
  process(inputs) {
    const input = inputs[0]?.[0];
    if (!input?.length) return true;
    const int16 = new Int16Array(input.length);
    for (let i = 0; i < input.length; i++) {
      int16[i] = Math.max(-32768, Math.min(32767, Math.round(input[i] * 32767)));
    }
    this.port.postMessage(int16.buffer, [int16.buffer]);
    return true;
  }
}
registerProcessor('pcm-processor', PcmProcessor);
```

---

## Streaming STT (Speech-to-Text only)

Stream raw PCM audio and receive real-time transcript events — no LLM, no TTS. Use this when you need live transcription without a full voice conversation: push-to-talk input, ambient listeners, onboarding interviews.

### Endpoint

```
WebSocket /v1/voice/stt
```

Same auth convention as the voice agent — bearer token in the subprotocol:

```js
new WebSocket("wss://your-tie-host/v1/voice/stt", ["tie.bearer", "<bearer-token>"])
```

### Session Lifecycle

```mermaid
sequenceDiagram
    participant Client
    participant Truememory
    participant Gemini

    Client->>Truememory: WebSocket connect (Sec-WebSocket-Protocol: tie.bearer, token)
    Truememory-->>Client: {"type": "ready"}

    loop While mic is open
        Client->>Truememory: Binary frame (PCM 16kHz audio chunk)
        Truememory->>Gemini: Audio relay
        Gemini-->>Truememory: inputTranscription event
        Truememory-->>Client: {"type": "transcript", "text": "...", "isFinal": false}
    end

    Truememory-->>Client: {"type": "speech-ended"}  (VAD detected end of utterance)
    Truememory-->>Client: {"type": "session_end", "duration_seconds": 42.5, "model": "..."}
```

### Audio Format

Send raw PCM audio: **16-bit signed, 16 kHz, mono**. Use the same `pcm-processor.js` AudioWorklet as the voice agent.

### Server → Client Events

| Event | Payload | When |
|-------|---------|------|
| `ready` | `{"type": "ready"}` | Gemini session open, audio accepted |
| `transcript` | `{"type": "transcript", "text": "...", "isFinal": bool}` | Transcription chunk received |
| `speech-ended` | `{"type": "speech-ended", "usage": {...}}` | VAD detected end of utterance. `usage` is the running per-session token total (same shape as `session_end.usage`). |
| `session_end` | `{"type": "session_end", "duration_seconds": float, "model": "...", "usage": {...}}` | Sent right before the WebSocket closes — use this to record session usage |
| `go_away` | `{"type": "go_away", "time_left": "..."}` | Gemini signaling imminent disconnect |
| `error` | `{"type": "error", "code": "...", "message": "..."}` | Auth failure, session limit, or upstream error |

The `usage` object on `session_end` carries per-turn-summed token counts (`prompt_tokens`, `completion_tokens`, `total_tokens`, `cached_content_tokens`, `thoughts_tokens`) from Gemini Live's `usage_metadata`. `duration_seconds` is included as a secondary proxy. The frame is best-effort — if the client has already disconnected, it is dropped silently.

The same `usage` object is also attached to every `speech-ended` frame so callers can record token totals even when the WebSocket disconnects before `session_end` arrives (e.g. client-initiated stop). Each frame carries the latest accumulator snapshot — callers should keep the most recent value and treat it as the canonical session total.

`isFinal` is `true` when Gemini marks the transcription as finished for that chunk. The `speech-ended` event marks the utterance boundary — use it to trigger submission in push-to-talk flows.

**VAD tuning:** STT is configured for **dictation**, not conversational turn-taking. The `speech-ended` event fires after **1000 ms of silence** (with low end-of-speech sensitivity and 200 ms prefix padding) so users can pause to think mid-sentence without their utterance being chopped up. The full voice agent (`/v1/voice/agent`) uses tighter VAD (500 ms silence, high sensitivity) for fast back-and-forth.

### Example (push-to-talk)

```ts
const ws = new WebSocket('wss://your-tie-host/v1/voice/stt', ['tie.bearer', token]);
ws.binaryType = 'arraybuffer';

let transcript = '';

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data as string);

  if (msg.type === 'ready') {
    // Start sending audio chunks from AudioWorklet
    workletNode.port.onmessage = (e: MessageEvent<ArrayBuffer>) => {
      if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
    };
  }

  if (msg.type === 'transcript') {
    transcript = msg.text; // update live preview
  }

  if (msg.type === 'speech-ended') {
    submitTranscript(transcript); // user stopped speaking — submit
    transcript = '';
  }
};
```

### Session Limits

| Limit | Value |
|-------|-------|
| Max session duration | `VOICE_SESSION_TIMEOUT` (default 900s) |
| Max concurrent sessions per user | `VOICE_MAX_CONCURRENT_SESSIONS_PER_USER` (default 3, shared with voice agent) |
| Audio send timeout | 5s (slow clients are disconnected) |

---

## Voice Preferences

Store a per-persona preferred voice and language server-side so every session picks them up automatically without the client having to pass them each time. Each persona a user owns can have its own voice.

### Get preferences

```
GET /v1/voice/preferences/{persona_id}
Authorization: Bearer <token>
```

```json
{
  "provider": "gemini",
  "voice": "Kore",
  "language": "en"
}
```

Returns stored preferences for that persona, or defaults (`provider: "gemini"`, `voice: "Kore"`, `language: "en"`) if none have been saved yet. The persona must be a preset or owned by the caller — otherwise returns `404 PERSONA_NOT_FOUND`.

### Update preferences

```
PUT /v1/voice/preferences/{persona_id}
Authorization: Bearer <token>
Content-Type: application/json
```

```json
{
  "provider": "gemini",
  "voice": "Aoede",
  "language": "en"
}
```

| Field | Type | Description |
|-------|------|-------------|
| `provider` | string | TTS provider. Currently only `"gemini"` is supported |
| `voice` | string | Voice name (see [Voices](#voices)) |
| `language` | string | BCP-47 language code (e.g. `"en"`, `"es"`, `"ms"`) |

Responds with the saved preferences.

Preferences apply to WebSocket sessions when `voice` and `language` are omitted (or null) in the session config. Truememory looks up the row keyed by the session's `persona_id` (or the user's active persona if `persona_id` is omitted). Pass `voice` / `language` explicitly in the session config to override for a single session.

---

## Voices

For the full list of available Gemini voices and their character descriptions, see [Audio — Gemini Voices](/ai-gateway/audio/#gemini-voices).

Source: https://truememory-docs.mvstg.com/ai-gateway/voice/index.mdx
