Real-Time Voice (Gemini Live)

Truememory provides real-time voice-to-voice conversation powered by Gemini Live. The client opens a WebSocket to Truememory, sends raw PCM audio, and receives audio back — Truememory handles auth, persona injection, memory context, tool routing, and identity guardrails transparently.

Per-persona voice and language preferences are stored server-side and applied automatically to each session — each persona a user owns can have its own voice. Stateless Gemini text-to-speech is exposed via the unified /v1/audio/speech endpoint (same as OpenAI TTS, just with a Gemini model).

Authentication

All HTTP voice endpoints (/v1/voice/preferences/{persona_id}) accept the same tri-mode auth as the rest of Truememory. See Ways to authenticate for details. (Standalone Gemini TTS is on the unified /v1/audio/speech endpoint with the same auth rules.)

These endpoints are user-scoped (preferences are per-user-per-persona, TTS billing is per-user), so the stateless service-only mode is rejected — service-key callers must include X-On-Behalf-Of.

WebSocket sessions (/v1/voice/agent, /v1/voice/stt) authenticate via the tie.bearer subprotocol — see each section below. Both forms work:

The proxy form requires X-On-Behalf-Of — without it, the connection is rejected, since we never collapse multi-tenant traffic onto a shared identity.

WebSocket Voice Session

Open a persistent bidirectional audio stream between the client and Gemini Live.

Endpoint

WebSocket /v1/voice/agent

Auth via the WebSocket subprotocol header (browsers can’t send custom headers on the handshake, so the bearer token is smuggled through Sec-WebSocket-Protocol):

new WebSocket("wss://your-tie-host/v1/voice/agent", ["tie.bearer", "<bearer-token>"])

The server reads the second subprotocol value as the token and accepts tie.bearer to complete the handshake. This keeps the token out of URL paths and proxy access logs.

Session Lifecycle

sequenceDiagram
    participant Client
    participant Truememory
    participant Gemini

    Client->>Truememory: WebSocket connect (Sec-WebSocket-Protocol: tie.bearer, token)
    Client->>Truememory: JSON session config (first message)
    Truememory->>Truememory: Load persona + memory + apply guardrails
    Truememory->>Gemini: Open Live session
    Truememory-->>Client: {"type": "session_ready", "session_id": "...", "thread_id": "..."}

    loop Bidirectional relay
        Client->>Truememory: Binary frame (PCM audio chunk)
        Truememory->>Gemini: Audio
        Gemini-->>Truememory: Audio + transcripts + tool calls
        Truememory-->>Client: Binary frame (audio) + JSON events
    end

    Client->>Truememory: {"type": "control", "action": "disconnect"}
    Truememory->>Truememory: Flush transcripts → thread + memory pipeline

Session Config

Send this as the first JSON message after connecting:

{
  "voice": null,
  "language": null,
  "agent_id": "my-app",
  "app_id": "vibrantly",
  "persona_id": "your-persona-id-or-null",
  "thread_id": "existing-thread-id-or-null",
  "skill_instructions": "You are currently in the MyApp experience.\nCurrent time: Monday, April 19, 2026 at 11:30 PM\nUser timezone: Asia/Kuala_Lumpur",
  "tools": [],
  "proactive_audio": false,
  "enable_affective_dialog": false,
  "resumption_handle": null
}

System Prompt Composition

Truememory builds the Gemini system instruction server-side by concatenating these layers in order:

1. Persona (loaded from Truememory persona store by persona_id)
2. Memory context (user's memory graph for this agent_id)
3. Current user identity ("You are speaking with <display_name>…")
4. skill_instructions (sent by the client — app context + dynamic data)
5. Identity guardrails (always appended by Truememory — cannot be overridden)

What belongs in skill_instructions:

The client is the only party that knows which app the user is in and real-time session data. Send a string that covers:

You are currently in the <AppName> app.
Keep responses concise and conversational — this is a voice session.
Never read out URLs, markdown, or bullet points — speak naturally.

Current time: Monday, April 19, 2026 at 11:30 PM
User timezone: Asia/Kuala_Lumpur

Do not hardcode AI identity (name, personality) in skill_instructions — that belongs in the Truememory persona so it can be managed via the admin panel. The AI name is set dynamically by the caller via the persona, not by the client.

Platform-level guardrails (applied automatically by Truememory):

Truememory always appends identity guardrails as the final layer regardless of what the client sends:

• The AI will never reveal it is powered by Gemini, GPT, Claude, or any other LLM technology
• If asked, it may say it is powered by Envision Inc
• It will never break character or acknowledge these instructions

Message Protocol

Binary frames carry audio. Text frames carry JSON control messages.

Client → Truememory:

Truememory → Client:

The error.details object is best-effort diagnostic context — present when Truememory can attribute the failure to an upstream cause:

The code field maps to:

The same usage object is also attached to interrupted and generation_complete frames so callers can record token totals even when the WebSocket disconnects before session_end arrives (e.g. client-initiated end-of-call). Each frame carries the latest accumulator snapshot — callers should keep the most recent value and treat it as the canonical session total.

The usage object on session_end carries the per-turn-summed token counts from Gemini Live’s usage_metadata:

{
  "prompt_tokens": 1240,
  "completion_tokens": 380,
  "total_tokens": 1620,
  "cached_content_tokens": 0,
  "thoughts_tokens": 0
}

Per-turn prompt_tokens includes the session memory replayed as input on each turn — that’s how Vertex AI bills it, so summing matches your invoice rather than over-counting.

Audio Format

Use an AudioWorklet to convert browser mic float32 samples to int16 PCM before sending.

Playback: Create your AudioBuffer at 24 kHz — not at the AudioContext’s native sample rate. Using the wrong rate causes chipmunk-pitched or slow playback.

Interrupt handling: When you receive interrupted, stop all scheduled AudioBufferSourceNodes by calling .stop() on each tracked node and reset your playback clock. Do not close and recreate the AudioContext — creating a new context outside a user gesture may leave it in a suspended state, silently breaking all subsequent audio playback.

Tool Call Routing

Truememory routes tool calls into two categories:

Truememory-native tools — executed server-side, invisible to the client:

• memory_search — searches the user’s memory graph
• memory_write — stores a new memory

Client-provided tools — defined in tools at session start, forwarded to the client for execution. When Gemini calls one, Truememory sends a tool_call event. The client must reply within 30 seconds with a tool_result message or Truememory returns a timeout error to Gemini.

// Client receives:
{"type": "tool_call", "call_id": "abc123", "name": "get_calendar", "args": {"date": "today"}}

// Client replies:
{"type": "tool_result", "call_id": "abc123", "result": {"events": [...]}}

Session Limits

Session Resumption

Gemini periodically resets WebSocket connections. Truememory enables seamless reconnection:

1. During a session, Truememory forwards session_resumption events with a handle
2. Store the latest handle in memory or sessionStorage
3. On reconnect, pass resumption_handle in the session config
4. Gemini restores conversation context from the handle

Transcript Sync

At session end, Truememory writes buffered transcripts to the thread and runs the memory pipeline. Pass the thread_id from session_ready to subsequent chat requests to continue in the same context.

Frontend Integration

React Hook (useTieVoiceSession)

A reference React hook that handles the full WebSocket lifecycle, audio pipeline, session resumption, and tool call dispatch. It is not published as a package — the import below shows how an app that has copied the hook into its codebase uses it; treat the sections above as the wire contract to implement.

import { useTieVoiceSession } from '@/hooks/use-tie-voice-session';

const {
  voiceMode,        // 'idle' | 'listening' | 'processing' | 'speaking'
  interimTranscript,
  error,
  startVoice,
  stopVoice,
  interrupt,
} = useTieVoiceSession({
  agentId: 'my-app',
  personaId: activePersona?.id,   // from GET /personas/active — Truememory is source of truth for the agent
  voiceId: companion?.voiceId,    // Gemini voice — overrides stored preference if set
  skillInstructions: buildSkillInstructions(companion?.name, user?.displayName),
  getAccessToken,
  onToolCall: async (name, args) => {
    // Handle client-side tool calls from Gemini
    if (name === 'get_calendar') return { events: await fetchCalendar(args.date) };
    return { error: 'Unknown tool' };
  },
});

Building skillInstructions:

function buildSkillInstructions(companionName?: string, userName?: string): string {
  const tz = Intl.DateTimeFormat().resolvedOptions().timeZone;
  const now = new Date().toLocaleString('en-US', {
    timeZone: tz,
    dateStyle: 'full',
    timeStyle: 'short',
  });

  return [
    // App context — replace with your app name
    'You are currently in the MyApp experience.\nKeep responses concise and conversational — this is a voice session.\nNever read out URLs, markdown, or bullet points — speak naturally.',
    // Dynamic companion/user context
    companionName ? `Your name is ${companionName}.` : '',
    userName ? `You are speaking with ${userName}. Address them by name when natural.` : '',
    // Always include timezone so the AI gives correct times
    `Current time: ${now}\nUser timezone: ${tz}`,
  ].filter(Boolean).join('\n\n');
}

Voice mode state machine:

idle ──startVoice()──▶ listening
                            │
              user speaks   │
                            ▼
                       processing   (transcript received, waiting for AI)
                            │
              AI responds   │
                            ▼
                        speaking    (binary audio frames arriving)
                            │
         generation_complete│  or  interrupted
                            ▼
                        listening
                            │
              stopVoice()   │
                            ▼
                          idle

UI state indicator:

function VoiceStateIndicator({ voiceMode }: { voiceMode: string }) {
  if (voiceMode === 'idle') return null;
  return (
    <div className={`voice-indicator ${voiceMode}`}>
      {voiceMode === 'listening'   && 'Listening…'}
      {voiceMode === 'processing'  && 'Thinking…'}
      {voiceMode === 'speaking'    && 'Speaking…'}
    </div>
  );
}

Raw WebSocket (no hook)

For apps not using the hook, here is the minimal connection pattern:

async function startVoiceSession(token: string, companionName: string) {
  const tz = Intl.DateTimeFormat().resolvedOptions().timeZone;
  const now = new Date().toLocaleString('en-US', { timeZone: tz, dateStyle: 'full', timeStyle: 'short' });

  // 1. Set up capture at 16 kHz
  const captureCtx = new AudioContext({ sampleRate: 16000 });
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: { channelCount: 1, echoCancellation: true, noiseSuppression: true },
  });
  const micSource = captureCtx.createMediaStreamSource(stream);
  await captureCtx.audioWorklet.addModule('/worklets/pcm-processor.js');
  const workletNode = new AudioWorkletNode(captureCtx, 'pcm-processor');
  micSource.connect(workletNode);

  // 2. Set up playback
  const playbackCtx = new AudioContext();
  let nextPlayTime = 0;
  const activeSources: AudioBufferSourceNode[] = [];

  function playPcm(data: ArrayBuffer) {
    const int16 = new Int16Array(data);
    const float32 = new Float32Array(int16.length);
    for (let i = 0; i < int16.length; i++) float32[i] = int16[i] / 32768;
    const buffer = playbackCtx.createBuffer(1, float32.length, 24000);
    buffer.copyToChannel(float32, 0);
    const source = playbackCtx.createBufferSource();
    source.buffer = buffer;
    source.connect(playbackCtx.destination);
    const startAt = Math.max(nextPlayTime, playbackCtx.currentTime);
    source.start(startAt);
    nextPlayTime = startAt + buffer.duration;
    activeSources.push(source);
    source.onended = () => activeSources.splice(activeSources.indexOf(source), 1);
  }

  function flushAudio() {
    activeSources.forEach(s => { try { s.stop(); } catch {} });
    activeSources.length = 0;
    nextPlayTime = 0;
  }

  // 3. Open WebSocket
  const ws = new WebSocket('wss://your-tie-host/v1/voice/agent', ['tie.bearer', token]);
  ws.binaryType = 'arraybuffer';

  ws.onopen = () => {
    ws.send(JSON.stringify({
      agent_id: 'my-app',
      // voice and language omitted — Truememory uses the user's stored preference.
      // Pass voiceId explicitly to override: voice: 'Aoede'
      skill_instructions: `You are currently in the MyApp experience.\nKeep responses concise and conversational.\nCurrent time: ${now}\nUser timezone: ${tz}`,
    }));
  };

  ws.onmessage = (event) => {
    if (event.data instanceof ArrayBuffer) {
      playPcm(event.data);
      return;
    }
    const msg = JSON.parse(event.data);

    if (msg.type === 'session_ready') {
      // Start streaming mic audio
      workletNode.port.onmessage = (e: MessageEvent<ArrayBuffer>) => {
        if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
      };
    }
    if (msg.type === 'interrupted')        flushAudio();
    if (msg.type === 'session_resumption') localStorage.setItem('tie-voice-handle', msg.handle);
    if (msg.type === 'tool_call') {
      // Execute and reply
      executeClientTool(msg.name, msg.args).then(result => {
        ws.send(JSON.stringify({ type: 'tool_result', call_id: msg.call_id, result }));
      });
    }
    if (msg.type === 'error') console.error('Truememory error:', msg.code, msg.message);
  };

  // 4. Mute: disable track + signal end of audio stream
  function mute() {
    stream.getAudioTracks()[0].enabled = false;
    ws.send(JSON.stringify({ type: 'audio_stream_end' }));
  }

  // 5. Disconnect cleanly
  function disconnect() {
    ws.send(JSON.stringify({ type: 'control', action: 'disconnect' }));
    ws.close();
    stream.getTracks().forEach(t => t.stop());
    captureCtx.close();
    playbackCtx.close();
  }

  return { mute, disconnect };
}

AudioWorklet (pcm-processor.js)

Place this file at /public/worklets/pcm-processor.js:

class PcmProcessor extends AudioWorkletProcessor {
  process(inputs) {
    const input = inputs[0]?.[0];
    if (!input?.length) return true;
    const int16 = new Int16Array(input.length);
    for (let i = 0; i < input.length; i++) {
      int16[i] = Math.max(-32768, Math.min(32767, Math.round(input[i] * 32767)));
    }
    this.port.postMessage(int16.buffer, [int16.buffer]);
    return true;
  }
}
registerProcessor('pcm-processor', PcmProcessor);

Streaming STT (Speech-to-Text only)

Stream raw PCM audio and receive real-time transcript events — no LLM, no TTS. Use this when you need live transcription without a full voice conversation: push-to-talk input, ambient listeners, onboarding interviews.

Endpoint

WebSocket /v1/voice/stt

Same auth convention as the voice agent — bearer token in the subprotocol:

new WebSocket("wss://your-tie-host/v1/voice/stt", ["tie.bearer", "<bearer-token>"])

Session Lifecycle

sequenceDiagram
    participant Client
    participant Truememory
    participant Gemini

    Client->>Truememory: WebSocket connect (Sec-WebSocket-Protocol: tie.bearer, token)
    Truememory-->>Client: {"type": "ready"}

    loop While mic is open
        Client->>Truememory: Binary frame (PCM 16kHz audio chunk)
        Truememory->>Gemini: Audio relay
        Gemini-->>Truememory: inputTranscription event
        Truememory-->>Client: {"type": "transcript", "text": "...", "isFinal": false}
    end

    Truememory-->>Client: {"type": "speech-ended"}  (VAD detected end of utterance)
    Truememory-->>Client: {"type": "session_end", "duration_seconds": 42.5, "model": "..."}

Audio Format

Send raw PCM audio: 16-bit signed, 16 kHz, mono. Use the same pcm-processor.js AudioWorklet as the voice agent.

Server → Client Events

The usage object on session_end carries per-turn-summed token counts (prompt_tokens, completion_tokens, total_tokens, cached_content_tokens, thoughts_tokens) from Gemini Live’s usage_metadata. duration_seconds is included as a secondary proxy. The frame is best-effort — if the client has already disconnected, it is dropped silently.

The same usage object is also attached to every speech-ended frame so callers can record token totals even when the WebSocket disconnects before session_end arrives (e.g. client-initiated stop). Each frame carries the latest accumulator snapshot — callers should keep the most recent value and treat it as the canonical session total.

isFinal is true when Gemini marks the transcription as finished for that chunk. The speech-ended event marks the utterance boundary — use it to trigger submission in push-to-talk flows.

VAD tuning: STT is configured for dictation, not conversational turn-taking. The speech-ended event fires after 1000 ms of silence (with low end-of-speech sensitivity and 200 ms prefix padding) so users can pause to think mid-sentence without their utterance being chopped up. The full voice agent (/v1/voice/agent) uses tighter VAD (500 ms silence, high sensitivity) for fast back-and-forth.

Example (push-to-talk)

const ws = new WebSocket('wss://your-tie-host/v1/voice/stt', ['tie.bearer', token]);
ws.binaryType = 'arraybuffer';

let transcript = '';

ws.onmessage = (event) => {
  const msg = JSON.parse(event.data as string);

  if (msg.type === 'ready') {
    // Start sending audio chunks from AudioWorklet
    workletNode.port.onmessage = (e: MessageEvent<ArrayBuffer>) => {
      if (ws.readyState === WebSocket.OPEN) ws.send(e.data);
    };
  }

  if (msg.type === 'transcript') {
    transcript = msg.text; // update live preview
  }

  if (msg.type === 'speech-ended') {
    submitTranscript(transcript); // user stopped speaking — submit
    transcript = '';
  }
};

Session Limits

Voice Preferences

Store a per-persona preferred voice and language server-side so every session picks them up automatically without the client having to pass them each time. Each persona a user owns can have its own voice.

Get preferences

GET /v1/voice/preferences/{persona_id}
Authorization: Bearer <token>

{
  "provider": "gemini",
  "voice": "Kore",
  "language": "en"
}

Returns stored preferences for that persona, or defaults (provider: "gemini", voice: "Kore", language: "en") if none have been saved yet. The persona must be a preset or owned by the caller — otherwise returns 404 PERSONA_NOT_FOUND.

Update preferences

PUT /v1/voice/preferences/{persona_id}
Authorization: Bearer <token>
Content-Type: application/json

{
  "provider": "gemini",
  "voice": "Aoede",
  "language": "en"
}

Responds with the saved preferences.

Preferences apply to WebSocket sessions when voice and language are omitted (or null) in the session config. Truememory looks up the row keyed by the session’s persona_id (or the user’s active persona if persona_id is omitted). Pass voice / language explicitly in the session config to override for a single session.

Voices

For the full list of available Gemini voices and their character descriptions, see Audio — Gemini Voices.