Skip to main content
Realtime voice is available on request. Contact your AI Reserve representative to have realtimeVoiceEnabled turned on for your organization — until then, session mints return 403. In environments where the voice relay is not deployed they return 503.
Realtime voice gives your application a live speech-to-speech session over a WebSocket with your choice of model: xAI Grok Voice or ElevenLabs Eleven Realtime. The grok-voice model is served by Grok Voice 3.0, xAI’s current flagship voice model. Billing, budgets, and usage reporting are handled by AI Reserve the same way as every other surface. The gateway acts as a voice-bound relay, not a generic LiteLLM passthrough. When you mint a session you select a model and a voice; both are locked at mint and the voice is injected server-side the moment the upstream WebSocket opens. Your AI Reserve API key never travels to the WebSocket — only the short-lived ephemeral key does. Neither provider’s key ever reaches your code.

Models

Both ElevenLabs models bill $0.08 per connected minute on top of their token rates. All three models speak the same client protocol (the OpenAI Realtime event shapes below), so switching is a change to the model and voice fields of the mint call. Sessions are WebSocket only for every model — there is no WebRTC or SIP transport. The Eleven Realtime models sit behind your organization’s unverified-retention opt-in, the same consent that unlocks the ElevenLabs text-to-speech models (ElevenLabs retains session content by default and publishes no deletion window). Until an administrator has accepted it, Eleven Realtime mints answer 403; grok-voice needs only realtimeVoiceEnabled.

How it works

Because a WebSocket cannot ride the normal request/response gateway, the flow has two steps:
  1. Mint a session credential — one API call that authenticates you, reserves funds, selects the model and voice, and returns a short-lived ephemeral_key and wss_url.
  2. Connect your WebSocket directly to wss_url using ephemeral_key. Drive the session with audio I/O events only.

Step 1: Mint a session

POST /v1/realtime/sessions authenticates the caller, checks the org gate, resolves voice ownership, reserves worst-case session funds from your wallet, and returns the credentials. model selects the upstream and defaults to "grok-voice". voice selects which voice the relay will use for this session and depends on the model:
  • grok-voice — any built-in voice ID from GET /v1/voices (e.g. "ara", "atlas", "luna") with no extra org gate, or one of your organization’s custom voice IDs from GET /v1/custom-voices, which requires the customVoicesEnabled org gate in addition to realtimeVoiceEnabled. Defaults to "ara".
  • eleven-realtime-v1-mini / eleven-realtime-v1-max — an ElevenLabs voice ID (1–64 characters). Defaults to "brian". GET /v1/voices and custom voice clones are xAI-only and are not accepted on Eleven Realtime sessions.
For an ElevenLabs session, name the model and (optionally) an ElevenLabs voice:
A successful mint returns 201:
The ephemeral_key is an HMAC-SHA-256 signed, opaque, one-use relay ticket — not an API key. The token itself encodes only an expiry timestamp and a random nonce; it does not carry a voice ID. Voice authorization lives in the session record: the relay stores the authorized model and voice and the ticket state in a DB row keyed by the ticket’s SHA-256 hash at mint time. The ticket is valid for 5 minutes and consumed on first use — a replay is rejected even within the TTL. Mint a new session for each conversation; do not log or share the key. An Eleven Realtime mint answers 503 with Retry-After when the deployment has no ElevenLabs credential configured. Nothing is charged; use grok-voice or contact support.

Step 2: Connect the WebSocket

From a server — pass the key as a bearer token header:
From a browser — browsers cannot set WebSocket headers; pass the browser_subprotocols array from the mint response verbatim:
The relay verifies the HMAC signature and expiry, reads the session record by the ticket’s SHA-256 hash to retrieve the authorized model and voice (and atomically marks the ticket consumed to prevent replay), then opens a connection to that model’s provider — xAI Grok Voice, or ElevenLabs Eleven Realtime through a server-minted ElevenLabs client secret — and sends an initial session.update event with the authorized voice before forwarding any client audio. The voice is bound in the session record — the client cannot change it mid-session.

Using the OpenAI Realtime SDK

Eleven Realtime speaks the OpenAI Realtime protocol, so the OpenAI Agents SDK WebSocket transport works against the relay: point it at the minted wss_url and hand it the ephemeral_key as the API key. Mint on your server and send only the mint response to the client.
In Node.js the SDK sends the key as a bearer header; in browsers it rides the WebSocket subprotocols (realtime, openai-insecure-api-key.…). The relay accepts both. Use the SDK with the Eleven Realtime models: its default session.update carries fields (tools, tracing, model, …) that the relay drops for Eleven Realtime but refuses on grok-voice, whose stricter event allowlist is described next.

Supported WebSocket events

The relay enforces a strict event allowlist. Only the following client→relay events are accepted; anything else closes the connection with code 1008: session.update is additionally sanitized. On grok-voice only instructions, turn_detection (server_vad), and audio fields are forwarded and any other field closes the connection. On Eleven Realtime the relay re-shapes the update into the OpenAI GA session object and forwards instructions, output_modalities, turn_detection (server_vad or semantic_vad, with eagerness), the audio input/output format, and the input transcription.language; unsupported fields such as tools, tool_choice, model, speed, and tracing are dropped rather than refused. On every model voice is forced to the value selected at mint regardless of what the client sends. conversation.item.create accepts message items only, with a role of user, system, or assistant and content parts of type input_text or text (up to 32,768 characters each). An optional item.id and previous_item_id are forwarded. Audio parts (input_audio), function_call and function_call_output items, and any other field close the connection with code 1008; audio goes through input_audio_buffer.append. conversation.item.truncate requires item_id, content_index, and audio_end_ms (non-negative integers). conversation.item.delete and conversation.item.retrieve require item_id. All are forwarded verbatim; the provider answers an error event for an unknown item_id. Relay→client messages are forwarded as-is from the provider. Eleven Realtime emits the OpenAI GA event names (response.output_audio.delta, response.done, …); handle both the beta and GA audio delta names if your client must work with every model.
Tool calling is not yet supported on any realtime model, and transcription and session resumption are not supported in this release. tools sent via session.update are stripped, and function_call_output items are never forwarded. See Deferred features.

Available voices

grok-voice — pass a supported built-in voice ID or one of your organization’s custom voice IDs (requires customVoicesEnabled). The default when voice is omitted is ara. Use GET /v1/voices to retrieve the current catalog with names, languages, and genders; catalog additions are enabled on the synthesis surfaces after AI Reserve verifies and deploys them. To create custom voice clones for your organization, see Text-to-speech. Eleven Realtime — pass an ElevenLabs voice ID. The default is brian. The relay pins the voice for the whole session. The mint only checks the ID’s shape; an unknown voice is rejected by ElevenLabs when the session opens, so test a new voice before shipping it.

Billing and limits

Usage appears on your ledger and dashboards like any other model call and counts toward the same wallet and budget caps. A grok-voice row records audio seconds as its unit; an Eleven Realtime row records the session’s token counts, with connected minutes alongside them in the row’s metadata. Token counts are taken from ElevenLabs’ record of the session once it closes, so the ledger row for a voice session appears shortly after the session ends rather than during it.

Audio persistence

AI Reserve does not persist audio from realtime sessions. Session metadata (duration, cost, voice ID, session ID, and for Eleven Realtime the token totals) is stored for billing and usage reporting; no audio content is retained.

Enable realtime voice

  1. Contact your AI Reserve representative (or the Help page in the portal) to enable realtimeVoiceEnabled for your organization.
  2. Optionally enable customVoicesEnabled if you want to use custom voice clones in grok-voice sessions, and have an administrator accept the unverified-retention opt-in if you want the Eleven Realtime models.
  3. Mint a session and connect — the two-step flow above is the full integration.
  4. Watch your usage dashboard: each session appears as one audio usage row with duration and cost.

Deferred features

The following are not available in this release:
  • Standalone speech-to-text: transcription calls without voice synthesis. See POST /v1/audio/transcriptions for the existing OpenAI-compatible transcription endpoint.
  • Streaming TTS: text-in, audio-out streaming without a microphone. See POST /v1/audio/speech for synchronous (non-streaming) TTS with grok-tts.
  • SIP / telephony and WebRTC: sessions are WebSocket only; direct PSTN or SIP dial-in/dial-out and WebRTC transports are not available.
  • Tool calling in realtime sessions: function calling over the realtime session is not yet available on any model.
  • Session resumption: each POST /v1/realtime/sessions mint starts a fresh session; reconnecting to a prior session is not supported.