Realtime voice is available on request. Contact your AI Reserve representative to have
realtimeVoiceEnabled
turned on for your organization — until then, session mints return 403. In environments where the voice relay is not
deployed they return 503.grok-voice model is
served by Grok Voice 3.0, xAI’s current flagship voice model. Billing, budgets, and usage
reporting are handled by AI Reserve the same way as every other surface.
The gateway acts as a voice-bound relay, not a generic LiteLLM passthrough. When you mint a
session you select a model and a voice; both are locked at mint and the voice is injected
server-side the moment the upstream WebSocket opens. Your AI Reserve API key never travels to
the WebSocket — only the short-lived ephemeral key does. Neither provider’s key ever reaches your
code.
Models
Both ElevenLabs models bill $0.08 per connected minute on top of their token rates. All three
models speak the same client protocol (the OpenAI Realtime event shapes below), so switching is a
change to the
model and voice fields of the mint call. Sessions are WebSocket only for
every model — there is no WebRTC or SIP transport.
The Eleven Realtime models sit behind your organization’s unverified-retention opt-in, the
same consent that unlocks the ElevenLabs text-to-speech models (ElevenLabs retains session content
by default and publishes no deletion window). Until an administrator has accepted it, Eleven
Realtime mints answer 403; grok-voice needs only realtimeVoiceEnabled.
How it works
Because a WebSocket cannot ride the normal request/response gateway, the flow has two steps:- Mint a session credential — one API call that authenticates you, reserves funds, selects
the model and voice, and returns a short-lived
ephemeral_keyandwss_url. - Connect your WebSocket directly to
wss_urlusingephemeral_key. Drive the session with audio I/O events only.
Step 1: Mint a session
POST /v1/realtime/sessions authenticates the caller, checks the org gate, resolves voice
ownership, reserves worst-case session funds from your wallet, and returns the credentials.
model selects the upstream and defaults to "grok-voice". voice selects which voice the
relay will use for this session and depends on the model:
grok-voice— any built-in voice ID fromGET /v1/voices(e.g."ara","atlas","luna") with no extra org gate, or one of your organization’s custom voice IDs fromGET /v1/custom-voices, which requires thecustomVoicesEnabledorg gate in addition torealtimeVoiceEnabled. Defaults to"ara".eleven-realtime-v1-mini/eleven-realtime-v1-max— an ElevenLabs voice ID (1–64 characters). Defaults to"brian".GET /v1/voicesand custom voice clones are xAI-only and are not accepted on Eleven Realtime sessions.
201:
ephemeral_key is an HMAC-SHA-256 signed, opaque, one-use relay ticket — not an API
key. The token itself encodes only an expiry timestamp and a random nonce; it does not
carry a voice ID. Voice authorization lives in the session record: the relay stores the
authorized model and voice and the ticket state in a DB row keyed by the ticket’s SHA-256 hash
at mint time. The ticket is valid for 5 minutes and consumed on first use — a replay is rejected
even within the TTL. Mint a new session for each conversation; do not log or share the key.
An Eleven Realtime mint answers 503 with Retry-After when the deployment has no ElevenLabs
credential configured. Nothing is charged; use grok-voice or contact support.
Step 2: Connect the WebSocket
From a server — pass the key as a bearer token header:browser_subprotocols
array from the mint response verbatim:
session.update event with the authorized voice before forwarding any client audio. The
voice is bound in the session record — the client cannot change it mid-session.
Using the OpenAI Realtime SDK
Eleven Realtime speaks the OpenAI Realtime protocol, so the OpenAI Agents SDK WebSocket transport works against the relay: point it at the mintedwss_url and hand it the
ephemeral_key as the API key. Mint on your server and send only the mint response to the
client.
realtime, openai-insecure-api-key.…). The relay accepts both. Use the SDK with
the Eleven Realtime models: its default session.update carries fields (tools, tracing,
model, …) that the relay drops for Eleven Realtime but refuses on grok-voice, whose stricter
event allowlist is described next.
Supported WebSocket events
The relay enforces a strict event allowlist. Only the following client→relay events are accepted; anything else closes the connection with code1008:
session.update is additionally sanitized. On grok-voice only instructions,
turn_detection (server_vad), and audio fields are forwarded and any other field closes the
connection. On Eleven Realtime the relay re-shapes the update into the OpenAI GA session object
and forwards instructions, output_modalities, turn_detection (server_vad or
semantic_vad, with eagerness), the audio input/output format, and the input
transcription.language; unsupported fields such as tools, tool_choice, model, speed,
and tracing are dropped rather than refused. On every model voice is forced to the value
selected at mint regardless of what the client sends.
conversation.item.create accepts message items only, with a role of user, system, or
assistant and content parts of type input_text or text (up to 32,768 characters each).
An optional item.id and previous_item_id are forwarded. Audio parts (input_audio),
function_call and function_call_output items, and any other field close the connection with
code 1008; audio goes through input_audio_buffer.append.
conversation.item.truncate requires item_id, content_index, and audio_end_ms (non-negative
integers). conversation.item.delete and conversation.item.retrieve require item_id. All are
forwarded verbatim; the provider answers an error event for an unknown item_id.
Relay→client messages are forwarded as-is from the provider. Eleven Realtime emits the OpenAI GA
event names (response.output_audio.delta, response.done, …); handle both the beta and GA
audio delta names if your client must work with every model.
Tool calling is not yet supported on any realtime model, and transcription and session resumption are not
supported in this release.
tools sent via session.update are stripped, and function_call_output items are never
forwarded. See Deferred features.Available voices
grok-voice — pass a supported built-in voice ID or one of your organization’s custom voice
IDs (requires customVoicesEnabled). The default when voice is omitted is ara. Use
GET /v1/voices to retrieve the current catalog with names, languages, and genders; catalog
additions are enabled on the synthesis surfaces after AI Reserve verifies and deploys them. To
create custom voice clones for your organization, see Text-to-speech.
Eleven Realtime — pass an ElevenLabs voice ID. The default is brian. The relay pins the
voice for the whole session. The mint only checks the ID’s shape; an unknown voice is rejected by
ElevenLabs when the session opens, so test a new voice before shipping it.
Billing and limits
Usage appears on your ledger and dashboards like any other model call and counts toward the same
wallet and budget caps. A
grok-voice row records audio seconds as its unit; an Eleven Realtime
row records the session’s token counts, with connected minutes alongside them in the row’s
metadata. Token counts are taken from ElevenLabs’ record of the session once it closes, so the
ledger row for a voice session appears shortly after the session ends rather than during it.
Audio persistence
AI Reserve does not persist audio from realtime sessions. Session metadata (duration, cost, voice ID, session ID, and for Eleven Realtime the token totals) is stored for billing and usage reporting; no audio content is retained.Enable realtime voice
- Contact your AI Reserve representative (or the Help page in the portal) to enable
realtimeVoiceEnabledfor your organization. - Optionally enable
customVoicesEnabledif you want to use custom voice clones ingrok-voicesessions, and have an administrator accept the unverified-retention opt-in if you want the Eleven Realtime models. - Mint a session and connect — the two-step flow above is the full integration.
- Watch your usage dashboard: each session appears as one audio usage row with duration and cost.
Deferred features
The following are not available in this release:- Standalone speech-to-text: transcription calls without voice synthesis. See
POST /v1/audio/transcriptionsfor the existing OpenAI-compatible transcription endpoint. - Streaming TTS: text-in, audio-out streaming without a microphone. See
POST /v1/audio/speechfor synchronous (non-streaming) TTS withgrok-tts. - SIP / telephony and WebRTC: sessions are WebSocket only; direct PSTN or SIP dial-in/dial-out and WebRTC transports are not available.
- Tool calling in realtime sessions: function calling over the realtime session is not yet available on any model.
- Session resumption: each
POST /v1/realtime/sessionsmint starts a fresh session; reconnecting to a prior session is not supported.