Video Full-Duplex
Video full-duplex protocol for the Realtime API
Video full-duplex mode is for realtime audio/video conversation. The client continuously sends 16 kHz audio and can include JPEG video frames in each input packet; the server returns listen, text, and audio output as response.output.delta.
Before connecting, obtain your own API key and configure authentication as described in the overview. Browser clients also need an allowed page Origin.
Connection
wss://api.modelbest.ai/v1/realtime?mode=video&model=MiniCPM-o-4.5-Realtime| Item | Value |
|---|---|
| Frame format | JSON text frames |
| Input audio | 16 kHz mono float32 PCM, base64 encoded |
| Input video | JPEG, base64 encoded, under input.video_frames |
| Output audio | 24 kHz mono float32 PCM, base64 encoded |
| Total session duration limit | 300 seconds |
Initialize
After connecting, wait for session.queue_done, then send session.init:
{
"type": "session.init",
"payload": {
"system_prompt": "You are a helpful video voice assistant.",
"config": {
"length_penalty": 1.1
},
"voice": {
"ref_audio_base64": "<base64 float32 PCM>",
"tts_ref_audio_base64": "<base64 float32 PCM>"
}
}
}| Field | Type | Required | Description |
|---|---|---|---|
system_prompt | string | No | System prompt. instructions is also accepted |
config | object | No | Duplex inference settings; length_penalty is commonly used |
voice.ref_audio_base64 | string | No | LLM reference audio |
voice.tts_ref_audio_base64 | string | No | TTS reference audio; can fall back to the LLM reference audio |
The server sends session.created after initialization completes.
Send Input
{
"type": "input.append",
"input": {
"audio": "<base64 float32 PCM, 16 kHz mono>",
"video_frames": ["<base64 JPEG>"],
"force_listen": false,
"max_slice_nums": 1
}
}| Field | Type | Required | Description |
|---|---|---|---|
audio | string | Yes | 16 kHz mono float32 PCM. The browser demo usually sends one chunk per second |
video_frames | string[] | No | JPEG frame list. In video mode, send frames continuously when available |
force_listen | bool | No | Force the model back to listen state |
max_slice_nums | int | No | Video slice count for this input, default 1 |
hints.force_listen | bool | No | Equivalent position for force_listen |
hints.max_slice_nums | int | No | Equivalent position for max_slice_nums |
Receive Output
Model keeps listening:
{
"type": "response.output.delta",
"kind": "listen",
"session_id": "sess_xxx",
"metrics": {
"kv_cache_length": 1024,
"vision_slices": 1,
"vision_tokens": 64
}
}Text delta:
{
"type": "response.output.delta",
"kind": "text",
"text": "I can see",
"response_id": "resp_xxx",
"metrics": {}
}Audio delta:
{
"type": "response.output.delta",
"kind": "audio",
"audio": "<base64 float32 PCM, 24 kHz mono>",
"response_id": "resp_xxx",
"metrics": {}
}Text and audio are separate deltas and are not guaranteed to map one-to-one. Clients should update captions and the audio playback queue independently; kind=listen means the model has returned to listening for user input.
Timeline
Client connects
<- session.queued / session.queue_update optional
<- session.queue_done
-> session.init { payload: {...} }
<- session.created
-> input.append { input: { audio, video_frames } }
-> input.append { input: { audio, video_frames } }
<- response.output.delta { kind: "listen" }
-> input.append { input: { audio, video_frames } }
<- response.output.delta { kind: "text", text: "..." }
<- response.output.delta { kind: "audio", audio: "..." }
<- response.output.delta { kind: "listen" }
-> session.close
<- session.closedVideo full-duplex mode does not use response.done for every output turn. The boundary is represented by kind=listen, later input, and session.closed.
Close
{
"type": "session.close",
"reason": "user_stop"
}The server makes a best effort to return session.closed. If the 300 second limit is reached, the gateway sends:
{
"type": "session.closed",
"reason": "timeout"
}