Mode C — full-duplex agent
The full-duplex conversational transport: the hello and ready handshake, PCM in both directions, the mandatory mic-mute rule, button barge-in, and using TCP backpressure as your pacing mechanism.
13 min read
On this page
The richest mode: a real two-way conversation with an AI agent, PCM in both directions over one socket. This is the mode behind the table-side ordering terminal, and the default for the Table Top.
Transport
wss://<host>/ws- TLS 1.2 minimum, 1.3 preferred. Validate the server certificate. Ship a CA bundle in flash; pin the CA if you pin anything, never a leaf, or certificate rotation bricks your fleet.
- SNI required.
- Two frame types on one socket, distinguished by WebSocket opcode. Binary 0x2 carries raw PCM with no header and no envelope — the WebSocket frame is the envelope. Text 0x1 carries UTF-8 JSON control messages.
- Do not add your own length prefix or sequence number. Do not batch audio frames.
Audio
| Uplink (device → server) | Downlink (server → device) | |
|---|---|---|
| Encoding | s16le mono | s16le mono |
| Rate | 16 000 Hz exactly | 16 000 Hz |
| Frame | 20 ms = 320 samples = 640 B (10–40 ms acceptable) | 40 ms = 640 samples = 1280 B |
One WebSocket binary frame per audio frame. Batching adds latency directly onto turn detection, which is the part the user feels.
Downlink is not paced — use TCP backpressure
The server writes speech audio to the socket as fast as the socket accepts it. Text-to-speech generates far faster than real time, so a ten-second reply is roughly 320 KB arriving in a burst.
Two consequences worth designing around: your buffer occupancy is your pacing mechanism, and your playback buffer is the only jitter absorption anywhere in the system. Report its high-water mark in telemetry so the buffer can be sized on evidence rather than on a guess.
Handshake
The first message after the upgrade must be a text frame. Declare audio honestly — it is what you will actually send, not what you wish you sent. Same for capabilities.
{
"type": "hello",
"protocol": 1,
"mac": "AA:BB:CC:DD:EE:FF",
"session": "b7f3a1c2",
"property_id": 4,
"table": 5,
"firmware_version": "2.1.4",
"ota_version": "2.1.4",
"audio": { "encoding": "pcm_s16le", "sample_rate": 16000,
"channels": 1, "frame_ms": 20 },
"capabilities": { "aec": false, "mic_mute_on_playback": true,
"button_interrupt": true, "playback_buffer_ms": 400 }
}Send no audio until ready arrives.
{
"type": "ready",
"session_id": "srv-9f2c",
"property_id": 4,
"table": 5,
"currency": "PKR",
"tts": { "encoding": "pcm_s16le", "sample_rate": 16000,
"channels": 1, "frame_ms": 40 },
"tail_ms": 300,
"heartbeat_interval_s": 20
}Or a failure, after which the socket closes.
| Code | Meaning | Device must |
|---|---|---|
| unknown_device | MAC not provisioned | Stop. Fault indicator. No retry loop |
| unknown_property | property_id invalid | Stop. Provisioning fault |
| protocol_unsupported | Version mismatch | Stop. Needs an update |
| menu_unavailable | Backend unreachable | Retry with backoff |
| busy | Server at capacity | Retry with backoff |
Control messages
| Device → server | When | Payload |
|---|---|---|
| hello | First message | See above |
| interrupt | Button pressed while the agent speaks | {"type":"interrupt"} |
| bye | Clean end | {"type":"bye","reason":"user_ended"} |
| error | Device fault worth reporting | {"type":"error","code":"…","message":"…"} |
| Server → device | Meaning | Device action |
|---|---|---|
| ready | Handshake accepted | Start streaming mic audio |
| speaking_start | Agent audio follows | Mute the microphone |
| speaking_end | Agent audio finished | Unmute after tail_ms |
| order_update | Current transaction state | Optional: drive a display |
| error | Fault | Per the code table above |
| close | Server ending the session | Close cleanly |
The mic mute rule — mandatory, not tunable
The microphone must be muted whenever the speaker is producing sound, plus a tail of tail_ms (default 300 ms) afterwards.
Muted means stop sending uplink audio frames entirely. Do not send silence and do not send attenuated audio — anything you send keeps the recognition socket open and burns billable audio seconds. tail_ms arrives in ready so it can be tuned without an update; treat the server's value as authoritative and default to 300 ms if absent.
Button semantics
| State | Press | Action |
|---|---|---|
| IDLE | short | Start session: connect, handshake |
| SPEAKING | short | Send interrupt, stop playback immediately, flush the playback buffer, unmute |
| LISTENING | short | Reserved — no action in v1 |
| any | long (≥ 2 s) | Send bye, close the socket, return to IDLE |
On interrupt, stop audio locally without waiting for the server. The person pressed the button because they want the talking to stop now; a round trip is not an acceptable delay. Send interrupt in parallel so the server abandons the rest of the turn. Debounce 20 to 50 ms — a bouncing button sends a burst of interrupts.
Heartbeat and liveness
Use WebSocket protocol-level ping and pong (RFC 6455 opcodes 0x9 and 0xA), not application JSON.
| Parameter | Value |
|---|---|
| Server → device ping | Every 20 s (heartbeat_interval_s in ready) |
| Device pong deadline | 10 s |
| Declare dead | Two consecutive missed pongs → reconnect |
- TCP does not notice a device that lost power or left the Wi-Fi. The socket stays open server-side until TCP keepalive fires, which on a Linux default is two hours — and until then the server holds a pipeline and a recognition socket for a terminal with nobody at it.
- Load balancers kill idle connections. Nginx, ALB and Azure Application Gateway commonly default to 60 seconds. The design deliberately goes silent when idle, so without heartbeat traffic the proxy drops a perfectly healthy connection.
A reference client
import asyncio, json, websockets
URL = "wss://api.yourcompany.com/ws"
HELLO = {
"type": "hello", "protocol": 1,
"mac": "AA:BB:CC:DD:EE:FF", "session": "b7f3a1c2",
"property_id": 4, "table": 5,
"firmware_version": "2.1.4", "ota_version": "2.1.4",
"audio": {"encoding": "pcm_s16le", "sample_rate": 16000,
"channels": 1, "frame_ms": 20},
"capabilities": {"aec": False, "mic_mute_on_playback": True,
"button_interrupt": True, "playback_buffer_ms": 400},
}
PERMANENT = {"unknown_device", "unknown_property", "protocol_unsupported"}
async def session(mic, speaker, button):
async with websockets.connect(URL, ping_interval=None, # server pings us
max_queue=8) as ws: # bounded: backpressure
await ws.send(json.dumps(HELLO))
ready = json.loads(await ws.recv())
if ready["type"] == "error":
if ready["code"] in PERMANENT:
raise SystemExit(f"permanent: {ready['code']}") # never retry
raise ConnectionError(ready["code"]) # retry w/ backoff
tail_ms = ready.get("tail_ms", 300)
muted = asyncio.Event()
async def uplink():
async for frame in mic: # 640-byte s16le buffers
if muted.is_set():
continue # drop, do NOT send silence
await ws.send(frame)
async def downlink():
async for msg in ws:
if isinstance(msg, bytes):
# Awaiting playback is the pacing mechanism: the receive
# window closes behind us and TCP throttles the server.
await speaker.play(msg)
continue
evt = json.loads(msg)
if evt["type"] == "speaking_start":
muted.set()
elif evt["type"] == "speaking_end":
await asyncio.sleep(tail_ms / 1000)
muted.clear()
elif evt["type"] == "close":
return
async def barge_in():
async for press in button:
if press.long:
await ws.send(json.dumps({"type": "bye", "reason": "user_ended"}))
return
if speaker.active:
speaker.stop(); speaker.flush() # locally, first
muted.clear()
await ws.send(json.dumps({"type": "interrupt"}))
await asyncio.gather(uplink(), downlink(), barge_in())// A site's access point reboots and thirty devices reconnect in the same
// second. Each reconnect costs a TLS handshake, a device lookup, a context
// fetch and a recognition socket. Thirty at once can wedge a process.
let attempt = 0;
function nextDelay() {
const base = Math.min(2 ** attempt, 60) * 1000; // 1,2,4,8,16,32,60 s cap
attempt += 1;
return base * (0.75 + Math.random() * 0.5); // ±25 % jitter
}
function onConnected() {
// Reset only after the connection has survived a while — a socket that
// opens and dies immediately is still a failing connection.
setTimeout(() => { attempt = 0; }, 60_000);
}Something wrong or missing on this page? Tell us.

