Realtime Voice — Yophone → Hoory

Status: design. Not implemented. This is the contract both sides build against; nothing described here answers on production yet. The websocket host is TBD — confirmed before Yophone builds against it.

A user taps the mic in Yophone and talks to the AI assistant. Speech in, speech out, interruptible. Captions appear live while they talk. When they hang up, the conversation is in the chat as text.

Yophone owns the mic, the speaker and the voice screen. Hoory owns the session, the assistant, the tools and the transcript.

1. Open a session

Server-to-server. The bot token never reaches the app.

POST /webhooks/yophone/{bot_token}/voice/session

{
  "chatId": "0193fcf9-0152-7821-93cb-a4c05ebc17e2",
  "sender": { "id": "0193fcf9-…", "firstName": "Anna", "lastName": "Petrosyan" },
  "conversation_id": 987
}

Field	Notes
`chatId` / `sender`	As on every other call.
`conversation_id`	Required. The conversation the session belongs to, exactly as you already send it on every message.

The bot token is a path segment, same as on the message webhook. It is never in the body.

200 — go ahead:

{
  "session_id": "vs_01JB8F2K9T",
  "conversation_id": 987,
  "ws_url": "wss://<tbd-host>/svc/voice/v1/session",
  "token": "eyJhbGciOi…",
  "expires_at": "2026-09-17T11:14:58Z",
  "max_duration_s": 600
}

token is single-use and expires in 60 seconds. Mint it when the user taps the mic, not when the screen loads. Hand it to the app; it is scoped to this one session and carries no account access.

Refusals — all final, none retryable. Every one carries a body, so the app never has to read meaning into a status code alone:

{ "error": "human_assigned", "message": "An agent owns this conversation" }

Code	`error`	What the app should show
`402`	`credits_exhausted`	Voice unavailable, keep typing.
`409`	`human_assigned`	Voice unavailable, an agent is replying.
`409`	`session_active`	A session is already open — close it first.
`404`	`unknown_bot_token` / `inbox_disabled` / `conversation_not_found`	Hide the mic button.
`429`	`too_many_sessions`	Try again in a moment.

2. Connect

wss://<tbd-host>/svc/voice/v1/session

Authenticate with Authorization: Bearer <token>. A ?token= query parameter is accepted for clients that cannot set headers on the handshake, but prefer the header — query strings end up in access logs.

The first frame Hoory sends is session.ready. Do not send audio before it.

Audio, both directions: PCM16, 24 kHz, mono, little-endian, sent as binary frames. 20–40 ms per frame (960–1920 bytes). No container, no header. Everything else is text frames carrying one JSON object.

Send audio continuously while the mic is live. Do not do client-side silence detection — Hoory decides when a turn ends. Muting means: stop sending binary frames. Nothing else.

3. Frames from the app

`type`	Payload	When
`client.text`	`{ "turn_id": "c1", "text": "…" }`	The user typed into the field instead of speaking. Treated as a full turn. `turn_id` is yours; it comes back on the matching `transcript.*` and `turn.persisted` frames.
`client.interrupt`	—	The user tapped to stop the assistant talking. Also happens automatically when they speak over it.
`client.close`	`{ "reason": "user_hangup" }`	The ✕ button. Then close the socket.

4. Frames from Hoory

`type`	Payload	Meaning
`session.ready`	`{ "session_id", "conversation_id" }`	Start sending audio.
`speech.started`	—	The user was detected speaking. Stop playback, drop queued audio.
`transcript.user.delta`	`{ "turn_id", "text" }`	Caption for what the user is saying. Append.
`transcript.user.done`	`{ "turn_id", "text" }`	Final text for that turn. Replace the appended text with it.
`transcript.assistant.delta`	`{ "turn_id", "text" }`	Caption for the reply, arrives with the audio.
`transcript.assistant.done`	`{ "turn_id", "text" }`	Final text for the reply.
`turn.persisted`	`{ "turn_id", "message_id", "role" }`	That turn is now a message in the conversation.
`handoff`	`{ "reason": "agent" }`	The assistant is handing over to a human. The session closes right after.
`warning.limit`	`{ "seconds_left": 60 }`	The duration cap is near.
`error`	`{ "code", "message" }`	See below.
`session.closed`	`{ "reason" }`	Final frame. `user_hangup`, `idle_timeout`, `max_duration`, `handoff`, `credits_exhausted`, `internal_error`.

Assistant audio arrives as binary frames interleaved with these. Play it in order. On speech.started, discard anything not yet played — the user interrupted, and the assistant’s own turn is cut short on our side too.

5. How the chat fills up

Two different mechanisms, because Hoory cannot write into the Yophone chat as the user:

The assistant’s turns are sent to the chat the same way every other bot reply is — a normal outbound sendMessage, one per completed turn, arriving moments after transcript.assistant.done. Nothing new to implement.
The user’s turns must be inserted by the app, from transcript.user.done. Hoory has them, but only as the contact’s messages inside the conversation; there is no API that makes them appear in a YoAI chat as sent by that user.

So the app renders both sides live from the transcript.* frames, and then has to dedupe the assistant’s bubbles: the one it drew from captions and the one that arrives over sendMessage are the same turn. turn.persisted carries the message_id for exactly this.

turn.persisted is the durable receipt. Captions are for the live screen; the messages are what the conversation, search, reports and the agent dashboard see.

6. Messages sent while a session is live

The text field on the voice screen must go over the socket as client.text, not through the message webhook — that is what keeps it in the same conversation the assistant is currently hearing.

If a message arrives on the normal webhook anyway (another device, a race with hang-up), it is always accepted with 200 — the response body says what happened:

{ "conversation_id": 987, "queued_until_voice_ends": true }

It is stored immediately and answered once the session closes. It is never dropped and never a 4xx. Voice notes sent as attachments behave the same way.

Rules

One live session per conversation. A second POST while one is open returns 409 session_active. Close the socket before opening another.
Voice only while the bot owns the conversation. The moment a human agent replies, the session closes with handoff and the mic button should go away until the conversation is resolved.
Hard caps. max_duration_s (600 by default) and a 30-second idle timeout. Both are announced — warning.limit, then session.closed.
No reconnect. A dropped socket ends the session. Ask for a new token and start a new one; the conversation and every completed turn are unchanged, so nothing is lost but the turn in flight.
Voice and text share one memory. Messages typed in the chat before the session are context for it, and everything said during it is context for what is typed after.
Language. The assistant answers in the language it is spoken to. Armenian quality is being measured before launch; treat it as unconfirmed.

If something is wrong

error frames are informational unless followed by session.closed. The app should show a short “voice unavailable” state and fall back to typing — never a retry loop. Codes: audio_format, rate_limited, credits_exhausted, upstream_unavailable, internal_error.

A session that fails mid-way still leaves every completed turn in the chat. Nothing is rolled back.