VAD and turn taking

Sleeptalk detects speech, decides when your turn is complete, and handles interruptions as separate decisions.

Detecting speech

Voice activity detection, or VAD, decides which incoming audio contains speech. The shared capture pipeline uses a bundled Silero model together with an audio-level threshold. The default hybrid capture mode accepts either detector's speech evidence, which helps keep quieter words.

Once an utterance begins, the assembler keeps its audio through pauses until the turn is committed. If Silero fails, capture reports the failure and falls back to the audio-level detector, which is less selective about noise.

The word hybrid in capture settings refers to combining these detectors. It is unrelated to the OpenAI voice routes.

Deciding when you are done

A pause is not always the end of a sentence. Endpointing combines elapsed silence with the transcript and any completion signals available. Complete phrases can finish sooner. Trailing fillers and floor-holding words such as "um", "and", "I mean", or "wait" get more time, and so do incomplete phrases and longer monologues. Those word cues match English only.

A bundled Smart Turn v3 model can inspect the audio when the text-based decision is uncertain. It can shorten an ambiguous wait. It cannot override a phrase that explicitly holds the turn, and it does not interrupt ongoing speech on its own. If Smart Turn fails, the runtime reports it and disables that scorer for the session; silence and transcript endpointing continue.

Choosing a turn-taking preset

The profile's policy.endpointing has three presets. The settings page shows them under Behavior, as Turn taking:

[policy]
endpointing = "relaxed"

Use Patient if the agent answers during your pauses. Use Snappy if the wait after a finished sentence is too long. Changes apply to your next call.

Some speech services hear nothing until the turn is over, then upload and decode it. These include the OpenAI-compatible audio services and Moonshine Tiny. With these batch recognizers, Sleeptalk tightens Balanced silence windows automatically. Choosing Patient, Snappy, or explicit overrides turns that adjustment off. Local speech now defaults to streaming Moonshine Small, which supplies revisable partial transcripts while you speak.

Presets change thresholds, not a guaranteed response time. Provider timing and model work also affect when you hear a reply.

Speaking over a reply

Barge-in detects your speech while the agent is talking. It uses sustained speech, a volume threshold, a short grace period, and a cooldown to reduce false interruptions from brief noises or playback echo.

The interruption mode is [policy.interrupt] mode, with off, soft, and immediate. The default is immediate. Soft means pause and let the transcript decide whether you really took the turn. A surface that cannot pause and resume playback downgrades soft to a stop and says so.

Terminal interruption depends on your audio setup. With headphones, the microphone cannot hear the agent, so barge-in uses the same speech detector as browser and Discord calls. With speakers, it uses a volume-based detector so the agent's own voice does not interrupt it. See terminal audio.

Interrupting speech and stopping work

Talking over an answer interrupts its presentation. It does not guarantee cancellation of every tool or background worker behind that answer. Pending findings can remain available for a later reply.

The browser's Stop button also clears pending output. Targeted work cancellation is a separate adapter capability. Check the agent's reported status before assuming a background task stopped, and see tool calls and sub-agents.

Advanced tuning

Advanced profiles expose capture confidence, minimum volume, silence windows, and interruption duration. See settings and profiles for the fields. Lower thresholds can admit more background noise; shorter waits can split one thought into several turns. Change one setting at a time and test a real spoken sentence with pauses.

Implementation: utterance assembly, endpointing, Smart Turn scoring, and presets.