Show HN: RealtimeVAD – trading false positives for 110ms earlier end-of-speech

RealtimeVAD Know that the speaker stopped, about 100 ms before Silero does. RealtimeSTT · RealtimeTTS · Live demo · Spec · LiveKit plugin A voice assistant can't do anything until it knows the user has stopped talking. Speech recognition, the LLM call and the voice output all wait for that one decision, and a normal voice activity detector takes a few hundred milliseconds to make it. RealtimeVAD makes it sooner. It keeps Silero VAD to decide what is speech, and adds a second, faster check: every 10 ms it compares the audio level with the room's noise and with the speaker's own level. The moment the audio goes quiet, you get an event, long before Silero's probability has fallen. Input: 16 kHz mono audio in chunks of any size (float32, int16 or int16 bytes). Output: start , pause and end events, plus the audio of the running segment at any time. Lead: speech-off a median 100 ms before Silero's probability drops, and 206 ms before Silero's VADIterator reports the end, on 3,279 real turns. The end itself comes before Silero's in 82 % of turns, by a median 70 ms. Price: it also fires in short gaps between words, about ten times as often as Silero. Quiet input: a quiet speaker's soft word endings are not silence; on a noisy quiet line the lead is gone instead ( details ). Runs on: one CPU thread, far faster than realtime. Needs only numpy and onnxruntime; the Silero model is bundled, no torch. Status: alpha (0.0.5). The API may still change. Live demo demo.mp4 The video uses a synthetic voice and shows a median lead of 92 ms. The 100 ms above is the median on real voice-assistant turns. demo/ runs this page on your own microphone. It shows, turn by turn, how much sooner RealtimeVAD knew that you stopped than Silero alone, on the same audio. Quick start pip install realtimevad from RealtimeVAD import VAD vad = VAD () for chunk in microphone_chunks (): for event in vad . feed ( chunk ): if event . kind == "start" : stream_to_asr ( vad . audio ()) # includes the audio just before speech began elif event . kind == "pause" : speculate ( vad . audio ()) # the speaker may be done; this isn't the end yet elif event . kind == "end" : finalize ( event . segment . audio ) Events Event Meaning When start Speech has begun After about 64 ms of confident speech, so clicks don't trigger it. Speech Silero is never confident about (a short "mm-hmm", a quiet phone voice) starts after about 0.13 s at half confidence, if it is clearly louder than the room pause The speaker may have stopped (speech-off) Within about 10 ms of the audio going quiet end The segment is over 0.24 s after speech-off, unless speech resumes. Never later than Silero alone onset Speech may have begun At the first confident frame, before start confirms it. Off by default: VAD(onset_events=True) vad.audio() returns the running segment at any time, starting about 0.3 s before the speech. end carries the finished segment with its audio, start time, end time and the time the speech stopped. If your assistant talks while the microphone is open, pass vad.feed(chunk, playback=True) for those chunks so its own voice is handled as possible echo. Every threshold and duration can be configured; SPEC.md lists what is guaranteed. How to use pause pause is early and often wrong, on purpose. It fires on every breath, every short gap between words and every stop consonant, so most pause events are followed by more speech. Use it to start work you can cancel: finalize the transcript, call the LLM, prepare the voice output. Cancel when speech continues. A wrong pause then costs almost nothing, and a right one saves hundreds of milliseconds. Never treat it as the end of the turn. Only end closes a segment. Can't cancel quickly? Ignore pause and wait for end . The extra false alarms would cost you more than they save. Already using Silero? Change the import: from silero_vad import load_silero_vad from RealtimeVAD import VADIterator # was: from silero_vad import VADIterator vad_iterator = VADIterator ( load_silero_vad (), sampling_rate = 16000 ) speech_dict = vad_iterator ( chunk , return_seconds = True ) # {'start': ...} / {'end': ...} / None The constructor and the results stay the same, but end arrives as soon as the audio goes quiet. Without a model argument, it uses the bundled Silero model and doesn't need torch. Using LiveKit Agents? livekit-plugins-realtimevad/ is a VAD plugin: realtimevad.VAD.load() in place of silero.VAD.load() . Its README shows how the events map and what it measured against the Silero plugin on the same audio. In short: against the Silero plugin's defaults the end of speech comes about 380 ms sooner, but most of that is the shorter silence setting (0.25 s instead of 0.55 s), which Silero can be given too. At the same setting RealtimeVAD's end is 65 to 110 ms earlier, and LiveKit's turn detector starts 95 to 180 ms earlier. Inside a LiveKit session the turn was committed about 150 ms sooner than with Silero's defaults and about 70 ms sooner than with Silero at 0.25 s. Results Measured on 3,279 turns from a live voice assistant, each with its trailing silence, with default settings. Silero's decisions are computed from the very same frame probabilities, so both see identical input. "Speech-off" is RealtimeVAD's last pause before the turn ended, against the first frame of Silero's final run below its threshold. RealtimeVAD speech-off earlier than … median p25 – p75 p90 earlier in Silero, probability below 0.5 (its default threshold) 100 ms 80 – 150 ms 214 ms 98 % Silero, probability below 0.35 ( VADIterator 's end threshold) 110 ms 90 – 176 ms 230 ms 98 % Silero VADIterator end, default 100 ms minimum silence 206 ms 186 – 272 ms 336 ms 99 % Speech-off signals followed by more speech per second of speech turns without any RealtimeVAD pause 1.5 17 % Silero, probability below 0.5 0.14 86 % Until 0.0.4 the first row read 110 ms. The 10 ms went into not cutting quiet speakers, see the next section. The end event, against the end Silero's own silence count gives with the same thresholds and the same 0.24 s (2,323 turns where both closed the same segments inside the recording): RealtimeVAD end earlier than Silero alone earlier in median p25 – p75 later in 0.0.5 82 % 70 ms 40 – 92 ms 0 the same without restarting the count after a click 51 % 18 ms 0 – 76 ms 0 Up to 0.0.4 a single loud 10 ms block inside the quiet after a word (a click, a lip smack, a breath) stopped the quiet count for good and left the end to Silero. That happened in a third of 680 sampled turns. The count now starts again after it. Reproduce with your own recordings (16 kHz mono WAVs, one turn each, with trailing silence): python benchmarks/speech_off_lead.py < wav dir > out.json Quiet input and phone audio Up to 0.0.4, "quiet" had a fixed floor ( quiet_rms , about −54 dBFS). For a speaker around −32 dBFS that is fine. For a quiet one (a phone line recorded at source level, a distant microphone) the floor sits only a few dB under the speech, their soft word endings counted as silence, and 0.24 s later the segment ended while they were still talking. Since 0.0.5 the floor follows the speaker: it is quiet_rms or 20 dB below the speaker's own level, whichever is lower ( floor_margin_db ). The same 3,401 turns, attenuated and sent through a simulated phone line (300–3400 Hz, 8 kHz, μ-law, line noise). "Ended while talking" counts turns with an end that the same detector does not make at the recorded level and that is followed by more speech. "Silero alone" is the same VAD with the quiet clock off, on the same audio. Speech level Ended while talking: 0.0.4 0.0.5 Silero alone Lost ≥ 100 ms of speech: 0.0.4 0.0.5 Silero a