An open, real-time voice agent pipeline with a browser dashboard.
Now with Hermes Agent integration (proxy, tab, LLM read-timeout knob) and a Pascal GPU restart-loop fix.
Every stage is swappable. Each one runs in its own thread, connected by queues, and the whole thing speaks the OpenAI Realtime GA protocol on ws://<host>:8765/v1/realtime.
A FastAPI + vanilla-JS web app that introspects every CLI flag the pipeline knows about and renders a form for each one. No build step. No per-flag hardcoding.
Mode, VAD, STT, LLM, TTS, Advanced, Status & Logs, Guide, Settings, Control — every backend has its own sub-form that swaps in when you pick it.
Live /v1/pool polling on the Status tab — see which pipeline units are idle, busy, and which session is on each.
Filter by level, toggle verbose (--log-level debug), watch the pipeline come up in real time.
Cyberpunk Neon (default), Matrix, Synthwave, Blade Runner, Tron, Vaporwave, Hacker Terminal, Dark Glass — drop a new .css in web_ui/themes/ and it appears in the picker.
Settings live at web_ui_settings.json in the repo root (gitignored). Atomic write — a crash mid-save can't corrupt the file. A working example is checked in as web_ui_settings.example.json — import it from the Settings tab to start with a real config instead of bare defaults.
One click to start, one click to kill. Shutdown closes both the pipeline and the dashboard. Cross-platform: CREATE_NEW_PROCESS_GROUP on Windows, SIGTERM+SIGKILL on Linux.
New tab for Hermes Agent — start/stop the Hermes subprocess from the dashboard, view live status, and chat with the agent through a reverse proxy at /hermes-proxy/v1/*. Switching --llm-backend-type to hermes wires the pipeline straight into Hermes's OpenAI-compatible API on port 8642.
The pipeline's hardcoded 20 s OpenAI read timeout made Hermes look "stuck in a loop" on long passages. The dashboard now ships a hermes.read_timeout_s setting (0 = wait forever). Implemented as a zero-touch monkey-patch loaded into the venv via sitecustomize.py on pipeline startup — no edits to src/.
Hermes Agent (Nous Research) is now a first-class LLM backend option. The dashboard owns the Hermes subprocess via web_ui/hermes_manager.py (start/stop/restart, log capture, exit codes) and exposes Hermes through a reverse proxy at /hermes-proxy/v1/* so the browser doesn't have to whitelist http://127.0.0.1:8642. The API key is read from Hermes's own config and auto-injected as a Bearer token by the proxy.
A dedicated Hermes tab in the dashboard shows live status, the currently loaded model (polled every 5 s from /v1/models), and an embedded chat console for ad-hoc testing. In web_ui_settings.json, switching --llm-backend-type to hermes rewires the pipeline to talk to Hermes and swaps the editable --model-name field for a read-only label — because the model is chosen via hermes model in the terminal, not the dashboard.
Hermes repeatedly looked "stuck in a loop" because the pipeline hardcodes a 20 s read timeout (src/.../base_openai_compatible_language_model.py:140). When the LLM takes longer than 20 s on a long passage, the OpenAI client times out and the pipeline emits the canned "Wow I'm a bit slow today, could you repeat that?" fallback — which then gets spoken by the TTS. Every subsequent slow turn produces the same canned line.
New setting hermes.read_timeout_s on the Hermes tab (0 = wait forever, >0 = read timeout in seconds). The dashboard drops a tiny sitecustomize.py into the venv's site-packages on pipeline start that auto-imports a monkey-patch wrapping the OpenAI SDK's Completions.create and Responses.create — replacing the hardcoded 20 s with the configured value. Cleanup runs on pipeline stop. Zero edits to src/speech_to_speech/.
On Pascal (sm_61) GPUs, clicking Start Pipeline on the Control tab triggered an infinite restart loop: torch's CUDA context blew up on init, the watchdog saw a non-zero exit, relaunched, watched it die again, and the only way out was killing the dashboard.
Four-file fix: web_ui/gpu.py now detects Pascal and pins torch to the bundled CPU-friendly wheel (torch==2.4.1+cpu) on install, plus a runtime CPU fallback for any other CUDA init failure. process_manager.py tracks the restart count and surfaces it on the Control tab so you can see the loop in the dashboard. qwentts_installer.py defers its heavy CUDA work until the first TTS request instead of at pipeline start (so a CUDA-init failure no longer kills startup before the pipeline ever gets a chance to run). Startup logs now print a one-line "GPU init failed, falling back to CPU (…)" banner so the cause is obvious.
web_ui/llm_keepalive.py — a tiny thread-safe helper that pings the configured Ollama endpoint every keepalive_interval_s (default 60 s) so the model stays loaded in VRAM between turns. Without it, Ollama unloads the model after 5 minutes of idle, and the next user turn pays a multi-second cold-load penalty.
The dashboard's existing Settings form renders the interval field automatically (auto-introspection, no JS special cases). Disable it by setting the interval to 0. Hermes backends are skipped automatically — Hermes manages its own model lifecycle.
When --tts qwen3 is selected, the TTS tab now shows a card grid of the 9 CustomVoice preset speakers — Vivian, Serena, Uncle_Fu, Dylan, Eric (Chinese), Ryan, Aiden (English), Ono_Anna (Japanese), Sohee (Korean). Each card has a language badge, a sample sentence in that speaker's native language, a Set active button, and a Test button that synthesizes the sample so you can hear the voice before committing.
Realtime clients (browser, app, robot) send voice: "alloy" in session.update; qwen3-TTS doesn't recognize OpenAI's voice names, so the dashboard silently ignores the client's voice field and uses whatever you set in Settings → TTS → Qwen3-TTS → Speaker. Pick from the dropdown, save, the Realtime pipe speaks in that voice.
--qwen3-tts-language is now a dropdown with auto plus 10 languages (English, Chinese, Japanese, Korean, German, French, Russian, Portuguese, Spanish, Italian). The Test modal also has a Language selector pre-filled with the speaker's native language. Pinning the language per-request fixes the cross-lingual artefact where, e.g., sending French text to Ono_Anna would emit French-across-Japanese-voice.
CustomVoice 1.7B/0.6B (the 9-voice models), Base 1.7B/0.6B (voice cloning via ref_audio + ref_text), and VoiceDesign 1.7B (voice style via instruct text) are all selectable from the dashboard's --qwen3-tts-model-name dropdown.
Base and VoiceDesign paths need ABI v2 symbols (qt_extract_voice_ref, qt_voice_ref_free) that aren't in the public andimarafioti/qwentts.cpp source. The bundled Pascal qwentts-cpp-python wheel is ABI v1, so picking one of those models renders a yellow info banner explaining the limitation and why the Test button will fail at synthesis time. On RTX 30/40 series GPUs where the upstream wheel ships ABI v2, voice cloning and voice design light up automatically — no dashboard work needed.
Without the venv shim, every OpenAI Realtime session.update raised QwenTTSError: Speaker enumeration requires qwentts.cpp ABI v2 because the upstream get_supported_speakers() calls into the missing ABI v2 runtime.speaker_names() symbol. web_ui/qwentts_installer.py now patches the venv binding (idempotent, marker comment for traceability) to return [], which makes the upstream handler's existing "ignore client voice, use configured" branch fire. The configured --qwen3-tts-speaker wins. Shim is reapplied automatically on every uv sync via the installer.
The Pascal (sm_61) build of qwentts-cpp-python (126 MB) is checked into web_ui/wheels/ so the dashboard can install it without a network round-trip. The installer runs once before pipeline start; on non-Pascal GPUs it falls through to the upstream PyPI wheel as before.
Resemble AI's Chatterbox, the sixth TTS backend. Upload a short reference audio (5–30s of clean speech), the dashboard extracts the speaker embedding, and the robot speaks in that voice for every reply. Three variants:
| Variant | Size | Notes |
|---|---|---|
chatterbox (English 500M) | ~1 GB | Best quality. exaggeration, cfg_weight, repetition_penalty, min_p all apply. |
chatterbox-turbo (350M) | ~700 MB | Single-step decoder, low latency. exaggeration and cfg_weight are no-ops (grayed in the form). |
chatterbox-nano (110M) | ~250 MB | CPU-friendly. Same restrictions as Turbo. |
The dashboard's Voice Library shows all your cloned voices as cards. Set active, test with the current form settings, or delete. Variant-conditional params are visually disabled, not removed.
Chatterbox Turbo holds ~700 MB of RAM; the full English variant is ~1 GB. If you're RAM-constrained, click Unload TTS Model on the Control tab — the model is dropped from RAM without killing the pipeline. The next TTS request reloads it transparently (~15–20s on CPU).
Implemented as a UNLOAD_TTS control message that walks the entire handler chain (VAD → STT → … → TTS). Each non-TTS handler's on_unload() is a no-op; only the TTS handler actually drops the model.
Pick Chatterbox in the TTS tab and click Start — if the package isn't installed yet, the dashboard shows the exact uv pip install command for your platform and streams the install log into the Status tab. No terminal needed.
# Install (Chatterbox requires the optional extra)
git clone https://github.com/huggingface/speech-to-speech.git
cd speech-to-speech
uv sync --extra chatterbox
# Linux / macOS
./start_web_ui.sh
# Windows
start_web_ui.batThe browser opens to http://localhost:8050. Set SPEECH_TO_SPEECH_WEB_PORT to change the port. See the in-app Guide for Ollama recipes, OpenAI provider setup, and troubleshooting.
| Path | What |
|---|---|
web_ui/ | The browser dashboard (FastAPI + vanilla JS, no build step) |
web_ui/static/ | HTML, CSS, theme files, voice-library UI modules (voice_library_ui.js, qwen3_voice_library_ui.js) |
web_ui/static/qwen3_models.json | All 5 qwen3-TTS model IDs with variant metadata + ABI v2 flags |
web_ui/static/field_choices.json | Curated dropdown values for plain-str fields (devices, dtypes, voice names, languages) |
web_ui/Guide.md | The in-app walkthrough (Quickstart, Ollama, qwen3-TTS troubleshooting, common issues) |
web_ui/voice_library.py | Chatterbox on-disk voices/<name>.pt library: list / clone / delete / test |
web_ui/qwentts_voice_library.py | qwen3-TTS preset library + reference voices + WAV test synthesis |
web_ui/qwentts_installer.py | Auto-installs the bundled Pascal wheel on sm_61 GPUs and patches the venv binding for the ABI v1 realtime crash |
web_ui/wheels/ | Bundled CUDA wheels (currently: qwentts_cpp_python-0.3.1 for Pascal sm_61) |
web_ui/hermes_manager.py | Hermes Agent subprocess lifecycle (start / stop / restart, log capture, exit codes) |
web_ui/hermes_proxy.py | Reverse proxy at /hermes-proxy/v1/* so the browser doesn't need to allowlist http://127.0.0.1:8642; injects the Hermes API key as a Bearer token |
web_ui/llm_keepalive.py | Background thread that pings the configured Ollama endpoint so the model stays loaded in VRAM between turns |
web_ui/gpu.py | Pascal detection + CPU torch fallback to keep the pipeline startable on sm_61 GPUs |
web_ui/_openai_timeout_patch.py | Zero-touch OpenAI-SDK monkey-patch (Hermes-only) injected via sitecustomize.py on pipeline start, removed on stop |
demo/ | The Reachy Mini conversation UI (orb, settings, tools, WebRTC) |
src/speech_to_speech/TTS/ | All six TTS backends, including chatterbox_tts_handler.py and qwen3_tts_handler.py |
src/speech_to_speech/api/openai_realtime/ | The Realtime WebSocket / WebRTC server |
voices/ | Cloned Chatterbox + uploaded qwen3 reference audio (gitignored, created on first upload) |
web_ui_settings.example.json | Working settings example (Ollama + qwen3-TTS + parakeet STT). Import from Settings tab. |
web_ui_settings.json | Saved dashboard settings (gitignored, atomic write, created on first Save) |
This fork powers the conversation backend for the Reachy Mini robot. The conversation app in demo/ talks to the same ws://<host>:8765/v1/realtime endpoint, so the moment your dashboard's pipeline is up, the robot can use your cloned Chatterbox voice.
# In one terminal: the dashboard + pipeline
./start_web_ui.sh
# Click "Start Pipeline" on the Control tab.
# In another terminal: the conversation app
uv run uvicorn --app-dir demo server:app --port 7860
# Open http://localhost:7860/ and click the orb.