7.9 KiB
CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
Running the app
pip install -r requirements.txt
cp .env.example .env # configure GROQ_API_KEY and ELEVENLABS_API_KEY
python main.py
# → http://localhost:8000
live2dcubismcore.min.js (~202 KB) is downloaded automatically on first run if missing.
Architecture
Backend (main.py) connected to a vanilla JS frontend via one persistent WebSocket (/ws). Each connection holds a SessionState with conversation history, busy/cancel flags, TTS preference, and optional geo coordinates.
Supporting modules:
| File | Role |
|---|---|
websearch.py |
web_search tool — Serper → DuckDuckGo → Wikipedia |
weather.py |
Open-Meteo forecast + parse_weather_request() intent detection |
workspace.py |
Terminal tools when workspace mode is enabled |
Message flow
Browser mic → PCM Int16 binary frame → /ws
→ Groq Whisper STT → { type: "transcript" }
→ (optional) weather prefetch → { type: "weather_widget" }
→ Groq LLM (history + tools) → { type: "chunk" } × N → { type: "done" }
→ ElevenLabs TTS (if tts_enabled) → { type: "tts_audio", data: base64mp3 }
Text input skips STT: browser sends { type: "question", text } directly.
Wake word flow (Safari-compatible — no Web Speech API):
Continuous mic + VAD → { type: "wake_check", pcm: base64 }
→ Whisper on server → { type: "wake_ok", text? } | { type: "no_wake" }
→ if text: auto-send question; else: start follow-up recording with post-buffer
WebSocket message types
| Direction | Type | Meaning |
|---|---|---|
| client→server | audio_config |
Set lang, sample_rate, tts_enabled |
| client→server | geo_config |
Browser lat/lon for local weather |
| client→server | wake_check |
Base64 PCM to detect wake word «Arisu» |
| client→server | workspace_config |
Enable terminal + base directory path |
| client→server | question |
Text prompt |
| client→server | cancel |
Cancel in-flight LLM/TTS |
| client→server | clear_history |
Reset session conversation history |
| client→server | binary | Raw PCM-16LE at 16 kHz (manual recording) |
| server→client | transcript |
Whisper result |
| server→client | wake_ok |
Wake word detected; optional text = remainder question |
| server→client | no_wake |
Wake word not found |
| server→client | thinking |
LLM started |
| server→client | chunk |
LLM streaming delta (may contain [tag]) |
| server→client | done |
LLM finished |
| server→client | cancelled |
Response was cancelled |
| server→client | busy |
Server already processing a request |
| server→client | tts_audio |
Base64 MP3 |
| server→client | tts_error |
ElevenLabs failed |
| server→client | no_speech |
Audio below threshold or too short |
| server→client | tool_event |
Tool invocation started |
| server→client | tool_result |
Tool result preview |
| server→client | weather_widget |
Structured weather data for chat card |
| server→client | workspace_ok |
Workspace validated |
| server→client | workspace_error |
Invalid workspace path |
| server→client | history_cleared |
History reset confirmed |
| server→client | error |
Generic error |
Weather
weather.py uses Open-Meteo (no API key). Do not expose weather via LLM tool calling — llama-3.1-8b-instant often generates malformed tool calls (tool_use_failed).
Instead, _prefetch_weather() in main.py:
- Runs
parse_weather_request(user_text)— regex intent + city extraction (aliases like Tokio→Tokyo). - Calls
get_weather(location, lang, session.geo_lat, session.geo_lon). - Sends
{ type: "weather_widget", data: {...} }to the browser. - Injects weather text into the LLM messages as a system note (
DATOS DE CLIMA).
Frontend renders the widget in appendWeatherWidget() with Bootstrap Icons mapped by WMO code icon key.
Geo: browser sends { type: "geo_config", lat, lon } on WebSocket open if geolocation is allowed.
Web search
websearch.py provides web_search(query, lang) via LLM tool calling. Fallback chain: Serper (if SERPER_API_KEY) → DuckDuckGo → Wikipedia.
current_date_context(lang) and enrich_search_query() inject today's date from APP_TIMEZONE (default America/Mexico_City) so «hoy» queries use the real year.
Terminal / workspace mode
When terminal_enabled and a valid workspace_path are set via workspace_config, terminal tools are added alongside the always-available web_search tool.
LLM tool calling
- Tools per session:
[WEB_SEARCH_TOOL]+ terminal tools if enabled. _llm_completion()retries without tools on Groqtool_use_failed(400) instead of crashing.
Session memory
SessionState.historystores up toMAX_HISTORY(20) user/assistant messages per WebSocket.- History is trimmed FIFO when exceeded.
- Cleared via
clear_historymessage or frontend "Clear chat" button.
Wake word (browser)
Key constants in app.js:
| Constant | Value | Purpose |
|---|---|---|
WAKE_SILENCE_MS |
1700 | End capture after pause (allows gap between «Arisu» and question) |
WAKE_MAX_MS |
9000 | Max utterance length for wake capture |
WAKE_PREROLL_MS |
350 | Pre-buffer before speech detection |
WAKE_POST_BUFFER_MS |
6000 | Buffer audio during server wake_check STT |
While wakeChecking, audio is stored in wakePostBuffer and prepended to follow-up recording on wake_ok without text.
Audio pipeline (browser)
getUserMedia → AudioContext → ScriptProcessor(4096) → Float32 chunks → downsample to 16 kHz → Int16Array → binary WebSocket frame. The server always wraps received bytes as 16 kHz WAV regardless of the audio_config sample_rate field.
Audio tags
The LLM is prompted to include ElevenLabs v3 emotion tags ([laughs], [giggles], [excited], [whispers], [sighs], [gasps]) inline. app.js:detectExpressions() parses tags from streaming chunks before display and calls avatar.setExpression(tag). Tags are stripped from chat text via stripAudioTags(). The full tagged text is sent to ElevenLabs TTS unchanged.
Live2D (live2d.js)
Live2DAvatar wraps pixi-live2d-display. Key methods: setLook(x,y), startSpeaking(analyser?), stopSpeaking(), setExpression(tag). Lip-sync uses AnalyserNode RMS from TTS playback when available; falls back to sinusoidal mouth animation. Model is anchored at (0.5, 0) with Y offset to show face+torso.
TTS playback
Uses AudioContext.decodeAudioData() + BufferSourceNode + AnalyserNode (not new Audio()) to avoid browser autoplay blocking and drive lip-sync. Reuses state.audioCtx from the mic pipeline.
i18n
Frontend UI strings are bilingual ES/EN via I18N object in app.js. UI language is independent of STT language (langSelect). Preference stored in localStorage key arisu_ui_lang.
Key constraints
- API keys must be in
.env— never hardcode inmain.py. - The server always uses
16000Hz for WAV wrapping — the client always downsamples to 16 kHz before sending. - RMS check (
< 0.015) instopRecording()discards silent/noise-only recordings before sending to Whisper. GROQ_STT_MODEL = "whisper-large-v3-turbo"— do not switch to OpenAI Whisper; it hallucinates more on this audio pipeline.- TTS voice ID and ElevenLabs model are in
.env/main.pyconstants (ELEVEN_VOICE_ID,ELEVEN_MODEL = "eleven_v3"). - WebSocket URL uses
wss://when page is served over HTTPS. - Only one in-flight LLM request per WebSocket session; concurrent requests receive
{ type: "busy" }. - Weather must use server-side prefetch, not LLM
get_weathertool — small Groq models fail tool formatting.
Plan document
See PLAN.md for audit findings and implementation status.