# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. ## Running the app ```bash pip install -r requirements.txt cp .env.example .env # configure GROQ_API_KEY and ELEVENLABS_API_KEY python main.py # → http://localhost:8000 ``` `live2dcubismcore.min.js` (~202 KB) is downloaded automatically on first run if missing. ## Architecture **Backend** (`main.py`) connected to a vanilla JS frontend via one persistent WebSocket (`/ws`). Each connection holds a `SessionState` with conversation history, busy/cancel flags, TTS preference, and optional geo coordinates. Supporting modules: | File | Role | |------|------| | `websearch.py` | `web_search` tool — Serper → DuckDuckGo → Wikipedia | | `weather.py` | Open-Meteo forecast + `parse_weather_request()` intent detection | | `workspace.py` | Terminal tools when workspace mode is enabled | ### Message flow ``` Browser mic → PCM Int16 binary frame → /ws → Groq Whisper STT → { type: "transcript" } → (optional) weather prefetch → { type: "weather_widget" } → Groq LLM (history + tools) → { type: "chunk" } × N → { type: "done" } → ElevenLabs TTS (if tts_enabled) → { type: "tts_audio", data: base64mp3 } ``` Text input skips STT: browser sends `{ type: "question", text }` directly. Wake word flow (Safari-compatible — no Web Speech API): ``` Continuous mic + VAD → { type: "wake_check", pcm: base64 } → Whisper on server → { type: "wake_ok", text? } | { type: "no_wake" } → if text: auto-send question; else: start follow-up recording with post-buffer ``` ### WebSocket message types | Direction | Type | Meaning | |-----------|------|---------| | client→server | `audio_config` | Set `lang`, `sample_rate`, `tts_enabled` | | client→server | `geo_config` | Browser lat/lon for local weather | | client→server | `wake_check` | Base64 PCM to detect wake word «Arisu» | | client→server | `workspace_config` | Enable terminal + base directory path | | client→server | `question` | Text prompt | | client→server | `cancel` | Cancel in-flight LLM/TTS | | client→server | `clear_history` | Reset session conversation history | | client→server | binary | Raw PCM-16LE at 16 kHz (manual recording) | | server→client | `transcript` | Whisper result | | server→client | `wake_ok` | Wake word detected; optional `text` = remainder question | | server→client | `no_wake` | Wake word not found | | server→client | `thinking` | LLM started | | server→client | `chunk` | LLM streaming delta (may contain `[tag]`) | | server→client | `done` | LLM finished | | server→client | `cancelled` | Response was cancelled | | server→client | `busy` | Server already processing a request | | server→client | `tts_audio` | Base64 MP3 | | server→client | `tts_error` | ElevenLabs failed | | server→client | `no_speech` | Audio below threshold or too short | | server→client | `tool_event` | Tool invocation started | | server→client | `tool_result` | Tool result preview | | server→client | `weather_widget` | Structured weather data for chat card | | server→client | `workspace_ok` | Workspace validated | | server→client | `workspace_error` | Invalid workspace path | | server→client | `history_cleared` | History reset confirmed | | server→client | `error` | Generic error | ### Weather [`weather.py`](weather.py) uses Open-Meteo (no API key). **Do not expose weather via LLM tool calling** — `llama-3.1-8b-instant` often generates malformed tool calls (`tool_use_failed`). Instead, `_prefetch_weather()` in `main.py`: 1. Runs `parse_weather_request(user_text)` — regex intent + city extraction (aliases like Tokio→Tokyo). 2. Calls `get_weather(location, lang, session.geo_lat, session.geo_lon)`. 3. Sends `{ type: "weather_widget", data: {...} }` to the browser. 4. Injects weather text into the LLM messages as a system note (`DATOS DE CLIMA`). Frontend renders the widget in `appendWeatherWidget()` with Bootstrap Icons mapped by WMO code icon key. Geo: browser sends `{ type: "geo_config", lat, lon }` on WebSocket open if geolocation is allowed. ### Web search [`websearch.py`](websearch.py) provides `web_search(query, lang)` via LLM tool calling. Fallback chain: Serper (if `SERPER_API_KEY`) → DuckDuckGo → Wikipedia. `current_date_context(lang)` and `enrich_search_query()` inject today's date from `APP_TIMEZONE` (default `America/Mexico_City`) so «hoy» queries use the real year. ### Terminal / workspace mode When `terminal_enabled` and a valid `workspace_path` are set via `workspace_config`, terminal tools are added alongside the always-available `web_search` tool. ### LLM tool calling - Tools per session: `[WEB_SEARCH_TOOL]` + terminal tools if enabled. - `_llm_completion()` retries **without tools** on Groq `tool_use_failed` (400) instead of crashing. ### Session memory - `SessionState.history` stores up to `MAX_HISTORY` (20) user/assistant messages per WebSocket. - History is trimmed FIFO when exceeded. - Cleared via `clear_history` message or frontend "Clear chat" button. ### Wake word (browser) Key constants in `app.js`: | Constant | Value | Purpose | |----------|-------|---------| | `WAKE_SILENCE_MS` | 1700 | End capture after pause (allows gap between «Arisu» and question) | | `WAKE_MAX_MS` | 9000 | Max utterance length for wake capture | | `WAKE_PREROLL_MS` | 350 | Pre-buffer before speech detection | | `WAKE_POST_BUFFER_MS` | 6000 | Buffer audio during server wake_check STT | While `wakeChecking`, audio is stored in `wakePostBuffer` and prepended to follow-up recording on `wake_ok` without `text`. ### Audio pipeline (browser) `getUserMedia` → `AudioContext` → `ScriptProcessor(4096)` → Float32 chunks → downsample to 16 kHz → `Int16Array` → binary WebSocket frame. The server always wraps received bytes as 16 kHz WAV regardless of the `audio_config` sample_rate field. ### Audio tags The LLM is prompted to include ElevenLabs v3 emotion tags (`[laughs]`, `[giggles]`, `[excited]`, `[whispers]`, `[sighs]`, `[gasps]`) inline. `app.js:detectExpressions()` parses tags from streaming chunks before display and calls `avatar.setExpression(tag)`. Tags are stripped from chat text via `stripAudioTags()`. The full tagged text is sent to ElevenLabs TTS unchanged. ### Live2D (`live2d.js`) `Live2DAvatar` wraps pixi-live2d-display. Key methods: `setLook(x,y)`, `startSpeaking(analyser?)`, `stopSpeaking()`, `setExpression(tag)`. Lip-sync uses `AnalyserNode` RMS from TTS playback when available; falls back to sinusoidal mouth animation. Model is anchored at `(0.5, 0)` with Y offset to show face+torso. ### TTS playback Uses `AudioContext.decodeAudioData()` + `BufferSourceNode` + `AnalyserNode` (not `new Audio()`) to avoid browser autoplay blocking and drive lip-sync. Reuses `state.audioCtx` from the mic pipeline. ### i18n Frontend UI strings are bilingual ES/EN via `I18N` object in `app.js`. UI language is independent of STT language (`langSelect`). Preference stored in `localStorage` key `arisu_ui_lang`. ## Key constraints - API keys must be in `.env` — never hardcode in `main.py`. - The server **always** uses `16000` Hz for WAV wrapping — the client always downsamples to 16 kHz before sending. - RMS check (`< 0.015`) in `stopRecording()` discards silent/noise-only recordings before sending to Whisper. - `GROQ_STT_MODEL = "whisper-large-v3-turbo"` — do not switch to OpenAI Whisper; it hallucinates more on this audio pipeline. - TTS voice ID and ElevenLabs model are in `.env` / `main.py` constants (`ELEVEN_VOICE_ID`, `ELEVEN_MODEL = "eleven_v3"`). - WebSocket URL uses `wss://` when page is served over HTTPS. - Only one in-flight LLM request per WebSocket session; concurrent requests receive `{ type: "busy" }`. - Weather must use server-side prefetch, not LLM `get_weather` tool — small Groq models fail tool formatting. ## Plan document See [`PLAN.md`](PLAN.md) for audit findings and implementation status.