waifu/CLAUDE.md
2026-05-19 22:13:49 -06:00

161 lines
7.9 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Running the app
```bash
pip install -r requirements.txt
cp .env.example .env # configure GROQ_API_KEY and ELEVENLABS_API_KEY
python main.py
# → http://localhost:8000
```
`live2dcubismcore.min.js` (~202 KB) is downloaded automatically on first run if missing.
## Architecture
**Backend** (`main.py`) connected to a vanilla JS frontend via one persistent WebSocket (`/ws`). Each connection holds a `SessionState` with conversation history, busy/cancel flags, TTS preference, and optional geo coordinates.
Supporting modules:
| File | Role |
|------|------|
| `websearch.py` | `web_search` tool — Serper → DuckDuckGo → Wikipedia |
| `weather.py` | Open-Meteo forecast + `parse_weather_request()` intent detection |
| `workspace.py` | Terminal tools when workspace mode is enabled |
### Message flow
```
Browser mic → PCM Int16 binary frame → /ws
→ Groq Whisper STT → { type: "transcript" }
→ (optional) weather prefetch → { type: "weather_widget" }
→ Groq LLM (history + tools) → { type: "chunk" } × N → { type: "done" }
→ ElevenLabs TTS (if tts_enabled) → { type: "tts_audio", data: base64mp3 }
```
Text input skips STT: browser sends `{ type: "question", text }` directly.
Wake word flow (Safari-compatible — no Web Speech API):
```
Continuous mic + VAD → { type: "wake_check", pcm: base64 }
→ Whisper on server → { type: "wake_ok", text? } | { type: "no_wake" }
→ if text: auto-send question; else: start follow-up recording with post-buffer
```
### WebSocket message types
| Direction | Type | Meaning |
|-----------|------|---------|
| client→server | `audio_config` | Set `lang`, `sample_rate`, `tts_enabled` |
| client→server | `geo_config` | Browser lat/lon for local weather |
| client→server | `wake_check` | Base64 PCM to detect wake word «Arisu» |
| client→server | `workspace_config` | Enable terminal + base directory path |
| client→server | `question` | Text prompt |
| client→server | `cancel` | Cancel in-flight LLM/TTS |
| client→server | `clear_history` | Reset session conversation history |
| client→server | binary | Raw PCM-16LE at 16 kHz (manual recording) |
| server→client | `transcript` | Whisper result |
| server→client | `wake_ok` | Wake word detected; optional `text` = remainder question |
| server→client | `no_wake` | Wake word not found |
| server→client | `thinking` | LLM started |
| server→client | `chunk` | LLM streaming delta (may contain `[tag]`) |
| server→client | `done` | LLM finished |
| server→client | `cancelled` | Response was cancelled |
| server→client | `busy` | Server already processing a request |
| server→client | `tts_audio` | Base64 MP3 |
| server→client | `tts_error` | ElevenLabs failed |
| server→client | `no_speech` | Audio below threshold or too short |
| server→client | `tool_event` | Tool invocation started |
| server→client | `tool_result` | Tool result preview |
| server→client | `weather_widget` | Structured weather data for chat card |
| server→client | `workspace_ok` | Workspace validated |
| server→client | `workspace_error` | Invalid workspace path |
| server→client | `history_cleared` | History reset confirmed |
| server→client | `error` | Generic error |
### Weather
[`weather.py`](weather.py) uses Open-Meteo (no API key). **Do not expose weather via LLM tool calling** — `llama-3.1-8b-instant` often generates malformed tool calls (`tool_use_failed`).
Instead, `_prefetch_weather()` in `main.py`:
1. Runs `parse_weather_request(user_text)` — regex intent + city extraction (aliases like Tokio→Tokyo).
2. Calls `get_weather(location, lang, session.geo_lat, session.geo_lon)`.
3. Sends `{ type: "weather_widget", data: {...} }` to the browser.
4. Injects weather text into the LLM messages as a system note (`DATOS DE CLIMA`).
Frontend renders the widget in `appendWeatherWidget()` with Bootstrap Icons mapped by WMO code icon key.
Geo: browser sends `{ type: "geo_config", lat, lon }` on WebSocket open if geolocation is allowed.
### Web search
[`websearch.py`](websearch.py) provides `web_search(query, lang)` via LLM tool calling. Fallback chain: Serper (if `SERPER_API_KEY`) → DuckDuckGo → Wikipedia.
`current_date_context(lang)` and `enrich_search_query()` inject today's date from `APP_TIMEZONE` (default `America/Mexico_City`) so «hoy» queries use the real year.
### Terminal / workspace mode
When `terminal_enabled` and a valid `workspace_path` are set via `workspace_config`, terminal tools are added alongside the always-available `web_search` tool.
### LLM tool calling
- Tools per session: `[WEB_SEARCH_TOOL]` + terminal tools if enabled.
- `_llm_completion()` retries **without tools** on Groq `tool_use_failed` (400) instead of crashing.
### Session memory
- `SessionState.history` stores up to `MAX_HISTORY` (20) user/assistant messages per WebSocket.
- History is trimmed FIFO when exceeded.
- Cleared via `clear_history` message or frontend "Clear chat" button.
### Wake word (browser)
Key constants in `app.js`:
| Constant | Value | Purpose |
|----------|-------|---------|
| `WAKE_SILENCE_MS` | 1700 | End capture after pause (allows gap between «Arisu» and question) |
| `WAKE_MAX_MS` | 9000 | Max utterance length for wake capture |
| `WAKE_PREROLL_MS` | 350 | Pre-buffer before speech detection |
| `WAKE_POST_BUFFER_MS` | 6000 | Buffer audio during server wake_check STT |
While `wakeChecking`, audio is stored in `wakePostBuffer` and prepended to follow-up recording on `wake_ok` without `text`.
### Audio pipeline (browser)
`getUserMedia` → `AudioContext` → `ScriptProcessor(4096)` → Float32 chunks → downsample to 16 kHz → `Int16Array` → binary WebSocket frame. The server always wraps received bytes as 16 kHz WAV regardless of the `audio_config` sample_rate field.
### Audio tags
The LLM is prompted to include ElevenLabs v3 emotion tags (`[laughs]`, `[giggles]`, `[excited]`, `[whispers]`, `[sighs]`, `[gasps]`) inline. `app.js:detectExpressions()` parses tags from streaming chunks before display and calls `avatar.setExpression(tag)`. Tags are stripped from chat text via `stripAudioTags()`. The full tagged text is sent to ElevenLabs TTS unchanged.
### Live2D (`live2d.js`)
`Live2DAvatar` wraps pixi-live2d-display. Key methods: `setLook(x,y)`, `startSpeaking(analyser?)`, `stopSpeaking()`, `setExpression(tag)`. Lip-sync uses `AnalyserNode` RMS from TTS playback when available; falls back to sinusoidal mouth animation. Model is anchored at `(0.5, 0)` with Y offset to show face+torso.
### TTS playback
Uses `AudioContext.decodeAudioData()` + `BufferSourceNode` + `AnalyserNode` (not `new Audio()`) to avoid browser autoplay blocking and drive lip-sync. Reuses `state.audioCtx` from the mic pipeline.
### i18n
Frontend UI strings are bilingual ES/EN via `I18N` object in `app.js`. UI language is independent of STT language (`langSelect`). Preference stored in `localStorage` key `arisu_ui_lang`.
## Key constraints
- API keys must be in `.env` — never hardcode in `main.py`.
- The server **always** uses `16000` Hz for WAV wrapping — the client always downsamples to 16 kHz before sending.
- RMS check (`< 0.015`) in `stopRecording()` discards silent/noise-only recordings before sending to Whisper.
- `GROQ_STT_MODEL = "whisper-large-v3-turbo"` — do not switch to OpenAI Whisper; it hallucinates more on this audio pipeline.
- TTS voice ID and ElevenLabs model are in `.env` / `main.py` constants (`ELEVEN_VOICE_ID`, `ELEVEN_MODEL = "eleven_v3"`).
- WebSocket URL uses `wss://` when page is served over HTTPS.
- Only one in-flight LLM request per WebSocket session; concurrent requests receive `{ type: "busy" }`.
- Weather must use server-side prefetch, not LLM `get_weather` tool — small Groq models fail tool formatting.
## Plan document
See [`PLAN.md`](PLAN.md) for audit findings and implementation status.