161 lines
7.9 KiB
Markdown
161 lines
7.9 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Running the app
|
||
|
||
```bash
|
||
pip install -r requirements.txt
|
||
cp .env.example .env # configure GROQ_API_KEY and ELEVENLABS_API_KEY
|
||
python main.py
|
||
# → http://localhost:8000
|
||
```
|
||
|
||
`live2dcubismcore.min.js` (~202 KB) is downloaded automatically on first run if missing.
|
||
|
||
## Architecture
|
||
|
||
**Backend** (`main.py`) connected to a vanilla JS frontend via one persistent WebSocket (`/ws`). Each connection holds a `SessionState` with conversation history, busy/cancel flags, TTS preference, and optional geo coordinates.
|
||
|
||
Supporting modules:
|
||
|
||
| File | Role |
|
||
|------|------|
|
||
| `websearch.py` | `web_search` tool — Serper → DuckDuckGo → Wikipedia |
|
||
| `weather.py` | Open-Meteo forecast + `parse_weather_request()` intent detection |
|
||
| `workspace.py` | Terminal tools when workspace mode is enabled |
|
||
|
||
### Message flow
|
||
|
||
```
|
||
Browser mic → PCM Int16 binary frame → /ws
|
||
→ Groq Whisper STT → { type: "transcript" }
|
||
→ (optional) weather prefetch → { type: "weather_widget" }
|
||
→ Groq LLM (history + tools) → { type: "chunk" } × N → { type: "done" }
|
||
→ ElevenLabs TTS (if tts_enabled) → { type: "tts_audio", data: base64mp3 }
|
||
```
|
||
|
||
Text input skips STT: browser sends `{ type: "question", text }` directly.
|
||
|
||
Wake word flow (Safari-compatible — no Web Speech API):
|
||
|
||
```
|
||
Continuous mic + VAD → { type: "wake_check", pcm: base64 }
|
||
→ Whisper on server → { type: "wake_ok", text? } | { type: "no_wake" }
|
||
→ if text: auto-send question; else: start follow-up recording with post-buffer
|
||
```
|
||
|
||
### WebSocket message types
|
||
|
||
| Direction | Type | Meaning |
|
||
|-----------|------|---------|
|
||
| client→server | `audio_config` | Set `lang`, `sample_rate`, `tts_enabled` |
|
||
| client→server | `geo_config` | Browser lat/lon for local weather |
|
||
| client→server | `wake_check` | Base64 PCM to detect wake word «Arisu» |
|
||
| client→server | `workspace_config` | Enable terminal + base directory path |
|
||
| client→server | `question` | Text prompt |
|
||
| client→server | `cancel` | Cancel in-flight LLM/TTS |
|
||
| client→server | `clear_history` | Reset session conversation history |
|
||
| client→server | binary | Raw PCM-16LE at 16 kHz (manual recording) |
|
||
| server→client | `transcript` | Whisper result |
|
||
| server→client | `wake_ok` | Wake word detected; optional `text` = remainder question |
|
||
| server→client | `no_wake` | Wake word not found |
|
||
| server→client | `thinking` | LLM started |
|
||
| server→client | `chunk` | LLM streaming delta (may contain `[tag]`) |
|
||
| server→client | `done` | LLM finished |
|
||
| server→client | `cancelled` | Response was cancelled |
|
||
| server→client | `busy` | Server already processing a request |
|
||
| server→client | `tts_audio` | Base64 MP3 |
|
||
| server→client | `tts_error` | ElevenLabs failed |
|
||
| server→client | `no_speech` | Audio below threshold or too short |
|
||
| server→client | `tool_event` | Tool invocation started |
|
||
| server→client | `tool_result` | Tool result preview |
|
||
| server→client | `weather_widget` | Structured weather data for chat card |
|
||
| server→client | `workspace_ok` | Workspace validated |
|
||
| server→client | `workspace_error` | Invalid workspace path |
|
||
| server→client | `history_cleared` | History reset confirmed |
|
||
| server→client | `error` | Generic error |
|
||
|
||
### Weather
|
||
|
||
[`weather.py`](weather.py) uses Open-Meteo (no API key). **Do not expose weather via LLM tool calling** — `llama-3.1-8b-instant` often generates malformed tool calls (`tool_use_failed`).
|
||
|
||
Instead, `_prefetch_weather()` in `main.py`:
|
||
|
||
1. Runs `parse_weather_request(user_text)` — regex intent + city extraction (aliases like Tokio→Tokyo).
|
||
2. Calls `get_weather(location, lang, session.geo_lat, session.geo_lon)`.
|
||
3. Sends `{ type: "weather_widget", data: {...} }` to the browser.
|
||
4. Injects weather text into the LLM messages as a system note (`DATOS DE CLIMA`).
|
||
|
||
Frontend renders the widget in `appendWeatherWidget()` with Bootstrap Icons mapped by WMO code icon key.
|
||
|
||
Geo: browser sends `{ type: "geo_config", lat, lon }` on WebSocket open if geolocation is allowed.
|
||
|
||
### Web search
|
||
|
||
[`websearch.py`](websearch.py) provides `web_search(query, lang)` via LLM tool calling. Fallback chain: Serper (if `SERPER_API_KEY`) → DuckDuckGo → Wikipedia.
|
||
|
||
`current_date_context(lang)` and `enrich_search_query()` inject today's date from `APP_TIMEZONE` (default `America/Mexico_City`) so «hoy» queries use the real year.
|
||
|
||
### Terminal / workspace mode
|
||
|
||
When `terminal_enabled` and a valid `workspace_path` are set via `workspace_config`, terminal tools are added alongside the always-available `web_search` tool.
|
||
|
||
### LLM tool calling
|
||
|
||
- Tools per session: `[WEB_SEARCH_TOOL]` + terminal tools if enabled.
|
||
- `_llm_completion()` retries **without tools** on Groq `tool_use_failed` (400) instead of crashing.
|
||
|
||
### Session memory
|
||
|
||
- `SessionState.history` stores up to `MAX_HISTORY` (20) user/assistant messages per WebSocket.
|
||
- History is trimmed FIFO when exceeded.
|
||
- Cleared via `clear_history` message or frontend "Clear chat" button.
|
||
|
||
### Wake word (browser)
|
||
|
||
Key constants in `app.js`:
|
||
|
||
| Constant | Value | Purpose |
|
||
|----------|-------|---------|
|
||
| `WAKE_SILENCE_MS` | 1700 | End capture after pause (allows gap between «Arisu» and question) |
|
||
| `WAKE_MAX_MS` | 9000 | Max utterance length for wake capture |
|
||
| `WAKE_PREROLL_MS` | 350 | Pre-buffer before speech detection |
|
||
| `WAKE_POST_BUFFER_MS` | 6000 | Buffer audio during server wake_check STT |
|
||
|
||
While `wakeChecking`, audio is stored in `wakePostBuffer` and prepended to follow-up recording on `wake_ok` without `text`.
|
||
|
||
### Audio pipeline (browser)
|
||
|
||
`getUserMedia` → `AudioContext` → `ScriptProcessor(4096)` → Float32 chunks → downsample to 16 kHz → `Int16Array` → binary WebSocket frame. The server always wraps received bytes as 16 kHz WAV regardless of the `audio_config` sample_rate field.
|
||
|
||
### Audio tags
|
||
|
||
The LLM is prompted to include ElevenLabs v3 emotion tags (`[laughs]`, `[giggles]`, `[excited]`, `[whispers]`, `[sighs]`, `[gasps]`) inline. `app.js:detectExpressions()` parses tags from streaming chunks before display and calls `avatar.setExpression(tag)`. Tags are stripped from chat text via `stripAudioTags()`. The full tagged text is sent to ElevenLabs TTS unchanged.
|
||
|
||
### Live2D (`live2d.js`)
|
||
|
||
`Live2DAvatar` wraps pixi-live2d-display. Key methods: `setLook(x,y)`, `startSpeaking(analyser?)`, `stopSpeaking()`, `setExpression(tag)`. Lip-sync uses `AnalyserNode` RMS from TTS playback when available; falls back to sinusoidal mouth animation. Model is anchored at `(0.5, 0)` with Y offset to show face+torso.
|
||
|
||
### TTS playback
|
||
|
||
Uses `AudioContext.decodeAudioData()` + `BufferSourceNode` + `AnalyserNode` (not `new Audio()`) to avoid browser autoplay blocking and drive lip-sync. Reuses `state.audioCtx` from the mic pipeline.
|
||
|
||
### i18n
|
||
|
||
Frontend UI strings are bilingual ES/EN via `I18N` object in `app.js`. UI language is independent of STT language (`langSelect`). Preference stored in `localStorage` key `arisu_ui_lang`.
|
||
|
||
## Key constraints
|
||
|
||
- API keys must be in `.env` — never hardcode in `main.py`.
|
||
- The server **always** uses `16000` Hz for WAV wrapping — the client always downsamples to 16 kHz before sending.
|
||
- RMS check (`< 0.015`) in `stopRecording()` discards silent/noise-only recordings before sending to Whisper.
|
||
- `GROQ_STT_MODEL = "whisper-large-v3-turbo"` — do not switch to OpenAI Whisper; it hallucinates more on this audio pipeline.
|
||
- TTS voice ID and ElevenLabs model are in `.env` / `main.py` constants (`ELEVEN_VOICE_ID`, `ELEVEN_MODEL = "eleven_v3"`).
|
||
- WebSocket URL uses `wss://` when page is served over HTTPS.
|
||
- Only one in-flight LLM request per WebSocket session; concurrent requests receive `{ type: "busy" }`.
|
||
- Weather must use server-side prefetch, not LLM `get_weather` tool — small Groq models fail tool formatting.
|
||
|
||
## Plan document
|
||
|
||
See [`PLAN.md`](PLAN.md) for audit findings and implementation status.
|