waifu/CLAUDE.md
2026-05-19 22:13:49 -06:00

7.9 KiB
Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Running the app

pip install -r requirements.txt
cp .env.example .env   # configure GROQ_API_KEY and ELEVENLABS_API_KEY
python main.py
# → http://localhost:8000

live2dcubismcore.min.js (~202 KB) is downloaded automatically on first run if missing.

Architecture

Backend (main.py) connected to a vanilla JS frontend via one persistent WebSocket (/ws). Each connection holds a SessionState with conversation history, busy/cancel flags, TTS preference, and optional geo coordinates.

Supporting modules:

File Role
websearch.py web_search tool — Serper → DuckDuckGo → Wikipedia
weather.py Open-Meteo forecast + parse_weather_request() intent detection
workspace.py Terminal tools when workspace mode is enabled

Message flow

Browser mic → PCM Int16 binary frame → /ws
  → Groq Whisper STT → { type: "transcript" }
  → (optional) weather prefetch → { type: "weather_widget" }
  → Groq LLM (history + tools) → { type: "chunk" } × N → { type: "done" }
  → ElevenLabs TTS (if tts_enabled) → { type: "tts_audio", data: base64mp3 }

Text input skips STT: browser sends { type: "question", text } directly.

Wake word flow (Safari-compatible — no Web Speech API):

Continuous mic + VAD → { type: "wake_check", pcm: base64 }
  → Whisper on server → { type: "wake_ok", text? } | { type: "no_wake" }
  → if text: auto-send question; else: start follow-up recording with post-buffer

WebSocket message types

Direction Type Meaning
client→server audio_config Set lang, sample_rate, tts_enabled
client→server geo_config Browser lat/lon for local weather
client→server wake_check Base64 PCM to detect wake word «Arisu»
client→server workspace_config Enable terminal + base directory path
client→server question Text prompt
client→server cancel Cancel in-flight LLM/TTS
client→server clear_history Reset session conversation history
client→server binary Raw PCM-16LE at 16 kHz (manual recording)
server→client transcript Whisper result
server→client wake_ok Wake word detected; optional text = remainder question
server→client no_wake Wake word not found
server→client thinking LLM started
server→client chunk LLM streaming delta (may contain [tag])
server→client done LLM finished
server→client cancelled Response was cancelled
server→client busy Server already processing a request
server→client tts_audio Base64 MP3
server→client tts_error ElevenLabs failed
server→client no_speech Audio below threshold or too short
server→client tool_event Tool invocation started
server→client tool_result Tool result preview
server→client weather_widget Structured weather data for chat card
server→client workspace_ok Workspace validated
server→client workspace_error Invalid workspace path
server→client history_cleared History reset confirmed
server→client error Generic error

Weather

weather.py uses Open-Meteo (no API key). Do not expose weather via LLM tool calling — llama-3.1-8b-instant often generates malformed tool calls (tool_use_failed).

Instead, _prefetch_weather() in main.py:

  1. Runs parse_weather_request(user_text) — regex intent + city extraction (aliases like Tokio→Tokyo).
  2. Calls get_weather(location, lang, session.geo_lat, session.geo_lon).
  3. Sends { type: "weather_widget", data: {...} } to the browser.
  4. Injects weather text into the LLM messages as a system note (DATOS DE CLIMA).

Frontend renders the widget in appendWeatherWidget() with Bootstrap Icons mapped by WMO code icon key.

Geo: browser sends { type: "geo_config", lat, lon } on WebSocket open if geolocation is allowed.

websearch.py provides web_search(query, lang) via LLM tool calling. Fallback chain: Serper (if SERPER_API_KEY) → DuckDuckGo → Wikipedia.

current_date_context(lang) and enrich_search_query() inject today's date from APP_TIMEZONE (default America/Mexico_City) so «hoy» queries use the real year.

Terminal / workspace mode

When terminal_enabled and a valid workspace_path are set via workspace_config, terminal tools are added alongside the always-available web_search tool.

LLM tool calling

  • Tools per session: [WEB_SEARCH_TOOL] + terminal tools if enabled.
  • _llm_completion() retries without tools on Groq tool_use_failed (400) instead of crashing.

Session memory

  • SessionState.history stores up to MAX_HISTORY (20) user/assistant messages per WebSocket.
  • History is trimmed FIFO when exceeded.
  • Cleared via clear_history message or frontend "Clear chat" button.

Wake word (browser)

Key constants in app.js:

Constant Value Purpose
WAKE_SILENCE_MS 1700 End capture after pause (allows gap between «Arisu» and question)
WAKE_MAX_MS 9000 Max utterance length for wake capture
WAKE_PREROLL_MS 350 Pre-buffer before speech detection
WAKE_POST_BUFFER_MS 6000 Buffer audio during server wake_check STT

While wakeChecking, audio is stored in wakePostBuffer and prepended to follow-up recording on wake_ok without text.

Audio pipeline (browser)

getUserMedia → AudioContext → ScriptProcessor(4096) → Float32 chunks → downsample to 16 kHz → Int16Array → binary WebSocket frame. The server always wraps received bytes as 16 kHz WAV regardless of the audio_config sample_rate field.

Audio tags

The LLM is prompted to include ElevenLabs v3 emotion tags ([laughs], [giggles], [excited], [whispers], [sighs], [gasps]) inline. app.js:detectExpressions() parses tags from streaming chunks before display and calls avatar.setExpression(tag). Tags are stripped from chat text via stripAudioTags(). The full tagged text is sent to ElevenLabs TTS unchanged.

Live2D (live2d.js)

Live2DAvatar wraps pixi-live2d-display. Key methods: setLook(x,y), startSpeaking(analyser?), stopSpeaking(), setExpression(tag). Lip-sync uses AnalyserNode RMS from TTS playback when available; falls back to sinusoidal mouth animation. Model is anchored at (0.5, 0) with Y offset to show face+torso.

TTS playback

Uses AudioContext.decodeAudioData() + BufferSourceNode + AnalyserNode (not new Audio()) to avoid browser autoplay blocking and drive lip-sync. Reuses state.audioCtx from the mic pipeline.

i18n

Frontend UI strings are bilingual ES/EN via I18N object in app.js. UI language is independent of STT language (langSelect). Preference stored in localStorage key arisu_ui_lang.

Key constraints

  • API keys must be in .env — never hardcode in main.py.
  • The server always uses 16000 Hz for WAV wrapping — the client always downsamples to 16 kHz before sending.
  • RMS check (< 0.015) in stopRecording() discards silent/noise-only recordings before sending to Whisper.
  • GROQ_STT_MODEL = "whisper-large-v3-turbo" — do not switch to OpenAI Whisper; it hallucinates more on this audio pipeline.
  • TTS voice ID and ElevenLabs model are in .env / main.py constants (ELEVEN_VOICE_ID, ELEVEN_MODEL = "eleven_v3").
  • WebSocket URL uses wss:// when page is served over HTTPS.
  • Only one in-flight LLM request per WebSocket session; concurrent requests receive { type: "busy" }.
  • Weather must use server-side prefetch, not LLM get_weather tool — small Groq models fail tool formatting.

Plan document

See PLAN.md for audit findings and implementation status.