gptme-voice

Voice interface for gptme agents using OpenAI or xAI Grok Realtime APIs.

Features

• Real-time voice conversations with low-latency audio streaming
• Agent personality loading from gptme.toml project config (ABOUT.md, etc.)
• Subagent tool dispatches tasks to gptme for workspace interaction (read files, check tasks, run commands)
• workspace_search (opt-in GPTME_VOICE_RAG=1) — fast gptme-rag lookup over recent journals for recap / "last hour" questions, without a subagent
• Auto-detection of agent repo when installed in gptme-contrib
• Feedback loop prevention by muting mic during playback
• Twilio integration for phone call support via Media Streams
• Local testing with direct microphone/speaker I/O
• BobBrain camera tool — the /local session exposes look and runs VLM inference only after the edge node returns an on-demand frame
• Goal-level body tools — in-process MAVSDK or an authenticated remote bob-body/0 node can handle status, bounded movement, stop, turn, and local interaction directly in the realtime tool bridge (never via a subagent)

Installation

# Install with poetry (from gptme-contrib)
cd packages/gptme-voice
poetry install

# For local mic/speaker testing
poetry install -E local

Usage

Start the server

# Auto-detects agent repo and loads personality
gptme-voice-server

# Use xAI Grok
gptme-voice-server --provider grok

# With debug logging
gptme-voice-server --debug

# Explicit workspace
gptme-voice-server --workspace /path/to/agent-repo

The server auto-detects the agent repo by walking up from gptme-contrib to find gptme.toml, and loads personality files (prioritizing ABOUT.md).

Connect with local client

# In a separate terminal
gptme-voice-client

Speak into your microphone. The agent responds with its configured personality and can use the subagent tool to interact with its workspace.

Tip: Use headphones to enable interrupting the agent mid-sentence (see Limitations below).

Receive phone calls via Twilio

1. Start the server with a public URL (e.g. via ngrok):

   gptme-voice-server --port 8080
   ngrok http 8080
   
   

2. In the Twilio console, set your phone number's Voice webhook to: https://<your-ngrok-url>/incoming (HTTP POST)
3. Call the Twilio number — Twilio connects the call to the voice server.

Missed-call context persistence

When a trusted operator calls back within 30 minutes of an unanswered outbound call, the inbound session receives the context that was prepared for the original call — as if the call took place but the operator was silent. The original call must also have been placed on the same UTC calendar day: a call placed at 23:50 UTC that is returned at 00:10 UTC the next day will not restore context.

Context note format — the outbound call path writes state/voice-calls/missed-call-context.json:

{
  "type": "standup",
  "sid": "CA...",
  "date": "2026-09-15",
  "placed_at": "2026-09-15T10:00:00+00:00",
  "caller": "+15551212",
  "context_file": "state/standup-brief.json"
}

The inbound reader treats all type values the same. It reads the referenced context_file (workspace-relative; path traversal is rejected) and injects that content into the callback session, including the file path so the session can see the link. A standup-brief-shaped JSON file (generated_at + text) keeps the existing freshness checks. Other JSON and text files are loaded as-is — generated_at is taken from the missed call so a long-lived notes file still restores on callback. An optional inlined context snapshot is a fallback when the file is missing, stale, or unreadable (for example a replacement brief generated after the missed call).

The outbound call path (create_outbound_call / gptme-voice-call --context-file) writes the note when callers pass workspace plus a context_file and/or a prepared missed_call_context snapshot. The inbound loader still requires Twilio to confirm the outbound leg ended unanswered.

gptme-voice-call +46701234567 \
  --workspace /path/to/agent-repo \
  --context-file state/standup-brief.json \
  --call-type standup

Trust requirements — the callback path requires a signed /incoming webhook, an exact TWILIO_CALLER_ALLOWLIST match, and Call role: operator in the caller's people file. The WebSocket must present the webhook's grant bound to both number and CallSid. A bounded two-second Twilio lookup must confirm the stamped outbound call went to this caller and ended no-answer, busy, failed, or canceled. Answered, unresolved, stale, missing, or malformed evidence leaves normal inbound behavior intact. API errors fail closed.

Callback sessions bypass generic number-keyed prewarms. While a trusted caller has fresh local callback evidence, /incoming skips prewarming so it cannot consume recent-call state before the routing decision. An intervening call takes precedence and resumes normally.

Legacy standup files — if no missed-call-context.json exists, the loader falls back to reading state/voice-calls/last-standup-call-sid.txt + state/standup-brief.json so existing deployments continue to work.

Place outbound phone calls via Twilio

Set these values in your environment or gptme config:

TWILIO_ACCOUNT_SID=...
TWILIO_AUTH_TOKEN=...
TWILIO_PHONE_NUMBER=...
GPTME_VOICE_PUBLIC_BASE_URL=https://<your-ngrok-url>

Then place a call:

gptme-voice-call +46701234567

Pass --context-file (and optionally --workspace / --call-type) to drop a missed-call context note so a trusted callback can pick up that file if the call goes unanswered.

Use --dry-run to print the generated TwiML without dialing.

Connect a remote body node

Configure a private body endpoint and pass its token separately so credentials do not appear in URLs or logs:

GPTME_VOICE_BODY_URL=tcp://127.0.0.1:7777
GPTME_VOICE_BODY_TOKEN=<body-node-token>
GPTME_VOICE_BODY_CONTROLLER_ID=gptme-voice-local  # optional

Plaintext tcp:// is loopback-only (127.0.0.1 / ::1; hostnames including localhost are refused). A non-loopback host is refused so the bearer token and physical commands never cross the network in the clear.

The remote node negotiates its actual capabilities during the controller-authenticated handshake. gptme-voice registers only the corresponding realtime tools. The body node remains responsible for controller leases, command TTLs, idempotency, deadman behavior, and collision/local safety.

API keys

Keys are loaded from gptme config (~/.config/gptme/config.toml or config.local.toml):

• OPENAI_API_KEY for the default openai provider
• XAI_API_KEY for --provider grok

No need to export them as shell env vars if they're already configured in gptme.

Workspace search (gptme-rag)

Set GPTME_VOICE_RAG=1 to advertise a workspace_search tool. Recap questions such as "what have you been doing in the last hour?" search recent journal/ files through gptme-rag (lexical first) and return in a few seconds instead of dispatching a subagent. Optional: GPTME_VOICE_RAG_TIMEOUT_SECONDS (default 8) and GPTME_VOICE_RAG_RECENCY_HOURS (default 24).

Voice latency tracing

Set GPTME_VOICE_LATENCY_SINK to a file path or - (stdout). Each utterance emits one JSONL utterance_trace with:

• asr_ms — VAD speech_stopped → user transcript completed
• tts_first_audio_ms — response.created → first audio chunk
• round_trip_ms — speech_stopped → first audio chunk

send_audio is counted (input_audio_chunks) but is not the round-trip clock; Twilio streams PCM continuously.

Architecture

• openai_client.py - WebSocket client for OpenAI Realtime API with VAD, audio streaming, and event handling
• xai_client.py - xAI Grok Voice Agent adapter (OpenAI-compatible WebSocket protocol)
• server.py - Starlette WebSocket server bridging clients to OpenAI or xAI
• tool_bridge.py - Async subagent dispatcher plus body/vision/RAG tool routing
• rag.py - Recency-scoped gptme-rag search for live recap queries
• vision.py - Correlated camera-frame requests, edge-event handling, and host-side VLM inference
• audio.py - Audio format conversion (PCM ↔ μ-law for Twilio)
• client.py - Local client with mic/speaker I/O and feedback loop prevention
• latency.py - Per-utterance ASR / TTS-first-audio / round-trip tracing

Limitations

• No interruption without headphones. The local test client mutes the mic while audio is playing to prevent feedback loops (speaker → mic → infinite loop). This means you can't interrupt the agent mid-sentence when using speakers. Use headphones to avoid this — with headphones there's no speaker bleed into the mic, so the client could skip muting. A proper fix would be acoustic echo cancellation (AEC), e.g. via speexdsp or WebRTC AEC.
• Subagent latency. Tool calls dispatch a full gptme subprocess, which takes a few seconds. The voice conversation continues while it runs, and the result is injected when ready.