Home Assistant Voice Agent (LiveKit)

A voice + text assistant for Home Assistant, built on

LiveKit Agents. This repo is the worker (the agent

brain): it connects to Home Assistant's native

MCP Server through the framework's

built-in `MCPToolset`, and serves STT / LLM / TTS via LiveKit Inference.

The dashboard UI is a Home Assistant custom integration + Lovelace card that runs the

session inside HA:

**ha-livekit-agent-frontend**. Run this

worker, install that integration, point both at the same LiveKit project, and you're set.

• Native MCP — Home Assistant tools (`HassTurnOn`, `HassLightSet`, `GetLiveContext`, …)

are exposed to the LLM directly through LiveKit's `MCPToolset`. No custom MCP client code.

• LiveKit Inference — STT, LLM, and TTS are served by LiveKit's inference gateway; no

per-provider API keys required.

• Cost-aware — STT is billed continuously, so it follows the mic: enabled while audio

input is live and torn down `STT_IDLE_TIMEOUT` seconds (default 120) after the mic is

gated. The worker boots dormant (STT off, no TTS), so an idle card on a dashboard costs

nothing beyond the worker connection, and text chat works without ever enabling STT.

• Turn modes — manual push-to-talk (`start_turn` / `end_turn` / `cancel_turn`) or auto

turn detection; switchable at runtime via `set_turn_mode`.

• Helper tools — a few function tools (`get_areas`, `get_device_domains`, `get_devices`,

`get_environment_info`) pre-process `GetLiveContext` into compact, area/domain-filtered

views so the LLM isn't flooded with the full home state.

Structure

.
├── agent/            # Python LiveKit worker (uv)
│   ├── src/          # main.py (entrypoint), agent.py, config.py, ha.py, scheduler_client.py, …
│   ├── prompt.yaml   # system prompt (bind-mounted, edit without a rebuild)
│   ├── pyproject.toml
│   └── Dockerfile
├── scheduler/        # task scheduler service (FastAPI + APScheduler)
├── docker-compose.yml
├── .env.example      # agent + scheduler env
└── README.md

Prerequisites

• Home Assistant with the MCP Server integration enabled, and a long-lived access token.
• A LiveKit server (self-hosted or LiveKit Cloud) for the realtime transport.
• LiveKit Inference credentials for STT/LLM/TTS (LiveKit Cloud).
• The **HA integration** installed for

the dashboard UI.

Run the worker

Uses uv.

cd agent
cp ../.env.example ../.env    # then fill in the values (see below)
uv sync
uv run src/main.py dev        # or `console` to test in the terminal

Or with Docker Compose (builds and runs the worker):

docker compose up --build

Environment (`.env` in the repo root)

The agent calls `load_dotenv()`, which walks up to the repo-root `.env`.

Models default to Chinese-friendly choices and can be overridden via

`STT_MODEL`, `STT_LANGUAGE`, `LLM_MODEL`, `TTS_MODEL`, `TTS_VOICE`, `TTS_LANGUAGE`

(see `.env.example`).

Turn modes & session control

The card configures the session at runtime over RPCs the worker registers:

• `set_turn_mode` (`manual` | `auto`) — `manual` is push-to-talk: the mic stays gated

until `start_turn`, then `end_turn` commits (agent replies) or `cancel_turn` discards.

`auto` keeps the mic live and lets the model detect turn ends.

• `start_turn` / `end_turn` / `cancel_turn` — manual turn lifecycle. `start_turn` opens

the mic (bringing STT back); `end_turn`/`cancel_turn` gate it (STT tears down after the

idle timeout). STT simply follows the mic.

• `set_audio_output` (`on` | `off`) — toggles the agent's spoken (TTS) replies; off means

text-only. Off by default, so text chat always works without any STT/TTS cost.

The worker broadcasts its live STT + audio-output state on the `ha.speech_state` data channel

so the card can mirror it (e.g. a "sleeping" indicator when STT is torn down), and the

tool-execution lifecycle on `ha.tool_call`.

Frontend / dashboard UI

Use the companion Home Assistant integration:

**ha-livekit-agent-frontend** — a HACS

custom integration + Lovelace card with native device tiles (tap to control), voice, text

chat, transcript, and tool-call status. It mints LiveKit tokens for logged-in HA users and

dispatches this worker by `AGENT_NAME`.