The Camera Agent
Talk to your cameras — and get a full operator product: events that click to the moment, leveled alarms, real-time WebRTC screens, recordings, and RBAC sign-in.
The camera agent is OpenNVR’s conversational front end — and a complete operator product in its own right. Ask by voice or text; everything runs on your hardware (Whisper → a local Ollama LLM with tool-calling → Piper, or any OpenAI-compatible endpoint you bring).
The screens
- All cameras: live thumbnails per camera. Tap to peek — the agent drops that camera’s current frame into the chat with “Open →” and “📌 Ask about this frame” actions. Pin a frame and your next question is answered about that exact moment, not whatever the camera sees by the time the tools run.
- Per-camera screen (
/demo/camera/<id>, deep-linkable): plays the same WebRTC stream as the NVR’s Live view (stills are only a labelled fallback), with a scrub-back review ring of the last few minutes, detection markers on the track, and the server’s recorded segments played inline — ● LIVE always snaps back. Talk/Ask are scoped to that camera; Watch/Alarm are one-tap popovers.
Standing work
| Panel | What it is |
|---|---|
| Events | ONE feed of alarm rings, watch hits, and app alerts. Click an event → the camera opens at that timestamp (review ring first, covering recording second, live as fallback). |
| Alarms | Armed conditions with four annunciation levels: 🚨 siren (latches until silenced), ⚠️ pulse (loud ~1 min, stands down), 🔔 chime (one ding), 🤫 silent (log + push). Per-target defaults are site-configurable and editable in the UI (⚙ in the Alarms card). |
| Watching | Standing monitors — notify-when-seen, live counts, line crossings — hosted on the same SDK rules the store apps run. |
| Background tasks | Long jobs (“check every camera for a red shirt”) run off-thread; the answer lands back in chat. |
| Scheduled reports | “Every morning at 7, summarise overnight activity” — daily or every-N-minutes, delivered to chat and your alert channels. |
| Hardware | What your enabled skills run on vs. what makes them fast, derived from live adapter capabilities — advisory, never a gate. |
Notifications
Alarms, watch hits, finished tasks, and reports fan out to webhooks
(Slack, Discord, Teams, Home Assistant, n8n — one JSON POST) and to
Apprise (pip install apprise): email, Telegram, ntfy phone push,
Pushover, SMS — 100+ services from one config list. With
agent_public_url set, every push carries a tap-to-open deep link to the
right camera’s screen.
Authentication
auth_mode: opennvr delegates sign-in to your OpenNVR server — the agent
never mints credentials. Three tiers from your existing RBAC roles:
viewers look and chat (the mutating chat tools are physically removed
from their toolset), operators arm and manage, admins govern
skills and site policy. Bearer + refresh tokens, no cookies — the same
contract a future mobile app will use.
Latency on CPU
A spoken turn is STT → LLM → tools → LLM → TTS, and on an 8-core box
without a GPU each stage has a knob (.env; detail in
examples/camera-agent/MODELS_AND_LATENCY.md):
| Knob | Default | Why |
|---|---|---|
OLLAMA_MODEL |
granite4:micro |
must support tool-calling. The interactive installer picks from the model catalog by RAM and cores: granite4:micro on a box with 4+ cores or a GPU (the best measured at calling its tools before it answers), qwen3:1.7b for turns in half the time, qwen2.5:0.5b on a small box |
WHISPER_MODEL_SIZE |
base.en |
.en is faster and hallucinates less on quiet audio; the agent asks for a greedy decode |
PIPER_VOICE |
en_US-lessac-medium |
the quality tier is the CPU cost — a “high” voice took ~4 cores and 25 s for two sentences |
PIPER_THREADS / WHISPER_CPU_THREADS |
2 / all |
thread caps so one utterance does not take the whole box from detection and recording |
router_tier0 / router_hints (config.yml) |
on | simple camera questions (“is anyone at the front door?”, “did a car come by cam1 in the last hour?”) are decided by a router before the model sees them — the tool runs at once and the model only says the answer; anything with a side effect, or about who, still goes to the model |
stt_trailing_silence_secs (config.yml) |
0.5 |
silence padded onto each utterance so Whisper hears the end of the last word |
The streaming voice path runs on Pipecat 1.12: Silero VAD opens a turn, Smart Turn v3 (bundled, CPU) closes it, and transcription runs off the pipeline’s path so a two-second Whisper call never stalls turn-taking. The image downloads nothing at run time.
Quickstart
examples/camera-agent/quickstart.sh # voice
examples/camera-agent/quickstart.sh --chat # text-only, lightest CPU
See examples/camera-agent/README.md, ALARMS.md, and
NOTIFICATIONS.md in the main repository for configuration detail.