The Camera Agent

Talk to your cameras — and get a full operator product: events that click to the moment, leveled alarms, real-time WebRTC screens, recordings, and RBAC sign-in.

On this page

The camera agent is OpenNVR’s conversational front end — and a complete operator product in its own right. Ask by voice or text; everything runs on your hardware (Whisper → a local Ollama LLM with tool-calling → Piper, or any OpenAI-compatible endpoint you bring).

The screens

  • All cameras: live thumbnails per camera. Tap to peek — the agent drops that camera’s current frame into the chat with “Open →” and “📌 Ask about this frame” actions. Pin a frame and your next question is answered about that exact moment, not whatever the camera sees by the time the tools run.
  • Per-camera screen (/demo/camera/<id>, deep-linkable): plays the same WebRTC stream as the NVR’s Live view (stills are only a labelled fallback), with a scrub-back review ring of the last few minutes, detection markers on the track, and the server’s recorded segments played inline — ● LIVE always snaps back. Talk/Ask are scoped to that camera; Watch/Alarm are one-tap popovers.

Standing work

Panel What it is
Events ONE feed of alarm rings, watch hits, and app alerts. Click an event → the camera opens at that timestamp (review ring first, covering recording second, live as fallback).
Alarms Armed conditions with four annunciation levels: 🚨 siren (latches until silenced), ⚠️ pulse (loud ~1 min, stands down), 🔔 chime (one ding), 🤫 silent (log + push). Per-target defaults are site-configurable and editable in the UI (⚙ in the Alarms card).
Watching Standing monitors — notify-when-seen, live counts, line crossings — hosted on the same SDK rules the store apps run.
Background tasks Long jobs (“check every camera for a red shirt”) run off-thread; the answer lands back in chat.
Scheduled reports “Every morning at 7, summarise overnight activity” — daily or every-N-minutes, delivered to chat and your alert channels.
Hardware What your enabled skills run on vs. what makes them fast, derived from live adapter capabilities — advisory, never a gate.

Notifications

Alarms, watch hits, finished tasks, and reports fan out to webhooks (Slack, Discord, Teams, Home Assistant, n8n — one JSON POST) and to Apprise (pip install apprise): email, Telegram, ntfy phone push, Pushover, SMS — 100+ services from one config list. With agent_public_url set, every push carries a tap-to-open deep link to the right camera’s screen.

Authentication

auth_mode: opennvr delegates sign-in to your OpenNVR server — the agent never mints credentials. Three tiers from your existing RBAC roles: viewers look and chat (the mutating chat tools are physically removed from their toolset), operators arm and manage, admins govern skills and site policy. Bearer + refresh tokens, no cookies — the same contract a future mobile app will use.

Latency on CPU

A spoken turn is STT → LLM → tools → LLM → TTS, and on an 8-core box without a GPU each stage has a knob (.env; detail in examples/camera-agent/MODELS_AND_LATENCY.md):

Knob Default Why
OLLAMA_MODEL granite4:micro must support tool-calling. The interactive installer picks from the model catalog by RAM and cores: granite4:micro on a box with 4+ cores or a GPU (the best measured at calling its tools before it answers), qwen3:1.7b for turns in half the time, qwen2.5:0.5b on a small box
WHISPER_MODEL_SIZE base.en .en is faster and hallucinates less on quiet audio; the agent asks for a greedy decode
PIPER_VOICE en_US-lessac-medium the quality tier is the CPU cost — a “high” voice took ~4 cores and 25 s for two sentences
PIPER_THREADS / WHISPER_CPU_THREADS 2 / all thread caps so one utterance does not take the whole box from detection and recording
router_tier0 / router_hints (config.yml) on simple camera questions (“is anyone at the front door?”, “did a car come by cam1 in the last hour?”) are decided by a router before the model sees them — the tool runs at once and the model only says the answer; anything with a side effect, or about who, still goes to the model
stt_trailing_silence_secs (config.yml) 0.5 silence padded onto each utterance so Whisper hears the end of the last word

The streaming voice path runs on Pipecat 1.12: Silero VAD opens a turn, Smart Turn v3 (bundled, CPU) closes it, and transcription runs off the pipeline’s path so a two-second Whisper call never stalls turn-taking. The image downloads nothing at run time.

Quickstart

examples/camera-agent/quickstart.sh          # voice
examples/camera-agent/quickstart.sh --chat   # text-only, lightest CPU

See examples/camera-agent/README.md, ALARMS.md, and NOTIFICATIONS.md in the main repository for configuration detail.