Supported AI Adapters

Every AI model in OpenNVR is a standalone microservice behind one HTTP contract. KAI-C talks to it over POST /infer; nothing else in the core knows what is inside. This page is the reference for the adapters that ship, what tasks they advertise, and what their responses actually look like on the wire.

🛠️ Official Registry

These nine adapters ship in v0.1, maintained by the OpenNVR core team with pre-built images on GHCR. They run locally by default — no cloud round-trip.

Adapter	Advertised task(s)	Type	Primary use case
YOLOv8	object_detection	Vision	Low-latency person and object detection (ONNX, CPU + GPU). The default detector most apps ride.
YOLO Pose	pose_estimation	Vision	COCO-17 body keypoints per person — shoulders, elbows, wrists, hips — so a rule can reason about what someone is doing. CPU-first, ~12 fps on 8 cores at 448px.
ByteTrack	multi_object_tracking	Vision	Stable track IDs across frames, so a rule can tell “one person for thirty seconds” from “thirty people for one second”.
InsightFace	face_detection, face_recognition	Vision	Face detection and embedding with a REST-managed face DB. Faces are PII — installing this is an operator decision.
fast-plate-ocr	license_plate_recognition	Vision	Plate text from a cropped plate, ~30 MB on top of the SDK, CPU-only.
BLIP	scene_caption	Vision	One-sentence scene captions, local. Feeds footage search.
Moondream	visual_qa, scene_caption	Vision	Answers plain-language questions about a frame — the agent’s “what do you see?”.
Whisper	audio_transcription, audio_translation	Audio	Speech-to-text with translation (faster-whisper, CPU + GPU).
Piper	speech_synthesis	Audio	Fast, local text-to-speech. What the OpenNVR Agent speaks with.

Advertised names vs. the canonical taxonomy

The column above is what each adapter puts in tasks_advertised and what you pass as task — it is the literal wire string. OpenNVR also keeps a curated taxonomy in server/config/tasks.yml, and folds the common spellings into it as aliases, so an adapter advertising either name is routed and rendered identically:

Advertised	Canonical task
scene_caption	image_captioning
visual_qa	vqa
audio_transcription, audio_translation	speech_to_text
speech_synthesis	text_to_speech

An adapter may advertise any string it likes. One that matches neither a canonical task nor an alias still registers and works — it is simply uncategorised in the UI until it is promoted into tasks.yml.

An Ollama integration for local LLM chat ships in the bundled reference server. Cloud providers such as Hugging Face are supported as an explicit opt-in — under the default local_only sovereignty policy, any adapter declaring network egress is refused registration.

---

🤝 Community Contributions

Anyone can publish an adapter and list it here. The entries below are illustrative of the shape community contributions take — they are open for contribution, not shipped. Submit yours via the steps in Adding Your Name to the List.

Adapter	Advertised task	Type	Notes
PPE / Hard-Hat Detector	ppe_compliance	Vision	Construction-site safety compliance — a popular community target.
Pose / Fall Detection	fall_detection	Vision	Pose-based fall detection for elder care — on the roadmap, open for contribution.

Two more adapters live in the ai-adapter repository but are not yet in the shipped registry index: OWL-ViT open-vocabulary detection (open_vocab_detection) and an Ollama VLM router (visual_qa, scene_caption) for bring-your-own weights.

---

🏗️ Response Schemas

Every adapter returns the same envelope. The task-specific payload lives under result; model_name, model_version and inference_ms are contract fields the core records against every inference.

Object detection — YOLOv8

Task: object_detection

Bounding boxes are normalised to [0, 1] as an {x, y, w, h} object with x, y at the top-left corner — not pixels. frame_dimensions carries the source frame size if you need to scale back up.

{
  "model_name": "yolov8n",
  "model_version": "onnxruntime/yolov8n",
  "inference_ms": 42,
  "result": {
    "detections": [
      {
        "label": "person",
        "confidence": 0.9412,
        "bbox": { "x": 0.104, "y": 0.203, "w": 0.089, "h": 0.240 },
        "track_id": null,
        "attributes": { "class_id": 0 }
      }
    ],
    "frame_dimensions": { "w": 1920, "h": 1080 },
    "raw_prediction_count": 12
  }
}

track_id is null from a bare detector; it is populated once ByteTrack runs over the detections.

Scene caption — BLIP

Task: scene_caption

{
  "model_name": "Salesforce/blip-image-captioning-base",
  "model_version": "blip/Salesforce/blip-image-captioning-base",
  "inference_ms": 310,
  "result": {
    "task": "scene_caption",
    "caption": "a large truck is parked in a driveway next to a warehouse",
    "model_id": "Salesforce/blip-image-captioning-base",
    "device": "cpu"
  }
}

Facial analysis — InsightFace

Task: face_detection (also face_embedding, face_recognition)

Face boxes are pixel [x1, y1, x2, y2], unlike the detector’s normalised boxes — InsightFace reports in source-image coordinates. landmarks, age and gender appear only when the model pack provides them.

{
  "model_name": "buffalo_l",
  "model_version": "insightface/buffalo_l",
  "inference_ms": 88,
  "result": {
    "task": "face_detection",
    "faces": [
      {
        "bbox": [50, 50, 120, 120],
        "confidence": 0.9912,
        "landmarks": [[55, 60], [65, 60], [60, 70], [56, 78], [64, 78]],
        "age": 32,
        "gender": "F"
      }
    ],
    "face_count": 1,
    "model_pack": "buffalo_l"
  }
}

Errors

A failure returns the §7 failure envelope rather than a result, with a category (MODEL_ERROR, TRANSPORT_ERROR, NOT_SUPPORTED, …), a stable code, and a transient flag telling the caller whether a retry is worth attempting. Streaming (§6 WebSocket) results carry the same envelope, so one parser handles both paths.

---

🚀 Adding Your Name to the List

Don’t see your specialised model here? We encourage developers to list their industry-specific adapters (retail analytics, medical safety, pet tracking, and the rest) in the official registry.

1. Build your adapter — follow the Adapter SDK reference, starting with its quickstart. The SDK is Apache-2.0, so your adapter can carry any licence you like.
2. Register it — add your adapter’s metadata to server/config/adapters_index.yml in the open-nvr repository. That file is the canonical registry: it is what the in-product catalog reads, and this page follows it.
3. Approval — once verified by the core maintainers, your model appears on the AI Registry page for every OpenNVR operator to discover.