What we measured on the Ventuno Q: detector, captions, speech, search and plates on the Hexagon NPU
Between 30 October and 11 November we moved every AI role in OpenNVR onto the Hexagon NPU of an Arduino Ventuno Q and measured each one against the same board's ARM cores. This post is the numbers. Each section links to the report in the repository that holds the full log, the commands and the caveats, so you can rerun any of it. The announcement has the short version and the thanks.
The board and the method
Arduino Ventuno Q: Qualcomm Dragonwing IQ-8275 (QCS8275), eight ARM cores, Adreno 623 GPU, a Hexagon NPU (HTP v75) that Arduino rates at up to 40 dense TOPS, 16 GB LPDDR5, 64 GB eMMC, Ubuntu 24.04 with Qualcomm's 6.8 kernel. Runtime: ONNX Runtime 1.31.1 with the onnxruntime-qnn 2.7.0 plugin (Qualcomm AI Runtime 2.51) for the detector and the ONNX adapters, and Google's LiteRT-LM 0.18 for Gemma. Every CPU figure is the same model on the same board's cores; nothing here was compared across machines. Cameras came from the fake-camera rig (looping clips served over RTSP on the box itself, which costs about half a core the real deployment would not pay) and, for the real-camera run, one office camera. Every measurement was taken after the service reported the accelerator in use; the first "NPU result" we ever saw was a stub detector after a failed session, which is why that check exists.
1. The always-on detector
The Tier-0 detector (YOLO26n, 640 px) runs on every camera at about 2 fps. On the ARM cores it ceilinged at 8.6 inferences a second for the whole pipeline:
| CPU, cameras | Frames detected per camera | Dropped per camera | Wall per inference | Host idle |
|---|---|---|---|---|
| 1 | 2.0 fps | 0 | 164 ms | 46% |
| 4 | 2.0 fps | 0.01 fps | 284 ms | 21% |
| 10 | 0.93 fps | 1.07 fps | 1.05 to 1.13 s | 4% |
Four cameras clean; ten cameras lose more than half their frames. On the NPU the same model, run as fp16, takes 8.6 ms an inference standalone (median; 7.2 min, 11.8 p90), against 609 ms through ONNX Runtime's CPU provider in the same probe container. In the stack:
| NPU, cameras | Frames detected, total | Dropped | Inferences per second | Wall per inference | Host idle |
|---|---|---|---|---|---|
| 16 | 32.1 fps | 0 | 24.1 | 25 ms | 40% |
Sixteen cameras, nothing dropped, and the ARM cores 40% idle. Detections agree with the CPU run: the same boxes within half a pixel, scores within 0.01. The remaining CPU is video decode, which is software on this board (see "A real camera" below). Report: ventuno-q-benchmark-2026-10-30.md.
2. The ONNX adapters
The five ONNX adapters reach the NPU through one provider shim in the adapter SDK. These are end-to-end /infer medians with a 1080p JPEG, so they include JPEG decode, letterboxing and HTTP on the ARM cores; the NPU's own share of the object detector's 63 ms is about 9 ms.
| Adapter | CPU | Hexagon | Note |
|---|---|---|---|
| Object detection (YOLO26n) | 152 ms | 59 to 67 ms | graph compiles in 10 s, 4 s cached |
| Pose (YOLO11n-pose) | 113 ms | 48 to 69 ms | dynamic export pinned to 448 px |
| Package detection | 103 ms | 55 ms | pinned to 416 px |
| Plate OCR (one crop) | 272 ms | 66 to 84 ms | the QNN GPU backend hung at load; HTP only |
| Face recognition (InsightFace buffalo_l) | 436 to 485 ms | 57 to 60 ms | detector pinned to 640, recogniser batch to 1 |
Parakeet, the CPU default for speech, loads on the NPU and fails at inference (its graphs have dynamic lengths), so it stays on the cores; the next section is what replaced it there.
3. Captions and visual questions
A vision-language model on four ARM cores is too slow to caption what a camera saw before the next event arrives: the Qwen3-VL we use on x86 took 60 to 90 s a photo here, past every deadline in the stack. Google's LiteRT-LM ships Gemma 4 E2B compiled for this exact SoC, with the image encoder on the NPU too.
| Model | Where | Caption | Visual question | Resident |
|---|---|---|---|---|
| Gemma 4 E2B (LiteRT-LM) | NPU | 1.6 to 1.8 s | 2.4 s | 1.3 GiB |
| Moondream 0.5B int8 | CPU | 16.0 to 17.4 s | 14.5 s | 2.0 GiB |
| Qwen3-VL 2B (Ollama) | CPU | 60 to 90 s |
While a caption runs, the detector sharing the NPU slows from 38 to 44 ms; the captions themselves were right (a person "in a casual t-shirt, sitting on a chair, using a laptop" was exactly that). Report: ventuno-q-npu-captioner-2026-11-05.md.
4. The camera agent's language model
The same Gemma bundle serves as the agent's LLM, with native tool calling. On the agent's tool-choice questions it picked the right tool 10 of 10 times at 0.8 to 2.0 s a turn, where the CPU default took 20 to 25 s. On the 52-case live evaluation, same box, same cases:
| Model | Cases right | Seconds a turn, by tier (chat / look / investigate) |
|---|---|---|
| Gemma 4 E2B on the NPU | 52 / 52 | 0.6 / 3.1 / 3.2 |
| granite4:micro on the ARM cores | 50 / 52 | 16 / 195 / 218 |
Granite still answers correctly on the cores; it answers three minutes late. The agent's 96-turn evaluation went 96 of 96 on the NPU in under four minutes, after two deterministic fixes the first run exposed, and a live look at a camera dropped from 6 s to 0.6 to 0.8 s once we found the time was in the RTSP grab, not the model. The window is the limit on this bundle: 4,096 tokens, fixed at compile time, which fits 13 of the agent's 14 tool schemas or 7 schemas plus eight turns of history. Reports: agent LLM, agent eval, CPU vs NPU.
5. Speech to text
Qualcomm's AI Hub publishes Whisper Small as precompiled w8a16 context binaries for this chip. We wrapped them as an adapter and compared it with Parakeet on the cores, on five synthesised sentences of two to five seconds:
| Model | Where | Per utterance | Word error rate |
|---|---|---|---|
| Whisper Small w8a16 | NPU | 361 ms | 0.03 (digits written as words) |
| Parakeet TDT 0.6B v3 | CPU | 774 ms | 0.00 |
Half the time, the same words, and Whisper brings a no-speech probability that gates the microphone. STT_ADAPTER=auto now picks it on Qualcomm boards and Parakeet everywhere else. What we have not measured is accented speech on a real microphone; that is the open caveat in the report. Report: ventuno-q-stt-2026-11-09.md.
6. Footage search
Search by description needs an image embedding for every event and a text embedding for every query. SigLIP 2 (w8a16, both towers) compiled for the NPU in 11 s and encodes a frame in 6 to 8 ms and a query in 4 to 6 ms; the ranking was right on five of five probe queries. One number had to change with it: the similarity floor that decides whether anything matches at all dropped from 0.22 to 0.08, because the quantised model's scores sit lower than the CPU model's did. The caption side of search runs a sentence encoder (bge-small) on two ARM cores at 37.6 ms a caption, which is fine there; on the labelled 33-case evaluation the pair took mean reciprocal rank from 0.46 to 0.71 and hit-at-5 from 0.57 to 0.82. Reports: embedder, caption arm.
7. Plates, and what sharing one NPU costs
Reading a plate from a vehicle crop takes 71 to 77 ms on the NPU when the plate reader has the device to itself. The interesting measurement is what happens when it does not. We replayed one vehicle frame through the production path at two reads a second, with and without the captioner running beside it:
| Run | Plate read | Detector inference | CPU, plate adapter |
|---|---|---|---|
| Quiet | 24 to 26 ms | 0.03 cores | |
| Plates alone | 71 to 77 ms | 20 to 23 ms | 0.12 cores |
| Plates and captions | 370 ms | 124 to 138 ms | 0.12 cores |
A caption holds the NPU for about a second; while it does, a plate read and a detector inference each take five times longer, with the CPU idle everywhere. The dashboard had reported this as "NPU 100%", which was misleading in both directions: the device was never idle, and requests were queued behind each other (2.6 in flight on average), so the tile now says "queued" with that number. Setting QNN's context priority high for the detector changed nothing, because LiteRT-LM is not a QNN session and does not yield to one. For a plate camera on a busy site the practical answer today is captions off on that camera and the plate app's fast scan mode; the scheduler that makes captions yield is the next piece of work. Report: ventuno-q-anpr-htp-2026-11-11.md.
8. A real camera
Fake cameras understate decode, so we pushed one real CP Plus office camera at the board: an H.265 "1080N" main stream (960×1080) at about 15 fps with a one-second GOP. Over 30 minutes with the detector on the NPU the box averaged 14.5% busy (median 14.7, peak 20.8), with the detector holding its 2 fps target and 98 person visits recorded. Of that, the pipeline worker and ffmpeg each took about a quarter of one core; decoding that stream in software costs roughly 24% of a core on its own. The extrapolation, which is an extrapolation and not a measurement: ten to twelve such cameras on their main streams before the detector's budget is reached, and considerably more if they are detected on their substreams, where decode is about five times cheaper, which OpenNVR does by itself when the camera has one. Hardware decode would change this picture; the Iris video core is present and the kernel fix for it is upstream, not yet on the board.
What cost us a day each
- Measure accuracy after every accelerator change, not only latency. Pinning the plate OCR graph's batch dimension put all 377 nodes on the NPU and cut a read from 62 ms to 10. It also made the model answer the same plate string for every crop. The change is gated off until the export is fixed.
- Silent fallbacks are the enemy. A missing
libatomic1in a slim image, a provider named instead of attached by device, a missing DMA heap node: each one left the service running on the CPU while listing the NPU as its provider. Every service now reports requested versus effective and the compiled-graph cache doubles as proof. - Dynamic shapes do not compile. Every export with a symbolic axis (pose, package, the face detector) had to be pinned before the NPU would take it; the SDK does that now, and fixed-size exports are what we ask of new adapters.
- The compose version matters. The board's distro Compose (2.24) randomly drops a container's second network at creation; 2.26.1 fixes it, and the installer now checks.
Every report above lives in bench/ with the commands that produced it. The boards were lent by Indus Technologies, who are also working the video-decoder kernel issue with us; the thanks are in the announcement and they are meant. If you have a different Dragonwing or Snapdragon board, the protocol in docs/HARDWARE.md gives you a row to add, and if you would rather we did the measuring, the hardware programme says how to lend us the board.