# AI Models for Frigate Enrichments

Frigate uses a handful of AI models — the object detector, plus the
"enrichment" features that build on it (semantic search, face recognition,
license plates, bird and audio detection, and more). This page lists every
model, **grouped by feature**, and **where** each runs — the **Neural Engine
(ANE)**, the **GPU**, or the **CPU** — on Fregata and, for comparison, in Docker.

The short version: **object detection — the one thing that runs on every single
frame — gets the Neural Engine to itself.** For everything else, Fregata uses the
**GPU** for the models Frigate accelerates *that Apple's CoreML can run*, and the
**CPU** for the rest. A few models that upstream Frigate accelerates on a GPU use
dynamic input shapes the Mac's CoreML GPU path can't take yet, so they run on the
CPU for now — none of them a hot path (see
["A few models run on the CPU for now"](#a-few-models-run-on-the-cpu-for-now)).

You don't configure any of this — Fregata picks the right place automatically.
The only related knob is `model_size` on the features that offer one (for
example face recognition's `small` vs `large`).

## Where each model runs

The full picture — every model, the feature it powers, and where it runs across
the three ways you might run Frigate:

| Feature | Model (variant) | macOS Docker¹ | Linux / Windows Docker² | Fregata (macOS) |
| --- | --- | --- | --- | --- |
| **Object detection**³ | YOLOv9-t (the detector) | CPU | GPU / NPU | **Neural Engine** |
| **Semantic search** | jina-clip — vision (images) | CPU | GPU | **GPU** |
| | jina-clip — text (query) | CPU | GPU | CPU † |
| **Face recognition** | FaceDet — finding faces | CPU | GPU | **GPU** |
| | FaceNet — `small` embedding (default) | CPU | CPU | CPU |
| | ArcFace — `large` embedding | CPU | GPU | **GPU** |
| **License plate** | YOLOv9-256 — plate detection | CPU | GPU | **GPU** |
| | PaddleOCR — orientation | CPU | GPU | **GPU** |
| | PaddleOCR — recognition_v4 (OCR) | CPU | GPU | CPU † |
| **Bird classification** | MobileNetV2 (iNaturalist) | CPU | CPU | CPU |
| **Audio detection** | YAMNet | CPU | CPU | CPU |
| **Audio transcription** | sherpa-onnx — `small` (default) | CPU | CPU | CPU |
| | Whisper — `large` | CPU | GPU | CPU † |
| **Custom classification** | MobileNetV2 — training | CPU | GPU | CPU |
| | trained model (`.tflite`) — inference | CPU | CPU | CPU |

¹ **Docker Desktop on a Mac runs everything on the CPU.** Docker's Linux VM
can't reach the Mac's Neural Engine or GPU, so nothing is
accelerated — the main reason Fregata exists.

² With a supported GPU or accelerator (NVIDIA/TensorRT, Intel/OpenVINO, a Coral,
etc.). *Without* one, Linux/Windows Docker also runs everything on the CPU —
same as the macOS Docker column.

³ The detector model differs by platform: Fregata bundles **YOLOv9-t** and runs
it on the ANE; Docker images default to a CPU SSD/MobileDet model (or an
accelerator-specific one).

† **GPU on a GPU-equipped Frigate, but CPU on Fregata today.** These three use
dynamic input shapes Apple's CoreML GPU path can't take yet. None is a hot path
(they run per search query, per plate, and per spoken phrase — not per frame),
so the impact is small; GPU support is a planned optimization in a future release. See
[below](#a-few-models-run-on-the-cpu-for-now).

The takeaway: **Fregata is the only way to optimally hardware-accelerate Frigate on a
Mac.** macOS Docker is CPU-only across the board; Fregata puts object detection
on the Neural Engine and most enrichment models on the GPU — matching a
GPU-equipped Linux/Windows box except for three dynamic-shape models that fall
back to the CPU for now.

## Why object detection owns the Neural Engine

The ANE is Apple's dedicated, extremely power-efficient inference chip — but it
works best running **one** model at a time. Object detection runs multiple times on every frame
with motion from every camera, so it's by far the most demanding and latency-sensitive job
Fregata does. Pinning detection to the ANE — and routing everything else to the
GPU or CPU — means a burst of face recognition or search indexing can never
stall the detector. It's also why Fregata needs Apple Silicon: Intel Macs have
no Neural Engine, and the CPU fallback (~50 ms/frame versus ~2 ms on the ANE) is
too slow to be a real NVR.

## Why some models use the GPU and others the CPU

Fregata is a faithful port of Frigate and keeps the same accuracy. Frigate
already decides, per model, where it runs; Fregata honors those choices on Apple
Silicon wherever the Mac's GPU path can:

- **GPU** (Apple's CoreML, via ONNX Runtime) — the models Frigate accelerates
  that have fixed input shapes CoreML can compile: image search (vision),
  `large` face recognition, and license-plate detection + orientation.
- **CPU, same as upstream** — the models Frigate runs on the CPU everywhere:
  bird, audio detection, `small` face recognition, and the small speech-to-text
  model. These are small / TFLite models with no GPU path upstream either, so they run on the CPU like in all version of Frigate.

### A few models run on the CPU for now

Three models that upstream Frigate accelerates on a GPU use **dynamic input
shapes** the Mac's CoreML GPU path can't take yet, so today they run on the CPU:
the **text side of semantic search**, **license-plate OCR**, and the **`large`
(Whisper) transcription** model. On a GPU-equipped Frigate those steps use the
GPU, so they're slower on Fregata for now — but none is a hot path (they run
per search query, per plate, and per spoken phrase, not per frame), so the
real-world impact is small. GPU support for them is a planned optimization in a future release.

## When models download

Object detection and audio detection ship **inside the app**, so they work
offline the moment you install Fregata. The other models download automatically
the first time you turn on the feature that needs them, into Fregata's model
cache — a one-time download per model. Custom-model **training** additionally
fetches TensorFlow on first use (a larger, one-time download); the model you
train then runs on the CPU like any other.