T

gx a1090a5f4c controller: Phase 4a — overlay infrastructure (data models + API + Frigate bridge skeleton)

Phase 4a deliverable (no filter rendering yet — это Phase 4b).
End-to-end pipeline: HA/HTTP/MQTT → controller → ZMQ → FFmpeg (logged).

Modules:
- overlays.py — 7 discriminated union types через pydantic:
  rect, text, icon, image, dim, graph, chat. Normalized coords (0.0-1.0),
  optional cell binding, z_order, opacity, visible.
- state.py — overlay storage per instance (CRUD: add/remove/update/get/clear)
- dispatch.py — overlay.add/remove/clear actions:
  - parses JSON payload в Overlay через TypeAdapter
  - serializes to ZMQ string: "<id> <type> <full-json>"
  - sends via FFmpeg process_command (filter will парсить в Phase 4b)
  - updates state + publishes events (overlay_added, overlay_removed, overlays_cleared)
- http_api.py — REST endpoints:
  - POST /overlay/{inst}/add (body = Overlay JSON, returns id)
  - GET /overlay/{inst} — list all
  - DELETE /overlay/{inst}/{id} — single
  - DELETE /overlay/{inst} — clear all
  - PATCH /overlay/{inst}/{id} — update
- mqtt_loop.py — already subscribes cuda_grid/cmd/<inst>/+/+; teper handles
  overlay/add (JSON payload), overlay/remove (id), overlay/clear
- frigate_bridge.py — FrigateBridge skeleton:
  - subscribe frigate/+/motion + frigate/events
  - mapping camera_name → target_instance + cell index
  - Phase 4a: log received events (rendering в Phase 4b)
- config.py — frigate: optional section
- examples/controller.yaml — frigate mappings для 4 наших камер

State management:
- ControllerState.add/remove/update/get/clear_overlay (asyncio.Lock guarded)
- InstanceState.overlays: dict[str, Overlay]
- IDs generated via uuid4()[:8]

Phase 4a limitations:
- Filter side ничего не рендерит (just logs ZMQ commands)
- Frigate bridge принимает events но не auto-generates overlays
- HA Discovery не имеет overlay-specific entities (overlays через REST API)

Phase 4b: filter-side AVFrame side data + CUDA kernels (rect first, NPP-based,
потом text via freetype atlas, потом icon sprite blit).

2026-05-19 22:03:20 +01:00

controller

controller: Phase 4a — overlay infrastructure (data models + API + Frigate bridge skeleton)

2026-05-19 22:03:20 +01:00

docs

initial: README + design document (architect-reviewed)

2026-05-19 20:36:47 +01:00

.gitignore

Initial commit

2026-05-19 20:35:31 +01:00

LICENSE

Initial commit

2026-05-19 20:35:31 +01:00

README.md

readme: project overview + architecture diagram + phase table

2026-05-19 20:37:41 +01:00

README.md

vf-cuda-grid

GPU-native video grid composer — FFmpeg filter + control-plane sidecar для multi-camera mosaic composition с runtime layout switching, rich overlays и интеграцией с Home Assistant / Frigate / любой MQTT-системой.

Статус: дизайн зафиксирован — см. docs/design.md. Имплементация не начата.

Что это

vf_cuda_grid — FFmpeg video filter (libavfilter/vf_cuda_grid.c), out-of-tree patch для FFmpeg n7.1+. Принимает N CUDA-frames на входе, выдаёт один composed frame с N-cell layout. End-to-end CUDA (без CPU round-trip).
cuda-grid-controller — Python sidecar process (FastAPI + asyncio + pyzmq + aiomqtt). Принимает commands через ZeroMQ / MQTT / HTTP REST / HA Discovery, транслирует в FFmpeg process_command через zmq filter. Также publishes events наружу — для bidirectional integration с Home Assistant и Node-RED.

Ключевые возможности (planned)

✅ Multi-input — N CUDA-frames в filter ([in0][in1][in2]...cuda_grid=...)
✅ Multi-output — несколько filter instances (разные TV, public stream, private)
✅ Per-cell camera binding — cells привязаны к camera_id, не к input slot
✅ Layout DSL — predefined templates + runtime-created custom layouts
✅ Runtime layout switching без teardown filter graph
✅ Rich overlays — rectangles, text, icons, image overlays, dim/darken, графики, чаты с alpha blending. Всё GPU-side.
✅ Control plane — ZeroMQ, MQTT, HTTP REST, HA Discovery
✅ Bidirectional events — controller publishes layout_switched, cell_camera_changed, fps_drop, overlay_added etc.
✅ Audio orchestration — controller координирует video grid + audio (amix, sidechaincompress через стандартные FFmpeg filters)
✅ Privacy filtering — на public screen overlays можно выборочно отключить

Use cases

NVR с несколькими TV — каждый TV имеет свой layout (например full-house quad в гостиной + door-focus в холле + private-only mosaic в спальне)
Public stream + private monitor — на public stream без LPR-text и без privacy-camera; на private — всё
Domofon scenario — звонок в дверь → controller переключает layout на focus-door + ducked music + door audio focused
Frigate detection visualization — overlay bounding boxes, LPR numbers, face names с alpha blending поверх video grid

Architecture (high level)

        ┌────────── Home Assistant / Node-RED / custom apps ──────────┐
        │            (MQTT / ZeroMQ / HTTP REST / SSE)                │
        └─────────────────────────────┬───────────────────────────────┘
                                      │
                          ┌───────────▼────────────┐
                          │   cuda-grid-controller │  (Python sidecar)
                          │  (FastAPI + pyzmq +    │
                          │   aiomqtt + pycairo)   │
                          └───────────┬────────────┘
                                      │  process_command
                                      │  via FFmpeg's zmq filter
                                      ▼
            ┌─────────────────── FFmpeg ───────────────────┐
   cuframes://cam1 ─┐                                       │
   cuframes://cam2 ─┼─►  vf_cuda_grid  ──►  h264_nvenc  ──► output
   cuframes://cam3 ─┤    (instance N)                       │
   cuframes://camN ─┘    runtime sendcmd via zmq            │
            └───────────────────────────────────────────────┘

Полная архитектура, layout DSL, CUDA kernels, overlay system, multi-instance behaviour, audio orchestration, 6 phase implementation plan и migration path для cctv-processor — см. docs/design.md.

Связано

gx/cuframes (repo) — zero-copy CUDA IPC frame sharing. Frames из cuframes — основной input source для vf_cuda_grid в нашей экосистеме.
gx/ffmpeg-patched — FFmpeg fork с cuframes:// demuxer. vf_cuda_grid будет добавлен сюда (или в отдельный patched fork).
gx/cctv #22 — performance investigation для текущего custom C++ GridComposer. vf_cuda_grid v1.0 закрывает Phase 4 (end-to-end GPU).
gx/cctv #24 — initial MQTT plugin design для cctv-processor. Superseded — controller cuda-grid-controller покрывает 60-70% этого scope, см. design §14.

Status / Phases

См. epic issue #1 для tracking прогресса.

Phase	Что	Status
1	MVP filter — fixed quad layout, 4 CUDA inputs → 1 output	planned
2	Dynamic layouts (DSL) + scaling per cell	planned
3	`cuda-grid-controller` sidecar (ZMQ + MQTT + HTTP + HA Discovery)	planned
4	Overlay primitives (rect/text/icon) — CUDA rendering	planned
5	Image overlays, dim areas, graphs/charts (Cairo→texture pipeline)	planned
6	Audio orchestration use cases в controller	planned

License

LGPL-2.1+ — для совместимости с FFmpeg LGPL builds и cuframes.

Контакты

Issues и discussions — в этом репо. Project maintainer — gx@goldix.org.

Description

GPU-native video grid composer — FFmpeg filter + control-plane sidecar для multi-camera mosaic compositions с runtime layout switching, rich overlays и MQTT/ZeroMQ/HTTP/HA Discovery integration

Readme LGPL-2.1 452 KiB

README.md Unescape Escape

vf-cuda-grid

Что это

Ключевые возможности (planned)

Use cases

Architecture (high level)

Связано

Status / Phases

License

Контакты

README.md