cuframes-composer/docs/en/user.md

cfc-grid — user guide

Audience: installation admin. Configures cameras, layouts, overlays, watches the RTSP stream on TV or browser. Does not edit C++ code.

Developer extending Cell/Decoration/widget types → see developer.md.

1. What this is

cfc-grid is a CUDA compositor that combines N cameras into a single RTSP stream rtsp://192.168.88.23:554/cfc-grid (1920×1080, H.264, NVENC). Layout is selected automatically by motion (from Frigate) or manually via ONVIF presets on the TV.

In addition to camera frames, the following are drawn on top:

• Borders — grey 2 px borders around each cell
• Labels — name prio=N caption in the corner of each camera
• Detection boxes — object rectangles from Frigate, tracking camera position when layout changes
• MQTT overlays — text fields bound to MQTT topics (temperature, statuses, chats)

2. Architecture in one sentence

cuframes-pub-* (per camera)
       ↓ shared VMM
   cfc-grid (composer) ── ZMQ control ──┐
       ↓ pipe (H.264)                   │
   cfc-grid-ffmpeg (relay) ─→ mediamtx ─┴─→ TV / VLC / Frigate / ...
                                ↑
                          ONVIF discovery from cctv-onvif

Frames go through cuframes zero-copy (a single VMM buffer shared between publisher and composer). The composer takes the NV12 surface, resizes/blits into its output, adds decorations, hands off to NVENC, NVENC writes H.264 into a pipe, cfc-grid-ffmpeg re-muxes the pipe → RTSP push to mediamtx.

3. Motion mode — the main operating mode

3.1 What happens

Each frame the composer:

1. Reads last_motion_ms for each camera (updated from Frigate MQTT frigate/events)
2. Treats as "active" any camera with (now - last_motion_ms) < motion_ttl_ms (default 45 seconds)
3. Sorts active by priority (integer; higher = more important)
4. Selects template from templates.json by best-fit rule: minimal template with nb_camera_cells >= active_count
5. If template has more camera-cells than active — extra ones are filled with remaining drawable cameras from pool (by priority)
6. Applies asymmetric hysteresis: growth in active count switches layout immediately, shrinkage waits 3 seconds (to avoid flicker)

3.2 What "drawable" means

A camera is excluded from pool if its cfc_source_state_t = CONNECTING, DISCONNECTED or DEAD (cuframes publisher silent for longer than dead_threshold_ms, default 5 seconds).

STALE (frames arrive infrequently) — counts, last available frame is drawn.

3.3 Manual override via PTZ

In the TV, ONVIF PTZ presets list template names (tpl_1, tpl_3, tpl_4, ..., tpl_16). Pressing GotoPreset or movement keys:

• Applies the chosen layout immediately
• Freezes motion-mode for 60 seconds
• After expiry — returns to auto mode

ContinuousMove (arrows): pan/tilt cycles through preset list, zoom-in = tpl_1 (full screen), zoom-out = tpl_16 (4×4 grid).

4. templates.json — screen layout

4.1 Schema

{
  "version": 1,
  "grid_cols": 8,
  "grid_rows": 8,
  "templates": [
    {
      "name": "tpl_N",
      "_desc": "Description",
      "priority": 0,
      "cells": [
        {
          "col": 0, "row": 0,
          "cs": 4, "rs": 4,
          "role": "camera",
          "order": 0
        },
        {
          "col": 6, "row": 0,
          "cs": 2, "rs": 6,
          "role": "widget",
          "widget": "temp_chart"
        }
      ]
    }
  ]
}

Grid: 8×8 microcells (240×135 px each = 16:9 on 1920×1080 output). Any N×N square of microcells is also 16:9.

Field	Value
col, row	Top-left corner of cell in microcells (0..7)
cs, rs	Size in microcells
role	camera or widget
order	For camera: placement order for active cameras (0 = main, usually the largest cell)
widget	For widget: placeholder name (caption text)

4.2 Best-fit selection

The composer selects a template for current active count:

candidates = [t for t in templates if t.nb_camera_cells >= n_active]
pick = min(candidates, key=lambda t: t.nb_camera_cells - n_active)
# On tie: higher priority wins

If active count exceeds the largest template's cells — the largest is taken, extra cameras are dropped (lowest priority).

4.3 Built-in templates

By default 9 templates in /opt/templates.json:

Name	Cells	Description
tpl_1	1 cam	One camera fullscreen
tpl_3	3 cam + 2 widgets	Main 1440×810 + 2 previews + 2 widget areas
tpl_4	4 cam	Quad 2×2, 960×540 each
tpl_5	5 cam + 1 widget	Main + 4 previews stacked right + widget bottom
tpl_6	6 cam + 1 widget	Main + 3 right + 2 bottom + widget
tpl_7	7 cam + 1 widget	Main + 3 right + 3 bottom + widget
tpl_8	8 cam (1+3+4)	Main + 3 right + 4 bottom row
tpl_9	9 cam + 2 widgets	3×3 mains + widget strips
tpl_16	16 cam	4×4 grid, 480×270 each

Details — see docker/templates.json in the repo.

4.4 Adding your own template

1. Open docker/templates.json (or mounted override)
2. Add a block to "templates": [...] per schema above
3. Restart cfc-grid (or invoke ZMQ when hot-reload is available — Phase 12)

4.5 Coordinate math

Each microcell = 1920/8 = 240 px wide, 1080/8 = 135 px tall (16:9). Cell {col=2, row=4, cs=4, rs=2}:

• pixel x = 2 * 240 = 480
• pixel y = 4 * 135 = 540
• pixel w = 4 * 240 = 960
• pixel h = 2 * 135 = 270

Aspect cell = cs/rs * 16/9. For 16:9 cells: cs == rs.

5. mqtt_overlays.json — text overlays from MQTT

5.1 Schema

{
  "version": 1,
  "overlays": [
    {
      "id": "temp_outside",
      "topic": "zigbee2mqtt/Outdoor temp sensor",
      "json_field": "temperature",
      "format": "%+.1f°C",
      "anchor": "right-bottom",
      "margin_x": 32, "margin_y": 24,
      "pixel_size": 32,
      "color": [255, 255, 255],
      "alpha": 230,
      "bg_alpha": 160,
      "bg_y": 16, "bg_u": 128, "bg_v": 128,
      "bg_pad": 10,
      "placeholder": "—",
      "font_path": "/fonts/DejaVuSans-Bold.ttf"
    }
  ]
}

Field	Description
id	Unique overlay identifier (used by ZMQ for lookup)
topic	MQTT topic to subscribe (via cctv-mosquitto)
json_field	If payload is JSON — field name to extract; empty = raw payload as string
format	printf-style for the formatted value (e.g. "%+.1f°C", "%s")
anchor	Positioning anchor: right-bottom, right-top, left-bottom, left-top, center
margin_x, margin_y	Offset from nearest screen edge (px)
pixel_size	Font size in pixels
color	RGB text color
alpha	Overall text opacity (0..255)
bg_alpha	Background opacity (0..255); 0 = no background
bg_y, bg_u, bg_v	BT.709 limited-range background color; default black
bg_pad	Background padding around text (px)
placeholder	What to show before first MQTT message; empty = "—"
font_path	Path to font (.ttf/.otf) inside container

5.2 Supported symbols

Font DejaVuSans-Bold.ttf — standard from fonts-dejavu package. Covers Basic Multilingual Plane (latin, cyrillic, basic symbols), including:

• ❯ (U+276F), ✎ (U+270E), ➤ (U+27A4), → (U+2192)
• ★ (U+2605), ▶ (U+25B6), ✉ (U+2709)

Emoji from Supplementary Multilingual Plane (>U+10000) — e.g. 🗣 (U+1F5E3), 🤖 (U+1F916), 💬 (U+1F4AC) — are not rendered: font lacks those glyphs. Placeholder square is drawn instead.

To add color emoji — bind Noto Color Emoji and extend renderer for COLR/CPAL/SBIX (see developer.md).

5.3 Examples

Plain number: payload = "23.5" (raw), json_field: "", format: "%s°"

JSON with field: payload = {"temperature": 23.5, "humidity": 45}, json_field: "temperature", format: "%+.1f°C"

Multiple overlays stacked top-right: first with margin_y: 24, second margin_y: 72, third margin_y: 120.

5.4 How to add

1. Open docker/mqtt_overlays.json (or mounted override)
2. Add block to "overlays": [...] array
3. Restart cfc-grid

Hot-reload via ZMQ — Phase 12 (reload_overlays verb).

6. Composer CLI flags

In compose override docker/cctv/cuframes-composer/docker-compose.override.yml:

command:
  - "--out=/out/grid.h264"          # named pipe for ffmpeg-relay
  - "--fps=25"
  - "--bitrate=6000"                # kbps
  - "--width=1920"
  - "--height=1080"
  - "--intra-refresh"               # instead of IDR-bursts (low-latency)
  - "--control=tcp://0.0.0.0:5599"  # ZMQ control plane
  - "--mqtt=cctv-mosquitto:1883"    # MQTT for health publishing
  - "--mqtt-instance=cfc-grid"
  - "--mqtt-user=composer"
  - "--mqtt-pass=${COMPOSER_MQTT_PASSWORD}"

  # Sources (cameras) — repeatable
  - "--source=cam-parking,frigate=parking_overview,priority=100,zones=parking_zone:canopy:private_area"
  - "--source=cam-back_yard,frigate=back_yard,priority=70"
  # ...

  - "--motion-mode"                  # enable auto-layout
  - "--motion-ttl=45000"             # ms

  - "--templates=/opt/templates.json"
  - "--mqtt-overlays=/opt/mqtt_overlays.json"

  # Frigate motion-driver and detection boxes
  - "--frigate-mqtt=cctv-mosquitto:1883"
  - "--frigate-topic=frigate/events"
  - "--detection-cell=parking,parking_overview,0,0,960,540,640,480,parking_zone:canopy:private_area"

6.1 --source syntax

--source=<cuframes_key>,frigate=<camera>,priority=<N>[,zones=<z1>:<z2>:...]

• cuframes_key — name of cuframes publisher (e.g. cam-parking)
• frigate=NAME — camera name in Frigate (for motion event matching)
• priority=N — integer, higher = more important
• zones=... — optional whitelist of Frigate zones; motion counts only if event.current_zones intersects the list (filters street flood)

6.2 --detection-cell syntax

--detection-cell=<key>,<frigate_camera>,<x>,<y>,<w>,<h>,<detect_w>,<detect_h>[,zones]

• key — arbitrary overlay identifier for logs
• frigate_camera — Frigate name (for event.camera matching)
• x,y,w,h — initial geometry (composer recalculates dynamically)
• detect_w,detect_h — Frigate detector resolution (e.g. 640×480)
• zones — bbox whitelist

7. ZMQ control plane

Default endpoint: tcp://192.168.88.23:5599. All verbs — JSON request/reply.

7.1 Verb list

Command	Parameters	What it does
ping	—	Health check
health	—	{total, active, stale, dead} over pool
set_text	id, text, r, g, b, x, y, visible	Update text overlay (for CLI --text=...)
set_visible	id, visible	Hide/show overlay
list_overlays	—	List of overlays
set_layout	name	Apply named template (manual override 60s in motion-mode)
list_layouts	—	List of templates with cells
get_layout	—	Name of current template
set_motion_mode	on, ttl_ms	Toggle motion mode
get_motion_mode	—	Motion mode state
get_template	name	Full template JSON
reload_templates	path?	Reload templates from file

7.2 Example

python3 <<EOF
import zmq, json
s = zmq.Context().socket(zmq.REQ)
s.connect("tcp://192.168.88.23:5599")
s.send_json({"cmd": "list_layouts"})
print(json.dumps(s.recv_json(), indent=2, ensure_ascii=False))
EOF

Plain nc does not work — REP socket expects ZMQ wire-protocol. Use zmq Python/Go/JS or mosquitto_pub via RPC-bridge (Phase 12).

8. ONVIF and PTZ

cctv-onvif service binds to host network, responds to WS-Discovery multicast (239.255.255.250:3702) and SOAP requests over HTTP :8085.

8.1 Adding to TV

In the client (TV / IP-CamViewer / Synology / Frigate):

• ONVIF host: 192.168.88.23
• Port: 8085
• User / Password: empty (auth not configured)

WS-Discovery in LAN 192.168.88.0/24 finds device cfc-grid (Goldix). RTSP URL is automatic — rtsp://192.168.88.23:554/cfc-grid.

8.2 PTZ presets

List (GetPresets): tpl_1, tpl_3, tpl_4, tpl_5, tpl_6, tpl_7, tpl_8, tpl_9, tpl_16.

GotoPreset(name) → composer applies template + freezes motion-mode for 60 seconds → auto-return.

8.3 PTZ movement (ContinuousMove)

Command	Action
Pan right / Tilt down	Next template in list
Pan left / Tilt up	Previous
Zoom in (+)	tpl_1 (full screen)
Zoom out (−)	tpl_16 (4×4 grid)

9. Where to view RTSP

Method	URL
VLC / mpv / ffplay	rtsp://192.168.88.23:554/cfc-grid
Browser (HLS)	http://192.168.88.23:8888/cfc-grid
WebRTC	http://192.168.88.23:8889/cfc-grid
OBS / FFmpeg input	rtsp://192.168.88.23:554/cfc-grid
ONVIF clients	via WS-Discovery (see §8)

10. Known limitations

• Color emoji not rendered (needs Noto Color Emoji + COLR/CPAL support in text renderer — Phase 12)
• Hot-reload mqtt_overlays.json — no ZMQ verb, requires cfc-grid restart
• Per-overlay broker — all MQTT overlays use the common broker (set as --mqtt); subscribing to a foreign broker separately — not yet
• Widget rendering — placeholder (dark rect + label), real widgets (graph, chat) — Phase 12+

11. FAQ

Q: TV shows old layout names (quad, dual_horizontal). What to do?

A: TV cached ONVIF presets. In the client, delete the camera and add again — it will re-read GetPresets with current names.

Q: Parking camera is DEAD but logs show active=3. Why?

A: cfc_composer_get_health shows pool-wide state, but motion-active counts by last_motion_ms independently of source state. DEAD is excluded in is_camera_drawable() inside compose_motion_relayout.

Q: Pressed PTZ on TV, layout switched, but reverted after a minute.

A: This is by design — set_layout in motion-mode freezes auto for 60 seconds (manual_override_duration_ms). To pin layout permanently — disable motion-mode entirely via ZMQ:

{"cmd": "set_motion_mode", "on": 0}

Q: I want text overlay backgrounds in different colors.

A: bg_y/bg_u/bg_v fields accept BT.709 limited-range. For red — Y≈80, U≈90, V≈240. For cyan — Y≈170, U≈170, V≈100. Calculator: https://www.rapidtables.com/convert/color/rgb-to-yuv.html (use BT.709 limited).

Q: With motion on 5 cameras, the layout doesn't change, stays at quad.

A: Check docker logs cfc-grid | grep "loaded N templates" — must be ≥5 (should include tpl_5..tpl_8, tpl_9, tpl_16). If not — templates.json didn't load (check syntax via jq or python3 -m json.tool).

Q: Frigate bbox is drawn on the wrong camera.

A: Check --detection-cell — frigate_camera must match event.after.camera. Composer binds detbox-overlay to pool-entry via frigate_camera (see cfc_composer::pool::by_frigate_camera).

12. Next

• developer.md — internals, extension
• operations.md — build, deploy, troubleshooting
• repo README: brief overview