cuframes-composer/docs/en/developer.md

cfc-grid — developer guide

Audience: developer who edits composer C++ code, adds new Cell/Decoration/overlay types, or changes auto-layout logic.

If you're a user — see user.md. For deploy/troubleshooting see operations.md.

1. Architecture (40000-foot view)

                ┌──────────────────────────────────────────────┐
                │            cfc::Composer (C++)              │
                │                                              │
                │  ┌──────────┐ ┌──────────┐ ┌──────────────┐  │
                │  │SourcePool│ │  Layout  │ │OutputSurface │  │
                │  │ (pool of │ │ vector<  │ │  CudaBuffer  │  │
                │  │  cfc_    │ │ unique_  │ │   (VMM NV12) │  │
                │  │source_t*)│ │ ptr<Cell>│ │              │  │
                │  └──────────┘ └──────────┘ └──────────────┘  │
                │       ↓             ↓              ↑          │
                │   motion_pulse   apply()         NVENC        │
                └──────────────────────────────────────────────┘
                       ↑                              ↓
              extern "C" ABI shim                  H.264 pipe
              (composer_c_api.cpp,                    ↓
               layouts_c_api.cpp,                ffmpeg-relay
               mqtt_overlay_c_api.cpp)                ↓
                       ↑                          mediamtx
              ┌────────┴────────┐                     ↓
              │                 │                     ↓
        grid_record.c       control.c              clients
        (C, CLI parsing)    (ZMQ verbs)
              ↑                 ↑
              │                 │
         frigate_mqtt.c   health.c, audio.c, writer.c, source.c
         (Frigate events)  (C-only modules)

Principles:

• C++17 host-side, CUDA kernels remain on C
• Zero-copy: output VMM buffer is passed as NV12Ref to all cells/decorations (read-write reference, not copy)
• Coexistence: C modules (source.c, nvenc.c, frigate_mqtt.c, health.c, writer.c, audio.c, overlay.c) are kept, accessed via extern "C". Only composer.c and layouts.c were removed (replaced by C++ + ABI shim).
• grid_record.c (CLI entry point) remains in C, uses public cfc_composer_* API unchanged

2. Source tree

include/cuframes_composer/
├── *.h                            # Public C API (extern "C")
│   ├── composer.h                 # cfc_composer_* ABI
│   ├── layouts.h                  # cfc_layout_* ABI
│   ├── overlay.h                  # cfc_overlay_* (text/png/border/detbox)
│   ├── source.h, nvenc.h, frigate_mqtt.h, ...
│   └── ...
└── cpp/                           # C++ headers — public classes
    ├── cuda_raii.hpp              # CudaBuffer, CudaStream (RAII)
    ├── types.hpp                  # Rect, NV12Ref
    ├── cell.hpp                   # abstract Cell + draw()
    ├── camera_cell.hpp            # CameraCell : Cell
    ├── widget_cell.hpp            # WidgetCell : Cell
    ├── blank_cell.hpp             # BlankCell : Cell
    ├── decoration.hpp             # abstract Decoration
    ├── border_decoration.hpp      # BorderDecoration
    ├── label_decoration.hpp       # LabelDecoration (FreeType internal)
    ├── template.hpp               # LayoutTemplate, CellTemplate, kGridCols/Rows
    ├── template_loader.hpp        # JSON parser, current_templates()
    ├── source_pool.hpp            # SourcePool, PoolEntry
    ├── layout.hpp                 # Layout
    ├── composer.hpp               # Composer (main class)
    └── mqtt_overlay.hpp           # MqttOverlayItem + Manager

src/
├── *.c                            # C modules (source/nvenc/health/...)
├── overlay.c                      # All cfc_overlay_t types
├── frigate_mqtt.c                 # Subscribe + motion_pulse + zone-filter
├── cugrid/cugrid.cu               # CUDA kernels (resize, fill, blit)
└── cpp/                           # C++ implementations
    ├── camera_cell.cpp, widget_cell.cpp, blank_cell.cpp
    ├── border_decoration.cpp, label_decoration.cpp
    ├── source_pool.cpp
    ├── template_loader.cpp
    ├── layout.cpp
    ├── composer.cpp
    ├── composer_c_api.cpp         # extern "C" shim for composer
    ├── layouts_c_api.cpp          # extern "C" shim for layouts
    ├── mqtt_overlay.cpp
    └── mqtt_overlay_c_api.cpp

examples/
├── simple_record.c                # 1-source → H.264 file
├── grid_record.c                  # Main CLI entry point (C)
└── grid_record_cpp.cpp            # C++ smoke (uses cfc::Composer directly)

3. Single frame lifecycle

// In compose loop (grid_record.c, via cfc_composer_compose):
NV12Ref ref = composer.compose_frame();
//   1. maybe_relayout():  motion-mode? best-fit template; apply Layout
//   2. compose_clear():    fill output buffer BT.709 black
//   3. Layout::render():   for each cell: draw_content() + decorations[]
//   4. for each overlay:   sync detbox geom + cfc_overlay_draw()
// cudaStreamSynchronize(0);
// nvenc.encode(ref.y_ptr, ref.pitch_y, ref.uv_ptr, ref.pitch_uv, ...);

All operations execute on CUDA default stream. Zero-copy: ref.y_ptr / ref.uv_ptr are the same CUdeviceptrs inside composer.output_ (RAII CudaBuffer).

4. Cell — hierarchy and extension

4.1 Abstraction

class Cell {
public:
    explicit Cell(const Rect& geom);
    virtual ~Cell() = default;
    Cell(const Cell&) = delete;

    const Rect& geometry() const noexcept;
    void set_geometry(const Rect& r) noexcept;
    void add_decoration(std::unique_ptr<Decoration>);

    void draw(CUstream stream, NV12Ref& dst);   // public — calls draw_content() + decorations

protected:
    virtual void draw_content(CUstream stream, NV12Ref& dst) = 0;
    Rect geom_;
    std::vector<std::unique_ptr<Decoration>> decorations_;
};

Cell::draw() is final (non-virtual — NVI pattern):

void Cell::draw(CUstream stream, NV12Ref& dst) {
    if (geom_.empty()) return;
    draw_content(stream, dst);                 // subclass renders content
    for (auto& d : decorations_) {
        d->draw(stream, dst, geom_);            // overlay decorations on top
    }
}

4.2 Existing subclasses

Class	draw_content implementation
CameraCell	cfc_source_get_latest() + cfc_cugrid_resize_nv12
WidgetCell	cfc_cugrid_fill_nv12 dark grey Y=40 (placeholder)
BlankCell	cfc_cugrid_fill_nv12 BT.709 black Y=16

4.3 How to add a new Cell type

Example: GraphCell — draws scrolling chart from MQTT subscription.

1. Create include/cuframes_composer/cpp/graph_cell.hpp:

#ifndef CUFRAMES_COMPOSER_CPP_GRAPH_CELL_HPP
#define CUFRAMES_COMPOSER_CPP_GRAPH_CELL_HPP
#include "cell.hpp"
#include <deque>

namespace cfc {

class GraphCell : public Cell {
public:
    GraphCell(const Rect& geom, std::size_t max_points = 60)
        : Cell(geom), max_points_(max_points) {}

    void push_value(double v) {
        if (values_.size() >= max_points_) values_.pop_front();
        values_.push_back(v);
    }

protected:
    void draw_content(CUstream stream, NV12Ref& dst) override;

private:
    std::deque<double> values_;
    std::size_t max_points_;
};

} // namespace cfc
#endif

2. Implementation src/cpp/graph_cell.cpp:

#include "../../include/cuframes_composer/cpp/graph_cell.hpp"
#include "../../include/cuframes_composer/cugrid.h"

namespace cfc {

void GraphCell::draw_content(CUstream stream, NV12Ref& dst) {
    if (geom_.empty() || values_.empty()) return;

    // 1. BG fill
    cfc_cugrid_fill_nv12(stream, dst.y_ptr, dst.pitch_y, dst.uv_ptr, dst.pitch_uv,
                          geom_.x, geom_.y, geom_.w, geom_.h,
                          /*Y*/ 30, 128, 128, 255);

    // 2. Compute min/max
    double mn = *std::min_element(values_.begin(), values_.end());
    double mx = *std::max_element(values_.begin(), values_.end());
    if (mx == mn) mx = mn + 1.0;

    // 3. Render as thin filled rects (lazy, no bitmap kernel)
    int n = static_cast<int>(values_.size());
    int dx = geom_.w / n;
    for (int i = 0; i < n; i++) {
        double norm = (values_[i] - mn) / (mx - mn);
        int y_px = geom_.y + geom_.h - 2 - (int)(norm * (geom_.h - 4));
        cfc_cugrid_fill_nv12(stream, dst.y_ptr, dst.pitch_y, dst.uv_ptr, dst.pitch_uv,
                              geom_.x + i * dx, y_px, dx & ~1, 2,
                              /*Y*/ 235, 128, 64, 255);  // limited-range bright
    }
}

} // namespace cfc

3. Add to src/CMakeLists.txt:

set(COMPOSER_SOURCES_CPP
    ...
    cpp/graph_cell.cpp
)

4. Use in Layout::apply():

// In layout.cpp, widget cells handling:
if (wt->widget == "graph_temp") {
    cells_.push_back(std::make_unique<GraphCell>(r, /*max_points=*/120));
} else {
    cells_.push_back(std::make_unique<WidgetCell>(r, wt->widget));
}

5. Wire MQTT → GraphCell::push_value() — either via MqttOverlayManager extension, or new API cfc::Composer::register_graph_feed().

5. Decoration — composition in Cell

5.1 Abstraction

class Decoration {
public:
    virtual ~Decoration() = default;
    virtual void draw(CUstream stream, NV12Ref& dst, const Rect& parent_rect) = 0;
};

Decoration knows only parent cell's pixel-rect — positions relative to it. Has no access to Layout/Composer.

5.2 Existing subclasses

• LabelDecoration — FreeType atlas + cfc_cugrid_blit_rgba_nv12. Persistent VRAM (rebuild on set_text). Supports background fill (via cfc_overlay_text_config_t.bg_* parameters).
• BorderDecoration — 4 calls of cfc_cugrid_fill_nv12 (top, bottom, left, right rects by thickness).

5.3 How to add a new Decoration type

Example: BadgeDecoration — corner icon (e.g. red "recording" dot).

1. Header badge_decoration.hpp:

class BadgeDecoration : public Decoration {
public:
    struct Style {
        int corner = 0;          // 0=top-left, 1=top-right, 2=bottom-right, 3=bottom-left
        int size = 12;
        int margin = 8;
        int color_y = 80, color_u = 90, color_v = 240;  // red-ish
        int alpha = 240;
        bool visible = true;
    };
    explicit BadgeDecoration(const Style& s) : style_(s) {}
    void set_visible(bool v) noexcept { style_.visible = v; }
    void draw(CUstream stream, NV12Ref& dst, const Rect& p) override;
private:
    Style style_;
};

2. Implementation:

void BadgeDecoration::draw(CUstream s, NV12Ref& dst, const Rect& p) {
    if (!style_.visible || style_.alpha <= 0) return;
    int x, y;
    switch (style_.corner) {
        case 0: x = p.x + style_.margin; y = p.y + style_.margin; break;
        case 1: x = p.x + p.w - style_.size - style_.margin; y = p.y + style_.margin; break;
        case 2: x = p.x + p.w - style_.size - style_.margin; y = p.y + p.h - style_.size - style_.margin; break;
        case 3: x = p.x + style_.margin; y = p.y + p.h - style_.size - style_.margin; break;
    }
    x &= ~1; y &= ~1;
    cfc_cugrid_fill_nv12(s, dst.y_ptr, dst.pitch_y, dst.uv_ptr, dst.pitch_uv,
                          x, y, style_.size, style_.size,
                          style_.color_y, style_.color_u, style_.color_v, style_.alpha);
}

3. Wire in Layout::apply():

if (cc->source()->state() == CFC_SOURCE_STALE) {
    BadgeDecoration::Style bs;
    bs.corner = 1; bs.color_y = 180; bs.color_v = 100;  // yellow
    cell->add_decoration(std::make_unique<BadgeDecoration>(bs));
}

6. SourcePool and motion state

struct PoolEntry {
    std::string cuframes_key;
    std::string frigate_camera;
    int priority;
    cfc_source_t* source;
    std::atomic<int64_t> last_motion_ms;
    std::vector<std::string> required_zones;

    cfc_source_state_t state() const;     // calls cfc_source_get_latest
    bool drawable() const;                 // ACTIVE || STALE
};

class SourcePool {
public:
    int add(key, frigate_camera, priority, zones, SubscribeOpts);
    PoolEntry* by_key(const std::string&);
    PoolEntry* by_frigate_camera(const std::string&);
    template<typename F> void for_each(F&&);
    void motion_pulse(const std::string& frigate_camera,
                       const std::vector<std::string>& current_zones);
};

motion_pulse is called from frigate_mqtt.c for each event. If required_zones is non-empty — match by intersection with event.current_zones, otherwise accept all.

7. Auto-layout algorithms

See cfc::Composer::maybe_relayout() (src/cpp/composer.cpp).

7.1 Best-fit selection

const LayoutTemplate* Composer::pick_best_fit(int need) const {
    const auto& reg = current_templates();
    const LayoutTemplate* best = nullptr;
    int best_waste = -1, best_prio = -1;
    for (auto& t : reg) {
        int n = t.nb_camera_cells();
        if (n < need) continue;
        int waste = n - need;
        if (!best || waste < best_waste ||
            (waste == best_waste && t.priority > best_prio)) {
            best = &t; best_waste = waste; best_prio = t.priority;
        }
    }
    if (best) return best;
    // overflow → largest
    for (auto& t : reg) {
        if (!best || t.nb_camera_cells() > best->nb_camera_cells()) best = &t;
    }
    return best;
}

7.2 Collect active

std::vector<PoolEntry*> Composer::collect_active() const {
    std::vector<PoolEntry*> active;
    int64_t now = now_ms_mono();
    pool_.for_each([&](PoolEntry& e) {
        if (!e.drawable()) return;            // DEAD/CONNECTING — skip
        int64_t last = e.last_motion_ms.load();
        if (last == 0) return;                 // never had motion
        if (now - last > cfg_.motion_ttl_ms) return;  // TTL expired
        active.push_back(&e);
    });
    // idle fallback: top-priority drawable
    if (active.empty()) {
        PoolEntry* best = nullptr;
        pool_.for_each([&](PoolEntry& e) {
            if (!e.drawable()) return;
            if (!best || e.priority > best->priority) best = &e;
        });
        if (best) active.push_back(best);
    }
    std::sort(active.begin(), active.end(),
              [](PoolEntry* a, PoolEntry* b) { return a->priority > b->priority; });
    return active;
}

7.3 Asymmetric hysteresis

Signature template_name + "|" + sorted_keys is remembered. If new set ⊇ committed (growth) → commit immediately. Otherwise wait shrink_hysteresis_ms (default 3000) → commit.

bool is_grow = std::includes(nkeys.begin(), nkeys.end(),
                              ckeys.begin(), ckeys.end());
if (is_grow && nkeys.size() == ckeys.size()) is_grow = false;
if (is_grow) { /* commit */ } else {
    if (sig != pending_signature_) {
        pending_signature_ = sig;
        pending_first_seen_ms_ = now;
    }
    if (now - pending_first_seen_ms_ < cfg_.shrink_hysteresis_ms) return;
    /* commit */
}

7.4 Fill free cells

After picking template and capping by nb_camera_cells:

if (active.size() < cap) {
    std::vector<PoolEntry*> extras;
    pool_.for_each([&](PoolEntry& e) {
        if (!e.drawable()) return;
        if (e ∈ already_active) return;
        extras.push_back(&e);
    });
    std::sort(extras.begin(), extras.end(), priority_desc);
    while (active.size() < cap && !extras.empty()) {
        active.push_back(extras.front());
        extras.erase(extras.begin());
    }
}

7.5 Manual PTZ override

Composer::set_layout(name) in motion-mode:

manual_override_until_ms_ = now + manual_override_duration_ms_;  // default 60s

maybe_relayout() skips work while now < manual_override_until_ms_. After expiry — committed_signature_.clear() → forced relayout.

8. extern "C" ABI shim

composer_c_api.cpp — thin wrapper:

extern "C" int cfc_composer_create(const cfc_composer_config_t* cfg,
                                     cfc_composer_t** out)
{
    cfc::ComposerConfig cpp_cfg = {/* ... */};
    auto* comp = new (std::nothrow) cfc::Composer(cpp_cfg);
    if (!comp->ok()) { delete comp; return -1; }
    // manual_cells from cfg->cells → set_manual_cells()
    *out = reinterpret_cast<cfc_composer_t*>(comp);
    return 0;
}

cfc_composer_t is opaque in C, reinterpret_cast to/from cfc::Composer* in shim.

layouts_c_api.cpp is analogous for cfc_layout_*. Holds static cache vector<cfc_layout_t> resynced with cfc::current_templates() on reload.

9. Build

9.1 CMake

project(cuframes-composer LANGUAGES C CXX CUDA)
set(CMAKE_CXX_STANDARD 17)

COMPOSER_SOURCES_CPP in src/CMakeLists.txt lists all .cpp files.

9.2 Host build (for CI / dev machine Ubuntu 24.04)

cd /home/claude/projects/cuframes-composer
mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Release ..
make -j$(nproc)

9.3 Production build (Ubuntu 22.04 jammy)

See operations.md. Briefly:

docker run --rm --gpus all \
  -v "$PWD":/src -w /src/build-jammy \
  cuframes-composer-builder:cached \
  bash -c 'cmake .. && make -j$(nproc)'

10. Performance

10.1 Zero-copy guarantees

• One CudaBuffer output_ per Composer, passed as NV12Ref to all cells/decorations
• Cell does not create VRAM allocations per frame (only Decoration::draw may do FreeType rebuild on set_text — that's offline)
• Layout::apply() recreates vector<unique_ptr<Cell>> only on template change; typically every N seconds

10.2 Virtual call overhead

Cell::draw() → draw_content() virtual call: 1 indirect call per cell per frame. At 25 fps × 16 cells = 400 calls/sec — irrelevant.

10.3 Hot path

CUDA kernels (cugrid.cu):

• fill_nv12 — 1 kernel launch per rect
• resize_nv12 — bilinear via 2 kernels (Y plane + UV plane)
• blit_rgba_nv12 — 1 kernel, RGBA → NV12 + alpha-blend

All Cell operations convert to N×fill_nv12 + 1×resize_nv12 + M×blit_rgba_nv12 — batching not yet needed (GPU is not sticky on current load).

11. Where to edit common tasks

I want to…	File
Change hysteresis	composer.hpp → ComposerConfig::shrink_hysteresis_ms
Change cell border color	layout.cpp → border_style.color_y/u/v
Change label font	mqtt_overlay.cpp → LabelDecoration style + LabelStyle::font_path
Add ZMQ verb	control.c::dispatch() + new cmd_* function
Change manual override duration	composer.hpp → manual_override_duration_ms_
Add new MQTT-overlay anchor	mqtt_overlay.cpp::reposition_overlay() switch
Support color emoji	overlay.c::text_rebuild_atlas() — handle FT_LOAD_COLOR + bitmap BGRA → blit as RGBA

12. Testing

12.1 Unit (not implemented)

Planned: catch2 tests for pick_best_fit, collect_active, hysteresis. CUDA dependency — mock via cfc_source_get_latest shim.

12.2 Integration smoke (examples/grid_record_cpp.cpp)

Minimal C++ smoke: init Composer, compose loop, dump NV12 → file. Does not use NVENC (tests composition only).

build/examples/grid_record_cpp \
  --out=/tmp/dump.nv12 --frames=10 --width=1920 --height=1080 \
  --templates=docker/templates.json \
  --source=cam-parking,frigate=parking_overview,priority=100 \
  --motion-mode

12.3 Production smoke

docker logs cfc-grid | grep -E "loaded|template|motion|grow|shrink" — live composer telemetry.

13. Known pitfalls

• cfc_overlay_t is not RAII — managed via cfc_composer_add_overlay / cfc_overlay_destroy (Composer.cpp::~Composer destroys all).
• pthread_mutex_t in SourcePool vs std::mutex — chose std::mutex for C++ layer, pthread_mutex_t for C layer (PoolEntry::last_motion_ms uses std::atomic).
• Compose loop is NOT blocking on CUDA ops — cudaStreamSynchronize called by caller (grid_record.c) before NVENC.
• Frigate event may arrive BEFORE Layout::apply — motion_pulse updates last_motion_ms, but cell for that camera may not exist yet. On next frame maybe_relayout recalculates.
• dynamic_cast<CameraCell*> in Layout::find_camera_cell_rect — uses RTTI. Enabled -frtti by default in g++/nvcc.

14. Phase task change workflow

1. Branch feature/<phase>-<feature> off main
2. Implementation, host build PASS, jammy build PASS (see operations.md)
3. Bake image gx/cuframes-composer:<phase>-stepN
4. Deploy to dev target → smoke verify via VLC/logs
5. Commit + push branch
6. (If needed) — merge to main via --no-ff PR-style

For multi-commit phase: one WIP merge commit on main describing key changes.