--- title: FFmpeg cuframes:// demuxer sidebar_position: 1 --- # FFmpeg `cuframes://` demuxer cuframes ships two FFmpeg input demuxers, both delivered as a patch on top of upstream FFmpeg: - **`cuframes`** — subscribes to a decoded NV12 frame ring and exposes it as a `rawvideo` stream (one stream per URL). - **`cuframes_packets`** — subscribes to an encoded packet ring and exposes it as an `h264` / `hevc` byte-stream, with `extradata` taken from the publisher's handshake. Both demuxers are pure consumers. They never decode, never re-encode, and never touch the network — the actual RTSP pull happens once in the publisher (typically [`cuframes-rtsp-source`](/docs/getting-started/install)). FFmpeg just attaches to the existing ring via a Unix socket. ## URL scheme ```text cuframes:// # decoded NV12 frames (raw GPU surfaces) cuframes_packets:// # encoded H264/HEVC packets (Annex-B) ``` `` is the publisher's key — the same string passed to `cuframes_publisher_create()` or to `--key` on `cuframes-rtsp-source`. The legacy `cuframes:` form (no `//`) is also accepted. The two ring types are independent. A single publisher can expose both: decoded frames for compositors / AI, encoded packets for recorders that want to skip a re-encode. ## What the demuxer does On open, the demuxer: 1. Connects to `/run/cuframes/.sock`. 2. Receives N POSIX file descriptors via `SCM_RIGHTS` (frame slots) plus the shm metadata header. 3. For `cuframes`: imports each FD as a CUDA VMM allocation, advertises a single `AV_PIX_FMT_NV12` stream at the publisher's width/height/framerate. 4. For `cuframes_packets`: reads `extradata` (SPS/PPS for H264, VPS/SPS/PPS for HEVC) from the handshake and advertises one `AV_CODEC_ID_H264` or `AV_CODEC_ID_HEVC` stream. In the read loop the demuxer polls the publisher's `global_seq`, copies the frame / packet into the pipeline, and stamps `pts` from the publisher's clock. See [Protocol reference](/docs/reference/protocol) for the wire format. ## Pipeline examples ### Single source — decoded ring → NVENC → MPEG-TS Re-encode a published camera into an H.264 MPEG-TS UDP stream. No NVDEC happens in this ffmpeg — the publisher already decoded once, this process just NVENCs the shared NV12 surface. ```bash ffmpeg -hwaccel cuda -hwaccel_output_format cuda \ -f cuframes -i cuframes://cam-parking \ -c:v h264_nvenc -preset p4 -b:v 4M \ -f mpegts udp://192.168.88.50:5000 ``` ### Packet ring — true copy, no decode and no encode When you only need to record or restream an existing camera and do not care about the decoded pixels, subscribe to the **packet** ring with `-c:v copy`. Both NVDEC and NVENC stay idle. ```bash ffmpeg -f cuframes_packets -i cuframes_packets://cam-parking \ -c:v copy -f mp4 /var/recordings/cam-parking.mp4 ``` This is the cheapest way to fan out a single decode to N recorders. ### Composition — 4 inputs into a CUDA grid A short example of multi-input wiring. The full filter reference lives in [FFmpeg `vf_cuda_grid` filter](/docs/integration/ffmpeg-filter). ```bash ffmpeg \ -hwaccel cuda -hwaccel_output_format cuda \ -f cuframes -i cuframes://cam1 \ -f cuframes -i cuframes://cam2 \ -f cuframes -i cuframes://cam3 \ -f cuframes -i cuframes://cam4 \ -filter_complex "[0:v][1:v][2:v][3:v]cuda_grid=layout=quad[out]" \ -map "[out]" -c:v h264_nvenc -preset p4 \ -f rtsp rtsp://127.0.0.1:8554/grid ``` ## Build / run with the patched FFmpeg The demuxers live in `libavformat/cuframesdec.c` and `libavformat/cuframes_packetsdec.c` and are not in upstream FFmpeg. You have two options. ### Option A — pre-built Docker image A production-tested image is published as `ffmpeg-vf-cuda-grid:phase8`. It contains the patched ffmpeg binary, `libcuframes.so`, and the `vf_cuda_grid` filter. The reference `docker compose` setup that wires it together with a publisher container lives in the `localhost-infra` repo — copy and adapt, do not pin it as a public dependency. ```bash docker run --rm --runtime=nvidia \ --ipc=container:cuframes-publisher \ ffmpeg-vf-cuda-grid:phase8 \ ffmpeg -f cuframes -i cuframes://cam1 -c:v copy -f null - ``` The `--ipc=container:...` flag matches the publisher's IPC namespace so the POSIX shm header is visible. PID namespace sharing is **not** required since cuframes v0.4. ### Option B — build it yourself Use the `ffmpeg-builds` toolchain (a fork of `BtbN/FFmpeg-Builds`). The script `scripts.d/50-libcuframes.sh` clones cuframes, builds it static, and `--enable-libcuframes` is appended automatically when the `cuframes` addin is active. ```bash git clone ffmpeg-builds cd ffmpeg-builds ADDITIONAL_SCRIPTS=50-libcuframes.sh ./build.sh ``` The patched ffmpeg source tree (with both demuxers and the filter) lives in `ffmpeg-fresh/`. If you want to vendor the patch into your own FFmpeg fork, copy the three files (`libavformat/cuframesdec.c`, `libavformat/cuframes_packetsdec.c`, `libavfilter/vf_cuda_grid.c`) plus the matching `Makefile` and `allformats.c` / `allfilters.c` registrations. ## Reconnect behaviour Publishers come and go — container restarts, RTSP camera reboots, the host's `cuframes-rtsp-source` is updated. The demuxer is designed to **survive a publisher restart without tearing down the FFmpeg pipeline**. When the subscriber sees `CUFRAMES_ERR_DISCONNECTED`: - The demuxer does **not** return `AVERROR_EOF`. - It releases the dead subscriber and tries `cuframes_subscriber_create()` again, rate-limited to **one attempt every 2 seconds**. - While reconnecting, `av_read_frame()` returns `AVERROR(EAGAIN)`. The pipeline blocks but stays alive. - On success the demuxer logs `cuframes: reconnected to ''` at `INFO` level and resumes delivering frames. This matters for long-running consumers (NVR recorders, RTSP restreamers, NVENC composers) that would otherwise need an external supervisor to restart ffmpeg on every publisher hiccup. If you actually want EOF on disconnect — e.g. a one-shot transcode that should stop when the source dies — wrap the demuxer with `-timeout` or your own watchdog. The built-in behaviour is "wait forever", not "fail fast". ## See also - [First publisher](/docs/getting-started/first-publisher) — minimal C producer. - [Protocol reference](/docs/reference/protocol) — wire format and handshake. - [`vf_cuda_grid` filter](/docs/integration/ffmpeg-filter) — multi-camera composition.