cuframes-docs/site/docs/integration/ffmpeg-demuxer.md

title, sidebar_position

title	sidebar_position
FFmpeg cuframes:// demuxer	1

FFmpeg cuframes:// demuxer

cuframes ships two FFmpeg input demuxers, both delivered as a patch on top of upstream FFmpeg:

• cuframes — subscribes to a decoded NV12 frame ring and exposes it as a rawvideo stream (one stream per URL).
• cuframes_packets — subscribes to an encoded packet ring and exposes it as an h264 / hevc byte-stream, with extradata taken from the publisher's handshake.

Both demuxers are pure consumers. They never decode, never re-encode, and never touch the network — the actual RTSP pull happens once in the publisher (typically cuframes-rtsp-source). FFmpeg just attaches to the existing ring via a Unix socket.

URL scheme

cuframes://<key>            # decoded NV12 frames (raw GPU surfaces)
cuframes_packets://<key>    # encoded H264/HEVC packets (Annex-B)

<key> is the publisher's key — the same string passed to cuframes_publisher_create() or to --key on cuframes-rtsp-source. The legacy cuframes:<key> form (no //) is also accepted.

The two ring types are independent. A single publisher can expose both: decoded frames for compositors / AI, encoded packets for recorders that want to skip a re-encode.

What the demuxer does

On open, the demuxer:

1. Connects to /run/cuframes/<key>.sock.
2. Receives N POSIX file descriptors via SCM_RIGHTS (frame slots) plus the shm metadata header.
3. For cuframes: imports each FD as a CUDA VMM allocation, advertises a single AV_PIX_FMT_NV12 stream at the publisher's width/height/framerate.
4. For cuframes_packets: reads extradata (SPS/PPS for H264, VPS/SPS/PPS for HEVC) from the handshake and advertises one AV_CODEC_ID_H264 or AV_CODEC_ID_HEVC stream.

In the read loop the demuxer polls the publisher's global_seq, copies the frame / packet into the pipeline, and stamps pts from the publisher's clock. See Protocol reference for the wire format.

Pipeline examples

Single source — decoded ring → NVENC → MPEG-TS

Re-encode a published camera into an H.264 MPEG-TS UDP stream. No NVDEC happens in this ffmpeg — the publisher already decoded once, this process just NVENCs the shared NV12 surface.

ffmpeg -hwaccel cuda -hwaccel_output_format cuda \
  -f cuframes -i cuframes://cam-parking \
  -c:v h264_nvenc -preset p4 -b:v 4M \
  -f mpegts udp://192.168.88.50:5000

Packet ring — true copy, no decode and no encode

When you only need to record or restream an existing camera and do not care about the decoded pixels, subscribe to the packet ring with -c:v copy. Both NVDEC and NVENC stay idle.

ffmpeg -f cuframes_packets -i cuframes_packets://cam-parking \
  -c:v copy -f mp4 /var/recordings/cam-parking.mp4

This is the cheapest way to fan out a single decode to N recorders.

Composition — 4 inputs into a CUDA grid

A short example of multi-input wiring. The full filter reference lives in FFmpeg vf_cuda_grid filter.

ffmpeg \
  -hwaccel cuda -hwaccel_output_format cuda \
  -f cuframes -i cuframes://cam1 \
  -f cuframes -i cuframes://cam2 \
  -f cuframes -i cuframes://cam3 \
  -f cuframes -i cuframes://cam4 \
  -filter_complex "[0:v][1:v][2:v][3:v]cuda_grid=layout=quad[out]" \
  -map "[out]" -c:v h264_nvenc -preset p4 \
  -f rtsp rtsp://127.0.0.1:8554/grid

Build / run with the patched FFmpeg

The demuxers live in libavformat/cuframesdec.c and libavformat/cuframes_packetsdec.c and are not in upstream FFmpeg. You have two options.

Option A — pre-built Docker image

A production-tested image is published as ffmpeg-vf-cuda-grid:phase8. It contains the patched ffmpeg binary, libcuframes.so, and the vf_cuda_grid filter. The reference docker compose setup that wires it together with a publisher container lives in the localhost-infra repo — copy and adapt, do not pin it as a public dependency.

docker run --rm --runtime=nvidia \
  --ipc=container:cuframes-publisher \
  ffmpeg-vf-cuda-grid:phase8 \
  ffmpeg -f cuframes -i cuframes://cam1 -c:v copy -f null -

The --ipc=container:... flag matches the publisher's IPC namespace so the POSIX shm header is visible. PID namespace sharing is not required since cuframes v0.4.

Option B — build it yourself

Use the ffmpeg-builds toolchain (a fork of BtbN/FFmpeg-Builds). The script scripts.d/50-libcuframes.sh clones cuframes, builds it static, and --enable-libcuframes is appended automatically when the cuframes addin is active.

git clone <ffmpeg-builds repo> ffmpeg-builds
cd ffmpeg-builds
ADDITIONAL_SCRIPTS=50-libcuframes.sh ./build.sh <target> <variant>

The patched ffmpeg source tree (with both demuxers and the filter) lives in ffmpeg-fresh/. If you want to vendor the patch into your own FFmpeg fork, copy the three files (libavformat/cuframesdec.c, libavformat/cuframes_packetsdec.c, libavfilter/vf_cuda_grid.c) plus the matching Makefile and allformats.c / allfilters.c registrations.

Reconnect behaviour

Publishers come and go — container restarts, RTSP camera reboots, the host's cuframes-rtsp-source is updated. The demuxer is designed to survive a publisher restart without tearing down the FFmpeg pipeline.

When the subscriber sees CUFRAMES_ERR_DISCONNECTED:

• The demuxer does not return AVERROR_EOF.
• It releases the dead subscriber and tries cuframes_subscriber_create() again, rate-limited to one attempt every 2 seconds.
• While reconnecting, av_read_frame() returns AVERROR(EAGAIN). The pipeline blocks but stays alive.
• On success the demuxer logs cuframes: reconnected to '<key>' at INFO level and resumes delivering frames.

This matters for long-running consumers (NVR recorders, RTSP restreamers, NVENC composers) that would otherwise need an external supervisor to restart ffmpeg on every publisher hiccup.

If you actually want EOF on disconnect — e.g. a one-shot transcode that should stop when the source dies — wrap the demuxer with -timeout or your own watchdog. The built-in behaviour is "wait forever", not "fail fast".

See also

• First publisher — minimal C producer.
• Protocol reference — wire format and handshake.
• vf_cuda_grid filter — multi-camera composition.