Hardware transcoding: measured, built, and deliberately not adopted

Decision, 2026-09-12: vibe-live does not use hardware transcoding, and will not implement it against OvenMediaEngine. The work was done — the cost was measured, a working image was built and published, the GPU was made schedulable — and then rejected on the trade it actually asks for. This document is why, so the question does not get reopened from scratch.

The architecture should stay aware of it. Nothing here says hardware encoding is a bad idea; it says OvenMediaEngine is the wrong place to get it now. Anything that assumes transcoding is inherently a CPU cost, or that bakes software-encode timings into a design, is making an assumption this decision does not license.

Reopened 2026-09-12, through ffmpeg rather than through the engine

That last paragraph stopped being hypothetical the same day it was written. The project's direction changed to an ffmpeg-first media path with engines as optional delegates (docs/design/engines-and-players.md), and ffmpeg's hardware paths — NVENC, VAAPI, QuickSync, V4L2 — are first-class and maintained. Doing the transcode ourselves reaches the GPU without asking OvenMediaEngine for anything.

So this decision is extended, not reversed. Its scope was always the route, never the capability: the GPU was unreachable through the engine, at the price of pinning that engine at v0.20.5 forever. It is reachable through a transcoder this project already has to run for the engine-less path to exist at all. Every number below still stands and is the baseline the new path will be measured against.

Two things this does not license, both recorded so the next reader does not have to re-derive them:

• Software first, hardware after. Hardware encoding is an optimisation with a large portability surface. The built-in path is proved in software before any vendor path is added, which is step 8 of the design doc's plan and deliberately not step 1.
• An ffmpeg carrying every vendor's encoder is not a small image, so "no engine" does not automatically mean "slim". The likely shape is software plus one vendor path by default, with multi-vendor as its own variant — the same shape as the engine tags.

Why not

Upstream discontinued it in the open-source edition. Per ovenmedia.com/docs/ome/transcoding: "Hardware acceleration is no longer supported in the open-source version in releases after v0.20.5." The capability has moved behind a paid edition.

Using it therefore means pinning the media server at v0.20.5 forever — no upstream fixes, no security patches, no new features — and maintaining a from-source build of an ageing release against a project that has deliberately moved on. The code paths may be removed outright later, not just the build flags.

And the saving does not justify that. Measured, not estimated:

28.9x cheaper, and worth 0.57 cores per stream. On the media node's 8 CPUs that is 7% of the node per stream. At one to three concurrent streams this project spends under two cores to stay on a supportable version. That is a good trade.

The figure is trustworthy: 0.595 came from Prometheus over 2026-09-04..11 (55 samples with a channel live against 5,419 idle at 0.0007), and an independent synthetic encode on the same node gave 0.599 — two unrelated methods 0.7% apart.

The better lever, which costs no version freedom

llhls_transcode re-encodes to H.264 720p30 unconditionally. When a streamer already publishes 720p30 H.264 — which OBS defaults make common — that entire transcode is waste, and the bypass profile already proves OvenMediaEngine can pass a stream through untouched for WebRTC.

The cheapest transcode is the one not performed. What this needs before building anything: how often the incoming stream already matches, and whether LLHLS segmentation tolerates a bypassed stream's keyframe cadence. Both are measurements, not guesses.

What was kept

Only these numbers, so the question can be reopened with evidence rather than re-measured from nothing.

The GPU time-slicing added for this was also reverted: the media node advertises one GPU again and jellyfin holds it undivided. Time-slicing is not partitioning, so with nothing else meant to be on the card, advertising two replicas only creates the possibility of a workload scheduling onto a GPU it then contends for, with no isolation and no warning.

What was removed

deploy/ome-image/ (the Dockerfile that built OvenMediaEngine with --enable-nvc --enable-qsv), .forgejo/workflows/ome-image.yml, and the guards in cmd/vibe-live/ome_config_test.go that protected them. The image it produced was published and works: forgejo.gmatte.xyz/gmatte/ovenmediaengine-hw:0.18.0 — 401MB against airensoft's 377MB, both vendors compiled in, starting cleanly on hosts with no GPU.

If this is ever reopened

The hard-won details, because they cost a day and none is guessable:

• OvenMediaEngine v0.18.0 supports NV, QSV, XMA, NILOGAN. No VAAPI path and no AMD path at all — an AMD GPU is unreachable however it is built.
• The flags are independent booleans, so ONE image serves both vendors: misc/prerequisites.sh --enable-nvc --enable-qsv.
• <Modules> must stay EMPTY. OME only appends the software fallback candidate when it is unset; pinning NV with no usable device empties the encoder candidate list, every frame fails to decode, and the config looks correct. Found live, twice.
• The CUDA toolkit is a BUILD dependency only. Driver libraries are injected at runtime by the container runtime — jellyfin ships none and supports 24 hardware encoders in 0.74GB. Build on CUDA devel, ship on plain ubuntu.
• But OME links libcuda.so.1 and libnvidia-ml.so.1 at LOAD time, so a hardware build refuses to start on any host without them. It needs the 100KB of toolkit link stubs, behind a wrapper that exposes them only when neither real library is present.
• prerequisites.sh calls sudo, and sudo's env_reset is load-bearing: the CUDA base exports LIBRARY_PATH, which hiredis's Makefile reads as its install subdirectory. Replacing sudo with a bare exec shim puts hiredis somewhere pkg-config cannot find it, and the build dies 200 lines later linking with undefined redisCommand.
• Verify a hardware encode by watching nvidia-smi --query-gpu=encoder.stats.sessionCount DURING a publish. Never by reading OME's log: every hardware failure in this project looked like success from outside, and one ffmpeg run reported 589x faster than software while timing a crash that produced zero frames.