A Real Weekend Outage, a No-Signal Slate, Two OBS Bugs, and a Rebuilt Wiki
A stretch of days that started with real live-production firefighting on SRT Matrix — a boot-time race that killed every source for a whole weekend without anyone noticing, plus a batch of failures that were failing completely silently — grew into a proper "no signal" slate feature, then turned into actually starting OBS SRT Toolkit: a design spec that had sat untouched since late July became a real, working OBS plugin in a single day, including two genuine bugs found and fixed in how it registers with libobs. Finished by rebuilding jpps/wiki to support GitLab-style namespace/project URLs and updating every doc to point people at the plugin.
A weekend outage nobody noticed, and three bugs that were failing silently
Found a real production bug the hard way: a source's SRT ingest listener only ever started when you explicitly told it to over the API — destinations already auto-resumed on every gateway startup, but nothing replayed that same step for sources already sitting in the loaded config. A crash, an update, or a plain restart silently killed every source's listener with no way back short of manually re-triggering each one by hand. Fixed it so sources resume exactly the same way destinations already did, plus a regression test to keep it that way.
That fix uncovered a second, uglier one: a gateway that reboots before its network interface
actually has an IP assigned gets EADDRNOTAVAIL trying to bind its listeners, and
every listener bind only ever tried once. One real reboot left every source's ingest port
permanently dead — for an entire weekend, since nothing was actively trying to connect in the
meantime to notice. Added a bounded retry (30 attempts, 2s apart) specifically for that one
error code, leaving genuine problems like a port already in use failing immediately as before.
Chasing a separate live incident — a return connection over a VPN link that kept retrying forever with zero log output anywhere, even though a packet capture proved real UDP traffic was flowing both directions — turned up a pattern repeated in three different places: a failed connection accept, a rejected connection with no matching stream ID, and a failed program-out pull were all being silently discarded, updating an in-memory diagnostics counter but never actually logging anything. Fixed all three, so the next version of this exact problem shows up in the logs instead of requiring a packet capture to even start narrowing it down.
Smaller but real: presets could already save and restore which source feeds which destination, but had no way to represent a source's return route at all — applying a preset silently left return routing exactly as it was, regardless of what the preset's forward routes did. Added return-route support to presets with the same full-snapshot semantics the forward side already had.
Building a real "no signal" slate, live, during another incident
A field decoder receiving nothing at all (a station that's down, a route nobody's patched in yet) just reports a bare connection timeout — which looks identical to an actual network problem, even when the real cause is something as mundane as "the upstream just hasn't started streaming yet." Built a proper fix: any idle route now automatically falls back, after 5 seconds with no real payload, to a shared synthetic "technical difficulties" video loop instead of sending silence — and switches back to the real picture the instant it resumes. Slate frames are deliberately excluded from the traffic stats and the idle-detection clock, so they never get mistaken for real signal.
A few real refinements followed once it was actually in use: return feeds now show the slate even when genuinely unrouted (a field operator's confidence monitor should never just be dark because nobody's patched a return route yet, which is a different situation from a forward push slot nobody's touched — that one correctly stays blank, since that's still someone's own explicit choice). Clearing an already-idle return route used to leave the decoder showing a stale-frame error for the full 5-second grace period before the slate kicked in; now it jumps straight to the slate on the very next frame, since there's nothing to gain by waiting on a route that can no longer produce anything real. The slate image itself is fully configurable per install from the dashboard, with a live preview so you're never uploading blind, and each site gets its own return-feed thumbnail so an operator can tell at a glance whether a site is showing real content or the slate without checking the field decoder directly.
Then, cleaning up SRT Matrix's own builder
Small housekeeping before turning to OBS:
SRT Matrix's
Windows installer was
still showing "srt-matrix" — lowercase, hyphenated — in Add/Remove Programs and the
installer's own title bar, instead of the real product name. Fixed the WiX product name.
Ported a CI speedup I'd already proven on obs-srt-toolkit's runner over to SRT Matrix too:
GitLab Runner's default pre-checkout git clean -ffdx was wiping the
target/ directory on every single job, so even a full sccache hit still forced
every crate back through rustc. Excluding target/ from that clean
step took one job from several minutes down to under 20 seconds.
Also fixed a small but real terminology bug on the operator dashboard: the license panel was still calling capacity limits "servers," a holdover from before a naming pass that had already renamed everything else to "destinations." Fixed both the badge and the full license section.
Actually starting OBS SRT Toolkit — design spec to a working plugin, in one day
OBS SRT Toolkit had existed as an idea since late July — a design spec for an OBS Studio
plugin that would give OBS proper SRT input and output, replacing the old trick of repurposing
OBS's Custom FFmpeg Output to fake an SRT stream — but nothing beyond that spec had actually
been written. Today's the day that changed: scaffolded the Rust cdylib project,
then built out a real SRT Output (registration, a settings UI, listener and caller mode over
real srt-tokio sockets, hand-rolled MPEG-TS muxing verified against an independent
decoder), an independent Start/Stop dock as a small C++/Qt shim, a real SRT Source that hands
off to OBS's own FFmpeg-backed decoder, and a Windows .msi installer that finds an
existing OBS install automatically — fixing a string of genuine Windows-specific build failures
(a C++17/C++20 mismatch in Qt6's SFINAE helpers, a missing import library, CI silently
recompiling every dependency on every run) along the way.
With both directions actually built, the obvious next step was proving they worked against something real rather than just compiling cleanly — SRT Matrix, running live, was right there. Pushing a source in worked cleanly on the first try. Getting OBS's own Program Out feed back to SRT Matrix did not: SRT Matrix's dashboard showed a connected socket, but zero bytes ever moved.
First bug: OBS's log was complaining about "trying to use a video encoder on a raw output" —
traced it to the Rust obs-wrapper crate never actually setting the
OBS_OUTPUT_ENCODED flag OBS itself needs to treat an output as encoded rather
than raw, even when you ask for encoded-packet callbacks. Worked around it by building the
output info normally, then reaching into the raw obs_output_info struct after
the fact to patch the flag before registering it directly with libobs.
Fixing that got the connection further, but still zero packets — a temporary log line dropped
straight into the plugin's own packet callback confirmed libobs was calling it exactly zero
times, despite a fully established SRT session. That sent me into libobs's own C source rather
than guessing: turns out a real OBS output plugin is expected to call
obs_output_begin_data_capture() itself once it's ready — that's the actual switch
that wires up encoder-to-output dispatch — and obs-wrapper never calls it on your
behalf. One line fixed it. After both patches, SRT Matrix's dashboard showed a live thumbnail
of OBS's actual program feed and real sustained throughput, not just an idle-but-connected
socket — a real plugin, built and proven working end to end, same day it went from spec to
code.
Rebuilding jpps/wiki to work like GitLab does
Wanted a wiki page documenting all of this for real users, which meant first fixing a design
gap in jpps/wiki
itself (the self-hosted wiki
I built to replace GitLab's built-in one). It
only supported a flat wiki.jpps.us/<project> URL shape — fine when
everything belonged to one group, not fine going forward, since I want a personal namespace
alongside the company one eventually. Rebuilt it to work the way GitLab itself does:
wiki.jpps.us/<namespace>/<project>, with a short-link fallback —
going straight to wiki.jpps.us/srt-matrix still works and auto-resolves to the
right namespace, unless two namespaces both have a project by that name, in which case it
shows a real disambiguation page instead of guessing.
Added a proper migration (SQLite can't add a column into an existing unique constraint
directly, so it's a rebuild-the-table-underneath migration, backfilling every existing page to
the real jpps namespace it already belonged to), a left sidebar listing every page
in the current project, and fixed the nav bar to show the actual project's title instead of a
static "jpps-wiki" label. Tested the whole migration against a real copy of the production
database before ever touching the live one, then deployed for real with a backup taken first.
68 tests passing, verified live with zero errors in the service log afterward.
Publishing the wiki page, updating the Software page, and making the docs agree
With both the plugin and the wiki actually working, wrote a real user-facing wiki page for OBS SRT Toolkit and added it to jpps.us's Software page — a new card plus a full detail page, following the same structure as SRT Matrix's own page but without a pricing section, since this one's GPL-2.0 and free.
Then went back through every place that still described the plugin as a "planned, not yet implemented" idea and fixed it, since that was true a week ago and isn't anymore: SRT Matrix's own OBS Setup Guide got restructured to lead with "install OBS SRT Toolkit" as the recommended path, with the old manual OBS workarounds (hand-crafted SRT URLs in a Media Source, repurposing the Recording tab to fake an SRT output) kept further down as a documented fallback for anyone who can't install a plugin. The roadmap page moved the toolkit from "on the roadmap" to "shipped." And the plugin's own README had two dead links pointing at GitLab's old built-in wiki, which hasn't been the real wiki in a while — fixed those too.