A Peplink Integration, a New Support-Ticket System, and a Real Outage I Caused Myself

Building a support/diagnostics-report system from nothing

The bigger new thing: a whole new service, jpps/software-support, built from a blank repo to handle diagnostics reports across every JPPS product going forward, not just OmniBridge. Axum + SQLite, a real three-state ticket lifecycle (waiting on support, waiting on reporter, closed) that flips automatically depending on who replied last, a public no-login status page gated by an unguessable ticket ID instead of a whole account system, and an admin side with real form-based login — deliberately not Basic Auth, since that breaks password-manager autofill.

Then built the other half inside OmniBridge itself: a "Send diagnostics report" button on the About page that bundles recent logs and the current routing config — with every passphrase, stream key, and API secret redacted before it ever leaves the machine, not scrubbed server-side after the fact — plus a universal crash hook that submits a best-effort report automatically if the app ever panics, on every platform (previously this only existed on Windows release builds, and even then didn't really work).

That crash hook alone was a small saga on its own. First version spawned a detached thread to send the report and returned immediately — and never actually sent anything, because the process exits the moment the panicking thread finishes unwinding, whether or not some other thread it spawned is still running. Fixed version two by dropping the thread spawn and calling the network client directly from the hook — which also didn't work, because reqwest::blocking can't build its own async runtime from inside an app that's already running one. The actual fix needed both: spawn the thread and join it, blocking the panic sequence until the report either sends or times out. Confirmed for real by triggering actual panics against the live service and watching real tickets show up. A second real bug also only showed up by clicking the button in an actual browser against a real running instance: the new endpoint had landed inside the license-gated part of the router, so an unlicensed install — exactly the install most likely to need to file a report — couldn't use it at all. Neither of those was visible to a fully green 243-test suite; both would have shipped broken without the manual pass.

A real outage, self-inflicted, while chasing "wasted disk space"

Not every part of this stretch went cleanly. Asked to check the shared services host for wasted disk space, I found a 15GB anomalous log file and went hunting for what was writing to it with awk | sort | uniq -c run directly against the live 126-million-line file — on the same box serving the store, the wiki, this new support system, and the reverse proxy fronting all of it. That exhausted the box's memory and swap and made every one of those services unresponsive for several minutes, a real production incident I caused with a single bad command. Flagged it immediately rather than quietly trying to paper over it; the actual recovery was the user raising the LXC's swap via Proxmox while I killed the stray process once SSH came back.

The actual root cause behind that oversized log turned out to be worth finding for real, once I did it properly — downloaded a gzip'd copy locally instead of touching the live file again. rust-proxy's accept loop had no backoff on a transient EMFILE error, so once it hit the open-file limit it busy-spun forever, logging as fast as the CPU could go. Fixed with a small backoff plus a raised LimitNOFILE in the systemd unit, deployed, and confirmed the disk usage back down from 51% to 13%. Lesson learned the expensive way: never run a memory-hungry command against a large file on a live production host again — download it and work on the copy.