A Real Loss Rate Finally Caught With My Own Packet-Capture Tool
Four more mlvpn-rs releases in one night, a fix that honestly didn't work, a rollback that turned out to be a false alarm caught just in time, and — after ruling out nearly everything else I could think of — finally proving with hard numbers that real data is disappearing between my two routers in a way the tunnel itself never caught. Plus a first day back at the hardware store.
Two more real fixes, then one that honestly didn't help
Picked back up right where the last post left off: the tunnel's own internal bottleneck was still costing real speed under load. I split one shared resource into two, so checking the connection's security setup — which happens on every single piece of data sent — no longer has to wait behind the much rarer moments that actually change that setup. The idea came from looking at how WireGuard itself handles the same problem. I followed it with a smaller fix to a background task that was grabbing the same resource a second time, unnecessarily, every time data arrived out of order.
Then I tried something that looked promising on paper: shrinking how long the tunnel waits for out-of-order data once it's confident only one connection is being used, since there's nothing left to reorder at that point. I shipped it, deployed it to my real home-to-cloud routers, and it genuinely did shrink that wait time the way it was supposed to. It just didn't move the actual problem at all — same slow speeds, if anything a bit more retransmitting than before. Rather than call that a win because the code worked as designed, I said so plainly and moved on to the next idea. That's a better use of an evening than pretending a fix helped when the numbers say otherwise.
Borrowing WireGuard's whole approach — and a real bug caught before it shipped
The next idea was bigger: WireGuard doesn't bother holding data back to fix ordering at all on a single connection — it just delivers everything the instant it arrives and lets the receiving computer sort out anything that came in out of order, which it already does fine on its own. mlvpn-rs's own "wait and reorder" step only exists because it can spread traffic across multiple connections at once, where real reordering is possible. Once it's settled on using just one connection, that whole step is pure overhead. So I built exactly that: skip the reorder step entirely once the tunnel has settled on a single active connection.
I sent it out for a second opinion before shipping, same as always for anything touching how data actually gets delivered — and it was the right call. The reviewer found that one of my own safety assumptions was wrong: I'd assumed a certain cleanup step would always run at least once per connection, but it might never run again once this new fast path kicked in. Left alone, that could have left the tunnel permanently stuck favoring the wrong connection after a routine security refresh, with no way back short of a restart. I fixed it, got a clean second look, and shipped it as its own release.
A rollback that turned out to be a false alarm — caught in time
Ran the usual real-world speed test right after deploying that fix and got a genuinely bad number — worse than before, by every measure. Rolled straight back to the previous version to be safe. Then, before concluding anything, I ran the exact same test again on the version I'd just rolled back to, a minute later — and it was just as bad. The new version was never the problem. Whatever's actually wrong with the connection tonight had simply gotten worse in the few minutes between tests, and my first comparison was against an old, stale number from earlier in the night instead of a fair one taken right then. I put the new version back. Small thing, but a good reminder: never trust a "this got worse" conclusion without a fresh, same-moment comparison against whatever you're rolling back to.
Proving it's the software, not the internet connection
With four releases now hunting the same mystery and none of them moving the needle, it was time to stop guessing and start ruling things out one at a time. I logged into the cable modem directly and pulled its error history — completely clean, no sign of a bad physical connection. I renewed the internet lease on the router just in case — no change, same result.
Then the real turning point: I already had a plain WireGuard connection set up between the same two routers, over the exact same internet line, for exactly this kind of comparison. I brought it up and ran the identical speed test through it, back to back with mlvpn-rs, on the same connection, at the same moment. WireGuard: fast, clean, barely any retransmitting. mlvpn-rs, seconds later, same connection: slow, and heavily retransmitting. That's about as close to a controlled experiment as you get outside a lab — same wire, same two computers, same instant, wildly different result. Whatever's wrong, it's mine to fix, not the internet provider's.
Ruling things out one at a time, then building a tool to actually prove it
From there it became a real process of elimination. A theory that the tunnel's own speed-measuring feature was flooding the connection and starving real traffic — checked by turning it off entirely and testing again; still slow, so not that (though it did turn up a real, separate bug worth fixing later). A theory that mlvpn-rs sends data in uneven bursts while WireGuard sends smoothly — checked by measuring the actual gaps between data on both tunnels; turned out WireGuard is actually the burstier of the two, so that wasn't it either. A theory that the computer itself was silently dropping data internally — checked against the exact internal counter that would catch that; it hadn't moved once during an active bad run.
Every one of those came back clean, which was frustrating but also narrowing things down fast. So I wrote a small tool from scratch: capture everything the tunnel actually sends on both routers at once, and match it up to see whether data that genuinely left one side ever actually showed up on the other. My first pass looked like it had caught something alarming — the same piece of data appearing to be sent twice — but a closer look showed that was a mistake in my own tool, not a real bug: both directions of traffic share the same numbering, so it was occasionally matching up two unrelated pieces of data that just happened to share a number by coincidence. I fixed the tool, confirmed the false match disappeared, and ran it again for real.
The corrected result: of everything the sending router genuinely transmitted, about 23 percent of it never showed up on the receiving end at all — real, measurable loss, on a connection the tunnel's own health checks report as essentially perfect. That's the real number that's been hiding behind every one of tonight's failed fixes. As of writing this, mid-investigation, the next test in progress is changing which network port the tunnel uses, to check whether something along the path is specifically picking out this one connection rather than treating all traffic the same way. I don't have an answer yet — but for the first time tonight, there's a real, solid number to chase instead of a guess.
Personal note
First day back at the Ace Hardware in Zion landed on a delivery day, so most of the shift was spent getting new stock out onto the shelves, with a couple of customers helped along the way. Good, simple contrast to the rest of tonight.