A Real-Browser-Only Login Bug, DMX Automation at Church, and Bridging the Gap

The GitLab login bug that only ever broke for real browsers

GitLab behind rust-proxy had a login failure that made no sense on paper: a 422 InvalidAuthenticityToken, 100% reproducible in real Chrome and Edge, on desktop and mobile, every single time — and completely unreproducible with curl or a scripted httpx client using the exact same credentials. That split is the kind of clue that usually means the bug lives in something a script simply doesn't do the way a real browser does, so the fastest way to find it was to stop guessing and drive an actual browser engine through the problem: installed Playwright, ran headless Chromium, and pointed it at the real login form.

The investigation ruled out a long list of reasonable suspects along the way — stale browser cache, extensions, different networks and devices, IPv6, connection reuse, concurrent request bursts, special characters in the password, timing — before the real culprit showed up in a raw TCP byte capture. I temporarily redirected rust-proxy's upstream for the GitLab host to a throwaway Python socket listener, captured the exact bytes rust-proxy was forwarding, and restored the real upstream immediately after. Sitting right there in the capture: four separate cookie: header lines, instead of one. That's legal — RFC 7540 §8.1.2.5 lets an HTTP/2 client split its Cookie header into multiple fields for better compression, and real Chromium and Firefox both do it. Curl, httpx, and Node's fetch don't, which is exactly why every scripted reproduction attempt sailed through. GitLab's HTTP layer, like most HTTP/1.1 server frameworks, only reads the first Cookie: line off the wire — and the session cookie happened to be sitting in the second of the four. GitLab was never seeing a session at all, so it rejected the CSRF token issued for a session it could no longer see. rust-proxy now merges every split cookie header into one line, per spec, before forwarding to an HTTP/1.1 upstream — confirmed the fix actually works by watching that same headless Chromium sail through the password step that used to 422 every time, then confirmed it for real logging in with a passkey in one browser and 2FA in another.

That capture technique paid off twice more along the way, closing out a few other real gaps in how rust-proxy talks to upstreams: an HTTP/2 request's absolute-form URI was being forwarded to HTTP/1.1 upstreams unchanged, which happened to work by accident against nginx but got correctly rejected as malformed by a stricter upstream (Proxmox's own pveproxy) — now downgraded to origin-form before forwarding. The real client IP is now forwarded to every upstream via X-Forwarded-For/-Proto/-Host, instead of every backend seeing every request as coming from the proxy itself. And a per-host option now lets an operator downgrade an upstream's own SameSite=None cookies to SameSite=Lax, for the case where the backend application sets that unconditionally and modern browsers quietly drop the cookie for it. All four fixes, tests, and a changelog write-up are pushed.

Bridging the Gap

After the workday wrapped up, took Lisa to her work's Bridging the Gap dinner — a nice way to close out a day that started with rewiring the church's AV and lighting integrations and ended with good food and good company.