Chute Devs

When Things Go Wrong: Building Chute's Debugging Infrastructure

Hello everyone.

A proxy engine is, by design, a black box. It takes your packets, routes them through rules you wrote, encrypts some of them, sends them out a network you do not control — and most of the time it Just Works. Right up until it doesn’t. Then the support messages start: “Chute is slow.” “This rule doesn’t fire.” “The tunnel died and I don’t know why.” And we are left asking for logs that are too large to send, screenshots of a console that was built to show status, and a lot of guessing.

Two weeks ago we wrote an internal capability assessment of our own diagnostic groundwork, and it was blunt: the ground floor was thick — we had a lot of machinery — but the last mile was broken in four specific, fixable ways. This post is about what we built in the two weeks since. It is the biggest unglamorous upgrade the engine has had, and it will change what happens the next time you report a bug.

The Four Things We Fixed First

The assessment’s first-pass list was a set of specific, nameable defects, not a vibe. Four of them:

1. Timestamps had no anchor. Every value the engine emitted counted from engine start, because that is the cheap clock the per-packet paths can afford. Nothing published the base. So the web UI did the only thing it could with a bare duration — rendered it as an epoch — and drew 1970. The fix ships the base with the values: the engine now publishes its start time alongside its uptime, the connection and log pages carry both, and the log ring buffer switched to the wall clock outright. Wall time is what you render; the monotonic counter is the cursor you page by.

2. The log had no bounds. One file per run, no rotation, no size cap. The only reclamation was deleting a whole run folder. On a device connected for days that file grew until it was too large to retrieve from the very process you needed to diagnose — the download cap inside the iOS extension is 8 MB, and the log was routinely larger. Shards are now sized to that cap: 8 MB, 8 shards per run, oldest dropped, so any completed shard can be downloaded whole. Each shard opens with a header naming the run and the wall-clock instant its uptime base refers to, so a shard in isolation is self-describing.

3. The debug endpoint was open. The engine’s internal HTTP listener, previously reachable with no token, is now bound to a per-run token that the host app resolves and the console consumes. A URL can now carry that token and sign you straight in.

4. A kill was invisible. When the system jetsam’d the extension for memory pressure, the next launch had no way to say so. The engine now writes a run marker and refreshes it under memory pressure; on launch it classifies the previous marker — fresh means “still running”, stale means “killed” — and says it out loud.

We also fixed monotonic log cursors — a “give me entries since X” that stopped matching the moment uptime wrapped — and recorded two pre-existing defects the work turned up. That last part is a habit now, not an afterthought: every batch ends with a committed doc naming what it found.

Giving the Engine a Voice

The machinery was all there; the question was how to interrogate it without an Xcode debugger attached. The answer: the engine’s own diagnostics, answerable over HTTP.

The web console stopped being a status page. It now has a page that explains a run — start time, shard list, memory watermark, the tunnels it carried, the crash marker of the run before it. The host app resolves the console’s address and token in one place and hands it to you. You stop asking “where do I even look” and start asking “what is this engine telling me.”

Three Things a Support Conversation Needs

The next batch was defined narrowly: three artifacts a support exchange could not exist without.

Say which rule fired. The connection records already carried flags saying a rewrite rule changed the message, but not which one. With a dozen rules in a module, that is the whole question, and the only way to answer it was to comment lines out one at a time. The engine now counts every application by rule text and attaches the rule to the connection it changed. The console names the culprit, and it also reports which rules have never fired at all — the usual reason a rewrite “doesn’t work”, previously invisible.

Export a HAR that isn’t a guess. The MITM payload files are a flat byte concatenation, so no byte carries a time and any HAR timing built from them would be invented. Each capture now writes a fixed-width sidecar index — time, offset, length — leaving the payload byte-exact for every reader that already parses it as HTTP. HAR bodies are bounded by their framing (no more “the connection died so the body is the rest of the file”).

Build a bundle you can send. One click packs the run’s shards, the rules, an anonymized config snapshot, the crash marker, and a manifest into a single diagnostics-… archive. Redaction is not optional.

The Case That Broke Open: an Offline Bundle

The runtime bundle is built by the process that runs the engine — on iOS, the packet-tunnel extension. Which means it exists only while the tunnel is up, and the moment a user most wants a bundle is the moment after the tunnel went down.

The app’s own report has the opposite shape: a host summary, the previous run’s crash marker, whatever logs the app can read, no live state. The two are now distinct kinds on every platform. An offline build packs only what it is handed: the host’s report, the previous run’s marker classified without adopting it (a marker whose heartbeat is still fresh describes a run that is alive, not a kill — the file alone cannot tell the two apart, so the age of the last write has to), and log shard tails. Same caps, same redaction, same manifest — but it exists now, after the crash, not whenever the engine deigned to run.

The Thing the Infrastructure Caught Immediately

The payoff was not theoretical. The same week the HAR exporter landed, it exposed a real bug: an HTTP-based tunnel’s response reader could outlive its own drain — a use-after-free that no amount of user testing had surfaced because the window is only a few instructions wide. We found it because the gate-run harness could finally see the shape of what the engine was doing at close time. A bug we would have chased for a sprint, diagnosed in a single run.

What This Means for You

Practically, two things:

  • Next time you report a problem, the answer is no longer “please screenshot the console.” It is “open the web console, hit Export Bundle, send the file.” If the extension already crashed, the app can still build one — that is the offline kind.
  • Next time we investigate, the question is no longer “can you reproduce it with a debugger attached.” It is “what does the run say.”

This is not a feature you will see in a marketing screenshot. It is the kind of work that makes every future feature more honest — because when something goes wrong, the engine can now tell you what it was doing.

The diagnostic bundle, the HAR export, the rule-attribution view, and the run-explainer page are all rolling out now on Chute iOS, macOS, and tvOS. The Chute Manual documents the bundle format and the console endpoints.

Thanks.

Chute Devs