Hello everyone.
A download through the tunnel on a link with 100 ms of round trip was moving at about half a megabyte a second. Four downloads sharing one connection were slower together than one of them alone. Nothing else looked wrong: the server was healthy, the handshake was fine, and the CPU was half idle.
The problem was two numbers that should have been the same size and were not.
Two Windows, Not One
HTTP/2 gives every stream a receive window, and gives the connection a receive window of its own. They are separate budgets. A sender may have plenty to put on a stream, but it can only send as much as both windows allow.
The defaults are modest: 65,535 bytes each. A client that wants more says so. For a stream, the number goes in SETTINGS. For the connection there is no SETTINGS field at all: the connection window starts at 65,535 and only grows when the client sends a WINDOW_UPDATE for the connection as a whole.
That asymmetry is the trap. We announced a 1 MiB window for every stream and left the connection at the protocol’s default. On paper each stream could have a megabyte in flight. In practice one connection carried 40–60 KiB per round trip for all of its streams together, because that was all the connection window allowed.
At 100 ms that is 0.4–0.6 MiB/s. At 200 ms it is 0.2. Four sessions sharing one connection managed 0.35 together — worse than a single session with a connection of its own. The streams were not slow. They were queued behind a budget nobody had raised.
It is easy to see why the stream window got all the attention. A stream is one user, one request, one visible thing; the connection is the plumbing. But a proxy is exactly the kind of program that puts many sessions behind one connection, so the plumbing is where the traffic goes.
Raising the Connection
The session now raises the connection’s receive window right after its SETTINGS: 16 MiB on macOS, 4 MiB on iOS and tvOS. The number sounds large, but a window is credit, not memory. What a connection can actually hold is still bounded by its streams’ windows and by how fast the reader drains them. Raising it allocates nothing.
We also let a request ask for a receive window of its own, and change it while the stream runs. Two boundaries matter here. A stream cannot be opened below the SETTINGS value — only raised from it — and the HTTP/2 library only knows a stream once its headers have gone out, so a window asked for earlier is kept and applied when the send buffer drains. One of our HTTP-carried transports uses this to grow its download window as the reader proves it can keep up; XHTTP’s streams stay at 1 MiB.
Measured in the test harness with 100 ms of round trip injected:
| Before | After | |
|---|---|---|
| HTTP-carried transport, iOS budget | 0.4–0.6 MiB/s | 9.7 MiB/s |
| same, macOS budget | 0.4–0.6 MiB/s | 19.3 MiB/s |
| four sessions on one connection | 0.35 MiB/s | 20.8 MiB/s |
| XHTTP over HTTP/2 | pinned at 64 KiB per round trip | 5.4 MiB/s |
| gRPC over HTTP/2 | pinned at 64 KiB per round trip | 4.9 MiB/s |
The last two run on the same session code and the same fix; they had been paying the same tax all along.
The Other Side of the Decryption
HTTPS decryption has the same problem from the other direction. When Chute decrypts a connection, it is a server to the app and a client to the origin. Its connection to the origin was running on the protocol’s default windows — 64 KiB for the connection and for each stream — so an 8 MiB download cost about 130 round trips to the origin no matter how fast the phone could take it.
The connection window is now the raised one. A stream starts at 64 KiB and doubles when the reader has taken half of it within a second and holds less than a quarter, up to 4 MiB (1 MiB on iOS and tvOS). Everything above 64 KiB comes out of one budget for the process, the way our HTTP-carried transport already budgets its downloads. The same 8 MiB now takes 10 round trips. The mobile numbers are budgeted the same way but have not been measured on a device.
A reader that stops leaves at most its earned window behind. A stream that ends with bytes its own client has not taken hands its share to the others; it goes back as they drain, when the stream is cleaned up, and when the interceptor closes.
The Parts That Bite
Three details cost more time than the arithmetic.
Credit cannot be taken back. When a new session needs part of the budget, we shrink the window of a download that is already running. But the peer may already have sent what we granted; HTTP/2 has no way to un-send credit. Returning the budget immediately would let the shrunk session still receive its old window while the freed budget went to someone else — up to twice the budget in flight. So the difference is recorded as owed and repaid a byte at a time as the reader drains, and cleared when the request ends.
A queued request is not a stalled one. A peer can cap concurrent streams. When it does, the library holds a request’s headers until a slot frees. Our watchdog used to start counting from the moment we handed the request over, so a request waiting in someone else’s line was declared stuck, its connection retired, and every other session on that connection paid for it. The transport now reports when the headers actually go out, and the clock starts there — with a ceiling, so a peer whose limit is below what the connection always holds cannot keep a request forever.
A graceful GOAWAY is ordinary. nginx sends one every thousand requests on a keep-alive connection. After it, the library will still hand out stream ids for new requests and then drop them when their turn comes; because the stream never existed, nobody is told. We now stop opening streams on that connection, raise one notice, retire it from the pool, and let the accepted streams finish.
What It Means
Flow control is not a detail to leave at the default. An HTTP/2 connection is a shared budget, and the default budget was sized for a browser on a good day, not for several tunnels, XHTTP, gRPC and the decrypted side of HTTPS sharing one session class.
The numbers here come from the harness, not from a phone on a desk. But the shape is not subtle: four sessions slower than one is a bug you can feel, and a window that never grows is a bug you can measure.
Thanks.
Chute Devs