A customer lit up their brand-new dedicated 1 Gbps link to the datacenter where everything gets backed up. First night of glory: the backup crawls along at exactly 100 Mbps. They call the provider; the provider runs an iperf with eight parallel streams and clocks 940 Mbps. "The link is perfect." And it is. It's also true that the backup runs at a tenth of what they're paying for. Nobody is lying and nothing is broken: that's TCP doing its job. TCP ships cargo only when the signed delivery slip comes back, and between that office and that datacenter the slip takes 20 ms to make the round trip. This guide is the invoice for that slip.
The signed delivery slip: why TCP waits
TCP promises something borderline miraculous: complete, in-order delivery over a network that drops packets and shuffles them. Its method is old-school warehouse clerk: number every byte and demand a receipt (the ACK). And since it can't ship infinite cargo without knowing what arrived, there's a cap on how much data can travel unconfirmed: the window. Think of a strict courier: he dispatches up to N boxes and, until the signed slip comes back from the recipient, he will not dispatch one more.
If the destination warehouse is next door, the slip comes right back and the flow never stops. If it's 20 ms away round-trip, here's what happens: pushing a 64 KB window down a 1 Gbps link takes a mere 0.52 ms… and waiting for the ACK takes 20 ms. The sender works 2.6% of the time and stares at the ceiling the remaining 97.4%. The link isn't slow — it's idle.
That gives us the most important formula in this guide: max throughput = window ÷ RTT. Notice what does not appear in it: the link's bandwidth. You can buy 10 Gbps and the math won't budge a single bit. That's why the first move on any "it's slow" ticket is never re-reading the contract — it's measuring the RTT.
The three delays that make up your RTT
That 20 ms RTT isn't divine punishment: it's the sum of three delays with very different personalities.
1. Serialization: the time it takes to push the bits onto the wire, one after another. A 1500-byte packet at 1 Gbps is 12,000 bits ÷ 10⁹ bps = 12 µs. At 100 Mbps, 120 µs; at 10 Gbps, 1.2 µs. You pay it at every hop that receives the full packet before forwarding it (store-and-forward), so a path with eight hops charges it eight times.
2. Propagation: physics, folks. Light in fiber doesn't travel at vacuum speed but at about 204,000 km/s (glass slows it down, refractive index ≈ 1.47). Hence the golden rule: ≈ 5 µs per kilometer. 100 km of fiber = 0.5 ms one way = 1 ms of RTT with nobody doing anything wrong. 2,000 km = 20 ms of RTT. And 8,000 km to a faraway datacenter = 80 ms of RTT on propagation alone. No upgrade fixes this one: it's distance, not equipment.
3. Queuing and processing: the moody one. Full buffers on a congested router, deep inspection on the firewall, a cheap switch with its CPU pegged. It's the only one of the three you can fight with QoS and proper sizing — and the only one that produces jitter.
The uncomfortable consequence: upgrading from 1 to 10 Gbps only improves serialization, which drops from 12 µs to 1.2 µs. Inside a 20 ms RTT, that's a rounding error. You can add up all three delays along your path, hop by hop, with the latency budget calculator.
BDP: your link is a pipe (and the window is the lid)
Now let's assemble the pieces. A link behaves like a pipe: the width is the bandwidth (how many bits per second fit) and the length is the RTT (how long they take to cross and get confirmed). The volume of that pipe is called the bandwidth-delay product (BDP): how many bytes must be "in flight" — sent but not yet acknowledged — for the pipe to run full. If your window is smaller than the BDP, the pipe travels with air in it.
Let's run the full numbers, exactly:
- You want to sustain 100 Mbps over a path with a 20 ms RTT. BDP = 100,000,000 bits/s × 0.020 s = 2,000,000 bits.
- 2,000,000 ÷ 8 = 250,000 bytes = 250 KB. That's how much must be traveling unacknowledged, at all times, so the pipe never runs dry.
- TCP's original window field is 16 bits: at most 64 KB (65,536 bytes). Throughput ceiling = 65,536 × 8 ÷ 0.020 s = 26,214,400 bps ≈ 26.2 Mbps.
- And here's the brutal part: it makes zero difference that the link is 1 Gbps. With a 64 KB window and 20 ms of RTT, the ceiling is still 26.2 Mbps. To fill 1 Gbps × 20 ms you need 2.5 MB in flight — 38 classic windows.
- The rescue is called window scaling (RFC 7323): an option negotiated in the handshake that multiplies the window by up to 2¹⁴, allowing windows of ~1 GB. Every operating system from this century ships it enabled — but one prehistoric middlebox stripping the option from the SYN sends you straight back to 1994.
How much window and how much RTT do you need to hit your target? That's exactly what the TCP throughput calculator computes: give it RTT, window and loss, and it returns the real ceiling of a single flow.
MTU and MSS: how much fits in each box
So far we've talked about how many boxes fly at once. Now, the size of each box. On Ethernet the maximum payload of a frame — the MTU — is 1500 bytes. Of those, the IP header takes 20 and the TCP header another 20: you're left with 1460 bytes of actual payload, the famous MSS (maximum segment size). So far, clean.
The problem is that every tunnel along the path charges a toll in bytes. PPPoE — the staple of residential GPON connections — inserts 8 bytes of session header and leaves the MTU at 1492. GRE takes 24. VXLAN, the datacenter standard, takes 50. And since the physical limit of the path is still 1500, everything the headers add gets deducted from your letter.
| Path | Usable MTU | MSS (IPv4) | Toll |
|---|---|---|---|
| Plain Ethernet | 1500 | 1460 | — |
| PPPoE (typical GPON) | 1492 | 1452 | 8 B |
| GRE | 1476 | 1436 | 24 B |
| VXLAN (over MTU 1500) | 1450 | 1410 | 50 B |
| IPsec tunnel (typical) | ≈1400 | ≈1360 | 50–73 B, cipher-dependent |
And what if a packet leaves bigger than the path can carry? Two scenarios, both ugly. With fragmentation, the router chops the packet into pieces: lose one fragment and TCP retransmits the whole packet, while reassembly burns CPU at the destination. Fragmentation isn't a workaround — it's slow poison. And with the DF bit set (the default in modern TCP), the router drops the packet and reports back via ICMP so the sender can try a smaller one — the mechanism known as Path MTU Discovery.
Finding the real MTU of a path takes thirty seconds with ping and the DF bit:
$ ping -M do -s 1472 vps.example.com From 192.168.1.1 icmp_seq=1 Frag needed and DF set (mtu = 1492) $ ping -M do -s 1464 vps.example.com 1472 bytes from vps.example.com: icmp_seq=1 ttl=55 time=19.8 ms
1464 of payload + 8 of ICMP + 20 of IP = 1492: there's PPPoE on the path. The full header math, for any stack of tunnels, is what the MTU and MSS calculator does.
Firewalls don't die of Gbps: they die of pps
Last piece of the puzzle, and the most expensive one to ignore. A network device doesn't process "megabits": it processes packets. Every packet — tiny or full-size — demands the same choreography: route lookup, session, policies, NAT. That's why the metric that kills firewalls is pps (packets per second), not Gbps.
The numbers speak for themselves. At 1 Gbps with minimum 64-byte frames (each occupies 84 B on the wire, counting preamble and interframe gap), the device faces 1.49 Mpps. With full 1518-byte frames, a mere 81,274 pps: 18 times less work for the same "bandwidth". The datasheet that says "1 Gbps firewall" almost always measured with big packets; your real traffic — VoIP, DNS, gaming, TCP ACKs — is full of small ones.
To size honestly there's IMIX, a statistical blend of real traffic: 7 frames of 64 B + 4 of 570 + 1 of 1518, average 353.83 bytes. At 1 Gbps that's about 334,000 pps — four times more demanding than the full-frame scenario. You can dissect per-frame overhead in the Ethernet frame and PPS calculator and size your edge device with the firewall sizing calculator.
Recipes: latency first, parallelism second
With all the physics on the table, the field recipes come down to three.
For interactive apps, low RTT beats high bandwidth. Opening an HTTPS session costs about 4 round trips before the first useful byte: at 10 ms of RTT that's 40 ms and feels instant; at 150 ms it's 600 ms and the user is already complaining. Remote desktop, ERPs, video calls: they all live off the RTT. Picking the closest datacenter is worth more than doubling the link.
When the RTT is unavoidable, parallelize. The window ÷ RTT limit is per flow: four parallel TCP connections, each with its own 64 KB window, move 104.9 Mbps where a single one moves 26.2. That's why the provider's iperf with -P 8 clocked 940 Mbps while the single-stream backup dragged its feet — and why serious transfer tools (parallel rsync, multithreaded robocopy, browsers with their 6 connections) open several flows.
And size with all three units. A link is bought in Gbps, saturated in pps, and exploited according to window ÷ RTT. Before signing off on an upgrade, run the transfer through the transfer time calculator and confirm which of the three is actually holding you back.
Next time a link "runs slow", don't start with the contract: measure the RTT, compute the BDP and check the path's MSS. In that order. Nine times out of ten, the gigabit was intact all along — what was missing was window.