200 OK IT'S THE LATENCY, STUPID. OH, AND PACKET LOSS TOO. The Problems with 3G Mobile Networks.

Posted by Mark Lindsey


All the carriers brag about their 3G Networks. I mostly attend to AT&T 3G and Sprint EV-DO; these are the two carriers I use.

Email and file transfer performance is usually disappointing. Transfer speeds as measured by the Ookla "Speedtest" application vary wildly, but the round-trip-time (RTT) latency is consistently quite high.

I recently did a study for a client about the effects of network delay in cross-country file transmission. With fixed networks (such as normal Internet links), cross-country latency is often 40 to 50 ms; much of that time due to the speed of light to get from New York to Seattle, for example. So the round-trip time (RTT) is 80 ms to 100 ms, typically. The overall goal is to transfer a 4 GB file across the US in a matter of minutes. We demonstrated through real lab tests that nontrivial delays under 100 ms do slow TCP transfers, so that FTP will not get the job done. The effects are exaggerated as the network throughput increases. E.g., at 100 Mbps, a link with 16 ms RTT slows to about 52 Mbps (assuming all the TCP buffers/windows are tuned properly.)

Many national labs, such as LANL and CERN, deal with problems of large data transfers. Their scientific instruments output gigabytes of data within seconds, and they have to transport it across the network or around the world. They have the money to buy 100 Mbps links from the US to Switzerland.

But many ordinary folks are on the other end of the spectrum, below 1 Mbps, and experiencing large delays. The vaunted 3G networks fall into this category: even with 3 Mbps or better downstream paths, the large delays can also kill performance.

A reasonable AT&T 3G speed is around 350 Kbps, i.e., 0.35 Mbps. Consider these lab tests with varying levels of delay:

  • 350 Kbps downstream, 45 Kbps upstream, 0 ms RTT delay, 10 kB TCP window: 330 Kbps TCP connection throughput
  • 350 Kbps downstream, 45 Kbps upstream, 200 ms RTT delay, 10 kB TCP window: 230 Kbps TCP connection throughput
  • 350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 10 kB TCP window: 135 Kbps TCP connection throughput
  • 350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window: 309 Kbps TCP connection throughput

The TCP window size is normally configured in the operating system by default; 64 kB (i.e., 65,536 bytes) is a typical value.

In the AT&T 3G and Sprint EV-DO networks, 400 ms RTT to the first-hop router is quite typical. Consider these examples with higher capacity and 400 ms RTT:

  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 10 kB TCP window: 150 Kbps TCP connection throughput
  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window: 430 Kbps TCP connection throughput
  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 64 kB TCP window, 2% downstream packet loss: 220 Kbps TCP connection throughput
  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 527 kB TCP window: 600 Kbps TCP connection throughput
  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 782 kB TCP window: 650 Kbps TCP connection throughput
  • 1,350 Kbps downstream, 45 Kbps upstream, 400 ms RTT delay, 3500 kB TCP window: 980 Kbps TCP connection throughput

As you can see, as network delay increases, the TCP window must also increase. But even a tiny amount of packet loss -- 2%, fully reasonable in my tests of these networks -- can slow things down as well.

Conclusions

  • The TCP Buffer sizes need to be tuned to 3G networks, not just high-performance networks: OS Vendors need to dynamically adjust TCP buffer sizing based on the first-hop round trip time.
  • Users may need to explore non-TCP mechanisms to fully exploit network transport.
  • High latency and even moderate packet loss can significantly reduce the usefulness of 3G wireless networks.

I performed these lab tests using two Mac OS X 10.5 machines and the Dummynet Delay/bandwidth emulator in the kernel, using 1 MB buffers on each end of the link. iperf was used to generate traffic.