« Thinking about my visit to The Library | Main | Networks and Government »
March 27, 2003
Jumbo Frames and Jumbo Bandwidth
I've been researching lately some of the limitations to fully utilizing a Gigabit (or faster) network.
The congestion-avoidance algorithms of TCP create one very significant limitation. The equation for the maximum performance on a given TCP session is:
TCP Throughput <= ~0.7 * MSS / (rtt * sqrt(packet_loss))
MSS = Maximum segment size, usually 1460 on an Ethernet network
RTT = round-trip (end-to-end) time, in milli-seconds
On a typical TCP transfer with RTT of 40ms (very typical), the maximum throughput is 6 Mb/sec. Upgrading to GigE or 10GbE, upgrading the server and workstation--the upper limit is still 6Mb/sec. And performance issues on the workstation, server or within the network result in typical peformance much lower.
Throughtput is dramatically increased by reducing RTT (sometimes, but not often, possible through peering or local caching), or more likely, by increasing the MSS. A Jumbo Frame MSS of 9000 bytes would increase maximum throughput to 40 Mb/sec.
UDP and other protocols don't necessarily have these limitations, but any protocol or application with error-control and congestion-avoidance mechanisms (which any widely-deployed protocol has to have on a statistically-multiplexed network like the Internet) will have similar performance limitations.
Another critical benefit of Jumbo Frames is lower processing requirements, on the workstation, server and network infrastructure (switches and routers). With 1500-byte packets, a full 1000 Mb/s would be 83,330 packets/second (or 833,330/sec at 10GbE). With 9000-byte packets, it's only 13,890 packets/sec. Jumbo frames use 1/6th the processing power of 1500-byte frames, which is absolutely necessary to ensure that the workstations, servers, routers and switches can actually achieve Gigabit speeds and higher.
Since GigE is still usually deployed just within the core of a network, these issues haven't received a lot of attention yet. However, with most workstations now shipping with 10/100/1000 network cards, and networks like Abilene encouraging wide-scale deployment of jumbo frames (the median bulk TCP flow throughput on Abilene, a high-speed research network, is 1.9Mb/s), and users pushing higher-speed applications, and the wide-scale availability of broadband and high-speed networks--all of these point to the inevitability that jumbo frames will be an absolute necessity within the next couple of years, particularly within networks that are aggressively deploying GigE.
We are testing jumbo frames over the next couple of weeks, in preparation for deploying jumbo frames within the University of Utah campus and hopefully in some parts of the UEN network (where the equipment supports jumbo frames). As we deploy 100Mb and 1000Mb Ethernet over the next few months, jumbo frames will be an important capability of the new network, to ensure that users will be able to utilize this additional bandwidth.
Posted by pete at March 27, 2003 10:13 PM