Automatic TCP Buffer Tuning

Automatic TCP Buffer Tuning
Jeffrey Semke
Jamshid Mahdavi
Matthew Mathis
Pittsburgh Supercomputing Center

September 4, 1998
PSC/CMU/NLANR http://www.psc.edu/networking/auto.html 1
Acknowledgments
● Work made possible by funding from the
National Science Foundation
● Much assistance from
– Greg Miller and the MCI vBNS Team
– Kevin Lahey (NASA Ames)
– kc claffy & Jambi Ganbar (SDSC)
High Speed File Transfers
Server
Client
vBNS
10 MB File Transfer
Path Properties:
- 155 Mb/s Bandwidth
- 68 msec Round Trip Time
Expected Performance
● Transfer time = 10 MB/(155 Mb/s) = .5 sec
● Typical transfer time: 43 sec (!)
● Why? TCP Default tuning limits file
transfer to a window of 16kB
Default Tuning:
Performance vs. Delay
TCP Assumptions
● Assume that TCP/IP stack already includes
modifications for higher performance:
– RFC 1191 Path MTU Discovery
– RFC 1323 TCP Large Windows
– RFC 2018 Selective Acknowledgment (SACK)
Socket Buffers
Sender Receiver
Applic. Applic.
Data Data
Send Recv
Data 40 Socket Data 40
Socket
Buffer Buffer Data 23
Data 22
Data 0 Data 40 Data 25 Data 0

Data 20 Data 23
Network
TCP/IP TCP/IP
Under-buffered send socket buffer
Sender Receiver
Applic.
Data
Send
Data 100 Socket
Buffer Ack 20 Ack 21 Ack 22
Data 0 Data 26
Data 20
Data
26
TCP/IP TCP/IP
Hand Tuning:
● Increase sender and receiver socket buffers
to enable high performance.
● Required socket buffer size is 1 to 2 times
BW * Delay
● In this example, 2 * 155 Mb/s * 68 msec =
2.6 MB
● Can be done system wide, or by each user.
Problems with Hand Tuning
● Per User:
– Requires users to be network wizards.
● System Wide:
– All connections default to same buffer size
– Can result in overuse of system memory
(specifically MBUF Clusters) and in some
cases crashes.
Over-buffered send socket buffer
Sender Receiver
Send
Applic.
Socket
Data
Buffer
Data
999,999 Data Ack 20 Ack 21 Ack 22
999,999
Data 28
Data 0 Data 20 Data Data Data
28 27 26
TCP/IP TCP/IP
System Memory Usage
Goals of Sender Autotuning
● Each TCP connection gets the best possible

performance without any configuration
● Use system memory more effectively and
prevent memory starvation
● No need to modify existing applications
Congestion Window
● Sender uses Congestion Avoidance
algorithm to attempt to find the appropriate
window
– Factors:
● Distance between hosts (delay)
● Bottleneck link rate (bandwidth)
● Queue lengths of routers between hosts
Sender auto-tuning based on cwnd
Sender Receiver
Send
Applic.
Socket
Data
Buffer
Data
999,999 Data Ack 20 Ack 21 Ack 22
20 + S
Data 28
Data 0 Data 20 Data Data Data
28 27 26
TCP/IP TCP/IP
2*cwnd <= S < 4*cwnd
sb_net_target vs. cwnd over time
Fair Share Algorithm
● sb_net_target may not be attainable for all
connections
● Fair share is periodically calculated
● Small connections (sb_net_target < fair share)
donate unused memory to the pool
● Large connections (sb_net_target >= fair share) are
limited to the fair share.
Illustration of fair share algorithm
Memory pool for TCP sender socket buffers
conn0 conn1 conn2 conn3
conn4
conn5
conn6
= Actual allocation = Desired allocation

Test Topology
40 Mbps vBNS
R R
MAN Testnet
R
100Mbps R
FDDI
ring FDDI ring
10Mbps ether.
NetBSD
sender Pittsburgh San
Diego
DEC Alpha DEC Alpha

Receiver Receiver
Tuning Types
● Under-buffered (default) tuning
– 16kB send socket buffer
● Over-buffered (hiperf) tuning
– Hand tuned for SDSC path
● 68 msec * 40 Mb/s = 340 kB
– 400kB send socket buffer
● Automatically tuned buffers (auto)
Basic Functionality Test
● From 1 to 30 concurrent connections from
sender in Pittsburgh to receiver in San
Diego
● 40 Mbps bottleneck link, 68 ms delay
– 340 kB bandwidth delay product
Basic Functionality
Basic Functionality
Basic Functionality
System Memory Usage
Diverse concurrent connections
● Half the connections go to local receiver,

while half go to remote receiver
● Local connections have a 10Mbps
bottleneck link, with ~1ms delay
– 1.25 kB bandwidth delay product
System Memory Usage
Impact
Conclusion
● Better performance
● More efficient use of memory
● More concurrent connections
● Great for servers that have many
connections over very diverse
bandwidth*delay paths

Automatic TCP Buffer Tuning

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Automatic TCP Buffer Tuning

Uploaded by

Copyright:

Available Formats

Automatic TCP Buffer Tuning

Pittsburgh Supercomputing Center

Data 0 Data 40 Data 25 Data 0

● Each TCP connection gets the best possible

● Queue lengths of routers between hosts

conn0 conn1 conn2 conn3

= Actual allocation = Desired allocation

DEC Alpha DEC Alpha

● Half the connections go to local receiver,

You might also like