Tuning the TCP stack in Ubuntu Operating System

Transmission Control Protocol and Internet Protocol (TCP/IP) is a standard set of protocols used by every network-enabled device. TCP/IP defines the standards to communicate over a network. TCP/IP is a set of protocols and is divided in two parts: TCP and IP. IP defines the rules for IP addressing and routing packets over network and provides an identity IP address to each host on the network. TCP deals with the interconnection between two hosts and enables them to exchange data over network. TCP is a connection-oriented protocol and controls the ordering of packets, retransmission, error detection, and other reliability tasks.

TCP stack is designed to be very general in nature so that it can be used by anyone for any network conditions. Servers use the same TCP/IP stack as used by their clients. For this reason, the default values are configured for general uses and not optimized for high-load server environments. New Linux kernel provides a tool called sysctl that can be used to modify kernel parameters at runtime without recompiling the entire kernel. We can use sysctl to modify and TCP/IP parameters to match our needs.

In this article, we will look at various kernel parameters that control the network. It is not required to modify all parameters listed here. You can choose ones that are required and suitable for your system and network environment.

It is advisable to test these modifications on local systems before doing any changes on live environment. A lot of these parameters directly deal with network connections and related CPU and memory uses. This can result in connection drops and/or sudden increases in resource use. Make sure that you have read the documentation for the parameter before you change anything.

Also, it is a good idea to set benchmarks before and after making any changes to sysctl parameters. This will give you a base to compare improvements, if any. Again, benchmarks may not reveal all the effects of parameter changes. Make sure that you have read the respective documentation.

Prerequisites

You will need root access.

Note down basic performance metrics with the tool of your choice.

TCP stack tune:

Follow these steps to tune the TCP stack:

  • Set the maximum open files limit:
$ ulimit -n # check existing limits for logged in user 
# ulimit -n 65535 # root change values above hard limits
  • To permanently set limits for a user, open /etc/security/limits.conf and add the following lines at end of the file. Make sure to replace values in brackets, <>:
<username> soft nofile <value> # soft limits 
<username> hard nofile <value> # hard limits

  • Save limits.conf and exit. Then restart the user session.

  • View all available parameters:

# sysctl -a
  • Set the TCP default read-write buffer:
# echo 'net.core.rmem_default=65536' >> /etc/sysctl.conf 
# echo 'net.core.wmem_default=65536' >> /etc/sysctl.conf
  • Set the TCP read and write buffers to 8 MB:
# echo 'net.core.rmem_max=8388608' >> /etc/sysctl.conf 
# echo 'net.core.wmem_max=8388608' >> /etc/sysctl.conf
  • Increase the maximum TCP orphans:
# echo 'net.ipv4.tcp_max_orphans=4096' >> /etc/sysctl.conf
  • Disable slow start after being idle:
# echo 'net.ipv4.tcp_slow_start_after_idle=0' >> /etc/sysctl.conf
  • Minimize TCP connection retries:
# echo 'net.ipv4.tcp_synack_retries=3' >> /etc/sysctl.conf 
# echo 'net.ipv4.tcp_syn_retries =3' >> /etc/sysctl.conf
  • Set the TCP window scaling:
# echo 'net.ipv4.tcp_window_scaling=1' >> /etc/sysctl.conf
  • Enable timestamps:
# echo 'net.ipv4.tcp_timestamp=1' >> /etc/sysctl.conf
  • Enable selective acknowledgements:
# echo 'net.ipv4.tcp_sack=0' >> /etc/sysctl.conf
  • Set the maximum number of times the IPV4 packet can be reordered in the TCP packet stream:
# echo 'net.ipv4.tcp_reordering=3' >> /etc/sysctl.conf
  • Send data in the opening SYN packet:
# echo 'net.ipv4.tcp_fastopen=1' >> /etc/sysctl.conf
  • Set the number of opened connections to be remembered before receiving acknowledgement:
# echo 'tcp_max_syn_backlog=1500' >> /etc/sysctl.conf
  • Set the number of TCP keep-alive probes to send before deciding the connection is broken:
# echo 'tcp_keepalive_probes=5' >> /etc/sysctl.conf
  • Set the keep-alive time, which is a timeout value after the broken connection is killed:
# echo 'tcp_keepalive_time=1800' >> /etc/sysctl.conf
  • Set intervals to send keep-alive packets:
# echo 'tcp_keepalive_intvl=60' >> /etc/sysctl.conf
  • Set to reuse or recycle connections in the wait state:
# echo 'net.ipv4.tcp_tw_reuse=1' >> /etc/sysctl.conf 
# echo 'net.ipv4.tcp_tw_recycle=1' >> /etc/sysctl.conf
  • Increase the maximum number of connections:
# echo 'net.ipv4.ip_local_port_range=32768 65535' >> /etc/sysctl.conf
  • Set TCP FIN timeout:
# echo 'tcp_fin_timeout=60' >> /etc/sysctl.conf

How TCP stack works:

The behavior of Linux kernel can be fine tuned with the help of various Linux kernel parameters. These are the options passed to the kernel in order to control various aspects of the system. These parameters can be passed while compiling the kernel, at boot time, or at runtime using the /proc filesystem and tools such as sysctl.

In this recipe, we have used sysctl to configure network-related kernel parameters to fine tune network settings. Again, you need to cross check each configuration to see if it’s working as expected.

Along with network parameters, tons of other kernel parameters can be configured with the sysctl command. The -a flag to sysctl will list all the available parameters:

$ sysctl -a

All these configurations are stored in a filesystem at the /proc directory, grouped in their respective categories. You can directly read/write these files or use the sysctl command:

ubuntu@ubuntu:~$ sysctl fs.file-max 
fs.file-max = 98869 
ubuntu@ubuntu:~$ cat /proc/sys/fs/file-max 
98869

Related Articles