Optimize Kernel TCP Parameters on Linux to Improve System Performance

Kernel optimization, like server optimization, should follow the principles of stability and security. Using a Squid server as an example, once the client and server establish a TCP/IP connection, the Socket will then be closed, and the port state on the server side will become TIME_WAIT. So, do all Sockets that perform an active close enter the TIME_WAIT state? Is there any situation in which a Socket performing an active close can go directly into the CLOSED state? The answer is that the side performing the active close enters the TIME_WAIT state after sending the final ACK and remains there for 2MSL (Maximum Segment Lifetime). This is an essential part of TCP/IP, which means this cannot be “solved.”
The TCP/IP designers made it this way for two main reasons:

• To prevent packets from the previous connection from getting lost and then reappearing, which could affect a new connection (after 2MSL, all duplicate packets from the previous connection will have disappeared).
• To ensure the reliable closure of a TCP connection. The final ACK (FIN) sent by the side performing the active close may be lost. If it is lost, the passive side will resend the FIN. At that point, if the active side is in the CLOSED state, it will respond with RST instead of ACK. Therefore, the active side must remain in the TIME_WAIT state rather than CLOSED. In addition, TIME_WAIT does not consume many resources unless the system is under attack.

On a Squid server, you can enter the following command to view the current connection statistics:

netstat -n | awk '/^tcp/ {++S[$NF]} END{for(a in S) print a, S[a]}'

The command output is shown below:

LAST_ACK 14
SYN_RECV 348
ESTABLISHED 70
FIN_WAIT1 229
FIN_WAIT2 30
CLOSING 33
TIME_WAIT 18122

The meanings of the states in the command are as follows.

• CLOSED: No active or ongoing connection.
• LISTEN: The server is waiting for an incoming call.
• SYN_RECV: A connection request has arrived and is awaiting acknowledgment.
• SYN_SENT: The application has started and opened a connection.
• ESTABLISHED: Normal data transfer state.
• FIN_WAIT1: The application says it has finished.
• FIN_WAIT2: The other side has agreed to release.
• ITMED_WAIT: Waiting for all packets to expire.
• CLOSING: Both sides are attempting to close at the same time.
• TIME_WAIT: The other side has initiated a release.
• LAST_ACK: Waiting for all packets to expire.

In other words, this command can classify and summarize the current system’s network connection status.

On high-concurrency Squid servers running on Linux, the number of TCP
TIME_WAIT sockets can often reach twenty to thirty thousand, and the server can easily be dragged down. However, you can reduce the number of TIME_WAIT sockets on a Squid server by modifying Linux kernel parameters, as follows:
vim /etc/sysctl.conf
Then add the following parameters:

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 1200
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000
net.ipv4.tcp_max_syn_backlog = 8192
net.ipv4.tcp_max_tw_buckets = 5000

The following briefly explains the meaning of each parameter above:

• net.ipv4.tcp_syncookies＝1 means enabling SYN
  Cookies. When the SYN wait queue overflows, cookies are enabled for handling it, which can help defend against a small number of SYN attacks. The default value of this parameter is 0, which means disabled.
• net.ipv4.tcp_tw_reuse＝1 means enabling reuse; that is, allowing TIME-WAIT sockets to be reused for new TCP connections. The default value of this parameter is 0, which means disabled.
• net.ipv4.tcp_tw_recycle＝1 means enabling fast recycling of TIME-WAIT sockets in TCP connections. The default value of this parameter is 0, which means disabled.
• net.ipv4.tcp_fin_timeout＝30 means that if the socket is closed by the local end, this parameter determines how long it remains in the FIN-WAIT-2 state.
• net.ipv4.tcp_keepalive_time＝1200 means that when Keepalived is enabled, the frequency of TCP Keepalived messages is changed to 20 minutes; the default value is 2 hours.
• net.ipv4.ip_local_port_range＝1000065000 indicates the port range used by the CentOS system for outbound connections. Its default value is quite small, so here it is changed to 10000 to 65000. It is recommended not to set the minimum value here too low, otherwise it may occupy normal ports.
• net.ipv4.tcp_max_syn_backlog＝8192 indicates the length of the SYN queue. The default value is 1024; increasing it here to 8192 allows more pending network connections to be accommodated.
• net.ipv4.tcp_max_tw_buckets＝5000 indicates the maximum number of TIME_WAIT sockets the system can keep at the same time. If this number is exceeded, TIME_WAIT sockets will be cleared immediately and a warning message will be printed. The default value is 180000, and here it is changed to 5000. For servers such as Apache and Nginx, the several parameters introduced above can already reduce the number of TIME_WAIT sockets quite effectively, but for Squid the effect is not very significant. With this parameter, you can control the maximum number of TIME_WAIT sockets and prevent the Squid server from being dragged down by a large number of TIME_WAIT sockets.
  Run the following command to make the kernel configuration take effect immediately:
  /sbin/sysctl –p
  If it is used for a Web server such as Apache or Nginx, then you only need to change the following items:

net.ipv4.tcp_syncookies=1
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000

Run the following command to make the kernel configuration take effect immediately:
/sbin/sysctl –p
If it is a Postfix mail server, the following kernel optimization settings are recommended:

net.ipv4.tcp_fin_timeout = 30
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
net.ipv4.ip_local_port_range = 10000 65000
kernel.shmmax = 134217728

Run the following command to make the kernel configuration take effect immediately:
/sbin/sysctl –p
Of course, these are only the most basic changes. You can also modify the kernel settings according to your own needs. For example, on our production machines under high concurrency, the error “TCP: too many
orpharned
sockets” often appears. At the same time, you should always follow the highest principle of server stability. If the server is unstable, all work and effort will be in vain. If the above optimizations still cannot meet operational requirements, you may need to customize your server kernel or upgrade the server hardware.