How keepalive_timeout directive works in Nginx?

实验观察Nginx中keepalive_timeout参数的具体作用

2016-09-13

Background

In TCP/IP, when a TCP connection is established, both client and server can send a FIN request to end the connection. Client wants to keep the ESTABLISHED state the long the better, for there are so many files are needed to get, such as HTML/JS/CSS/Image, etc.

However, In the interest of the server, the ESTABLISHED state the short the better, because the there are many other clients are waiting for service and the resources of a server is limited.

So, generally, after all the files have been downloaded, the browser will not close the connection actively, and the server often close the connection actively.

How to close a connection after a certain amount of time in Nginx ? the keepalive_timeout directive kicks in.

Syntax:	keepalive_timeout timeout [header_timeout];
Default:	
keepalive_timeout 75s;
Context:	http, server, location

The first parameter sets a timeout during which a keep-alive client connection will stay open on the server side. The zero value disables keep-alive client connections. The optional second parameter sets a value in the “Keep-Alive: timeout=time” response header field. Two parameters may differ.

The “Keep-Alive: timeout=time” header field is recognized by Mozilla and Konqueror. MSIE closes keep-alive connections by itself in about 60 seconds.

在默认情况下，浏览器向nginx发起请求时，http请求头中设置的是connection: keep-alive，可以利用这个连接来传输多个文件，例如js/html/css/jpg等，从而避免重复建立多个连接的开销，所以默认是keepalive，所以在通信完成后，浏览器不会主动关闭连接的。即使客户端设置了保持连接，服务器也可以主动关闭此连接(这里暂时先不讨论)。

这样就产生一个问题，在高并发场景下，短时间内会产生大量的这样的连接，每个连接会占用一定的服务器资源（主要占用内存，占用打开文件数，同时如果是代理的话，要考虑nginx向后端服务，例如php发起连接，这样的连接应该不多），这不是我们希望看到的，服务端希望客户端如果完成访问，就尽快释放连接，避免长时间消耗资源。

所以调整一个合适值，对于服务效能很重要，一方面要避免重复建立连接的开销，另一方面要避免客户端长期占用连接资源不放。

关于nginx中keepalive_timeout的用法：

使用环境：http, server, location

使用格式：keepalive_timeout arg1 [arg2]

keepalive_timeout接收2个参数:

arg1必选：nginx向客户端发起断开连接的超时时间
arg2可选：nginx会在响应头信息中设置Keep-Alive:timeout=arg2，让客户端主动关闭连接。

所以有些情况下，浏览器会忽略这个设置。下面的情况可以看到，设置了arg2后并无效果。

该指令定义一个keep-alive的时长，keep-alive能够使客户端到服务器的连接在一定时间内持续有效。在这个时间内，客户端对服务器的访问不需要再次建立连接或重新连接。

Experiment Environment

Client IP：192.168.0.16
Server IP：192.168.0.21

Setup a Nginx server, which serves a html page, we are going to focus on TCP protocol.

# file: nginx.conf
http {
  keepalive_timeout  15;
  #这里单位是s
}

Using tcpdump on Nginx Server

nginx start at port 80，using tcpdump

$ tcpdump tcp port 80

Experiment 1：keepalive_timout: 15s

The following package shows that client sents ESTABLISH request to the server at 11:53:12, after 15s, the server at 11:53:27, send FIN to clinet. 客户端在12s时刻发起请求，27s时刻nginx向客户端发送FIN，请求断开连接。

11:53:12.013881 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [S], seq 1647038819, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
11:53:12.013953 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [S.], seq 2440006518, ack 1647038820, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
11:53:12.014227 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [.], ack 1, win 16425, length 0
11:53:12.015674 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [P.], seq 1:534, ack 1, win 16425, length 533
11:53:12.015720 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [.], ack 534, win 123, length 0
11:53:12.015908 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [P.], seq 1:173, ack 534, win 123, length 172
11:53:12.213230 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [.], ack 173, win 16382, length 0
11:53:27.031171 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [F.], seq 173, ack 534, win 123, length 0
11:53:27.031388 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [.], ack 174, win 16382, length 0


11:54:12.036632 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [.], seq 533:534, ack 174, win 16382, length 1
11:54:12.036678 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [.], ack 534, win 123, length 0
11:54:57.028996 IP 192.168.0.16.52119 > 192.168.0.21.http: Flags [.], seq 533:534, ack 174, win 16382, length 1
11:54:57.029037 IP 192.168.0.21.http > 192.168.0.16.52119: Flags [R], seq 2440006692, win 0, length 0

Exp 2：keepalive_timout:6s

The following package shows that client sents ESTABLISH request to the server at 16:13:07, after 6s, the server at 16:13:13, send FIN to clinet.

客户端在07s时刻发起请求，13s时刻nginx向客户端发送FIN，请求断开连接。

16:13:07.986580 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [S], seq 141352550, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:13:07.986653 IP 192.168.0.21.http > 192.168.0.16.53518: Flags [S.], seq 3684334522, ack 141352551, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:13:07.986853 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [.], ack 1, win 16425, length 0
16:13:07.988210 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [P.], seq 1:534, ack 1, win 16425, length 533
16:13:07.988239 IP 192.168.0.21.http > 192.168.0.16.53518: Flags [.], ack 534, win 123, length 0
16:13:07.988911 IP 192.168.0.21.http > 192.168.0.16.53518: Flags [P.], seq 1:173, ack 534, win 123, length 172
16:13:08.186352 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [.], ack 173, win 16382, length 0
16:13:13.995178 IP 192.168.0.21.http > 192.168.0.16.53518: Flags [F.], seq 173, ack 534, win 123, length 0
16:13:13.995424 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [.], ack 174, win 16382, length 0
16:13:55.937133 IP 192.168.0.16.53518 > 192.168.0.21.http: Flags [F.], seq 534, ack 174, win 16382, length 0
16:13:55.937168 IP 192.168.0.21.http > 192.168.0.16.53518: Flags [.], ack 535, win 123, length 0

Exp 3：keepalive_timout: 0s

The connection will be closed by the server as soon as the data transfer was completed.

那么长连接在传输结束后就会被关闭。返回时设置头Connection: close

16:36:33.526022 IP 192.168.0.16.53988 > 192.168.0.21.http: Flags [S], seq 453414335, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:36:33.526097 IP 192.168.0.21.http > 192.168.0.16.53988: Flags [S.], seq 1445206998, ack 453414336, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:36:33.526391 IP 192.168.0.16.53988 > 192.168.0.21.http: Flags [.], ack 1, win 16425, length 0
16:36:33.528002 IP 192.168.0.16.53988 > 192.168.0.21.http: Flags [P.], seq 1:534, ack 1, win 16425, length 533
16:36:33.528032 IP 192.168.0.21.http > 192.168.0.16.53988: Flags [.], ack 534, win 123, length 0
16:36:33.528402 IP 192.168.0.21.http > 192.168.0.16.53988: Flags [P.], seq 1:168, ack 534, win 123, length 167
16:36:33.528497 IP 192.168.0.21.http > 192.168.0.16.53988: Flags [F.], seq 168, ack 534, win 123, length 0
16:36:33.528748 IP 192.168.0.16.53988 > 192.168.0.21.http: Flags [.], ack 169, win 16383, length 0
16:36:33.529182 IP 192.168.0.16.53988 > 192.168.0.21.http: Flags [F.], seq 534, ack 169, win 16383, length 0
16:36:33.529200 IP 192.168.0.21.http > 192.168.0.16.53988: Flags [.], ack 535, win 123, length 0

Exp 4：keepalive_timeout 30s, Keep-Alive:timeout=5;

keepalive_timeout 30 5;

16:42:32.941249 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [S], seq 1359017244, win 8192, options [mss 1460,nop,wscale 2,nop,nop,sackOK], length 0
16:42:32.941316 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [S.], seq 2951328040, ack 1359017245, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
16:42:32.941524 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [.], ack 1, win 16425, length 0
16:42:32.947169 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [P.], seq 1:534, ack 1, win 16425, length 533
16:42:32.947218 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [.], ack 534, win 123, length 0
16:42:32.947747 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [P.], seq 1:196, ack 534, win 123, length 195
16:42:33.148722 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [P.], seq 1:196, ack 534, win 123, length 195
16:42:33.148886 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [.], ack 196, win 16376, options [nop,nop,sack 1 {1:196}], length 0
16:43:02.979017 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [F.], seq 196, ack 534, win 123, length 0
16:43:02.979265 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [.], ack 197, win 16376, length 0
16:43:21.010604 IP 192.168.0.16.54128 > 192.168.0.21.http: Flags [F.], seq 534, ack 197, win 16376, length 0
16:43:21.010644 IP 192.168.0.21.http > 192.168.0.16.54128: Flags [.], ack 535, win 123, length 0

这里看到第二个参数并没有让客户端自己发起关闭。也就是并没有起作用，存疑待考。

同时在客户端Windows使用wireshark抓的包

备注：这里wireshark抓的包没有tcpdump详细，并没有显示出，我这里使用的过滤参数为：ip.addr == 192.168.0.21 && tcp.port == 80

keepalive_timout 为15s的情况

keepalive_timout 为6s的情况

Summary of the C/S Communication Process

  1. Clinet send syn to server
  2. Data transfer
  3. keepalive_time time is up，nginx send fin to clinet，the connection closed

After the connection is closed, the ESTABLISHED status will disappear，using netstat -antp | grep nginx

tcp        0      0 192.168.0.21:80             192.168.0.16:53424          ESTABLISHED 23034/nginx

keepalive_timeout什么时候会触发执行？

keepalive_timeout value  =  the time of `FIN` sent by server - the time of `SYN` start from client

如果连接断开，客户端会重新请求开启连接。

几个疑问

  • wireshark并没有显示双向通信过程，暂时不知怎么设置参数。
  • wireshark最后回有一个RST的包，不知是何故，而tcpdump并没有这样的情况。所以建议还是以tcpdump命令为准。
  • 如果keepalive_timout设置为0，则响应头信息返回Connection: close，连接立即关闭。
  • 第二个参数并没有起作用