Firstly I noticed multiple entries of the following in kern.log:
nf_conntrack: table full, dropping packet
After checking the existing table size:
# /sbin/sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 76768
…it seemed sensible to double it:
# cat /proc/sys/net/nf_conntrack_max
65526
# sysctl -w net.netfilter.nf_conntrack_max=131072
# echo net.netfilter.nf_conntrack_max=131072 >> /etc/sysctl.conf
OK, this gave me some more headroom, but my munin graphs still aren’t being rendered for fw_conntrack. Checking /var/log/munin/munin-node.log showed the following every 5 minutes:
2015/10/27-11:35:15 [1788] Service 'fw_conntrack' timed out.
2015/10/27-11:40:15 [2954] Service 'fw_conntrack' timed out.
Each munin graph has a 5 second timeout, according to /usr/share/munin/plugins/fw_conntrack , /proc/net/ip_conntrack is munged to build the graph. How long does that take to cat?:
# time cat /proc/net/nf_conntrack | wc -l
79590
real 0m22.027s
user 0m0.012s
sys 0m21.925s
So lets increase the timeout for the graph (/etc/munin/plugin-conf.d/munin-node) :
[fw_conntrack]
user root
timeout 60
24 hours pass….
I have my graph, now my graph is showing 55,000 established connections at night, this is not normal! Lets check take a look at the connection table:
# cat /proc/net/nf_conntrack > fook_connections.txt
A quick look through the file shows ~52k TCP connections going to a google server on port 5228:
$ cat fook_connections.txt | grep 2a00:1450:400b:0c02:0000:0000:0000:00bc | grep tcp | wc -l
52501
…out of interest how many to Google?:
$ cat fook_connections.txt | grep 2a00:1450 | wc -l
62327
I came across this post relating to outbound traffic spamming the port
OK, time to adjust the timeout value for established connections, currently it is set to 5 days! :
# /sbin/sysctl net.netfilter.nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
If this is not remedied each day the number of connections remaining in an established state will increase on a daily basis until the conntrack table is full again.
Lets tune that down to 10 minutes:
# /sbin/sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=600
# echo net.netfilter.nf_conntrack_tcp_timeout_established=600 >> /etc/sysctl.conf
Unfortunately this new timeout value will only take effect on new connections, exiting ones will still be using the 5 day value. I could always flush the table (conntrack -F conntrack) but I like my users. We’ll check back in a five days…
Finally the connections start ageing out:
A normal semester day:
Leave a Reply