IPv6 conntrack and munin

Argh, my beloved linux IPv6 firewall was suffering, too many connections, munin graphs not updating; this needed looking at…

Firstly I noticed multiple entries of the following in kern.log:

nf_conntrack: table full, dropping packet

After checking the existing table size:

# /sbin/sysctl net.netfilter.nf_conntrack_count
net.netfilter.nf_conntrack_count = 76768

…it seemed sensible to double it:

# cat /proc/sys/net/nf_conntrack_max
65526
# sysctl -w net.netfilter.nf_conntrack_max=131072
# echo net.netfilter.nf_conntrack_max=131072 >> /etc/sysctl.conf

OK, this gave me some more headroom, but my munin graphs still aren’t being rendered for fw_conntrack. Checking /var/log/munin/munin-node.log showed the following every 5 minutes:

2015/10/27-11:35:15 [1788] Service 'fw_conntrack' timed out.
2015/10/27-11:40:15 [2954] Service 'fw_conntrack' timed out.

Each munin graph has a 5 second timeout, according to /usr/share/munin/plugins/fw_conntrack , /proc/net/ip_conntrack is munged to build the graph. How long does that take to cat?:

# time cat /proc/net/nf_conntrack | wc -l
79590

real 0m22.027s
user 0m0.012s
sys 0m21.925s

So lets increase the timeout for the graph (/etc/munin/plugin-conf.d/munin-node) :

[fw_conntrack]
user root
timeout 60

24 hours pass….
I have my graph, now my graph is showing 55,000 established connections at night, this is not normal! Lets check take a look at the connection table:

# cat /proc/net/nf_conntrack > fook_connections.txt

A quick look through the file shows ~52k TCP connections going to a google server on port 5228:

$ cat fook_connections.txt | grep 2a00:1450:400b:0c02:0000:0000:0000:00bc | grep tcp | wc -l
52501

…out of interest how many to Google?:

$ cat fook_connections.txt | grep 2a00:1450 | wc -l
62327

I came across this post relating to outbound traffic spamming the port

OK, time to adjust the timeout value for established connections, currently it is set to 5 days! :

# /sbin/sysctl net.netfilter.nf_conntrack_tcp_timeout_established
net.netfilter.nf_conntrack_tcp_timeout_established = 432000

If this is not remedied each day the number of connections remaining in an established state will increase on a daily basis until the conntrack table is full again.
Lets tune that down to 10 minutes:

# /sbin/sysctl -w net.netfilter.nf_conntrack_tcp_timeout_established=600
# echo net.netfilter.nf_conntrack_tcp_timeout_established=600 >> /etc/sysctl.conf

Unfortunately this new timeout value will only take effect on new connections, exiting ones will still be using the 5 day value. I could always flush the table (conntrack -F conntrack) but I like my users. We’ll check back in a five days…

Finally the connections start ageing out:

A normal semester day: