Join Free
Results 1 to 1 of 1

Threaded View

  1. #1

    kernel: nf_conntrack: table full, dropping packet.

    Ran into an interesting server issue today. I noticed strange behavior across the cluster, and found the following error in /var/log/message (thousands of them)

    kernel: nf_conntrack: table full, dropping packet.

    This happens when your IPtables or CSF firewall is tracking too many connections. This can happen when you are being attacked, or is also very likely to happen on a busy server even if there is no malicious activity. Connections will be tracked if you have a firewall rule that does NAT or SNAT, or if you are tracking the number of connections per IP for rate limiting reasons. These scenarios are common either in linux router / firewalls, or in the case of firewall rules that are there for brute force protection / ddos protection.

    By default, Centos will set this maximum to 65,536 connections. This is enough for lightly loaded servers, but can easily be exhausted on heavily trafficked servers with a lot of firewall rules. On our heavy production servers, we’ve increased this limit to half a million, which has made a big improvement on the amount of workload those servers can handle.

    It is interesting to note, that the kind of servers most likely to have this problem, are ones where the user has set a lot of strict firewall rules to “help ward off attacks”. Unfortunately, the reality is that the firewall rules themselves are causing the downtime, not any attack! One way to solve the problem is to disable your firewall entirely, but before you go to that extreme, it is worth trying to increase the maximum connections here.

    How to Fix

    View the current maximum configured connections

    $ cat /proc/sys/net/netfilter/nf_conntrack_max
    To see the current used connections

    $ cat /proc/sys/net/netfilter/nf_conntrack_count
    Increase maximum configured connections limit

    # Temporarily Solution
    echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

    # Permanent Solution
    # Add following line on /etc/rc.d/rc.local

    $ vim /etc/rc.d/rc.local
    echo 524288 > /proc/sys/net/netfilter/nf_conntrack_max

    $ chmod a+x /etc/rc.d/rc.local

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
coupons | coupons and deals