Tuxifans Website

Some ISPs have this issue where TCP connections are closed after a couple of seconds of inactivity. I’ve experienced this on a VPS, apparently due to an incorrectly configured DDoS filter.

Applications like sshd can circumvent this by enabling a feature called “keepalive” on their TCP sockets to circumvent this by sending dummy TCP acknowledgements every couple of seconds while the connection is inactive.

However, there’s no immediately obvious way to make this the default system wide. In this post I’m documenting a way to do just that using an eBPF program for all incoming connections.

Configure default keepalive settings

First of all, figure out how for long your TCP connections are staying open until they are getting closed. Wireshark and netcat may help.

Then tweak the following settings in /etc/sysctl.conf accordingly:

net.ipv4.tcp_keepalive_time = 30
net.ipv4.tcp_keepalive_intvl = 10
net.ipv4.tcp_keepalive_probes = 3

• tcp_keepalive_time: the interval between the last data packet sent and the first keepalive probe
• tcp_keepalive_intvl: the interval between subsequential keepalive probes
• tcp_keepalive_probes: the number of unacknowledged probes to send before considering the connection dead

Source: https://tldp.org/HOWTO/TCP-Keepalive-HOWTO/usingkeepalive.html

You can then apply the settings using sudo sysctl -p.

Prepare eBPF usage

Before you can build and load eBPF programs some packages need to be installed. On Debian, those are:

clang llvm libbpf-dev bpftool linux-headers-amd64 linux-libc-dev

Then you can create a work directory of your choice for the steps following.

sudo mkdir -p /opt/bpf
cd /opt/bpf

(it really doesn’t matter as long as the location is persistent, though for security reasons I’d strongly recommend to make it owned and only writable by root)

Write and build the eBPF program

Write the following to keepalive.c:

#include <linux/bpf.h>
#include <bpf/bpf_helpers.h>

#ifndef SOL_SOCKET
#define SOL_SOCKET 1
#endif
#ifndef SO_KEEPALIVE
#define SO_KEEPALIVE 9
#endif

SEC("sockops")
int bpf_force_keepalive(struct bpf_sock_ops *skops)
{
    // Intercept when the server successfully establishes an incoming connection
    if (skops->op == BPF_SOCK_OPS_PASSIVE_ESTABLISHED_CB) {
        int val = 1;
        // Force SO_KEEPALIVE = 1
        bpf_setsockopt(skops, SOL_SOCKET, SO_KEEPALIVE, &val, sizeof(val));
    }
    return 0;
}

char _license[] SEC("license") = "GPL";

And build the program using Clang:

clang -O2 -target bpf -I/usr/include/$(uname -m)-linux-gnu -c keepalive.c -o keepalive.o

Load the program

Finally, just write the SystemD service file to /etc/systemd/system/bpf-keepalive.service:

[Unit]
Description=eBPF Default TCP Keepalive
After=network.target

[Service]
Type=oneshot
RemainAfterExit=yes

# Clean up existing loaded program
ExecStartPre=-/usr/sbin/bpftool cgroup detach /sys/fs/cgroup sock_ops pinned /sys/fs/bpf/tcp_keepalive
ExecStartPre=-/bin/rm -f /sys/fs/bpf/tcp_keepalive

# Load object into kernel and pin it
ExecStart=/usr/sbin/bpftool prog load /opt/bpf/keepalive.o /sys/fs/bpf/tcp_keepalive
#                                     ^ Replace this with your working directory!

# Attach it to root cgroups v2 mount (so affects all processes)
ExecStart=/usr/sbin/bpftool cgroup attach /sys/fs/cgroup sock_ops pinned /sys/fs/bpf/tcp_keepalive

# Cleanup on stop
ExecStop=/usr/sbin/bpftool cgroup detach /sys/fs/cgroup sock_ops pinned /sys/fs/bpf/tcp_keepalive
ExecStop=/bin/rm -f /sys/fs/bpf/tcp_keepalive

[Install]
WantedBy=multi-user.target

And enable+start the service:

sudo systemctl daemon-reload
sudo systemctl enable --now bpf-keepalive.service

You can verify that it’s working using:

sudo bpftool cgroup tree