Context

Traditionally, the data path for sending HTTPS traffic is:

• Read data from a file into a user-space buffer
• Encrypt that data using a user-space cryptography routine
• Send that encrypted data to a socket
  • The data is copied to a kernel-space socket buffer
• The kernel sends the encrypted data to the NIC
• The NIC writes the encrypted data to the wire

With Kernel TLS, this becomes:

• Read data from a file into a user-space buffer
• Send that plaintext data to a socket
  • The kernel encrypts the data on-the-fly while it copies it to a kernel-space socket buffer
• The kernel sends the encrypted data to the NIC
• The NIC writes the encrypted data to the wire

With Kernel TLS + TLS NIC (network interface card) offloading, this becomes:

• Read data from a file into a user-space buffer
• Send that plaintext data to a socket
  • The data is copied to a kernel-space socket buffer
• The kernel sends the plaintext data to the NIC
• The NIC encrypts the data using a hardware AES chip
• The NIC writes the encrypted data to the wire

Kernel TLS also enables using sendfile:

• The kernel reads data from a file into a kernel-space buffer
  • The kernel encrypts the data on-the-fly while it copies it to a kernel-space socket buffer
• The kernel sends the encrypted data to the NIC
• The NIC writes the encrypted data to the wire

Kernel TLS + TLS NIC offloading + sendfile is particularly powerful:

• The kernel reads data from a file into a kernel-space buffer
• The kernel sends the plaintext data to the NIC
• The NIC encrypts the data using a hardware AES chip
• The NIC writes the encrypted data to the wire

Kernel TLS + TLS NIC offloading + zerocopy sendfile:

• The kernel copies data from a file into the NIC
• The NIC encrypts the data using a hardware AES chip
• The NIC writes the encrypted data to the wire

The advantages of Kernel TLS are:

• enabling the use of sendfile for serving encrypted content
• enabling the use of TLS offloading to the NIC
• reducing the number of copies of the data, because the data is encrypted as it is copied

The advantages of sendfile are:

• reducing user-space/kernel-space context switches by running the logic in kernel-space only, reducing CPU load
• reducing the number of copies of the data, greatly reducing the RAM bandwidth usage

The advantage of TLS offloading is to greatly reduce the CPU load of the server by running the cipher logic on a hardware chip on the NIC.

There are multiple tutorials on enabling kernel TLS with NGINX and with low-level socket calls but not so many on enabling it with OpenSSL, which is the main purpose of this article.

Kernel TLS usage

Kernel configuration

Kernel TLS requires:

• Linux >= 4.13 or FreeBSD >= 13.0 (check with uname -a)
• Kernel TLS support to be built (check that zgrep CONFIG_TLS= /proc/config.gz or grep 'CONFIG_TLS=' /boot/config-$(uname -r) returns y or m)

Kernel TLS is usually built as a module, which must be explicitly loaded before starting the TLS program.

On FreeBSD, run as root:

kldload ktls_ocf.ko
sysctl kern.ipc.tls.enable=1

Additional configuration might be required. See the FreeBSD man page.

On Linux, run as root:

modprobe tls

To load it automatically on boot:

echo tls >/etc/modules-load.d/tls.conf

OpenSSL configuration

Kernel TLS requires:

• OpenSSL >= 3.0.0
• The kTLS option to be enabled at build time in OpenSSL
  • It is enabled by default on debian >= 12
  • Not sure about other distributions. Let me know!

Kernel TLS must then be enabled at run-time with either of:

• SSL_CTX_set_options(ctx, SSL_OP_ENABLE_KTLS)
• SSL_set_options(ssl, SSL_OP_ENABLE_KTLS)

Only some encryption ciphers are supported by kernel TLS.

This translates to the following ciphers & ciphersuites.

This list can be obtained with openssl ciphers -s -v, grepping on the Enc= value.

For TLS v1.3, TLS_AES_128_GCM_SHA256 support is mandatory, so enabling just this ciphersuite is enough.

To set the appropriate ciphers & ciphersuites, use:

• TLS ≤ v1.2 ciphers: SSL_CTX_set_cipher_list(ctx, "cipher1:cipher2:...")
• TLS v1.3 ciphersuites: SSL_CTX_set_ciphersuites(ctx, "ciphersuite1:ciphersuite2:...")

These algorithms are not supported in TLS versions earlier than TLS v1.2. To support earlier clients, an older cipher can be added, for example ECDHE-RSA-AES256-SHA, which is supported from TLS v1.0. But connections using this cipher will not use kernel TLS.

In order to support both new and older ciphers, but ensure that clients supporting newer ciphers are forced to use them, OpenSSL can be configured to pick its favorite common cipher rather than letting the client decide. This can be done with: SSL_CTX_set_options(ctx, SSL_OP_CIPHER_SERVER_PREFERENCE)

After these steps are done, OpenSSL will automatically use kernel TLS internally if available. No specific kernel TLS calls are needed.

Checking kernel TLS usage

Linux offers the /proc/net/tls_stat file to monitor kernel TLS usage.

The TlsTxSw and TlsRxSw counters are incremented for each connection that has used TX or RX kernel TLS.

This file is namespace-scoped! When running inside a container, only stats for sockets in that container will be shown.

Troubleshooting kernel TLS usage

If TlsTxSw or TlsRxSw do not increase, check the ciphers used for the connection.

bpftrace can be used to check whether the appropriate syscalls are being called by OpenSSL.

My command was:

sudo bpftrace -e 'tracepoint:syscalls:sys_enter_setsockopt  { printf("%d@%d: %d=%ld %d\n", args->fd, args->level, args->optname, args->optval, args->optlen); }'

• grep '31=' checks whether OpenSSL init kernel TLS. If it does not, it was not compiled with kernel TLS support.
• grep '@282' checks whether OpenSSL sets the kernel TLS data. If it does not, the algorithm is not supported or kernel TLS was not enabled at run-time.

TLS NIC offload usage

This article refers to NIC offload to NVIDIA Mellanox ConnectX6-DX cards. Other NICs can be used.

TLS NIC offload requires:

• a modern kernel version (see below)
• TLS NIC offload support to be built
  • check that zgrep CONFIG_TLS_DEVICE= /proc/config.gz or grep 'CONFIG_TLS_DEVICE=' /boot/config-$(uname -r) returns y or m
  • check that zgrep CONFIG_MLX5_EN_TLS= /proc/config.gz or grep 'CONFIG_MLX5_EN_TLS=' /boot/config-$(uname -r) returns y or m

NIC configuration

TLS NIC offload can be enabled on each network interface with:

• ethtool -K <interface> tls-hw-tx-offload on for TLS TX
• ethtool -K <interface> tls-hw-rx-offload on for TLS RX

The current status can be queried with ethtool -k <interface> | grep tls

Bonding configuration

Support for TLS NIC offload over bond interfaces is supported on Linux ≥ 5.12, FreeBSD ≥ ?.

Bonding is supported for modes balance-xor and 802.3ad, with xmit_hash_policy layer3+4.

No specific configuration is needed. TLS offload is implicitly enabled when the interfaces it uses have it enabled.

Specifically, ethtool TLS offload parameters must not be used for bonding interfaces. TLS values it exports are incorrect. What matters is what the underlying interfaces support.

Cipher support

NVIDIA Mellanox ConnectX6-DX cards only support AES-GCM(128).

On my server, I configured the corresponding TLS ciphers and ciphersuites (see above), added a few additional fallback ciphers, and enabled cipher server preference.

No other configuration is needed. The kernel (and OpenSSL) will automatically make use of TLS NIC offloading.

Checking TLS NIC offload usage

The /proc/net/tls_stat file contains TLS NIC offload metrics as well.

The TlsTxDevice and TlsRxDevice counters are incremented for each connection that has used TX or RX TLS NIC offload.

Troubleshooting TLS NIC offload usage

If TlsTxDevice or TlsRxDevice do not increase, check if TlsTxSw or TlsRxSw are increasing. If they do not, kernel TLS must be fixed first.

If kernel TLS is used but is not offloaded to NICs, it could be related to the cipher used, or the kernel version.

Make sure that the tls-hw-tx-offload / tls-hw-rx-offload ethtool features are enabled.

Make sure that the server does send packets to the test client through that NIC! Packets routed through the loopback interface will not make use of NIC offloading.

Sendfile support

Regular sendfile

Linux supports using sendfile to copy data from a fd to another in kernel-space only.

This can be used for TLS fd as well in OpenSSL with SSL_sendfile.

Zero-copy sendfile for TLS NIC offloading

On Linux >= 6.0, OpenSSL >= 3.2.0, SSL_sendfile can make use of zero-copy sendfile for TLS NIC offloading. This only works for the TX path.

This must enabled at run-time with either of:

• SSL_CTX_set_options(ctx, SSL_OP_ENABLE_KTLS_TX_ZEROCOPY_SENDFILE)
• SSL_set_options(ssl, SSL_OP_ENABLE_KTLS_TX_ZEROCOPY_SENDFILE)

This is an advanced optimization option, that must only be used if the contents of the fd that is being copied from are guaranteed not to change.

Enjoy your (near-)terabit HTTPS throughput! ;-)

Credits

• https://www.nginx.com/blog/improving-nginx-performance-with-kernel-tls/
• https://docs.nvidia.com/doca/sdk/tls-offload/index.html
• https://docs.nvidia.com/networking/display/OFEDv521040/Kernel+Transport+Layer+Security+(kTLS)+Offloads
• https://www.kernel.org/doc/html/latest/networking/tls.html
• https://www.kernel.org/doc/html/latest/networking/tls-offload.html
• https://www.openssl.org/docs/manmaster/man3/SSL_set_options.html
• https://kernelnewbies.org/LinuxVersions :-)