[PATCH RFT 5/5] passt.1: Add note about tuning rmem_max and wmem_max for throughput

[Stefano Brivio <sbrivio@redhat.com>]

Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
 passt.1 | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/passt.1 b/passt.1
index 1ad4276..bcbe6fd 100644
--- a/passt.1
+++ b/passt.1
@@ -926,6 +926,39 @@ If the sending window cannot be queried, it will always be announced as the
 current sending buffer size to guest or target namespace. This might affect
 throughput of TCP connections.
 
+.SS Tuning for high throughput
+
+On Linux, by default, the maximum memory that can be set for receive and send
+socket buffers is 208 KiB. Those limits are set by the
+\fI/proc/sys/net/core/rmem_max\fR and \fI/proc/sys/net/core/wmem_max\fR files,
+see \fBsocket\fR(7).
+
+As of Linux 6.5, while the TCP implementation can dynamically shrink buffers
+depending on utilisation even above those limits, such a small limit will
+reflect on the advertised TCP window at the beginning of a connection, and the
+buffer size of the UNIX domain socket buffer used by \fBpasst\fR cannot exceed
+these limits anyway.
+
+Further, as of Linux 6.5, using socket options \fBSO_RCVBUF\fR and
+\fBSO_SNDBUF\fR will prevent TCP buffers to expand above the \fIrmem_max\fR and
+\fIwmem_max\fR limits because the automatic adjustment provided by the TCP
+implementation is then disabled.
+
+As a consequence, \fBpasst\fR and \fBpasta\fR probe these limits at start-up and
+will not set TCP socket buffer sizes if they are lower than 2 MiB, because this
+would affect the maximum size of TCP buffers for the whole duration of a
+connection.
+
+Note that 208 KiB is, accounting for kernel overhead, enough to fit less than
+three TCP packets at the default MSS. In applications where high throughput is
+expected, it is therefore advisable to increase those limits to at least 2 MiB,
+or even 16 MiB:
+
+.nf
+	sysctl -w net.core.rmem_max=$((16 << 20)
+	sysctl -w net.core.wmem_max=$((16 << 20)
+.fi
+
 .SH LIMITATIONS
 
 Currently, IGMP/MLD proxying (RFC 4605) and support for SCTP (RFC 4960) are not
-- 
@@ -926,6 +926,39 @@ If the sending window cannot be queried, it will always be announced as the
 current sending buffer size to guest or target namespace. This might affect
 throughput of TCP connections.
 
+.SS Tuning for high throughput
+
+On Linux, by default, the maximum memory that can be set for receive and send
+socket buffers is 208 KiB. Those limits are set by the
+\fI/proc/sys/net/core/rmem_max\fR and \fI/proc/sys/net/core/wmem_max\fR files,
+see \fBsocket\fR(7).
+
+As of Linux 6.5, while the TCP implementation can dynamically shrink buffers
+depending on utilisation even above those limits, such a small limit will
+reflect on the advertised TCP window at the beginning of a connection, and the
+buffer size of the UNIX domain socket buffer used by \fBpasst\fR cannot exceed
+these limits anyway.
+
+Further, as of Linux 6.5, using socket options \fBSO_RCVBUF\fR and
+\fBSO_SNDBUF\fR will prevent TCP buffers to expand above the \fIrmem_max\fR and
+\fIwmem_max\fR limits because the automatic adjustment provided by the TCP
+implementation is then disabled.
+
+As a consequence, \fBpasst\fR and \fBpasta\fR probe these limits at start-up and
+will not set TCP socket buffer sizes if they are lower than 2 MiB, because this
+would affect the maximum size of TCP buffers for the whole duration of a
+connection.
+
+Note that 208 KiB is, accounting for kernel overhead, enough to fit less than
+three TCP packets at the default MSS. In applications where high throughput is
+expected, it is therefore advisable to increase those limits to at least 2 MiB,
+or even 16 MiB:
+
+.nf
+	sysctl -w net.core.rmem_max=$((16 << 20)
+	sysctl -w net.core.wmem_max=$((16 << 20)
+.fi
+
 .SH LIMITATIONS
 
 Currently, IGMP/MLD proxying (RFC 4605) and support for SCTP (RFC 4960) are not
-- 
2.39.2