From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by passt.top (Postfix, from userid 1000) id A7D405A0277; Sat, 23 Sep 2023 00:06:10 +0200 (CEST) From: Stefano Brivio To: David Gibson , Matej Hrica Subject: [PATCH RFT 5/5] passt.1: Add note about tuning rmem_max and wmem_max for throughput Date: Sat, 23 Sep 2023 00:06:10 +0200 Message-Id: <20230922220610.58767-6-sbrivio@redhat.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230922220610.58767-1-sbrivio@redhat.com> References: <20230922220610.58767-1-sbrivio@redhat.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: PRDSENJHQR7YI6WP2A52SDAQAD75EENW X-Message-ID-Hash: PRDSENJHQR7YI6WP2A52SDAQAD75EENW X-MailFrom: sbrivio@passt.top X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Signed-off-by: Stefano Brivio --- passt.1 | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/passt.1 b/passt.1 index 1ad4276..bcbe6fd 100644 --- a/passt.1 +++ b/passt.1 @@ -926,6 +926,39 @@ If the sending window cannot be queried, it will always be announced as the current sending buffer size to guest or target namespace. This might affect throughput of TCP connections. +.SS Tuning for high throughput + +On Linux, by default, the maximum memory that can be set for receive and send +socket buffers is 208 KiB. Those limits are set by the +\fI/proc/sys/net/core/rmem_max\fR and \fI/proc/sys/net/core/wmem_max\fR files, +see \fBsocket\fR(7). + +As of Linux 6.5, while the TCP implementation can dynamically shrink buffers +depending on utilisation even above those limits, such a small limit will +reflect on the advertised TCP window at the beginning of a connection, and the +buffer size of the UNIX domain socket buffer used by \fBpasst\fR cannot exceed +these limits anyway. + +Further, as of Linux 6.5, using socket options \fBSO_RCVBUF\fR and +\fBSO_SNDBUF\fR will prevent TCP buffers to expand above the \fIrmem_max\fR and +\fIwmem_max\fR limits because the automatic adjustment provided by the TCP +implementation is then disabled. + +As a consequence, \fBpasst\fR and \fBpasta\fR probe these limits at start-up and +will not set TCP socket buffer sizes if they are lower than 2 MiB, because this +would affect the maximum size of TCP buffers for the whole duration of a +connection. + +Note that 208 KiB is, accounting for kernel overhead, enough to fit less than +three TCP packets at the default MSS. In applications where high throughput is +expected, it is therefore advisable to increase those limits to at least 2 MiB, +or even 16 MiB: + +.nf + sysctl -w net.core.rmem_max=$((16 << 20) + sysctl -w net.core.wmem_max=$((16 << 20) +.fi + .SH LIMITATIONS Currently, IGMP/MLD proxying (RFC 4605) and support for SCTP (RFC 4960) are not -- 2.39.2