From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202502 header.b=DlgLtiGD; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id F245E5A0635 for ; Fri, 14 Feb 2025 03:24:49 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202502; t=1739499885; bh=NqPkhkxir/ZfStmpbGiM5ZyNv+nPhDV91Xv1Ur8CVOg=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=DlgLtiGD0iEAGMex4cmI3gPZ0wLmmtNNddg+RarqRuqiWXi4J3svq0dw5un7YF9tn dU8DnU0lr64vJTM1iGjVk/qfW2bJ/Tf3grGTxT4aNGpvpwm742J0UTKHyK5zNweZCc 20bwZEvGLuTLo0ieJMaGvryVWLzIJJALHVAz0FpkEnpadMVWfI5ZWdhVztHYAX8H9H 4hl1ZBbK59ZjGlR2TtVsRH/aYpHaYq4ELmtTM7NeO1QR8bhHLiecD8kDAqej7BJKNH d+uVZbLVkswQliBWlKmxMlY+3NQ/5FtrKUvNH/3R516HYcv5xW3V4IZz6TDH34W2yN 9EQb8pUHgrQ7A== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4YvG8s3vyJz4x2J; Fri, 14 Feb 2025 13:24:45 +1100 (AEDT) Date: Fri, 14 Feb 2025 13:24:40 +1100 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH] tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to maximum values Message-ID: References: <20250213221650.4086105-1-sbrivio@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="CQKJAsWWGN+FycD5" Content-Disposition: inline In-Reply-To: Message-ID-Hash: MIQ4WG6U2QHJALZY45K3OEJDHAB44I42 X-Message-ID-Hash: MIQ4WG6U2QHJALZY45K3OEJDHAB44I42 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --CQKJAsWWGN+FycD5 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Feb 14, 2025 at 10:02:56AM +1100, David Gibson wrote: > On Thu, Feb 13, 2025 at 11:16:50PM +0100, Stefano Brivio wrote: > > I added this a long long time ago because it dramatically improved > > throughput back then: with rmem_max and wmem_max >=3D 4 MiB, we would > > force send and receive buffer sizes for TCP sockets to the maximum > > allowed value. > >=20 > > This effectively disables TCP auto-tuning, which would otherwise allow > > us to exceed those limits, as crazy as it might sound. But in any > > case, it made sense. > >=20 > > Now that we have zero (internal) copies on every path, plus vhost-user > > support, it turns out that these settings are entirely obsolete. I get > > substantially the same throughput in every test we perform, even with > > very short durations (one second). > >=20 > > The settings are not just useless: they actually cause us quite some > > trouble on guest state migration, because they lead to huge queues > > that need to be moved as well. > >=20 > > Drop those settings. > >=20 > > Signed-off-by: Stefano Brivio >=20 > Hooray! >=20 > Reviewed-by: David Gibson So, I still think this is a good idea in general, but it does cause a new issue for migration. On the source, our buffers may have been auto-tuned up, but on the target, of course, we have a new socket with the default initial buffer size. So, when we try to put the source's buffer data into the target's buffer we can hit the (current) limit. I'm currently seeing what I think is that problem with iperf3_bidir6 - I'm getting EAGAIN on the target filling the sndbuf - this time the already sent portion (repair mode). It might have been one of the intermittent problems you hit as well. Some early experiments suggest we might be able to deal with this by moving the socket into blocking mode during the migration, although I'm certainly concerned that might let us block indefinitely while in the migration window if the peer isn't recv()ing for a while. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --CQKJAsWWGN+FycD5 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmeuqVkACgkQzQJF27ox 2GdXdw//bhxcMB6O7Ac+F62nKDe/TTZWKhldEFf8JFJkKvbJ0yQm04STeEbw833h d8vZwGgiRtOgMVbKEXt2p6cj9Cg2ML8Md1aKTRcbX08vEVvJqe4JyHWunXNvP0is k3wLWzd9mXR5kYgn57crx76Y4SOcGbmSaFC7eDg/6DuzF6eysKWd/YAHwvA9uiho mvK6vK3CGGdnVwHiGLGheJM+f4ewvAQMFoeRaWGjxFvTxgC6eooz1AyWHQ6NPh3h reoekGmH574CbYDPjO/ufDghRy/+BUIA2D3iOAqUWSNpLrYHPeUBjXx7aoleHxqx qsaYHIyN4049EJ59a4A88NCYYC4yu4T4k8jSOppDV8pxrALk2qe0jyQk4NfVZbr3 EgbNwQC7fa/QPdabejACSKJWhycYBl8cAUvxw0+keDuluXyMAFYg35Js/VNAos6w yTOxe3GG2j18PxX8BLCMi1LO06EXBbhjW6GGdhnresnIMmaNK3aEYsMhszhBMrUJ katqT/rsSLpUJI4qPBlO4qHx9WNgx1TvADJPuhb/RjtTZ9c5cP2CUBvYshg+D5Gg uTatRCXMaR2JMLzsgu6Smuiq3zS7WY3WAfc2lmtXQID8e7lYl37u2BZCmqZ7RBmZ WJZrGvyscBzpuV7GIh3PudagOpT4wKKf7LfGU9LQWV/RXfw9Tv8= =PcwW -----END PGP SIGNATURE----- --CQKJAsWWGN+FycD5--