From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 50E355A0275 for ; Mon, 25 Sep 2023 07:52:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=201602; t=1695621133; bh=S90QP71ChfzMJH7HOKROwrAHbdlgbrd6N4H+TnzUDmM=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=OW2Nj2BCN86IIaFk0vt5NQoBjsM4Osm5RCLMBP4DRhwwlDeLT+TQJSwSkh3M/RdaO kY8gcXqnpZ9e+gjSaJgZkmjkafBUyy+R4EP86qjW2ryGtN8t59pNTNFC7a1agN2J5h Nfv7yyxlcGq+mS6VgR1IcDQ4niJ0VOVvi0YClLdM= Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4RvBpj2CkHz4xM0; Mon, 25 Sep 2023 15:52:13 +1000 (AEST) Date: Mon, 25 Sep 2023 14:21:47 +1000 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH RFT 3/5] tcp: Force TCP_WINDOW_CLAMP before resetting STALLED flag Message-ID: References: <20230922220610.58767-1-sbrivio@redhat.com> <20230922220610.58767-4-sbrivio@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Pm75h3qIrCeOL8mE" Content-Disposition: inline In-Reply-To: Message-ID-Hash: BHRXUPA6OCD6DHXCGMI5K3GY3ONXFRSK X-Message-ID-Hash: BHRXUPA6OCD6DHXCGMI5K3GY3ONXFRSK X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Matej Hrica , passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --Pm75h3qIrCeOL8mE Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 25, 2023 at 02:09:41PM +1000, David Gibson wrote: > On Sat, Sep 23, 2023 at 12:06:08AM +0200, Stefano Brivio wrote: > > It looks like we need it as workaround for this situation, readily > > reproducible at least with a 6.5 Linux kernel, with default rmem_max > > and wmem_max values: > >=20 > > - an iperf3 client on the host sends about 160 KiB, typically > > segmented into five frames by passt. We read this data using > > MSG_PEEK > >=20 > > - the iperf3 server on the guest starts receiving > >=20 > > - meanwhile, the host kernel advertised a zero-sized window to the > > receiver, as expected > >=20 > > - eventually, the guest acknowledges all the data sent so far, and > > we drop it from the buffer, courtesy of tcp_sock_consume(), using > > recv() with MSG_TRUNC > >=20 > > - the client, however, doesn't get an updated window value, and > > even keepalive packets are answered with zero-window segments, > > until the connection is closed > >=20 > > It looks like dropping data from a socket using MSG_TRUNC doesn't > > cause a recalculation of the window, which would be expected as a > > result of any receiving operation that invalidates data on a buffer > > (that is, not with MSG_PEEK). > >=20 > > Strangely enough, setting TCP_WINDOW_CLAMP via setsockopt(), even to > > the previous value we clamped to, forces a recalculation of the > > window which is advertised to the guest. > >=20 > > I couldn't quite confirm this issue by following all the possible > > code paths in the kernel, yet. If confirmed, this should be fixed in > > the kernel, but meanwhile this workaround looks robust to me (and it > > will be needed for backward compatibility anyway). >=20 > So, I tested this, and things got a bit complicated. >=20 > First, I reproduced the "read side" problem by setting > net.core.rmem_max to 256kiB while setting net.core.wmem_max to 16MiB. > The "160kiB" stall happened almost every time. Applying this patch > appears to fix it completely, getting GiB/s throughput consistently. > So, yah. >=20 > Then I tried reproducing it differently: by setting both > net.core.rmem_max and net.core.wmem_max to 16MiB, but setting > SO_RCVBUF to 128kiB explicitly in tcp_sock_set_bufsize() (which > actually results in a 256kiB buffer, because of the kernel's weird > interpretation). >=20 > With the SO_RCVBUF clamp and without this patch, I don't get the > 160kiB stall consistently any more. What I *do* get is nearly every > time - but not *every* time - is slow transfers, ~40Mbps vs. ~12Gbps. > Sometimes it stalls after several seconds. The stall is slightly > different from the 160kiB stall though: the 160kiB stall seems 0 bytes > transferred on both sides. With the RCVBUF stall I get a trickle of > bytes (620 bytes/s) on the receiver/guest side, with mostly 0 bytes > per interval on the sender but occasionally an interval with several > hundred KB. >=20 > That is it seems like there's a buffer somewhere that's very slowly > draining into the receiver, then getting topped up in an instant once > it gets low enough. >=20 > When I have both this patch and the RCVBUF clamp, I don't seem to be > able to reproduce the trickle-stall anymore, but I still get the slow > transfer speeds most, but not every time. Sometimes, but only rarely, > I do seem to still get a complete stall (0 bytes on both sides). I noted another oddity. With this patch, _no_ RCVBUF clamp and 16MiB wmem_max fixed, things seem to behave much better with a small rmem_max than large. With rmem_max=3D256KiB I get pretty consistent 37Gbps throughput and iperf3 -c reports 0 retransmits. With rmem_max=3D16MiB, the throughput fluctuates from second to second between ~3Gbps and ~30Gbps. The client reports retransmits in some intervals, which is pretty weird over lo. Urgh... so many variables. --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --Pm75h3qIrCeOL8mE Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmURCtEACgkQzQJF27ox 2GfyeQ/+OgAvINNH6GVJhkisAeo/5b16ev40wRJ/rnK1zKjX5BjPmjny4FojRXqG 4TMVFO48i4lSkVMld6uMpmKYfHxO6LvEP49C+s3hy+UaOywQrNpiV4AG290RK9fz 2+iRh7TePVPEfWN3ZV1qk3Lb96ekl1GNqO+HgPIoYY8t7XxMIMcPPkxDXAsc7wRH nJ7R3bd2IxMpCnc0+BXqrUv6M65IxGRJC8oYcbwXQfljJHM82xENhV6N65CYwm47 6d8TTgkWXB/+XvsjymrZJMX8k8qa/Avg7ViGLEiZQpqVTOxdgKer3O8P33fXxGtt 0+WEIOi1gi66W6lBeIV6cULoZ1mByV5Al3/pTZ0CS1CIW3r/4Ro6v5nCXHMSWl/7 2xjjHxawt0voqHynd0tNldi86OKl90EL47fNZcqCbZitjUwtp92Y2bSp9gJBhlRf mqQ856HaUzn1dcELYo68M0JcWB6HZqUkVUl/h04UXCTYEeSJwYpHNhL1gFtLMpOn DWweKyD7OpE+rxgLOhT6ozBPWzOy1j7Y3HIZReGFQyJrG2wErxcPejgJL4D12wLl ugCb3djcgK/lgF7luRfGs0kB/8U9Hsx1XXP3E5/4tTv3jnDjiGvbjNXy798obvBv 1tqPD5igKfJG2qWda4NHtiytI5I3H2p8ZfqaauKtQamr4zJCR8g= =0F/C -----END PGP SIGNATURE----- --Pm75h3qIrCeOL8mE--