From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 471955A026D for ; Mon, 25 Sep 2023 06:15:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=201602; t=1695615303; bh=XUo9Om2wMxPZ+kw0NIBAWceL8N2iwdW8E6k7hu+oRek=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=H2bC26xUJrVEGzauzrKXh+60sgL6UOY3BP5JnkS0PjjFeKYt1JO3Jhit5PwZ3PQVK nqL2Plj99Wj3c1oyWYgMMjybYyrdK+a1pML5d/Wmm9hk9Le32Md3hA4TeGKGZQeZOB cY0LamOQZMpoJ4SWwKmo1CLWQ2CxQShL6++r3z7E= Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4Rv8fb6MNwz4xKl; Mon, 25 Sep 2023 14:15:03 +1000 (AEST) Date: Mon, 25 Sep 2023 14:10:50 +1000 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH RFT 3/5] tcp: Force TCP_WINDOW_CLAMP before resetting STALLED flag Message-ID: References: <20230922220610.58767-1-sbrivio@redhat.com> <20230922220610.58767-4-sbrivio@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="BL6NS8TaPc3ByKxB" Content-Disposition: inline In-Reply-To: Message-ID-Hash: 6QJNZIPP4ZG57ODA7CV2YEBBY2KDVL22 X-Message-ID-Hash: 6QJNZIPP4ZG57ODA7CV2YEBBY2KDVL22 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Matej Hrica , passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --BL6NS8TaPc3ByKxB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Sep 25, 2023 at 02:09:41PM +1000, David Gibson wrote: > On Sat, Sep 23, 2023 at 12:06:08AM +0200, Stefano Brivio wrote: > > It looks like we need it as workaround for this situation, readily > > reproducible at least with a 6.5 Linux kernel, with default rmem_max > > and wmem_max values: > >=20 > > - an iperf3 client on the host sends about 160 KiB, typically > > segmented into five frames by passt. We read this data using > > MSG_PEEK > >=20 > > - the iperf3 server on the guest starts receiving > >=20 > > - meanwhile, the host kernel advertised a zero-sized window to the > > receiver, as expected > >=20 > > - eventually, the guest acknowledges all the data sent so far, and > > we drop it from the buffer, courtesy of tcp_sock_consume(), using > > recv() with MSG_TRUNC > >=20 > > - the client, however, doesn't get an updated window value, and > > even keepalive packets are answered with zero-window segments, > > until the connection is closed > >=20 > > It looks like dropping data from a socket using MSG_TRUNC doesn't > > cause a recalculation of the window, which would be expected as a > > result of any receiving operation that invalidates data on a buffer > > (that is, not with MSG_PEEK). > >=20 > > Strangely enough, setting TCP_WINDOW_CLAMP via setsockopt(), even to > > the previous value we clamped to, forces a recalculation of the > > window which is advertised to the guest. > >=20 > > I couldn't quite confirm this issue by following all the possible > > code paths in the kernel, yet. If confirmed, this should be fixed in > > the kernel, but meanwhile this workaround looks robust to me (and it > > will be needed for backward compatibility anyway). >=20 > So, I tested this, and things got a bit complicated. >=20 > First, I reproduced the "read side" problem by setting > net.core.rmem_max to 256kiB while setting net.core.wmem_max to 16MiB. > The "160kiB" stall happened almost every time. Applying this patch > appears to fix it completely, getting GiB/s throughput consistently. > So, yah. >=20 > Then I tried reproducing it differently: by setting both > net.core.rmem_max and net.core.wmem_max to 16MiB, but setting > SO_RCVBUF to 128kiB explicitly in tcp_sock_set_bufsize() (which > actually results in a 256kiB buffer, because of the kernel's weird > interpretation). >=20 > With the SO_RCVBUF clamp and without this patch, I don't get the > 160kiB stall consistently any more. What I *do* get is nearly every > time - but not *every* time - is slow transfers, ~40Mbps vs. ~12Gbps. > Sometimes it stalls after several seconds. The stall is slightly > different from the 160kiB stall though: the 160kiB stall seems 0 bytes > transferred on both sides. With the RCVBUF stall I get a trickle of > bytes (620 bytes/s) on the receiver/guest side, with mostly 0 bytes > per interval on the sender but occasionally an interval with several > hundred KB. >=20 > That is it seems like there's a buffer somewhere that's very slowly > draining into the receiver, then getting topped up in an instant once > it gets low enough. >=20 > When I have both this patch and the RCVBUF clamp, I don't seem to be > able to reproduce the trickle-stall anymore, but I still get the slow > transfer speeds most, but not every time. Sometimes, but only rarely, > I do seem to still get a complete stall (0 bytes on both sides). >=20 > So it looks like there are two different things going on here. It > looks like this patch fixes at least something, but there might be > some more things to investigate afterwards. On that basis: >=20 > Tested-by: David Gibson > Reviewed-by: David Gibson With the exception of the previously noted s/receiver/sender/ fixes, of course. --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --BL6NS8TaPc3ByKxB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmURCEMACgkQzQJF27ox 2GfyYBAAm/x0jQg/b3TfA1pJVB3QpwmHF3CS7f7BKb/5D6Ok96Wr7Emouf64pzSW 4VF+4vhVogcXcLgmOpYF6iwrzIe44JF4whhEzGoOm8oowZgiMWeBJZ0PVntkmt2i 4Q0o7ecueqBOhAHodQEwZ33V4SB/dfAy6hRuH1YsIGBIWnH5qcvhTMNxDoWd51+q nd/S4cq652UY4dUzPxhUmSyNOK55bZA+FIhd5ENWUstSF8uZc1E8GobpsA96PpZF gBFGm09j9UJiNnQPLu8W+0lJfF0Fgg9Xr2+89QDEHz8gqgwDoudwzjW22UB6fdqO +7+dJMbN15+TJQLs70axAPWNjBSF7VFd4AdbFZPYTo/dPfddc96+tc/TsLDJ9J/A HmGIb98uxZ+FZH/8uGgUT3BnUWLotTLmWTyr7LUm66ZQlegVbFIfFeelJMCcdEaj hT2dyC3mnFg2nAmBMqVjUMIFFW5qtGR6XF5OxkLhFWfTChkBcPJzkkMQMx3hiAvZ 1hYX8U/ZJnIrpZlJhmnOsHvzgrJzkvonyldoPgXC91A8+1IpvIVDoYS0A+BWWvif hRQoQxW4L+eEjZ6SH3NUU4PHd4VgXtjUYhwV1uLNVy6YjCqou+9IEqbojc/WyIgP WVsrNFNq6NR38mBCRL9SwD6hJbfz0K2OgWjKLMdEz1puZmlCUwo= =jMmX -----END PGP SIGNATURE----- --BL6NS8TaPc3ByKxB--