From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202512 header.b=pRSPMJRz; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id C87EA5A0271 for ; Mon, 08 Dec 2025 07:43:23 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202512; t=1765176201; bh=cGLsHrTBi//0R3VoheMZ6jV8aRArUwy6sk/xZROZlf4=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=pRSPMJRzSHknEA3Bx5pNfJi1JdeLZnG1m7Ed7W3sfETFFErgdqPGzt+U8lWf6PVPO 4Mqgt4DSoSfEluh9++nEpTG30GcA+X+eVr9L/GkkBRttnVgF8IsQjWmhm+ZK3vUDup S5T5jVIBHgi7nBmtKcRc/UHP51nYNpiGqevruv2w/Af9CozA8eLbOZKDJ6y66O+rSj 2+57CTDaPLtPOocOIu8MO3ZmxD1HMVC+xIWWT40PJru+JWVCjllZAvtx1bIc2wDxmU 28MxyBUF3bjXutUkEPiTYJeQUlqgHE9FmT8EY6sYMEWswxF0J3JgS505PqUz1rm9/m 0JzCEtm+MduJg== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4dPsr938dTz4wCx; Mon, 08 Dec 2025 17:43:21 +1100 (AEDT) Date: Mon, 8 Dec 2025 17:25:02 +1100 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH v2 7/9] tcp: Allow exceeding the available sending buffer size in window advertisements Message-ID: References: <20251208002229.391162-1-sbrivio@redhat.com> <20251208002229.391162-8-sbrivio@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="taqgqOi3AReVpdEs" Content-Disposition: inline In-Reply-To: <20251208002229.391162-8-sbrivio@redhat.com> Message-ID-Hash: P4N75KP5XBOJLPRREPOG5XBKA5U3JSLG X-Message-ID-Hash: P4N75KP5XBOJLPRREPOG5XBKA5U3JSLG X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Max Chernoff X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --taqgqOi3AReVpdEs Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Dec 08, 2025 at 01:22:15AM +0100, Stefano Brivio wrote: > If the remote peer is advertising a bigger value than our current > sending buffer, it means that a bigger sending buffer is likely to > benefit throughput. >=20 > We can get a bigger sending buffer by means of the buffer size > auto-tuning performed by the Linux kernel, which is triggered by > aggressively filling the sending buffer. >=20 > Use an adaptive boost factor, up to 150%, depending on: >=20 > - how much data we sent so far: we don't want to risk retransmissions > for short-lived connections, as the latency cost would be > unacceptable, and >=20 > - the current RTT value, as we need a bigger buffer for higher > transmission delays >=20 > The factor we use is not quite a bandwidth-delay product, as we're > missing the time component of the bandwidth, which is not interesting > here: we are trying to make the buffer grow at the beginning of a > connection, progressively, as more data is sent. >=20 > The tuning of the amount of boost factor we want to apply was done > somewhat empirically but it appears to yield the available throughput > in rather different scenarios (from ~ 10 Gbps bandwidth with 500ns to > ~ 1 Gbps with 300 ms RTT) and it allows getting there rather quickly, > within a few seconds for the 300 ms case. >=20 > Note that we want to apply this factor only if the window advertised > by the peer is bigger than the current sending buffer, as we only need > this for auto-tuning, and we absolutely don't want to incur > unnecessary retransmissions otherwise. >=20 > The related condition in tcp_update_seqack_wnd() is not redundant as > there's a subtractive factor, sendq, in the calculation of the window > limit. If the sending buffer is smaller than the peer's advertised > window, the additional limit we might apply might be lower than we > would do otherwise. >=20 > Assuming that the sending buffer is reported as 100k, sendq is > 20k, we could have these example cases: >=20 > 1. tinfo->tcpi_snd_wnd is 120k, which is bigger than the sending > buffer, so we boost its size to 150k, and we limit the window > to 120k >=20 > 2. tinfo->tcpi_snd_wnd is 90k, which is smaller than the sending > buffer, so we aren't trying to trigger buffer auto-tuning and > we'll stick to the existing, more conservative calculation, > by limiting the window to 100 - 20 =3D 80k >=20 > If we omitted the new condition, we would always use the boosted > value, that is, 120k, even if potentially causing unnecessary > retransmissions. >=20 > Signed-off-by: Stefano Brivio > --- > tcp.c | 38 ++++++++++++++++++++++++++++++++++++++ > 1 file changed, 38 insertions(+) >=20 > diff --git a/tcp.c b/tcp.c > index 3c046a5..60a9687 100644 > --- a/tcp.c > +++ b/tcp.c > @@ -353,6 +353,13 @@ enum { > #define LOW_RTT_TABLE_SIZE 8 > #define LOW_RTT_THRESHOLD 10 /* us */ > =20 > +/* Parameters to temporarily exceed sending buffer to force TCP auto-tun= ing */ > +#define SNDBUF_BOOST_BYTES_RTT_LO 2500 /* B * s: no boost until here */ > +/* ...examples: 5 MB sent * 500 ns RTT, 250 kB * 10 ms, 8 kB * 300 ms = */ > +#define SNDBUF_BOOST_FACTOR 150 /* % */ > +#define SNDBUF_BOOST_BYTES_RTT_HI 6000 /* apply full boost factor */ > +/* 12 MB sent * 500 ns RTT, 600 kB * 10 ms, 20 kB * 300 ms */ > + > /* Ratio of buffer to bandwidth * delay product implying interactive tra= ffic */ > #define SNDBUF_TO_BW_DELAY_INTERACTIVE /* > */ 20 /* (i.e. < 5% of buffe= r) */ > =20 > @@ -1033,6 +1040,35 @@ void tcp_fill_headers(const struct ctx *c, struct = tcp_tap_conn *conn, > tap_hdr_update(taph, MAX(l3len + sizeof(struct ethhdr), ETH_ZLEN)); > } > =20 > +/** > + * tcp_sndbuf_boost() - Calculate limit of sending buffer to force auto-= tuning > + * @conn: Connection pointer > + * @tinfo: tcp_info from kernel, must be pre-fetched > + * > + * Return: increased sending buffer to use as a limit for advertised win= dow > + */ > +static unsigned long tcp_sndbuf_boost(struct tcp_tap_conn *conn, > + struct tcp_info_linux *tinfo) > +{ > + unsigned long bytes_rtt_product; > + > + if (!bytes_acked_cap) > + return SNDBUF_GET(conn); > + > + /* This is *not* a bandwidth-delay product, but it's somewhat related: > + * as we send more data (usually at the beginning of a connection), we > + * try to make the sending buffer progressively grow, with the RTT as a > + * factor (longer delay, bigger buffer needed). > + */ > + bytes_rtt_product =3D (long long)tinfo->tcpi_bytes_acked * > + tinfo->tcpi_rtt / 1000 / 1000; I only half follow the reasoning in the commit message, but this doesn't see quite right to me. Assuming the RTT is roughly-fixed, as you'd expect, this will always trend to infinity for long-lived connections - regardless of whether they're high throughput or interactive. So, we'll always trend towards using 150% of the send buffer size. > + return scale_x_to_y_slope(SNDBUF_GET(conn), bytes_rtt_product, > + SNDBUF_BOOST_BYTES_RTT_LO, > + SNDBUF_BOOST_BYTES_RTT_HI, > + SNDBUF_BOOST_FACTOR); > +} > + > /** > * tcp_update_seqack_wnd() - Update ACK sequence and window to guest/tap > * @c: Execution context > @@ -1152,6 +1188,8 @@ int tcp_update_seqack_wnd(const struct ctx *c, stru= ct tcp_tap_conn *conn, > =20 > if ((int)sendq > SNDBUF_GET(conn)) /* Due to memory pressure? */ > limit =3D 0; > + else if ((int)tinfo->tcpi_snd_wnd > SNDBUF_GET(conn)) > + limit =3D tcp_sndbuf_boost(conn, tinfo) - (int)sendq; Now that 5/9 has pointed out to be the existence of tcpi_delivery_rate, would it make more sense to do a limit +=3D tcpi_delivery_rate * rtt; The idea being to allow the guest to send as much as the receiver can accomodate itself, plus as much as we can fit "in the air" between us and the peer. > else > limit =3D SNDBUF_GET(conn) - (int)sendq; > =20 > --=20 > 2.43.0 >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --taqgqOi3AReVpdEs Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmk2bz0ACgkQzQJF27ox 2GeHZg/+IRYEmrbVAKN8dzbPMwdw3UJIow7X1b3MEAI97RHHxT6XLsUez3ua1lpV X9CsBfsy17RTrrG2ZV5Q7NMx01WvH7EUwkj4aSuSeEhQul3MmzBZzwydR2gLfXpX /glQ41OWqzI3/XTDiY+uukpdceFBRpLwkZulAUubZyYbufTHS0CSWZxUUUUk3rp0 JffdrJLvxNlduy7sw+ce1fqM8d8cuGane/1p8dF3nZnr5OOzXQZCC4loLhIeKJBI VMGIIYGZHQZAEvKyk1f4RMjbwAeya/oiPrsyWkDM/hmdjYNyl0Vdzl77JHOqXcdV moNDvFIJm0fDnh/65jTPJwz5/3e111qlylDt47b4otnQJufpUo6lje4RjiwYx6BO 0eoiKSBd6rYCD561e2lo5D7D0GjFmq+TrQA0ASMrl0eXRGO6Y52EE7j7VzwPwF74 WLJb8lM0qzG26H3X1oiEwIsnL7CKXNsUzjvCIOnzl1tRBHGFuRFDltdfPNQn1ZiZ 8V6n7gOMRNOQYuVuKF5MUv1sTkv2VKOPuTQoLKa1kebMFmRpgCSTxAwx6uOa5MyL dmnO16mLx2Iq5nVepjo05zspHL0k/xNtPG8K7kfxsguIHxTDSOga9XHh3IZgDHvM nkSbUpyJ0GWbD/dyhHmDPLEv21hJpNk+u86u4iqfk07t9phJgIE= =jlbw -----END PGP SIGNATURE----- --taqgqOi3AReVpdEs--