From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=maxchernoff.ca Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=maxchernoff.ca header.i=@maxchernoff.ca header.a=rsa-sha256 header.s=key1 header.b=9sP1TGPF; dkim-atps=neutral Received: from out-184.mta0.migadu.com (out-184.mta0.migadu.com [91.218.175.184]) by passt.top (Postfix) with ESMTPS id F2EBF5A095E for ; Thu, 13 Nov 2025 01:30:41 +0100 (CET) Message-ID: <4fefd0965271d496dac3e9bc7d33c91f10125f74.camel@maxchernoff.ca> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maxchernoff.ca; s=key1; t=1762993840; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=thM8/nu/DchT5IySLCBAgcsC3nKdU0Vh7pMhYCV/S1A=; b=9sP1TGPFzpOaKgLbwKSm6naIGsxffW7Zm+EV81klmDH0HcBv3NGUOvI9xyWg4fu1yl7oSk QI5Yx4X51kGjPUl7+APDlJ6p5Z1wRvVPmlFkOKKcV4hVf2Km5t8nQBVYtBTL+Yrx+GBe41 /rV49oCNXPecjA10nWWyhFrvG/Uo7sFvyd5UMTiNqLtlSpsaCxbSqylWD0wDsPUbVzfZlx VtENIqbwlee3CjLEjhqonADLyzOQBBlv9k5oKkU3TtGdttE23SrQSrOp3Lh5pbUaRxZxDC 0I6afvC8w0StHp2FPBVv/Mb9pE0Uhx92xxvFocrVkG9wDiNUr2xVh/eg243Kwg== Subject: Re: pasta slow at HTTP upload X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Max Chernoff To: Stefano Brivio Date: Wed, 12 Nov 2025 17:30:35 -0700 In-Reply-To: <20251112135312.595c9a44@elisabeth> References: <176293029592.2033508.497353982367240204@maja> <20251112075548.0c05a25e@elisabeth> <20251112113201.3bcabc6c@elisabeth> <043088ef8bdc2d2c7a910617eb58d494cd9761e0.camel@maxchernoff.ca> <20251112135312.595c9a44@elisabeth> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-MailFrom: git@maxchernoff.ca X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: 6STUAOL5SFKIAWO2N2PCZFAN5YPLKI7F X-Message-ID-Hash: 6STUAOL5SFKIAWO2N2PCZFAN5YPLKI7F X-Mailman-Approved-At: Thu, 13 Nov 2025 08:43:57 +0100 CC: passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Stefano, On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote: > As a further hack, you could probably do something like this on top: > > --- > diff --git a/tcp.c b/tcp.c > index 697f80d..8c50ee0 100644 > --- a/tcp.c > +++ b/tcp.c > @@ -339,7 +339,7 @@ enum { > #define MSS_DEFAULT 536 > #define WINDOW_DEFAULT 14600 /* > RFC 6928 */ > > -#define ACK_INTERVAL 1 /* ms */ > +#define ACK_INTERVAL 200 /* us */ > #define SYN_TIMEOUT 10 /* s */ > #define ACK_TIMEOUT 2 > #define FIN_TIMEOUT 60 > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) > } > > if (conn->flags & ACK_TO_TAP_DUE) { > - it.it_value.tv_nsec =3D (long)ACK_INTERVAL * 1000 * 1000; > + it.it_value.tv_nsec =3D (long)ACK_INTERVAL * 1000; > } else if (conn->flags & ACK_FROM_TAP_DUE) { > if (!(conn->events & ESTABLISHED)) > it.it_value.tv_sec =3D SYN_TIMEOUT; > --- That actually makes it worse again, about as bad as before the patch. But I've just tried rebuilding with original patch again, and also with the exact same binary that I used yesterday, and that's slow now too. I've verified with pgrep that Podman is using the correct pasta version, so I have no idea what's happening. However, I do remember that for the past few months, some uploads would randomly go really quickly, so maybe the problem happens sporadically, and when I was testing the patched version I just happened to get (un)lucky? > > net.core.wmem_max=3D7500000 > > net.core.rmem_max=3D7500000 > > Those were settings we recommended for KubeVirt until > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to > necessarily make sense as we seem to have made peace with the TCP > auto-tuning mechanism in Linux meanwhile. > > See also https://bugs.passt.top/show_bug.cgi?id=3D138 and commit > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to > maximum values"). > > As the issue here is about socket (kernel) buffers being "too small" for > a while, I guess that those settings plus reverting that commit would > "fix" the issue entirely for you. But it's impractical to rely on users > to set those, that's why I'm looking for something adaptive which still > plays nicely with TCP auto-tuning instead. Ah, I didn't know that those (used to be) recommended for pasta; I set those for Caddy since it complains on startup if those aren't set https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes > > net.ipv4.tcp_notsent_lowat=3D131072 > > net.core.default_qdisc=3Dcake > > net.ipv4.tcp_congestion_control=3Dbbr I set those as a general performance tuning thing (not for pasta specifically) based off of https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-= latency/ https://grapheneos.org/articles/server-traffic-shaping > I'm not sure if those really matter for pasta, but I haven't really > thought about them. Aha, those do seem to be the issue. Using the original (unpatched) pasta: (Set everything to my original settings) $ sudo sysctl -w net.core.wmem_max=3D7500000 net.core.rmem_max=3D750000= 0 net.ipv4.tcp_notsent_lowat=3D131072 net.core.default_qdisc=3Dcake net.ipv= 4.tcp_congestion_control=3Dbbr (Test with --network=3Dhost) $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dhost quay.io/fedora/fedora-minimal curl --output= /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan.= org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-= - 20.4M (Test with --network=3Dpasta) $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:0= 7 26188 (Stopped early since I got sick of waiting) (Set everything to the kernel defaults) $ sudo sysctl -w net.core.wmem_max=3D212992 net.core.rmem_max=3D212992 = net.ipv4.tcp_notsent_lowat=3D4294967295 net.core.default_qdisc=3Dfq_codel n= et.ipv4.tcp_congestion_control=3Dcubic $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-= - 1399k (tcp_congestion_control=3Ddefault, tcp_notsent_lowat=3Dcustom, [rw]mem_= max=3Ddefault) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dcubic net.ipv4.tcp_n= otsent_lowat=3D131072 net.core.rmem_max=3D212992 net.core.wmem_max=3D212992 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-= - 1400k (tcp_congestion_control=3Dcustom, tcp_notsent_lowat=3Ddefault, [rw]mem_= max=3Ddefault) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dbbr net.ipv4.tcp_not= sent_lowat=3D4294967295 net.core.rmem_max=3D212992 net.core.wmem_max=3D2129= 92 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-= - 14.9M (tcp_congestion_control=3Dcustom, tcp_notsent_lowat=3Dcustom, [rw]mem_m= ax=3Ddefault) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dbbr net.ipv4.tcp_not= sent_lowat=3D131072 net.core.rmem_max=3D212992 net.core.wmem_max=3D212992 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-= - 17.1M (tcp_congestion_control=3Ddefault, tcp_notsent_lowat=3Ddefault, [rw]mem= _max=3Dcustom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dcubic net.ipv4.tcp_n= otsent_lowat=3D4294967295 net.core.rmem_max=3D7500000 net.core.wmem_max=3D7= 500000 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-= - 8310k (tcp_congestion_control=3Dcustom, tcp_notsent_lowat=3Ddefault, [rw]mem_= max=3Dcustom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dbbr net.ipv4.tcp_not= sent_lowat=3D4294967295 net.core.rmem_max=3D7500000 net.core.wmem_max=3D750= 0000 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-= - 12.1M (tcp_congestion_control=3Dcustom, tcp_notsent_lowat=3Dcustom, [rw]mem_m= ax=3Dcustom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=3Dbbr net.ipv4.tcp_not= sent_lowat=3D131072 net.core.rmem_max=3D7500000 net.core.wmem_max=3D7500000 $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:2= 3 0 (Stopped early since I got sick of waiting) (Test with --network=3Dhost again) $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dhost quay.io/fedora/fedora-minimal curl --output= /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan.= org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-= - 24.3M I know fairly little about networking and the kernel, so if the answer is just "don't set those sysctls together", I'd be okay with that. But I haven't changed these sysctls since February, my upload speeds via pasta were fine up until a few months ago, and the upload speeds are still okay with --network=3Dhost, so I suspect that this is a bug somewhere. I also find it quite interesting that setting any of the sysctls individually or in pairs improves the upload speeds, but setting all 3 at once slows it down drastically. I bisected a kernel a few weeks ago, so I can try that here if you think that this is a kernel bug and not a pasta bug. Thanks, -- Max