* pasta slow at HTTP upload
@ 2025-11-12 6:11 Max Chernoff
0 siblings, 0 replies; 8+ messages in thread
From: Max Chernoff @ 2025-11-12 6:11 UTC (permalink / raw)
To: passt-user
Hi,
For the past few months, I've noticed that my HTTP uploads from
containers are really slow. Reproduction:
$ cd "$(mktemp -d)"
$ head --bytes=100M /dev/urandom > test.bin
$ tar cf ./test.tar.gz ./test.bin
$ chmod -R a+rx .
$ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 23.9M
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
6 100M 0 0 6 6912k 0 9517 3:03:39 0:12:23 2:51:16 0
(Stopped after 10 minutes since I got sick of waiting)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 64 18.7M 0:00:05 0:00:05 --:--:-- 23.8M
Some comments about the above:
- "www.ctan.org" is a web server running Apache; it isn't my server, but
it's where I first noticed this issue, and I have permission from its
owners to use it for testing.
- The upload speed is constant on the host and with --network=host, but
with --network=pasta, it starts high-ish, but quickly and continuously
decreases. But the "bytes transferred" number keeps increasing, so the
upload hasn't stalled out completely.
- I've also had problems with uploads to GitHub from a custom Go
binary---it takes ~3 minutes to upload ~600MB, split across 5 files,
which is much slower than I would expect, but still faster than the
curl/CTAN upload speeds. So I don't think that this is a bug with curl
or this specific web server.
- The host that I ran this from is a VM in a datacenter, supposedly with
a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of
2ms, and pinging www.ctan.org shows an RTT of 100ms.
- This problem seems to have started a few months ago. I don't have an
exact date (sorry), but it seems to have been fine before then.
- The host is running Fedora 43 IoT on x86_64, and I'm using the
following software versions:
$ rpm --query --queryformat='%{NAME}\t%{VERSION}\n' passt podman containers-common netavark | sort | column --table
containers-common 0.64.2
netavark 1.16.1
passt 0^20250919.g623dbf6
podman 5.6.2
I had the same issue when the host/containers were using Fedora 42.
- The host has fairly little CPU/network load when I ran these tests,
and pasta's CPU usage was low enough that it didn't show up on the
first page of htop.
Does anybody have any advice on troubleshooting this? I'm willing to
test with other servers/commands, compile/test any patches, or provide
any other information that might be helpful.
Thanks,
-- Max
^ permalink raw reply [flat|nested] 8+ messages in thread[parent not found: <176293029592.2033508.497353982367240204@maja>]
* Re: pasta slow at HTTP upload [not found] <176293029592.2033508.497353982367240204@maja> @ 2025-11-12 6:55 ` Stefano Brivio 2025-11-12 10:32 ` Stefano Brivio 0 siblings, 1 reply; 8+ messages in thread From: Stefano Brivio @ 2025-11-12 6:55 UTC (permalink / raw) To: Max Chernoff; +Cc: passt-user Hi Max, On Tue, 11 Nov 2025 23:11:17 -0700 Max Chernoff <git@maxchernoff.ca> wrote: > Hi, > > For the past few months, I've noticed that my HTTP uploads from > containers are really slow. Reproduction: > > [...] > > - The host that I ran this from is a VM in a datacenter, supposedly with > a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of > 2ms, and pinging www.ctan.org shows an RTT of 100ms. Thanks for reporting this. Coincidentally, I'm currently debugging an issue that looks similar to this, and I should have some patches for you to test in a couple of days, I'll keep you posted. -- Stefano ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload 2025-11-12 6:55 ` Stefano Brivio @ 2025-11-12 10:32 ` Stefano Brivio 2025-11-12 11:22 ` Max Chernoff 0 siblings, 1 reply; 8+ messages in thread From: Stefano Brivio @ 2025-11-12 10:32 UTC (permalink / raw) To: Max Chernoff; +Cc: passt-user [-- Attachment #1: Type: text/plain, Size: 1040 bytes --] On Wed, 12 Nov 2025 07:55:48 +0100 Stefano Brivio <sbrivio@redhat.com> wrote: > Hi Max, > > On Tue, 11 Nov 2025 23:11:17 -0700 > Max Chernoff <git@maxchernoff.ca> wrote: > > > Hi, > > > > For the past few months, I've noticed that my HTTP uploads from > > containers are really slow. Reproduction: > > > > [...] > > > > - The host that I ran this from is a VM in a datacenter, supposedly with > > a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of > > 2ms, and pinging www.ctan.org shows an RTT of 100ms. > > Thanks for reporting this. Coincidentally, I'm currently debugging an > issue that looks similar to this, and I should have some patches for > you to test in a couple of days, I'll keep you posted. Hmm, actually, I have a hack that's not quite correct (we should make ACK_INTERVAL adaptive instead, which is one of the other bits I'm working on), but if it fixes the issue for you, it should at least mean that we're talking about the same issue. Patch attached. Can you give that a try? -- Stefano [-- Attachment #2: tcp_account_for_sendq_in_window.patch --] [-- Type: text/x-patch, Size: 1336 bytes --] diff --git a/tcp.c b/tcp.c index e91c0cf..697f80d 100644 --- a/tcp.c +++ b/tcp.c @@ -339,7 +339,7 @@ enum { #define MSS_DEFAULT 536 #define WINDOW_DEFAULT 14600 /* RFC 6928 */ -#define ACK_INTERVAL 10 /* ms */ +#define ACK_INTERVAL 1 /* ms */ #define SYN_TIMEOUT 10 /* s */ #define ACK_TIMEOUT 2 #define FIN_TIMEOUT 60 @@ -1097,9 +1097,22 @@ int tcp_update_seqack_wnd(const struct ctx *c, struct tcp_tap_conn *conn, if ((conn->flags & LOCAL) || tcp_rtt_dst_low(conn)) { new_wnd_to_tap = tinfo->tcpi_snd_wnd; } else { + uint32_t sendq; + int limit; + + ioctl(s, SIOCOUTQ, &sendq); tcp_get_sndbuf(conn); + + if ((int)sendq > SNDBUF_GET(conn)) + limit = 0; + else + limit = SNDBUF_GET(conn) - (int)sendq; + + if (limit < MSS_GET(conn)) + limit = 0; + new_wnd_to_tap = MIN((int)tinfo->tcpi_snd_wnd, - SNDBUF_GET(conn)); + limit); } new_wnd_to_tap = MIN(new_wnd_to_tap, MAX_WINDOW); @@ -1240,7 +1253,8 @@ int tcp_prepare_flags(const struct ctx *c, struct tcp_tap_conn *conn, th->fin = !!(flags & FIN); if (th->ack) { - if (SEQ_GE(conn->seq_ack_to_tap, conn->seq_from_tap)) + if (SEQ_GE(conn->seq_ack_to_tap, conn->seq_from_tap) && + conn->wnd_to_tap > MSS_GET(conn)) conn_flag(c, conn, ~ACK_TO_TAP_DUE); else conn_flag(c, conn, ACK_TO_TAP_DUE); ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload 2025-11-12 10:32 ` Stefano Brivio @ 2025-11-12 11:22 ` Max Chernoff 2025-11-12 12:53 ` Stefano Brivio 0 siblings, 1 reply; 8+ messages in thread From: Max Chernoff @ 2025-11-12 11:22 UTC (permalink / raw) To: Stefano Brivio; +Cc: passt-user Hi Stefano, On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote: > Hmm, actually, I have a hack that's not quite correct (we should make > ACK_INTERVAL adaptive instead, which is one of the other bits I'm > working on), but if it fixes the issue for you, it should at least mean > that we're talking about the same issue. > > Patch attached. Can you give that a try? That seemed to help quite a bit---it's now 200x faster than before, but still 10x slower than --network=host: $ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 63 18.4M 0:00:05 0:00:05 --:--:-- 20.3M (With the original pasta, stopped early) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 4 100M 0 0 4 5056k 0 78152 0:22:21 0:01:06 0:21:15 39298⏎ (With the patch applied) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 8 2393k 0:00:42 0:00:42 --:--:-- 4729k $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M Also, I should mention that I'm using the following networking-related sysctls: net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr I read some articles that suggested that those were a good idea, and I've been using them for about a year now, but I can disable those for testing if you want. I'm also using systemd's IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux port restrictions; I can easily disable those too. Thanks, -- Max ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload 2025-11-12 11:22 ` Max Chernoff @ 2025-11-12 12:53 ` Stefano Brivio 2025-11-13 0:30 ` Max Chernoff [not found] ` <176301983731.2033508.12381101277059600955@maja> 0 siblings, 2 replies; 8+ messages in thread From: Stefano Brivio @ 2025-11-12 12:53 UTC (permalink / raw) To: Max Chernoff; +Cc: passt-user On Wed, 12 Nov 2025 04:22:32 -0700 Max Chernoff <git@maxchernoff.ca> wrote: > Hi Stefano, > > On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote: > > Hmm, actually, I have a hack that's not quite correct (we should make > > ACK_INTERVAL adaptive instead, which is one of the other bits I'm > > working on), but if it fixes the issue for you, it should at least mean > > that we're talking about the same issue. > > > > Patch attached. Can you give that a try? > > That seemed to help quite a bit---it's now 200x faster than before, but > still 10x slower than --network=host: Thanks for testing. I'm fairly sure it's that problem, then. Setting 1 ms as interval between checks for socket-side ACKs (reported by the kernel) as my hack does is not the appropriate solution, I'm implementing something based on the reported round-trip-time (RTT) instead. As a further hack, you could probably do something like this on top: --- diff --git a/tcp.c b/tcp.c index 697f80d..8c50ee0 100644 --- a/tcp.c +++ b/tcp.c @@ -339,7 +339,7 @@ enum { #define MSS_DEFAULT 536 #define WINDOW_DEFAULT 14600 /* RFC 6928 */ -#define ACK_INTERVAL 1 /* ms */ +#define ACK_INTERVAL 200 /* us */ #define SYN_TIMEOUT 10 /* s */ #define ACK_TIMEOUT 2 #define FIN_TIMEOUT 60 @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) } if (conn->flags & ACK_TO_TAP_DUE) { - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; } else if (conn->flags & ACK_FROM_TAP_DUE) { if (!(conn->events & ESTABLISHED)) it.it_value.tv_sec = SYN_TIMEOUT; --- ...but again, I'm going to fix that properly in a bit. > Also, I should mention that I'm using the following networking-related > sysctls: > > net.core.wmem_max=7500000 > net.core.rmem_max=7500000 Those were settings we recommended for KubeVirt until https://github.com/kubevirt/user-guide/pull/933, but they don't seem to necessarily make sense as we seem to have made peace with the TCP auto-tuning mechanism in Linux meanwhile. See also https://bugs.passt.top/show_bug.cgi?id=138 and commit 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to maximum values"). As the issue here is about socket (kernel) buffers being "too small" for a while, I guess that those settings plus reverting that commit would "fix" the issue entirely for you. But it's impractical to rely on users to set those, that's why I'm looking for something adaptive which still plays nicely with TCP auto-tuning instead. > net.ipv4.tcp_notsent_lowat=131072 > net.core.default_qdisc=cake > net.ipv4.tcp_congestion_control=bbr I'm not sure if those really matter for pasta, but I haven't really thought about them. > I read some articles that suggested that those were a good idea, and > I've been using them for about a year now, but I can disable those for > testing if you want. I'm also using systemd's > IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux > port restrictions; I can easily disable those too. No need, I don't think those make a difference. -- Stefano ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload 2025-11-12 12:53 ` Stefano Brivio @ 2025-11-13 0:30 ` Max Chernoff [not found] ` <176301983731.2033508.12381101277059600955@maja> 1 sibling, 0 replies; 8+ messages in thread From: Max Chernoff @ 2025-11-13 0:30 UTC (permalink / raw) To: Stefano Brivio; +Cc: passt-user Hi Stefano, On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote: > As a further hack, you could probably do something like this on top: > > --- > diff --git a/tcp.c b/tcp.c > index 697f80d..8c50ee0 100644 > --- a/tcp.c > +++ b/tcp.c > @@ -339,7 +339,7 @@ enum { > #define MSS_DEFAULT 536 > #define WINDOW_DEFAULT 14600 /* > RFC 6928 */ > > -#define ACK_INTERVAL 1 /* ms */ > +#define ACK_INTERVAL 200 /* us */ > #define SYN_TIMEOUT 10 /* s */ > #define ACK_TIMEOUT 2 > #define FIN_TIMEOUT 60 > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) > } > > if (conn->flags & ACK_TO_TAP_DUE) { > - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; > + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; > } else if (conn->flags & ACK_FROM_TAP_DUE) { > if (!(conn->events & ESTABLISHED)) > it.it_value.tv_sec = SYN_TIMEOUT; > --- That actually makes it worse again, about as bad as before the patch. But I've just tried rebuilding with original patch again, and also with the exact same binary that I used yesterday, and that's slow now too. I've verified with pgrep that Podman is using the correct pasta version, so I have no idea what's happening. However, I do remember that for the past few months, some uploads would randomly go really quickly, so maybe the problem happens sporadically, and when I was testing the patched version I just happened to get (un)lucky? > > net.core.wmem_max=7500000 > > net.core.rmem_max=7500000 > > Those were settings we recommended for KubeVirt until > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to > necessarily make sense as we seem to have made peace with the TCP > auto-tuning mechanism in Linux meanwhile. > > See also https://bugs.passt.top/show_bug.cgi?id=138 and commit > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to > maximum values"). > > As the issue here is about socket (kernel) buffers being "too small" for > a while, I guess that those settings plus reverting that commit would > "fix" the issue entirely for you. But it's impractical to rely on users > to set those, that's why I'm looking for something adaptive which still > plays nicely with TCP auto-tuning instead. Ah, I didn't know that those (used to be) recommended for pasta; I set those for Caddy since it complains on startup if those aren't set https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes > > net.ipv4.tcp_notsent_lowat=131072 > > net.core.default_qdisc=cake > > net.ipv4.tcp_congestion_control=bbr I set those as a general performance tuning thing (not for pasta specifically) based off of https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/ https://grapheneos.org/articles/server-traffic-shaping > I'm not sure if those really matter for pasta, but I haven't really > thought about them. Aha, those do seem to be the issue. Using the original (unpatched) pasta: (Set everything to my original settings) $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr (Test with --network=host) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M (Test with --network=pasta) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188 (Stopped early since I got sick of waiting) (Set everything to the kernel defaults) $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default) $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default) $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default) $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom) $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000 $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0 (Stopped early since I got sick of waiting) (Test with --network=host again) $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M I know fairly little about networking and the kernel, so if the answer is just "don't set those sysctls together", I'd be okay with that. But I haven't changed these sysctls since February, my upload speeds via pasta were fine up until a few months ago, and the upload speeds are still okay with --network=host, so I suspect that this is a bug somewhere. I also find it quite interesting that setting any of the sysctls individually or in pairs improves the upload speeds, but setting all 3 at once slows it down drastically. I bisected a kernel a few weeks ago, so I can try that here if you think that this is a kernel bug and not a pasta bug. Thanks, -- Max ^ permalink raw reply [flat|nested] 8+ messages in thread
[parent not found: <176301983731.2033508.12381101277059600955@maja>]
* Re: pasta slow at HTTP upload [not found] ` <176301983731.2033508.12381101277059600955@maja> @ 2025-11-17 5:13 ` David Gibson 2025-11-23 9:12 ` Max Chernoff 0 siblings, 1 reply; 8+ messages in thread From: David Gibson @ 2025-11-17 5:13 UTC (permalink / raw) To: Max Chernoff; +Cc: Stefano Brivio, passt-user [-- Attachment #1: Type: text/plain, Size: 13366 bytes --] On Wed, Nov 12, 2025 at 05:30:35PM -0700, Max Chernoff via user wrote: > Date: Wed, 12 Nov 2025 17:30:35 -0700 > From: Max Chernoff <git@maxchernoff.ca> > To: Stefano Brivio <sbrivio@redhat.com> > CC: passt-user@passt.top > Subject: Re: pasta slow at HTTP upload > List-Id: "For passt users: support, questions and answers" > <passt-user.passt.top> > > Hi Stefano, > > On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote: > > As a further hack, you could probably do something like this on top: > > > > --- > > diff --git a/tcp.c b/tcp.c > > index 697f80d..8c50ee0 100644 > > --- a/tcp.c > > +++ b/tcp.c > > @@ -339,7 +339,7 @@ enum { > > #define MSS_DEFAULT 536 > > #define WINDOW_DEFAULT 14600 /* > > RFC 6928 */ > > > > -#define ACK_INTERVAL 1 /* ms */ > > +#define ACK_INTERVAL 200 /* us */ > > #define SYN_TIMEOUT 10 /* s */ > > #define ACK_TIMEOUT 2 > > #define FIN_TIMEOUT 60 > > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) > > } > > > > if (conn->flags & ACK_TO_TAP_DUE) { > > - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; > > + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; > > } else if (conn->flags & ACK_FROM_TAP_DUE) { > > if (!(conn->events & ESTABLISHED)) > > it.it_value.tv_sec = SYN_TIMEOUT; > > --- > > That actually makes it worse again, about as bad as before the patch. > But I've just tried rebuilding with original patch again, and also with > the exact same binary that I used yesterday, and that's slow now too. > I've verified with pgrep that Podman is using the correct pasta version, > so I have no idea what's happening. > > However, I do remember that for the past few months, some uploads would > randomly go really quickly, so maybe the problem happens sporadically, > and when I was testing the patched version I just happened to get > (un)lucky? > > > > net.core.wmem_max=7500000 > > > net.core.rmem_max=7500000 > > > > Those were settings we recommended for KubeVirt until > > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to > > necessarily make sense as we seem to have made peace with the TCP > > auto-tuning mechanism in Linux meanwhile. > > > > See also https://bugs.passt.top/show_bug.cgi?id=138 and commit > > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to > > maximum values"). > > > > As the issue here is about socket (kernel) buffers being "too small" for > > a while, I guess that those settings plus reverting that commit would > > "fix" the issue entirely for you. But it's impractical to rely on users > > to set those, that's why I'm looking for something adaptive which still > > plays nicely with TCP auto-tuning instead. > > Ah, I didn't know that those (used to be) recommended for pasta; I set > those for Caddy since it complains on startup if those aren't set > > https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes > > > > net.ipv4.tcp_notsent_lowat=131072 > > > net.core.default_qdisc=cake > > > net.ipv4.tcp_congestion_control=bbr > > I set those as a general performance tuning thing (not for pasta > specifically) based off of > > https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/ > > https://grapheneos.org/articles/server-traffic-shaping > > > I'm not sure if those really matter for pasta, but I haven't really > > thought about them. > > Aha, those do seem to be the issue. Using the original (unpatched) > pasta: Huh, interesting. > (Set everything to my original settings) > $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr > > (Test with --network=host) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M > > (Test with --network=pasta) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188 > (Stopped early since I got sick of waiting) > > (Set everything to the kernel defaults) > $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k > > (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M > > (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k > > (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M > > (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom) > $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000 > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0 > (Stopped early since I got sick of waiting) > > (Test with --network=host again) > $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate" > % Total % Received % Xferd Average Speed Time Time Time Current > Dload Upload Total Spent Left Speed > 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat is the biggest culprit... except for the outlier where it seems ok with both custom notsent_lowat and custom congestion control. How repeatable are each of these results? > I know fairly little about networking and the kernel, so if the answer > is just "don't set those sysctls together", I'd be okay with that. But I > haven't changed these sysctls since February, my upload speeds via pasta > were fine up until a few months ago, and the upload speeds are still > okay with --network=host, so I suspect that this is a bug somewhere. I have a vague theory why tcp_notsent_lowat might be bad for pasta but not --network=host. I'm far from certain about it though, and I don't have a good idea what change on the pasta side might have changed. Actually, is it just pasta that's changed in that time? Or could it be a new kernel version or something else? I'm having some trouble wrapping my head around what tcp_notsent_lowat does, but I _think_ the upshot of it will be to make the socket send path more "bursty": we'll fill the socket send buffer (which could be very large), then stop. We won't wake up again to refill until the amount in the socket buffer drops below tcp_notsend_lowat. Assuming we've been getting plenty of data from the other side, we'll then refill the buffer very quickly. Less wakups sounds like a good thing in the abstract, but I'm wondering if this burstiness is messing with the flow control on the other side, causing the guest side peer to slow down to nothing. If that is the case, it would make sense that it gets even worse with large socket buffer sizes, since that would make the "bursts" even bigger. Likewise I was wondering if having different congestion control algorithms on each side might somehow conflict causing a slowdown. I don't think the data you have above really suggest that though. > I also find it quite interesting that setting any of the sysctls > individually or in pairs improves the upload speeds, but setting all 3 > at once slows it down drastically. I'm curious about the case with default congestion control, but custom lowat and mem_max. I think that one's missing from the table above. > I bisected a kernel a few weeks ago, > so I can try that here if you think that this is a kernel bug and not a > pasta bug. Could be either, I have no real intuition either way so far. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload 2025-11-17 5:13 ` David Gibson @ 2025-11-23 9:12 ` Max Chernoff 0 siblings, 0 replies; 8+ messages in thread From: Max Chernoff @ 2025-11-23 9:12 UTC (permalink / raw) To: David Gibson; +Cc: Stefano Brivio, passt-user Hi David, On Mon, 2025-11-17 at 16:13 +1100, David Gibson wrote: > Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat > is the biggest culprit... except for the outlier where it seems ok > with both custom notsent_lowat and custom congestion control. > > How repeatable are each of these results? Very repeatable. I reran the tests from a different host (my laptop over home Wifi) targeting different servers (two servers that I control) and I got similar results. > Actually, is it just pasta that's changed in that time? Or could it > be a new kernel version or something else? I'm always just using the most recent packages in Fedora, so both pasta and the kernel would have upgraded in that time. Specifically: - Around June 6th, I didn't notice any issues; I was using "kernel-6.14.9-300.fc42.x86_64" and "passt-0^20250512.g8ec1341-1.fc42.x86_64". - Around August 20th, I started noticing intermittent issues; I was using "kernel-6.15.9-201.fc42.x86_64" and "passt-0^20250805.g309eefd-2.fc42.x86_64". - Around September 26th, the problem became persistent; I was using "kernel-6.16.8-200.fc42.x86_64" and "passt-0^20250919.g623dbf6-1.fc42.x86_64". - I ran the tests in this email with "kernel-6.17.8-300.fc43.x86_64" and "passt-0^20250919.g623dbf6-1.fc43.x86_64". > > I also find it quite interesting that setting any of the sysctls > > individually or in pairs improves the upload speeds, but setting all 3 > > at once slows it down drastically. > > I'm curious about the case with default congestion control, but custom > lowat and mem_max. I think that one's missing from the table above. Ok, I ran some more tests. The problematic combination seems to be network=pasta direction=upload rtt/ping=large net.core.wmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.ipv4.tcp_congestion_control=bbr and all other parameters seem to not make much difference. The data is somewhat noisy, but still fairly consistent. I ran the tests for all combinations for 10MB twice and 100MB once. The full data is below: network ping size wmem rmem lowat cngst qdisc down up pasta 170ms 100MB # . # # # 17.69 180.00 pasta 170ms 100MB # # # # . 15.41 180.00 pasta 170ms 100MB . # # . . 15.23 180.00 pasta 170ms 100MB # . # # . 11.86 180.00 pasta 170ms 100MB # # # # # 10.90 180.00 pasta 170ms 100MB # . # . . 23.91 128.18 pasta 170ms 10MB . . . # # 6.98 120.00 pasta 170ms 10MB # # # # # 6.26 120.00 pasta 170ms 10MB # . # # # 6.19 120.00 pasta 170ms 10MB # . # # # 6.10 120.00 pasta 170ms 10MB # . # # . 6.05 120.00 pasta 170ms 10MB # . # # . 5.93 120.00 pasta 170ms 10MB # # # # . 5.57 120.00 pasta 170ms 10MB . # . # . 4.45 120.00 pasta 170ms 100MB . # # . # 17.46 118.70 pasta 50ms 100MB # . # # # 6.57 117.59 pasta 170ms 100MB . . # . # 12.91 94.23 pasta 170ms 100MB . . # . . 12.46 89.22 pasta 170ms 100MB . # # # # 15.05 85.79 pasta 170ms 100MB . . . # # 12.92 82.36 pasta 170ms 100MB . . # # . 15.03 76.69 pasta 170ms 100MB . # . # . 18.91 74.79 pasta 170ms 100MB . . # # # 21.75 66.75 pasta 170ms 100MB # # # . . 23.02 66.36 pasta 50ms 100MB . . # # # 7.57 58.58 pasta 50ms 100MB . . # # . 8.46 58.24 pasta 170ms 100MB # # # . # 12.15 55.27 pasta 170ms 100MB . # # # . 17.17 54.86 pasta 50ms 100MB . # # . . 6.80 51.66 pasta 170ms 100MB # . # . # 11.89 51.40 pasta 170ms 10MB . . . # . 7.25 51.27 pasta 170ms 10MB . . # # # 6.98 49.34 pasta 50ms 100MB . # # # # 6.79 47.94 pasta 170ms 100MB . . . # . 11.98 45.94 pasta 50ms 100MB . . # . . 8.62 45.85 pasta 50ms 100MB # . # # . 7.05 45.24 pasta 50ms 100MB . . # . # 8.25 44.98 pasta 170ms 10MB # # # # # 6.91 44.95 pasta 170ms 10MB # # . . . 6.16 44.43 pasta 50ms 100MB . # # . # 8.47 44.18 pasta 50ms 100MB # . # . . 6.97 43.38 pasta 170ms 10MB # . # . . 5.27 42.54 pasta 170ms 10MB . # # . . 4.92 42.46 pasta 170ms 10MB # # # . # 6.15 42.36 pasta 170ms 10MB . # # . # 7.06 41.61 pasta 50ms 100MB . # # # . 6.66 40.83 pasta 170ms 100MB . # . # # 14.50 40.45 pasta 170ms 10MB # # # . # 7.10 39.84 pasta 170ms 10MB . # # . # 5.76 38.34 pasta 50ms 100MB # # # . # 6.81 37.96 pasta 170ms 10MB # # . # . 8.50 37.88 pasta 170ms 10MB . # # # . 5.47 37.85 pasta 170ms 10MB # . # . # 3.82 37.46 pasta 170ms 10MB # # # . . 6.79 37.19 pasta 170ms 10MB . # # # . 5.34 37.03 host 170ms 100MB # . # # # 15.09 36.65 pasta 170ms 10MB # # # . . 7.19 36.25 pasta 50ms 100MB # . # . # 7.63 35.72 pasta 170ms 10MB # . # . # 6.53 35.69 pasta 170ms 10MB # # # # . 8.35 35.69 pasta 170ms 100MB # . . # # 13.78 35.65 pasta 170ms 100MB # # . # # 16.32 35.20 host 170ms 100MB . . # # # 13.61 33.89 host 170ms 100MB . . . # . 14.76 33.87 host 170ms 100MB # . . # # 16.60 33.79 host 170ms 100MB # . # # . 12.71 33.41 pasta 170ms 100MB # # . # . 15.82 33.28 pasta 170ms 100MB . # . . # 13.55 33.25 host 170ms 100MB # # . # . 13.97 33.19 host 170ms 100MB . . # # . 17.49 33.15 host 170ms 100MB . # # # . 14.68 32.89 host 170ms 100MB . # # # # 17.12 32.71 host 170ms 100MB # . . # . 22.21 32.71 host 170ms 100MB # # # # . 12.18 32.44 pasta 170ms 100MB # . . # . 16.30 32.23 host 170ms 100MB # # . # # 15.46 32.08 host 170ms 100MB . # . # # 14.47 31.97 host 170ms 100MB . . . # # 16.80 31.84 host 170ms 100MB . # . # . 13.04 31.77 pasta 170ms 100MB . . . . # 15.85 31.74 host 170ms 100MB # # # # # 19.09 31.57 pasta 170ms 10MB . . # # . 5.62 30.59 pasta 170ms 10MB . # # # # 5.58 30.46 pasta 170ms 100MB . . . . . 11.22 30.22 pasta 170ms 100MB # . . . # 18.42 30.17 pasta 170ms 100MB . # . . . 14.93 30.09 pasta 170ms 100MB # # . . # 14.24 29.86 pasta 50ms 100MB # # # . . 6.57 29.81 host 170ms 100MB . # . . . 15.52 29.10 pasta 170ms 10MB . . # # . 5.11 28.77 host 170ms 100MB . # . . # 15.73 28.63 host 170ms 100MB # # # . . 16.57 28.54 host 170ms 100MB . # # . # 15.03 28.41 host 170ms 100MB . # # . . 14.29 28.19 host 170ms 100MB . . . . . 15.26 28.04 pasta 170ms 100MB # # . . . 14.35 27.76 pasta 170ms 10MB . . # . # 5.60 27.74 pasta 50ms 10MB # . # # . 1.37 27.68 pasta 170ms 10MB . # . # # 4.47 27.56 host 170ms 100MB . . . . # 16.84 27.43 host 170ms 100MB # # # . # 14.88 27.27 host 170ms 100MB . . # . # 17.53 26.75 host 170ms 100MB # . # . # 15.83 26.69 pasta 170ms 10MB . # . # # 5.29 26.13 host 170ms 100MB # # . . # 16.73 26.04 host 170ms 100MB # # . . . 19.30 25.79 pasta 170ms 100MB # . . . . 14.31 25.78 host 170ms 100MB # . # . . 20.52 25.61 pasta 170ms 10MB # . # . . 6.37 24.39 pasta 170ms 10MB . . # . . 7.07 24.33 host 170ms 100MB # . . . # 17.69 24.23 host 170ms 100MB # . . . . 19.94 23.75 pasta 50ms 100MB # # # # # 6.37 23.43 pasta 50ms 10MB # # # # . 1.52 22.99 pasta 50ms 100MB # # # # . 6.31 22.65 pasta 170ms 10MB . . # . # 6.52 22.01 pasta 50ms 10MB # . # # . 1.35 21.59 pasta 170ms 10MB . . . # . 7.15 21.35 pasta 170ms 10MB . . # . . 6.40 20.86 pasta 50ms 100MB . . . # . 8.43 20.49 pasta 50ms 100MB # # . # . 6.50 19.70 host 170ms 10MB # . . # # 4.71 19.61 pasta 50ms 100MB . # . # # 8.03 19.47 pasta 50ms 100MB . . . # # 8.40 19.29 pasta 170ms 10MB # # . # # 7.63 18.92 pasta 50ms 100MB . # . # . 7.09 18.88 host 50ms 100MB . . # # . 7.94 18.85 host 50ms 100MB . # . # . 7.15 18.74 host 170ms 100MB . . # . . 13.75 18.71 host 170ms 10MB . . . # . 4.52 18.57 host 170ms 10MB . . . # . 6.24 18.52 pasta 170ms 10MB . . # # # 7.04 18.48 host 50ms 100MB # . . # . 6.38 18.45 host 50ms 100MB . # . # # 7.37 17.89 pasta 170ms 10MB . # . # . 7.35 17.88 host 170ms 10MB # # . # . 6.11 17.76 host 50ms 100MB . . # . . 8.20 17.56 host 170ms 10MB . . # # . 6.56 17.51 pasta 170ms 10MB # . . # # 6.39 17.51 host 170ms 10MB . # # # # 4.86 17.46 host 170ms 10MB # # . # # 6.91 17.45 host 50ms 100MB # # # # # 7.11 17.45 host 170ms 10MB . # . # . 4.93 17.45 host 170ms 10MB . # # # . 5.14 17.44 pasta 170ms 10MB . # # # # 5.69 17.41 host 170ms 10MB # . # # # 3.64 17.39 pasta 170ms 10MB # # . # # 6.59 17.33 host 170ms 10MB . # . # # 4.83 17.33 pasta 170ms 10MB # . . # . 5.46 17.30 pasta 50ms 100MB # . . # # 6.23 17.30 pasta 170ms 10MB # . . # . 4.30 17.25 host 170ms 10MB . . # # # 5.64 17.25 pasta 50ms 100MB # . . # . 6.76 17.20 pasta 170ms 10MB . . . # # 7.77 17.19 host 50ms 100MB . . . # # 7.61 17.18 host 170ms 10MB . . . # # 6.01 17.16 host 50ms 100MB . # # # # 6.44 17.04 host 170ms 10MB # # # # . 5.08 17.04 host 170ms 10MB . # # # . 4.62 17.02 host 50ms 100MB # . # # # 6.51 17.02 host 50ms 100MB # . . # # 6.20 16.99 host 50ms 100MB # . # # . 6.43 16.98 pasta 170ms 10MB # # . # . 5.87 16.88 pasta 50ms 100MB # # . # # 6.36 16.86 host 170ms 10MB . . . # # 5.03 16.85 host 170ms 10MB # # # # # 3.44 16.83 host 170ms 10MB # # . # # 6.12 16.83 host 170ms 10MB # . # # . 5.62 16.83 pasta 170ms 10MB # . . # # 14.82 16.83 host 170ms 10MB # . . # . 5.14 16.82 host 170ms 10MB # . # # . 6.47 16.81 host 170ms 10MB . # . # # 5.13 16.79 host 170ms 10MB # . . # . 5.72 16.78 host 170ms 10MB # # # # # 5.49 16.78 host 170ms 10MB # # # # . 4.72 16.77 host 170ms 10MB # # . # . 5.79 16.76 host 50ms 100MB . . # # # 7.41 16.71 host 170ms 10MB . . # # # 6.10 16.70 host 50ms 100MB . # . . # 6.69 16.68 host 170ms 10MB . # # # # 6.06 16.67 host 170ms 10MB # . . # # 6.01 16.65 host 170ms 10MB . # . # . 5.32 16.58 host 170ms 10MB . . # # . 4.90 16.55 host 50ms 100MB # # . # # 6.47 16.40 pasta 170ms 10MB . # # . . 4.81 16.19 host 170ms 10MB # . # # # 5.77 16.06 host 50ms 100MB . # # # . 6.67 15.97 pasta 170ms 10MB # # . . # 6.78 15.23 host 50ms 100MB # # # # . 6.56 14.97 pasta 50ms 100MB . . . . . 8.19 14.95 pasta 170ms 10MB . . . . . 6.10 14.69 pasta 170ms 10MB # . . . . 5.78 14.51 pasta 170ms 10MB . # . . # 6.42 14.22 pasta 170ms 10MB . # . . . 6.30 14.20 host 50ms 100MB # # . # . 6.12 14.08 pasta 170ms 10MB . . . . # 6.63 14.07 pasta 170ms 10MB . # . . # 4.93 13.53 pasta 50ms 10MB # # # # # 1.29 13.41 pasta 170ms 10MB # . . . # 6.31 13.11 host 50ms 100MB . # # . # 7.62 13.07 pasta 170ms 10MB # # . . . 6.41 13.03 host 50ms 10MB # # # # . 1.80 12.98 host 50ms 100MB . . . # . 7.67 12.88 pasta 170ms 10MB # . . . . 5.65 12.76 pasta 170ms 10MB . . . . # 5.88 12.59 pasta 170ms 10MB # # . . # 5.69 12.55 host 50ms 100MB # . # . . 6.67 12.52 pasta 50ms 10MB # . # # # 1.22 12.50 host 170ms 10MB # . # . . 5.41 12.41 pasta 50ms 100MB # # . . . 6.58 12.28 pasta 50ms 10MB # # # # # 2.12 12.20 pasta 170ms 10MB . # . . . 6.56 12.16 pasta 170ms 10MB . . . . . 5.38 12.11 host 170ms 10MB # # # . # 5.60 12.00 pasta 170ms 10MB # . . . # 10.77 11.98 pasta 50ms 10MB # # # # . 1.19 11.92 pasta 50ms 100MB # . . . . 6.63 11.81 host 170ms 10MB . . # . # 5.55 11.80 host 50ms 100MB . # . . . 6.64 11.71 host 170ms 10MB # # # . # 5.35 11.70 pasta 50ms 100MB . # . . # 7.40 11.69 host 170ms 10MB # . # . # 6.18 11.66 host 50ms 100MB # # # . . 6.40 11.60 pasta 50ms 10MB . # # # . 1.24 11.51 host 170ms 10MB . # . . # 5.54 11.40 pasta 50ms 10MB . . # # # 1.35 11.38 host 170ms 10MB . # . . . 6.33 11.23 host 170ms 10MB # . # . # 4.25 11.22 host 50ms 100MB # # . . . 6.90 11.19 host 170ms 10MB . # # . . 6.59 11.16 host 50ms 100MB . # # . . 7.29 11.09 host 50ms 100MB . . # . # 8.63 11.07 host 170ms 10MB . . . . # 6.00 10.99 pasta 50ms 100MB # . . . # 7.90 10.86 host 170ms 10MB . . . . . 6.66 10.86 pasta 50ms 10MB # . # . # 2.61 10.84 host 170ms 10MB # . # . . 5.41 10.83 host 170ms 10MB # . . . # 5.80 10.76 host 170ms 10MB . . # . . 5.38 10.68 pasta 50ms 100MB . # . . . 7.13 10.68 host 170ms 10MB # # . . . 8.34 10.68 pasta 50ms 10MB . # # # # 1.39 10.66 pasta 50ms 10MB . # # # . 1.39 10.64 host 170ms 10MB # . . . . 4.92 10.64 host 170ms 10MB # # . . # 5.70 10.60 pasta 50ms 100MB . . . . # 6.92 10.56 host 50ms 100MB . . . . # 6.86 10.56 pasta 50ms 100MB # # . . # 6.32 10.53 host 170ms 10MB . . # . # 6.53 10.52 host 50ms 100MB # . . . . 6.67 10.52 host 170ms 10MB . . . . # 5.04 10.48 host 50ms 100MB # # # . # 6.39 10.45 host 170ms 10MB . # . . # 5.09 10.36 host 170ms 10MB # # # . . 6.01 10.29 host 50ms 100MB # . # . # 8.37 10.19 host 170ms 10MB . # # . . 7.35 10.14 host 170ms 10MB . # . . . 5.03 10.13 host 170ms 10MB # # # . . 6.11 10.08 host 170ms 10MB . # # . # 6.63 10.04 host 50ms 100MB # . . . # 7.07 10.04 host 50ms 100MB . . . . . 7.68 10.01 host 170ms 10MB # # . . # 4.41 9.96 host 170ms 10MB . . . . . 6.56 9.92 host 170ms 10MB # . . . . 5.72 9.88 pasta 50ms 10MB # . # . # 1.64 9.82 host 50ms 100MB # # . . # 6.32 9.72 host 170ms 10MB # . . . # 6.53 9.60 pasta 50ms 10MB # # # . # 1.53 9.25 host 170ms 10MB # # . . . 5.07 9.16 pasta 50ms 10MB . . # # # 1.41 9.10 host 170ms 10MB . . # . . 6.06 8.95 host 170ms 10MB . # # . # 6.71 8.83 pasta 50ms 10MB . . # . . 1.42 8.81 pasta 50ms 10MB . . # # . 1.45 8.65 pasta 50ms 10MB . # # # # 1.28 8.35 pasta 50ms 10MB # . # . . 1.47 7.92 pasta 50ms 10MB . # # . # 1.51 7.78 pasta 50ms 10MB # # # . . 1.87 7.61 pasta 50ms 10MB . . # # . 1.53 7.32 pasta 50ms 10MB # . # # # 1.33 7.22 pasta 50ms 10MB # # # . # 1.31 7.15 pasta 50ms 10MB . . . . . 1.34 7.12 pasta 50ms 10MB # # # . . 1.42 6.95 pasta 50ms 10MB . # # . . 1.51 6.83 pasta 50ms 10MB . # . . # 1.79 6.76 host 50ms 10MB . . . # # 1.42 6.66 pasta 50ms 10MB # . # . . 1.60 6.26 pasta 50ms 10MB # . . # # 1.55 5.66 pasta 50ms 10MB . # # . . 1.55 5.62 pasta 50ms 10MB . # . . . 1.44 5.53 pasta 50ms 10MB # # . # # 1.30 5.52 pasta 50ms 10MB . . # . # 1.43 5.28 pasta 50ms 10MB . . . . # 1.35 5.20 pasta 50ms 10MB . . . . # 1.49 5.15 pasta 50ms 10MB . # # . # 1.47 5.13 pasta 50ms 10MB . . # . . 1.40 5.10 pasta 50ms 10MB . . . . . 1.35 5.10 pasta 50ms 10MB . . # . # 1.40 5.08 host 50ms 10MB . # . # # 1.64 4.99 pasta 50ms 10MB # . . . # 1.44 4.69 pasta 50ms 10MB . # . # # 2.60 4.67 pasta 50ms 10MB . # . . . 1.45 4.63 pasta 50ms 10MB . # . . # 1.34 4.63 host 50ms 10MB . . # # . 1.31 4.57 host 50ms 10MB # . . # # 1.37 4.51 host 50ms 10MB # # . # # 1.24 4.48 pasta 50ms 10MB # # . . # 1.55 4.44 host 50ms 10MB # # . . # 1.49 4.42 pasta 50ms 10MB # # . . . 1.22 4.35 host 50ms 10MB . # . # . 1.29 4.33 host 50ms 10MB . # # # . 1.36 4.27 host 50ms 10MB # # # # # 1.32 4.04 pasta 50ms 10MB . . . # # 1.31 3.89 pasta 50ms 10MB . # . # . 1.33 3.88 pasta 50ms 10MB . . . # # 1.50 3.85 host 50ms 10MB # . # # . 1.43 3.78 host 50ms 10MB . . . # . 1.28 3.75 host 50ms 10MB # . # . # 1.52 3.74 pasta 50ms 10MB # . . # . 1.38 3.73 host 50ms 10MB . # # # # 1.36 3.68 host 50ms 10MB # # . # . 1.21 3.68 host 50ms 10MB . # . # # 1.41 3.66 pasta 50ms 10MB # . . # . 1.43 3.66 host 50ms 10MB . . . # # 1.62 3.65 pasta 50ms 10MB # # . # . 1.51 3.64 host 50ms 10MB . . # # # 1.31 3.64 host 50ms 10MB . . # . . 1.48 3.61 pasta 50ms 10MB # . . . . 1.30 3.60 pasta 50ms 10MB # . . . # 1.49 3.56 host 50ms 10MB . . . . # 1.30 3.54 pasta 50ms 10MB # # . # # 1.33 3.53 host 50ms 10MB # # . # . 1.38 3.52 host 50ms 10MB # . # # # 1.18 3.51 host 50ms 10MB . . . # . 1.59 3.51 pasta 50ms 10MB . . . # . 1.58 3.49 host 50ms 10MB . . . . . 1.48 3.48 host 50ms 10MB . # . . # 1.57 3.46 pasta 50ms 10MB # . . . . 1.49 3.45 pasta 50ms 10MB . # . # # 1.46 3.44 host 50ms 10MB . # # . . 1.41 3.41 host 50ms 10MB # # # . . 1.34 3.40 host 50ms 10MB # # # . # 1.34 3.40 host 50ms 10MB # # # # . 1.29 3.39 host 50ms 10MB # . # . . 1.87 3.39 host 50ms 10MB # # # . # 1.38 3.38 host 50ms 10MB . . . . . 1.43 3.37 pasta 50ms 10MB # # . . . 1.29 3.36 host 50ms 10MB # . # . # 1.59 3.33 host 50ms 10MB . . # . # 1.46 3.32 host 50ms 10MB # . . . . 1.47 3.29 host 50ms 10MB . # . . . 1.49 3.28 host 50ms 10MB . # . . # 1.38 3.28 host 50ms 10MB . . # . . 1.39 3.28 host 50ms 10MB # . . # . 1.43 3.27 host 50ms 10MB # . . # . 1.35 3.27 pasta 50ms 10MB # . . # # 1.39 3.26 host 50ms 10MB # # . . . 1.28 3.24 host 50ms 10MB . # # . # 1.43 3.24 host 50ms 10MB # . . . # 1.57 3.19 host 50ms 10MB . # # # . 1.27 3.19 host 50ms 10MB . # # # # 1.54 3.18 host 50ms 10MB . . # # . 1.34 3.14 host 50ms 10MB # . # # . 1.32 3.14 pasta 50ms 10MB . . . # . 1.58 3.12 host 50ms 10MB . # # . . 1.59 3.09 host 50ms 10MB . # . # . 1.43 3.08 pasta 50ms 10MB # # . . # 1.30 3.07 pasta 50ms 10MB # # . # . 1.35 3.01 host 50ms 10MB # # # # # 1.53 2.99 host 50ms 10MB . # # . # 1.54 2.94 host 50ms 10MB # . # . . 1.58 2.94 host 50ms 10MB . # . . . 1.43 2.87 host 50ms 10MB . . # . # 1.46 2.87 host 50ms 10MB # . # # # 1.36 2.87 host 50ms 10MB # # # . . 1.40 2.84 host 50ms 10MB # # . . # 1.29 2.79 host 50ms 10MB # . . . . 1.32 2.78 host 50ms 10MB # . . . # 1.37 2.68 host 50ms 10MB # # . . . 1.21 2.67 host 50ms 10MB . . . . # 1.37 2.66 host 50ms 10MB # # . # # 1.34 2.39 host 50ms 10MB # . . # # 1.43 2.15 pasta 50ms 10MB . # . # . 1.45 2.04 host 50ms 10MB . . # # # 1.54 1.93 column description network podman --network= argument ping the RTT to the remote server size the size of the file {up,down}loaded wmem sysctl rmem sysctl lowat sysctl cngst sysctl qdisc sysctl down download time in seconds up upload time in seconds (timed out at 120s for the 10MB tests and 180s for the 100MB tests ) column sysctl "#" (custom) "." (default) wmem net.core.wmem_max 7500000 212992 rmem net.core.rmem_max 7500000 212992 lowat net.ipv4.tcp_notsent_lowat 131072 4294967295 cngst net.ipv4.tcp_congestion_control bbr cubic qdisc net.core.default_qdisc cake fq_codel And this is the messy script that I used to generate this data (I'm hosting both servers myself, so feel free to use them for testing): #!/usr/bin/bash set -euo pipefail # The location of the files script_dir="$(dirname "$(realpath "$0")")" output_file="$script_dir/pasta-upload-test-results.tsv" # The size of the file to upload/download file_size="100M" # Define the sysctl parameters declare -a -r sysctl_keys=( "custom" "default" ) declare -A -r wmem_maxes=( [default]=212992 [custom]=7500000 ) declare -A -r rmem_maxes=( [default]=212992 [custom]=7500000 ) declare -A -r tcp_notsent_lowats=( [default]=4294967295 [custom]=131072 ) declare -A -r tcp_congestion_controls=( [default]="cubic" [custom]="bbr" ) declare -A -r default_qdiscs=( [default]="fq_codel" [custom]="cake" ) # Define the network parameters declare -a -r podman_networks=("host" "pasta") declare -A -r servers=( [50ms]="https://www.maxchernoff.ca/tools/speedtest" [170ms]="https://svn.tug.org:8369/speedtest" ) # Initialize the temporary directory temp_dir=$(mktemp -d) cd "$temp_dir" trap 'rm -rf "$temp_dir"' EXIT # Define the network test commands download() { local network=$1 local server=$2 podman run \ --quiet \ --rm \ --pull="never" \ --volume="$temp_dir:/srv/:Z" \ --workdir="/srv/" \ --network="$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=SIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ "$server?bytes=$file_size" \ || echo "180.000000" } upload() { local network=$1 local server=$2 podman run \ --quiet \ --rm \ --pull="never" \ --volume="$temp_dir:/srv/:Z" \ --workdir="/srv/" \ --network="$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=SIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ -X POST \ --upload-file test.bin \ "$server" \ || echo "180.000000" } # Pull the container podman pull --quiet quay.io/fedora/fedora-minimal >/dev/null 2>&1 # Create a sample file to upload head --bytes="$file_size" /dev/urandom > test.bin # Run the tests printf 'network\tping_time\twmem_max\trmem_max\ttcp_notsent_lowat\ttcp_congestion_control\tdefault_qdisc\tdownload_time\tupload_time\n' \ | tee "$output_file" for wmem_max in "${sysctl_keys[@]}"; do for rmem_max in "${sysctl_keys[@]}"; do for tcp_notsent_lowat in "${sysctl_keys[@]}"; do for tcp_congestion_control in "${sysctl_keys[@]}"; do for default_qdisc in "${sysctl_keys[@]}"; do sudo sysctl -w \ net.core.wmem_max="${wmem_maxes[$wmem_max]}" \ net.core.rmem_max="${rmem_maxes[$rmem_max]}" \ net.ipv4.tcp_notsent_lowat="${tcp_notsent_lowats[$tcp_notsent_lowat]}" \ net.ipv4.tcp_congestion_control="${tcp_congestion_controls[$tcp_congestion_control]}" \ net.core.default_qdisc="${default_qdiscs[$default_qdisc]}" \ > /dev/null for server in "${!servers[@]}"; do for network in "${podman_networks[@]}"; do printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \ "$network" \ "$server" \ "$wmem_max" \ "$rmem_max" \ "$tcp_notsent_lowat" \ "$tcp_congestion_control" \ "$default_qdisc" \ "$(download "$network" "${servers[$server]}")" \ "$(upload "$network" "${servers[$server]}")" \ | tee -a "$output_file" done done done done done done done # Sort the lines cat "$output_file" | \ sort -k9rn -k8rn | \ column --table --table-right=2,8,9 --output-width=999 | \ tac | awk 'NR==1 {line=$0; next} 1; END{print line}' | tac | \ sponge "$output_file" > > I bisected a kernel a few weeks ago, > > so I can try that here if you think that this is a kernel bug and not a > > pasta bug. > > Could be either, I have no real intuition either way so far. Ok, I'll try testing with some older versions of both and see if that narrows down the issue. Thanks, -- Max ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-11-23 9:12 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-11-12 6:11 pasta slow at HTTP upload Max Chernoff
[not found] <176293029592.2033508.497353982367240204@maja>
2025-11-12 6:55 ` Stefano Brivio
2025-11-12 10:32 ` Stefano Brivio
2025-11-12 11:22 ` Max Chernoff
2025-11-12 12:53 ` Stefano Brivio
2025-11-13 0:30 ` Max Chernoff
[not found] ` <176301983731.2033508.12381101277059600955@maja>
2025-11-17 5:13 ` David Gibson
2025-11-23 9:12 ` Max Chernoff
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for IMAP folder(s).