* pasta slow at HTTP upload
@ 2025-11-12 6:11 Max Chernoff
0 siblings, 0 replies; 8+ messages in thread
From: Max Chernoff @ 2025-11-12 6:11 UTC (permalink / raw)
To: passt-user
Hi,
For the past few months, I've noticed that my HTTP uploads from
containers are really slow. Reproduction:
$ cd "$(mktemp -d)"
$ head --bytes=100M /dev/urandom > test.bin
$ tar cf ./test.tar.gz ./test.bin
$ chmod -R a+rx .
$ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 23.9M
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
6 100M 0 0 6 6912k 0 9517 3:03:39 0:12:23 2:51:16 0
(Stopped after 10 minutes since I got sick of waiting)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 64 18.7M 0:00:05 0:00:05 --:--:-- 23.8M
Some comments about the above:
- "www.ctan.org" is a web server running Apache; it isn't my server, but
it's where I first noticed this issue, and I have permission from its
owners to use it for testing.
- The upload speed is constant on the host and with --network=host, but
with --network=pasta, it starts high-ish, but quickly and continuously
decreases. But the "bytes transferred" number keeps increasing, so the
upload hasn't stalled out completely.
- I've also had problems with uploads to GitHub from a custom Go
binary---it takes ~3 minutes to upload ~600MB, split across 5 files,
which is much slower than I would expect, but still faster than the
curl/CTAN upload speeds. So I don't think that this is a bug with curl
or this specific web server.
- The host that I ran this from is a VM in a datacenter, supposedly with
a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of
2ms, and pinging www.ctan.org shows an RTT of 100ms.
- This problem seems to have started a few months ago. I don't have an
exact date (sorry), but it seems to have been fine before then.
- The host is running Fedora 43 IoT on x86_64, and I'm using the
following software versions:
$ rpm --query --queryformat='%{NAME}\t%{VERSION}\n' passt podman containers-common netavark | sort | column --table
containers-common 0.64.2
netavark 1.16.1
passt 0^20250919.g623dbf6
podman 5.6.2
I had the same issue when the host/containers were using Fedora 42.
- The host has fairly little CPU/network load when I ran these tests,
and pasta's CPU usage was low enough that it didn't show up on the
first page of htop.
Does anybody have any advice on troubleshooting this? I'm willing to
test with other servers/commands, compile/test any patches, or provide
any other information that might be helpful.
Thanks,
-- Max
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
2025-11-17 5:13 ` David Gibson
@ 2025-11-23 9:12 ` Max Chernoff
0 siblings, 0 replies; 8+ messages in thread
From: Max Chernoff @ 2025-11-23 9:12 UTC (permalink / raw)
To: David Gibson; +Cc: Stefano Brivio, passt-user
Hi David,
On Mon, 2025-11-17 at 16:13 +1100, David Gibson wrote:
> Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat
> is the biggest culprit... except for the outlier where it seems ok
> with both custom notsent_lowat and custom congestion control.
>
> How repeatable are each of these results?
Very repeatable. I reran the tests from a different host (my laptop over
home Wifi) targeting different servers (two servers that I control) and
I got similar results.
> Actually, is it just pasta that's changed in that time? Or could it
> be a new kernel version or something else?
I'm always just using the most recent packages in Fedora, so both pasta
and the kernel would have upgraded in that time. Specifically:
- Around June 6th, I didn't notice any issues; I was using
"kernel-6.14.9-300.fc42.x86_64" and
"passt-0^20250512.g8ec1341-1.fc42.x86_64".
- Around August 20th, I started noticing intermittent issues; I was
using "kernel-6.15.9-201.fc42.x86_64" and
"passt-0^20250805.g309eefd-2.fc42.x86_64".
- Around September 26th, the problem became persistent; I was using
"kernel-6.16.8-200.fc42.x86_64" and
"passt-0^20250919.g623dbf6-1.fc42.x86_64".
- I ran the tests in this email with "kernel-6.17.8-300.fc43.x86_64" and
"passt-0^20250919.g623dbf6-1.fc43.x86_64".
> > I also find it quite interesting that setting any of the sysctls
> > individually or in pairs improves the upload speeds, but setting all 3
> > at once slows it down drastically.
>
> I'm curious about the case with default congestion control, but custom
> lowat and mem_max. I think that one's missing from the table above.
Ok, I ran some more tests. The problematic combination seems to be
network=pasta
direction=upload
rtt/ping=large
net.core.wmem_max=7500000
net.ipv4.tcp_notsent_lowat=131072
net.ipv4.tcp_congestion_control=bbr
and all other parameters seem to not make much difference. The data is
somewhat noisy, but still fairly consistent. I ran the tests for all
combinations for 10MB twice and 100MB once. The full data is below:
network ping size wmem rmem lowat cngst qdisc down up
pasta 170ms 100MB # . # # # 17.69 180.00
pasta 170ms 100MB # # # # . 15.41 180.00
pasta 170ms 100MB . # # . . 15.23 180.00
pasta 170ms 100MB # . # # . 11.86 180.00
pasta 170ms 100MB # # # # # 10.90 180.00
pasta 170ms 100MB # . # . . 23.91 128.18
pasta 170ms 10MB . . . # # 6.98 120.00
pasta 170ms 10MB # # # # # 6.26 120.00
pasta 170ms 10MB # . # # # 6.19 120.00
pasta 170ms 10MB # . # # # 6.10 120.00
pasta 170ms 10MB # . # # . 6.05 120.00
pasta 170ms 10MB # . # # . 5.93 120.00
pasta 170ms 10MB # # # # . 5.57 120.00
pasta 170ms 10MB . # . # . 4.45 120.00
pasta 170ms 100MB . # # . # 17.46 118.70
pasta 50ms 100MB # . # # # 6.57 117.59
pasta 170ms 100MB . . # . # 12.91 94.23
pasta 170ms 100MB . . # . . 12.46 89.22
pasta 170ms 100MB . # # # # 15.05 85.79
pasta 170ms 100MB . . . # # 12.92 82.36
pasta 170ms 100MB . . # # . 15.03 76.69
pasta 170ms 100MB . # . # . 18.91 74.79
pasta 170ms 100MB . . # # # 21.75 66.75
pasta 170ms 100MB # # # . . 23.02 66.36
pasta 50ms 100MB . . # # # 7.57 58.58
pasta 50ms 100MB . . # # . 8.46 58.24
pasta 170ms 100MB # # # . # 12.15 55.27
pasta 170ms 100MB . # # # . 17.17 54.86
pasta 50ms 100MB . # # . . 6.80 51.66
pasta 170ms 100MB # . # . # 11.89 51.40
pasta 170ms 10MB . . . # . 7.25 51.27
pasta 170ms 10MB . . # # # 6.98 49.34
pasta 50ms 100MB . # # # # 6.79 47.94
pasta 170ms 100MB . . . # . 11.98 45.94
pasta 50ms 100MB . . # . . 8.62 45.85
pasta 50ms 100MB # . # # . 7.05 45.24
pasta 50ms 100MB . . # . # 8.25 44.98
pasta 170ms 10MB # # # # # 6.91 44.95
pasta 170ms 10MB # # . . . 6.16 44.43
pasta 50ms 100MB . # # . # 8.47 44.18
pasta 50ms 100MB # . # . . 6.97 43.38
pasta 170ms 10MB # . # . . 5.27 42.54
pasta 170ms 10MB . # # . . 4.92 42.46
pasta 170ms 10MB # # # . # 6.15 42.36
pasta 170ms 10MB . # # . # 7.06 41.61
pasta 50ms 100MB . # # # . 6.66 40.83
pasta 170ms 100MB . # . # # 14.50 40.45
pasta 170ms 10MB # # # . # 7.10 39.84
pasta 170ms 10MB . # # . # 5.76 38.34
pasta 50ms 100MB # # # . # 6.81 37.96
pasta 170ms 10MB # # . # . 8.50 37.88
pasta 170ms 10MB . # # # . 5.47 37.85
pasta 170ms 10MB # . # . # 3.82 37.46
pasta 170ms 10MB # # # . . 6.79 37.19
pasta 170ms 10MB . # # # . 5.34 37.03
host 170ms 100MB # . # # # 15.09 36.65
pasta 170ms 10MB # # # . . 7.19 36.25
pasta 50ms 100MB # . # . # 7.63 35.72
pasta 170ms 10MB # . # . # 6.53 35.69
pasta 170ms 10MB # # # # . 8.35 35.69
pasta 170ms 100MB # . . # # 13.78 35.65
pasta 170ms 100MB # # . # # 16.32 35.20
host 170ms 100MB . . # # # 13.61 33.89
host 170ms 100MB . . . # . 14.76 33.87
host 170ms 100MB # . . # # 16.60 33.79
host 170ms 100MB # . # # . 12.71 33.41
pasta 170ms 100MB # # . # . 15.82 33.28
pasta 170ms 100MB . # . . # 13.55 33.25
host 170ms 100MB # # . # . 13.97 33.19
host 170ms 100MB . . # # . 17.49 33.15
host 170ms 100MB . # # # . 14.68 32.89
host 170ms 100MB . # # # # 17.12 32.71
host 170ms 100MB # . . # . 22.21 32.71
host 170ms 100MB # # # # . 12.18 32.44
pasta 170ms 100MB # . . # . 16.30 32.23
host 170ms 100MB # # . # # 15.46 32.08
host 170ms 100MB . # . # # 14.47 31.97
host 170ms 100MB . . . # # 16.80 31.84
host 170ms 100MB . # . # . 13.04 31.77
pasta 170ms 100MB . . . . # 15.85 31.74
host 170ms 100MB # # # # # 19.09 31.57
pasta 170ms 10MB . . # # . 5.62 30.59
pasta 170ms 10MB . # # # # 5.58 30.46
pasta 170ms 100MB . . . . . 11.22 30.22
pasta 170ms 100MB # . . . # 18.42 30.17
pasta 170ms 100MB . # . . . 14.93 30.09
pasta 170ms 100MB # # . . # 14.24 29.86
pasta 50ms 100MB # # # . . 6.57 29.81
host 170ms 100MB . # . . . 15.52 29.10
pasta 170ms 10MB . . # # . 5.11 28.77
host 170ms 100MB . # . . # 15.73 28.63
host 170ms 100MB # # # . . 16.57 28.54
host 170ms 100MB . # # . # 15.03 28.41
host 170ms 100MB . # # . . 14.29 28.19
host 170ms 100MB . . . . . 15.26 28.04
pasta 170ms 100MB # # . . . 14.35 27.76
pasta 170ms 10MB . . # . # 5.60 27.74
pasta 50ms 10MB # . # # . 1.37 27.68
pasta 170ms 10MB . # . # # 4.47 27.56
host 170ms 100MB . . . . # 16.84 27.43
host 170ms 100MB # # # . # 14.88 27.27
host 170ms 100MB . . # . # 17.53 26.75
host 170ms 100MB # . # . # 15.83 26.69
pasta 170ms 10MB . # . # # 5.29 26.13
host 170ms 100MB # # . . # 16.73 26.04
host 170ms 100MB # # . . . 19.30 25.79
pasta 170ms 100MB # . . . . 14.31 25.78
host 170ms 100MB # . # . . 20.52 25.61
pasta 170ms 10MB # . # . . 6.37 24.39
pasta 170ms 10MB . . # . . 7.07 24.33
host 170ms 100MB # . . . # 17.69 24.23
host 170ms 100MB # . . . . 19.94 23.75
pasta 50ms 100MB # # # # # 6.37 23.43
pasta 50ms 10MB # # # # . 1.52 22.99
pasta 50ms 100MB # # # # . 6.31 22.65
pasta 170ms 10MB . . # . # 6.52 22.01
pasta 50ms 10MB # . # # . 1.35 21.59
pasta 170ms 10MB . . . # . 7.15 21.35
pasta 170ms 10MB . . # . . 6.40 20.86
pasta 50ms 100MB . . . # . 8.43 20.49
pasta 50ms 100MB # # . # . 6.50 19.70
host 170ms 10MB # . . # # 4.71 19.61
pasta 50ms 100MB . # . # # 8.03 19.47
pasta 50ms 100MB . . . # # 8.40 19.29
pasta 170ms 10MB # # . # # 7.63 18.92
pasta 50ms 100MB . # . # . 7.09 18.88
host 50ms 100MB . . # # . 7.94 18.85
host 50ms 100MB . # . # . 7.15 18.74
host 170ms 100MB . . # . . 13.75 18.71
host 170ms 10MB . . . # . 4.52 18.57
host 170ms 10MB . . . # . 6.24 18.52
pasta 170ms 10MB . . # # # 7.04 18.48
host 50ms 100MB # . . # . 6.38 18.45
host 50ms 100MB . # . # # 7.37 17.89
pasta 170ms 10MB . # . # . 7.35 17.88
host 170ms 10MB # # . # . 6.11 17.76
host 50ms 100MB . . # . . 8.20 17.56
host 170ms 10MB . . # # . 6.56 17.51
pasta 170ms 10MB # . . # # 6.39 17.51
host 170ms 10MB . # # # # 4.86 17.46
host 170ms 10MB # # . # # 6.91 17.45
host 50ms 100MB # # # # # 7.11 17.45
host 170ms 10MB . # . # . 4.93 17.45
host 170ms 10MB . # # # . 5.14 17.44
pasta 170ms 10MB . # # # # 5.69 17.41
host 170ms 10MB # . # # # 3.64 17.39
pasta 170ms 10MB # # . # # 6.59 17.33
host 170ms 10MB . # . # # 4.83 17.33
pasta 170ms 10MB # . . # . 5.46 17.30
pasta 50ms 100MB # . . # # 6.23 17.30
pasta 170ms 10MB # . . # . 4.30 17.25
host 170ms 10MB . . # # # 5.64 17.25
pasta 50ms 100MB # . . # . 6.76 17.20
pasta 170ms 10MB . . . # # 7.77 17.19
host 50ms 100MB . . . # # 7.61 17.18
host 170ms 10MB . . . # # 6.01 17.16
host 50ms 100MB . # # # # 6.44 17.04
host 170ms 10MB # # # # . 5.08 17.04
host 170ms 10MB . # # # . 4.62 17.02
host 50ms 100MB # . # # # 6.51 17.02
host 50ms 100MB # . . # # 6.20 16.99
host 50ms 100MB # . # # . 6.43 16.98
pasta 170ms 10MB # # . # . 5.87 16.88
pasta 50ms 100MB # # . # # 6.36 16.86
host 170ms 10MB . . . # # 5.03 16.85
host 170ms 10MB # # # # # 3.44 16.83
host 170ms 10MB # # . # # 6.12 16.83
host 170ms 10MB # . # # . 5.62 16.83
pasta 170ms 10MB # . . # # 14.82 16.83
host 170ms 10MB # . . # . 5.14 16.82
host 170ms 10MB # . # # . 6.47 16.81
host 170ms 10MB . # . # # 5.13 16.79
host 170ms 10MB # . . # . 5.72 16.78
host 170ms 10MB # # # # # 5.49 16.78
host 170ms 10MB # # # # . 4.72 16.77
host 170ms 10MB # # . # . 5.79 16.76
host 50ms 100MB . . # # # 7.41 16.71
host 170ms 10MB . . # # # 6.10 16.70
host 50ms 100MB . # . . # 6.69 16.68
host 170ms 10MB . # # # # 6.06 16.67
host 170ms 10MB # . . # # 6.01 16.65
host 170ms 10MB . # . # . 5.32 16.58
host 170ms 10MB . . # # . 4.90 16.55
host 50ms 100MB # # . # # 6.47 16.40
pasta 170ms 10MB . # # . . 4.81 16.19
host 170ms 10MB # . # # # 5.77 16.06
host 50ms 100MB . # # # . 6.67 15.97
pasta 170ms 10MB # # . . # 6.78 15.23
host 50ms 100MB # # # # . 6.56 14.97
pasta 50ms 100MB . . . . . 8.19 14.95
pasta 170ms 10MB . . . . . 6.10 14.69
pasta 170ms 10MB # . . . . 5.78 14.51
pasta 170ms 10MB . # . . # 6.42 14.22
pasta 170ms 10MB . # . . . 6.30 14.20
host 50ms 100MB # # . # . 6.12 14.08
pasta 170ms 10MB . . . . # 6.63 14.07
pasta 170ms 10MB . # . . # 4.93 13.53
pasta 50ms 10MB # # # # # 1.29 13.41
pasta 170ms 10MB # . . . # 6.31 13.11
host 50ms 100MB . # # . # 7.62 13.07
pasta 170ms 10MB # # . . . 6.41 13.03
host 50ms 10MB # # # # . 1.80 12.98
host 50ms 100MB . . . # . 7.67 12.88
pasta 170ms 10MB # . . . . 5.65 12.76
pasta 170ms 10MB . . . . # 5.88 12.59
pasta 170ms 10MB # # . . # 5.69 12.55
host 50ms 100MB # . # . . 6.67 12.52
pasta 50ms 10MB # . # # # 1.22 12.50
host 170ms 10MB # . # . . 5.41 12.41
pasta 50ms 100MB # # . . . 6.58 12.28
pasta 50ms 10MB # # # # # 2.12 12.20
pasta 170ms 10MB . # . . . 6.56 12.16
pasta 170ms 10MB . . . . . 5.38 12.11
host 170ms 10MB # # # . # 5.60 12.00
pasta 170ms 10MB # . . . # 10.77 11.98
pasta 50ms 10MB # # # # . 1.19 11.92
pasta 50ms 100MB # . . . . 6.63 11.81
host 170ms 10MB . . # . # 5.55 11.80
host 50ms 100MB . # . . . 6.64 11.71
host 170ms 10MB # # # . # 5.35 11.70
pasta 50ms 100MB . # . . # 7.40 11.69
host 170ms 10MB # . # . # 6.18 11.66
host 50ms 100MB # # # . . 6.40 11.60
pasta 50ms 10MB . # # # . 1.24 11.51
host 170ms 10MB . # . . # 5.54 11.40
pasta 50ms 10MB . . # # # 1.35 11.38
host 170ms 10MB . # . . . 6.33 11.23
host 170ms 10MB # . # . # 4.25 11.22
host 50ms 100MB # # . . . 6.90 11.19
host 170ms 10MB . # # . . 6.59 11.16
host 50ms 100MB . # # . . 7.29 11.09
host 50ms 100MB . . # . # 8.63 11.07
host 170ms 10MB . . . . # 6.00 10.99
pasta 50ms 100MB # . . . # 7.90 10.86
host 170ms 10MB . . . . . 6.66 10.86
pasta 50ms 10MB # . # . # 2.61 10.84
host 170ms 10MB # . # . . 5.41 10.83
host 170ms 10MB # . . . # 5.80 10.76
host 170ms 10MB . . # . . 5.38 10.68
pasta 50ms 100MB . # . . . 7.13 10.68
host 170ms 10MB # # . . . 8.34 10.68
pasta 50ms 10MB . # # # # 1.39 10.66
pasta 50ms 10MB . # # # . 1.39 10.64
host 170ms 10MB # . . . . 4.92 10.64
host 170ms 10MB # # . . # 5.70 10.60
pasta 50ms 100MB . . . . # 6.92 10.56
host 50ms 100MB . . . . # 6.86 10.56
pasta 50ms 100MB # # . . # 6.32 10.53
host 170ms 10MB . . # . # 6.53 10.52
host 50ms 100MB # . . . . 6.67 10.52
host 170ms 10MB . . . . # 5.04 10.48
host 50ms 100MB # # # . # 6.39 10.45
host 170ms 10MB . # . . # 5.09 10.36
host 170ms 10MB # # # . . 6.01 10.29
host 50ms 100MB # . # . # 8.37 10.19
host 170ms 10MB . # # . . 7.35 10.14
host 170ms 10MB . # . . . 5.03 10.13
host 170ms 10MB # # # . . 6.11 10.08
host 170ms 10MB . # # . # 6.63 10.04
host 50ms 100MB # . . . # 7.07 10.04
host 50ms 100MB . . . . . 7.68 10.01
host 170ms 10MB # # . . # 4.41 9.96
host 170ms 10MB . . . . . 6.56 9.92
host 170ms 10MB # . . . . 5.72 9.88
pasta 50ms 10MB # . # . # 1.64 9.82
host 50ms 100MB # # . . # 6.32 9.72
host 170ms 10MB # . . . # 6.53 9.60
pasta 50ms 10MB # # # . # 1.53 9.25
host 170ms 10MB # # . . . 5.07 9.16
pasta 50ms 10MB . . # # # 1.41 9.10
host 170ms 10MB . . # . . 6.06 8.95
host 170ms 10MB . # # . # 6.71 8.83
pasta 50ms 10MB . . # . . 1.42 8.81
pasta 50ms 10MB . . # # . 1.45 8.65
pasta 50ms 10MB . # # # # 1.28 8.35
pasta 50ms 10MB # . # . . 1.47 7.92
pasta 50ms 10MB . # # . # 1.51 7.78
pasta 50ms 10MB # # # . . 1.87 7.61
pasta 50ms 10MB . . # # . 1.53 7.32
pasta 50ms 10MB # . # # # 1.33 7.22
pasta 50ms 10MB # # # . # 1.31 7.15
pasta 50ms 10MB . . . . . 1.34 7.12
pasta 50ms 10MB # # # . . 1.42 6.95
pasta 50ms 10MB . # # . . 1.51 6.83
pasta 50ms 10MB . # . . # 1.79 6.76
host 50ms 10MB . . . # # 1.42 6.66
pasta 50ms 10MB # . # . . 1.60 6.26
pasta 50ms 10MB # . . # # 1.55 5.66
pasta 50ms 10MB . # # . . 1.55 5.62
pasta 50ms 10MB . # . . . 1.44 5.53
pasta 50ms 10MB # # . # # 1.30 5.52
pasta 50ms 10MB . . # . # 1.43 5.28
pasta 50ms 10MB . . . . # 1.35 5.20
pasta 50ms 10MB . . . . # 1.49 5.15
pasta 50ms 10MB . # # . # 1.47 5.13
pasta 50ms 10MB . . # . . 1.40 5.10
pasta 50ms 10MB . . . . . 1.35 5.10
pasta 50ms 10MB . . # . # 1.40 5.08
host 50ms 10MB . # . # # 1.64 4.99
pasta 50ms 10MB # . . . # 1.44 4.69
pasta 50ms 10MB . # . # # 2.60 4.67
pasta 50ms 10MB . # . . . 1.45 4.63
pasta 50ms 10MB . # . . # 1.34 4.63
host 50ms 10MB . . # # . 1.31 4.57
host 50ms 10MB # . . # # 1.37 4.51
host 50ms 10MB # # . # # 1.24 4.48
pasta 50ms 10MB # # . . # 1.55 4.44
host 50ms 10MB # # . . # 1.49 4.42
pasta 50ms 10MB # # . . . 1.22 4.35
host 50ms 10MB . # . # . 1.29 4.33
host 50ms 10MB . # # # . 1.36 4.27
host 50ms 10MB # # # # # 1.32 4.04
pasta 50ms 10MB . . . # # 1.31 3.89
pasta 50ms 10MB . # . # . 1.33 3.88
pasta 50ms 10MB . . . # # 1.50 3.85
host 50ms 10MB # . # # . 1.43 3.78
host 50ms 10MB . . . # . 1.28 3.75
host 50ms 10MB # . # . # 1.52 3.74
pasta 50ms 10MB # . . # . 1.38 3.73
host 50ms 10MB . # # # # 1.36 3.68
host 50ms 10MB # # . # . 1.21 3.68
host 50ms 10MB . # . # # 1.41 3.66
pasta 50ms 10MB # . . # . 1.43 3.66
host 50ms 10MB . . . # # 1.62 3.65
pasta 50ms 10MB # # . # . 1.51 3.64
host 50ms 10MB . . # # # 1.31 3.64
host 50ms 10MB . . # . . 1.48 3.61
pasta 50ms 10MB # . . . . 1.30 3.60
pasta 50ms 10MB # . . . # 1.49 3.56
host 50ms 10MB . . . . # 1.30 3.54
pasta 50ms 10MB # # . # # 1.33 3.53
host 50ms 10MB # # . # . 1.38 3.52
host 50ms 10MB # . # # # 1.18 3.51
host 50ms 10MB . . . # . 1.59 3.51
pasta 50ms 10MB . . . # . 1.58 3.49
host 50ms 10MB . . . . . 1.48 3.48
host 50ms 10MB . # . . # 1.57 3.46
pasta 50ms 10MB # . . . . 1.49 3.45
pasta 50ms 10MB . # . # # 1.46 3.44
host 50ms 10MB . # # . . 1.41 3.41
host 50ms 10MB # # # . . 1.34 3.40
host 50ms 10MB # # # . # 1.34 3.40
host 50ms 10MB # # # # . 1.29 3.39
host 50ms 10MB # . # . . 1.87 3.39
host 50ms 10MB # # # . # 1.38 3.38
host 50ms 10MB . . . . . 1.43 3.37
pasta 50ms 10MB # # . . . 1.29 3.36
host 50ms 10MB # . # . # 1.59 3.33
host 50ms 10MB . . # . # 1.46 3.32
host 50ms 10MB # . . . . 1.47 3.29
host 50ms 10MB . # . . . 1.49 3.28
host 50ms 10MB . # . . # 1.38 3.28
host 50ms 10MB . . # . . 1.39 3.28
host 50ms 10MB # . . # . 1.43 3.27
host 50ms 10MB # . . # . 1.35 3.27
pasta 50ms 10MB # . . # # 1.39 3.26
host 50ms 10MB # # . . . 1.28 3.24
host 50ms 10MB . # # . # 1.43 3.24
host 50ms 10MB # . . . # 1.57 3.19
host 50ms 10MB . # # # . 1.27 3.19
host 50ms 10MB . # # # # 1.54 3.18
host 50ms 10MB . . # # . 1.34 3.14
host 50ms 10MB # . # # . 1.32 3.14
pasta 50ms 10MB . . . # . 1.58 3.12
host 50ms 10MB . # # . . 1.59 3.09
host 50ms 10MB . # . # . 1.43 3.08
pasta 50ms 10MB # # . . # 1.30 3.07
pasta 50ms 10MB # # . # . 1.35 3.01
host 50ms 10MB # # # # # 1.53 2.99
host 50ms 10MB . # # . # 1.54 2.94
host 50ms 10MB # . # . . 1.58 2.94
host 50ms 10MB . # . . . 1.43 2.87
host 50ms 10MB . . # . # 1.46 2.87
host 50ms 10MB # . # # # 1.36 2.87
host 50ms 10MB # # # . . 1.40 2.84
host 50ms 10MB # # . . # 1.29 2.79
host 50ms 10MB # . . . . 1.32 2.78
host 50ms 10MB # . . . # 1.37 2.68
host 50ms 10MB # # . . . 1.21 2.67
host 50ms 10MB . . . . # 1.37 2.66
host 50ms 10MB # # . # # 1.34 2.39
host 50ms 10MB # . . # # 1.43 2.15
pasta 50ms 10MB . # . # . 1.45 2.04
host 50ms 10MB . . # # # 1.54 1.93
column description
network podman --network= argument
ping the RTT to the remote server
size the size of the file {up,down}loaded
wmem sysctl
rmem sysctl
lowat sysctl
cngst sysctl
qdisc sysctl
down download time in seconds
up upload time in seconds (timed out at 120s for the 10MB tests and
180s for the 100MB tests )
column sysctl "#" (custom) "." (default)
wmem net.core.wmem_max 7500000 212992
rmem net.core.rmem_max 7500000 212992
lowat net.ipv4.tcp_notsent_lowat 131072 4294967295
cngst net.ipv4.tcp_congestion_control bbr cubic
qdisc net.core.default_qdisc cake fq_codel
And this is the messy script that I used to generate this data (I'm
hosting both servers myself, so feel free to use them for testing):
#!/usr/bin/bash
set -euo pipefail
# The location of the files
script_dir="$(dirname "$(realpath "$0")")"
output_file="$script_dir/pasta-upload-test-results.tsv"
# The size of the file to upload/download
file_size="100M"
# Define the sysctl parameters
declare -a -r sysctl_keys=(
"custom"
"default"
)
declare -A -r wmem_maxes=(
[default]=212992
[custom]=7500000
)
declare -A -r rmem_maxes=(
[default]=212992
[custom]=7500000
)
declare -A -r tcp_notsent_lowats=(
[default]=4294967295
[custom]=131072
)
declare -A -r tcp_congestion_controls=(
[default]="cubic"
[custom]="bbr"
)
declare -A -r default_qdiscs=(
[default]="fq_codel"
[custom]="cake"
)
# Define the network parameters
declare -a -r podman_networks=("host" "pasta")
declare -A -r servers=(
[50ms]="https://www.maxchernoff.ca/tools/speedtest"
[170ms]="https://svn.tug.org:8369/speedtest"
)
# Initialize the temporary directory
temp_dir=$(mktemp -d)
cd "$temp_dir"
trap 'rm -rf "$temp_dir"' EXIT
# Define the network test commands
download() {
local network=$1
local server=$2
podman run \
--quiet \
--rm \
--pull="never" \
--volume="$temp_dir:/srv/:Z" \
--workdir="/srv/" \
--network="$network" \
quay.io/fedora/fedora-minimal \
timeout --signal=SIGINT 180s \
curl \
--silent \
--output /dev/null \
--write-out '%{time_total}\n' \
"$server?bytes=$file_size" \
|| echo "180.000000"
}
upload() {
local network=$1
local server=$2
podman run \
--quiet \
--rm \
--pull="never" \
--volume="$temp_dir:/srv/:Z" \
--workdir="/srv/" \
--network="$network" \
quay.io/fedora/fedora-minimal \
timeout --signal=SIGINT 180s \
curl \
--silent \
--output /dev/null \
--write-out '%{time_total}\n' \
-X POST \
--upload-file test.bin \
"$server" \
|| echo "180.000000"
}
# Pull the container
podman pull --quiet quay.io/fedora/fedora-minimal >/dev/null 2>&1
# Create a sample file to upload
head --bytes="$file_size" /dev/urandom > test.bin
# Run the tests
printf 'network\tping_time\twmem_max\trmem_max\ttcp_notsent_lowat\ttcp_congestion_control\tdefault_qdisc\tdownload_time\tupload_time\n' \
| tee "$output_file"
for wmem_max in "${sysctl_keys[@]}"; do
for rmem_max in "${sysctl_keys[@]}"; do
for tcp_notsent_lowat in "${sysctl_keys[@]}"; do
for tcp_congestion_control in "${sysctl_keys[@]}"; do
for default_qdisc in "${sysctl_keys[@]}"; do
sudo sysctl -w \
net.core.wmem_max="${wmem_maxes[$wmem_max]}" \
net.core.rmem_max="${rmem_maxes[$rmem_max]}" \
net.ipv4.tcp_notsent_lowat="${tcp_notsent_lowats[$tcp_notsent_lowat]}" \
net.ipv4.tcp_congestion_control="${tcp_congestion_controls[$tcp_congestion_control]}" \
net.core.default_qdisc="${default_qdiscs[$default_qdisc]}" \
> /dev/null
for server in "${!servers[@]}"; do
for network in "${podman_networks[@]}"; do
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$network" \
"$server" \
"$wmem_max" \
"$rmem_max" \
"$tcp_notsent_lowat" \
"$tcp_congestion_control" \
"$default_qdisc" \
"$(download "$network" "${servers[$server]}")" \
"$(upload "$network" "${servers[$server]}")" \
| tee -a "$output_file"
done
done
done
done
done
done
done
# Sort the lines
cat "$output_file" | \
sort -k9rn -k8rn | \
column --table --table-right=2,8,9 --output-width=999 | \
tac | awk 'NR==1 {line=$0; next} 1; END{print line}' | tac | \
sponge "$output_file"
> > I bisected a kernel a few weeks ago,
> > so I can try that here if you think that this is a kernel bug and not a
> > pasta bug.
>
> Could be either, I have no real intuition either way so far.
Ok, I'll try testing with some older versions of both and see if that
narrows down the issue.
Thanks,
-- Max
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
[not found] ` <176301983731.2033508.12381101277059600955@maja>
@ 2025-11-17 5:13 ` David Gibson
2025-11-23 9:12 ` Max Chernoff
0 siblings, 1 reply; 8+ messages in thread
From: David Gibson @ 2025-11-17 5:13 UTC (permalink / raw)
To: Max Chernoff; +Cc: Stefano Brivio, passt-user
[-- Attachment #1: Type: text/plain, Size: 13366 bytes --]
On Wed, Nov 12, 2025 at 05:30:35PM -0700, Max Chernoff via user wrote:
> Date: Wed, 12 Nov 2025 17:30:35 -0700
> From: Max Chernoff <git@maxchernoff.ca>
> To: Stefano Brivio <sbrivio@redhat.com>
> CC: passt-user@passt.top
> Subject: Re: pasta slow at HTTP upload
> List-Id: "For passt users: support, questions and answers"
> <passt-user.passt.top>
>
> Hi Stefano,
>
> On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote:
> > As a further hack, you could probably do something like this on top:
> >
> > ---
> > diff --git a/tcp.c b/tcp.c
> > index 697f80d..8c50ee0 100644
> > --- a/tcp.c
> > +++ b/tcp.c
> > @@ -339,7 +339,7 @@ enum {
> > #define MSS_DEFAULT 536
> > #define WINDOW_DEFAULT 14600 /*
> > RFC 6928 */
> >
> > -#define ACK_INTERVAL 1 /* ms */
> > +#define ACK_INTERVAL 200 /* us */
> > #define SYN_TIMEOUT 10 /* s */
> > #define ACK_TIMEOUT 2
> > #define FIN_TIMEOUT 60
> > @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn)
> > }
> >
> > if (conn->flags & ACK_TO_TAP_DUE) {
> > - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000;
> > + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000;
> > } else if (conn->flags & ACK_FROM_TAP_DUE) {
> > if (!(conn->events & ESTABLISHED))
> > it.it_value.tv_sec = SYN_TIMEOUT;
> > ---
>
> That actually makes it worse again, about as bad as before the patch.
> But I've just tried rebuilding with original patch again, and also with
> the exact same binary that I used yesterday, and that's slow now too.
> I've verified with pgrep that Podman is using the correct pasta version,
> so I have no idea what's happening.
>
> However, I do remember that for the past few months, some uploads would
> randomly go really quickly, so maybe the problem happens sporadically,
> and when I was testing the patched version I just happened to get
> (un)lucky?
>
> > > net.core.wmem_max=7500000
> > > net.core.rmem_max=7500000
> >
> > Those were settings we recommended for KubeVirt until
> > https://github.com/kubevirt/user-guide/pull/933, but they don't seem to
> > necessarily make sense as we seem to have made peace with the TCP
> > auto-tuning mechanism in Linux meanwhile.
> >
> > See also https://bugs.passt.top/show_bug.cgi?id=138 and commit
> > 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to
> > maximum values").
> >
> > As the issue here is about socket (kernel) buffers being "too small" for
> > a while, I guess that those settings plus reverting that commit would
> > "fix" the issue entirely for you. But it's impractical to rely on users
> > to set those, that's why I'm looking for something adaptive which still
> > plays nicely with TCP auto-tuning instead.
>
> Ah, I didn't know that those (used to be) recommended for pasta; I set
> those for Caddy since it complains on startup if those aren't set
>
> https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
>
> > > net.ipv4.tcp_notsent_lowat=131072
> > > net.core.default_qdisc=cake
> > > net.ipv4.tcp_congestion_control=bbr
>
> I set those as a general performance tuning thing (not for pasta
> specifically) based off of
>
> https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/
>
> https://grapheneos.org/articles/server-traffic-shaping
>
> > I'm not sure if those really matter for pasta, but I haven't really
> > thought about them.
>
> Aha, those do seem to be the issue. Using the original (unpatched)
> pasta:
Huh, interesting.
> (Set everything to my original settings)
> $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr
>
> (Test with --network=host)
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M
>
> (Test with --network=pasta)
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188
> (Stopped early since I got sick of waiting)
>
> (Set everything to the kernel defaults)
> $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k
>
> (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k
>
> (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M
>
> (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M
>
> (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k
>
> (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M
>
> (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom)
> $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0
> (Stopped early since I got sick of waiting)
>
> (Test with --network=host again)
> $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
> % Total % Received % Xferd Average Speed Time Time Time Current
> Dload Upload Total Spent Left Speed
> 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M
Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat
is the biggest culprit... except for the outlier where it seems ok
with both custom notsent_lowat and custom congestion control.
How repeatable are each of these results?
> I know fairly little about networking and the kernel, so if the answer
> is just "don't set those sysctls together", I'd be okay with that. But I
> haven't changed these sysctls since February, my upload speeds via pasta
> were fine up until a few months ago, and the upload speeds are still
> okay with --network=host, so I suspect that this is a bug somewhere.
I have a vague theory why tcp_notsent_lowat might be bad for pasta but
not --network=host. I'm far from certain about it though, and I don't
have a good idea what change on the pasta side might have changed.
Actually, is it just pasta that's changed in that time? Or could it
be a new kernel version or something else?
I'm having some trouble wrapping my head around what tcp_notsent_lowat
does, but I _think_ the upshot of it will be to make the socket send
path more "bursty": we'll fill the socket send buffer (which could be
very large), then stop. We won't wake up again to refill until the
amount in the socket buffer drops below tcp_notsend_lowat. Assuming
we've been getting plenty of data from the other side, we'll then
refill the buffer very quickly.
Less wakups sounds like a good thing in the abstract, but I'm
wondering if this burstiness is messing with the flow control on the
other side, causing the guest side peer to slow down to nothing. If
that is the case, it would make sense that it gets even worse with
large socket buffer sizes, since that would make the "bursts" even
bigger.
Likewise I was wondering if having different congestion control
algorithms on each side might somehow conflict causing a slowdown. I
don't think the data you have above really suggest that though.
> I also find it quite interesting that setting any of the sysctls
> individually or in pairs improves the upload speeds, but setting all 3
> at once slows it down drastically.
I'm curious about the case with default congestion control, but custom
lowat and mem_max. I think that one's missing from the table above.
> I bisected a kernel a few weeks ago,
> so I can try that here if you think that this is a kernel bug and not a
> pasta bug.
Could be either, I have no real intuition either way so far.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
2025-11-12 12:53 ` Stefano Brivio
@ 2025-11-13 0:30 ` Max Chernoff
[not found] ` <176301983731.2033508.12381101277059600955@maja>
1 sibling, 0 replies; 8+ messages in thread
From: Max Chernoff @ 2025-11-13 0:30 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-user
Hi Stefano,
On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote:
> As a further hack, you could probably do something like this on top:
>
> ---
> diff --git a/tcp.c b/tcp.c
> index 697f80d..8c50ee0 100644
> --- a/tcp.c
> +++ b/tcp.c
> @@ -339,7 +339,7 @@ enum {
> #define MSS_DEFAULT 536
> #define WINDOW_DEFAULT 14600 /*
> RFC 6928 */
>
> -#define ACK_INTERVAL 1 /* ms */
> +#define ACK_INTERVAL 200 /* us */
> #define SYN_TIMEOUT 10 /* s */
> #define ACK_TIMEOUT 2
> #define FIN_TIMEOUT 60
> @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn)
> }
>
> if (conn->flags & ACK_TO_TAP_DUE) {
> - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000;
> + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000;
> } else if (conn->flags & ACK_FROM_TAP_DUE) {
> if (!(conn->events & ESTABLISHED))
> it.it_value.tv_sec = SYN_TIMEOUT;
> ---
That actually makes it worse again, about as bad as before the patch.
But I've just tried rebuilding with original patch again, and also with
the exact same binary that I used yesterday, and that's slow now too.
I've verified with pgrep that Podman is using the correct pasta version,
so I have no idea what's happening.
However, I do remember that for the past few months, some uploads would
randomly go really quickly, so maybe the problem happens sporadically,
and when I was testing the patched version I just happened to get
(un)lucky?
> > net.core.wmem_max=7500000
> > net.core.rmem_max=7500000
>
> Those were settings we recommended for KubeVirt until
> https://github.com/kubevirt/user-guide/pull/933, but they don't seem to
> necessarily make sense as we seem to have made peace with the TCP
> auto-tuning mechanism in Linux meanwhile.
>
> See also https://bugs.passt.top/show_bug.cgi?id=138 and commit
> 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to
> maximum values").
>
> As the issue here is about socket (kernel) buffers being "too small" for
> a while, I guess that those settings plus reverting that commit would
> "fix" the issue entirely for you. But it's impractical to rely on users
> to set those, that's why I'm looking for something adaptive which still
> plays nicely with TCP auto-tuning instead.
Ah, I didn't know that those (used to be) recommended for pasta; I set
those for Caddy since it complains on startup if those aren't set
https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
> > net.ipv4.tcp_notsent_lowat=131072
> > net.core.default_qdisc=cake
> > net.ipv4.tcp_congestion_control=bbr
I set those as a general performance tuning thing (not for pasta
specifically) based off of
https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/
https://grapheneos.org/articles/server-traffic-shaping
> I'm not sure if those really matter for pasta, but I haven't really
> thought about them.
Aha, those do seem to be the issue. Using the original (unpatched)
pasta:
(Set everything to my original settings)
$ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr
(Test with --network=host)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M
(Test with --network=pasta)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 100M 0 0 4 5120k 0 42440 0:41:10 0:02:03 0:39:07 26188
(Stopped early since I got sick of waiting)
(Set everything to the kernel defaults)
$ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 4 1431k 0:01:11 0:01:11 --:--:-- 1399k
(tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 5 1484k 0:01:08 0:01:08 --:--:-- 1400k
(tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 12 3576k 0:00:28 0:00:28 --:--:-- 14.9M
(tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 8 2595k 0:00:39 0:00:39 --:--:-- 17.1M
(tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 7 2107k 0:00:48 0:00:48 --:--:-- 8310k
(tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 15 4620k 0:00:22 0:00:22 --:--:-- 12.1M
(tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom)
$ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
5 100M 0 0 5 5376k 0 29899 0:58:27 0:03:04 0:55:23 0
(Stopped early since I got sick of waiting)
(Test with --network=host again)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 24.3M
I know fairly little about networking and the kernel, so if the answer
is just "don't set those sysctls together", I'd be okay with that. But I
haven't changed these sysctls since February, my upload speeds via pasta
were fine up until a few months ago, and the upload speeds are still
okay with --network=host, so I suspect that this is a bug somewhere.
I also find it quite interesting that setting any of the sysctls
individually or in pairs improves the upload speeds, but setting all 3
at once slows it down drastically. I bisected a kernel a few weeks ago,
so I can try that here if you think that this is a kernel bug and not a
pasta bug.
Thanks,
-- Max
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
2025-11-12 11:22 ` Max Chernoff
@ 2025-11-12 12:53 ` Stefano Brivio
2025-11-13 0:30 ` Max Chernoff
[not found] ` <176301983731.2033508.12381101277059600955@maja>
0 siblings, 2 replies; 8+ messages in thread
From: Stefano Brivio @ 2025-11-12 12:53 UTC (permalink / raw)
To: Max Chernoff; +Cc: passt-user
On Wed, 12 Nov 2025 04:22:32 -0700
Max Chernoff <git@maxchernoff.ca> wrote:
> Hi Stefano,
>
> On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote:
> > Hmm, actually, I have a hack that's not quite correct (we should make
> > ACK_INTERVAL adaptive instead, which is one of the other bits I'm
> > working on), but if it fixes the issue for you, it should at least mean
> > that we're talking about the same issue.
> >
> > Patch attached. Can you give that a try?
>
> That seemed to help quite a bit---it's now 200x faster than before, but
> still 10x slower than --network=host:
Thanks for testing. I'm fairly sure it's that problem, then.
Setting 1 ms as interval between checks for socket-side ACKs (reported
by the kernel) as my hack does is not the appropriate solution, I'm
implementing something based on the reported round-trip-time (RTT)
instead.
As a further hack, you could probably do something like this on top:
---
diff --git a/tcp.c b/tcp.c
index 697f80d..8c50ee0 100644
--- a/tcp.c
+++ b/tcp.c
@@ -339,7 +339,7 @@ enum {
#define MSS_DEFAULT 536
#define WINDOW_DEFAULT 14600 /*
RFC 6928 */
-#define ACK_INTERVAL 1 /* ms */
+#define ACK_INTERVAL 200 /* us */
#define SYN_TIMEOUT 10 /* s */
#define ACK_TIMEOUT 2
#define FIN_TIMEOUT 60
@@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn)
}
if (conn->flags & ACK_TO_TAP_DUE) {
- it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000;
+ it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000;
} else if (conn->flags & ACK_FROM_TAP_DUE) {
if (!(conn->events & ESTABLISHED))
it.it_value.tv_sec = SYN_TIMEOUT;
---
...but again, I'm going to fix that properly in a bit.
> Also, I should mention that I'm using the following networking-related
> sysctls:
>
> net.core.wmem_max=7500000
> net.core.rmem_max=7500000
Those were settings we recommended for KubeVirt until
https://github.com/kubevirt/user-guide/pull/933, but they don't seem to
necessarily make sense as we seem to have made peace with the TCP
auto-tuning mechanism in Linux meanwhile.
See also https://bugs.passt.top/show_bug.cgi?id=138 and commit
71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to
maximum values").
As the issue here is about socket (kernel) buffers being "too small" for
a while, I guess that those settings plus reverting that commit would
"fix" the issue entirely for you. But it's impractical to rely on users
to set those, that's why I'm looking for something adaptive which still
plays nicely with TCP auto-tuning instead.
> net.ipv4.tcp_notsent_lowat=131072
> net.core.default_qdisc=cake
> net.ipv4.tcp_congestion_control=bbr
I'm not sure if those really matter for pasta, but I haven't really
thought about them.
> I read some articles that suggested that those were a good idea, and
> I've been using them for about a year now, but I can disable those for
> testing if you want. I'm also using systemd's
> IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux
> port restrictions; I can easily disable those too.
No need, I don't think those make a difference.
--
Stefano
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
2025-11-12 10:32 ` Stefano Brivio
@ 2025-11-12 11:22 ` Max Chernoff
2025-11-12 12:53 ` Stefano Brivio
0 siblings, 1 reply; 8+ messages in thread
From: Max Chernoff @ 2025-11-12 11:22 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-user
Hi Stefano,
On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote:
> Hmm, actually, I have a hack that's not quite correct (we should make
> ACK_INTERVAL adaptive instead, which is one of the other bits I'm
> working on), but if it fixes the issue for you, it should at least mean
> that we're talking about the same issue.
>
> Patch attached. Can you give that a try?
That seemed to help quite a bit---it's now 200x faster than before, but
still 10x slower than --network=host:
$ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 63 18.4M 0:00:05 0:00:05 --:--:-- 20.3M
(With the original pasta, stopped early)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
4 100M 0 0 4 5056k 0 78152 0:22:21 0:01:06 0:21:15 39298⏎
(With the patch applied)
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 8 2393k 0:00:42 0:00:42 --:--:-- 4729k
$ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-- 20.4M
Also, I should mention that I'm using the following networking-related
sysctls:
net.core.wmem_max=7500000
net.core.rmem_max=7500000
net.ipv4.tcp_notsent_lowat=131072
net.core.default_qdisc=cake
net.ipv4.tcp_congestion_control=bbr
I read some articles that suggested that those were a good idea, and
I've been using them for about a year now, but I can disable those for
testing if you want. I'm also using systemd's
IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux
port restrictions; I can easily disable those too.
Thanks,
-- Max
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
2025-11-12 6:55 ` Stefano Brivio
@ 2025-11-12 10:32 ` Stefano Brivio
2025-11-12 11:22 ` Max Chernoff
0 siblings, 1 reply; 8+ messages in thread
From: Stefano Brivio @ 2025-11-12 10:32 UTC (permalink / raw)
To: Max Chernoff; +Cc: passt-user
[-- Attachment #1: Type: text/plain, Size: 1040 bytes --]
On Wed, 12 Nov 2025 07:55:48 +0100
Stefano Brivio <sbrivio@redhat.com> wrote:
> Hi Max,
>
> On Tue, 11 Nov 2025 23:11:17 -0700
> Max Chernoff <git@maxchernoff.ca> wrote:
>
> > Hi,
> >
> > For the past few months, I've noticed that my HTTP uploads from
> > containers are really slow. Reproduction:
> >
> > [...]
> >
> > - The host that I ran this from is a VM in a datacenter, supposedly with
> > a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of
> > 2ms, and pinging www.ctan.org shows an RTT of 100ms.
>
> Thanks for reporting this. Coincidentally, I'm currently debugging an
> issue that looks similar to this, and I should have some patches for
> you to test in a couple of days, I'll keep you posted.
Hmm, actually, I have a hack that's not quite correct (we should make
ACK_INTERVAL adaptive instead, which is one of the other bits I'm
working on), but if it fixes the issue for you, it should at least mean
that we're talking about the same issue.
Patch attached. Can you give that a try?
--
Stefano
[-- Attachment #2: tcp_account_for_sendq_in_window.patch --]
[-- Type: text/x-patch, Size: 1336 bytes --]
diff --git a/tcp.c b/tcp.c
index e91c0cf..697f80d 100644
--- a/tcp.c
+++ b/tcp.c
@@ -339,7 +339,7 @@ enum {
#define MSS_DEFAULT 536
#define WINDOW_DEFAULT 14600 /* RFC 6928 */
-#define ACK_INTERVAL 10 /* ms */
+#define ACK_INTERVAL 1 /* ms */
#define SYN_TIMEOUT 10 /* s */
#define ACK_TIMEOUT 2
#define FIN_TIMEOUT 60
@@ -1097,9 +1097,22 @@ int tcp_update_seqack_wnd(const struct ctx *c, struct tcp_tap_conn *conn,
if ((conn->flags & LOCAL) || tcp_rtt_dst_low(conn)) {
new_wnd_to_tap = tinfo->tcpi_snd_wnd;
} else {
+ uint32_t sendq;
+ int limit;
+
+ ioctl(s, SIOCOUTQ, &sendq);
tcp_get_sndbuf(conn);
+
+ if ((int)sendq > SNDBUF_GET(conn))
+ limit = 0;
+ else
+ limit = SNDBUF_GET(conn) - (int)sendq;
+
+ if (limit < MSS_GET(conn))
+ limit = 0;
+
new_wnd_to_tap = MIN((int)tinfo->tcpi_snd_wnd,
- SNDBUF_GET(conn));
+ limit);
}
new_wnd_to_tap = MIN(new_wnd_to_tap, MAX_WINDOW);
@@ -1240,7 +1253,8 @@ int tcp_prepare_flags(const struct ctx *c, struct tcp_tap_conn *conn,
th->fin = !!(flags & FIN);
if (th->ack) {
- if (SEQ_GE(conn->seq_ack_to_tap, conn->seq_from_tap))
+ if (SEQ_GE(conn->seq_ack_to_tap, conn->seq_from_tap) &&
+ conn->wnd_to_tap > MSS_GET(conn))
conn_flag(c, conn, ~ACK_TO_TAP_DUE);
else
conn_flag(c, conn, ACK_TO_TAP_DUE);
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: pasta slow at HTTP upload
[not found] <176293029592.2033508.497353982367240204@maja>
@ 2025-11-12 6:55 ` Stefano Brivio
2025-11-12 10:32 ` Stefano Brivio
0 siblings, 1 reply; 8+ messages in thread
From: Stefano Brivio @ 2025-11-12 6:55 UTC (permalink / raw)
To: Max Chernoff; +Cc: passt-user
Hi Max,
On Tue, 11 Nov 2025 23:11:17 -0700
Max Chernoff <git@maxchernoff.ca> wrote:
> Hi,
>
> For the past few months, I've noticed that my HTTP uploads from
> containers are really slow. Reproduction:
>
> [...]
>
> - The host that I ran this from is a VM in a datacenter, supposedly with
> a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of
> 2ms, and pinging www.ctan.org shows an RTT of 100ms.
Thanks for reporting this. Coincidentally, I'm currently debugging an
issue that looks similar to this, and I should have some patches for
you to test in a couple of days, I'll keep you posted.
--
Stefano
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-11-23 9:12 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-11-12 6:11 pasta slow at HTTP upload Max Chernoff
[not found] <176293029592.2033508.497353982367240204@maja>
2025-11-12 6:55 ` Stefano Brivio
2025-11-12 10:32 ` Stefano Brivio
2025-11-12 11:22 ` Max Chernoff
2025-11-12 12:53 ` Stefano Brivio
2025-11-13 0:30 ` Max Chernoff
[not found] ` <176301983731.2033508.12381101277059600955@maja>
2025-11-17 5:13 ` David Gibson
2025-11-23 9:12 ` Max Chernoff
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).