From: Max Chernoff <git@maxchernoff.ca>
To: David Gibson <david@gibson.dropbear.id.au>
Cc: Stefano Brivio <sbrivio@redhat.com>, passt-user@passt.top
Subject: Re: pasta slow at HTTP upload
Date: Sun, 23 Nov 2025 02:12:29 -0700 [thread overview]
Message-ID: <9aa57ffd40d34f81a80e649826828d431b31469c.camel@maxchernoff.ca> (raw)
In-Reply-To: <aRqu9o1FoIUWmDHD@zatzit>
Hi David,
On Mon, 2025-11-17 at 16:13 +1100, David Gibson wrote:
> Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat
> is the biggest culprit... except for the outlier where it seems ok
> with both custom notsent_lowat and custom congestion control.
>
> How repeatable are each of these results?
Very repeatable. I reran the tests from a different host (my laptop over
home Wifi) targeting different servers (two servers that I control) and
I got similar results.
> Actually, is it just pasta that's changed in that time? Or could it
> be a new kernel version or something else?
I'm always just using the most recent packages in Fedora, so both pasta
and the kernel would have upgraded in that time. Specifically:
- Around June 6th, I didn't notice any issues; I was using
"kernel-6.14.9-300.fc42.x86_64" and
"passt-0^20250512.g8ec1341-1.fc42.x86_64".
- Around August 20th, I started noticing intermittent issues; I was
using "kernel-6.15.9-201.fc42.x86_64" and
"passt-0^20250805.g309eefd-2.fc42.x86_64".
- Around September 26th, the problem became persistent; I was using
"kernel-6.16.8-200.fc42.x86_64" and
"passt-0^20250919.g623dbf6-1.fc42.x86_64".
- I ran the tests in this email with "kernel-6.17.8-300.fc43.x86_64" and
"passt-0^20250919.g623dbf6-1.fc43.x86_64".
> > I also find it quite interesting that setting any of the sysctls
> > individually or in pairs improves the upload speeds, but setting all 3
> > at once slows it down drastically.
>
> I'm curious about the case with default congestion control, but custom
> lowat and mem_max. I think that one's missing from the table above.
Ok, I ran some more tests. The problematic combination seems to be
network=pasta
direction=upload
rtt/ping=large
net.core.wmem_max=7500000
net.ipv4.tcp_notsent_lowat=131072
net.ipv4.tcp_congestion_control=bbr
and all other parameters seem to not make much difference. The data is
somewhat noisy, but still fairly consistent. I ran the tests for all
combinations for 10MB twice and 100MB once. The full data is below:
network ping size wmem rmem lowat cngst qdisc down up
pasta 170ms 100MB # . # # # 17.69 180.00
pasta 170ms 100MB # # # # . 15.41 180.00
pasta 170ms 100MB . # # . . 15.23 180.00
pasta 170ms 100MB # . # # . 11.86 180.00
pasta 170ms 100MB # # # # # 10.90 180.00
pasta 170ms 100MB # . # . . 23.91 128.18
pasta 170ms 10MB . . . # # 6.98 120.00
pasta 170ms 10MB # # # # # 6.26 120.00
pasta 170ms 10MB # . # # # 6.19 120.00
pasta 170ms 10MB # . # # # 6.10 120.00
pasta 170ms 10MB # . # # . 6.05 120.00
pasta 170ms 10MB # . # # . 5.93 120.00
pasta 170ms 10MB # # # # . 5.57 120.00
pasta 170ms 10MB . # . # . 4.45 120.00
pasta 170ms 100MB . # # . # 17.46 118.70
pasta 50ms 100MB # . # # # 6.57 117.59
pasta 170ms 100MB . . # . # 12.91 94.23
pasta 170ms 100MB . . # . . 12.46 89.22
pasta 170ms 100MB . # # # # 15.05 85.79
pasta 170ms 100MB . . . # # 12.92 82.36
pasta 170ms 100MB . . # # . 15.03 76.69
pasta 170ms 100MB . # . # . 18.91 74.79
pasta 170ms 100MB . . # # # 21.75 66.75
pasta 170ms 100MB # # # . . 23.02 66.36
pasta 50ms 100MB . . # # # 7.57 58.58
pasta 50ms 100MB . . # # . 8.46 58.24
pasta 170ms 100MB # # # . # 12.15 55.27
pasta 170ms 100MB . # # # . 17.17 54.86
pasta 50ms 100MB . # # . . 6.80 51.66
pasta 170ms 100MB # . # . # 11.89 51.40
pasta 170ms 10MB . . . # . 7.25 51.27
pasta 170ms 10MB . . # # # 6.98 49.34
pasta 50ms 100MB . # # # # 6.79 47.94
pasta 170ms 100MB . . . # . 11.98 45.94
pasta 50ms 100MB . . # . . 8.62 45.85
pasta 50ms 100MB # . # # . 7.05 45.24
pasta 50ms 100MB . . # . # 8.25 44.98
pasta 170ms 10MB # # # # # 6.91 44.95
pasta 170ms 10MB # # . . . 6.16 44.43
pasta 50ms 100MB . # # . # 8.47 44.18
pasta 50ms 100MB # . # . . 6.97 43.38
pasta 170ms 10MB # . # . . 5.27 42.54
pasta 170ms 10MB . # # . . 4.92 42.46
pasta 170ms 10MB # # # . # 6.15 42.36
pasta 170ms 10MB . # # . # 7.06 41.61
pasta 50ms 100MB . # # # . 6.66 40.83
pasta 170ms 100MB . # . # # 14.50 40.45
pasta 170ms 10MB # # # . # 7.10 39.84
pasta 170ms 10MB . # # . # 5.76 38.34
pasta 50ms 100MB # # # . # 6.81 37.96
pasta 170ms 10MB # # . # . 8.50 37.88
pasta 170ms 10MB . # # # . 5.47 37.85
pasta 170ms 10MB # . # . # 3.82 37.46
pasta 170ms 10MB # # # . . 6.79 37.19
pasta 170ms 10MB . # # # . 5.34 37.03
host 170ms 100MB # . # # # 15.09 36.65
pasta 170ms 10MB # # # . . 7.19 36.25
pasta 50ms 100MB # . # . # 7.63 35.72
pasta 170ms 10MB # . # . # 6.53 35.69
pasta 170ms 10MB # # # # . 8.35 35.69
pasta 170ms 100MB # . . # # 13.78 35.65
pasta 170ms 100MB # # . # # 16.32 35.20
host 170ms 100MB . . # # # 13.61 33.89
host 170ms 100MB . . . # . 14.76 33.87
host 170ms 100MB # . . # # 16.60 33.79
host 170ms 100MB # . # # . 12.71 33.41
pasta 170ms 100MB # # . # . 15.82 33.28
pasta 170ms 100MB . # . . # 13.55 33.25
host 170ms 100MB # # . # . 13.97 33.19
host 170ms 100MB . . # # . 17.49 33.15
host 170ms 100MB . # # # . 14.68 32.89
host 170ms 100MB . # # # # 17.12 32.71
host 170ms 100MB # . . # . 22.21 32.71
host 170ms 100MB # # # # . 12.18 32.44
pasta 170ms 100MB # . . # . 16.30 32.23
host 170ms 100MB # # . # # 15.46 32.08
host 170ms 100MB . # . # # 14.47 31.97
host 170ms 100MB . . . # # 16.80 31.84
host 170ms 100MB . # . # . 13.04 31.77
pasta 170ms 100MB . . . . # 15.85 31.74
host 170ms 100MB # # # # # 19.09 31.57
pasta 170ms 10MB . . # # . 5.62 30.59
pasta 170ms 10MB . # # # # 5.58 30.46
pasta 170ms 100MB . . . . . 11.22 30.22
pasta 170ms 100MB # . . . # 18.42 30.17
pasta 170ms 100MB . # . . . 14.93 30.09
pasta 170ms 100MB # # . . # 14.24 29.86
pasta 50ms 100MB # # # . . 6.57 29.81
host 170ms 100MB . # . . . 15.52 29.10
pasta 170ms 10MB . . # # . 5.11 28.77
host 170ms 100MB . # . . # 15.73 28.63
host 170ms 100MB # # # . . 16.57 28.54
host 170ms 100MB . # # . # 15.03 28.41
host 170ms 100MB . # # . . 14.29 28.19
host 170ms 100MB . . . . . 15.26 28.04
pasta 170ms 100MB # # . . . 14.35 27.76
pasta 170ms 10MB . . # . # 5.60 27.74
pasta 50ms 10MB # . # # . 1.37 27.68
pasta 170ms 10MB . # . # # 4.47 27.56
host 170ms 100MB . . . . # 16.84 27.43
host 170ms 100MB # # # . # 14.88 27.27
host 170ms 100MB . . # . # 17.53 26.75
host 170ms 100MB # . # . # 15.83 26.69
pasta 170ms 10MB . # . # # 5.29 26.13
host 170ms 100MB # # . . # 16.73 26.04
host 170ms 100MB # # . . . 19.30 25.79
pasta 170ms 100MB # . . . . 14.31 25.78
host 170ms 100MB # . # . . 20.52 25.61
pasta 170ms 10MB # . # . . 6.37 24.39
pasta 170ms 10MB . . # . . 7.07 24.33
host 170ms 100MB # . . . # 17.69 24.23
host 170ms 100MB # . . . . 19.94 23.75
pasta 50ms 100MB # # # # # 6.37 23.43
pasta 50ms 10MB # # # # . 1.52 22.99
pasta 50ms 100MB # # # # . 6.31 22.65
pasta 170ms 10MB . . # . # 6.52 22.01
pasta 50ms 10MB # . # # . 1.35 21.59
pasta 170ms 10MB . . . # . 7.15 21.35
pasta 170ms 10MB . . # . . 6.40 20.86
pasta 50ms 100MB . . . # . 8.43 20.49
pasta 50ms 100MB # # . # . 6.50 19.70
host 170ms 10MB # . . # # 4.71 19.61
pasta 50ms 100MB . # . # # 8.03 19.47
pasta 50ms 100MB . . . # # 8.40 19.29
pasta 170ms 10MB # # . # # 7.63 18.92
pasta 50ms 100MB . # . # . 7.09 18.88
host 50ms 100MB . . # # . 7.94 18.85
host 50ms 100MB . # . # . 7.15 18.74
host 170ms 100MB . . # . . 13.75 18.71
host 170ms 10MB . . . # . 4.52 18.57
host 170ms 10MB . . . # . 6.24 18.52
pasta 170ms 10MB . . # # # 7.04 18.48
host 50ms 100MB # . . # . 6.38 18.45
host 50ms 100MB . # . # # 7.37 17.89
pasta 170ms 10MB . # . # . 7.35 17.88
host 170ms 10MB # # . # . 6.11 17.76
host 50ms 100MB . . # . . 8.20 17.56
host 170ms 10MB . . # # . 6.56 17.51
pasta 170ms 10MB # . . # # 6.39 17.51
host 170ms 10MB . # # # # 4.86 17.46
host 170ms 10MB # # . # # 6.91 17.45
host 50ms 100MB # # # # # 7.11 17.45
host 170ms 10MB . # . # . 4.93 17.45
host 170ms 10MB . # # # . 5.14 17.44
pasta 170ms 10MB . # # # # 5.69 17.41
host 170ms 10MB # . # # # 3.64 17.39
pasta 170ms 10MB # # . # # 6.59 17.33
host 170ms 10MB . # . # # 4.83 17.33
pasta 170ms 10MB # . . # . 5.46 17.30
pasta 50ms 100MB # . . # # 6.23 17.30
pasta 170ms 10MB # . . # . 4.30 17.25
host 170ms 10MB . . # # # 5.64 17.25
pasta 50ms 100MB # . . # . 6.76 17.20
pasta 170ms 10MB . . . # # 7.77 17.19
host 50ms 100MB . . . # # 7.61 17.18
host 170ms 10MB . . . # # 6.01 17.16
host 50ms 100MB . # # # # 6.44 17.04
host 170ms 10MB # # # # . 5.08 17.04
host 170ms 10MB . # # # . 4.62 17.02
host 50ms 100MB # . # # # 6.51 17.02
host 50ms 100MB # . . # # 6.20 16.99
host 50ms 100MB # . # # . 6.43 16.98
pasta 170ms 10MB # # . # . 5.87 16.88
pasta 50ms 100MB # # . # # 6.36 16.86
host 170ms 10MB . . . # # 5.03 16.85
host 170ms 10MB # # # # # 3.44 16.83
host 170ms 10MB # # . # # 6.12 16.83
host 170ms 10MB # . # # . 5.62 16.83
pasta 170ms 10MB # . . # # 14.82 16.83
host 170ms 10MB # . . # . 5.14 16.82
host 170ms 10MB # . # # . 6.47 16.81
host 170ms 10MB . # . # # 5.13 16.79
host 170ms 10MB # . . # . 5.72 16.78
host 170ms 10MB # # # # # 5.49 16.78
host 170ms 10MB # # # # . 4.72 16.77
host 170ms 10MB # # . # . 5.79 16.76
host 50ms 100MB . . # # # 7.41 16.71
host 170ms 10MB . . # # # 6.10 16.70
host 50ms 100MB . # . . # 6.69 16.68
host 170ms 10MB . # # # # 6.06 16.67
host 170ms 10MB # . . # # 6.01 16.65
host 170ms 10MB . # . # . 5.32 16.58
host 170ms 10MB . . # # . 4.90 16.55
host 50ms 100MB # # . # # 6.47 16.40
pasta 170ms 10MB . # # . . 4.81 16.19
host 170ms 10MB # . # # # 5.77 16.06
host 50ms 100MB . # # # . 6.67 15.97
pasta 170ms 10MB # # . . # 6.78 15.23
host 50ms 100MB # # # # . 6.56 14.97
pasta 50ms 100MB . . . . . 8.19 14.95
pasta 170ms 10MB . . . . . 6.10 14.69
pasta 170ms 10MB # . . . . 5.78 14.51
pasta 170ms 10MB . # . . # 6.42 14.22
pasta 170ms 10MB . # . . . 6.30 14.20
host 50ms 100MB # # . # . 6.12 14.08
pasta 170ms 10MB . . . . # 6.63 14.07
pasta 170ms 10MB . # . . # 4.93 13.53
pasta 50ms 10MB # # # # # 1.29 13.41
pasta 170ms 10MB # . . . # 6.31 13.11
host 50ms 100MB . # # . # 7.62 13.07
pasta 170ms 10MB # # . . . 6.41 13.03
host 50ms 10MB # # # # . 1.80 12.98
host 50ms 100MB . . . # . 7.67 12.88
pasta 170ms 10MB # . . . . 5.65 12.76
pasta 170ms 10MB . . . . # 5.88 12.59
pasta 170ms 10MB # # . . # 5.69 12.55
host 50ms 100MB # . # . . 6.67 12.52
pasta 50ms 10MB # . # # # 1.22 12.50
host 170ms 10MB # . # . . 5.41 12.41
pasta 50ms 100MB # # . . . 6.58 12.28
pasta 50ms 10MB # # # # # 2.12 12.20
pasta 170ms 10MB . # . . . 6.56 12.16
pasta 170ms 10MB . . . . . 5.38 12.11
host 170ms 10MB # # # . # 5.60 12.00
pasta 170ms 10MB # . . . # 10.77 11.98
pasta 50ms 10MB # # # # . 1.19 11.92
pasta 50ms 100MB # . . . . 6.63 11.81
host 170ms 10MB . . # . # 5.55 11.80
host 50ms 100MB . # . . . 6.64 11.71
host 170ms 10MB # # # . # 5.35 11.70
pasta 50ms 100MB . # . . # 7.40 11.69
host 170ms 10MB # . # . # 6.18 11.66
host 50ms 100MB # # # . . 6.40 11.60
pasta 50ms 10MB . # # # . 1.24 11.51
host 170ms 10MB . # . . # 5.54 11.40
pasta 50ms 10MB . . # # # 1.35 11.38
host 170ms 10MB . # . . . 6.33 11.23
host 170ms 10MB # . # . # 4.25 11.22
host 50ms 100MB # # . . . 6.90 11.19
host 170ms 10MB . # # . . 6.59 11.16
host 50ms 100MB . # # . . 7.29 11.09
host 50ms 100MB . . # . # 8.63 11.07
host 170ms 10MB . . . . # 6.00 10.99
pasta 50ms 100MB # . . . # 7.90 10.86
host 170ms 10MB . . . . . 6.66 10.86
pasta 50ms 10MB # . # . # 2.61 10.84
host 170ms 10MB # . # . . 5.41 10.83
host 170ms 10MB # . . . # 5.80 10.76
host 170ms 10MB . . # . . 5.38 10.68
pasta 50ms 100MB . # . . . 7.13 10.68
host 170ms 10MB # # . . . 8.34 10.68
pasta 50ms 10MB . # # # # 1.39 10.66
pasta 50ms 10MB . # # # . 1.39 10.64
host 170ms 10MB # . . . . 4.92 10.64
host 170ms 10MB # # . . # 5.70 10.60
pasta 50ms 100MB . . . . # 6.92 10.56
host 50ms 100MB . . . . # 6.86 10.56
pasta 50ms 100MB # # . . # 6.32 10.53
host 170ms 10MB . . # . # 6.53 10.52
host 50ms 100MB # . . . . 6.67 10.52
host 170ms 10MB . . . . # 5.04 10.48
host 50ms 100MB # # # . # 6.39 10.45
host 170ms 10MB . # . . # 5.09 10.36
host 170ms 10MB # # # . . 6.01 10.29
host 50ms 100MB # . # . # 8.37 10.19
host 170ms 10MB . # # . . 7.35 10.14
host 170ms 10MB . # . . . 5.03 10.13
host 170ms 10MB # # # . . 6.11 10.08
host 170ms 10MB . # # . # 6.63 10.04
host 50ms 100MB # . . . # 7.07 10.04
host 50ms 100MB . . . . . 7.68 10.01
host 170ms 10MB # # . . # 4.41 9.96
host 170ms 10MB . . . . . 6.56 9.92
host 170ms 10MB # . . . . 5.72 9.88
pasta 50ms 10MB # . # . # 1.64 9.82
host 50ms 100MB # # . . # 6.32 9.72
host 170ms 10MB # . . . # 6.53 9.60
pasta 50ms 10MB # # # . # 1.53 9.25
host 170ms 10MB # # . . . 5.07 9.16
pasta 50ms 10MB . . # # # 1.41 9.10
host 170ms 10MB . . # . . 6.06 8.95
host 170ms 10MB . # # . # 6.71 8.83
pasta 50ms 10MB . . # . . 1.42 8.81
pasta 50ms 10MB . . # # . 1.45 8.65
pasta 50ms 10MB . # # # # 1.28 8.35
pasta 50ms 10MB # . # . . 1.47 7.92
pasta 50ms 10MB . # # . # 1.51 7.78
pasta 50ms 10MB # # # . . 1.87 7.61
pasta 50ms 10MB . . # # . 1.53 7.32
pasta 50ms 10MB # . # # # 1.33 7.22
pasta 50ms 10MB # # # . # 1.31 7.15
pasta 50ms 10MB . . . . . 1.34 7.12
pasta 50ms 10MB # # # . . 1.42 6.95
pasta 50ms 10MB . # # . . 1.51 6.83
pasta 50ms 10MB . # . . # 1.79 6.76
host 50ms 10MB . . . # # 1.42 6.66
pasta 50ms 10MB # . # . . 1.60 6.26
pasta 50ms 10MB # . . # # 1.55 5.66
pasta 50ms 10MB . # # . . 1.55 5.62
pasta 50ms 10MB . # . . . 1.44 5.53
pasta 50ms 10MB # # . # # 1.30 5.52
pasta 50ms 10MB . . # . # 1.43 5.28
pasta 50ms 10MB . . . . # 1.35 5.20
pasta 50ms 10MB . . . . # 1.49 5.15
pasta 50ms 10MB . # # . # 1.47 5.13
pasta 50ms 10MB . . # . . 1.40 5.10
pasta 50ms 10MB . . . . . 1.35 5.10
pasta 50ms 10MB . . # . # 1.40 5.08
host 50ms 10MB . # . # # 1.64 4.99
pasta 50ms 10MB # . . . # 1.44 4.69
pasta 50ms 10MB . # . # # 2.60 4.67
pasta 50ms 10MB . # . . . 1.45 4.63
pasta 50ms 10MB . # . . # 1.34 4.63
host 50ms 10MB . . # # . 1.31 4.57
host 50ms 10MB # . . # # 1.37 4.51
host 50ms 10MB # # . # # 1.24 4.48
pasta 50ms 10MB # # . . # 1.55 4.44
host 50ms 10MB # # . . # 1.49 4.42
pasta 50ms 10MB # # . . . 1.22 4.35
host 50ms 10MB . # . # . 1.29 4.33
host 50ms 10MB . # # # . 1.36 4.27
host 50ms 10MB # # # # # 1.32 4.04
pasta 50ms 10MB . . . # # 1.31 3.89
pasta 50ms 10MB . # . # . 1.33 3.88
pasta 50ms 10MB . . . # # 1.50 3.85
host 50ms 10MB # . # # . 1.43 3.78
host 50ms 10MB . . . # . 1.28 3.75
host 50ms 10MB # . # . # 1.52 3.74
pasta 50ms 10MB # . . # . 1.38 3.73
host 50ms 10MB . # # # # 1.36 3.68
host 50ms 10MB # # . # . 1.21 3.68
host 50ms 10MB . # . # # 1.41 3.66
pasta 50ms 10MB # . . # . 1.43 3.66
host 50ms 10MB . . . # # 1.62 3.65
pasta 50ms 10MB # # . # . 1.51 3.64
host 50ms 10MB . . # # # 1.31 3.64
host 50ms 10MB . . # . . 1.48 3.61
pasta 50ms 10MB # . . . . 1.30 3.60
pasta 50ms 10MB # . . . # 1.49 3.56
host 50ms 10MB . . . . # 1.30 3.54
pasta 50ms 10MB # # . # # 1.33 3.53
host 50ms 10MB # # . # . 1.38 3.52
host 50ms 10MB # . # # # 1.18 3.51
host 50ms 10MB . . . # . 1.59 3.51
pasta 50ms 10MB . . . # . 1.58 3.49
host 50ms 10MB . . . . . 1.48 3.48
host 50ms 10MB . # . . # 1.57 3.46
pasta 50ms 10MB # . . . . 1.49 3.45
pasta 50ms 10MB . # . # # 1.46 3.44
host 50ms 10MB . # # . . 1.41 3.41
host 50ms 10MB # # # . . 1.34 3.40
host 50ms 10MB # # # . # 1.34 3.40
host 50ms 10MB # # # # . 1.29 3.39
host 50ms 10MB # . # . . 1.87 3.39
host 50ms 10MB # # # . # 1.38 3.38
host 50ms 10MB . . . . . 1.43 3.37
pasta 50ms 10MB # # . . . 1.29 3.36
host 50ms 10MB # . # . # 1.59 3.33
host 50ms 10MB . . # . # 1.46 3.32
host 50ms 10MB # . . . . 1.47 3.29
host 50ms 10MB . # . . . 1.49 3.28
host 50ms 10MB . # . . # 1.38 3.28
host 50ms 10MB . . # . . 1.39 3.28
host 50ms 10MB # . . # . 1.43 3.27
host 50ms 10MB # . . # . 1.35 3.27
pasta 50ms 10MB # . . # # 1.39 3.26
host 50ms 10MB # # . . . 1.28 3.24
host 50ms 10MB . # # . # 1.43 3.24
host 50ms 10MB # . . . # 1.57 3.19
host 50ms 10MB . # # # . 1.27 3.19
host 50ms 10MB . # # # # 1.54 3.18
host 50ms 10MB . . # # . 1.34 3.14
host 50ms 10MB # . # # . 1.32 3.14
pasta 50ms 10MB . . . # . 1.58 3.12
host 50ms 10MB . # # . . 1.59 3.09
host 50ms 10MB . # . # . 1.43 3.08
pasta 50ms 10MB # # . . # 1.30 3.07
pasta 50ms 10MB # # . # . 1.35 3.01
host 50ms 10MB # # # # # 1.53 2.99
host 50ms 10MB . # # . # 1.54 2.94
host 50ms 10MB # . # . . 1.58 2.94
host 50ms 10MB . # . . . 1.43 2.87
host 50ms 10MB . . # . # 1.46 2.87
host 50ms 10MB # . # # # 1.36 2.87
host 50ms 10MB # # # . . 1.40 2.84
host 50ms 10MB # # . . # 1.29 2.79
host 50ms 10MB # . . . . 1.32 2.78
host 50ms 10MB # . . . # 1.37 2.68
host 50ms 10MB # # . . . 1.21 2.67
host 50ms 10MB . . . . # 1.37 2.66
host 50ms 10MB # # . # # 1.34 2.39
host 50ms 10MB # . . # # 1.43 2.15
pasta 50ms 10MB . # . # . 1.45 2.04
host 50ms 10MB . . # # # 1.54 1.93
column description
network podman --network= argument
ping the RTT to the remote server
size the size of the file {up,down}loaded
wmem sysctl
rmem sysctl
lowat sysctl
cngst sysctl
qdisc sysctl
down download time in seconds
up upload time in seconds (timed out at 120s for the 10MB tests and
180s for the 100MB tests )
column sysctl "#" (custom) "." (default)
wmem net.core.wmem_max 7500000 212992
rmem net.core.rmem_max 7500000 212992
lowat net.ipv4.tcp_notsent_lowat 131072 4294967295
cngst net.ipv4.tcp_congestion_control bbr cubic
qdisc net.core.default_qdisc cake fq_codel
And this is the messy script that I used to generate this data (I'm
hosting both servers myself, so feel free to use them for testing):
#!/usr/bin/bash
set -euo pipefail
# The location of the files
script_dir="$(dirname "$(realpath "$0")")"
output_file="$script_dir/pasta-upload-test-results.tsv"
# The size of the file to upload/download
file_size="100M"
# Define the sysctl parameters
declare -a -r sysctl_keys=(
"custom"
"default"
)
declare -A -r wmem_maxes=(
[default]=212992
[custom]=7500000
)
declare -A -r rmem_maxes=(
[default]=212992
[custom]=7500000
)
declare -A -r tcp_notsent_lowats=(
[default]=4294967295
[custom]=131072
)
declare -A -r tcp_congestion_controls=(
[default]="cubic"
[custom]="bbr"
)
declare -A -r default_qdiscs=(
[default]="fq_codel"
[custom]="cake"
)
# Define the network parameters
declare -a -r podman_networks=("host" "pasta")
declare -A -r servers=(
[50ms]="https://www.maxchernoff.ca/tools/speedtest"
[170ms]="https://svn.tug.org:8369/speedtest"
)
# Initialize the temporary directory
temp_dir=$(mktemp -d)
cd "$temp_dir"
trap 'rm -rf "$temp_dir"' EXIT
# Define the network test commands
download() {
local network=$1
local server=$2
podman run \
--quiet \
--rm \
--pull="never" \
--volume="$temp_dir:/srv/:Z" \
--workdir="/srv/" \
--network="$network" \
quay.io/fedora/fedora-minimal \
timeout --signal=SIGINT 180s \
curl \
--silent \
--output /dev/null \
--write-out '%{time_total}\n' \
"$server?bytes=$file_size" \
|| echo "180.000000"
}
upload() {
local network=$1
local server=$2
podman run \
--quiet \
--rm \
--pull="never" \
--volume="$temp_dir:/srv/:Z" \
--workdir="/srv/" \
--network="$network" \
quay.io/fedora/fedora-minimal \
timeout --signal=SIGINT 180s \
curl \
--silent \
--output /dev/null \
--write-out '%{time_total}\n' \
-X POST \
--upload-file test.bin \
"$server" \
|| echo "180.000000"
}
# Pull the container
podman pull --quiet quay.io/fedora/fedora-minimal >/dev/null 2>&1
# Create a sample file to upload
head --bytes="$file_size" /dev/urandom > test.bin
# Run the tests
printf 'network\tping_time\twmem_max\trmem_max\ttcp_notsent_lowat\ttcp_congestion_control\tdefault_qdisc\tdownload_time\tupload_time\n' \
| tee "$output_file"
for wmem_max in "${sysctl_keys[@]}"; do
for rmem_max in "${sysctl_keys[@]}"; do
for tcp_notsent_lowat in "${sysctl_keys[@]}"; do
for tcp_congestion_control in "${sysctl_keys[@]}"; do
for default_qdisc in "${sysctl_keys[@]}"; do
sudo sysctl -w \
net.core.wmem_max="${wmem_maxes[$wmem_max]}" \
net.core.rmem_max="${rmem_maxes[$rmem_max]}" \
net.ipv4.tcp_notsent_lowat="${tcp_notsent_lowats[$tcp_notsent_lowat]}" \
net.ipv4.tcp_congestion_control="${tcp_congestion_controls[$tcp_congestion_control]}" \
net.core.default_qdisc="${default_qdiscs[$default_qdisc]}" \
> /dev/null
for server in "${!servers[@]}"; do
for network in "${podman_networks[@]}"; do
printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n' \
"$network" \
"$server" \
"$wmem_max" \
"$rmem_max" \
"$tcp_notsent_lowat" \
"$tcp_congestion_control" \
"$default_qdisc" \
"$(download "$network" "${servers[$server]}")" \
"$(upload "$network" "${servers[$server]}")" \
| tee -a "$output_file"
done
done
done
done
done
done
done
# Sort the lines
cat "$output_file" | \
sort -k9rn -k8rn | \
column --table --table-right=2,8,9 --output-width=999 | \
tac | awk 'NR==1 {line=$0; next} 1; END{print line}' | tac | \
sponge "$output_file"
> > I bisected a kernel a few weeks ago,
> > so I can try that here if you think that this is a kernel bug and not a
> > pasta bug.
>
> Could be either, I have no real intuition either way so far.
Ok, I'll try testing with some older versions of both and see if that
narrows down the issue.
Thanks,
-- Max
next prev parent reply other threads:[~2025-11-23 9:12 UTC|newest]
Thread overview: 8+ messages / expand[flat|nested] mbox.gz Atom feed top
[not found] <176293029592.2033508.497353982367240204@maja>
2025-11-12 6:55 ` Stefano Brivio
2025-11-12 10:32 ` Stefano Brivio
2025-11-12 11:22 ` Max Chernoff
2025-11-12 12:53 ` Stefano Brivio
2025-11-13 0:30 ` Max Chernoff
[not found] ` <176301983731.2033508.12381101277059600955@maja>
2025-11-17 5:13 ` David Gibson
2025-11-23 9:12 ` Max Chernoff [this message]
2025-11-12 6:11 Max Chernoff
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=9aa57ffd40d34f81a80e649826828d431b31469c.camel@maxchernoff.ca \
--to=git@maxchernoff.ca \
--cc=david@gibson.dropbear.id.au \
--cc=passt-user@passt.top \
--cc=sbrivio@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).