From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=maxchernoff.ca Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=maxchernoff.ca header.i=@maxchernoff.ca header.a=rsa-sha256 header.s=key1 header.b=0Wv6WyNG; dkim-atps=neutral Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [91.218.175.182]) by passt.top (Postfix) with ESMTPS id B24255A0271 for ; Sun, 23 Nov 2025 10:12:34 +0100 (CET) Message-ID: <9aa57ffd40d34f81a80e649826828d431b31469c.camel@maxchernoff.ca> DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maxchernoff.ca; s=key1; t=1763889153; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=4tAiI5GhqluMNB4zJgJi04MEmw9166aOsZRd8zZG/38=; b=0Wv6WyNG13MjVPNhhcLbCQaSW4cSRjAyflW3/2KVin80dHzbHBtCChPOC/Ob/QUnO6O7cs 79VBNjkK9nrY4hhHrjJ/3mUI2WO80r3vcPV70tV9omLAF/JCJxymLsEE/L+pCgbBWKHoJw cRiUuBmPe1lYh9xhS3B7Sqie/R8WK71vIlKPHEPWxOa+O4axHraDwrzTOuuWtniJiwhY1z xwsF0qXJHBnUFg0Qlyfz2tmN0B3IxJXOLRVMsKtPVRlz9mC0weQ5Udc82iYPq4l2hUoGKj 7Q269eRQKxVougsjjCq6E7F3Y4K25DO2FlID7Swg2d5a/JcUiaax05emvRfdKw== Subject: Re: pasta slow at HTTP upload X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Max Chernoff To: David Gibson Date: Sun, 23 Nov 2025 02:12:29 -0700 In-Reply-To: References: <176293029592.2033508.497353982367240204@maja> <20251112075548.0c05a25e@elisabeth> <20251112113201.3bcabc6c@elisabeth> <043088ef8bdc2d2c7a910617eb58d494cd9761e0.camel@maxchernoff.ca> <20251112135312.595c9a44@elisabeth> <176301983731.2033508.12381101277059600955@maja> Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT Message-ID-Hash: DQ63HMCE2PYZZZE3WLUKWZUF6D54EAH5 X-Message-ID-Hash: DQ63HMCE2PYZZZE3WLUKWZUF6D54EAH5 X-MailFrom: git@maxchernoff.ca X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Stefano Brivio , passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi David, On Mon, 2025-11-17 at 16:13 +1100, David Gibson wrote: > Well, that sure is a confusing pattern. Looks like tcp_notsent_lowat > is the biggest culprit... except for the outlier where it seems ok > with both custom notsent_lowat and custom congestion control. > > How repeatable are each of these results? Very repeatable. I reran the tests from a different host (my laptop over home Wifi) targeting different servers (two servers that I control) and I got similar results. > Actually, is it just pasta that's changed in that time? Or could it > be a new kernel version or something else? I'm always just using the most recent packages in Fedora, so both pasta and the kernel would have upgraded in that time. Specifically: - Around June 6th, I didn't notice any issues; I was using "kernel-6.14.9-300.fc42.x86_64" and "passt-0^20250512.g8ec1341-1.fc42.x86_64". - Around August 20th, I started noticing intermittent issues; I was using "kernel-6.15.9-201.fc42.x86_64" and "passt-0^20250805.g309eefd-2.fc42.x86_64". - Around September 26th, the problem became persistent; I was using "kernel-6.16.8-200.fc42.x86_64" and "passt-0^20250919.g623dbf6-1.fc42.x86_64". - I ran the tests in this email with "kernel-6.17.8-300.fc43.x86_64" and "passt-0^20250919.g623dbf6-1.fc43.x86_64". > > I also find it quite interesting that setting any of the sysctls > > individually or in pairs improves the upload speeds, but setting all 3 > > at once slows it down drastically. > > I'm curious about the case with default congestion control, but custom > lowat and mem_max. I think that one's missing from the table above. Ok, I ran some more tests. The problematic combination seems to be network=3Dpasta direction=3Dupload rtt/ping=3Dlarge net.core.wmem_max=3D7500000 net.ipv4.tcp_notsent_lowat=3D131072 net.ipv4.tcp_congestion_control=3Dbbr and all other parameters seem to not make much difference. The data is somewhat noisy, but still fairly consistent. I ran the tests for all combinations for 10MB twice and 100MB once. The full data is below: network ping size wmem rmem lowat cngst qdisc down up pasta 170ms 100MB # . # # # 17.69 180.00 pasta 170ms 100MB # # # # . 15.41 180.00 pasta 170ms 100MB . # # . . 15.23 180.00 pasta 170ms 100MB # . # # . 11.86 180.00 pasta 170ms 100MB # # # # # 10.90 180.00 pasta 170ms 100MB # . # . . 23.91 128.18 pasta 170ms 10MB . . . # # 6.98 120.00 pasta 170ms 10MB # # # # # 6.26 120.00 pasta 170ms 10MB # . # # # 6.19 120.00 pasta 170ms 10MB # . # # # 6.10 120.00 pasta 170ms 10MB # . # # . 6.05 120.00 pasta 170ms 10MB # . # # . 5.93 120.00 pasta 170ms 10MB # # # # . 5.57 120.00 pasta 170ms 10MB . # . # . 4.45 120.00 pasta 170ms 100MB . # # . # 17.46 118.70 pasta 50ms 100MB # . # # # 6.57 117.59 pasta 170ms 100MB . . # . # 12.91 94.23 pasta 170ms 100MB . . # . . 12.46 89.22 pasta 170ms 100MB . # # # # 15.05 85.79 pasta 170ms 100MB . . . # # 12.92 82.36 pasta 170ms 100MB . . # # . 15.03 76.69 pasta 170ms 100MB . # . # . 18.91 74.79 pasta 170ms 100MB . . # # # 21.75 66.75 pasta 170ms 100MB # # # . . 23.02 66.36 pasta 50ms 100MB . . # # # 7.57 58.58 pasta 50ms 100MB . . # # . 8.46 58.24 pasta 170ms 100MB # # # . # 12.15 55.27 pasta 170ms 100MB . # # # . 17.17 54.86 pasta 50ms 100MB . # # . . 6.80 51.66 pasta 170ms 100MB # . # . # 11.89 51.40 pasta 170ms 10MB . . . # . 7.25 51.27 pasta 170ms 10MB . . # # # 6.98 49.34 pasta 50ms 100MB . # # # # 6.79 47.94 pasta 170ms 100MB . . . # . 11.98 45.94 pasta 50ms 100MB . . # . . 8.62 45.85 pasta 50ms 100MB # . # # . 7.05 45.24 pasta 50ms 100MB . . # . # 8.25 44.98 pasta 170ms 10MB # # # # # 6.91 44.95 pasta 170ms 10MB # # . . . 6.16 44.43 pasta 50ms 100MB . # # . # 8.47 44.18 pasta 50ms 100MB # . # . . 6.97 43.38 pasta 170ms 10MB # . # . . 5.27 42.54 pasta 170ms 10MB . # # . . 4.92 42.46 pasta 170ms 10MB # # # . # 6.15 42.36 pasta 170ms 10MB . # # . # 7.06 41.61 pasta 50ms 100MB . # # # . 6.66 40.83 pasta 170ms 100MB . # . # # 14.50 40.45 pasta 170ms 10MB # # # . # 7.10 39.84 pasta 170ms 10MB . # # . # 5.76 38.34 pasta 50ms 100MB # # # . # 6.81 37.96 pasta 170ms 10MB # # . # . 8.50 37.88 pasta 170ms 10MB . # # # . 5.47 37.85 pasta 170ms 10MB # . # . # 3.82 37.46 pasta 170ms 10MB # # # . . 6.79 37.19 pasta 170ms 10MB . # # # . 5.34 37.03 host 170ms 100MB # . # # # 15.09 36.65 pasta 170ms 10MB # # # . . 7.19 36.25 pasta 50ms 100MB # . # . # 7.63 35.72 pasta 170ms 10MB # . # . # 6.53 35.69 pasta 170ms 10MB # # # # . 8.35 35.69 pasta 170ms 100MB # . . # # 13.78 35.65 pasta 170ms 100MB # # . # # 16.32 35.20 host 170ms 100MB . . # # # 13.61 33.89 host 170ms 100MB . . . # . 14.76 33.87 host 170ms 100MB # . . # # 16.60 33.79 host 170ms 100MB # . # # . 12.71 33.41 pasta 170ms 100MB # # . # . 15.82 33.28 pasta 170ms 100MB . # . . # 13.55 33.25 host 170ms 100MB # # . # . 13.97 33.19 host 170ms 100MB . . # # . 17.49 33.15 host 170ms 100MB . # # # . 14.68 32.89 host 170ms 100MB . # # # # 17.12 32.71 host 170ms 100MB # . . # . 22.21 32.71 host 170ms 100MB # # # # . 12.18 32.44 pasta 170ms 100MB # . . # . 16.30 32.23 host 170ms 100MB # # . # # 15.46 32.08 host 170ms 100MB . # . # # 14.47 31.97 host 170ms 100MB . . . # # 16.80 31.84 host 170ms 100MB . # . # . 13.04 31.77 pasta 170ms 100MB . . . . # 15.85 31.74 host 170ms 100MB # # # # # 19.09 31.57 pasta 170ms 10MB . . # # . 5.62 30.59 pasta 170ms 10MB . # # # # 5.58 30.46 pasta 170ms 100MB . . . . . 11.22 30.22 pasta 170ms 100MB # . . . # 18.42 30.17 pasta 170ms 100MB . # . . . 14.93 30.09 pasta 170ms 100MB # # . . # 14.24 29.86 pasta 50ms 100MB # # # . . 6.57 29.81 host 170ms 100MB . # . . . 15.52 29.10 pasta 170ms 10MB . . # # . 5.11 28.77 host 170ms 100MB . # . . # 15.73 28.63 host 170ms 100MB # # # . . 16.57 28.54 host 170ms 100MB . # # . # 15.03 28.41 host 170ms 100MB . # # . . 14.29 28.19 host 170ms 100MB . . . . . 15.26 28.04 pasta 170ms 100MB # # . . . 14.35 27.76 pasta 170ms 10MB . . # . # 5.60 27.74 pasta 50ms 10MB # . # # . 1.37 27.68 pasta 170ms 10MB . # . # # 4.47 27.56 host 170ms 100MB . . . . # 16.84 27.43 host 170ms 100MB # # # . # 14.88 27.27 host 170ms 100MB . . # . # 17.53 26.75 host 170ms 100MB # . # . # 15.83 26.69 pasta 170ms 10MB . # . # # 5.29 26.13 host 170ms 100MB # # . . # 16.73 26.04 host 170ms 100MB # # . . . 19.30 25.79 pasta 170ms 100MB # . . . . 14.31 25.78 host 170ms 100MB # . # . . 20.52 25.61 pasta 170ms 10MB # . # . . 6.37 24.39 pasta 170ms 10MB . . # . . 7.07 24.33 host 170ms 100MB # . . . # 17.69 24.23 host 170ms 100MB # . . . . 19.94 23.75 pasta 50ms 100MB # # # # # 6.37 23.43 pasta 50ms 10MB # # # # . 1.52 22.99 pasta 50ms 100MB # # # # . 6.31 22.65 pasta 170ms 10MB . . # . # 6.52 22.01 pasta 50ms 10MB # . # # . 1.35 21.59 pasta 170ms 10MB . . . # . 7.15 21.35 pasta 170ms 10MB . . # . . 6.40 20.86 pasta 50ms 100MB . . . # . 8.43 20.49 pasta 50ms 100MB # # . # . 6.50 19.70 host 170ms 10MB # . . # # 4.71 19.61 pasta 50ms 100MB . # . # # 8.03 19.47 pasta 50ms 100MB . . . # # 8.40 19.29 pasta 170ms 10MB # # . # # 7.63 18.92 pasta 50ms 100MB . # . # . 7.09 18.88 host 50ms 100MB . . # # . 7.94 18.85 host 50ms 100MB . # . # . 7.15 18.74 host 170ms 100MB . . # . . 13.75 18.71 host 170ms 10MB . . . # . 4.52 18.57 host 170ms 10MB . . . # . 6.24 18.52 pasta 170ms 10MB . . # # # 7.04 18.48 host 50ms 100MB # . . # . 6.38 18.45 host 50ms 100MB . # . # # 7.37 17.89 pasta 170ms 10MB . # . # . 7.35 17.88 host 170ms 10MB # # . # . 6.11 17.76 host 50ms 100MB . . # . . 8.20 17.56 host 170ms 10MB . . # # . 6.56 17.51 pasta 170ms 10MB # . . # # 6.39 17.51 host 170ms 10MB . # # # # 4.86 17.46 host 170ms 10MB # # . # # 6.91 17.45 host 50ms 100MB # # # # # 7.11 17.45 host 170ms 10MB . # . # . 4.93 17.45 host 170ms 10MB . # # # . 5.14 17.44 pasta 170ms 10MB . # # # # 5.69 17.41 host 170ms 10MB # . # # # 3.64 17.39 pasta 170ms 10MB # # . # # 6.59 17.33 host 170ms 10MB . # . # # 4.83 17.33 pasta 170ms 10MB # . . # . 5.46 17.30 pasta 50ms 100MB # . . # # 6.23 17.30 pasta 170ms 10MB # . . # . 4.30 17.25 host 170ms 10MB . . # # # 5.64 17.25 pasta 50ms 100MB # . . # . 6.76 17.20 pasta 170ms 10MB . . . # # 7.77 17.19 host 50ms 100MB . . . # # 7.61 17.18 host 170ms 10MB . . . # # 6.01 17.16 host 50ms 100MB . # # # # 6.44 17.04 host 170ms 10MB # # # # . 5.08 17.04 host 170ms 10MB . # # # . 4.62 17.02 host 50ms 100MB # . # # # 6.51 17.02 host 50ms 100MB # . . # # 6.20 16.99 host 50ms 100MB # . # # . 6.43 16.98 pasta 170ms 10MB # # . # . 5.87 16.88 pasta 50ms 100MB # # . # # 6.36 16.86 host 170ms 10MB . . . # # 5.03 16.85 host 170ms 10MB # # # # # 3.44 16.83 host 170ms 10MB # # . # # 6.12 16.83 host 170ms 10MB # . # # . 5.62 16.83 pasta 170ms 10MB # . . # # 14.82 16.83 host 170ms 10MB # . . # . 5.14 16.82 host 170ms 10MB # . # # . 6.47 16.81 host 170ms 10MB . # . # # 5.13 16.79 host 170ms 10MB # . . # . 5.72 16.78 host 170ms 10MB # # # # # 5.49 16.78 host 170ms 10MB # # # # . 4.72 16.77 host 170ms 10MB # # . # . 5.79 16.76 host 50ms 100MB . . # # # 7.41 16.71 host 170ms 10MB . . # # # 6.10 16.70 host 50ms 100MB . # . . # 6.69 16.68 host 170ms 10MB . # # # # 6.06 16.67 host 170ms 10MB # . . # # 6.01 16.65 host 170ms 10MB . # . # . 5.32 16.58 host 170ms 10MB . . # # . 4.90 16.55 host 50ms 100MB # # . # # 6.47 16.40 pasta 170ms 10MB . # # . . 4.81 16.19 host 170ms 10MB # . # # # 5.77 16.06 host 50ms 100MB . # # # . 6.67 15.97 pasta 170ms 10MB # # . . # 6.78 15.23 host 50ms 100MB # # # # . 6.56 14.97 pasta 50ms 100MB . . . . . 8.19 14.95 pasta 170ms 10MB . . . . . 6.10 14.69 pasta 170ms 10MB # . . . . 5.78 14.51 pasta 170ms 10MB . # . . # 6.42 14.22 pasta 170ms 10MB . # . . . 6.30 14.20 host 50ms 100MB # # . # . 6.12 14.08 pasta 170ms 10MB . . . . # 6.63 14.07 pasta 170ms 10MB . # . . # 4.93 13.53 pasta 50ms 10MB # # # # # 1.29 13.41 pasta 170ms 10MB # . . . # 6.31 13.11 host 50ms 100MB . # # . # 7.62 13.07 pasta 170ms 10MB # # . . . 6.41 13.03 host 50ms 10MB # # # # . 1.80 12.98 host 50ms 100MB . . . # . 7.67 12.88 pasta 170ms 10MB # . . . . 5.65 12.76 pasta 170ms 10MB . . . . # 5.88 12.59 pasta 170ms 10MB # # . . # 5.69 12.55 host 50ms 100MB # . # . . 6.67 12.52 pasta 50ms 10MB # . # # # 1.22 12.50 host 170ms 10MB # . # . . 5.41 12.41 pasta 50ms 100MB # # . . . 6.58 12.28 pasta 50ms 10MB # # # # # 2.12 12.20 pasta 170ms 10MB . # . . . 6.56 12.16 pasta 170ms 10MB . . . . . 5.38 12.11 host 170ms 10MB # # # . # 5.60 12.00 pasta 170ms 10MB # . . . # 10.77 11.98 pasta 50ms 10MB # # # # . 1.19 11.92 pasta 50ms 100MB # . . . . 6.63 11.81 host 170ms 10MB . . # . # 5.55 11.80 host 50ms 100MB . # . . . 6.64 11.71 host 170ms 10MB # # # . # 5.35 11.70 pasta 50ms 100MB . # . . # 7.40 11.69 host 170ms 10MB # . # . # 6.18 11.66 host 50ms 100MB # # # . . 6.40 11.60 pasta 50ms 10MB . # # # . 1.24 11.51 host 170ms 10MB . # . . # 5.54 11.40 pasta 50ms 10MB . . # # # 1.35 11.38 host 170ms 10MB . # . . . 6.33 11.23 host 170ms 10MB # . # . # 4.25 11.22 host 50ms 100MB # # . . . 6.90 11.19 host 170ms 10MB . # # . . 6.59 11.16 host 50ms 100MB . # # . . 7.29 11.09 host 50ms 100MB . . # . # 8.63 11.07 host 170ms 10MB . . . . # 6.00 10.99 pasta 50ms 100MB # . . . # 7.90 10.86 host 170ms 10MB . . . . . 6.66 10.86 pasta 50ms 10MB # . # . # 2.61 10.84 host 170ms 10MB # . # . . 5.41 10.83 host 170ms 10MB # . . . # 5.80 10.76 host 170ms 10MB . . # . . 5.38 10.68 pasta 50ms 100MB . # . . . 7.13 10.68 host 170ms 10MB # # . . . 8.34 10.68 pasta 50ms 10MB . # # # # 1.39 10.66 pasta 50ms 10MB . # # # . 1.39 10.64 host 170ms 10MB # . . . . 4.92 10.64 host 170ms 10MB # # . . # 5.70 10.60 pasta 50ms 100MB . . . . # 6.92 10.56 host 50ms 100MB . . . . # 6.86 10.56 pasta 50ms 100MB # # . . # 6.32 10.53 host 170ms 10MB . . # . # 6.53 10.52 host 50ms 100MB # . . . . 6.67 10.52 host 170ms 10MB . . . . # 5.04 10.48 host 50ms 100MB # # # . # 6.39 10.45 host 170ms 10MB . # . . # 5.09 10.36 host 170ms 10MB # # # . . 6.01 10.29 host 50ms 100MB # . # . # 8.37 10.19 host 170ms 10MB . # # . . 7.35 10.14 host 170ms 10MB . # . . . 5.03 10.13 host 170ms 10MB # # # . . 6.11 10.08 host 170ms 10MB . # # . # 6.63 10.04 host 50ms 100MB # . . . # 7.07 10.04 host 50ms 100MB . . . . . 7.68 10.01 host 170ms 10MB # # . . # 4.41 9.96 host 170ms 10MB . . . . . 6.56 9.92 host 170ms 10MB # . . . . 5.72 9.88 pasta 50ms 10MB # . # . # 1.64 9.82 host 50ms 100MB # # . . # 6.32 9.72 host 170ms 10MB # . . . # 6.53 9.60 pasta 50ms 10MB # # # . # 1.53 9.25 host 170ms 10MB # # . . . 5.07 9.16 pasta 50ms 10MB . . # # # 1.41 9.10 host 170ms 10MB . . # . . 6.06 8.95 host 170ms 10MB . # # . # 6.71 8.83 pasta 50ms 10MB . . # . . 1.42 8.81 pasta 50ms 10MB . . # # . 1.45 8.65 pasta 50ms 10MB . # # # # 1.28 8.35 pasta 50ms 10MB # . # . . 1.47 7.92 pasta 50ms 10MB . # # . # 1.51 7.78 pasta 50ms 10MB # # # . . 1.87 7.61 pasta 50ms 10MB . . # # . 1.53 7.32 pasta 50ms 10MB # . # # # 1.33 7.22 pasta 50ms 10MB # # # . # 1.31 7.15 pasta 50ms 10MB . . . . . 1.34 7.12 pasta 50ms 10MB # # # . . 1.42 6.95 pasta 50ms 10MB . # # . . 1.51 6.83 pasta 50ms 10MB . # . . # 1.79 6.76 host 50ms 10MB . . . # # 1.42 6.66 pasta 50ms 10MB # . # . . 1.60 6.26 pasta 50ms 10MB # . . # # 1.55 5.66 pasta 50ms 10MB . # # . . 1.55 5.62 pasta 50ms 10MB . # . . . 1.44 5.53 pasta 50ms 10MB # # . # # 1.30 5.52 pasta 50ms 10MB . . # . # 1.43 5.28 pasta 50ms 10MB . . . . # 1.35 5.20 pasta 50ms 10MB . . . . # 1.49 5.15 pasta 50ms 10MB . # # . # 1.47 5.13 pasta 50ms 10MB . . # . . 1.40 5.10 pasta 50ms 10MB . . . . . 1.35 5.10 pasta 50ms 10MB . . # . # 1.40 5.08 host 50ms 10MB . # . # # 1.64 4.99 pasta 50ms 10MB # . . . # 1.44 4.69 pasta 50ms 10MB . # . # # 2.60 4.67 pasta 50ms 10MB . # . . . 1.45 4.63 pasta 50ms 10MB . # . . # 1.34 4.63 host 50ms 10MB . . # # . 1.31 4.57 host 50ms 10MB # . . # # 1.37 4.51 host 50ms 10MB # # . # # 1.24 4.48 pasta 50ms 10MB # # . . # 1.55 4.44 host 50ms 10MB # # . . # 1.49 4.42 pasta 50ms 10MB # # . . . 1.22 4.35 host 50ms 10MB . # . # . 1.29 4.33 host 50ms 10MB . # # # . 1.36 4.27 host 50ms 10MB # # # # # 1.32 4.04 pasta 50ms 10MB . . . # # 1.31 3.89 pasta 50ms 10MB . # . # . 1.33 3.88 pasta 50ms 10MB . . . # # 1.50 3.85 host 50ms 10MB # . # # . 1.43 3.78 host 50ms 10MB . . . # . 1.28 3.75 host 50ms 10MB # . # . # 1.52 3.74 pasta 50ms 10MB # . . # . 1.38 3.73 host 50ms 10MB . # # # # 1.36 3.68 host 50ms 10MB # # . # . 1.21 3.68 host 50ms 10MB . # . # # 1.41 3.66 pasta 50ms 10MB # . . # . 1.43 3.66 host 50ms 10MB . . . # # 1.62 3.65 pasta 50ms 10MB # # . # . 1.51 3.64 host 50ms 10MB . . # # # 1.31 3.64 host 50ms 10MB . . # . . 1.48 3.61 pasta 50ms 10MB # . . . . 1.30 3.60 pasta 50ms 10MB # . . . # 1.49 3.56 host 50ms 10MB . . . . # 1.30 3.54 pasta 50ms 10MB # # . # # 1.33 3.53 host 50ms 10MB # # . # . 1.38 3.52 host 50ms 10MB # . # # # 1.18 3.51 host 50ms 10MB . . . # . 1.59 3.51 pasta 50ms 10MB . . . # . 1.58 3.49 host 50ms 10MB . . . . . 1.48 3.48 host 50ms 10MB . # . . # 1.57 3.46 pasta 50ms 10MB # . . . . 1.49 3.45 pasta 50ms 10MB . # . # # 1.46 3.44 host 50ms 10MB . # # . . 1.41 3.41 host 50ms 10MB # # # . . 1.34 3.40 host 50ms 10MB # # # . # 1.34 3.40 host 50ms 10MB # # # # . 1.29 3.39 host 50ms 10MB # . # . . 1.87 3.39 host 50ms 10MB # # # . # 1.38 3.38 host 50ms 10MB . . . . . 1.43 3.37 pasta 50ms 10MB # # . . . 1.29 3.36 host 50ms 10MB # . # . # 1.59 3.33 host 50ms 10MB . . # . # 1.46 3.32 host 50ms 10MB # . . . . 1.47 3.29 host 50ms 10MB . # . . . 1.49 3.28 host 50ms 10MB . # . . # 1.38 3.28 host 50ms 10MB . . # . . 1.39 3.28 host 50ms 10MB # . . # . 1.43 3.27 host 50ms 10MB # . . # . 1.35 3.27 pasta 50ms 10MB # . . # # 1.39 3.26 host 50ms 10MB # # . . . 1.28 3.24 host 50ms 10MB . # # . # 1.43 3.24 host 50ms 10MB # . . . # 1.57 3.19 host 50ms 10MB . # # # . 1.27 3.19 host 50ms 10MB . # # # # 1.54 3.18 host 50ms 10MB . . # # . 1.34 3.14 host 50ms 10MB # . # # . 1.32 3.14 pasta 50ms 10MB . . . # . 1.58 3.12 host 50ms 10MB . # # . . 1.59 3.09 host 50ms 10MB . # . # . 1.43 3.08 pasta 50ms 10MB # # . . # 1.30 3.07 pasta 50ms 10MB # # . # . 1.35 3.01 host 50ms 10MB # # # # # 1.53 2.99 host 50ms 10MB . # # . # 1.54 2.94 host 50ms 10MB # . # . . 1.58 2.94 host 50ms 10MB . # . . . 1.43 2.87 host 50ms 10MB . . # . # 1.46 2.87 host 50ms 10MB # . # # # 1.36 2.87 host 50ms 10MB # # # . . 1.40 2.84 host 50ms 10MB # # . . # 1.29 2.79 host 50ms 10MB # . . . . 1.32 2.78 host 50ms 10MB # . . . # 1.37 2.68 host 50ms 10MB # # . . . 1.21 2.67 host 50ms 10MB . . . . # 1.37 2.66 host 50ms 10MB # # . # # 1.34 2.39 host 50ms 10MB # . . # # 1.43 2.15 pasta 50ms 10MB . # . # . 1.45 2.04 host 50ms 10MB . . # # # 1.54 1.93 column description network podman --network=3D argument ping the RTT to the remote server size the size of the file {up,down}loaded wmem sysctl rmem sysctl lowat sysctl cngst sysctl qdisc sysctl down download time in seconds up upload time in seconds (timed out at 120s for the 10MB tests a= nd 180s for the 100MB tests = ) column sysctl "#" (custom) "." (default) wmem net.core.wmem_max 7500000 212992 rmem net.core.rmem_max 7500000 212992 lowat net.ipv4.tcp_notsent_lowat 131072 4294967295 cngst net.ipv4.tcp_congestion_control bbr cubic qdisc net.core.default_qdisc cake fq_codel And this is the messy script that I used to generate this data (I'm hosting both servers myself, so feel free to use them for testing): #!/usr/bin/bash set -euo pipefail # The location of the files script_dir=3D"$(dirname "$(realpath "$0")")" output_file=3D"$script_dir/pasta-upload-test-results.tsv" # The size of the file to upload/download file_size=3D"100M" # Define the sysctl parameters declare -a -r sysctl_keys=3D( "custom" "default" ) declare -A -r wmem_maxes=3D( [default]=3D212992 [custom]=3D7500000 ) declare -A -r rmem_maxes=3D( [default]=3D212992 [custom]=3D7500000 ) declare -A -r tcp_notsent_lowats=3D( [default]=3D4294967295 [custom]=3D131072 ) declare -A -r tcp_congestion_controls=3D( [default]=3D"cubic" [custom]=3D"bbr" ) declare -A -r default_qdiscs=3D( [default]=3D"fq_codel" [custom]=3D"cake" ) # Define the network parameters declare -a -r podman_networks=3D("host" "pasta") declare -A -r servers=3D( [50ms]=3D"https://www.maxchernoff.ca/tools/speedtest" [170ms]=3D"https://svn.tug.org:8369/speedtest" ) # Initialize the temporary directory temp_dir=3D$(mktemp -d) cd "$temp_dir" trap 'rm -rf "$temp_dir"' EXIT # Define the network test commands download() { local network=3D$1 local server=3D$2 podman run \ --quiet \ --rm \ --pull=3D"never" \ --volume=3D"$temp_dir:/srv/:Z" \ --workdir=3D"/srv/" \ --network=3D"$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=3DSIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ "$server?bytes=3D$file_size" \ || echo "180.000000" } upload() { local network=3D$1 local server=3D$2 podman run \ --quiet \ --rm \ --pull=3D"never" \ --volume=3D"$temp_dir:/srv/:Z" \ --workdir=3D"/srv/" \ --network=3D"$network" \ quay.io/fedora/fedora-minimal \ timeout --signal=3DSIGINT 180s \ curl \ --silent \ --output /dev/null \ --write-out '%{time_total}\n' \ -X POST \ --upload-file test.bin \ "$server" \ || echo "180.000000" } # Pull the container podman pull --quiet quay.io/fedora/fedora-minimal >/dev/null 2>&1 # Create a sample file to upload head --bytes=3D"$file_size" /dev/urandom > test.bin # Run the tests printf 'network\tping_time\twmem_max\trmem_max\ttcp_notsent_lowat\ttcp_= congestion_control\tdefault_qdisc\tdownload_time\tupload_time\n' \ | tee "$output_file" for wmem_max in "${sysctl_keys[@]}"; do for rmem_max in "${sysctl_keys[@]}"; do for tcp_notsent_lowat in "${sysctl_keys[@]}"; do for tcp_congestion_control in "${sysctl_keys[@]}"; do for default_qdisc in "${sysctl_keys[@]}"; do sudo sysctl -w \ net.core.wmem_max=3D"${wmem_maxes[$wmem_max]}" = \ net.core.rmem_max=3D"${rmem_maxes[$rmem_max]}" = \ net.ipv4.tcp_notsent_lowat=3D"${tcp_notsent_low= ats[$tcp_notsent_lowat]}" \ net.ipv4.tcp_congestion_control=3D"${tcp_conges= tion_controls[$tcp_congestion_control]}" \ net.core.default_qdisc=3D"${default_qdiscs[$def= ault_qdisc]}" \ > /dev/null for server in "${!servers[@]}"; do for network in "${podman_networks[@]}"; do printf '%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\= n' \ "$network" \ "$server" \ "$wmem_max" \ "$rmem_max" \ "$tcp_notsent_lowat" \ "$tcp_congestion_control" \ "$default_qdisc" \ "$(download "$network" "${servers[$serv= er]}")" \ "$(upload "$network" "${servers[$server= ]}")" \ | tee -a "$output_file" done done done done done done done # Sort the lines cat "$output_file" | \ sort -k9rn -k8rn | \ column --table --table-right=3D2,8,9 --output-width=3D999 | \ tac | awk 'NR=3D=3D1 {line=3D$0; next} 1; END{print line}' | tac | = \ sponge "$output_file" > > I bisected a kernel a few weeks ago, > > so I can try that here if you think that this is a kernel bug and not a > > pasta bug. > > Could be either, I have no real intuition either way so far. Ok, I'll try testing with some older versions of both and see if that narrows down the issue. Thanks, -- Max