From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=maxchernoff.ca Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=maxchernoff.ca header.i=@maxchernoff.ca header.a=rsa-sha256 header.s=key1 header.b=N+7AVE/E; dkim-atps=neutral Received: from out-182.mta0.migadu.com (out-182.mta0.migadu.com [IPv6:2001:41d0:1004:224b::b6]) by passt.top (Postfix) with ESMTPS id EA1A45A08B5 for ; Wed, 12 Nov 2025 07:11:22 +0100 (CET) Message-ID: DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=maxchernoff.ca; s=key1; t=1762927882; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=naYlJZAEDPn5aib9TqCIrBPpwiI9ot3iiGfo5Kksq60=; b=N+7AVE/EWRkIzBEqjFX7LsiPNBzWsUIM8ZEwfy/oSM1bs+U7N77xWDMj6LmOUdrbBRjlJr trrZOYo21Q6TO/DTQopyhP5r/HZIhqbYobPrtqv/p3PR3x8PqpZc7omcXTOJZptNU2SpYU OWXYpwaJGiog8XeTNrHyToK91d2xAUtgJzrnRH8eVx4ST+uDzHjLMYiJvkfiyw2o1iigbl tSLNwy2RSu2xf25kp6+RPn2mVQk/Y95i+6fMex4i+BxqhR7j5TwTCWYRQgbYppe2QtzjVP b4Mq2lv+OBrPb9PU+wJvrA+QaqXgvxCkvivaAuMVSICYDTKQyGSOueHeut4rsg== Subject: pasta slow at HTTP upload X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Max Chernoff To: passt-user@passt.top Date: Tue, 11 Nov 2025 23:11:17 -0700 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Migadu-Flow: FLOW_OUT X-MailFrom: git@maxchernoff.ca X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: 6JE5NGG5L726ZOIPZVH2E7CVXPIFEHVO X-Message-ID-Hash: 6JE5NGG5L726ZOIPZVH2E7CVXPIFEHVO X-Mailman-Approved-At: Wed, 12 Nov 2025 07:51:35 +0100 X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, For the past few months, I've noticed that my HTTP uploads from containers are really slow. Reproduction: $ cd "$(mktemp -d)" $ head --bytes=3D100M /dev/urandom > test.bin $ tar cf ./test.tar.gz ./test.bin $ chmod -R a+rx . $ curl --output /dev/null --progress-meter --form file=3D@./test.tar.gz= "https://www.ctan.org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 69 20.0M 0:00:04 0:00:04 --:--:-= - 23.9M $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dpasta quay.io/fedora/fedora-minimal curl --outpu= t /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan= .org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 6 100M 0 0 6 6912k 0 9517 3:03:39 0:12:23 2:51:1= 6 0 (Stopped after 10 minutes since I got sick of waiting) $ podman run --rm --pull=3Dnewer --volume=3D"$(realpath .):/srv/:Z" --w= orkdir=3D/srv/ --network=3Dhost quay.io/fedora/fedora-minimal curl --output= /dev/null --progress-meter --form file=3D@./test.tar.gz "https://www.ctan.= org/submit/validate" % Total % Received % Xferd Average Speed Time Time Time = Current Dload Upload Total Spent Left = Speed 100 100M 0 345 100 100M 64 18.7M 0:00:05 0:00:05 --:--:-= - 23.8M Some comments about the above: - "www.ctan.org" is a web server running Apache; it isn't my server, but it's where I first noticed this issue, and I have permission from its owners to use it for testing. - The upload speed is constant on the host and with --network=3Dhost, but with --network=3Dpasta, it starts high-ish, but quickly and continuously decreases. But the "bytes transferred" number keeps increasing, so the upload hasn't stalled out completely. - I've also had problems with uploads to GitHub from a custom Go binary---it takes ~3 minutes to upload ~600MB, split across 5 files, which is much slower than I would expect, but still faster than the curl/CTAN upload speeds. So I don't think that this is a bug with curl or this specific web server. - The host that I ran this from is a VM in a datacenter, supposedly with a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of 2ms, and pinging www.ctan.org shows an RTT of 100ms. - This problem seems to have started a few months ago. I don't have an exact date (sorry), but it seems to have been fine before then. - The host is running Fedora 43 IoT on x86_64, and I'm using the following software versions: $ rpm --query --queryformat=3D'%{NAME}\t%{VERSION}\n' passt podman co= ntainers-common netavark | sort | column --table containers-common 0.64.2 netavark 1.16.1 passt 0^20250919.g623dbf6 podman 5.6.2 I had the same issue when the host/containers were using Fedora 42. - The host has fairly little CPU/network load when I ran these tests, and pasta's CPU usage was low enough that it didn't show up on the first page of htop. Does anybody have any advice on troubleshooting this? I'm willing to test with other servers/commands, compile/test any patches, or provide any other information that might be helpful. Thanks, -- Max