public inbox for passt-user@passt.top
 help / color / mirror / Atom feed
From: Max Chernoff <git@maxchernoff.ca>
To: passt-user@passt.top
Subject: pasta slow at HTTP upload
Date: Tue, 11 Nov 2025 23:11:17 -0700	[thread overview]
Message-ID: <c8eff1fa02018ad0960290f68b553019cb7e5a4d.camel@maxchernoff.ca> (raw)

Hi,

For the past few months, I've noticed that my HTTP uploads from
containers are really slow. Reproduction:

    $ cd "$(mktemp -d)"
    $ head --bytes=100M /dev/urandom > test.bin
    $ tar cf ./test.tar.gz ./test.bin
    $ chmod -R a+rx .

    $ curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     69  20.0M  0:00:04  0:00:04 --:--:-- 23.9M

    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      6  100M    0     0    6 6912k      0   9517  3:03:39  0:12:23  2:51:16     0
    (Stopped after 10 minutes since I got sick of waiting)

    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     64  18.7M  0:00:05  0:00:05 --:--:-- 23.8M

Some comments about the above:

- "www.ctan.org" is a web server running Apache; it isn't my server, but
  it's where I first noticed this issue, and I have permission from its
  owners to use it for testing.

- The upload speed is constant on the host and with --network=host, but
  with --network=pasta, it starts high-ish, but quickly and continuously
  decreases. But the "bytes transferred" number keeps increasing, so the
  upload hasn't stalled out completely.

- I've also had problems with uploads to GitHub from a custom Go
  binary---it takes ~3 minutes to upload ~600MB, split across 5 files,
  which is much slower than I would expect, but still faster than the
  curl/CTAN upload speeds. So I don't think that this is a bug with curl
  or this specific web server.

- The host that I ran this from is a VM in a datacenter, supposedly with
  a symmetrical 1Gb/s connection. Pinging github.com shows an RTT of
  2ms, and pinging www.ctan.org shows an RTT of 100ms.

- This problem seems to have started a few months ago. I don't have an
  exact date (sorry), but it seems to have been fine before then.

- The host is running Fedora 43 IoT on x86_64, and I'm using the
  following software versions:

      $ rpm --query --queryformat='%{NAME}\t%{VERSION}\n' passt podman containers-common netavark | sort | column --table
      containers-common  0.64.2
      netavark           1.16.1
      passt              0^20250919.g623dbf6
      podman             5.6.2

  I had the same issue when the host/containers were using Fedora 42.

- The host has fairly little CPU/network load when I ran these tests,
  and pasta's CPU usage was low enough that it didn't show up on the
  first page of htop.

Does anybody have any advice on troubleshooting this? I'm willing to
test with other servers/commands, compile/test any patches, or provide
any other information that might be helpful.

Thanks,
-- Max

             reply	other threads:[~2025-11-12  6:11 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-11-12  6:11 Max Chernoff [this message]
     [not found] <176293029592.2033508.497353982367240204@maja>
2025-11-12  6:55 ` Stefano Brivio
2025-11-12 10:32   ` Stefano Brivio
2025-11-12 11:22     ` Max Chernoff
2025-11-12 12:53       ` Stefano Brivio
2025-11-13  0:30         ` Max Chernoff
     [not found]         ` <176301983731.2033508.12381101277059600955@maja>
2025-11-17  5:13           ` David Gibson
2025-11-23  9:12             ` Max Chernoff

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=c8eff1fa02018ad0960290f68b553019cb7e5a4d.camel@maxchernoff.ca \
    --to=git@maxchernoff.ca \
    --cc=passt-user@passt.top \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).