Re: pasta slow at HTTP upload

[Max Chernoff <git@maxchernoff.ca>]

Hi Stefano,

On Wed, 2025-11-12 at 13:53 +0100, Stefano Brivio wrote:
> As a further hack, you could probably do something like this on top:
>
> ---
> diff --git a/tcp.c b/tcp.c
> index 697f80d..8c50ee0 100644
> --- a/tcp.c
> +++ b/tcp.c
> @@ -339,7 +339,7 @@ enum {
>  #define MSS_DEFAULT			536
>  #define WINDOW_DEFAULT			14600		/*
> RFC 6928 */
>
> -#define ACK_INTERVAL			1		/* ms */
> +#define ACK_INTERVAL			200		/* us */
>  #define SYN_TIMEOUT			10		/* s */
>  #define ACK_TIMEOUT			2
>  #define FIN_TIMEOUT			60
> @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn)
>  	}
>
>  	if (conn->flags & ACK_TO_TAP_DUE) {
> -		it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000;
> +		it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000;
>  	} else if (conn->flags & ACK_FROM_TAP_DUE) {
>  		if (!(conn->events & ESTABLISHED))
>  			it.it_value.tv_sec = SYN_TIMEOUT;
> ---

That actually makes it worse again, about as bad as before the patch.
But I've just tried rebuilding with original patch again, and also with
the exact same binary that I used yesterday, and that's slow now too.
I've verified with pgrep that Podman is using the correct pasta version,
so I have no idea what's happening.

However, I do remember that for the past few months, some uploads would
randomly go really quickly, so maybe the problem happens sporadically,
and when I was testing the patched version I just happened to get
(un)lucky?

> >     net.core.wmem_max=7500000
> >     net.core.rmem_max=7500000
>
> Those were settings we recommended for KubeVirt until
> https://github.com/kubevirt/user-guide/pull/933, but they don't seem to
> necessarily make sense as we seem to have made peace with the TCP
> auto-tuning mechanism in Linux meanwhile.
>
> See also https://bugs.passt.top/show_bug.cgi?id=138 and commit
> 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to
> maximum values").
>
> As the issue here is about socket (kernel) buffers being "too small" for
> a while, I guess that those settings plus reverting that commit would
> "fix" the issue entirely for you. But it's impractical to rely on users
> to set those, that's why I'm looking for something adaptive which still
> plays nicely with TCP auto-tuning instead.

Ah, I didn't know that those (used to be) recommended for pasta; I set
those for Caddy since it complains on startup if those aren't set

    https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes

> >     net.ipv4.tcp_notsent_lowat=131072
> >     net.core.default_qdisc=cake
> >     net.ipv4.tcp_congestion_control=bbr

I set those as a general performance tuning thing (not for pasta
specifically) based off of

    https://blog.cloudflare.com/optimizing-tcp-for-high-throughput-and-low-latency/

    https://grapheneos.org/articles/server-traffic-shaping

> I'm not sure if those really matter for pasta, but I haven't really
> thought about them.

Aha, those do seem to be the issue. Using the original (unpatched)
pasta:

    (Set everything to my original settings)
    $ sudo sysctl -w net.core.wmem_max=7500000 net.core.rmem_max=7500000 net.ipv4.tcp_notsent_lowat=131072 net.core.default_qdisc=cake net.ipv4.tcp_congestion_control=bbr

    (Test with --network=host)
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     69  20.0M  0:00:04  0:00:04 --:--:-- 20.4M

    (Test with --network=pasta)
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      4  100M    0     0    4 5120k      0  42440  0:41:10  0:02:03  0:39:07 26188
    (Stopped early since I got sick of waiting)

    (Set everything to the kernel defaults)
    $ sudo sysctl -w net.core.wmem_max=212992 net.core.rmem_max=212992 net.ipv4.tcp_notsent_lowat=4294967295 net.core.default_qdisc=fq_codel net.ipv4.tcp_congestion_control=cubic
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M      4  1431k  0:01:11  0:01:11 --:--:-- 1399k

    (tcp_congestion_control=default, tcp_notsent_lowat=custom, [rw]mem_max=default)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M      5  1484k  0:01:08  0:01:08 --:--:-- 1400k

    (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=default)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=212992 net.core.wmem_max=212992
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     12  3576k  0:00:28  0:00:28 --:--:-- 14.9M

    (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=default)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=212992 net.core.wmem_max=212992
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M      8  2595k  0:00:39  0:00:39 --:--:-- 17.1M

    (tcp_congestion_control=default, tcp_notsent_lowat=default, [rw]mem_max=custom)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=cubic net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M      7  2107k  0:00:48  0:00:48 --:--:-- 8310k

    (tcp_congestion_control=custom, tcp_notsent_lowat=default, [rw]mem_max=custom)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=4294967295 net.core.rmem_max=7500000 net.core.wmem_max=7500000
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     15  4620k  0:00:22  0:00:22 --:--:-- 12.1M

    (tcp_congestion_control=custom, tcp_notsent_lowat=custom, [rw]mem_max=custom)
    $ sudo sysctl -w net.ipv4.tcp_congestion_control=bbr net.ipv4.tcp_notsent_lowat=131072 net.core.rmem_max=7500000 net.core.wmem_max=7500000
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=pasta quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
      5  100M    0     0    5 5376k      0  29899  0:58:27  0:03:04  0:55:23     0
    (Stopped early since I got sick of waiting)

    (Test with --network=host again)
    $ podman run --rm --pull=newer --volume="$(realpath .):/srv/:Z" --workdir=/srv/ --network=host quay.io/fedora/fedora-minimal curl --output /dev/null --progress-meter --form file=@./test.tar.gz "https://www.ctan.org/submit/validate"
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100  100M    0   345  100  100M     69  20.0M  0:00:04  0:00:04 --:--:-- 24.3M

I know fairly little about networking and the kernel, so if the answer
is just "don't set those sysctls together", I'd be okay with that. But I
haven't changed these sysctls since February, my upload speeds via pasta
were fine up until a few months ago, and the upload speeds are still
okay with --network=host, so I suspect that this is a bug somewhere.

I also find it quite interesting that setting any of the sysctls
individually or in pairs improves the upload speeds, but setting all 3
at once slows it down drastically. I bisected a kernel a few weeks ago,
so I can try that here if you think that this is a kernel bug and not a
pasta bug.

Thanks,
-- Max