Re: pasta slow at HTTP upload

[Stefano Brivio <sbrivio@redhat.com>]

On Wed, 12 Nov 2025 04:22:32 -0700
Max Chernoff <git@maxchernoff.ca> wrote:

> Hi Stefano,
> 
> On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote:
> > Hmm, actually, I have a hack that's not quite correct (we should make
> > ACK_INTERVAL adaptive instead, which is one of the other bits I'm
> > working on), but if it fixes the issue for you, it should at least mean
> > that we're talking about the same issue.
> >
> > Patch attached. Can you give that a try?  
> 
> That seemed to help quite a bit---it's now 200x faster than before, but
> still 10x slower than --network=host:

Thanks for testing. I'm fairly sure it's that problem, then.

Setting 1 ms as interval between checks for socket-side ACKs (reported
by the kernel) as my hack does is not the appropriate solution, I'm
implementing something based on the reported round-trip-time (RTT)
instead.

As a further hack, you could probably do something like this on top:

---
diff --git a/tcp.c b/tcp.c
index 697f80d..8c50ee0 100644
--- a/tcp.c
+++ b/tcp.c
@@ -339,7 +339,7 @@ enum {
 #define MSS_DEFAULT			536
 #define WINDOW_DEFAULT			14600		/*
RFC 6928 */
 
-#define ACK_INTERVAL			1		/* ms */
+#define ACK_INTERVAL			200		/* us */
 #define SYN_TIMEOUT			10		/* s */
 #define ACK_TIMEOUT			2
 #define FIN_TIMEOUT			60
@@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn)
 	}
 
 	if (conn->flags & ACK_TO_TAP_DUE) {
-		it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000;
+		it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000;
 	} else if (conn->flags & ACK_FROM_TAP_DUE) {
 		if (!(conn->events & ESTABLISHED))
 			it.it_value.tv_sec = SYN_TIMEOUT;
---

...but again, I'm going to fix that properly in a bit.

> Also, I should mention that I'm using the following networking-related
> sysctls:
> 
>     net.core.wmem_max=7500000
>     net.core.rmem_max=7500000

Those were settings we recommended for KubeVirt until
https://github.com/kubevirt/user-guide/pull/933, but they don't seem to
necessarily make sense as we seem to have made peace with the TCP
auto-tuning mechanism in Linux meanwhile.

See also https://bugs.passt.top/show_bug.cgi?id=138 and commit
71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to
maximum values").

As the issue here is about socket (kernel) buffers being "too small" for
a while, I guess that those settings plus reverting that commit would
"fix" the issue entirely for you. But it's impractical to rely on users
to set those, that's why I'm looking for something adaptive which still
plays nicely with TCP auto-tuning instead.

>     net.ipv4.tcp_notsent_lowat=131072
>     net.core.default_qdisc=cake
>     net.ipv4.tcp_congestion_control=bbr

I'm not sure if those really matter for pasta, but I haven't really
thought about them.

> I read some articles that suggested that those were a good idea, and
> I've been using them for about a year now, but I can disable those for
> testing if you want. I'm also using systemd's
> IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux
> port restrictions; I can easily disable those too.

No need, I don't think those make a difference.

-- 
Stefano