From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=FV6i8w+Z; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 55F235A08B5 for ; Wed, 12 Nov 2025 13:53:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1762951998; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=o+Lw2/LkG0hYfkUoe6XFLC2XDIy74NPHh6CmX0bAouM=; b=FV6i8w+ZOAgBfIRo68Kpi3LYLkto6sXJ35FyNi1IUjxL7dB+PBRfTtYdJ4SqNM8YglhqIx OYeP+J3EDZaCuSqXPgMplIGhsiirMaFDfuNaWXb9/ec674VKn4g8sbyUbImY9leWXEl+la bRKY+SEKwBdpZhev5JEoWqsY15NYm/M= Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-526-LrxRIcxZMBqcif6eXMknWQ-1; Wed, 12 Nov 2025 07:53:17 -0500 X-MC-Unique: LrxRIcxZMBqcif6eXMknWQ-1 X-Mimecast-MFC-AGG-ID: LrxRIcxZMBqcif6eXMknWQ_1762951996 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4776b0ada3dso5947105e9.0 for ; Wed, 12 Nov 2025 04:53:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1762951995; x=1763556795; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=o+Lw2/LkG0hYfkUoe6XFLC2XDIy74NPHh6CmX0bAouM=; b=A4vMHumzu4gpkCurCW+XBYK0dT077s7J3OaNHclyh4GWvBdhmTx35ixpXcM1oyQeIw A0F9Ohy/xIytfAGdb2gKe+BlUKp3vgmpwmvZJMfyTKsHf3XDBcJplxx0y2eJtPwjUud+ YhqoZLNf1mO37uf3ZnVQEpKbeRYc1OVE9Kqb/1mkBw2cGdfoaQwL1qnYL4AC3X5eGihb aTUGSvMfUNl9l84TjRuxjAv4BqZv/s7n5oy8nnbZ33C2wNsN1SHhkWzdo8Z3mPacVibW YqM1tMlkdkUVkTB0ubAFLUHL6W45wV6j2BbyWn/oYZIT23IgPqAWKfx+VhhHtu0BA5NJ lWQw== X-Gm-Message-State: AOJu0Yzneew6WCHq7LCbaCxhq8fYhld6pyTTezajDEpDx/pFfg8UX94/ MMtW1BSU3rUnZ6S7kQNlYk6AwOnsegmLf+dNNash0W9iicKYPMEhLY/pkUDYGOZ7ZMnDU9Rr+pW /gJEFjSHiGNtP1fIBwRtSKXx5NyM8t4o0fBcakGL+P1o9e3QjrRWptXRxYkvscKE= X-Gm-Gg: ASbGncsySMdMAdKl+TfSH9k9Uw82wZ4y8g9NFonEPIE3FipAP5KeVUOmgIW2fhuUo/0 vE6eajcuTcwJAh7XbYxW0tE38fi7ONErkNywMOsbxossrs8SS4WE6k8L35+pX5bF7rechQfVSB9 FbMR0Suw9G0jn3kSREFnuvx/vlvLvQED2g47ZPNyMedsKGeJipqtoH8r4DTEQVUxZ/seVKN2En9 5UsB9noYSKlQrLE1tBo5BLo9DBZs6Rr2GTtxRcG5bzuUe22G9o0VTZy6SGzF7UGIolNyq3RgpMo hoVjb8CeOEQX6M9Xh6sbdC3cFxtY6VTNPXzzmuKHkchCS6wK6g9VRY9nO2Z4mCIcBLhmpkXmuhu aGofEcwtbJgFXVSl+2nvdBX0jtcA= X-Received: by 2002:a05:600c:2049:b0:477:8895:303c with SMTP id 5b1f17b1804b1-4778a84b632mr6241845e9.3.1762951995253; Wed, 12 Nov 2025 04:53:15 -0800 (PST) X-Google-Smtp-Source: AGHT+IHT5XFPSCrMyGZ6FRp7XXJBbyLSdTWN0mgtL6juppMeJmUQdWv7VKjY9DaXIJ0UOWWuhGekHA== X-Received: by 2002:a05:600c:2049:b0:477:8895:303c with SMTP id 5b1f17b1804b1-4778a84b632mr6241585e9.3.1762951994723; Wed, 12 Nov 2025 04:53:14 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-47788a92a9bsm13048725e9.8.2025.11.12.04.53.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Nov 2025 04:53:14 -0800 (PST) Date: Wed, 12 Nov 2025 13:53:12 +0100 From: Stefano Brivio To: Max Chernoff Subject: Re: pasta slow at HTTP upload Message-ID: <20251112135312.595c9a44@elisabeth> In-Reply-To: <043088ef8bdc2d2c7a910617eb58d494cd9761e0.camel@maxchernoff.ca> References: <176293029592.2033508.497353982367240204@maja> <20251112075548.0c05a25e@elisabeth> <20251112113201.3bcabc6c@elisabeth> <043088ef8bdc2d2c7a910617eb58d494cd9761e0.camel@maxchernoff.ca> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: df2F9wRDrUKSAEs9gRg8Bi2rJJDLvqW_A0pOkvyWrAM_1762951996 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: S3RKZPJ3GHZDOKRPD722BS43DPXTUPPK X-Message-ID-Hash: S3RKZPJ3GHZDOKRPD722BS43DPXTUPPK X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 12 Nov 2025 04:22:32 -0700 Max Chernoff wrote: > Hi Stefano, > > On Wed, 2025-11-12 at 11:32 +0100, Stefano Brivio wrote: > > Hmm, actually, I have a hack that's not quite correct (we should make > > ACK_INTERVAL adaptive instead, which is one of the other bits I'm > > working on), but if it fixes the issue for you, it should at least mean > > that we're talking about the same issue. > > > > Patch attached. Can you give that a try? > > That seemed to help quite a bit---it's now 200x faster than before, but > still 10x slower than --network=host: Thanks for testing. I'm fairly sure it's that problem, then. Setting 1 ms as interval between checks for socket-side ACKs (reported by the kernel) as my hack does is not the appropriate solution, I'm implementing something based on the reported round-trip-time (RTT) instead. As a further hack, you could probably do something like this on top: --- diff --git a/tcp.c b/tcp.c index 697f80d..8c50ee0 100644 --- a/tcp.c +++ b/tcp.c @@ -339,7 +339,7 @@ enum { #define MSS_DEFAULT 536 #define WINDOW_DEFAULT 14600 /* RFC 6928 */ -#define ACK_INTERVAL 1 /* ms */ +#define ACK_INTERVAL 200 /* us */ #define SYN_TIMEOUT 10 /* s */ #define ACK_TIMEOUT 2 #define FIN_TIMEOUT 60 @@ -582,7 +582,7 @@ static void tcp_timer_ctl(struct tcp_tap_conn *conn) } if (conn->flags & ACK_TO_TAP_DUE) { - it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000 * 1000; + it.it_value.tv_nsec = (long)ACK_INTERVAL * 1000; } else if (conn->flags & ACK_FROM_TAP_DUE) { if (!(conn->events & ESTABLISHED)) it.it_value.tv_sec = SYN_TIMEOUT; --- ...but again, I'm going to fix that properly in a bit. > Also, I should mention that I'm using the following networking-related > sysctls: > > net.core.wmem_max=7500000 > net.core.rmem_max=7500000 Those were settings we recommended for KubeVirt until https://github.com/kubevirt/user-guide/pull/933, but they don't seem to necessarily make sense as we seem to have made peace with the TCP auto-tuning mechanism in Linux meanwhile. See also https://bugs.passt.top/show_bug.cgi?id=138 and commit 71249ef3f9bc ("tcp, tcp_splice: Don't set SO_SNDBUF and SO_RCVBUF to maximum values"). As the issue here is about socket (kernel) buffers being "too small" for a while, I guess that those settings plus reverting that commit would "fix" the issue entirely for you. But it's impractical to rely on users to set those, that's why I'm looking for something adaptive which still plays nicely with TCP auto-tuning instead. > net.ipv4.tcp_notsent_lowat=131072 > net.core.default_qdisc=cake > net.ipv4.tcp_congestion_control=bbr I'm not sure if those really matter for pasta, but I haven't really thought about them. > I read some articles that suggested that those were a good idea, and > I've been using them for about a year now, but I can disable those for > testing if you want. I'm also using systemd's > IPAddressAllow/IPAddressDeny/RestrictAddressFamilies and some SELinux > port restrictions; I can easily disable those too. No need, I don't think those make a difference. -- Stefano