From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=fail (p=quarantine dis=none) header.from=redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 63B5F5A0265 for ; Wed, 04 Nov 2026 14:53:30 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1793800409; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=8rQaLwofMJShAvu2qLdWdarsFpRaztfECEv8NdemMXA=; b=YMiuTtZ4GyaonP1FWZtxSoQtxCqRWMb6XT5zCfitFLmHTywyDY2QcSmb1pTDrz48B6Ym9x Kelz8V2m2LGppOgubiEzXBRasSpZzhl+4s35RbXRU4RSHdSTzv57QxVxp+vPao+I8l4f8U e5DBpkrKxbi1cCVqgCf8Sua9masbLW0= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-257-BqukUsqRP7S8ji_xJeJcHQ-1; Wed, 04 Nov 2026 08:53:27 -0500 X-MC-Unique: BqukUsqRP7S8ji_xJeJcHQ-1 X-Mimecast-MFC-AGG-ID: BqukUsqRP7S8ji_xJeJcHQ_1793800407 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-48dfaaea30cso601213f8f.0 for ; Wed, 04 Nov 2026 05:53:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793800407; x=1794405207; h=date:content-transfer-encoding:content-type:mime-version :organization:references:in-reply-to:message-id:subject:cc:to:from :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to:content-type; bh=y7/YU/KxhDz6j/BXzjU/L+knVKktX6jIVS9HM0L8wbY=; b=X8U7a5GFYExJYhQgSm/4xVcnIHltOVRhgQAdVZ0w2vMWMn5Mj1h1rHpdLrjOCnm8Iw j53aCDf0RmyHg34NdRgcrFpOiBHU9Roj3XB3VHhziYc8nYF2nXU7qWt51e+IK8PudtDX Dq8nWJYwTUHIGxCB6ntIq+qPes0QfEPuUbwJ7XvThqvpGHE/x5DeJ7jFcJ5HsRpNIer+ jH1EjWSpqcoIh5PrRCN6QCybJw4opsz3zMdweKwrEdClVmSASMSVnMf75G+VIHtUw/oA mul+vx9ARLislw8MeZ5z2p8C5W7F1ieCGHkxr18dQhfovGRTsJkeBcSsXTSWUB/R4gZ9 ypqw== X-Gm-Message-State: APuU4Z9QCsrbTwx9dvJ0lgY1ESy+kEEP5oKTC6x1vZcZEJaXbZZgzuzQ zrnq1FLi0ufM4kJDHRtUaCpcEIjywnjfTHwt8yKPl4iw7tvAffLQZ3mCmVkxsHaf2ckUGCxIqW0 DKdwCMx61RJv43OWSOf+ZVX6LzlAPNtJstBgxE6/mFANI5lE4FOR6OWP6asyu8g== X-Gm-Gg: AYBFou0HgJV1aU+6Cp8+vi7SJIiMX2yMMzQzrXk594EQD9qT473rKoGCO1Cg5togvII g2A47wSAQEUoR/pypLP8AY3jCZdwhDiawFrFbLlEk3olfeoYIR90+zEDuqIYajbzRIWgikDUwN8 zwmTYQ89bWmVtU4uNfUGeavXyFL7IWnmYrJ7k1PBf8NSnf5tsiBu1Xlx4b8zx844ZM+FSt7Gg0K Goo67QH/HAUVu1pVF8y4B2o56J9SaocyLeBnigqOAJNwqhS1ale7AbSBHucldYGom0RXa7U8cOD aSDisJ0Pef9bUooCDl8MlHSg1JKZgNOT4k77Ob0LDu+reI+rwVjlx8qn4MCG6PzDOobJqvtKtc+ AyFT5s6Wfcn4IqmoHApUPmvHeHXZtOr+HFp8vLgh8anWZsr+dvNakbK8tfmCS7H1HoAgnz1+5Sn FyOd/J7jZd X-Received: by 2002:a05:6000:29ce:b0:48d:f7c0:97c0 with SMTP id ffacd0b85a97d-48e6e304627mr22318824f8f.50.1793800406373; Wed, 04 Nov 2026 05:53:26 -0800 (PST) X-Received: by 2002:a05:6000:29ce:b0:48d:f7c0:97c0 with SMTP id ffacd0b85a97d-48e6e304627mr22318788f8f.50.1793800405685; Wed, 04 Nov 2026 05:53:25 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-48fed552aaesm4229702f8f.40.2026.11.04.05.53.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 04 Nov 2026 05:53:24 -0800 (PST) From: Stefano Brivio To: Lynxes Exe Subject: Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs Message-ID: <20261104145322.2dc73741@elisabeth> In-Reply-To: References: Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 Date: Wed, 04 Nov 2026 14:53:22 +0100 (CET) X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Rly_0WgKowH2H29v4nZqDXOhjF6cNoW1ktj-PnujWt4_1793800407 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: KCTPYPCF7U3JFK65B3E2DJ3DHS7AARCR X-Message-ID-Hash: KCTPYPCF7U3JFK65B3E2DJ3DHS7AARCR X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi Lynxes, Thanks for the report, and sorry for the delay. Some comments inline: On Wed, 28 Oct 2026 14:15:59 +0100 Lynxes Exe wrote: > Hi, >=20 > A disclaimer before anything is owed: what you're about to read was > written by Claude, and is the result of a bug fixing session that occurre= d > on my personal home infrastructure. It was not intended as a bug hunt. > Since it seems like this issue has been unreported and I couldn't find > anything similar to it, and since what Claude reported seems potentially > plausible, I'm putting it here. > If AI assisted/generated bug reports are unwanted, I apologize. At least as far as I'm concerned, in term of bugs reports, whatever makes sense and is usable is welcome. This report has a few dubious aspects (see below) but it might still be valuable, I think. > Original report > -------------------------- > I think a peer address can get stuck in pasta's low-RTT destination > table and then lose window clamping for good. This part wasn't clearly demonstrated. > That caused every new > TCP connection from our LAN router's address to stall on RTOs until > pasta was restarted. Details below. This looks plausible and somewhat demonstrated. > I haven't found an existing report on Bugzilla or this list. Correct, no known issues in this sense so far. > Version: pasta 2026_10_02.cba3570-3-g8ba4132 (the logic below is > unchanged in current master) > Setup: rootless Docker 29.6.1 via rootlesskit, --net=3Dpasta, > --port-driver=3Dimplicit, --mtu=3D65520 (pasta --tcp-ports=3Daut= o > --udp-ports=3Dauto --no-map-gw --ipv4-only ...) > Kernel: 6.18.57 (Arch linux-lts), x86_64 >=20 > What we think is happening > -------------------------- > We suspect the low-RTT destination table (low_rtt_dst[], > tcp_rtt_dst_check() / tcp_rtt_dst_low()). Our LAN router's address, > 10.0.0.1, ended up in it even though the router is several > milliseconds away. Nothing clearly confirms the "ended up in it" part (yet). > [...] > > We could not read the table directly (no ptrace into pasta), so > "10.0.0.1 is in the table" is inferred from this behaviour and the > code path. ...I guess the problem here is training a language model about the difference between "inferred" and "suspected". :) One could ptrace() as root by the way, but that wouldn't help either. Could you make sure about this with a print in tcp_rtt_dst_check()? If that's not the case, you might need to further trace what's going on in tcp_update_seqack_wnd() instead. Rebuilding and using a modified version of passt is quick (it just depends on a C library): git clone git://passt.top/passt && cd passt [edit as needed] make && sudo make install # Or make install as root Both Podman and Docker / rootlesskit will pick up the local build under /usr/local/bin. You can cleanly remove it with 'make uninstall' as root once you're done. > Why we think 10.0.0.1 got into the table > ---------------------------------------- > A real RTT of 10 us or less to the router is impossible here > (measured RTT 2-6 ms). Our suspicion comes from reading the kernel > source; we have not caught it on a live socket: >=20 > - tcp_rtt_dst_check() tests > (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD. > - The kernel initialises rtt_min to ~0U, both in tcp_init_sock() > (net/ipv4/tcp.c) and for accepted sockets in > tcp_create_openreq_child() (tcp_minisocks.c). tcpi_min_rtt stays > ~0U until the first valid RTT sample. > - tcp_synack_rtt_meas() takes no sample when the SYN-ACK was > retransmitted (rtt_us =3D -1 unless !req->num_retrans). The same > applies on the active side when the SYN was retransmitted (Karn). > - Cast to int, ~0U becomes -1, which passes the "<=3D 10 us" test. > - pasta runs this check in tcp_prepare_flags() when it sends the SYN > (inbound) or SYN-ACK (outbound) to the guest. That is right after > the host-side handshake, before any data ACK could provide a > sample. ...maybe, yes. > We measured exactly such handshake retransmissions on this > Wi-Fi/NAT path: 6 of 1,800 fresh connections took ~1.05 s, i.e. one > 1 s retry. With hundreds of inbound and outbound connections to > 10.0.0.1 per day, one retried handshake would be enough to insert it > permanently. >=20 > The night the stalls became severe, the published container was also > restarted about ten times while clients kept connecting, which made > unusual handshakes more likely. We cannot tell exactly when the > insertion happened. >=20 >=20 > Suggested fixes > --------------- > 1. In tcp_rtt_dst_check(), treat tcpi_min_rtt =3D=3D ~0U (and 0) as > "no sample" and skip the insertion. Compare as unsigned, or check > tcpi_rtt / tcpi_rcv_rtt as well. I'm not sure if we actually risk hitting 0: strictly speaking that would mean less than 1 =C2=B5s which looks unlikely (even for simple veth pairs or local guests), but in practice there might be rounding errors somewhere. Do we need to skip 0? From a quick glance of the kernel paths involved I'd say we don't. Could you test this perhaps? Not build tested but I think you could modify the initial early return condition to: =09if (!min_rtt_cap || =09 (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD || =09 tinfo->tcpi_min_rtt =3D=3D ~0U /* not measured yet */) =09=09return; > 2. Don't trust a single sample forever: give entries a lifetime, or > evict an address when a later connection to it shows a min RTT > above the threshold. Reasonable. I guess we could add a timestamp and check if the entry expired (minutes?) where we update the RTT in tcp_update_seqack_wnd()... or even in tcp_rtt_dst_check(). No strong preference on my side, I didn't really think it through. > 3. Log (at debug level) when an address enters the table. It was > invisible here; the only visible signs were retransmissions > inside the namespace. Yeah, that would be good. I added this feature before we started logging anything, I think. Feel free to send a patch for that. > Workaround: restarting pasta clears the table. >=20 > Happy to test a patch or collect more data. Let me know if you can check those few things I suggested, thanks. --=20 Stefano