From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=fail (p=none dis=none) header.from=gmail.com Received: from mail-oi1-x230.google.com (mail-oi1-x230.google.com [IPv6:2607:f8b0:4864:20::230]) by passt.top (Postfix) with ESMTPS id 5B2395A061B for ; Wed, 28 Oct 2026 14:16:12 +0100 (CET) Received: by mail-oi1-x230.google.com with SMTP id 5614622812f47-4af81963f35so650993b6e.0 for ; Wed, 28 Oct 2026 06:16:12 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1793193371; cv=none; d=google.com; s=arc-20260327; b=UEYZM9cPbnHX9Lk0LCRjAnLvBiVJGrdtja/Ey7DkaSOKE6psWUsEeyyYGruB1TIGDz PfiFUBHpbr0c/gi+6CbIuB5K6p1jj+TsdMhnlh8XKspzsB4+qTBf44MdO/PnxB0fvdIj EdCbZ/wkTKYm67ODjo3me1alnppG1a4uaOb3EbW7a8FdtpUyOyWaBD3iae5GJQjH9wJh hH+6W8lIlCcIfA43gnZ0dZvbc11jo2NZRzyI6kg7qE60ltcOI68nildQxokNXAjJPEMp a9mVz0IhfnXVPQgmyy4jL2OukPb15BY1HnAUQmZP58mgHV2ffSM0UyxGwdm6Vb1wMxBa TBzA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20260327; h=to:subject:message-id:date:from:mime-version:dkim-signature; bh=jAkk7MbQpWTbmWkKVu7+/Ka9wKtZwLdtnlcvLY4II3A=; fh=MP6pDY7h4lyla7quOTgkRjeKcXyvTic0y98AGmvb8DU=; b=OJP4iZKiyIQsYW2KWZYDyxCpbEJ8aMJPBpos4nE+CSQUMAfC87VOsDRR1MZYA+VSco jSalZg9Cv9KAXnRKXLtt7fhEpvQxHAYL1OWPXxVMXZQ3zFCdpht9VQTfuvk8FRjgrMII 5Goph1nV7RSjlwCACFo9eZ/lGQeHyS1HrLW7iWEVj3dxff5RcT02tkY//DVbtKyG8k3j zNSOeI12DuvTZK4iUZaJcenQPwQkrDzjSOYnNOKnOsHFpkQltnz4w4ChFiDHasc1SO58 ijai1bVBIZiOgaDkS93QefV0r2ENacNZFp0bLIpE7i/QOrFBma7zE2O7SJirC1P3Qn02 LxWg==; darn=passt.top ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20260707; t=1793193371; x=1793798171; darn=passt.top; h=content-type:to:subject:message-id:date:from:mime-version:from:to :cc:subject:date:message-id:reply-to:content-type; bh=jAkk7MbQpWTbmWkKVu7+/Ka9wKtZwLdtnlcvLY4II3A=; b=XSNRfw63va6WlFI0cvxcYLpoz7iKwrjoTw7PU4NrChNZHJxb6MHe9VswI5X8y7NOwJ uWrfiqlfRemcfl9qDEvFtmB1B9BAgrSBzc1f9xY+Gx4xPRcLJ2ugvDhZ4U15EEHrc37e u2G62AKa/sN+ls6whQssMKkH/VSWP4zGskEWFvbErljDDDjUuNcSSUDiZkEOlKK1cKs5 1+oD/Z1KsiLny667xxi+O5qn+jn9vDWB6cXCBh7iLYpc81XLEuuSapRkpp8HQWd1k9mc q5Mlk2yF55PLgmimhnGglNnf8v3/rgqfTMWfTUYybXJn4VM9GR5n1JJP4VImEUF7RwWK dO7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1793193371; x=1793798171; h=content-type:to:subject:message-id:date:from:mime-version:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=jAkk7MbQpWTbmWkKVu7+/Ka9wKtZwLdtnlcvLY4II3A=; b=0AvECPz60uVzPq7N2toYb18xTdrNO9KrKcvQX4nZi1yx6bdL8cEOlBibMsak40HY1S wm9HV+hTJZ7rnnG0TUsHJ024Hf+ajV0GICsYB3d6GyosNhxN8VEzSNGT2IPd/Z3XHIC9 QECjj8MjcCzrNm4TwSrC5gj8+3CjhhY4XBz3RnqL6vRA/YWquppieIvySDmwOBnwu7Q5 EXr8f4R2CKxp9lkLPSqm3z1R97R663b/S5KF7r29eBLDN0Gg4+phpgMbjeLjRIb2pCV7 0eQNAjxC89D04cyCs0j5HrQc5c96qSHGehA1GV6EZTY59x000r+pIT1CJh8ML33sigZb Rt0g== X-Gm-Message-State: AFq9FYLhy4bV0xuyz2MbgVY6rsryWnTGpztsHzOLBGyxhvDTqT3aPLJa mtxOzYoMjZDWmsTMZUlvCZY+grApr3H9/SUKZeKjbnnayUF/9HkcDcwXN9TbMCNfXFLq0PSdqpo NZpbQZKMGfuUvb6R9wFinybGrAA9MsjTQo/7h X-Gm-Gg: AYBFou0yOtRldVPa2JhT6JZemWX1GPh0v4PLeFO6iXt6pxgBHKi318jzYFVagVmaHxk Q/FeRRGX5Gd3dlrmmlBpmKYndSisPd6mz2VENO8NX7UMyXzjAvom9F9NKJkZ2uD8ey5zMPRLmVR ayNrrPAzV115TEmsmw0uJDm0d+NOcspj1N2rhYqYm91rnGo5/Q41bxDW7Q/R5ovGqAAZX8rXwS0 RR6eMtoYMBo4FfjL9mtw+xd2KxhpYWFhRJtDmBK9rASOFLpny40dF465sPtPb6vquACSX7yCk67 zxGdR1MdpQHjA6CrYhY6At3umxn6tl+RMZpWg2PK9rEEpJJLpk4DC02VsgC+XZlWTITTXxXx5Ah q9CwdN03IRDYRTMt1+ero5txOECy86pXt4iy3ofF9m63B9fYIeDTCU8JVV7Y= X-Received: by 2002:a05:6808:23d3:b0:4c1:e6f4:f2c5 with SMTP id 5614622812f47-5639fa6a4b9mr1271068b6e.2.1793193370731; Wed, 28 Oct 2026 06:16:10 -0700 (PDT) MIME-Version: 1.0 From: Lynxes Exe Date: Wed, 28 Oct 2026 14:15:59 +0100 X-Gm-Features: AclHuK-hJBh--CDh9dKUlu9hgU7NtmYPNeQpdQBEYk7AQ0m6W-5HCuV1JP6fSpM Message-ID: Subject: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs To: passt-dev@passt.top Content-Type: text/plain; charset="UTF-8" X-MailFrom: lynxes.exe@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: EXMD6URGNGTFDMKERFRBNNBIYG2H6JCX X-Message-ID-Hash: EXMD6URGNGTFDMKERFRBNNBIYG2H6JCX X-Mailman-Approved-At: Wed, 28 Oct 2026 15:55:04 +0100 X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, A disclaimer before anything is owed: what you're about to read was written by Claude, and is the result of a bug fixing session that occurred on my personal home infrastructure. It was not intended as a bug hunt. Since it seems like this issue has been unreported and I couldn't find anything similar to it, and since what Claude reported seems potentially plausible, I'm putting it here. If AI assisted/generated bug reports are unwanted, I apologize. Original report -------------------------- I think a peer address can get stuck in pasta's low-RTT destination table and then lose window clamping for good. That caused every new TCP connection from our LAN router's address to stall on RTOs until pasta was restarted. Details below. I haven't found an existing report on Bugzilla or this list. Version: pasta 2026_10_02.cba3570-3-g8ba4132 (the logic below is unchanged in current master) Setup: rootless Docker 29.6.1 via rootlesskit, --net=pasta, --port-driver=implicit, --mtu=65520 (pasta --tcp-ports=auto --udp-ports=auto --no-map-gw --ipv4-only ...) Kernel: 6.18.57 (Arch linux-lts), x86_64 What we think is happening -------------------------- We suspect the low-RTT destination table (low_rtt_dst[], tcp_rtt_dst_check() / tcp_rtt_dst_low()). Our LAN router's address, 10.0.0.1, ended up in it even though the router is several milliseconds away. For a peer in that table, tcp_update_seqack_wnd() advertises the host socket's tcpi_snd_wnd to the guest instead of tcp_wnd_from_sndbuf(). The guest then sends more than the host socket can queue, pasta discards the excess, and the guest only recovers through RTOs. Entries are never removed while pasta runs, so one bad insertion degraded every connection from that address for days, until pasta was restarted. Network layout (all private addresses) -------------------------------------- - LAN 10.0.0.0/8. Host: 10.0.0.181 (DHCP) and 10.0.0.80/32. Default gateway: the router, 10.0.0.1, 2-6 ms away over Wi-Fi. - An nginx container's port is published by pasta on 10.0.0.80:4444. - The router forwards WAN 443 to 10.0.0.80:4444 with NAT reflection, so LAN clients using the public name reach the host with source address 10.0.0.1. To pasta, every LAN client is the same peer, 10.0.0.1. - A desktop at 10.0.0.187 can also connect straight to 10.0.0.80:4444 (no NAT, its own address). - A monitoring container opens an HTTPS connection to https://10.0.0.1/ every minute (outbound through pasta), so pasta also sees regular outbound flows to 10.0.0.1. Symptom ------- The first large response on every new connection from 10.0.0.1 stalled for 1.5-3.2 s (3-4 RTOs with exponential backoff). Later responses on the same connection were fine. Connections from 10.0.0.187 to the same port and container never stalled. Test: one connection fetching a 520 KB static file 25 times, counting TCPTimeouts in the container's network namespace. - peer 10.0.0.1, LAN client through NAT reflection: +4 timeouts and one ~3.1 s fetch, every run - peer 10.0.0.1, the router itself, direct, no NAT: 11 of 13 runs stalled the same way - peer 10.0.0.187, desktop, direct: 0 timeouts in every run So the trigger is the peer address, not the NAT path. ss -ti on both legs during one transfer: peer 10.0.0.1 (stalls) 10.0.0.187 (fine) host socket (pasta) retransmits 0 0 tcpi_snd_wnd 252928 146432 container socket snd_wnd 251904 1280 rwnd_limited 7 % 80 % retransmits 186 (backoff 4) 1 transfer time 3.2 s 0.26 s For 10.0.0.1 the window pasta advertises to the container is the host socket's tcpi_snd_wnd, shifted by the window scale (252928 -> 251904). That is the table branch in tcp_update_seqack_wnd(). For 10.0.0.187 it is clamped to buffer space. The host-side socket never retransmits, and the tap device's drop counter does not move. The segments the container retransmits are ones pasta took off the tap and discarded. After restarting pasta (by restarting the rootless Docker daemon), the same tests from 10.0.0.1 gave: snd_wnd 1280, rwnd_limited 73 %, 0 retransmissions, 0 timeouts, about 0.06 s per fetch. Nothing else changed. We could not read the table directly (no ptrace into pasta), so "10.0.0.1 is in the table" is inferred from this behaviour and the code path. Why we think 10.0.0.1 got into the table ---------------------------------------- A real RTT of 10 us or less to the router is impossible here (measured RTT 2-6 ms). Our suspicion comes from reading the kernel source; we have not caught it on a live socket: - tcp_rtt_dst_check() tests (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD. - The kernel initialises rtt_min to ~0U, both in tcp_init_sock() (net/ipv4/tcp.c) and for accepted sockets in tcp_create_openreq_child() (tcp_minisocks.c). tcpi_min_rtt stays ~0U until the first valid RTT sample. - tcp_synack_rtt_meas() takes no sample when the SYN-ACK was retransmitted (rtt_us = -1 unless !req->num_retrans). The same applies on the active side when the SYN was retransmitted (Karn). - Cast to int, ~0U becomes -1, which passes the "<= 10 us" test. - pasta runs this check in tcp_prepare_flags() when it sends the SYN (inbound) or SYN-ACK (outbound) to the guest. That is right after the host-side handshake, before any data ACK could provide a sample. We measured exactly such handshake retransmissions on this Wi-Fi/NAT path: 6 of 1,800 fresh connections took ~1.05 s, i.e. one 1 s retry. With hundreds of inbound and outbound connections to 10.0.0.1 per day, one retried handshake would be enough to insert it permanently. The night the stalls became severe, the published container was also restarted about ten times while clients kept connecting, which made unusual handshakes more likely. We cannot tell exactly when the insertion happened. Suggested fixes --------------- 1. In tcp_rtt_dst_check(), treat tcpi_min_rtt == ~0U (and 0) as "no sample" and skip the insertion. Compare as unsigned, or check tcpi_rtt / tcpi_rcv_rtt as well. 2. Don't trust a single sample forever: give entries a lifetime, or evict an address when a later connection to it shows a min RTT above the threshold. 3. Log (at debug level) when an address enters the table. It was invisible here; the only visible signs were retransmissions inside the namespace. Workaround: restarting pasta clears the table. Happy to test a patch or collect more data. Thanks, Lynxes