public inbox for passt-dev@passt.top
 help / color / mirror / code / Atom feed
* tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs
@ 2026-10-28 13:15 Lynxes Exe
  2026-10-29  1:11 ` David Gibson
  2026-11-04 13:53 ` Stefano Brivio
  0 siblings, 2 replies; 4+ messages in thread
From: Lynxes Exe @ 2026-10-28 13:15 UTC (permalink / raw)
  To: passt-dev

Hi,

A disclaimer before anything is owed: what you're about to read was
written by Claude, and is the result of a bug fixing session that occurred
on my personal home infrastructure. It was not intended as a bug hunt.
Since it seems like this issue has been unreported and I couldn't find
anything similar to it, and since what Claude reported seems potentially
plausible, I'm putting it here.
If AI assisted/generated bug reports are unwanted, I apologize.

Original report
--------------------------
I think a peer address can get stuck in pasta's low-RTT destination
table and then lose window clamping for good. That caused every new
TCP connection from our LAN router's address to stall on RTOs until
pasta was restarted. Details below. I haven't found an existing
report on Bugzilla or this list.

Version: pasta 2026_10_02.cba3570-3-g8ba4132 (the logic below is
         unchanged in current master)
Setup:   rootless Docker 29.6.1 via rootlesskit, --net=pasta,
         --port-driver=implicit, --mtu=65520 (pasta --tcp-ports=auto
         --udp-ports=auto --no-map-gw --ipv4-only ...)
Kernel:  6.18.57 (Arch linux-lts), x86_64


What we think is happening
--------------------------
We suspect the low-RTT destination table (low_rtt_dst[],
tcp_rtt_dst_check() / tcp_rtt_dst_low()). Our LAN router's address,
10.0.0.1, ended up in it even though the router is several
milliseconds away.

For a peer in that table, tcp_update_seqack_wnd() advertises the host
socket's tcpi_snd_wnd to the guest instead of tcp_wnd_from_sndbuf().
The guest then sends more than the host socket can queue, pasta
discards the excess, and the guest only recovers through RTOs.

Entries are never removed while pasta runs, so one bad insertion
degraded every connection from that address for days, until pasta
was restarted.


Network layout (all private addresses)
--------------------------------------
- LAN 10.0.0.0/8. Host: 10.0.0.181 (DHCP) and 10.0.0.80/32.
  Default gateway: the router, 10.0.0.1, 2-6 ms away over Wi-Fi.
- An nginx container's port is published by pasta on
  10.0.0.80:4444.
- The router forwards WAN 443 to 10.0.0.80:4444 with NAT
  reflection, so LAN clients using the public name reach the host
  with source address 10.0.0.1. To pasta, every LAN client is the
  same peer, 10.0.0.1.
- A desktop at 10.0.0.187 can also connect straight to
  10.0.0.80:4444 (no NAT, its own address).
- A monitoring container opens an HTTPS connection to
  https://10.0.0.1/ every minute (outbound through pasta), so pasta
  also sees regular outbound flows to 10.0.0.1.


Symptom
-------
The first large response on every new connection from 10.0.0.1
stalled for 1.5-3.2 s (3-4 RTOs with exponential backoff). Later
responses on the same connection were fine. Connections from
10.0.0.187 to the same port and container never stalled.

Test: one connection fetching a 520 KB static file 25 times,
counting TCPTimeouts in the container's network namespace.
- peer 10.0.0.1, LAN client through NAT reflection:
  +4 timeouts and one ~3.1 s fetch, every run
- peer 10.0.0.1, the router itself, direct, no NAT:
  11 of 13 runs stalled the same way
- peer 10.0.0.187, desktop, direct:
  0 timeouts in every run
So the trigger is the peer address, not the NAT path.

ss -ti on both legs during one transfer:

  peer                 10.0.0.1 (stalls)    10.0.0.187 (fine)
  host socket (pasta)
    retransmits        0                    0
    tcpi_snd_wnd       252928               146432
  container socket
    snd_wnd            251904               1280
    rwnd_limited       7 %                  80 %
    retransmits        186 (backoff 4)      1
  transfer time        3.2 s                0.26 s

For 10.0.0.1 the window pasta advertises to the container is the
host socket's tcpi_snd_wnd, shifted by the window scale
(252928 -> 251904). That is the table branch in
tcp_update_seqack_wnd(). For 10.0.0.187 it is clamped to buffer
space.

The host-side socket never retransmits, and the tap device's drop
counter does not move. The segments the container retransmits are
ones pasta took off the tap and discarded.

After restarting pasta (by restarting the rootless Docker daemon),
the same tests from 10.0.0.1 gave: snd_wnd 1280, rwnd_limited 73 %,
0 retransmissions, 0 timeouts, about 0.06 s per fetch. Nothing else
changed.

We could not read the table directly (no ptrace into pasta), so
"10.0.0.1 is in the table" is inferred from this behaviour and the
code path.


Why we think 10.0.0.1 got into the table
----------------------------------------
A real RTT of 10 us or less to the router is impossible here
(measured RTT 2-6 ms). Our suspicion comes from reading the kernel
source; we have not caught it on a live socket:

- tcp_rtt_dst_check() tests
  (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD.
- The kernel initialises rtt_min to ~0U, both in tcp_init_sock()
  (net/ipv4/tcp.c) and for accepted sockets in
  tcp_create_openreq_child() (tcp_minisocks.c). tcpi_min_rtt stays
  ~0U until the first valid RTT sample.
- tcp_synack_rtt_meas() takes no sample when the SYN-ACK was
  retransmitted (rtt_us = -1 unless !req->num_retrans). The same
  applies on the active side when the SYN was retransmitted (Karn).
- Cast to int, ~0U becomes -1, which passes the "<= 10 us" test.
- pasta runs this check in tcp_prepare_flags() when it sends the SYN
  (inbound) or SYN-ACK (outbound) to the guest. That is right after
  the host-side handshake, before any data ACK could provide a
  sample.

We measured exactly such handshake retransmissions on this
Wi-Fi/NAT path: 6 of 1,800 fresh connections took ~1.05 s, i.e. one
1 s retry. With hundreds of inbound and outbound connections to
10.0.0.1 per day, one retried handshake would be enough to insert it
permanently.

The night the stalls became severe, the published container was also
restarted about ten times while clients kept connecting, which made
unusual handshakes more likely. We cannot tell exactly when the
insertion happened.


Suggested fixes
---------------
1. In tcp_rtt_dst_check(), treat tcpi_min_rtt == ~0U (and 0) as
   "no sample" and skip the insertion. Compare as unsigned, or check
   tcpi_rtt / tcpi_rcv_rtt as well.
2. Don't trust a single sample forever: give entries a lifetime, or
   evict an address when a later connection to it shows a min RTT
   above the threshold.
3. Log (at debug level) when an address enters the table. It was
   invisible here; the only visible signs were retransmissions
   inside the namespace.

Workaround: restarting pasta clears the table.

Happy to test a patch or collect more data.

Thanks,
Lynxes

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs
  2026-10-28 13:15 tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs Lynxes Exe
@ 2026-10-29  1:11 ` David Gibson
  2026-11-04 13:53 ` Stefano Brivio
  1 sibling, 0 replies; 4+ messages in thread
From: David Gibson @ 2026-10-29  1:11 UTC (permalink / raw)
  To: Lynxes Exe; +Cc: passt-dev

[-- Attachment #1: Type: text/plain, Size: 1080 bytes --]

On Wed, Oct 28, 2026 at 02:15:59PM +0100, Lynxes Exe wrote:
> Hi,
> 
> A disclaimer before anything is owed: what you're about to read was
> written by Claude, and is the result of a bug fixing session that occurred
> on my personal home infrastructure. It was not intended as a bug hunt.
> Since it seems like this issue has been unreported and I couldn't find
> anything similar to it, and since what Claude reported seems potentially
> plausible, I'm putting it here.
> If AI assisted/generated bug reports are unwanted, I apologize.

Well, I can't speak for Stefano, but I strongly suspect he'll agree
that one phrased like this is much more welcome than the sort which
confidently and incorrectly claims the sky is falling and implies the
mythical passt security team should be right this instant be conveying
the report to the CVE bunker under armed guard :).

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs
  2026-10-28 13:15 tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs Lynxes Exe
  2026-10-29  1:11 ` David Gibson
@ 2026-11-04 13:53 ` Stefano Brivio
  2026-11-12 16:09   ` Lynxes Exe
  1 sibling, 1 reply; 4+ messages in thread
From: Stefano Brivio @ 2026-11-04 13:53 UTC (permalink / raw)
  To: Lynxes Exe; +Cc: passt-dev

Hi Lynxes,

Thanks for the report, and sorry for the delay.  Some comments inline:

On Wed, 28 Oct 2026 14:15:59 +0100
Lynxes Exe <lynxes.exe@gmail.com> wrote:

> Hi,
> 
> A disclaimer before anything is owed: what you're about to read was
> written by Claude, and is the result of a bug fixing session that occurred
> on my personal home infrastructure. It was not intended as a bug hunt.
> Since it seems like this issue has been unreported and I couldn't find
> anything similar to it, and since what Claude reported seems potentially
> plausible, I'm putting it here.
> If AI assisted/generated bug reports are unwanted, I apologize.

At least as far as I'm concerned, in term of bugs reports, whatever
makes sense and is usable is welcome.  This report has a few dubious
aspects (see below) but it might still be valuable, I think.

> Original report
> --------------------------
> I think a peer address can get stuck in pasta's low-RTT destination
> table and then lose window clamping for good.

This part wasn't clearly demonstrated.

> That caused every new
> TCP connection from our LAN router's address to stall on RTOs until
> pasta was restarted. Details below.

This looks plausible and somewhat demonstrated.

> I haven't found an existing report on Bugzilla or this list.

Correct, no known issues in this sense so far.

> Version: pasta 2026_10_02.cba3570-3-g8ba4132 (the logic below is
>          unchanged in current master)
> Setup:   rootless Docker 29.6.1 via rootlesskit, --net=pasta,
>          --port-driver=implicit, --mtu=65520 (pasta --tcp-ports=auto
>          --udp-ports=auto --no-map-gw --ipv4-only ...)
> Kernel:  6.18.57 (Arch linux-lts), x86_64
> 
> What we think is happening
> --------------------------
> We suspect the low-RTT destination table (low_rtt_dst[],
> tcp_rtt_dst_check() / tcp_rtt_dst_low()). Our LAN router's address,
> 10.0.0.1, ended up in it even though the router is several
> milliseconds away.

Nothing clearly confirms the "ended up in it" part (yet).

> [...]
>
> We could not read the table directly (no ptrace into pasta), so
> "10.0.0.1 is in the table" is inferred from this behaviour and the
> code path.

...I guess the problem here is training a language model about the
difference between "inferred" and "suspected". :)

One could ptrace() as root by the way, but that wouldn't help either.

Could you make sure about this with a print in tcp_rtt_dst_check()?  If
that's not the case, you might need to further trace what's going on in
tcp_update_seqack_wnd() instead.

Rebuilding and using a modified version of passt is quick (it just
depends on a C library):

  git clone git://passt.top/passt && cd passt
  [edit as needed]
  make && sudo make install # Or make install as root

Both Podman and Docker / rootlesskit will pick up the local build under
/usr/local/bin. You can cleanly remove it with 'make uninstall' as root
once you're done.

> Why we think 10.0.0.1 got into the table
> ----------------------------------------
> A real RTT of 10 us or less to the router is impossible here
> (measured RTT 2-6 ms). Our suspicion comes from reading the kernel
> source; we have not caught it on a live socket:
> 
> - tcp_rtt_dst_check() tests
>   (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD.
> - The kernel initialises rtt_min to ~0U, both in tcp_init_sock()
>   (net/ipv4/tcp.c) and for accepted sockets in
>   tcp_create_openreq_child() (tcp_minisocks.c). tcpi_min_rtt stays
>   ~0U until the first valid RTT sample.
> - tcp_synack_rtt_meas() takes no sample when the SYN-ACK was
>   retransmitted (rtt_us = -1 unless !req->num_retrans). The same
>   applies on the active side when the SYN was retransmitted (Karn).
> - Cast to int, ~0U becomes -1, which passes the "<= 10 us" test.
> - pasta runs this check in tcp_prepare_flags() when it sends the SYN
>   (inbound) or SYN-ACK (outbound) to the guest. That is right after
>   the host-side handshake, before any data ACK could provide a
>   sample.

...maybe, yes.

> We measured exactly such handshake retransmissions on this
> Wi-Fi/NAT path: 6 of 1,800 fresh connections took ~1.05 s, i.e. one
> 1 s retry. With hundreds of inbound and outbound connections to
> 10.0.0.1 per day, one retried handshake would be enough to insert it
> permanently.
> 
> The night the stalls became severe, the published container was also
> restarted about ten times while clients kept connecting, which made
> unusual handshakes more likely. We cannot tell exactly when the
> insertion happened.
> 
> 
> Suggested fixes
> ---------------
> 1. In tcp_rtt_dst_check(), treat tcpi_min_rtt == ~0U (and 0) as
>    "no sample" and skip the insertion. Compare as unsigned, or check
>    tcpi_rtt / tcpi_rcv_rtt as well.

I'm not sure if we actually risk hitting 0: strictly speaking that
would mean less than 1 µs which looks unlikely (even for simple veth
pairs or local guests), but in practice there might be rounding errors
somewhere.

Do we need to skip 0? From a quick glance of the kernel paths involved
I'd say we don't.

Could you test this perhaps?  Not build tested but I think you could
modify the initial early return condition to:

	if (!min_rtt_cap ||
	    (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD ||
	    tinfo->tcpi_min_rtt == ~0U /* not measured yet */)
		return;

> 2. Don't trust a single sample forever: give entries a lifetime, or
>    evict an address when a later connection to it shows a min RTT
>    above the threshold.

Reasonable.  I guess we could add a timestamp and check if the entry
expired (minutes?) where we update the RTT in
tcp_update_seqack_wnd()... or even in tcp_rtt_dst_check().  No strong
preference on my side, I didn't really think it through.

> 3. Log (at debug level) when an address enters the table. It was
>    invisible here; the only visible signs were retransmissions
>    inside the namespace.

Yeah, that would be good.  I added this feature before we started
logging anything, I think.  Feel free to send a patch for that.

> Workaround: restarting pasta clears the table.
> 
> Happy to test a patch or collect more data.

Let me know if you can check those few things I suggested, thanks.

-- 
Stefano


^ permalink raw reply	[flat|nested] 4+ messages in thread

* Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs
  2026-11-04 13:53 ` Stefano Brivio
@ 2026-11-12 16:09   ` Lynxes Exe
  0 siblings, 0 replies; 4+ messages in thread
From: Lynxes Exe @ 2026-11-12 16:09 UTC (permalink / raw)
  To: Stefano Brivio; +Cc: passt-dev

[-- Attachment #1: Type: text/plain, Size: 10438 bytes --]

Hi Stefano,

I apologize for the long wait. The issue took a while to represent itself.
As usual, this was done with the assistance of Claude.
After the issue occurred again the table got extracted to show that in fact
two IPs that were not supposed to be there got in the table (more details
below).

Also, thanks to both of you for the responses.

Original response
---------------------------

Thanks for going through it.

The stalls came back this week, about two weeks after the last pasta
restart, so we could check what you asked on a live instance.

1. The insertion, now demonstrated

The pasta in Docker's rootless bundle is static, non-PIE and not
stripped, so low_rtt_dst sits at a fixed address. We read it as root from
/proc/<pid>/mem while pasta kept running:

  slot 0  10.0.0.181     the host's own LAN address
  slot 1  192.168.5.3    a WireGuard peer, over the Internet
  slot 2  10.0.0.80      the host's second address
  slot 3  192.168.122.1  libvirt bridge, host side
  slot 4  10.0.0.187     a Windows desktop on Wi-Fi, RTT 2-6 ms
  slot 5-7               empty

The desktop was the peer that stalled: through pasta it got 14-33 KB/s,
with retransmissions and backoff inside the namespace, while SSH from
the same machine to the host ran at about 9 MB/s. 10.0.0.1 from the
first report wasn't there, because pasta had been restarted since.

2. Why: the ~0U case, reproduced

For a passively opened socket, the handshake gives no RTT sample if the
SYN-ACK was retransmitted (tcp_synack_rtt_meas() only measures when
req->num_retrans is 0, and a retransmitted SYN from the client also
increments it, via tcp_rtx_synack()). The only fallback is the
timestamp echo in tcp_ack_update_rtt(), so if the client didn't
negotiate timestamps, tcpi_min_rtt is still ~0U when pasta checks it.
Windows doesn't offer timestamps on the connections it opens, by
default, which fits the desktop.

Reproducer: from a Linux client with net.ipv4.tcp_timestamps = 0, drop
incoming SYN-ACKs from the pasta host for 2 seconds, with an nftables
input rule on the client's fixed local port, while connecting to a port
that pasta forwards. The client's SYN retries make the server
retransmit the SYN-ACK, and the connection completes after about 2 s.

Results, on Linux 6.18.57, with pasta built from 8ba4132 plus your
check and some logging:

- A plain listener on the same host, reading TCP_INFO right after
  accept(): tcpi_min_rtt 0xffffffff, tcpi_rtt 0, tcpi_total_retrans 3.
  The same client without the drop: tcpi_min_rtt 5423 us.
- pasta: your check fired, the client stayed out of the table, and
  transfers from it stayed at full speed. Without the check, it would
  have been added.
- With timestamps enabled on the client, the same test gets a measured
  RTT from the timestamp echo, and nothing is skipped.

We never saw 0, so skipping only ~0U looks right to me.

3. Patches

Attached, against current master (7380393):

  0001 tcp: Don't add peers to low RTT table before RTT is measured
  0002 tcp: Log additions to low RTT table

0001 is your condition as you wrote it. 0002 logs only additions, with
flow_dbg(): the skip itself happens on every tcp_prepare_flags() call
until the first sample, which was 28 times for one connection in the
test above, so logging it would be noisy. Each patch builds without
warnings; I haven't run the test suite.

I didn't do the expiry part. I'm happy to try it if you have a
preference between tcp_rtt_dst_check() and tcp_update_seqack_wnd().

We're keeping the patched build running and will report if a remote
peer still gets into the table.

Thanks,
Lynxes


On Wed, Nov 4, 2026 at 2:53 PM Stefano Brivio <sbrivio@redhat.com> wrote:
>
> Hi Lynxes,
>
> Thanks for the report, and sorry for the delay.  Some comments inline:
>
> On Wed, 28 Oct 2026 14:15:59 +0100
> Lynxes Exe <lynxes.exe@gmail.com> wrote:
>
> > Hi,
> >
> > A disclaimer before anything is owed: what you're about to read was
> > written by Claude, and is the result of a bug fixing session that occurred
> > on my personal home infrastructure. It was not intended as a bug hunt.
> > Since it seems like this issue has been unreported and I couldn't find
> > anything similar to it, and since what Claude reported seems potentially
> > plausible, I'm putting it here.
> > If AI assisted/generated bug reports are unwanted, I apologize.
>
> At least as far as I'm concerned, in term of bugs reports, whatever
> makes sense and is usable is welcome.  This report has a few dubious
> aspects (see below) but it might still be valuable, I think.
>
> > Original report
> > --------------------------
> > I think a peer address can get stuck in pasta's low-RTT destination
> > table and then lose window clamping for good.
>
> This part wasn't clearly demonstrated.
>
> > That caused every new
> > TCP connection from our LAN router's address to stall on RTOs until
> > pasta was restarted. Details below.
>
> This looks plausible and somewhat demonstrated.
>
> > I haven't found an existing report on Bugzilla or this list.
>
> Correct, no known issues in this sense so far.
>
> > Version: pasta 2026_10_02.cba3570-3-g8ba4132 (the logic below is
> >          unchanged in current master)
> > Setup:   rootless Docker 29.6.1 via rootlesskit, --net=pasta,
> >          --port-driver=implicit, --mtu=65520 (pasta --tcp-ports=auto
> >          --udp-ports=auto --no-map-gw --ipv4-only ...)
> > Kernel:  6.18.57 (Arch linux-lts), x86_64
> >
> > What we think is happening
> > --------------------------
> > We suspect the low-RTT destination table (low_rtt_dst[],
> > tcp_rtt_dst_check() / tcp_rtt_dst_low()). Our LAN router's address,
> > 10.0.0.1, ended up in it even though the router is several
> > milliseconds away.
>
> Nothing clearly confirms the "ended up in it" part (yet).
>
> > [...]
> >
> > We could not read the table directly (no ptrace into pasta), so
> > "10.0.0.1 is in the table" is inferred from this behaviour and the
> > code path.
>
> ...I guess the problem here is training a language model about the
> difference between "inferred" and "suspected". :)
>
> One could ptrace() as root by the way, but that wouldn't help either.
>
> Could you make sure about this with a print in tcp_rtt_dst_check()?  If
> that's not the case, you might need to further trace what's going on in
> tcp_update_seqack_wnd() instead.
>
> Rebuilding and using a modified version of passt is quick (it just
> depends on a C library):
>
>   git clone git://passt.top/passt && cd passt
>   [edit as needed]
>   make && sudo make install # Or make install as root
>
> Both Podman and Docker / rootlesskit will pick up the local build under
> /usr/local/bin. You can cleanly remove it with 'make uninstall' as root
> once you're done.
>
> > Why we think 10.0.0.1 got into the table
> > ----------------------------------------
> > A real RTT of 10 us or less to the router is impossible here
> > (measured RTT 2-6 ms). Our suspicion comes from reading the kernel
> > source; we have not caught it on a live socket:
> >
> > - tcp_rtt_dst_check() tests
> >   (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD.
> > - The kernel initialises rtt_min to ~0U, both in tcp_init_sock()
> >   (net/ipv4/tcp.c) and for accepted sockets in
> >   tcp_create_openreq_child() (tcp_minisocks.c). tcpi_min_rtt stays
> >   ~0U until the first valid RTT sample.
> > - tcp_synack_rtt_meas() takes no sample when the SYN-ACK was
> >   retransmitted (rtt_us = -1 unless !req->num_retrans). The same
> >   applies on the active side when the SYN was retransmitted (Karn).
> > - Cast to int, ~0U becomes -1, which passes the "<= 10 us" test.
> > - pasta runs this check in tcp_prepare_flags() when it sends the SYN
> >   (inbound) or SYN-ACK (outbound) to the guest. That is right after
> >   the host-side handshake, before any data ACK could provide a
> >   sample.
>
> ...maybe, yes.
>
> > We measured exactly such handshake retransmissions on this
> > Wi-Fi/NAT path: 6 of 1,800 fresh connections took ~1.05 s, i.e. one
> > 1 s retry. With hundreds of inbound and outbound connections to
> > 10.0.0.1 per day, one retried handshake would be enough to insert it
> > permanently.
> >
> > The night the stalls became severe, the published container was also
> > restarted about ten times while clients kept connecting, which made
> > unusual handshakes more likely. We cannot tell exactly when the
> > insertion happened.
> >
> >
> > Suggested fixes
> > ---------------
> > 1. In tcp_rtt_dst_check(), treat tcpi_min_rtt == ~0U (and 0) as
> >    "no sample" and skip the insertion. Compare as unsigned, or check
> >    tcpi_rtt / tcpi_rcv_rtt as well.
>
> I'm not sure if we actually risk hitting 0: strictly speaking that
> would mean less than 1 µs which looks unlikely (even for simple veth
> pairs or local guests), but in practice there might be rounding errors
> somewhere.
>
> Do we need to skip 0? From a quick glance of the kernel paths involved
> I'd say we don't.
>
> Could you test this perhaps?  Not build tested but I think you could
> modify the initial early return condition to:
>
>         if (!min_rtt_cap ||
>             (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD ||
>             tinfo->tcpi_min_rtt == ~0U /* not measured yet */)
>                 return;
>
> > 2. Don't trust a single sample forever: give entries a lifetime, or
> >    evict an address when a later connection to it shows a min RTT
> >    above the threshold.
>
> Reasonable.  I guess we could add a timestamp and check if the entry
> expired (minutes?) where we update the RTT in
> tcp_update_seqack_wnd()... or even in tcp_rtt_dst_check().  No strong
> preference on my side, I didn't really think it through.
>
> > 3. Log (at debug level) when an address enters the table. It was
> >    invisible here; the only visible signs were retransmissions
> >    inside the namespace.
>
> Yeah, that would be good.  I added this feature before we started
> logging anything, I think.  Feel free to send a patch for that.
>
> > Workaround: restarting pasta clears the table.
> >
> > Happy to test a patch or collect more data.
>
> Let me know if you can check those few things I suggested, thanks.
>
> --
> Stefano
>

[-- Attachment #2: 0001-tcp-Don-t-add-peers-to-low-RTT-table-before-RTT-is-m.patch --]
[-- Type: application/octet-stream, Size: 2081 bytes --]

From 172138c0b005fe92e8f0220b38912685b63a5882 Mon Sep 17 00:00:00 2001
From: Pietro De Domenico <lynxes.exe@gmail.com>
Date: Thu, 12 Nov 2026 16:56:44 +0100
Subject: [PATCH 1/2] tcp: Don't add peers to low RTT table before RTT is
 measured

The kernel initialises the minimum RTT of a socket to ~0U (see
tcp_init_sock() and tcp_create_openreq_child()), and tcpi_min_rtt keeps
that value until the first RTT sample.

For a passively opened socket, the handshake gives no sample if the
SYN-ACK was retransmitted (Karn's algorithm, see tcp_synack_rtt_meas()),
and the only fallback is the timestamp echo, which isn't there if the
peer didn't negotiate TCP timestamps. Windows clients don't, by default.

tcp_rtt_dst_check() casts tcpi_min_rtt to int, so ~0U becomes -1, the
peer passes the threshold check, and it's added to the low RTT table,
which it never leaves: from then on, every connection with that peer
advertises an unclamped window to the guest and stalls on
retransmissions.

Skip the check while the minimum RTT hasn't been measured yet.

Reproduced on Linux 6.18.57 by dropping SYN-ACKs at a client with
net.ipv4.tcp_timestamps set to 0, for two seconds: tcpi_min_rtt of the
accepted socket is 0xffffffff, with tcpi_total_retrans 3, and pasta
would have added the client to the table.

Link: https://archives.passt.top/passt-dev/CAJPNJUuUYSkW776P0hKmweHnKOqXtsXqwm4HXP-+jorXrSfFFg@mail.gmail.com/
Suggested-by: Stefano Brivio <sbrivio@redhat.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pietro De Domenico <lynxes.exe@gmail.com>
---
 tcp.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tcp.c b/tcp.c
index 95f4f3e..d88ebd4 100644
--- a/tcp.c
+++ b/tcp.c
@@ -758,7 +758,8 @@ static void tcp_rtt_dst_check(const struct tcp_tap_conn *conn,
 	int i, hole = -1;
 
 	if (!min_rtt_cap ||
-	    (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD)
+	    (int)tinfo->tcpi_min_rtt > LOW_RTT_THRESHOLD ||
+	    tinfo->tcpi_min_rtt == ~0U /* not measured yet */)
 		return;
 
 	for (i = 0; i < LOW_RTT_TABLE_SIZE; i++) {
-- 
2.56.0


[-- Attachment #3: 0002-tcp-Log-additions-to-low-RTT-table.patch --]
[-- Type: application/octet-stream, Size: 1387 bytes --]

From 3f71b8ba1255e83c5702d4769230d5007c828a3d Mon Sep 17 00:00:00 2001
From: Pietro De Domenico <lynxes.exe@gmail.com>
Date: Thu, 12 Nov 2026 16:56:44 +0100
Subject: [PATCH 2/2] tcp: Log additions to low RTT table

Entries in the low RTT table never expire, and a wrong one makes every
connection with that peer stall, but nothing shows that a peer was
added: the only visible symptom is retransmissions inside the
namespace.

Print a debug message with the address and the measured minimum RTT
whenever a peer is added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Pietro De Domenico <lynxes.exe@gmail.com>
---
 tcp.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/tcp.c b/tcp.c
index d88ebd4..4aef854 100644
--- a/tcp.c
+++ b/tcp.c
@@ -755,6 +755,7 @@ static void tcp_rtt_dst_check(const struct tcp_tap_conn *conn,
 			      const struct tcp_info_linux *tinfo)
 {
 	const struct flowside *tapside = TAPFLOW(conn);
+	char buf[INANY_ADDRSTRLEN];
 	int i, hole = -1;
 
 	if (!min_rtt_cap ||
@@ -775,6 +776,10 @@ static void tcp_rtt_dst_check(const struct tcp_tap_conn *conn,
 	if (hole == -1)
 		return;
 
+	flow_dbg(conn, "Adding %s to low RTT table, minimum RTT: %u us",
+		 inany_ntop(&tapside->oaddr, buf, sizeof(buf)),
+		 tinfo->tcpi_min_rtt);
+
 	low_rtt_dst[hole++] = tapside->oaddr;
 	if (hole == LOW_RTT_TABLE_SIZE)
 		hole = 0;
-- 
2.56.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-12 16:09 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2026-10-28 13:15 tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs Lynxes Exe
2026-10-29  1:11 ` David Gibson
2026-11-04 13:53 ` Stefano Brivio
2026-11-12 16:09   ` Lynxes Exe

Code repositories for project(s) associated with this public inbox

	https://passt.top/passt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).