From: Stefano Brivio <sbrivio@redhat.com>
To: Lynxes Exe <lynxes.exe@gmail.com>
Cc: passt-dev@passt.top
Subject: Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs
Date: Tue, 17 Nov 2026 12:04:28 +0100 (CET) [thread overview]
Message-ID: <20261117120427.776ab215@elisabeth> (raw)
In-Reply-To: <CAJPNJUsp2MNfxBTuiO+WwtXN-QN-XMz+EBe6_54dB0SSidbWBg@mail.gmail.com>
On Thu, 12 Nov 2026 17:09:20 +0100
Lynxes Exe <lynxes.exe@gmail.com> wrote:
> Hi Stefano,
>
> I apologize for the long wait. The issue took a while to represent itself.
No worries, it wasn't that long, thanks for following up.
> [...]
>
> 3. Patches
>
> Attached, against current master (7380393):
>
> 0001 tcp: Don't add peers to low RTT table before RTT is measured
> 0002 tcp: Log additions to low RTT table
>
> 0001 is your condition as you wrote it. 0002 logs only additions, with
> flow_dbg(): the skip itself happens on every tcp_prepare_flags() call
> until the first sample, which was 28 times for one connection in the
> test above, so logging it would be noisy. Each patch builds without
> warnings; I haven't run the test suite.
Thanks for the patches, and welcome to the git log!
I was about to merge both, but there's an issue with the first one, so
I could just merge 2/2 for the moment. I'm fairly confident that 1/2
is actually correct, but it makes the migrate/rampstream_out test fail.
The migration functionality gives a hypervisor (currently this is only
supported with QEMU) a way to "move" all our context, including TCP
connections (!), between a source guest and a target guest, so that
virtual machines can be moved to another host as they are, with a short
interruption. The stages_v2[] array in migrate.c gives an overview of
what data can move to another instance of passt, and how.
That test checks that an outbound stream sent by a guest remains
consistent as it's moved from a source instance to a target instance of
passt. With 1/2, the flow appears to stop mid way, or to not restart
on the destination.
We don't copy the "low RTT" table during migration. I didn't think it
was needed, because, I thought, eventually the destination instance of
passt should figure things out. But it looks like it doesn't, and that
bug with the initial ~0U value hid the issue until now, that is,
(some?) migrated connections are _probably_ inserted by default in the
table. This is just a wild guess for the moment.
Now, this isn't really relevant for real-world operation, because any
useful TCP endpoint, in case there's a migration between hosts, is in
general non-local. But it breaks tests and that's problematic per se,
simply because we need a way to test this.
I can try to look into it in a few days (but it might take longer).
Maybe the solution is to copy the table (define a new version of the
migration protocol, which would be used after 1/2, which includes the
table information). Maybe there's another issue with congestion window
parameters that we don't properly transfer to the target instance.
If you want to have a look meanwhile, you can just execute that test (it
needs a few dependencies, see test/README.md) by commenting out (or
creating nested functions like skip() { ... }) other tests in test/run,
say:
diff --git a/test/run b/test/run
index 14ddbbf..1881e4d 100755
--- a/test/run
+++ b/test/run
@@ -72,4 +72,5 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
+skip1() {
exeter smoke/smoke.sh
exeter build/build.py
@@ -168,8 +170,13 @@ run() {
test migrate/rampstream_in
teardown migrate
+}
+ VHOST_USER=1
+ VALGRIND=1
+
setup migrate
test migrate/rampstream_out
teardown migrate
+skip2() {
VALGRIND=0
VHOST_USER=0
@@ -192,4 +199,5 @@ run() {
test passt_vu_in_ns/shutdown
teardown passt_in_ns
+}
# TODO: Make those faster by at least pre-installing gcc and make on
By the way, for the future, it would be nice if you could submit them
using git send-email, not as attachments, for ease of review. See also
CONTRIBUTING.md and:
https://git-scm.com/docs/git-send-email#_examples_of_smtp_servers
in this case, as we already discussed your patches and they were fairly
trivial, I took the liberty to apply the second one anyway.
> I didn't do the expiry part. I'm happy to try it if you have a
> preference between tcp_rtt_dst_check() and tcp_update_seqack_wnd().
Maybe tcp_rtt_dst_check() would make more sense as it's called once per
connection, plus every time we want to send a flag (ACK, typically)
without data, so it's called just before we need the information, but
not too frequently.
Regardless of the current fix, an expiration mechanism might make sense
in general because hosts might actually move. Given that we fetch
struct tcp_info from time to time, I start wondering if there could be
a simpler mechanism to take care of that.
On the other hand we don't want to mark a destination as non-local
because of some temporary condition (say, memory pressure, or CPU
load), so we shouldn't push a destination out of the table just because
the RTT was high for a couple of packets at the beginning of a
connection, or maybe even a few seconds.
With that regard, the expiration mechanism is probably more robust. But
perhaps there's some other way to have it equally robust and a bit more
convenient, I'm not sure.
--
Stefano
prev parent reply other threads:[~2026-11-17 11:04 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-10-28 13:15 Lynxes Exe
2026-10-29 1:11 ` David Gibson
2026-11-04 13:53 ` Stefano Brivio
2026-11-12 16:09 ` Lynxes Exe
2026-11-17 11:04 ` Stefano Brivio [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117120427.776ab215@elisabeth \
--to=sbrivio@redhat.com \
--cc=lynxes.exe@gmail.com \
--cc=passt-dev@passt.top \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://passt.top/passt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).