From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=fail (p=quarantine dis=none) header.from=redhat.com Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id A96AD5A061B for ; Tue, 17 Nov 2026 12:04:34 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1794913473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dVu6AK/mziaUXFvvuuCGpW9QrXzRl1UoyH5mUyNwZvk=; b=KqzawjGJUP0DY3khxmeDGKjeVmTGsaKSYXMlC4RBPhlrzpPQnSgAO1gb9tej7HE7EdLssi p9FLzx/a0Iwt1GqXI6YHUdFWeZQfnoWSXjhwFTYnVYr1mEB6HaZYJZ1kdHTVTjDUhBRjGp 3NUvRUvI0I+cUPuriVv3DAsTJOpAm90= X-MC-Rep: binary re single Received: from mail-wm1-f70.google.com (mail-wm1-f70.google.com [209.85.128.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-13-7n9KR0rePvKlOJomchASlw-1; Tue, 17 Nov 2026 06:04:32 -0500 X-MC-Unique: 7n9KR0rePvKlOJomchASlw-1 X-Mimecast-MFC-AGG-ID: 7n9KR0rePvKlOJomchASlw_1794913471 Received: by mail-wm1-f70.google.com with SMTP id 5b1f17b1804b1-4bb69d84053so17618975e9.0 for ; Tue, 17 Nov 2026 03:04:31 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1794913471; x=1795518271; h=date:content-transfer-encoding:content-type:mime-version :organization:references:in-reply-to:message-id:subject:cc:to:from :x-gm-gg:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to:content-type; bh=dVu6AK/mziaUXFvvuuCGpW9QrXzRl1UoyH5mUyNwZvk=; b=ar+EvX4QThMAvBRkFN/O5GT8Iebzvk+ALp5ovBqzUgq3JFZlW0p++9GyyZeVZT5NKi DRWzU8Ni+PGwh0fHDXF0CA61mOKNkX0ibGZrDceITKyhKBMD/Gb96505Ogs0aPczOXpU 8I4+8s7qD6yxzYBKM88MDJ4+YQHJVM39kCWxzpdwVlcj+PFxGf4f3tGwpo5isi3R4ZUU YWOyK28alc7ch61fQKw+bTjUNfl0b6B+0QRCbebDoblA5OTcuSjumxkTQX9rWsRYBqV1 oSmIWZEU6wIv4IuBDJdmLa6aUPNL18jBJSgt7hPdCn0xosv+8yCt3jsBR6mo0iOg55iq OrLA== X-Gm-Message-State: APuU4Z/pgCdgkMJe8P2lxtziANt5QO7fPzpR7Q6MH/vjTduyZ1aV21/B ZF2YqXpgTrxwf1MUkZJG3eveFAjJeGuSaRrxudigmSlHqCBO/PKixIiNnixsc5fy0nz+/0Y23Yv NIloFrCDLVYBzQ40e+D/UsZLsY3kiTyy7337+wZX8L5l29lr+SO6FonfB8LipWw== X-Gm-Gg: AYBFou2iLTka7LNELDu36RR1cNPokxOWlN39UIrG3w2PM7FCM0ZqLifDSYRZg/9RHzO zoyb7bzdnpoCxeNmlERzQLaokec7anm7RcjYYVIEtcXtxzJ9wZf7wriVVCLd8DUDQz6HzuTeBSz voTxfPBLon6Q4dK/ZErWGPGKIgQ0yrFsPRFLwfZcbFc0o5CRNfaLYQKfbJTLjKwM42xAHjg9Gwx 78y8xsOziKodZ7K3TvVNptcGM/JlQ3LqSs3uB8H5pL/+3WY0vVyeowzP1rvz9QzOx7HSLtXrsB4 0837SrO6t8s26oEkC2q7t8h2hTBd8lIBFdYGNtPTtDmVgkBgTGv0rVPw8s+P+n+r+/hEYynf1/m BjVEP7bBFTC885ecFWqzpNwf0YHy52VS2Ey0arcrMeiRVmvIRzM10HyUnPxVdv0Mwf6E= X-Received: by 2002:a05:600c:19d1:b0:4bb:7571:5743 with SMTP id 5b1f17b1804b1-4bb757157a7mr96714295e9.27.1794913470535; Tue, 17 Nov 2026 03:04:30 -0800 (PST) X-Received: by 2002:a05:600c:19d1:b0:4bb:7571:5743 with SMTP id 5b1f17b1804b1-4bb757157a7mr96713705e9.27.1794913469867; Tue, 17 Nov 2026 03:04:29 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4bb78430752sm62440685e9.12.2026.11.17.03.04.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 17 Nov 2026 03:04:29 -0800 (PST) From: Stefano Brivio To: Lynxes Exe Subject: Re: tcp: peer stuck in low_rtt_dst table loses window clamping, every new connection stalls on RTOs Message-ID: <20261117120427.776ab215@elisabeth> In-Reply-To: References: <20261104145322.2dc73741@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 Date: Tue, 17 Nov 2026 12:04:28 +0100 (CET) X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: sFa5GrTYSPddTA6CU9lCL8xzXQRcsm2WwIOpuMxC2_4_1794913471 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: YSJBBRAVSQHIAGSFF5QT5KWYS2NV76DS X-Message-ID-Hash: YSJBBRAVSQHIAGSFF5QT5KWYS2NV76DS X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 12 Nov 2026 17:09:20 +0100 Lynxes Exe wrote: > Hi Stefano, > > I apologize for the long wait. The issue took a while to represent itself. No worries, it wasn't that long, thanks for following up. > [...] > > 3. Patches > > Attached, against current master (7380393): > > 0001 tcp: Don't add peers to low RTT table before RTT is measured > 0002 tcp: Log additions to low RTT table > > 0001 is your condition as you wrote it. 0002 logs only additions, with > flow_dbg(): the skip itself happens on every tcp_prepare_flags() call > until the first sample, which was 28 times for one connection in the > test above, so logging it would be noisy. Each patch builds without > warnings; I haven't run the test suite. Thanks for the patches, and welcome to the git log! I was about to merge both, but there's an issue with the first one, so I could just merge 2/2 for the moment. I'm fairly confident that 1/2 is actually correct, but it makes the migrate/rampstream_out test fail. The migration functionality gives a hypervisor (currently this is only supported with QEMU) a way to "move" all our context, including TCP connections (!), between a source guest and a target guest, so that virtual machines can be moved to another host as they are, with a short interruption. The stages_v2[] array in migrate.c gives an overview of what data can move to another instance of passt, and how. That test checks that an outbound stream sent by a guest remains consistent as it's moved from a source instance to a target instance of passt. With 1/2, the flow appears to stop mid way, or to not restart on the destination. We don't copy the "low RTT" table during migration. I didn't think it was needed, because, I thought, eventually the destination instance of passt should figure things out. But it looks like it doesn't, and that bug with the initial ~0U value hid the issue until now, that is, (some?) migrated connections are _probably_ inserted by default in the table. This is just a wild guess for the moment. Now, this isn't really relevant for real-world operation, because any useful TCP endpoint, in case there's a migration between hosts, is in general non-local. But it breaks tests and that's problematic per se, simply because we need a way to test this. I can try to look into it in a few days (but it might take longer). Maybe the solution is to copy the table (define a new version of the migration protocol, which would be used after 1/2, which includes the table information). Maybe there's another issue with congestion window parameters that we don't properly transfer to the target instance. If you want to have a look meanwhile, you can just execute that test (it needs a few dependencies, see test/README.md) by commenting out (or creating nested functions like skip() { ... }) other tests in test/run, say: diff --git a/test/run b/test/run index 14ddbbf..1881e4d 100755 --- a/test/run +++ b/test/run @@ -72,4 +72,5 @@ run() { [ ${CI} -eq 1 ] && video_start ci +skip1() { exeter smoke/smoke.sh exeter build/build.py @@ -168,8 +170,13 @@ run() { test migrate/rampstream_in teardown migrate +} + VHOST_USER=1 + VALGRIND=1 + setup migrate test migrate/rampstream_out teardown migrate +skip2() { VALGRIND=0 VHOST_USER=0 @@ -192,4 +199,5 @@ run() { test passt_vu_in_ns/shutdown teardown passt_in_ns +} # TODO: Make those faster by at least pre-installing gcc and make on By the way, for the future, it would be nice if you could submit them using git send-email, not as attachments, for ease of review. See also CONTRIBUTING.md and: https://git-scm.com/docs/git-send-email#_examples_of_smtp_servers in this case, as we already discussed your patches and they were fairly trivial, I took the liberty to apply the second one anyway. > I didn't do the expiry part. I'm happy to try it if you have a > preference between tcp_rtt_dst_check() and tcp_update_seqack_wnd(). Maybe tcp_rtt_dst_check() would make more sense as it's called once per connection, plus every time we want to send a flag (ACK, typically) without data, so it's called just before we need the information, but not too frequently. Regardless of the current fix, an expiration mechanism might make sense in general because hosts might actually move. Given that we fetch struct tcp_info from time to time, I start wondering if there could be a simpler mechanism to take care of that. On the other hand we don't want to mark a destination as non-local because of some temporary condition (say, memory pressure, or CPU load), so we shouldn't push a destination out of the table just because the RTT was high for a couple of packets at the beginning of a connection, or maybe even a few seconds. With that regard, the expiration mechanism is probably more robust. But perhaps there's some other way to have it equally robust and a bit more convenient, I'm not sure. -- Stefano