public inbox for passt-dev@passt.top
 help / color / mirror / code / Atom feed
* Migration failure across bridge
@ 2025-03-18  5:21 David Gibson
  2025-03-18  8:28 ` Stefano Brivio
  0 siblings, 1 reply; 2+ messages in thread
From: David Gibson @ 2025-03-18  5:21 UTC (permalink / raw)
  To: Stefano Brivio; +Cc: passt-dev

[-- Attachment #1: Type: text/plain, Size: 1292 bytes --]

Continued investigating the problem with migration failing across a
bridge.

Good news is I've found the problem... or at least one problem.  Bad
news is we'll have to change the migration stream format to fix it.

The packets are being dropped in tcp_validate_incoming() due to a
failed PAWS check (skb drop reason "TCP_RFC7323_PAWS").  That in turn
looks to be because we don't preserve TCP timestamp state across the
migration.  We preserve _whether_ TCP timestamps are active on the
connection (TCPOPT_TIMESTAMP entry in TCP_REPAIR_OPTIONS), but we
don't preserve the current timestamp values (TCP_TIMESTAMP socket
option).  The equivalent CRIU code is

https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa3575691977eb21753/soccr/soccr.c#L266

and

https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa3575691977eb21753/soccr/soccr.c#L572

I'll work on writing a fix tomorrow.

Not yet sure why we didn't hit this with a local migration.  I'm
guessing some part of being a local connection means we're bypassing
the PAWS check.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: Migration failure across bridge
  2025-03-18  5:21 Migration failure across bridge David Gibson
@ 2025-03-18  8:28 ` Stefano Brivio
  0 siblings, 0 replies; 2+ messages in thread
From: Stefano Brivio @ 2025-03-18  8:28 UTC (permalink / raw)
  To: David Gibson; +Cc: passt-dev

On Tue, 18 Mar 2025 16:21:58 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> Continued investigating the problem with migration failing across a
> bridge.
> 
> Good news is I've found the problem... or at least one problem.

\o/

> Bad
> news is we'll have to change the migration stream format to fix it.

Whoops, sorry, my bad. And now, RFC 7323, section 3.2, contrary to RFC
1323 (also section 3.2), requires that we keep sending timestamps if we
negotiated them:

   Once TSopt has been successfully negotiated, that is both <SYN> and
   <SYN,ACK> contain TSopt, the TSopt MUST be sent in every non-<RST>
   segment for the duration of the connection

...so we can't just disable them for migrated flows.

Strictly speaking, I don't think it's necessary to define a new version
of the format, because I'm really really sure nobody is using this yet,
other than for tests.

If you want to use this as a chance to play with/test a version bump,
we can do it. My preference would be to keep this as v1 anyway for the
moment, regardless of the *non*-breakage, for simplicity. That is,
whoops, migration is broken on 2025_02_17.a1e48a0.

> The packets are being dropped in tcp_validate_incoming() due to a
> failed PAWS check (skb drop reason "TCP_RFC7323_PAWS").  That in turn
> looks to be because we don't preserve TCP timestamp state across the
> migration.  We preserve _whether_ TCP timestamps are active on the
> connection (TCPOPT_TIMESTAMP entry in TCP_REPAIR_OPTIONS), but we
> don't preserve the current timestamp values (TCP_TIMESTAMP socket
> option).  The equivalent CRIU code is
> 
> https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa3575691977eb21753/soccr/soccr.c#L266
> 
> and
> 
> https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa3575691977eb21753/soccr/soccr.c#L572
> 
> I'll work on writing a fix tomorrow.
> 
> Not yet sure why we didn't hit this with a local migration.  I'm
> guessing some part of being a local connection means we're bypassing
> the PAWS check.

The TCP_TIMESTAMP option is documented... not where it should be
documented, grr:

  https://criu.org/index.php?title=TCP_connection#Timestamp

and I _guess_ that two guests using kvm-clock as clock source might
actually have the same jiffies, and from this description, same
jiffies, same timestamps.

Perhaps in your nested case not all guests are using kvm-clock, or
there's something else to it.

-- 
Stefano


^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2025-03-18  8:28 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-03-18  5:21 Migration failure across bridge David Gibson
2025-03-18  8:28 ` Stefano Brivio

Code repositories for project(s) associated with this public inbox

	https://passt.top/passt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).