From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202502 header.b=cXCICudO; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 48DC05A0274 for ; Wed, 19 Mar 2025 06:22:44 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202502; t=1742361761; bh=Si9Fn26wSZagFcWlxvrPmBvrrpcMsFRI47MzCHj9U0Y=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=cXCICudODxNqe2ziC84wBscTerDOJ9mzrqRTToEk/i6tjBiUH+3R1G+CQPOdLKrYE InJ7cI/8P9DSWYjFDxAcvdLpqQNzd+mfUL37o6goIp9hNKk5nInK4FBCHxsPOFclSP ZKH5vKjdW2vCcqPSmaAM/RgHUMPTUCmf2G9aedfH8+cKyFEkFKKMD/+AlqvYZNqooQ PB2oUAV5Yf+carbddBY8XPFHOncXpy1HJo6Bp3eT8WWUcrBzAdSXES/KEnvpZGo515 SK/TsLRAjuv7p3lmVjdPJp/KPnG0yYh5hSJCt6bvN6oEFiOiuM4S19fd0wtzXxmf75 tDe1MIU0RFw8w== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4ZHcXx6MrSz4x2g; Wed, 19 Mar 2025 16:22:41 +1100 (AEDT) Date: Wed, 19 Mar 2025 16:22:32 +1100 From: David Gibson To: Stefano Brivio Subject: Re: Migration failure across bridge Message-ID: References: <20250318092823.0bca8887@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="N4ZROIBa4gXNxccT" Content-Disposition: inline In-Reply-To: <20250318092823.0bca8887@elisabeth> Message-ID-Hash: O7NSOJVDYI3AUWRSKWJ6RVLRORCHTWZK X-Message-ID-Hash: O7NSOJVDYI3AUWRSKWJ6RVLRORCHTWZK X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --N4ZROIBa4gXNxccT Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 18, 2025 at 09:28:23AM +0100, Stefano Brivio wrote: > On Tue, 18 Mar 2025 16:21:58 +1100 > David Gibson wrote: >=20 > > Continued investigating the problem with migration failing across a > > bridge. > >=20 > > Good news is I've found the problem... or at least one problem. >=20 > \o/ >=20 > > Bad > > news is we'll have to change the migration stream format to fix it. >=20 > Whoops, sorry, my bad. And now, RFC 7323, section 3.2, contrary to RFC > 1323 (also section 3.2), requires that we keep sending timestamps if we > negotiated them: >=20 > Once TSopt has been successfully negotiated, that is both and > contain TSopt, the TSopt MUST be sent in every non- > segment for the duration of the connection >=20 > ...so we can't just disable them for migrated flows. Right. I hadn't dug up that section, but I was kind of assuming that was the case. > Strictly speaking, I don't think it's necessary to define a new version > of the format, because I'm really really sure nobody is using this yet, > other than for tests. > If you want to use this as a chance to play with/test a version bump, > we can do it. My preference would be to keep this as v1 anyway for the > moment, regardless of the *non*-breakage, for simplicity. That is, > whoops, migration is broken on 2025_02_17.a1e48a0. Right. I did decide to bump the version, my reasoning is explained in the commit message, but I didn't attempt to maintain backwards compatibility with v1 for this reason. > > The packets are being dropped in tcp_validate_incoming() due to a > > failed PAWS check (skb drop reason "TCP_RFC7323_PAWS"). That in turn > > looks to be because we don't preserve TCP timestamp state across the > > migration. We preserve _whether_ TCP timestamps are active on the > > connection (TCPOPT_TIMESTAMP entry in TCP_REPAIR_OPTIONS), but we > > don't preserve the current timestamp values (TCP_TIMESTAMP socket > > option). The equivalent CRIU code is > >=20 > > https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa= 3575691977eb21753/soccr/soccr.c#L266 > >=20 > > and > >=20 > > https://github.com/checkpoint-restore/criu/blob/d18912fc88f3dc7bde5fdfa= 3575691977eb21753/soccr/soccr.c#L572 > >=20 > > I'll work on writing a fix tomorrow. > >=20 > > Not yet sure why we didn't hit this with a local migration. I'm > > guessing some part of being a local connection means we're bypassing > > the PAWS check. >=20 > The TCP_TIMESTAMP option is documented... not where it should be > documented, grr: Yeah. It's not in tcp(7), unfortunately. > https://criu.org/index.php?title=3DTCP_connection#Timestamp >=20 > and I _guess_ that two guests using kvm-clock as clock source might > actually have the same jiffies, and from this description, same > jiffies, same timestamps. > Perhaps in your nested case not all guests are using kvm-clock, or > there's something else to it. So... the L2 clock is irrelevant as this problem is on the socket side. In my test I'm using namespaces, not VMs for the L1 so they'd be even more likely to have the same jiffies, but it still breaks. That makes sense, because with net.ipv4.tcp_timestamps=3D1 (which is the default on both my Fedora and Debian system) there's supposed to be a random offset for every connection (that *is* documented in tcp(7)). What I don't understand is why it _doesn't_ fail when going fully local to local. I wondered if it might be that timestamps were disabled by default for local connections, but that doesn't appear to be the case. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --N4ZROIBa4gXNxccT Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmfaVJQACgkQzQJF27ox 2GdV4RAApCbBhFTfVS1TNn9/Fppbhdh1LAPVP6iJJxgt5+Hq7XmXQKtoXoSVyRBD pKeMuDL4l0Md8TGNUtk3xOxnx0yfDeq9cg33HchYfWN683rpvG0wJI1w9xcI+j34 xHoyEIXCEiHGc0WzXctlrgE78QAKMbjZ3Nz3pKMCn6zmV3xmnXElinpVxkDchnEY uVMvVml7Q4MGUSfxaxHrx6MOHtulsDefU/PDuOCr0rQvaSIHow99ooVaf4RgGCdZ VKVuItLMSVjvcBC6aqrNO+hDT6hGayTePa00+MhvMrQZkMGpsgJ43AhJNu+giHID syV7M7kS1aiqOGGtDL3l2WOI+79UTHiPe8NTAIpL2ZXdbIqcFNmXGlysvbHecEwc veaYHA4RGqtSwDQ08DboEfG73F6nMavXl8QwUI8dwbNP8diUOP9XdGWgQwlnrMLC IKcow6N4cuK+8XOVX33M217IGoj5hKdzKZj9d0fHPYeTtiPx9wvtBw154p+VwkfA ePqpewZsNZjsHOm14Q4X6is18O6lOLPHMr/tinX0jSNqa/hJ8GsfUhJyDcM22Hc/ 6CJ98dVRMWooNXIlv9M7JOVib/LIJG6/gSvQup+lDCoZG/hK2Hplg9a3ZGOSU1kK MAZApuZpBaHXnJKT3GHqPks0iR7gPnk3p12vu5f3VF+qK0obU5k= =AXS5 -----END PGP SIGNATURE----- --N4ZROIBa4gXNxccT--