From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202602 header.b=DwlM0phw; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 947395A0262 for ; Fri, 22 May 2026 08:15:18 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202602; t=1779430515; bh=RyWgPBjQ/ZNhUwfPk51lV6TnGyYZJiM1CGrvRtkkmO4=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=DwlM0phw56D04K+xkOilSxlgW0a4y4YQ4OCcJd+U6z/SBbQMijIJ08dQx9Muz6Hlg 7I6p8eav3yt8LqbIwxUOaQKXSx+6t2qA8QfSuNxry6tfSalFTg9BMV+0NFK7kevT+v FrHN9WzzDvsa5LXqDUCkNKylzkIf7PmL2qkVFaOFXTNVNhiCRRInfc7MwlqVJJISy3 /w0cUw/A51gTGFG11CE+/rz2R9cG8BNrK2Gl4A3j2UInvFkhhzbzWxhItZJLsG/gnC p0sB/Jt7fG81Qt6XLXgX2e8zXAZVSuZuOEJx6J7RAvq06kMZamVy4ODsbbUqugqk96 bWTzLTS+BMvzw== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4gMFPb0dH1z4wTN; Fri, 22 May 2026 16:15:15 +1000 (AEST) Date: Fri, 22 May 2026 16:15:08 +1000 From: David GIbson To: Stefano Brivio Subject: Re: [PATCH v4 00/10] vhost-user: Preparatory series for multiple iovec entries per virtqueue element Message-ID: References: <20260513115218.1662850-1-lvivier@redhat.com> <20260520173445.0658dfef@elisabeth> <20260520180708.275ec4de@elisabeth> <20260520181852.1f0119ff@elisabeth> <20260520225340.54490a21@elisabeth> <50d79312-0493-4af0-b0bc-7c590885cbd2@redhat.com> <20260522062239.4fcd3314@elisabeth> <20260522074455.15e6cc3e@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nVJfHD17sYl4FUIt" Content-Disposition: inline In-Reply-To: <20260522074455.15e6cc3e@elisabeth> Message-ID-Hash: COIXKTFQCW7N7AIX3OJLHJXQLMKUBO22 X-Message-ID-Hash: COIXKTFQCW7N7AIX3OJLHJXQLMKUBO22 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Laurent Vivier , passt-dev@passt.top, Jon Maloy X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --nVJfHD17sYl4FUIt Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, May 22, 2026 at 07:44:56AM +0200, Stefano Brivio wrote: > On Fri, 22 May 2026 06:22:39 +0200 > Stefano Brivio wrote: >=20 > > On Fri, 22 May 2026 01:13:33 +0200 > > Laurent Vivier wrote: > >=20 > > > On 5/21/26 10:30, Laurent Vivier wrote: =20 > > > > On 5/20/26 22:53, Stefano Brivio wrote: =20 > > > >> On Wed, 20 May 2026 18:18:52 +0200 > > > >> Stefano Brivio wrote: > > > >> =20 > > > >>> On Wed, 20 May 2026 18:07:08 +0200 > > > >>> Stefano Brivio wrote: > > > >>> =20 > > > >>>> On Wed, 20 May 2026 17:34:45 +0200 > > > >>>> Stefano Brivio wrote: =20 > > > >>>>> On Wed, 13 May 2026 13:52:08 +0200 > > > >>>>> Laurent Vivier wrote: =20 > > > >>>>>> Currently, the vhost-user path assumes each virtqueue element = contains > > > >>>>>> exactly one iovec entry covering the entire frame.=A0 This ass= umption > > > >>>>>> breaks as some virtio-net drivers (notably iPXE) provide descr= iptors where the > > > >>>>>> vnet header and the frame payload are in separate buffers, res= ulting in > > > >>>>>> two iovec entries per virtqueue element. > > > >>>>>> > > > >>>>>> This series refactors the vhost-user data path so that frame l= engths, > > > >>>>>> header sizes, and padding are tracked and passed explicitly ra= ther than > > > >>>>>> being derived from iovec sizes.=A0 This decoupling is a prereq= uisite for > > > >>>>>> correctly handling padding of multi-buffer frames. =20 > > > >>>>> > > > >>>>> Sorry to bring (likely) bad news, but this series seems to intr= oduce a > > > >>>>> regression: I got the migration/rampstream_in tests fail twice = in a > > > >>>>> row, which I've never saw happening (I think I saw a single fai= lure a > > > >>>>> long time ago when the machine had a high CPU load, but nothing= else). > > > >>>>> > > > >>>>> I'm currently bisecting and the bisect seems to point towards t= he end > > > >>>>> of the series (probably 10/10), but I haven't finished yet. I'l= l keep > > > >>>>> you posted. I haven't spotted anything that might cause issues = there. =20 > > > >>>> > > > >>>> Yeah, that's the one :( > > > >>>> > > > >>>> $ git bisect bad > > > >>>> db798fc60f4c5869cb53168354e068fb4dabd91a is the first bad commit > > > >>>> commit db798fc60f4c5869cb53168354e068fb4dabd91a > > > >>>> Author: Laurent Vivier > > > >>>> Date:=A0=A0 Wed May 13 13:52:18 2026 +0200 > > > >>>> > > > >>>> =A0=A0=A0=A0 vhost-user: Centralise Ethernet frame padding in vu= _collect() and vu_pad() =20 > > > >=20 > > > > I checked on my system with the commit previous to this series, > > > > bcc3d37a6e01 ("util: Fix changes to assert_with_msg()") and rampstr= eam_in fails too (not=20 > > > > everytime). > > > > =20 > > > > > TCP/IPv4: sequence check, ramps, inbound =20 > > > > ...failed. > > > >=20 > > > > and rampstream_out hangs sometime too. > > > >=20 > > > > I'm going to try with ealier commits. =20 > > >=20 > > > For me the problem can happen with any commit... > > >=20 > > > As it depends on the execution path and on the load and speed of the = system it looks like=20 > > > a race condition. =20 > >=20 > > Hah, thanks for checking. Maybe... > >=20 > > > Did you try to test on a host with a kernel patched with > > > "[PATCH net v2 0/2] Fix race condition between TCP_REPAIR dump and da= ta receive" ? =20 > >=20 > > Now I tried, and yes, the test doesn't hang anymore! I seem to have an > > issue with teardown functions on recent kernels (current net.git HEAD > > more or less): > >=20 > > --- > > + teardown_migrate > > + cat /tmp/passt-tests-VVtLn0/migrate/qemu_1.pid > > + /home/sbrivio/passt/test/nstool exec /tmp/passt-tests-VVtLn0/migrate/= ns1.hold -- kill 16 > > qemu-system-x86_64: terminating on signal 15 from pid 34 () > > + cat /tmp/passt-tests-VVtLn0/migrate/qemu_2.pid > > + /home/sbrivio/passt/test/nstool exec /tmp/passt-tests-VVtLn0/migrate/= ns1.hold -- kill 15 > > 18.8974: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Vhost user me= ssage =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 18.8974: Request: VHOST_USER_GET_VRING_BASE (11) > > 18.8974: Flags: 0x1 > > 18.8974: Size: 8 > > 18.8974: State.index: 0 > > 18.8975: =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Vhost user me= ssage =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > 18.8975: Request: VHOST_USER_GET_VRING_BASE (11) > > 18.8975: Flags: 0x1 > > 18.8975: Size: 8 > > 18.8975: State.index: 1 > > qemu-system-x86_64: terminating on signal 15 from pid 35 () > > 18.7961: Client connection closed > > 18.7962: Closing TCP_REPAIR helper socket > > + context_wait qemu_1 > > + __name=3Dqemu_1 > > + __pidfile=3D/tmp/passt-tests-VVtLn0/migrate/context_qemu_1.pid > > + cat /tmp/passt-tests-VVtLn0/migrate/context_qemu_1.pid > > + rc=3D0 > > + rm /tmp/passt-tests-VVtLn0/migrate/context_passt_repair_2.stdout.9pwp= VbQr /tmp/passt-tests-VVtLn0/migrate/context_passt_repair_2.stderr.dSY5hBu1 > > + __pid=3D67766 > > + rm /tmp/passt-tests-VVtLn0/migrate/context_qemu_1.pid > > + [ 1 -eq 1 ] > > + echo [Exit code: 0] > > + echo -n passt_repair_2$=20 > > + return 0 > > 18.9016: Client connection closed > > 18.9018: Closing TCP_REPAIR helper socket > > + wait 67766 > > + rc=3D0 > > + rm /tmp/passt-tests-VVtLn0/migrate/context_passt_repair_1.stdout.JEyD= GxXe /tmp/passt-tests-VVtLn0/migrate/context_passt_repair_1.stderr.WU550iEI > > + [ 1 -eq 1 ] > > + echo [Exit code: 0] > > + echo -n passt_repair_1$=20 > > + return 0 > > + rc=3D0 > > + rm /tmp/passt-tests-VVtLn0/migrate/context_qemu_2.stdout.Dm8EAhfl /tm= p/passt-tests-VVtLn0/migrate/context_qemu_2.stderr.207qJYPA > > + [ 1 -eq 1 ] > > + echo [Exit code: 0] > > + echo -n qemu_2$=20 > > + return 0 > > 2026/05/22 04:08:23 socat[73089] E connect(5, AF=3D40 cid:94558 port:22= , 16): Connection timed out > > Connection closed by UNKNOWN port 65535 > > ... > > --- > >=20 > > it looks like we stop QEMU a bit too early. But it should be unrelated. > >=20 > > I'm now trying to find some kind of workaround for existing (not fixed) > > kernel versions. Maybe stopping rampstream_in for a moment or something > > like that. >=20 > For some weird reason even very blatant throttling (100 ms - 1 s delays > every 10000 ramps, or an explicit 500 ms pause via signal before > migration) doesn't help. >=20 > So it doesn't seem to be *that* kind of race. I should probably check > the same exact kernel version with fix and without... If it's due to the kernel not stopping the queues on REPAIR, then the only real way to fix the test is to cut off the source machine's network before we trigger migration. That could be done with netfilter (in a user+netns). But probably more natural would be to not do the migration between local passt instances, but actually between two host namespaces, with separate netifs for external connectivity and for the migration. Remove the external netif on the source, then trigger migration, then add the external netif on the destination. It's quite a bit of hassle :(. But it does model something much closer to a real migration scenario. As a bonus it would mean we'd no longer rely on the hack of guessing when to exit the source passt in order to allow the destination passt to bind. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --nVJfHD17sYl4FUIt Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmoP9F8ACgkQzQJF27ox 2GcsihAArJ3iSzHMfghx4M0wHwAEKEVaqGqg3/1TrGp3v8v7VoD4o3CPzcCyjOnz l4t41I3lZup7AsfJeDyCNjlzB2g7JVHS0Lw0+FQQLeus/OcikSRma7xy/eyxsSjZ OI5/m8cYAAIByMQL/XqBvhtGWXAiiA7UrBw7jqYQ6W1gMbP5GAZgQyNV79smdtfp Jz9xoKvf7T8JqAn8mut9axKogisEwqC3DPstuuCpPgFnave1JlILtVUisM1YEnjg bPZUtTREbeveIHn0SiqNdhwT32hEpwbSAzRFDNXlqISbaCJwLHe5BRVfhWOwX8cY PlEJveZ+/noJXj/8iyrlVS0E4II7YlBsWPTRksnqNmt+POtBaLzrn+zmDafRqC7B 3Om2nPJrXm8Ub0qYBodZGKSyP1FCOC5FhSAmqKPGW+9/XjU4e67WRM11nOBso+Ux QzhDgmAQAW+i6RxgBvL2Mcw3TTEJqrC8uU6rpQ/1Sgv7YEX67CiuQRmzH1LOk/Rf mqHEliLF3Qv8FmTrNXZ1FcM/L9dlOs5oHYoqL0cDdrz2fZxf3I61MpH+fr/tbTC7 89Ye+DjTTWuAmPnnbkRepubLcb8dQ1AS407Be26ANgmxY9kUCKzUc6foLCiKgZw4 ylOkJKWnRNw9qe7OJv0LamT02jp2SS3DI2G71H5mRBUYsiCzQEw= =jV+V -----END PGP SIGNATURE----- --nVJfHD17sYl4FUIt--