From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dkim=fail reason="key not found in DNS" header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202312 header.b=iaQgH7rW; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 926405A0275 for ; Wed, 14 Aug 2024 12:01:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202312; t=1723629692; bh=r324AwuqUV7i2ul0AdaEC8Qq5FM4Pb+FkRIuP53hJTk=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=iaQgH7rWyW5bXPMGkVA9scnXrPCSHZ65ArPlF4Y30YxCOLiT03mE1XatiS9lgcABF cX+5u/V8DXka2ilRLl9Vh8LbmsMZSIZI1mINM7wDYpbIZFOwbXq2bGKXr1Q/zjNYWe qQCkDoIg6Q47BzfIjlaX7gXsKN+JEOXUf+kcNT8J3Uvc84MonDgC2GB+Ca58vWWKVa 31W491axvranC7HnwPr7QQwFkdj0u0b8ssS4rZTlsJCYLHyAXCtVpszSeRUTJiEJ5O dSdvUqW3rRXbviUgI9IbAOYiq1tPEL2qZ8KwrKY/9saJzqvq+Hw66JuYEjz8Fp88H8 04VXMPfMLAeSA== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4WkP0r6j8hz4x6l; Wed, 14 Aug 2024 20:01:32 +1000 (AEST) Date: Wed, 14 Aug 2024 20:01:28 +1000 From: David Gibson To: Stefano Brivio Subject: Re: Pasta 20240726 and newer crash with ASSERTION FAILED in flow_hash Message-ID: References: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> <20240814084022.02e39e31@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="zG1FZ9V56NqrmTgp" Content-Disposition: inline In-Reply-To: <20240814084022.02e39e31@elisabeth> Message-ID-Hash: YYOPNJNHMCIRWFWFHHIA6YPMTTTZNV62 X-Message-ID-Hash: YYOPNJNHMCIRWFWFHHIA6YPMTTTZNV62 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Matt Hamilton , passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --zG1FZ9V56NqrmTgp Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Aug 14, 2024 at 08:40:22AM +0200, Stefano Brivio wrote: > Hi Matt, >=20 > On Tue, 13 Aug 2024 22:58:42 -0700 > Matt Hamilton wrote: >=20 > > I am using Podman in Fedora 40, which uses pasta by default for rootles= s=20 > > container networking. > >=20 > > Fedora 40's base version of passt is `passt-0^20240326.g4988e2b-1.fc40`= ,=20 > > but recently two newer versions were released,=20 > > `passt-0^20240726.g57a21d2-1.fc40` and `0^20240806.gee36266-1.fc40`. > >=20 > > After upgrading, one pod kept going offline after a few minutes. The=20 > > containers remained running, but could not make outbound connections.= =20 > > Journalctl revealed that the pasta process for the pod had crashed with: > >=20 > > Aug 08 23:07:55 dev pasta[95859]: ASSERTION FAILED in flow_hash > > (flow.c:566): pif !=3D PIF_NONE && !inany_is_unspecified(&side->ead= dr) > > && side->eport !=3D 0 && side->fport !=3D 0 > > Aug 08 23:07:55 dev audit[95859]: SECCOMP auid=3D1000 uid=3D1000 > > gid=3D1000 ses=3D1 > > subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 > > pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D31 > > arch=3Dc000003e syscall=3D186 compat=3D0 ip=3D0x7f8f8c23b64f code= =3D0x80000000 > > Aug 08 23:07:55 dev audit[95859]: ANOM_ABEND auid=3D1000 uid=3D1000 > > gid=3D1000 ses=3D1 > > subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 > > pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D3= 1 res=3D1 > >=20 > > After much debugging, I isolated the trigger to a particular container= =20 > > making a peer-to-peer TCP connection to a remote address with port 0. >=20 > Thanks for the analysis and for the report! >=20 > > Reverting passt to version 20240326 works as expected, and the containe= r=20 > > stays online. It's been a long time since I wrote any C, but the code= =20 > > seems clear and checks that the endpoint and forwarding ports do not=20 > > equal 0. I assume that a port 0 connection is not realistic or useful,= =A0=20 > > and that actual attempt to connect over this port indicate a bug in the= =20 > > client code. Is this correct? >=20 > Right, that's somehow unexpected because TCP port zero is reserved > and not assigned, so it should never be used. However, I'm not sure how > we can even reach flow_hash() with it. >=20 > David, this seems to come from 163a339214dd ("tcp, flow: Replace TCP > specific hash function with general flow hash"), any clue? Stefano reproduced, and I've found the issue. The assert was intended to check that we never created flows with 0 port - and we don't. Unfortunately it was also invoked when searching for an existing flow matching a new packet. Patch coming shortly. Note that this will fix the crash, but it still won't permit the connection to port 0 to go through. I don't know if that will allow your application to run, or whether it relies on that port 0 connection. Actually allowing the connection to go through would be much harder. It's easy to remove the explicit checks, obviously, but making sure we never pass that 0 to an API where it doesn't mean what we want it to would require some time. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --zG1FZ9V56NqrmTgp Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAma8gHcACgkQzQJF27ox 2GcBSg/9HIhgjmUC+wzeN5lqf1YmGppEzmKQS7vxZOWlqhvlOwN+/++NY3MQr7Iu pfMePrrK+GOfSBeaI3tzUTshFrMqQA2LywHlwZEqOd2poDiBVoFkEXvXncQsGIl7 tGdHoBaU6WvCW4jLitkUHw3ogL7HAH2VduNhkb/4CwQFh79KCrM9U6pAPpsBdDlp ZBcOcw1yLkq0YDFGj/6y8DyC9EZvDgtXbNfgXiSMkSL3RXgbbvqemAetqVYs3otS hmSGj2pmAIhDnCEQqhtJ80ScSoqeOhgtptXCBmg4tXyXq0/E9tZptohX2G02GbIw zZAVwi8q5u25pK2b2hQmZwNcKSron2TweqcemyKYbSt65I2CybYGXUABbs08AdmD WxCNynf9oXy/mgTWE0CEqMnuPXQDhDmKeQpVrsB/9fTHOlFaI0D9iKzK20A0T+Mm hjDYQvLpJmVe7q0Ij46b9xvvMFhCS7aAgNjQHIaw/uqTaZbNd4PcBvH/Qj3irIVl 3pa8Lybh8pPaf2jf0+OBylNymcTYgj9NXFZyunzTsnJ7xYsy16fJbyFXlMwk2WO+ FZxERVS9Q8R1rEACsuugZKOIO5FqcnKVohn4E6C9kka+TXZJPIjkkvgwiYa6NBM3 t1XRXkfcb/LLa/OsnGDZNBXDELVBLk4xPI6uXa35w27xGEbqyBU= =OHEw -----END PGP SIGNATURE----- --zG1FZ9V56NqrmTgp--