From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 505535A0319 for ; Wed, 14 Aug 2024 08:39:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202312; t=1723617562; bh=PilPrE2L0DE0Q/DWHL8Aju3LOxy/wlaohrdVqvet3DI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=SDBd2X0ro9t86B5XYe/pGZ7iH0066A/KeSMHqpZswo+uZVYecjCn4/ikdJgMWCvpp +3fO2gW84NtGlAycnzXD21nVRECaPO8lvIMVDuTPqYjKNvuGbZrMOtXC0ubbXLKyn1 zO2PcGwheyTwTbvDD6RMCifK2T0G592zKytKIVGH3qOXj6puPlFu09skubIg5Hz7pn HOVGwdvzcr1pQ26ecA/KhCK1hOb83/lYn+s0F68GHNkkpPNd3qkYak/AtKd0Ji9dqT CZxmivI5835I5VV7blRrB8PQVq5w3QJZJZgHUudJHPCqUEtXYmnH1LWzAcC8b+Z7mI aZQAvjFvSWAAg== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4WkJWZ6QmTz4x3q; Wed, 14 Aug 2024 16:39:22 +1000 (AEST) Date: Wed, 14 Aug 2024 16:39:18 +1000 From: David Gibson To: Matt Hamilton Subject: Re: Pasta 20240726 and newer crash with ASSERTION FAILED in flow_hash Message-ID: References: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="n+ZzhQ3KHoQcAcbt" Content-Disposition: inline In-Reply-To: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> Message-ID-Hash: O2BLSKB3K7D4KGBOIGNZUVVNVKBBIWBP X-Message-ID-Hash: O2BLSKB3K7D4KGBOIGNZUVVNVKBBIWBP X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --n+ZzhQ3KHoQcAcbt Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 13, 2024 at 10:58:42PM -0700, Matt Hamilton wrote: > I am using Podman in Fedora 40, which uses pasta by default for rootless > container networking. >=20 > Fedora 40's base version of passt is `passt-0^20240326.g4988e2b-1.fc40`, = but > recently two newer versions were released, > `passt-0^20240726.g57a21d2-1.fc40` and `0^20240806.gee36266-1.fc40`. >=20 > After upgrading, one pod kept going offline after a few minutes. The > containers remained running, but could not make outbound connections. > Journalctl revealed that the pasta process for the pod had crashed with: >=20 > Aug 08 23:07:55 dev pasta[95859]: ASSERTION FAILED in flow_hash > (flow.c:566): pif !=3D PIF_NONE && !inany_is_unspecified(&side->eaddr) > && side->eport !=3D 0 && side->fport !=3D 0 Ouch. > Aug 08 23:07:55 dev audit[95859]: SECCOMP auid=3D1000 uid=3D1000 > gid=3D1000 ses=3D1 > subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 > pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D31 > arch=3Dc000003e syscall=3D186 compat=3D0 ip=3D0x7f8f8c23b64f code=3D0x= 80000000 > Aug 08 23:07:55 dev audit[95859]: ANOM_ABEND auid=3D1000 uid=3D1000 > gid=3D1000 ses=3D1 > subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 > pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D31 r= es=3D1 >=20 > After much debugging, I isolated the trigger to a particular container > making a peer-to-peer TCP connection to a remote address with port 0. Huh. > Reverting passt to version 20240326 works as expected, and the container > stays online. It's been a long time since I wrote any C, but the code see= ms > clear and checks that the endpoint and forwarding ports do not equal 0. I > assume that a port 0 connection is not realistic or useful,=A0 and that a= ctual > attempt to connect over this port indicate a bug in the client code. Is t= his > correct? So, AFAICT the RFCs don't preclude using port 0 for connections on the wire. However, it's usually not really sensible to do so: at least on systems with a BSD-like socket interface, a port of 0 usually means "unspecified" or "kernel, please pick for me". Obviously this client is making it happen - my guess would be that a 0 port in connect() is interpreted as a literal port 0, but I'm not sure how the server is receiving it in thie case, since a bind() with port 0 will cause the kernel to pick a port. So, it does look like the client is doing something weird, although whether it's technically invalid is debateable. Even if it is valid for the client to do this, pasta can't really handle that case, because it's using the sockets interface to do the forwarding. BUT, it absolutely should not be crashing - it should log a debug message, drop the connection and carry on. We have code which is supposed to handle this case gracefully before reaching that assertion. I'm not immediately sure why that's not working. One possibility is that the client _isn't_ doing something weird, but an unusual port forwarding configuration on pasta is remapping a sensible port to port 0, thus causing the crash. Getting the full podman command line for the failing container would be the next step here. If you could file a bug at https://bugs.passt.top that would be most helpful. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --n+ZzhQ3KHoQcAcbt Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAma8URUACgkQzQJF27ox 2GdEpA//WW8z4aCB2a4kcVOPrLnKZBb7oebplK8Sjqwy+HrAeHekS9T437Mdn2tn KP3DCIqaUvDUjRII8aVbTOawSxhYb9FCjr/J0ANlw4l2D3BuuGaAjrt6mE9a6Jym S8uEysWFzu9qN3sJ6kiWyQ5kVIqFZL4cRTnka+nUZpyMHJtH/iujI0pXuTnT2JHz 6FiQAWpKMBPBkDuDOoqpuyhpWP2Kiu9TchzFDGLwvVJbBoQBb8hPnz0Plt4YgxIY gHw/EEBynFnt6hVinwKyxXbEcKV83ZERlPEGVjMxCjhLNHuNF625wH8ZQNJR+8ft +mGZfpaGljv7J7a75PObnpYDymhkKRTBj0XARrkgHeRRz58CIgeDMctJpEPJwO+4 BlD02g3zOEvS2OY5sf4iYKKNrbNzGE/u29t8K0uKZvfA8Ng1r/cbX3JnNpR4i6bt dkCEv2AbdRsvkR67kDCZqGvBTLse6jzQfzsoK7+hSo5KjwU8KlMixhxzwL0mXQ29 7UOZcSQF+0MzDuLt7mB0EI4pM13DxdAnRtpNB5xP6KOvkS6fk/rdcU+ZSEzgZgLa 49rS1ixSeQME0yFeGSE5WHunBFJ5kITtRiWz9crvktgizL2Mav03mgbifmtsFvpp RuuYp24icvNtBoH4pwV+di7k815vI2cexqYUBi7zN7OSqdN48Uk= =AbS9 -----END PGP SIGNATURE----- --n+ZzhQ3KHoQcAcbt--