From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Ggb2YdZp; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTP id 8F1F25A0269 for ; Wed, 14 Aug 2024 11:57:54 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723629473; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=tiKmrVfYrBvtwFlG48Y0YcsJtvmettQfFrzBCr+MOQo=; b=Ggb2YdZpFIBav1H6jH1aLTECTyT8Zs5W/jMkXqfJ+xSqj6xCgDPkFf92zyiFIpJi6gMAe4 3ucXd66+2GffFz1B2h5Jd/u17YfppMV47JEYj8mJfwDnjZhdw1twa1hMI6VREqvbz19K5e XvUwJLZilO6NNcOcKocnns1MfdbSoJA= Received: from mail-pj1-f71.google.com (mail-pj1-f71.google.com [209.85.216.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-606-iY6aN07kMl24fXuo-mpfOA-1; Wed, 14 Aug 2024 05:57:51 -0400 X-MC-Unique: iY6aN07kMl24fXuo-mpfOA-1 Received: by mail-pj1-f71.google.com with SMTP id 98e67ed59e1d1-2cb696be198so6685938a91.3 for ; Wed, 14 Aug 2024 02:57:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723629470; x=1724234270; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=izYjvXdhEUNYjdxYkdiZG65vhtovBcBOBu1KbQr8buk=; b=lccYF30uSL8HdiKXo5gAjfOTzbMMe/rAYNll80Vn04nlt3LcHtjoxjyIQuE11TNbMG QY0HcWSxEp0qkUE2OUX8Oc3Mx5OpLvGxajJYdk3pkTVAbQGMA6xBaid6bnCk0BTh6ITC znnfq34mwrjN+RFU1k1HFo3q9hFE3zyAkO3q4iBPjlatD2JKZBqqPu0xhBTK4yj1iZS+ rLpbbncmfsnVRqE2Ys4JOe5G4j9MSPhC67WqJgip458Xp0PtWtLln6bKaNnaTzSC28Ad rAoheLlIs6vh199Clb6ctgSeZJjPBF0iMXIH1tqeYKz6/0Y9TKvesIJL/OiF/v5iMTbE +oig== X-Forwarded-Encrypted: i=1; AJvYcCXNeZOYnDFetHvG64Nc3USEcbmFBpjGzxRqP/cxMCqorOKAZ6AQGT2b5gAwZ1D7QMwlbEzjACKGfBcgaBc2cLYT3Razew== X-Gm-Message-State: AOJu0Yxts3JbgdVotNVbttXJKkRa05wi1fo0l6kGuyLs6jvyEBc+2jST kBfdgfPw/6S7R4pV/+KhmMu2EnIVOxs72Kizo/KLRvg+0uKq0ZGMDf+JJk2rkoaPFQ2cjVZj06y fnlXlerkL9ZbeN6Bnm3pVLNUKHUEdQVx2ZRxhLHBcykpDi/ilZCw= X-Received: by 2002:a17:902:ea06:b0:1fb:9cb0:3e23 with SMTP id d9443c01a7336-201d63c3d9amr26907355ad.26.1723629470507; Wed, 14 Aug 2024 02:57:50 -0700 (PDT) X-Google-Smtp-Source: AGHT+IG36beE+U251QtAlppCSGVUbmKuJDiDJnCGIwFyE3rgsZwoina8iJKsXE4w8UHrrKsaJHijqA== X-Received: by 2002:a17:902:ea06:b0:1fb:9cb0:3e23 with SMTP id d9443c01a7336-201d63c3d9amr26907205ad.26.1723629469977; Wed, 14 Aug 2024 02:57:49 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd1b4640sm26323635ad.192.2024.08.14.02.57.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 02:57:48 -0700 (PDT) Date: Wed, 14 Aug 2024 11:57:44 +0200 From: Stefano Brivio To: Matt Hamilton Subject: Re: Pasta 20240726 and newer crash with ASSERTION FAILED in flow_hash Message-ID: <20240814115744.56b528f4@elisabeth> In-Reply-To: <6d484b93-9bd4-4ab4-88cd-017b99a1df6e@thmail.io> References: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> <6d484b93-9bd4-4ab4-88cd-017b99a1df6e@thmail.io> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: GBS7YTAWH3HWCWEK2XAHXMH7YK4ELCL4 X-Message-ID-Hash: GBS7YTAWH3HWCWEK2XAHXMH7YK4ELCL4 X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: David Gibson , passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, 13 Aug 2024 23:56:56 -0700 Matt Hamilton wrote: > On 8/13/24 11:39 PM, David Gibson wrote: > > On Tue, Aug 13, 2024 at 10:58:42PM -0700, Matt Hamilton wrote: =20 > >> I am using Podman in Fedora 40, which uses pasta by default for rootle= ss > >> container networking. > >> > >> Fedora 40's base version of passt is `passt-0^20240326.g4988e2b-1.fc40= `, but > >> recently two newer versions were released, > >> `passt-0^20240726.g57a21d2-1.fc40` and `0^20240806.gee36266-1.fc40`. > >> > >> After upgrading, one pod kept going offline after a few minutes. The > >> containers remained running, but could not make outbound connections. > >> Journalctl revealed that the pasta process for the pod had crashed wit= h: > >> > >> Aug 08 23:07:55 dev pasta[95859]: ASSERTION FAILED in flow_hash > >> (flow.c:566): pif !=3D PIF_NONE && !inany_is_unspecified(&side->ea= ddr) > >> && side->eport !=3D 0 && side->fport !=3D 0 =20 > > Ouch. > > =20 > >> Aug 08 23:07:55 dev audit[95859]: SECCOMP auid=3D1000 uid=3D1000 > >> gid=3D1000 ses=3D1 > >> subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c102= 3 > >> pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D= 31 > >> arch=3Dc000003e syscall=3D186 compat=3D0 ip=3D0x7f8f8c23b64f code= =3D0x80000000 > >> Aug 08 23:07:55 dev audit[95859]: ANOM_ABEND auid=3D1000 uid=3D100= 0 > >> gid=3D1000 ses=3D1 > >> subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c102= 3 > >> pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D= 31 res=3D1 > >> > >> After much debugging, I isolated the trigger to a particular container > >> making a peer-to-peer TCP connection to a remote address with port 0. = =20 > > Huh. > > =20 > >> Reverting passt to version 20240326 works as expected, and the contain= er > >> stays online. It's been a long time since I wrote any C, but the code = seems > >> clear and checks that the endpoint and forwarding ports do not equal 0= . I > >> assume that a port 0 connection is not realistic or useful,=C2=A0 and = that actual > >> attempt to connect over this port indicate a bug in the client code. I= s this > >> correct? =20 > > So, AFAICT the RFCs don't preclude using port 0 for connections on the > > wire. However, it's usually not really sensible to do so: at least on > > systems with a BSD-like socket interface, a port of 0 usually means > > "unspecified" or "kernel, please pick for me". Obviously this client > > is making it happen - my guess would be that a 0 port in connect() is > > interpreted as a literal port 0, but I'm not sure how the server is > > receiving it in thie case, since a bind() with port 0 will cause the > > kernel to pick a port. > > > > So, it does look like the client is doing something weird, although > > whether it's technically invalid is debateable. > > > > Even if it is valid for the client to do this, pasta can't really > > handle that case, because it's using the sockets interface to do the > > forwarding. BUT, it absolutely should not be crashing - it should log > > a debug message, drop the connection and carry on. > > > > We have code which is supposed to handle this case gracefully before > > reaching that assertion. I'm not immediately sure why that's not worki= ng. > > > > One possibility is that the client _isn't_ doing something weird, but > > an unusual port forwarding configuration on pasta is remapping a > > sensible port to port 0, thus causing the crash. > > > > Getting the full podman command line for the failing container would > > be the next step here. If you could file a bug at > > https://bugs.passt.top that would be most helpful. =20 >=20 > I tried to make an account on bugzilla a day or two ago, but haven't=20 > received the email confirmation link - I tried signing up using my=20 > personal domain (used here) and a free service (gmail). I came here as a= =20 > second attempt to reach the devs! >=20 > If you can get me hooked up over there, I can file a bug with more=20 > detailed logs and the podman command to reproduce. I hope you got the Bugzilla confirmation request email by now, but anyway, we just managed to reproduce this, and a fix is on its way, so there's no need for you to collect more information. Thanks again! --=20 Stefano