From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTP id A40BE5A031A for ; Wed, 14 Aug 2024 09:01:43 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1723618902; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=k3xqYUPNAGwWLaqEfiqmjmdQL56SyRyct6bksA0/FKM=; b=DcPlyraFv5XoL59AsH6bScMos2tNxX10KywH317VhSAHAxydRDMKNNNTnIFoC5hM5P3GPs uVJaUSek8xaToTy5OgP0KZgWV3cwC4ca9cUzjEKF6V3zJ17LN+nVFT19FY9GHTYZ0NNo06 1fBgvX+vhxJpC8B6RHWc1LoPR/kzrVg= Received: from mail-pl1-f197.google.com (mail-pl1-f197.google.com [209.85.214.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-128-FlDEqjnEPz6wpIvvUxdtJw-1; Wed, 14 Aug 2024 03:01:39 -0400 X-MC-Unique: FlDEqjnEPz6wpIvvUxdtJw-1 Received: by mail-pl1-f197.google.com with SMTP id d9443c01a7336-1fd774c3b8eso5538755ad.0 for ; Wed, 14 Aug 2024 00:01:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1723618898; x=1724223698; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=58fZLgXMHnTiYyQpg2iNhxhFMrIinikxEW27xDzwtFM=; b=ZAu+llVLcln2kj+dLgLJ0ynGlkX2XsYk5WybQjVe5IZA+J5kei6vNpDj4tFmMQFHFE JmjnmG1QvKkyJk0S5A92DUaErFrNeGcuPk3mJ0JQLiX0UaPhs+kucCggnvFlfisRVCta rHN7wJzYfifTKNHrIaZlPhKqA+3ljRuLHZg3k92jf2dVGSVlLAmS5+l1y+VGrGQXQt0i ZunBrzSyKJ2lBXIedWDx7jD4CLhvxojMFTh5un4U5Pyvbi8JH/R/Q+qgIFPAYG5PUSmv BvgodDGFO4LJW8aExcY0CjNpfER1+hxipw9PlLfYrnWLzyjleZniKV6lsHCqBKW3n4Dv oOZQ== X-Forwarded-Encrypted: i=1; AJvYcCVj2HU5BUy8Yvnsb7ETV9GYW0xNGA9ogSuzYtiPV5uFNxrs1v/IvGMi0pjH+6V4j4x77U8CpmfHo0mTB/PvI9X8MqEv+w== X-Gm-Message-State: AOJu0Yx15wjBlucOyntyZRQqQnamnkgKOT1wvFEfvg2goTPUobVsxJAA Ixw+NmQkaT+Wp+rm1dopA2MsjqUjCgT5AV8LPUyn/bn4POwTOO6LDp6W/VwT6vpwbiU4j/kszau MYmjp2V47Xlqv/MO9BhpztUihpY73ycd9+uR4D1Jzy+vFenXIuGk= X-Received: by 2002:a17:903:234f:b0:1fd:96c7:24f5 with SMTP id d9443c01a7336-201cbba030bmr83603555ad.5.1723618897452; Wed, 14 Aug 2024 00:01:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFcUszQyYkYsh36l6SgVJaFfoPiL43HfsN8YNvSYhHl/WmK1VdIiVwje9Lvt8JQiuiB6FnxAg== X-Received: by 2002:a17:903:234f:b0:1fd:96c7:24f5 with SMTP id d9443c01a7336-201cbba030bmr83603005ad.5.1723618896650; Wed, 14 Aug 2024 00:01:36 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-201cd1a94b1sm23549275ad.122.2024.08.14.00.01.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 14 Aug 2024 00:01:36 -0700 (PDT) Date: Wed, 14 Aug 2024 09:01:33 +0200 From: Stefano Brivio To: Matt Hamilton Subject: Re: Pasta 20240726 and newer crash with ASSERTION FAILED in flow_hash Message-ID: <20240814090133.2d7f210c@elisabeth> In-Reply-To: <6d484b93-9bd4-4ab4-88cd-017b99a1df6e@thmail.io> References: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> <6d484b93-9bd4-4ab4-88cd-017b99a1df6e@thmail.io> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: MELDYRYGOHFVV4N4JQKUCJS75L34VIKM X-Message-ID-Hash: MELDYRYGOHFVV4N4JQKUCJS75L34VIKM X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: David Gibson , passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, 13 Aug 2024 23:56:56 -0700 Matt Hamilton wrote: > On 8/13/24 11:39 PM, David Gibson wrote: > > On Tue, Aug 13, 2024 at 10:58:42PM -0700, Matt Hamilton wrote: =20 > >> I am using Podman in Fedora 40, which uses pasta by default for rootle= ss > >> container networking. > >> > >> Fedora 40's base version of passt is `passt-0^20240326.g4988e2b-1.fc40= `, but > >> recently two newer versions were released, > >> `passt-0^20240726.g57a21d2-1.fc40` and `0^20240806.gee36266-1.fc40`. > >> > >> After upgrading, one pod kept going offline after a few minutes. The > >> containers remained running, but could not make outbound connections. > >> Journalctl revealed that the pasta process for the pod had crashed wit= h: > >> > >> Aug 08 23:07:55 dev pasta[95859]: ASSERTION FAILED in flow_hash > >> (flow.c:566): pif !=3D PIF_NONE && !inany_is_unspecified(&side->ea= ddr) > >> && side->eport !=3D 0 && side->fport !=3D 0 =20 > > Ouch. > > =20 > >> Aug 08 23:07:55 dev audit[95859]: SECCOMP auid=3D1000 uid=3D1000 > >> gid=3D1000 ses=3D1 > >> subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c102= 3 > >> pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D= 31 > >> arch=3Dc000003e syscall=3D186 compat=3D0 ip=3D0x7f8f8c23b64f code= =3D0x80000000 > >> Aug 08 23:07:55 dev audit[95859]: ANOM_ABEND auid=3D1000 uid=3D100= 0 > >> gid=3D1000 ses=3D1 > >> subj=3Dunconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c102= 3 > >> pid=3D95859 comm=3D"pasta.avx2" exe=3D"/usr/bin/pasta.avx2" sig=3D= 31 res=3D1 > >> > >> After much debugging, I isolated the trigger to a particular container > >> making a peer-to-peer TCP connection to a remote address with port 0. = =20 > > Huh. > > =20 > >> Reverting passt to version 20240326 works as expected, and the contain= er > >> stays online. It's been a long time since I wrote any C, but the code = seems > >> clear and checks that the endpoint and forwarding ports do not equal 0= . I > >> assume that a port 0 connection is not realistic or useful,=C2=A0 and = that actual > >> attempt to connect over this port indicate a bug in the client code. I= s this > >> correct? =20 > > So, AFAICT the RFCs don't preclude using port 0 for connections on the > > wire. However, it's usually not really sensible to do so: at least on > > systems with a BSD-like socket interface, a port of 0 usually means > > "unspecified" or "kernel, please pick for me". Obviously this client > > is making it happen - my guess would be that a 0 port in connect() is > > interpreted as a literal port 0, but I'm not sure how the server is > > receiving it in thie case, since a bind() with port 0 will cause the > > kernel to pick a port. > > > > So, it does look like the client is doing something weird, although > > whether it's technically invalid is debateable. > > > > Even if it is valid for the client to do this, pasta can't really > > handle that case, because it's using the sockets interface to do the > > forwarding. BUT, it absolutely should not be crashing - it should log > > a debug message, drop the connection and carry on. > > > > We have code which is supposed to handle this case gracefully before > > reaching that assertion. I'm not immediately sure why that's not worki= ng. > > > > One possibility is that the client _isn't_ doing something weird, but > > an unusual port forwarding configuration on pasta is remapping a > > sensible port to port 0, thus causing the crash. > > > > Getting the full podman command line for the failing container would > > be the next step here. If you could file a bug at > > https://bugs.passt.top that would be most helpful. =20 >=20 > I tried to make an account on bugzilla a day or two ago, but haven't=20 > received the email confirmation link - I tried signing up using my=20 > personal domain (used here) and a free service (gmail). I came here as a= =20 > second attempt to reach the devs! Sorry, my bad, I'm temporarily reviewing email confirmation requests because of an influx of spam and missed yours. You should have it now. --=20 Stefano