From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=reject dis=none) header.from=thmail.io Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=mailgun.thmail.io header.i=@mailgun.thmail.io header.a=rsa-sha256 header.s=smtp header.b=ow77J1Y6; dkim=pass (2048-bit key; unprotected) header.d=thmail.io header.i=@thmail.io header.a=rsa-sha256 header.s=dkim header.b=Am+tpK8b; dkim-atps=neutral Received: from so254-40.mailgun.net (so254-40.mailgun.net [198.61.254.40]) by passt.top (Postfix) with UTF8SMTPS id ED44E5A0262 for ; Wed, 14 Aug 2024 19:22:58 +0200 (CEST) DKIM-Signature: a=rsa-sha256; v=1; c=relaxed/relaxed; d=mailgun.thmail.io; q=dns/txt; s=smtp; t=1723656177; x=1723663377; h=Content-Transfer-Encoding: Content-Type: In-Reply-To: From: From: References: Cc: To: To: Subject: Subject: MIME-Version: Date: Message-ID: Sender: Sender; bh=4JVc+8cL7MLqY9hIEEQenykfOo/tFzH9KZ1p24q9sVE=; b=ow77J1Y6bu/UjSOgXEyD7+8WTSRNY8Hu20kAsxpVXJUWy+itk/h9sehgzAxc+OG83ORxfaykBo+4Khn8gOH1pbt+DFikB9ybKwu+53ABBEg9MuCZSEfxcVWNRaPjHszOSqPGbUHuz3c50t4tnz6lP107TelDjPX2/PlEozb8eoo= X-Mailgun-Sending-Ip: 198.61.254.40 X-Mailgun-Sid: WyI1YzUzMSIsInBhc3N0LXVzZXJAcGFzc3QudG9wIiwiY2ZjY2E0YiJd Received: from mail.thmail.io (mail.thmail.io [45.79.85.120]) by d802e597ecd9 with SMTP id 66bce7f04294deb0a496204a (version=TLS1.3, cipher=TLS_AES_128_GCM_SHA256); Wed, 14 Aug 2024 17:22:56 GMT Sender: matt=thmail.io@mailgun.thmail.io Received: from [127.0.0.1] (localhost [127.0.0.1]) by localhost (Mailerdaemon) with ESMTPSA id C1BDE48829; Wed, 14 Aug 2024 17:22:53 +0000 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thmail.io; s=dkim; t=1723656176; h=from:subject:date:message-id:to:cc:mime-version:content-type: content-transfer-encoding:content-language:in-reply-to:references; bh=4JVc+8cL7MLqY9hIEEQenykfOo/tFzH9KZ1p24q9sVE=; b=Am+tpK8bItffXjyxJuyuIzFDn0uKoqPfK21LgO0ZGEOQT4jyav/+8ZeF6uOF0t9N8l+miF rVrj6pZPj2lR/4sVYQzoTXyG+379Ora+CYlMk5FiFm1wra9sBmOjRjaFBgBIwcHEb1VJUJ nvMfund9LP8KLH8qJ7OAxbieqsXSFVshKplT3ZVRIkhh2poFdqBT8/27wTsFJevY6dooOF +FJCFeRp32Ky/lbJKbip/s03IQ+SGtqPIKjKJT5wjOSaZFZUOOhRsrcOfX6JrCXuIQXA9Z DCw6/sfWKuLz2OypeoiDjhNrPBLELoAREHuyB+tHJcQExnJrB9YjPhlRL4uQaQ== Message-ID: Date: Wed, 14 Aug 2024 10:22:52 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Pasta 20240726 and newer crash with ASSERTION FAILED in flow_hash To: David Gibson , Stefano Brivio References: <1f7aefdc-11e8-4993-b647-7429da67b26c@thmail.io> <20240814084022.02e39e31@elisabeth> Content-Language: en-US From: Matt Hamilton In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Last-TLS-Session-Version: TLSv1.3 Message-ID-Hash: RYYXZWZQCVGQCUQ4MP37OMOV3QN3JE2M X-Message-ID-Hash: RYYXZWZQCVGQCUQ4MP37OMOV3QN3JE2M X-MailFrom: bounce+f5b216.cfcca4b-passt-user=passt.top@mailgun.thmail.io X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-user@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: "For passt users: support, questions and answers" Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 8/14/24 3:01 AM, David Gibson wrote: > On Wed, Aug 14, 2024 at 08:40:22AM +0200, Stefano Brivio wrote: >> Hi Matt, >> >> On Tue, 13 Aug 2024 22:58:42 -0700 >> Matt Hamilton wrote: >> >>> I am using Podman in Fedora 40, which uses pasta by default for rootless >>> container networking. >>> >>> Fedora 40's base version of passt is `passt-0^20240326.g4988e2b-1.fc40`, >>> but recently two newer versions were released, >>> `passt-0^20240726.g57a21d2-1.fc40` and `0^20240806.gee36266-1.fc40`. >>> >>> After upgrading, one pod kept going offline after a few minutes. The >>> containers remained running, but could not make outbound connections. >>> Journalctl revealed that the pasta process for the pod had crashed with: >>> >>> Aug 08 23:07:55 dev pasta[95859]: ASSERTION FAILED in flow_hash >>> (flow.c:566): pif != PIF_NONE && !inany_is_unspecified(&side->eaddr) >>> && side->eport != 0 && side->fport != 0 >>> Aug 08 23:07:55 dev audit[95859]: SECCOMP auid=1000 uid=1000 >>> gid=1000 ses=1 >>> subj=unconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 >>> pid=95859 comm="pasta.avx2" exe="/usr/bin/pasta.avx2" sig=31 >>> arch=c000003e syscall=186 compat=0 ip=0x7f8f8c23b64f code=0x80000000 >>> Aug 08 23:07:55 dev audit[95859]: ANOM_ABEND auid=1000 uid=1000 >>> gid=1000 ses=1 >>> subj=unconfined_u:unconfined_r:container_runtime_t:s0-s0:c0.c1023 >>> pid=95859 comm="pasta.avx2" exe="/usr/bin/pasta.avx2" sig=31 res=1 >>> >>> After much debugging, I isolated the trigger to a particular container >>> making a peer-to-peer TCP connection to a remote address with port 0. >> Thanks for the analysis and for the report! >> >>> Reverting passt to version 20240326 works as expected, and the container >>> stays online. It's been a long time since I wrote any C, but the code >>> seems clear and checks that the endpoint and forwarding ports do not >>> equal 0. I assume that a port 0 connection is not realistic or useful, >>> and that actual attempt to connect over this port indicate a bug in the >>> client code. Is this correct? >> Right, that's somehow unexpected because TCP port zero is reserved >> and not assigned, so it should never be used. However, I'm not sure how >> we can even reach flow_hash() with it. >> >> David, this seems to come from 163a339214dd ("tcp, flow: Replace TCP >> specific hash function with general flow hash"), any clue? > Stefano reproduced, and I've found the issue. The assert was intended > to check that we never created flows with 0 port - and we don't. > Unfortunately it was also invoked when searching for an existing flow > matching a new packet. > > Patch coming shortly. Note that this will fix the crash, but it still > won't permit the connection to port 0 to go through. I don't know if > that will allow your application to run, or whether it relies on that > port 0 connection. > > Actually allowing the connection to go through would be much harder. > It's easy to remove the explicit checks, obviously, but making sure we > never pass that 0 to an API where it doesn't mean what we want it to > would require some time. > Thank you gents! I will pull the git repo and test a local build with the patch applied. I agree with your approach, pasta shouldn't be responsible for rewiring a badly formatted connection request. The zero port destination address is nonsensical so I'll keep running down the issue with the client devs - I think they should be discarding the address instead of letting it propagate to ultimately fail at the networking layer.