From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTP id 553345A0082 for ; Fri, 21 Oct 2022 19:05:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1666371908; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type; bh=476IH0mrtNq1xqumRbyQ8AUKvxgDbs6r3lQ4yjuV1ko=; b=DoVUsZWUG4QcExNRGY7tRCcunLSDWHWWOuVOE8/V/7Q9IPCJj2CnD1CxiAckSQDir4LN2Q xXPJ0+6JLvHVnj/JKQBH9JmeIck9aqpYyeMYZFSxQnpFkBuRmh67awCPoBrAQ51ddTMdxg aMzYMzPKDaeAd6ifMgD5Nx6B6ZqOao0= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-100-3-earMRmO5Cr6foFQwVM7g-1; Fri, 21 Oct 2022 13:05:06 -0400 X-MC-Unique: 3-earMRmO5Cr6foFQwVM7g-1 Received: by mail-pj1-f72.google.com with SMTP id m2-20020a17090a730200b0021020cce6adso4462061pjk.3 for ; Fri, 21 Oct 2022 10:05:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=to:subject:message-id:date:mime-version:from:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=476IH0mrtNq1xqumRbyQ8AUKvxgDbs6r3lQ4yjuV1ko=; b=V7LZehsCe9Rbd/JRJkLMY7zeS6f7Ei9D/6CQO4cLLsRWOrOU49uVB50Z97PivHAzvl AbC0zQclScesg+L38HHjtbmUQRhO4zD3npm3gTEJlHhTs8cS1BVFUcXYaKlxcBGcQF03 Rub7Qea05L4qxbZXuDpMvakzcTds8LKXoVxIzZy1Tm3dL9Ka/lwiewRjCCLoZXVcenkW L8UKXICwmHofc+TCJXAhQF70bmlr/tF3U9mM2YHFpub+akNiXt6iWk68a/9dJqQIcJQ6 URf86JUz3iF6p/yJQFJG14mCmQBB1XZ+ur4d/4IXi9OmTr+TXg11iWCo1OCI3Tc6roKZ K3Sg== X-Gm-Message-State: ACrzQf3u1OIvxqJ7UGGHjYHI12QxKBuiIDYsICketxurMGi3bSmaUsp8 WVm8CG0CQhqdxBEQ38EX9LlYIWFKf/y9p+lIHZALj32PjYCawttY0lss7fkcbYr4GnSUtAf89UP zK10k550T/m3WZxTAFVUvykr/KP27 X-Received: by 2002:a05:6a00:16c4:b0:535:890:d52 with SMTP id l4-20020a056a0016c400b0053508900d52mr20348917pfc.9.1666371904823; Fri, 21 Oct 2022 10:05:04 -0700 (PDT) X-Google-Smtp-Source: AMsMyM718EtO/3mpsmx2zGTDjiH9OZWHgToyCNyoWlOafCHYALhiWJ34pPDTKBohmydN+AMd5aGmZtoAw/2RV2OnhYs= X-Received: by 2002:a05:6a00:16c4:b0:535:890:d52 with SMTP id l4-20020a056a0016c400b0053508900d52mr20348878pfc.9.1666371904341; Fri, 21 Oct 2022 10:05:04 -0700 (PDT) Received: from 744723338238 named unknown by gmailapi.google.com with HTTPREST; Fri, 21 Oct 2022 10:05:03 -0700 From: Andrea Bolognani MIME-Version: 1.0 Date: Fri, 21 Oct 2022 10:05:03 -0700 Message-ID: Subject: passt crashes on CentOS Stream 9 To: passt-dev@passt.top X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Message-ID-Hash: 5WQJZOAJ7REWJSLW3RMBXXEQXRLXLMQV X-Message-ID-Hash: 5WQJZOAJ7REWJSLW3RMBXXEQXRLXLMQV X-MailFrom: abologna@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.3 Precedence: list List-Id: Development discussion and patches for passt Archived-At: <> Archived-At: List-Archive: <> List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: This should probably be filed on bugzilla but I can't be bothered signing up for yet another service, sorry! O:-) Short version: in a CentOS Stream 9 container, install the latest build (0^20221015.gb3f3591-1) from the official COPR, then run $ passt --runas 65534 -e -t 1234 Segmentation fault (core dumped) Doing the same thing in a CentOS Stream 8 container doesn't result in a crash, and the previous build (0^20220929.g06aa26f-1) is fine even on CentOS Stream 9. The backtrace produced by gdb doesn't look very illuminating, but maybe it will make more sense to a developer: Starting program: /usr/bin/passt --runas 65534 -e -t 1234 warning: Error disabling address space randomization: Operation not permitted [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". process 2856 is executing new program: /usr/bin/passt.avx2 warning: Could not load shared library symbols for linux-vdso.so.1. Do you need "set solib-search-path" or "set sysroot"? [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Program received signal SIGSEGV, Segmentation fault. 0x000055663d5307ff in nl_sock_init (c=0x7fffd7fe4ed0, ns=false) at /usr/src/debug/passt-0^20221015.gb3f3591-1.el9.x86_64/netlink.c:78 78 { (gdb) t a a bt Thread 1 (Thread 0x7fe8763da740 (LWP 2856) "passt.avx2"): #0 0x000055663d5307ff in nl_sock_init (c=0x7fffd7fe4ed0, ns=false) at /usr/src/debug/passt-0^20221015.gb3f3591-1.el9.x86_64/netlink.c:78 #1 0x000055663d531296 in conf (c=, argc=, argv=) at /usr/src/debug/passt-0^20221015.gb3f3591-1.el9.x86_64/conf.c:1547 #2 0x000055663d5262e6 in main (argc=6, argv=0x7fffd82b2a98) at /usr/src/debug/passt-0^20221015.gb3f3591-1.el9.x86_64/passt.c:243 A very interesting thing that I've noticed is that the crash doesn't occur when building from upstream sources (tag 2022_10_15.b3f3591, so it should match what's in the RPM). So I've tried looking into the compiler options used during the RPM build, and the gcc command line for passt.avx2 looks like gcc -Wall -Wextra -pedantic -std=c99 -D_XOPEN_SOURCE=700 \ -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -pie -fPIE -DPAGE_SIZE=4096 \ -DNETNS_RUN_DIR=\"/run/netns\" -DPASST_AUDIT_ARCH=AUDIT_ARCH_X86_64 \ -DRLIMIT_STACK_VAL=8192 -DARCH=\"x86_64\" \ -DVERSION=\"0^20221015.gb3f3591-1.el9.x86_64\" -DTCP_HASH_NOINLINE \ -DSIPHASH_20B_NOINLINE -DCSUM_UNALIGNED_NO_IPA -DHAS_SND_WND \ -DHAS_BYTES_ACKED -DHAS_MIN_RTT -DHAS_GETRANDOM \ -fstack-protector-strong -Ofast -mavx2 -ftree-vectorize \ -funroll-loops -flto=auto -ffat-lto-objects -fexceptions -g \ -grecord-gcc-switches -pipe -Wall -Werror=format-security \ -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS \ -specs=/usr/lib/rpm/redhat/redhat-hardened-cc1 \ -fstack-protector-strong \ -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 -m64 -march=x86-64-v2 \ -mtune=generic -fasynchronous-unwind-tables \ -fstack-clash-protection -fcf-protection arch.c arp.c checksum.c \ conf.c dhcp.c dhcpv6.c icmp.c igmp.c isolation.c lineread.c log.c \ mld.c ndp.c netlink.c packet.c passt.c pasta.c pcap.c siphash.c \ tap.c tcp.c tcp_splice.c udp.c util.c -o passt.avx2 -Wl,-z,relro \ -Wl,--as-needed -Wl,-z,now \ -specs=/usr/lib/rpm/redhat/redhat-hardened-ld \ -specs=/usr/lib/rpm/redhat/redhat-annobin-cc1 I tried making educated guesses at which ones among those could cause trouble, and pretty quickly landed on the LTO stuff. Indeed, dropping -flto=auto -ffat-lto-objects from the command results in a working binary, and adding %global _lto_cflags %nil to the top of the spec file produces a working RPM. Of course disabling LTO is a workaround, not a solution, especially considering that the previous version didn't have any problem with it, but hopefully there's enough information in here to allow the developers to track down and resolve the underlying issue :) -- Andrea Bolognani / Red Hat / Virtualization