From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=N7XiaIuC; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 5F4B65A027C for ; Thu, 21 Aug 2025 23:27:09 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1755811628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=YU4CAz8H9eOqyf/+jn/BEa7BUz20PiLRlKeC9JYy8Tw=; b=N7XiaIuCLAlg/9j0cl0rS5VmJfkzgK7zA4twzjacRz33Id2/tkzuX/eUyTSc2pg1d/JQFn ulUY37tIch0bc2TshmQOOKVcvNLE7UO4U/txX8TpPQXuOys37p+j1m4wJICwqOuE9r1e13 UCw2s7aSlpN96MUmIGwuLXIhBzgmFp4= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-522-mJxImylsOYqKDzePf3evDw-1; Thu, 21 Aug 2025 17:27:06 -0400 X-MC-Unique: mJxImylsOYqKDzePf3evDw-1 X-Mimecast-MFC-AGG-ID: mJxImylsOYqKDzePf3evDw_1755811626 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-3b9edf80ddcso527254f8f.3 for ; Thu, 21 Aug 2025 14:27:06 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1755811625; x=1756416425; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YU4CAz8H9eOqyf/+jn/BEa7BUz20PiLRlKeC9JYy8Tw=; b=DdOV3izmspbWQLLSmb5+NCLusL4DTtv0FlPyECnkKLu0O5AXaPPDGR/BbI46Eawju2 S9YIP6PjR58BSetz6g84Io82YuDwAXTqrKXJ6BBqpM/g77XatzFpQ1w/KuxOkk3NJkLY Y3BrDP8Q+X46kjbOSSjVIg1wfqPJb9/bAZE7wHpoJW1YZbMOrg2FSp67KVjFVEdxvLPI BSWQekahbbyHErwYaScjtoNHouovufLXOjoxyU0NBLYo9yS53DUwz4Tre2FmAF9yylyv Zyl5MnHtCOP2cg0+H94dVEybM09To9YTEDJyUG+fp+KLJ/eDUZT9xh29SfdeoXxn3cWZ XXrA== X-Gm-Message-State: AOJu0YwL5go7PZmNnfiEQubNLOXr0IuXv0KEH1bs5tLXlouFcTY1zTS9 WlotLN4he0HeHCz2GGILtvP3lzKEhW0s12jQiYzjY+NR7Yzcs5UfAsVaOA1K0u8e4p2Wl1HOcnm rRWptKnN5SkUeTl87M2uYtQ4/sbdSQ150OvZvluM8cGqmb7sDDkr/Nr5MLnxnPg== X-Gm-Gg: ASbGncvGTCS+fOAqYro02IhQFhBjMt3Qy1y//WtYsYgwH81yi8rNxe/VVL/3qAzdWuj CzaJfEpf+1esVhieTrJYIMMs98A2JXf/qliWBIfVYjfyi4ciAcZ+5o6V7c5Um7UEkjakKEqm/eB 0krD8hFkTUej9lC39fdVHUn5cqiZgmNrActm5bRFBff97lsY9VFTn5zstRdBcSsWis8GSceLWNv pZBHJ0wku34cbsfPNlOm4OiLgMTlazQpDK2A+MinfZt3e5OWpzvuvzLWnNuFpdu35iXa3jZ38hi ftz9SMwBRYHaBu+bwYXoyahaFrxMvSPNFnMNOuisyFZbdHD27gY= X-Received: by 2002:a05:6000:2282:b0:3b7:9af4:9c75 with SMTP id ffacd0b85a97d-3c5dc7315c3mr241857f8f.30.1755811625035; Thu, 21 Aug 2025 14:27:05 -0700 (PDT) X-Google-Smtp-Source: AGHT+IF/5ybKra0Z01IZQ3yXMYkYlMSp29dxanWxQQFtaXLbOpjqeQ7U3fv1rYBaJyUAr/BkTiBBXQ== X-Received: by 2002:a05:6000:2282:b0:3b7:9af4:9c75 with SMTP id ffacd0b85a97d-3c5dc7315c3mr241847f8f.30.1755811624609; Thu, 21 Aug 2025 14:27:04 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-45b50e2fa9csm11586495e9.16.2025.08.21.14.27.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 21 Aug 2025 14:27:03 -0700 (PDT) Date: Thu, 21 Aug 2025 23:27:02 +0200 From: Stefano Brivio To: David Gibson Subject: Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests Message-ID: <20250821232702.7d4a3a47@elisabeth> In-Reply-To: References: <20250820105456.1281906-1-david@gibson.dropbear.id.au> <20250820224048.413d9f2a@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: eLh_TY2lDfqJ_f8w8ZialliGjQTP1Cq5fDBgZ5rfaYc_1755811626 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: Q5FQCCSYQOTWVQJ37663CKNBW3PQBSFV X-Message-ID-Hash: Q5FQCCSYQOTWVQJ37663CKNBW3PQBSFV X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 21 Aug 2025 12:48:27 +1000 David Gibson wrote: > On Wed, Aug 20, 2025 at 10:40:48PM +0200, Stefano Brivio wrote: > > On Wed, 20 Aug 2025 20:54:52 +1000 > > David Gibson wrote: > > > > > Here's a new approach to building passt tests with exeter. This new > > > one no longer uses Avocado in the default case, although it would > > > still be possible to manually run the exeter based tests with Avocado. > > > > > > Here's another draft of my work on testing passt with Avocado and the > > > exeter library I recently created. It includes Cleber's patch adding > > > some basic Avocado tests and builds on that. > > > > > > For now this only does simple tests, to show how the integration could > > > work. It adds some new trivial "smoke tests" and converts the linter > > > and build checks to exeter. More complex tests will require building > > > the sinte/pesto library we've discussed. A lot of the work for that > > > already exists in my earlier exeter test series, but it will need some > > > rework to split it into a separate component. > > > > > > v5: > > > * Updated according to Stefano's review > > > - Fixed a number of whitespace errors > > > - Improved many comments and variable names to make things clearer > > > * New patch adding parallel test execution with BATS > > > * Improved autodetection of exeter tests using "exetool probe" > > > > This works on my setup and looks good to me, I just have two comments: > > > > - test names are still the same as before (not exactly descriptive, say, > > 'make_passt'). I already reported this on v4, I'm not sure what was > > your conclusion about it > > Sorry, I missed that comment on v4. > > exeter test ids are by design machine-friendly identifiers more than > they are human-friendly names or descriptions. There are a few > reasons for that: > > * The ids need to be passed around between test and runner both on > the command line and via stdio. Limiting them to characters that > are identifier friendly in most languages significantly reduces the > chances of screwing up quoting. > > * In some existing Python cases, and maybe more language cases in > future, the ids are auto-generated, e.g. for a matrix or > composition of tests. That works more naturally for > identifiers than names/descriptions. > > * Identifiers are more amenable to structured formatting grouping > related tests together, which is useful for filtering out groups of > test by glob/regexp. It looks perhaps a bit awkward to filter Bats-based pasta tests from Podman with, say, -f TCP, but I actually find it convenient. The test name is human-friendly, and regexps are still easy. > * I like having a succinct id to refer to tests by rather than a > waffly English description > > I'm not opposed to having an (optional) human-readable name or > description for tests in addition to the id. It would complexify the > exeter protocol, of course, which I'm trying to keep super simple. ...but yes, I see. On the other hand, let's pick something like: TCP/IPv4: host to ns (spliced): big transfer would you call that... tcp_v4_host_to_ns_spliced_big? To me that would look like an obvious regression. It's very hard on eyes, and much less informative to newcomers (unless you add "transfer", but then it gets quite long for a machine-friendly identifier). And we'll surely run into something worse than that... > Then again, I have several other things in mind that would need > per-test metadata, so it's probably is worth it. I guess we might even want to have some attributes to categorise tests, eventually. I'm rather clueless as to the amount of complexity it adds, but it sounds like an obvious choice to me. > > - I didn't check (yet) what happens when I run this as ./ci (for > > example, from the pre-push hook), if generated web links are still > > okay. I'll do that soon unless you can have a look first > > I don't really know how to check that. I don't think there's any > reason it wouldn't work. Run as ./ci, check that video_link_* links in web/ci.js make kind of sense. Anyway, never mind, I just checked, they still work. -- Stefano