From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202508 header.b=CZZBxGiX; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 6DAD05A0279 for ; Mon, 25 Aug 2025 03:55:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202508; t=1756086934; bh=8rJKAiw8LwiJMlJKh+Xz5+0mnQemoppazVPW+ECQImk=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=CZZBxGiXuQLpD/Q9oole6gnGoyeMGVuVHgg4nHLeG4uaLJRTAD0MoUpPsj/TNsbjW rM4M4LEpaBWm3lVTOqbzhWHlJpaeFGqq7LUz/Muhn3oa0FyF19sElwpdxrsL2054ma fvcWew3UtBRM7qZyQ3U8oTJ4cudNAhgWbdgljcpZ2tS/K/DWpAeliafjMCU81HUOUq 8uoeZGW6ozRut3fARjplw5C89aBgA/T4i45dfFSspdaZSN1kz0o4FrFvTscWQFYQa3 gWpKs/lNiCNpBg8v+xG751378hVvCHBDiL5oALa4n6z7mQw5VU5hs9SRbfNuvRNTah YcmcWddF8RXHA== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4c9DQZ6TxKz4x3M; Mon, 25 Aug 2025 11:55:34 +1000 (AEST) Date: Mon, 25 Aug 2025 11:55:21 +1000 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests Message-ID: References: <20250820105456.1281906-1-david@gibson.dropbear.id.au> <20250820224048.413d9f2a@elisabeth> <20250821232702.7d4a3a47@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="2t6EdyF3VhL7trbI" Content-Disposition: inline In-Reply-To: <20250821232702.7d4a3a47@elisabeth> Message-ID-Hash: XMAGTPFGEF7CLKMOIJ3UHHEBCF4Y4TBT X-Message-ID-Hash: XMAGTPFGEF7CLKMOIJ3UHHEBCF4Y4TBT X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --2t6EdyF3VhL7trbI Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Aug 21, 2025 at 11:27:02PM +0200, Stefano Brivio wrote: > On Thu, 21 Aug 2025 12:48:27 +1000 > David Gibson wrote: >=20 > > On Wed, Aug 20, 2025 at 10:40:48PM +0200, Stefano Brivio wrote: > > > On Wed, 20 Aug 2025 20:54:52 +1000 > > > David Gibson wrote: > > > =20 > > > > Here's a new approach to building passt tests with exeter. This new > > > > one no longer uses Avocado in the default case, although it would > > > > still be possible to manually run the exeter based tests with Avoca= do. > > > >=20 > > > > Here's another draft of my work on testing passt with Avocado and t= he > > > > exeter library I recently created. It includes Cleber's patch addi= ng > > > > some basic Avocado tests and builds on that. > > > >=20 > > > > For now this only does simple tests, to show how the integration co= uld > > > > work. It adds some new trivial "smoke tests" and converts the lint= er > > > > and build checks to exeter. More complex tests will require buildi= ng > > > > the sinte/pesto library we've discussed. A lot of the work for that > > > > already exists in my earlier exeter test series, but it will need s= ome > > > > rework to split it into a separate component. > > > >=20 > > > > v5: > > > > * Updated according to Stefano's review > > > > - Fixed a number of whitespace errors > > > > - Improved many comments and variable names to make things clear= er > > > > * New patch adding parallel test execution with BATS > > > > * Improved autodetection of exeter tests using "exetool probe" =20 > > >=20 > > > This works on my setup and looks good to me, I just have two comments: > > >=20 > > > - test names are still the same as before (not exactly descriptive, s= ay, > > > 'make_passt'). I already reported this on v4, I'm not sure what was > > > your conclusion about it =20 > >=20 > > Sorry, I missed that comment on v4. > >=20 > > exeter test ids are by design machine-friendly identifiers more than > > they are human-friendly names or descriptions. There are a few > > reasons for that: > >=20 > > * The ids need to be passed around between test and runner both on > > the command line and via stdio. Limiting them to characters that > > are identifier friendly in most languages significantly reduces the > > chances of screwing up quoting. > >=20 > > * In some existing Python cases, and maybe more language cases in > > future, the ids are auto-generated, e.g. for a matrix or > > composition of tests. That works more naturally for > > identifiers than names/descriptions. > >=20 > > * Identifiers are more amenable to structured formatting grouping > > related tests together, which is useful for filtering out groups of > > test by glob/regexp. >=20 > It looks perhaps a bit awkward to filter Bats-based pasta tests from > Podman with, say, -f TCP, but I actually find it convenient. The test > name is human-friendly, and regexps are still easy. Point taken. Consider human-readable names descriptions bumped several notches up my priority table (and at least the basics of metadata support with it). > > * I like having a succinct id to refer to tests by rather than a > > waffly English description > >=20 > > I'm not opposed to having an (optional) human-readable name or > > description for tests in addition to the id. It would complexify the > > exeter protocol, of course, which I'm trying to keep super simple. >=20 > ...but yes, I see. On the other hand, let's pick something like: >=20 > TCP/IPv4: host to ns (spliced): big transfer >=20 > would you call that... tcp_v4_host_to_ns_spliced_big? To me that would > look like an obvious regression. More likely I'd be looking at a name based on the grouping / structure, maybe 'host_to_splice;tcp4_xfer_big'. Which makes sense =66rom the point of view of constructing those tests from compositions of pieces (which should be possible for that case), but is even more of a readability regression. > It's very hard on eyes, and much less informative to newcomers (unless > you add "transfer", but then it gets quite long for a machine-friendly > identifier). >=20 > And we'll surely run into something worse than that... >=20 > > Then again, I have several other things in mind that would need > > per-test metadata, so it's probably is worth it. >=20 > I guess we might even want to have some attributes to categorise tests, > eventually. I'm rather clueless as to the amount of complexity it adds, > but it sounds like an obvious choice to me. Right. 'tags' of this type are a very common feature for test runners. Again, this is primarily just a matter of devising a means of exposing the metadata. The only thing that makes that tricky is the fact that every exeter implementation needs to do it, so it's important to design it so as to keep the implementation complexity to an absolute minimum. > > > - I didn't check (yet) what happens when I run this as ./ci (for > > > example, from the pre-push hook), if generated web links are still > > > okay. I'll do that soon unless you can have a look first =20 > >=20 > > I don't really know how to check that. I don't think there's any > > reason it wouldn't work. >=20 > Run as ./ci, check that video_link_* links in web/ci.js make kind of > sense. Anyway, never mind, I just checked, they still work. I'm not sure I know enough about those videos or js to know what makes sense. Anyway, thanks for checking it. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --2t6EdyF3VhL7trbI Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmirwogACgkQzQJF27ox 2GeD5Q/+O9cz8ui/s9Sun+w4kI8+F9pQLlFh2eTgcNrX90kR1+8L4uPrK6BlhAKV BYKEVZrM7h0WiyfePZ0oDwX1lwXgJEychJ+uLjIoociKeqkuKbM9m/3mT/de2HF2 poUruta6NJ2R1CGeI4n31Z8dyFiN55kIzu+Qyu6dy1HpvCAgOd3hEJfT9tPH3X/s l8jIaKvs3j21zMJYb1UNLP3+V74hHM+tk8pqxEMD5e1Q8yT/QXp+X2atnHgv8bai kfCa+hLkhnVnWACHDuhS0o9FkYiQJ7hotT+QSdilhMqLG5AHM3HDJl3XGLqehK7K WorOweAx1X/DF1yalcgKxfpUJpMq1Hv9NM69Qk4DUK3rUpSNZruckihFEBwRUwhZ nTpjC1dB6RcjXK3165uSddqpWhEZltc9TPnmXey8wTVvWZEd+rDjdmOTmTCjPIB0 0x1cIyptWpBKXEcmuqoSo88y+crXCcMbX1fcFg7PvkiQtLw1Fv0sClwd4p4k+h8Z CWLsYKXdxb1wQlcFshcmpTkjJrmMJ35gIX9iVst5IIjXA4OxgLR/dLiiU34Cy5kV g8RNUzz0/pjEnzgPaAXWvKWyEpFc5k8kCuwxnMqIrMBXz6n/stVqecE61nBhkpnW rbIlUwbrMfDa9kvwHR3NOARfHl5JzB5vTx6KDT9tiEVpcEMnyIk= =gi83 -----END PGP SIGNATURE----- --2t6EdyF3VhL7trbI--