From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202508 header.b=gXh4Ct6/; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 3C0D35A0271 for ; Wed, 06 Aug 2025 03:11:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202508; t=1754442692; bh=fBDJ5ee4vgexGbsPlCzgwelvuW+/vfdvWgMRIMOkrdI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=gXh4Ct6/DjSuKhtGZ1zhfNPKAiV8IhRJeKN0noVk/DfLptnF6Uwm1YulCiJos7MN0 abxhmk2RPF7zJ5QsI5YIHT5vnWWXNIakvN9YGqPJjHn62tueTbFg8hmY+AsVC21n3m Mslia0hLcm4bxXjFpfgqmrAddGOs0xKSTLe2MmdqUoYhdFWoh0lYm9cp6Yim46kadP xeHfMIIrxHO2y9bk/wvAtAMmSLkSaVPVQ7M5W6ZoUCuS178WkvQ0EVdWCVp0ftgizZ 9hHvavuoLmgsMZaGI9XmcMtxXK2jmE4HbvRyjcqLJB2+K0H0/7duMywW/g73QVovg+ fHLx0s5y24gKA== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4bxXLX2zjWz4wcd; Wed, 6 Aug 2025 11:11:32 +1000 (AEST) Date: Wed, 6 Aug 2025 11:11:24 +1000 From: David Gibson To: Stefano Brivio Subject: Re: Exeter testing, next steps Message-ID: References: <20250805095241.6cb715d5@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="KYjLLLZziYZk0II/" Content-Disposition: inline In-Reply-To: <20250805095241.6cb715d5@elisabeth> Message-ID-Hash: QRCKA6S5N5K4XAUZD3O3JONIJVTUOROX X-Message-ID-Hash: QRCKA6S5N5K4XAUZD3O3JONIJVTUOROX X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --KYjLLLZziYZk0II/ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote: > On Tue, 5 Aug 2025 16:09:34 +1000 > David Gibson wrote: >=20 > > Today I finally had a good look at what the next steps need to be if > > we are to use exeter based test "for real" in the passt repo. >=20 > \o/ >=20 > > I already have draft series which move a bunch of the simpler tests to > > exeter. But in order to actually use these we need some sort of > > runner. We're not tied to that runner, we can easily change - that's > > the whole point of exeter - but we need to have something. >=20 > I guess as an even earlier first step we could actually have a > hardcoded shell script similar to today's test/run... or am I > misrepresenting the problem? That's sort of option (2) below. I suppose we could just have a shell script that individually runs each exeter test. I don't love that idea though because a) exeter tests make no attempt to pretty print, so will show a lot of logging garbage even on successful runs and b) it means the script needs to be updated whenever the set of tests in an exeter program is changed. > > AIUI, Stefano is not happy with the idea of using either Avocado or > > Meson as the default which were the runners I initially focused on for > > exeter. >=20 > I'm concerned about compatibility. >=20 > I don't actually see a problem with Meson *as a test runner* as it's > widely packaged, but as a build system, it's arguably less compatible > and more complicated than Make is (and we don't need all those features > to build passt). Eh, maybe. More complex, perhaps, but in a number of ways it will tend to be more compatible than make alone, because it has built in a bunch of handling that you'd need to add to make with autoconf or explicit compatibility tests in the Makefile or whatever. > > I now suggest two more options: > >=20 > > 1) Add a tool as part of exeter that will generate a BATS test script > > from an exeter test program. So you'd do something like: > >=20 > > $ exetool --bats > foo.bats > > $ bats foo.bats > >=20 > > This should be pretty easy to do, it's basically what I already have > > for Avocado support, indeed a little easier. For reasons internal to > > exeter, Python is the obvious choice to implement exetool, but I could > > do it in shell if you really don't like that. >=20 > As long as it can reasonably run on all the distributions where we > might want to run tests, I don't see an issue with Python. It depends a > bit on what the required modules are, I would say. I'm not using anything outside the standard library. *looks* argparse, json, subprocess, sys and pathlib. That's for a current draft that generates Avocado job files. I wouldn't anticipate anything extra for bats. > > 2) Hand-roll a minimal exeter runner as part of passt's existing test > > scripts. I'm thinking you could essentially point our test stuff > > at an exeter program as an alternative to pointing it at a file > > with the existing test DSL. > >=20 > > This is more work, but still not too bad. It has the advantage of > > not adding another dependency, and means we could count exeter > > results along with our existing test results in the final summary. >=20 > It also has the advantage of being conceptually simpler in that it > avoids one additional step and one additional "language" (Bats). Yes. > > However, we'd lose parallel execution and filtering. >=20 > Aren't those sort of trivial once you have a "real" programming > language? This option wouldn't have a real programming language. Even in the bats case, the Python would just be generating a bats file, not actually running the tests. In this case the runner would be shell (the tests themselves could be anything). > > Stefano, would you be willing to merge patches which add some basic > > exeter tests using one of these approaches? This would probably just > > be static checked and build tests at this point, as a > > proof-of-concept. >=20 > Sure! As long as current tests keep working, of course. Of course. > > If we can start introducing some exeter tests, the next step would be > > to work on the support library stuff for constructing more complex > > network environments from namespaces. I have draft series with this > > as well, but I was looking at splitting it into another mini-project > > (tentative name "sinte" - Simulated Inter Network Test Environment). >=20 > Neat! That sounds like the juicy part and surely one part we really > miss at this point. I would go as far as proposing PESTO (playground > environment simplifies test orchestration / spurs test opportunities) > but "sinte" sounds good to me as well. I kind of love the "pesto" acronym, but the proposed expansions don't quite work for me though: they don't say anything specifically about networking, and it's really not about test orchestration - this is about how to write a single test. The infuriating thing is I'm pretty sure I came up with a much better tentative name months ago, and then forgot it. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --KYjLLLZziYZk0II/ Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmiSq5kACgkQzQJF27ox 2GfWNQ//XYT0aaHEZVdayrQUhXmhdyGi0/RYV4RtiExMzEXQP0D9yK9k7slw6DwS WMww49RC7Uw/+LOIdJO1tSrlwYL1oCpE57sG3400SSRDAYx7QktoNC3sYZnTkXIR pOAk0gZcKjt1eHKNlV4wc5XtEtkVM/+wj9+DvDvhkqS/XmPSJMZuWSXcsKuwP1ne e/ORu8r+D2Mf1owUnTtd8OqYhwGmQX6lqEXYxJfwvZJq6oiFHq6sajSYANtFzDxY 2buvX3uD/Pq6N9vXmgye4p1Sas/yLtdp4+4kLtYQx3sgBQaz4aa6s99Ijv+qZB4x ZmypVRZkP7BOoVH4RfaM1uvHSpSVgZDP0GZMJT0AAsqHG+ccnkQCMJyjGVJrrf/T nOzRUJkCyN+igk5faCzN29DmqzkPi0mpMtOcb8pS758zIPbpNMgSWVH/VlrLvSA6 PSB4zQ3igyv08jml6BVuZ6lZNWajCaICdvtURWqBIU7KVVZa5zSbd3bmBqSUTQoY Y4QeNUVIvpXlgKUuFi4EcnCkjM49HS7QEJqYyqp8LMnhEmIlige+J1f9VOE40kwg G3GaJLlsgXv7Kaa1v2siVKcCPsQU1ODWa1P96iZT37n3hzXJHPQKzs4s+c3kpHCd aawgXcb9ClSk4KsMwRuM6wLfDZkr3qpAtTS5832rUgZkA58seiw= =fNWT -----END PGP SIGNATURE----- --KYjLLLZziYZk0II/--