From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202508 header.b=BV/A79Pa; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 2871C5A0271 for ; Thu, 04 Sep 2025 04:43:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202508; t=1756953788; bh=WChUm93Rvh6iCKiZRLFcz17oGI3TjyXv7zsFYe4NT0c=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=BV/A79PaMj2m1CfnBSYu911w3txNwll+sk//DD/nHv7IrqKGfllpuG/JYoKUHYmFN kFKseoQQMmsSlX+aUwiyCbN2u1hY+9Vb8evps9TQnqjqix1ns2pQpOw6UFCQc8uZwI Safha12oeia/du1BG8XxH0nKDVF0ITopcYQxpjE452Znb7XXilir/AlbbdE+gm3lmO ibrcIBbsQrSCPdGhlUFrrWYWF9aLyw5Ij/epaJUPTDFhDIQf7HC7dAiM1/Elq47GaT HVX0OvWm82IQScXnAtMGQlalbJU1RcA1HTC8p7i53nWRkmXRAQEAJNbzdjHaTVVcL2 QhhfzbNNOurIg== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4cHP0r6hFRz4w9h; Thu, 4 Sep 2025 12:43:08 +1000 (AEST) Date: Thu, 4 Sep 2025 12:42:57 +1000 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH v6 0/4] New proof-of-concept based exeter tests Message-ID: References: <20250901042515.138861-1-david@gibson.dropbear.id.au> <20250903231435.11a1bdab@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="tqB4p0fXBC4dm6gf" Content-Disposition: inline In-Reply-To: <20250903231435.11a1bdab@elisabeth> Message-ID-Hash: RWHLK34FJU52QBG7QTOXZ4DP3RWCJMWJ X-Message-ID-Hash: RWHLK34FJU52QBG7QTOXZ4DP3RWCJMWJ X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --tqB4p0fXBC4dm6gf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote: > On Mon, 1 Sep 2025 14:25:11 +1000 > David Gibson wrote: >=20 > > Here's a new approach to building passt tests with exeter. This new > > one no longer uses Avocado in the default case, although it would > > still be possible to manually run the exeter based tests with Avocado. > >=20 > > Here's another draft of my work on testing passt with Avocado and the > > exeter library I recently created. It includes Cleber's patch adding > > some basic Avocado tests and builds on that. > >=20 > > For now this only does simple tests, to show how the integration could > > work. It adds some new trivial "smoke tests" and converts the linter > > and build checks to exeter. More complex tests will require building > > the sinte/pesto library we've discussed. A lot of the work for that > > already exists in my earlier exeter test series, but it will need some > > rework to split it into a separate component. > >=20 > > v6: > > * Use exeter's new metadata support to print nicer test names >=20 > Thanks, it looks much more readable now. >=20 > And to me it looks ready to merge, but I hit something during testing > that I'm not quite sure how to solve, assuming it can be considered an > issue at all. >=20 > Initially, I hadn't upgraded my local copy of exeter, so even smoke > tests would fail, until it occurred to me that of course I needed to > drop the 'exeter' directory and 'make exeter' again. It's an issue we > conceptually already have with mbuto (even though it didn't get > breaking changes for months now) and with Podman to some extent. Yes. > With exeter, I guess we're going to hit that kind of issue pretty > frequently at least in the near future. So I was wondering: should we > enforce some form of up-to-date check from test/Makefile? Good point. We actually already have a mechanism for this: the pull-% targets, which we already use for mbuto and podman. I pull a dependency on this for the "make bats" target, but not for regular "make assets" so we didn't update exeter. I'll respin for this, because I have one other trivial change to make. It does occur to me that there's another version of the problem too. Once again we already conceptually have it with mbuto and podman, but it will be much worse with exeter due to pace of development: If (for debugging) we check out an old version of the passt source tree, it will still pull the latest version of exeter, which it might not be compatible with[0]. The obvious solution - but you might not like it much - is git submodules. They (kind of rightly) have a reputation as being a pain to work with, but they do solve exactly this problem - qemu uses them heavily and pretty successfully for this. > I'm personally fine without it and, given that I plan to update > test/README.md soon anyway ('make' under test/ is not mentioned at > all), I can mention this kind of problem as well, so it shouldn't be a > big deal for others. But I wanted to know if you have thoughts or > proposals on the matter, before applying this. >=20 > --=20 > Stefano >=20 [0] At some point, I do want to stabilise the exeter interfaces, and be concerned with backwards compatibility, which would largely fix this. However, at present I'm still finding stuff that wants to change at the exeter level often enough that I think the effort of maintaining compatibility would be prohibitive. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --tqB4p0fXBC4dm6gf Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmi4/KAACgkQzQJF27ox 2GeSAg//RdgrY/aBVavNu4FUqj3OSVXtWOl+1cw7wwkqejv4GdxS3O7TnO9CfgAi myKhHCwyWrPH9LvaiDAhoaU4vyl7M+zBLog1lhAe9YqwZr+v13jkxYnNqdfytTl9 aEJ+s/d5bZ5Ff0kltyzksuK4k4kmV/jbd6V57DK3x8/Ij5kywhM06HR5egTaOQFu V9VyQO2jq9CDV3QPxy2Ycjyqo4YZCORUprYBnnKnMWB/5wpsFVuT1JdxB7+1JhRD TXHYhkB4fZa+4JpVGyesSfztdDXDEpEBvYkYOk+6AqWXoriZ+4PhTnFXnMY78n4m 0kpiMl2Cc4j6h93S3VXUUYAZSAZbh/FcV9UPA9XXLZqGHyUoMPv2VT4giw+q94WW cZ/wwHzclrvsoSLl/2oFyqgs3Ys+uGSmVrGbtTHOQhn26Ogfp/2YhKz/bVRErcoV 1OCjUbEFoJDENLjmgomfyyfsgjAaHV++9F7Sko2pnzEUbwCHjGADkekQwiQ3K/g2 uVaLHhguOPM81nDqI+CllPwaB1ohOIwaShdItVeDp4vhwSQXdYM7oQ8joSzrvU+h VFnnPaYAIfFb+hPpKZELWV99p0crHl9aeCZIi4WnYYSoBvuMSD6pyPMzp21epC5l b8OdIYXL5oIjNCQK7r4sK3m9EmRSAbD439WXMMfPlka/z8qq71Q= =7DCH -----END PGP SIGNATURE----- --tqB4p0fXBC4dm6gf--