Re: [PATCH v6 0/4] New proof-of-concept based exeter tests

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 3729 bytes --]

On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote:
> On Mon,  1 Sep 2025 14:25:11 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > Here's a new approach to building passt tests with exeter.  This new
> > one no longer uses Avocado in the default case, although it would
> > still be possible to manually run the exeter based tests with Avocado.
> > 
> > Here's another draft of my work on testing passt with Avocado and the
> > exeter library I recently created.  It includes Cleber's patch adding
> > some basic Avocado tests and builds on that.
> > 
> > For now this only does simple tests, to show how the integration could
> > work.  It adds some new trivial "smoke tests" and converts the linter
> > and build checks to exeter.  More complex tests will require building
> > the sinte/pesto library we've discussed.  A lot of the work for that
> > already exists in my earlier exeter test series, but it will need some
> > rework to split it into a separate component.
> > 
> > v6:
> >  * Use exeter's new metadata support to print nicer test names
> 
> Thanks, it looks much more readable now.
> 
> And to me it looks ready to merge, but I hit something during testing
> that I'm not quite sure how to solve, assuming it can be considered an
> issue at all.
> 
> Initially, I hadn't upgraded my local copy of exeter, so even smoke
> tests would fail, until it occurred to me that of course I needed to
> drop the 'exeter' directory and 'make exeter' again. It's an issue we
> conceptually already have with mbuto (even though it didn't get
> breaking changes for months now) and with Podman to some extent.

Yes.

> With exeter, I guess we're going to hit that kind of issue pretty
> frequently at least in the near future. So I was wondering: should we
> enforce some form of up-to-date check from test/Makefile?

Good point.  We actually already have a mechanism for this: the pull-%
targets, which we already use for mbuto and podman.  I pull a
dependency on this for the "make bats" target, but not for regular
"make assets" so we didn't update exeter.

I'll respin for this, because I have one other trivial change to make.

It does occur to me that there's another version of the problem too.
Once again we already conceptually have it with mbuto and podman, but
it will be much worse with exeter due to pace of development:

If (for debugging) we check out an old version of the passt source
tree, it will still pull the latest version of exeter, which it might
not be compatible with[0].  The obvious solution - but you might not
like it much - is git submodules.  They (kind of rightly) have a
reputation as being a pain to work with, but they do solve exactly
this problem - qemu uses them heavily and pretty successfully for
this.

> I'm personally fine without it and, given that I plan to update
> test/README.md soon anyway ('make' under test/ is not mentioned at
> all), I can mention this kind of problem as well, so it shouldn't be a
> big deal for others. But I wanted to know if you have thoughts or
> proposals on the matter, before applying this.
> 
> -- 
> Stefano
> 

[0] At some point, I do want to stabilise the exeter interfaces, and
    be concerned with backwards compatibility, which would largely fix
    this.  However, at present I'm still finding stuff that wants to
    change at the exeter level often enough that I think the effort of
    maintaining compatibility would be prohibitive.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]