Re: [PATCH v6 0/4] New proof-of-concept based exeter tests

[Stefano Brivio <sbrivio@redhat.com>]

On Thu, 4 Sep 2025 12:42:57 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote:
> > On Mon,  1 Sep 2025 14:25:11 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > Here's a new approach to building passt tests with exeter.  This new
> > > one no longer uses Avocado in the default case, although it would
> > > still be possible to manually run the exeter based tests with Avocado.
> > > 
> > > Here's another draft of my work on testing passt with Avocado and the
> > > exeter library I recently created.  It includes Cleber's patch adding
> > > some basic Avocado tests and builds on that.
> > > 
> > > For now this only does simple tests, to show how the integration could
> > > work.  It adds some new trivial "smoke tests" and converts the linter
> > > and build checks to exeter.  More complex tests will require building
> > > the sinte/pesto library we've discussed.  A lot of the work for that
> > > already exists in my earlier exeter test series, but it will need some
> > > rework to split it into a separate component.
> > > 
> > > v6:
> > >  * Use exeter's new metadata support to print nicer test names  
> > 
> > Thanks, it looks much more readable now.
> > 
> > And to me it looks ready to merge, but I hit something during testing
> > that I'm not quite sure how to solve, assuming it can be considered an
> > issue at all.
> > 
> > Initially, I hadn't upgraded my local copy of exeter, so even smoke
> > tests would fail, until it occurred to me that of course I needed to
> > drop the 'exeter' directory and 'make exeter' again. It's an issue we
> > conceptually already have with mbuto (even though it didn't get
> > breaking changes for months now) and with Podman to some extent.  
> 
> Yes.
> 
> > With exeter, I guess we're going to hit that kind of issue pretty
> > frequently at least in the near future. So I was wondering: should we
> > enforce some form of up-to-date check from test/Makefile?  
> 
> Good point.  We actually already have a mechanism for this: the pull-%
> targets, which we already use for mbuto and podman.  I pull a
> dependency on this for the "make bats" target, but not for regular
> "make assets" so we didn't update exeter.

Ah, right, of course. That actually looks good enough to me.

> I'll respin for this, because I have one other trivial change to make.
> 
> It does occur to me that there's another version of the problem too.
> Once again we already conceptually have it with mbuto and podman, but
> it will be much worse with exeter due to pace of development:
> 
> If (for debugging) we check out an old version of the passt source
> tree, it will still pull the latest version of exeter, which it might
> not be compatible with[0].  The obvious solution - but you might not
> like it much - is git submodules.  They (kind of rightly) have a
> reputation as being a pain to work with, but they do solve exactly
> this problem - qemu uses them heavily and pretty successfully for
> this.

To me it looks like a solution adding permanent complexity to solve a
problem that's, in practice, temporary.

Once exeter is stable enough, we'll just require a recent version. And
before that day, I doubt we'll want to debug / bisect issues by running
any of the exeter-based tests.

Every time I'm debugging a possible regression I come up with some
stand-alone reproducer or manual test. If the test suite had caught the
regression at a given point, then we wouldn't have hit the regression
in the first place.

> > I'm personally fine without it and, given that I plan to update
> > test/README.md soon anyway ('make' under test/ is not mentioned at
> > all), I can mention this kind of problem as well, so it shouldn't be a
> > big deal for others. But I wanted to know if you have thoughts or
> > proposals on the matter, before applying this.
> > 
> > -- 
> > Stefano
> >   
> 
> [0] At some point, I do want to stabilise the exeter interfaces, and
>     be concerned with backwards compatibility, which would largely fix
>     this.  However, at present I'm still finding stuff that wants to
>     change at the exeter level often enough that I think the effort of
>     maintaining compatibility would be prohibitive.

Right, I wouldn't care, I really can't see any practical need.

In practice, the kind of breakages we have right now are something much
more basic: outdated test/README.md and broken test/Makefile which I
couldn't find a moment to take care of, yet. And I still need to
killall -9 nstool every time.

-- 
Stefano