Exeter testing, next steps

Exeter testing, next steps

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2336 bytes --]

Today I finally had a good look at what the next steps need to be if
we are to use exeter based test "for real" in the passt repo.

I already have draft series which move a bunch of the simpler tests to
exeter.  But in order to actually use these we need some sort of
runner.  We're not tied to that runner, we can easily change - that's
the whole point of exeter - but we need to have something.

AIUI, Stefano is not happy with the idea of using either Avocado or
Meson as the default which were the runners I initially focused on for
exeter.  I now suggest two more options:

1) Add a tool as part of exeter that will generate a BATS test script
   from an exeter test program.  So you'd do something like:

	$ exetool --bats <exeter_test_program> > foo.bats
	$ bats foo.bats

This should be pretty easy to do, it's basically what I already have
for Avocado support, indeed a little easier.  For reasons internal to
exeter, Python is the obvious choice to implement exetool, but I could
do it in shell if you really don't like that.

2) Hand-roll a minimal exeter runner as part of passt's existing test
   scripts.  I'm thinking you could essentially point our test stuff
   at an exeter program as an alternative to pointing it at a file
   with the existing test DSL.

   This is more work, but still not too bad.  It has the advantage of
   not adding another dependency, and means we could count exeter
   results along with our existing test results in the final summary.
   However, we'd lose parallel execution and filtering.


Stefano, would you be willing to merge patches which add some basic
exeter tests using one of these approaches?  This would probably just
be static checked and build tests at this point, as a
proof-of-concept.

If we can start introducing some exeter tests, the next step would be
to work on the support library stuff for constructing more complex
network environments from namespaces.  I have draft series with this
as well, but I was looking at splitting it into another mini-project
(tentative name "sinte" - Simulated Inter Network Test Environment).

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Exeter testing, next steps

[Stefano Brivio]

On Tue, 5 Aug 2025 16:09:34 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> Today I finally had a good look at what the next steps need to be if
> we are to use exeter based test "for real" in the passt repo.

\o/

> I already have draft series which move a bunch of the simpler tests to
> exeter.  But in order to actually use these we need some sort of
> runner.  We're not tied to that runner, we can easily change - that's
> the whole point of exeter - but we need to have something.

I guess as an even earlier first step we could actually have a
hardcoded shell script similar to today's test/run... or am I
misrepresenting the problem?

> AIUI, Stefano is not happy with the idea of using either Avocado or
> Meson as the default which were the runners I initially focused on for
> exeter.

I'm concerned about compatibility.

I don't actually see a problem with Meson *as a test runner* as it's
widely packaged, but as a build system, it's arguably less compatible
and more complicated than Make is (and we don't need all those features
to build passt).

> I now suggest two more options:
> 
> 1) Add a tool as part of exeter that will generate a BATS test script
>    from an exeter test program.  So you'd do something like:
> 
> 	$ exetool --bats <exeter_test_program> > foo.bats
> 	$ bats foo.bats
> 
> This should be pretty easy to do, it's basically what I already have
> for Avocado support, indeed a little easier.  For reasons internal to
> exeter, Python is the obvious choice to implement exetool, but I could
> do it in shell if you really don't like that.

As long as it can reasonably run on all the distributions where we
might want to run tests, I don't see an issue with Python. It depends a
bit on what the required modules are, I would say.

> 2) Hand-roll a minimal exeter runner as part of passt's existing test
>    scripts.  I'm thinking you could essentially point our test stuff
>    at an exeter program as an alternative to pointing it at a file
>    with the existing test DSL.
> 
>    This is more work, but still not too bad.  It has the advantage of
>    not adding another dependency, and means we could count exeter
>    results along with our existing test results in the final summary.

It also has the advantage of being conceptually simpler in that it
avoids one additional step and one additional "language" (Bats).

>    However, we'd lose parallel execution and filtering.

Aren't those sort of trivial once you have a "real" programming
language?

> Stefano, would you be willing to merge patches which add some basic
> exeter tests using one of these approaches?  This would probably just
> be static checked and build tests at this point, as a
> proof-of-concept.

Sure! As long as current tests keep working, of course.

> If we can start introducing some exeter tests, the next step would be
> to work on the support library stuff for constructing more complex
> network environments from namespaces.  I have draft series with this
> as well, but I was looking at splitting it into another mini-project
> (tentative name "sinte" - Simulated Inter Network Test Environment).

Neat! That sounds like the juicy part and surely one part we really
miss at this point. I would go as far as proposing PESTO (playground
environment simplifies test orchestration / spurs test opportunities)
but "sinte" sounds good to me as well.

-- 
Stefano

Re: Exeter testing, next steps

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5382 bytes --]

On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote:
> On Tue, 5 Aug 2025 16:09:34 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > Today I finally had a good look at what the next steps need to be if
> > we are to use exeter based test "for real" in the passt repo.
> 
> \o/
> 
> > I already have draft series which move a bunch of the simpler tests to
> > exeter.  But in order to actually use these we need some sort of
> > runner.  We're not tied to that runner, we can easily change - that's
> > the whole point of exeter - but we need to have something.
> 
> I guess as an even earlier first step we could actually have a
> hardcoded shell script similar to today's test/run... or am I
> misrepresenting the problem?

That's sort of option (2) below.  I suppose we could just have a
shell script that individually runs each exeter test.  I don't love
that idea though because a) exeter tests make no attempt to pretty
print, so will show a lot of logging garbage even on successful runs
and b) it means the script needs to be updated whenever the set of
tests in an exeter program is changed.

> > AIUI, Stefano is not happy with the idea of using either Avocado or
> > Meson as the default which were the runners I initially focused on for
> > exeter.
> 
> I'm concerned about compatibility.
> 
> I don't actually see a problem with Meson *as a test runner* as it's
> widely packaged, but as a build system, it's arguably less compatible
> and more complicated than Make is (and we don't need all those features
> to build passt).

Eh, maybe.  More complex, perhaps, but in a number of ways it will
tend to be more compatible than make alone, because it has built in a
bunch of handling that you'd need to add to make with autoconf or
explicit compatibility tests in the Makefile or whatever.

> > I now suggest two more options:
> > 
> > 1) Add a tool as part of exeter that will generate a BATS test script
> >    from an exeter test program.  So you'd do something like:
> > 
> > 	$ exetool --bats <exeter_test_program> > foo.bats
> > 	$ bats foo.bats
> > 
> > This should be pretty easy to do, it's basically what I already have
> > for Avocado support, indeed a little easier.  For reasons internal to
> > exeter, Python is the obvious choice to implement exetool, but I could
> > do it in shell if you really don't like that.
> 
> As long as it can reasonably run on all the distributions where we
> might want to run tests, I don't see an issue with Python. It depends a
> bit on what the required modules are, I would say.

I'm not using anything outside the standard library. *looks* argparse,
json, subprocess, sys and pathlib.  That's for a current draft that
generates Avocado job files.  I wouldn't anticipate anything extra for
bats.

> > 2) Hand-roll a minimal exeter runner as part of passt's existing test
> >    scripts.  I'm thinking you could essentially point our test stuff
> >    at an exeter program as an alternative to pointing it at a file
> >    with the existing test DSL.
> > 
> >    This is more work, but still not too bad.  It has the advantage of
> >    not adding another dependency, and means we could count exeter
> >    results along with our existing test results in the final summary.
> 
> It also has the advantage of being conceptually simpler in that it
> avoids one additional step and one additional "language" (Bats).

Yes.

> >    However, we'd lose parallel execution and filtering.
> 
> Aren't those sort of trivial once you have a "real" programming
> language?

This option wouldn't have a real programming language.  Even in the
bats case, the Python would just be generating a bats file, not
actually running the tests.  In this case the runner would be shell
(the tests themselves could be anything).

> > Stefano, would you be willing to merge patches which add some basic
> > exeter tests using one of these approaches?  This would probably just
> > be static checked and build tests at this point, as a
> > proof-of-concept.
> 
> Sure! As long as current tests keep working, of course.

Of course.

> > If we can start introducing some exeter tests, the next step would be
> > to work on the support library stuff for constructing more complex
> > network environments from namespaces.  I have draft series with this
> > as well, but I was looking at splitting it into another mini-project
> > (tentative name "sinte" - Simulated Inter Network Test Environment).
> 
> Neat! That sounds like the juicy part and surely one part we really
> miss at this point. I would go as far as proposing PESTO (playground
> environment simplifies test orchestration / spurs test opportunities)
> but "sinte" sounds good to me as well.

I kind of love the "pesto" acronym, but the proposed expansions don't
quite work for me though: they don't say anything specifically about
networking, and it's really not about test orchestration - this is
about how to write a single test.

The infuriating thing is I'm pretty sure I came up with a much better
tentative name months ago, and then forgot it.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Exeter testing, next steps

[Stefano Brivio]

On Wed, 6 Aug 2025 11:11:24 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote:
> >
> > > If we can start introducing some exeter tests, the next step would be
> > > to work on the support library stuff for constructing more complex
> > > network environments from namespaces.  I have draft series with this
> > > as well, but I was looking at splitting it into another mini-project
> > > (tentative name "sinte" - Simulated Inter Network Test Environment).  
> > 
> > Neat! That sounds like the juicy part and surely one part we really
> > miss at this point. I would go as far as proposing PESTO (playground
> > environment simplifies test orchestration / spurs test opportunities)
> > but "sinte" sounds good to me as well.  
> 
> I kind of love the "pesto" acronym, but the proposed expansions don't
> quite work for me though: they don't say anything specifically about
> networking, and it's really not about test orchestration - this is
> about how to write a single test.

I couldn't come up with much better (and didn't get to actually review
the patches) yet, but I just wanted to quickly mention that by
"orchestration" I was referring to juggling namespaces and virtual
machines (at some point...?) with virtual links around, within a single
test.

And maybe at some point networking could be marginal in the whole thing.

> The infuriating thing is I'm pretty sure I came up with a much better
> tentative name months ago, and then forgot it.

I know you meant it for something else, but I'm wondering if you're
referring to NUDEL perhaps.

-- 
Stefano

Re: Exeter testing, next steps

[Stefano Brivio]

On Wed, 13 Aug 2025 20:04:23 +0200
Stefano Brivio <sbrivio@redhat.com> wrote:

> On Wed, 6 Aug 2025 11:11:24 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > The infuriating thing is I'm pretty sure I came up with a much better
> > tentative name months ago, and then forgot it.  
> 
> I know you meant it for something else, but I'm wondering if you're
> referring to NUDEL perhaps.

And, it just came back to my mind: you had at some point climns, and
then perhaps PASTEL / PASTLE...?

-- 
Stefano

Re: Exeter testing, next steps

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1126 bytes --]

On Thu, Aug 14, 2025 at 06:29:28AM +0200, Stefano Brivio wrote:
> On Wed, 13 Aug 2025 20:04:23 +0200
> Stefano Brivio <sbrivio@redhat.com> wrote:
> 
> > On Wed, 6 Aug 2025 11:11:24 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> > 
> > > The infuriating thing is I'm pretty sure I came up with a much better
> > > tentative name months ago, and then forgot it.  
> > 
> > I know you meant it for something else, but I'm wondering if you're
> > referring to NUDEL perhaps.
> 
> And, it just came back to my mind: you had at some point climns,

Ah, I had forgotten "climns".  I don't think that's the specific one I
forgot, but it's ok.

> and
> then perhaps PASTEL / PASTLE...?

That I had in mind for something different.  Specifically, an "engine"
version of passt/pasta which needs to be supplied with detailed
forwarding configuration - passt and pasta would become front-ends to
that.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Exeter testing, next steps

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3044 bytes --]

On Wed, Aug 13, 2025 at 08:04:23PM +0200, Stefano Brivio wrote:
> On Wed, 6 Aug 2025 11:11:24 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote:
> > >
> > > > If we can start introducing some exeter tests, the next step would be
> > > > to work on the support library stuff for constructing more complex
> > > > network environments from namespaces.  I have draft series with this
> > > > as well, but I was looking at splitting it into another mini-project
> > > > (tentative name "sinte" - Simulated Inter Network Test Environment).  
> > > 
> > > Neat! That sounds like the juicy part and surely one part we really
> > > miss at this point. I would go as far as proposing PESTO (playground
> > > environment simplifies test orchestration / spurs test opportunities)
> > > but "sinte" sounds good to me as well.  
> > 
> > I kind of love the "pesto" acronym, but the proposed expansions don't
> > quite work for me though: they don't say anything specifically about
> > networking, and it's really not about test orchestration - this is
> > about how to write a single test.
> 
> I couldn't come up with much better (and didn't get to actually review
> the patches) yet, but I just wanted to quickly mention that by
> "orchestration" I was referring to juggling namespaces and virtual
> machines (at some point...?) with virtual links around, within a single
> test.

That makes sense, but I don't think that meaning is obvious within the
name alone.

> And maybe at some point networking could be marginal in the whole thing.

Hm.  Maybe.  At this point handling network environments and
specifically *inter* network environments is really my focus though.

> > The infuriating thing is I'm pretty sure I came up with a much better
> > tentative name months ago, and then forgot it.
> 
> I know you meant it for something else, but I'm wondering if you're
> referring to NUDEL perhaps.

No, I think I had a tentative name specifically for this network
simulation thingy.  *brainstorms*

One idea I had was

"instead" - Inter Network Simulated Test Environment <something
something>.  "with Arbitary Devices" maybe?

That might be the one I'm thinking of, but I think I had one I liked
better.

Had a chat with Gemini and came up with a handful more options:

"mimic" - Multi-host Inter-network Mock Infrastructure Construct

"isthmus" - Inter-network Simulation & Test Harness for Multiple
Underlying Systems

"ross" - "Routing & Orchestration Simulation System"

Or, maybe..

"tunbridge" - <backronym pending>

Tunbridge (like Exeter) is a town in Tasmania, and a bridge of tun
devices would likely be a common construct to build with it (though
not as common as a bridge of veths, admittedly).

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Exeter testing, next steps

[Stefano Brivio]

On Thu, 14 Aug 2025 14:57:43 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, Aug 13, 2025 at 08:04:23PM +0200, Stefano Brivio wrote:
> > On Wed, 6 Aug 2025 11:11:24 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote:  
> > > >  
> > > > > If we can start introducing some exeter tests, the next step would be
> > > > > to work on the support library stuff for constructing more complex
> > > > > network environments from namespaces.  I have draft series with this
> > > > > as well, but I was looking at splitting it into another mini-project
> > > > > (tentative name "sinte" - Simulated Inter Network Test Environment).    
> > > > 
> > > > Neat! That sounds like the juicy part and surely one part we really
> > > > miss at this point. I would go as far as proposing PESTO (playground
> > > > environment simplifies test orchestration / spurs test opportunities)
> > > > but "sinte" sounds good to me as well.    
> > > 
> > > I kind of love the "pesto" acronym, but the proposed expansions don't
> > > quite work for me though: they don't say anything specifically about
> > > networking, and it's really not about test orchestration - this is
> > > about how to write a single test.  
> > 
> > I couldn't come up with much better (and didn't get to actually review
> > the patches) yet, but I just wanted to quickly mention that by
> > "orchestration" I was referring to juggling namespaces and virtual
> > machines (at some point...?) with virtual links around, within a single
> > test.  
> 
> That makes sense, but I don't think that meaning is obvious within the
> name alone.
> 
> > And maybe at some point networking could be marginal in the whole thing.  
> 
> Hm.  Maybe.  At this point handling network environments and
> specifically *inter* network environments is really my focus though.
> 
> > > The infuriating thing is I'm pretty sure I came up with a much better
> > > tentative name months ago, and then forgot it.  
> > 
> > I know you meant it for something else, but I'm wondering if you're
> > referring to NUDEL perhaps.  
> 
> No, I think I had a tentative name specifically for this network
> simulation thingy.  *brainstorms*
> 
> One idea I had was
> 
> "instead" - Inter Network Simulated Test Environment <something
> something>.  "with Arbitary Devices" maybe?  

averts disaster?

and description? deployment?

> That might be the one I'm thinking of, but I think I had one I liked
> better.
> 
> Had a chat with Gemini and came up with a handful more options:
> 
> "mimic" - Multi-host Inter-network Mock Infrastructure Construct
> 
> "isthmus" - Inter-network Simulation & Test Harness for Multiple
> Underlying Systems
> 
> "ross" - "Routing & Orchestration Simulation System"
> 
> Or, maybe..
> 
> "tunbridge" - <backronym pending>
> 
> Tunbridge (like Exeter) is a town in Tasmania, and a bridge of tun
> devices would likely be a common construct to build with it (though
> not as common as a bridge of veths, admittedly).

It might be a bit confusing, but I think it's worth the pun.

Or... if you want to stick to pasta that doesn't stick, I could propose
"aldente", arbitrary layers deliver effective network testing
environments.

It's two words (https://en.wiktionary.org/wiki/al_dente) actually (and
"to" + il "the" -> al), but... Alfresco already broke the taboo for us?

-- 
Stefano

Re: Exeter testing, next steps

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4716 bytes --]

On Thu, Aug 14, 2025 at 10:00:30AM +0200, Stefano Brivio wrote:
> On Thu, 14 Aug 2025 14:57:43 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > On Wed, Aug 13, 2025 at 08:04:23PM +0200, Stefano Brivio wrote:
> > > On Wed, 6 Aug 2025 11:11:24 +1000
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >   
> > > > On Tue, Aug 05, 2025 at 09:52:41AM +0200, Stefano Brivio wrote:  
> > > > >  
> > > > > > If we can start introducing some exeter tests, the next step would be
> > > > > > to work on the support library stuff for constructing more complex
> > > > > > network environments from namespaces.  I have draft series with this
> > > > > > as well, but I was looking at splitting it into another mini-project
> > > > > > (tentative name "sinte" - Simulated Inter Network Test Environment).    
> > > > > 
> > > > > Neat! That sounds like the juicy part and surely one part we really
> > > > > miss at this point. I would go as far as proposing PESTO (playground
> > > > > environment simplifies test orchestration / spurs test opportunities)
> > > > > but "sinte" sounds good to me as well.    
> > > > 
> > > > I kind of love the "pesto" acronym, but the proposed expansions don't
> > > > quite work for me though: they don't say anything specifically about
> > > > networking, and it's really not about test orchestration - this is
> > > > about how to write a single test.  
> > > 
> > > I couldn't come up with much better (and didn't get to actually review
> > > the patches) yet, but I just wanted to quickly mention that by
> > > "orchestration" I was referring to juggling namespaces and virtual
> > > machines (at some point...?) with virtual links around, within a single
> > > test.  
> > 
> > That makes sense, but I don't think that meaning is obvious within the
> > name alone.
> > 
> > > And maybe at some point networking could be marginal in the whole thing.  
> > 
> > Hm.  Maybe.  At this point handling network environments and
> > specifically *inter* network environments is really my focus though.
> > 
> > > > The infuriating thing is I'm pretty sure I came up with a much better
> > > > tentative name months ago, and then forgot it.  
> > > 
> > > I know you meant it for something else, but I'm wondering if you're
> > > referring to NUDEL perhaps.  
> > 
> > No, I think I had a tentative name specifically for this network
> > simulation thingy.  *brainstorms*
> > 
> > One idea I had was
> > 
> > "instead" - Inter Network Simulated Test Environment <something
> > something>.  "with Arbitary Devices" maybe?  
> 
> averts disaster?
> 
> and description? deployment?

Also possibilities.

> > That might be the one I'm thinking of, but I think I had one I liked
> > better.
> > 
> > Had a chat with Gemini and came up with a handful more options:
> > 
> > "mimic" - Multi-host Inter-network Mock Infrastructure Construct
> > 
> > "isthmus" - Inter-network Simulation & Test Harness for Multiple
> > Underlying Systems
> > 
> > "ross" - "Routing & Orchestration Simulation System"
> > 
> > Or, maybe..
> > 
> > "tunbridge" - <backronym pending>
> > 
> > Tunbridge (like Exeter) is a town in Tasmania, and a bridge of tun
> > devices would likely be a common construct to build with it (though
> > not as common as a bridge of veths, admittedly).
> 
> It might be a bit confusing, but I think it's worth the pun.

I'm leaning towards this one.  I like the pun quite a lot, and unlike
some of the others, my searches aren't coming across anything remotely
likely to cause a name conflict.

"Test Users of Networks with Bridges, Routers, Interconnects anD
General Environments"

"Try Universal Nodes with Bridges and Routers in Isolated Deployed
General Emulator"

"network emulater with TUNnels and BRIDGEs"

Or just give up on a backronym and let the name stand alone.

> Or... if you want to stick to pasta that doesn't stick, I could propose
> "aldente", arbitrary layers deliver effective network testing
> environments.

Also pretty good.  I'd probably go with "arbitrary links...".

..and there is another project with the name

https://github.com/AppHouseKitchen/AlDente-Battery_Care_and_Monitoring

> It's two words (https://en.wiktionary.org/wiki/al_dente) actually (and
> "to" + il "the" -> al), but... Alfresco already broke the taboo for us?

I'm aware; fwiw, I think it's usually styled as two words in
Australian cookbooks and the like.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]