From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202510 header.b=agH2t7cD; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 6A5305A0619 for ; Fri, 31 Oct 2025 06:21:10 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202510; t=1761888067; bh=tu27spRMcg3Jzdw5u88ytIpG5kb4lS9+57brPbIpCy8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=agH2t7cDGlF8C0jj0pJKQx1ooXpVegIOXINpLBaqskPkOay0yZtNMZiF+I+wXkl/c lxv8alLRM6C//gRqcYJEncySpDmOn5cLmp+HyNIn9o8eUwN6yeFkhNOhniKF+CbN5D wrjuV+tHIRNuJYcta8yJm2pzGRNmLTdPtBPMoFDeTTDzLE3H+HTRGZeLkYBf459Zax WUsOUlTBZ1I8qDp0NxdevAVAKB8wgNiMH742suf3CpKYm1Xdb8kvlIXxFGgvQ/ZGIg W0YQkNZWxyuRCjewqCDqUEc9imTsk77Ynj8WW5prMpQzVngLPU19iYxhdICRhn/Qqh xvB93jVHM83Gg== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4cyTpq4wVVz4xGn; Fri, 31 Oct 2025 16:21:07 +1100 (AEDT) Date: Fri, 31 Oct 2025 16:21:04 +1100 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH 3/3] test: Re-implement pasta NDP tests using tunbridge & exeter Message-ID: References: <20251002075708.461931-4-david@gibson.dropbear.id.au> <20251007220110.3c8bf21c@elisabeth> <20251009010248.1ebc1a50@elisabeth> <20251010012023.11f3a517@elisabeth> <20251016233119.4f5e155e@elisabeth> <20251029001722.12e7fe07@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="R4Y234D0F3ccSCI3" Content-Disposition: inline In-Reply-To: <20251029001722.12e7fe07@elisabeth> Message-ID-Hash: TGBEXR2CZEKOO5JKS67MPM3YP3ZFFTYW X-Message-ID-Hash: TGBEXR2CZEKOO5JKS67MPM3YP3ZFFTYW X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --R4Y234D0F3ccSCI3 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 29, 2025 at 12:17:22AM +0100, Stefano Brivio wrote: > On Fri, 17 Oct 2025 15:30:33 +1100 > David Gibson wrote: > > On Thu, Oct 16, 2025 at 11:31:19PM +0200, Stefano Brivio wrote: > > > On Fri, 10 Oct 2025 13:17:13 +1100 > > > David Gibson wrote: > > > =20 > > > > On Fri, Oct 10, 2025 at 01:20:23AM +0200, Stefano Brivio wrote: =20 > > > > > On Thu, 9 Oct 2025 15:47:01 +1100 > > > > > David Gibson wrote: > > > > > =20 > > > > > > On Thu, Oct 09, 2025 at 01:02:48AM +0200, Stefano Brivio wrote:= =20 > > > > > > > On Wed, 8 Oct 2025 13:32:27 +1100 > > > > > > > David Gibson wrote: > > > > > > > =20 > > > > > > > > On Tue, Oct 07, 2025 at 10:01:10PM +0200, Stefano Brivio wr= ote: =20 > > > > > > > > > On Thu, 2 Oct 2025 17:57:08 +1000 > > > > > > > > > David Gibson wrote: > > > > > > > > > =20 > > > > > > > > > > Convert the pasta NDP tests from shell and our own DSL = to Python using > > > > > > > > > > the exeter test protocol and tunbridge network simulati= on library. > > > > > > > > > >=20 > > > > > > > > > > Signed-off-by: David Gibson > > > > > > > > > > --- > > > > > > > > > > test/Makefile | 2 +- > > > > > > > > > > test/pasta/dhcp | 5 ++++ > > > > > > > > > > test/pasta/ndp.py | 59 ++++++++++++++++++++++++++= ++++++++++++++++ > > > > > > > > > > test/run | 6 +++-- > > > > > > > > > > test/tasst/__init__.py | 4 +++ > > > > > > > > > > test/tasst/pasta.py | 40 ++++++++++++++++++++++++++= ++ > > > > > > > > > > 6 files changed, 113 insertions(+), 3 deletions(-) > > > > > > > > > > create mode 100755 test/pasta/ndp.py > > > > > > > > > > create mode 100644 test/tasst/pasta.py > > > > > > > > > >=20 > > > > > > > > > > diff --git a/test/Makefile b/test/Makefile > > > > > > > > > > index f66c7e7e..95e3d75e 100644 > > > > > > > > > > --- a/test/Makefile > > > > > > > > > > +++ b/test/Makefile > > > > > > > > > > @@ -67,7 +67,7 @@ ASSETS =3D $(DOWNLOAD_ASSETS) $(LOCAL= _ASSETS) > > > > > > > > > > =20 > > > > > > > > > > EXETER_SH =3D smoke/smoke.sh build/static_checkers.sh > > > > > > > > > > EXETER_PYPATH =3D exeter/py3:tunbridge/:. > > > > > > > > > > -EXETER_PYTHON =3D smoke/smoke.py build/build.py > > > > > > > > > > +EXETER_PYTHON =3D smoke/smoke.py build/build.py pasta/= ndp.py > > > > > > > > > > EXETER_BATS =3D $(EXETER_SH:%=3D%.bats) $(EXETER_PYTHO= N:%=3D%.bats) > > > > > > > > > > BATS_FILES =3D $(EXETER_BATS) \ > > > > > > > > > > podman/test/system/505-networking-pasta.bats > > > > > > > > > > diff --git a/test/pasta/dhcp b/test/pasta/dhcp > > > > > > > > > > index e1c66be6..61279fbf 100644 > > > > > > > > > > --- a/test/pasta/dhcp > > > > > > > > > > +++ b/test/pasta/dhcp > > > > > > > > > > @@ -18,6 +18,11 @@ test Interface name > > > > > > > > > > nsout IFNAME ip -j link show | jq -rM '.[] | select(.l= ink_type =3D=3D "ether").ifname' > > > > > > > > > > check [ -n "__IFNAME__" ] > > > > > > > > > > =20 > > > > > > > > > > +# Bring up the interface > > > > > > > > > > +ns ip link set dev __IFNAME__ up > > > > > > > > > > +# Wait for SLAAC & DAD to complete > > > > > > > > > > +ns while ! ip -j -6 addr show dev __IFNAME__ | jq -e '= =2E[].addr_info.[] | select(.protocol =3D=3D "kernel_ra")'; do sleep 0.1; d= one > > > > > > > > > > + > > > > > > > > > > test DHCP: address > > > > > > > > > > ns /sbin/dhclient -4 --no-pid __IFNAME__ > > > > > > > > > > nsout ADDR ip -j -4 addr show|jq -rM '.[] | select(.if= name =3D=3D "__IFNAME__").addr_info[0].local' > > > > > > > > > > diff --git a/test/pasta/ndp.py b/test/pasta/ndp.py > > > > > > > > > > new file mode 100755 > > > > > > > > > > index 00000000..8c7ce31e > > > > > > > > > > --- /dev/null > > > > > > > > > > +++ b/test/pasta/ndp.py > > > > > > > > > > @@ -0,0 +1,59 @@ > > > > > > > > > > +#! /usr/bin/env python3 > > > > > > > > > > +# > > > > > > > > > > +# SPDX-License-Identifier: GPL-2.0-or-later > > > > > > > > > > +# > > > > > > > > > > +# test/pasta/ndp.py - pasta NDP functionality > > > > > > > > > > +# > > > > > > > > > > +# Copyright Red Hat > > > > > > > > > > +# Author: David Gibson > > > > > > > > > > + > > > > > > > > > > +import contextlib > > > > > > > > > > +import dataclasses > > > > > > > > > > +from typing import Iterator > > > > > > > > > > + > > > > > > > > > > +import exeter > > > > > > > > > > +import tunbridge > > > > > > > > > > +import tasst > > > > > > > > > > + > > > > > > > > > > + > > > > > > > > > > +@dataclasses.dataclass > > > > > > > > > > +class UnconfiguredScenario(exeter.Scenario): > > > > > > > > > > + """Tests for a pasta instance without --config-net= """ > > > > > > > > > > + > > > > > > > > > > + host: tunbridge.Site > > > > > > > > > > + guest: tunbridge.Site > > > > > > > > > > + ifname: str > > > > > > > > > > + addr6: tunbridge.ip.AddrMask6 > > > > > > > > > > + gw6: tunbridge.ip.Addr6 =20 > > > > > > > > >=20 > > > > > > > > > Until this point, it looks like stuff I can happily copy = and paste, > > > > > > > > > and grasp, even. But then: > > > > > > > > > =20 > > > > > > > > > > + @exeter.scenariotest > > > > > > > > > > + def test_ifname(self) -> None: > > > > > > > > > > + ifs =3D tunbridge.ip.ifs(self.guest) > > > > > > > > > > + exeter.assert_eq(set(ifs), {'lo', self.ifname}= ) =20 > > > > > > > > >=20 > > > > > > > > > ...why does a "Scenario" have a .ifname? =20 > > > > > > > >=20 > > > > > > > > Yeah, the readability of the Scenario mechanism was somethi= ng I was > > > > > > > > particularly concerned about. I think the concept is valua= ble, but > > > > > > > > I'm very open to different ways of naming or organising it,= if we can > > > > > > > > up with something better. =20 > > > > > > >=20 > > > > > > > From the description you give below, the name seems to fit. > > > > > > > =20 > > > > > > > > A "Scenario" (specifically a subclass of exeter.Scenario) i= s a group > > > > > > > > of tests with a common set of parameters. In this case > > > > > > > > UnconfiguredScenario is a bunch of tests about the behaviou= r of pasta > > > > > > > > without --config-net. Each of those tests has access to th= e host and > > > > > > > > guest sites, the expected interface name, address and gatew= ay in the > > > > > > > > guest - that is, the contents of an UncofiguredScenario ins= tance. =20 > > > > > > >=20 > > > > > > > I'm not sure if I understand this correctly, but if each gues= t has a > > > > > > > single interface, that sounds a bit limiting. =20 > > > > > >=20 > > > > > > Sorry, to be clear: a Scenario in the general sense can contain > > > > > > whatever parameters you like. This *particular* Scenario - > > > > > > UnconfiguredScenario - has just those things, because those are= all > > > > > > that its tests require. =20 > > > > >=20 > > > > > Ah, okay. Still, if I now want to take UnconfiguredScenario and a= dd a > > > > > couple of dummy interfaces to it for a quick test, I guess I have= the > > > > > choice to either do that with some "external" hack, or... copy and > > > > > rename it, so that it doesn't affect all the usages? =20 > > > >=20 > > > > No. A Scenario instance isn't responsible for managing the simulat= ed > > > > environment - that's the setup function - it's just conveying the > > > > information about it that the tests need. So, you can make a setup > > > > function that adds the dummy interfaces, and still yield an > > > > UnconfiguredScenario. It doesn't need to have information about the > > > > dummy interfaces because the tests carried by UnconfiguredScenario > > > > don't care about them. =20 > > >=20 > > > Oh, sorry, it's a class, of course, I see now. > > > =20 > > > > The scenario mechanism does several things: > > > > 1) Groups together some related (parameterized) tests > > > > 2) Allows all of those tests to be registered at once > > > > 3) Provides a mechanism for providing a bunch of information to > > > > those tests (without requiring them each to have a large set of > > > > direct parameters) > > > >=20 > > > > I'm aware that doing those things with the same construct may be > > > > confusing - it's just ways of doing them separately also seem > > > > confusing and/or awkward in their own ways. Maybe there's a better > > > > way, but I haven't spotted it yet. =20 > > >=20 > > > It really is confusing to me, but the description above is rather cle= ar > > > so I'll try to propose something once I get to write some kind of set= up > > > function and test cases myself. =20 > >=20 > > Makes sense. > >=20 > > > > > > > Actually, I think any abstraction that doesn't offer arbitrar= y sets of > > > > > > > (and relationships between) the objects shown via netlink (or= , at > > > > > > > least, namespaces, links, routes, addresses, neighbours) migh= t be > > > > > > > limiting and not generic enough. =20 > > > > > >=20 > > > > > > Absolutely, and the abstraction does allow that. > > > > > > =20 > > > > > > > > That instance describes a real (simulated) environment in w= hich we can > > > > > > > > run those tests. > > > > > > > >=20 > > > > > > > > You use this by supplying a function which sets things up, = then yields > > > > > > > > an UnconfiguredScenario instance describing what it set up.= exeter > > > > > > > > will run all of the UnconfiguredScenario tests on the envir= onment the > > > > > > > > setup function created, each one as a separate test case. = =20 > > > > > > >=20 > > > > > > > This part is now clear to me, and I think it's not complicate= d to grasp > > > > > > > the concept vaguely but enough to copy, paste, and modify cod= e doing > > > > > > > this. =20 > > > > > >=20 > > > > > > Ok. > > > > > > =20 > > > > > > > It would be even better to hide this entirely, because "yield= ing a > > > > > > > scenario" is a Python thing. In general, there's an imperativ= e part in > > > > > > > all this (bordering functional programming, but still, not de= scriptive) > > > > > > > which I struggle to see as beneficial. > > > > > > >=20 > > > > > > > Here the tasks at hand are, roughly: > > > > > > >=20 > > > > > > > 1. represent two network namespaces, with two interfaces each= (loopback > > > > > > > and non-loopback), with pasta connecting one of the interf= aces of the > > > > > > > inner one =20 > > > > > >=20 > > > > > > There's a bit more to it than that - we need to specify the hos= t's > > > > > > routing setup, because that will affect what pasta does. That'= s what > > > > > > simple_host() is about, creating a host with the single gateway > > > > > > routing that's our easiest / most common case. =20 > > > > >=20 > > > > > Okay, sure, by "interfaces" I meant configured interfaces with > > > > > addresses and a default route, too. But that doesn't really modif= y my > > > > > point, that is: > > > > > =20 > > > > > > > 2. bring up one of the interfaces > > > > > > >=20 > > > > > > > 3. compare addresses > > > > > > >=20 > > > > > > > ...and doing 1. like that is simply not... intuitive, I think= =2E =20 > > > > > >=20 > > > > > > I'm not really clear on what you're getting at here. There is = an > > > > > > unavoidable tradeoff here between obviousness for a single case, > > > > > > versus reuseability for multiple related cases. Is it just tha= t some > > > > > > of the relevant setup is hidden inside simple_host() that's the > > > > > > problem? Or is it something else? =20 > > > > >=20 > > > > > ...yes, one part is that it's hidden. Another part are, specifica= lly, > > > > > these lines: > > > > >=20 > > > > > host: tunbridge.Site > > > > > guest: tunbridge.Site > > > > > ifname: str > > > > >=20 > > > > > [...] > > > > >=20 > > > > > @exeter.scenariotest > > > > > def test_ifname(self) -> None: > > > > >=20 > > > > > [...] > > > > >=20 > > > > > None of these clearly links to "two network namespaces: A, with > > > > > interface a1 and address x1, ...". =20 > > > >=20 > > > > Fair. This needs a docstring explaining the parameters / fields. = =20 > > >=20 > > > That might help a tiny bit but I think the syntax and notations are > > > kind of self-explanatory. > > >=20 > > > My concern is at a more conceptual level, and it's better summarised > > > below, but here, specifically, we're writing: > > >=20 > > > host: tunbridge.Site > > >=20 > > > to say: > > >=20 > > > give me the "host" network namespace =20 > >=20 > > What's "me" in this sentence? >=20 > The writer of the test case, that is: So, I'm pretty sure we have a fundamental misunderstanding here, but I'm not sure on whose part. That line is not saying "give me" anything. It's a type declaration saying that an 'UnconfiguredScenario' instance includes a field 'host' of type tunbridge.Site. That field is used to store the host namespace; the docstring should say that, but doesn't yet. The tests (methods of UnconfiguredScenario) can then reference the host namespace as 'self.host'. The setup function is responsible for populating the field - that's the 'host=3Dsimh.site' in simh_pasta_setup() below. > > > and to say that, in my ideal world, I would probably go for something > > > on the line(s) of: > > >=20 > > > A =20 > >=20 > > Not really following, I hope it will make more sense to me below. >=20 > if "host" is a site in "host: tunbridge.Site", which I'm calling "A" > for brevity, I would like to say that with: >=20 > A >=20 > instead of: >=20 > A: tunbridge.Site In the test? So it's already 'self.host', or 'self.A' if you prefer. You only need the type annotation when defining a new Scenario subclass. Or we could disable mypy and not even need it then, but I think type checking catches enough bugs to make it worthwhile. > > > > > I understand this is probably very close to the bare minimum you = can > > > > > get by modelling this all with actual code, and that's why I think > > > > > actual (imperative/functional) code is usually not used to > > > > > model/describe things. =20 > > > >=20 > > > > Imperative/functional code as opposed to..? =20 > > >=20 > > > ...declarative. =20 > >=20 > > Ok, I've generally seen functional programming put under the > > declarative heading. Or as my exam invigilator famously said once > > "Declaraytive Programming Paradijjums". > >=20 > > So, I guess that makes me unclear what does and doesn't count as > > declarative for you. >=20 > Hmm yes I see your point. I would have said that "purely functional > programming" is declarative, and "functional programming" is a mix of > declarative and imperative (in practice). Eh, true. Kind of depends if you're talking Haskell or, say, Emacs Lisp. And even in Haskell 'do' constructs are quasi-imperative. > In any case, I guess I later clarified this aspect. >=20 > > [snip] > > > > > > Syntax certainly isn't irrelevant, but so far I haven't grasped= what > > > > > > you dislike about the function syntax versus a specialized gram= mer. > > > > > > Is it: > > > > > > - The irritating silly parentheses? > > > > > > - Longish (qualified) function names? > > > > > > - The indentation from the with syntax? > > > > > > - Something else? =20 > > > > >=20 > > > > > It's *also* the first two =20 > > > >=20 > > > > Ok. First I can't easily change. Second can be mitigated by handl= ing > > > > the imports differently. > > > > =20 > > > > > (the indentation looks actually convenient), =20 > > > >=20 > > > > Ok, good. I also think this is useful because it conveys the lifet= ime > > > > of each object, which will be important once we get to tests where = you > > > > need to change things part way through. > > > > =20 > > > > > but that's not my main point. My main point is that this isn't > > > > > fundamentally declarative. You're turning it into something that > > > > > resembles that, but the syntax is still from an imperative progra= mming > > > > > language. > > > > >=20 > > > > > And in my mind the main feature of a network (topology) simulator= is > > > > > that you describe the topology (and it will build it for you), no= t that > > > > > you... have to build a description? > > > > >=20 > > > > > Using an example that's obviously familiar to you: think of takin= g a > > > > > device tree for some system with a couple of USB and I=B2C busses= and a > > > > > flash controller, and writing all that in Python based on some fo= rm of > > > > > "bus" module/component. =20 > > > >=20 > > > > I mean... old school Open Firmware kind of is this, but with Forth > > > > instead of Python. =20 > > >=20 > > > Okay, you can model data structures in Python, obviously, but that > > > wasn't my point. Anyway, it's all in the example below. > > > =20 > > > > > Once one sees how practical device trees are for that, the Python > > > > > version would look wrong, wouldn't it? =20 > > > >=20 > > > > That really depends on the context. If I was making an API for > > > > building a device tree, I'd probably come up with something pretty > > > > like this. =20 > > >=20 > > > ...an API for building one, yes. But not if you were writing a device > > > tree itself. > > > =20 > > > > > Now, while I think that some single bits of DTS syntax are > > > > > unnecessarily complicated, conceptually, a "networking" device tr= ee > > > > > would look more usable to me than the approach you're taking. > > > > >=20 > > > > > Of course, we need the whole "testing" / exeter part as well, and= test > > > > > cases are fundamentally sequential/imperative. > > > > >=20 > > > > > But (sorry, it's been a few years I don't touch these): > > > > >=20 > > > > > namespace@1 { > > > > > interfaces { > > > > > lo { > > > > > address =3D 127.0.0.1; > > > > > }; > > > > > eth0 { > > > > > address =3D ...; > > > > > }; > > > > > }; > > > > > routes { > > > > > /* something simpler than ip -j ro sh ? */ > > > > > }; > > > > > } > > > > >=20 > > > > > ... > > > > >=20 > > > > > link@... { > > > > > vxlan { > > > > > endpoints { > > > > > a { > > > > > ns =3D <&namespace@1>; > > > > > }; > > > > > b ... > > > > >=20 > > > > > ... > > > > > =20 > > > > > this looks much more natural to me, as an input for a simulator (I > > > > > would personally make the syntax much more "elastic" by just thro= wing a > > > > > link into a namespace but I'm trying to keep it clean just for th= is > > > > > example). =20 > > > >=20 > > > > Aha, I think I finally get what you're saying. More below. > > > > =20 > > > > > Maybe tunbridge implements this somewhere and I missed it? Or wou= ld > > > > > this be part of a "Scenario" description eventually? =20 > > > >=20 > > > > This is entirely unrelated to what Scenario is trying to accomplish. > > > > That may cause you to reconsider whether "Scenario" is a good name, > > > > which is ok. > > > >=20 > > > >=20 > > > > So. A declarative way of defining networks would be nice to have. = =20 > > >=20 > > > From my perspective that's fundamental, rather. I gave it for granted. > > > =20 > > > > I think doing it with the flexibility we want is much harder than y= ou > > > > estimate. =20 > > >=20 > > > I'll pretend I'm not commenting on this line by... oops. :) =20 > >=20 > > Heh. Well, I'd love to be proved wrong on this one. > >=20 > > > > It looks easy for simple static situations like the > > > > examples above, but: > > > >=20 > > > > * If you want to describe a topology that changes partway through, > > > > that's a huge step up in complexity, and kind of necessarily > > > > reintroduces imperative elements. =20 > > >=20 > > > But you can use JSON or a ton of other formats that allow for ordering > > > of elements. =20 > >=20 > > Just ordering network elements w.r.t. each other isn't enough. If you > > have a test where you want /this/ network topology - do some stuff - > > then change the toplogy to /that/ - do some more stuff. I'm not sure > > how you encode that in a way that isn't imperative, or at least > > pseudo-imperative. >=20 > Yeah, I'll come up with a proposal, this is rather clear to me: simply > interleave declarative setups with imperative test steps. >=20 > > > Alternatively, one could add attributes for lifecycle and > > > timing (think of nftables sets) but it looks much less intuitive than > > > the alternative. =20 > >=20 > > I wasn't previously familiar with nftables sets. I had a quick look > > at the docs, but I don't see how it relates to the current proposal. >=20 > Sorry, I took this for granted: I meant timeout attributes for, say, > elements in sets. That is, making the lifecycle validity an attribute > of some declared object, rather than encapsulating declarative blocks > in a possibly imperative sequence (if needed). >=20 > > > > Device tree absolutely suffers > > > > from this - that's what motivated the godawful runtime overlay > > > > mechanism, and right now, I'm struggling to find time to > > > > participate in the latest of many discussions about how to better > > > > handle devices which can be runtime added and removed. =20 > > >=20 > > > I'm not suggesting that we use ANS Forth by the way. =20 > >=20 > > Well, we agree on that at least :). > >=20 > > > > * If you want to build complex scenarios out of simpler ones, you > > > > need what amounts to a macro system. =20 > > >=20 > > > There are a ton of ways. You can also use a filesystem and includes. = Or > > > simply definitions of blocks, not necessarily macros, =20 > >=20 > > You want to parameterise the blocks - at which point it's basically > > macros. > >=20 > > > and JSON > > > implicitly gives you all that. =20 > >=20 > > No, it doesn't. If you have a JSON sub-object that's repeated several > > times through a tree, you have to write it out, in full, each time. > > YAML does allow repeating, as do some other JSON extensions, but bare > > JSON does not. It still doesn't allow parameterisation of those > > blocks. >=20 > Until you meet RFC 6901. Only sort of. That's describing a convention for how to reference JSON nodes with a string. It's still not a notion in JSON proper - whatever's consuming the JSON has to know to expect an RFC 6901 pointer and process it accordingly. It doesn't help in the slightest with parameterization. > > > As it's nothing security relevant, I would actually go with something > > > that's in theory more complicated but in practice more digestible such > > > as YAML. =20 > >=20 > > I don't terribly like YAML, because I think there are a bunch of edge > > cases where it's not obvious reading it whether something is a list or > > object or somethine else. The typing and back-referencing would be > > useful for this problem, I'll grant you. >=20 > ...so perhaps YAML as an optional human-barely-tolerating format that we > translate to JSON? Maybe. > Or TOML? Maybe. I like TOML in general, but I feel like it's specific syntax isn't great for this use case. > > > > Again, a big leap up in > > > > complexity. Device tree struggles with this too - it originated > > > > primarily as a machine->machine format, where having heaps of > > > > repeated information is fine. As it transitioned to being a > > > > human->machine format, not so much. Hence /include/, expression > > > > support and semi-standard running dts files through cpp before > > > > compilation. It's still pretty clunky in this regard. =20 > > >=20 > > > It absolutely is, but that's because it was designed for a different > > > purpose. > > > =20 > > > > Plus.. I think the interpreter for this hypothetical declarative > > > > language would need an internal structure pretty similar to what > > > > tunbridge, so this is kind of already a first step towards it. =20 > > >=20 > > > Okay, that's good to know. > > >=20 > > > I'm estimating I'm currently writing about 5-10 scripts per month, > > > including pasta/iproute2 one-liners, setting up strange stuff, to > > > reproduce / debug issues. > > >=20 > > > Given that this looks so fundamental for my usage I'm thinking that I > > > could make at least part of this a priority of mine. > > >=20 > > > I realised I can implement netlink stuff and handling of networking > > > configuration concepts quite fast with Rust and neli, so I'm pondering > > > to write a proof of concept that can parse the example above (minus > > > Forth notations, but with some kind of pointer) and make it create at > > > least namespaces, links, addresses, and routes. > > >=20 > > > If it helps visualising how that could possibly look like with / in > > > tunbridge itself, I'll take care of it soon rather than later. > > >=20 > > > The only little problem is that I'm much faster with Rust (because of > > > neli) than I can possibly picture myself with Python, and that doesn't > > > play along with tunbridge. But perhaps as a proof of concept it helps > > > anyway? =20 > >=20 > > It would. At earlier points I did consider writing tunbridge (or > > whatever I was calling the idea at the time) in Rust. In principle at > > least, the lifetime tracking would make a very natural way for > > ensuring you bring up / pull down things in a sensible order. > >=20 > > In practice, however, I almost instantly ran into intractable borrow > > checker problems. Specifically, Rust's notorious difficulty with > > references between parts of the same structure. That arises almost > > immediately once you start building composite objects out of smaller > > components: > >=20 > > struct Node { ... } > > struct Veth<'a> { node0: &'a Node, node1: &'a Node, ... } > >=20 > > Then you want something that represents two nodes with a veth between > > them and you get: > >=20 > > struct TwoNodes { node0: Node, node1: Node, veth: Veth<'??> } > >=20 > > There are ways around it, of course, but everything I found so far was > > either hideously unergonomic, depended on obscure crates or both. > > There are a bunch of language extensions that would help, but while > > much discussed, none have landed yet. > >=20 > > If a way to solve this nicely appears, I'm not against moving > > tunbridge to Rust. I mean, slightly more against it than I was months > > ago before I made a start in Python, but still. >=20 > Well, let's see how my draft turns out. I think there are obvious > marketing reasons for Rust over Python, and a couple of technical ones > too (speed, plus what I mentioned about neli), but there are also > technical downsides as you point out. Oh, agreed. At the moment those technical downsides seem pretty fatal to me, though. Oh I missed an option there: hideously unergonomic, depend on obscure crates or neuter the borrow checker even in the cases it would be really useful. Or several of the above. > > > In general, do you think there's something in particular I could > > > contribute at this stage, if I want to see my declarative dream come > > > true? =20 > >=20 > > Yes. A bunch of pseudo-code examples - both the network declarations > > and example tests that might go with them. > >=20 > > [snip] > > > > > > > > Path I can easily add. Version would require an extra invo= cation of > > > > > > > > pasta, which I don't really want to do. =20 > > > > > > >=20 > > > > > > > Ah, right, never mind. The path will be good enough for that. > > > > > > > =20 > > > > > > > > > This part also looks > > > > > > > > > quite readable and intuitive to me without having looked = into tunbridge > > > > > > > > > recently. =20 > > > > > > > >=20 > > > > > > > > Ok, that's promising. =20 > > > > > > >=20 > > > > > > > I mean, I think it's all usable for the moment, and perhaps a= starting > > > > > > > point for some other kind of... front-end? I'm not sure. As I= mentioned > > > > > > > I'm a bit worried about the potential for universal intuitive= ness and > > > > > > > usability. =20 > > > > > >=20 > > > > > > So am I, but I have to weigh it against being able to re-use bo= th > > > > > > tests and setups without having to re-express both in each case= =2E =20 > > > > >=20 > > > > > I think setups written like that are reusable (or can be made > > > > > reusable). My usability point is about other project/usages. For = passt > > > > > and pasta themselves, this level or reusability looks enough to m= e for > > > > > the foreseeable future. > > > > >=20 > > > > > Even though, one day, I guess we might want to generate pseudo-ra= ndom > > > > > (fractal-tree-like?) topologies (and I was recently trying out a > > > > > pasta-in-pasta-in-pasta-in-pasta-in-pasta setup to reproduce that > > > > > HTTP/FIN issue). For that, a declarative approach would make thin= gs > > > > > easier, I suppose. =20 > > > >=20 > > > > Declarative, or imperative? I actually have something like that in > > > > tunbridge's selftests: a function that builds a stack of N nested > > > > namespaces. > > > > https://gitlab.com/dgibson/tunbridge/-/blob/main/tunbridge/unsh= are.py#L302 =20 > > >=20 > > > ...but they are all the same. Think, for example, of connecting every > > > odd-numbered pair with veth tunnels, and every even-numbered pair with > > > pasta. Say: n1 <-- veth --> n2 <-- pasta --> n3 <-- veth --> n4. =20 > >=20 > > That would certainly be possible. More complex, of course, but not > > dramtically so. > >=20 > > > What's really well suited for this situation, in my experience, is a > > > declarative description format that can be easily generated and > > > manipulated by imperative code. =20 > >=20 > > Ah, so there's both an imperative and declarative component. The idea > > in tunbridge is that you can do this, but rather than emit the > > "declarative part" as concrete text for another parser, it's emitted > > as a data structure (generally a bunch of wrappers around context > > managers). > >=20 > > It is true that as currently designed, tunbridge builds the data > > structure representation at the same time as building the actual > > simulated network. With a declarative language approach, building the > > description (language fragment) is separate from instantiating the > > simulation. Is that something you see as valuable? Or only a side > > effect of the other things about the declarative approach you like? >=20 > I see that as valuable by itself, mostly because those fragments can be > generated much more easily if they're separated from the imperative > part. Ok. I'll look into separating 'construct the describing data structure' from 'instantiating the simulation' in tunbridge. It's not everything you want, but it's a start (and would be a step closer to something that could implement a more declarative approach). Might not be for the next spin though, it's a pretty substantial rework. > > > The name of this kind of "indirection" in computer science research > > > currently escapes me, but I'm fairly sure there must be some theory > > > about it. In any case, I can include something like this in my (now > > > planned) proof of concept. >=20 > ...still planned... --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --R4Y234D0F3ccSCI3 Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmkERz8ACgkQzQJF27ox 2GcBMg/8DUdVT0eMKJl38mYCcyk3gXyu+1mQJhE8uGr/ZZE8kbFL7rsTVf7nwyBY 4lhBF/Kq26T+wc1hpYgWRJqtlTUEopZ9FTBDGavJ9ToWuyvxD/YBCV+fECQPkaxU B6rpWaYHuoodMdvHB52GPLeuU6juPZXELNCXXKAB/KFTq+D6ScNCWuFwtXaX1a6v 6ljitKfkiEZdiQ0pOKWsfoosXTjxq3OiKdjWvozfbJI6V0f2wh7XVUJxFtW9c2Yn qI8CGcai/r9IBoOyVA8y1hjiA2QEL6T39fJweR0osTlN3T9L0uEoiT/gfgLZ6N1u oK4j2e15cv8qnJUTuMdxNAKuARCo5K6NX9sR7zCeD4rXLOQsb8eJaaQmpTC+K6wZ /u8B7HMMbMBsPBaqvRhNl1V3MA7VZSEhM5sMCz0VoVIE7V8fkBJ5sWjTDHP25Oa3 YZstKMYn/rk80Hsunpelb/hpEtov+qbxIsdg6HqVGQuGlLqXcFRNJ4TTf+EnhpIY XNEv0WWl5KJio0yKJa+yOTeDFAQRg+nS+LzEL0XKvH4q10IsjtGwg++bgq5i0Hzz KM8hGsC1CRA3gDXHOvLgciiauu1z3U8V2/LGOJSZs5oKAnDQgotgKgoB8oyiavze IXtq6gxJ3A4bWXDIS7rYZEBujrICZID8ArT2/x+qfLcTy0ZCDmI= =OSFY -----END PGP SIGNATURE----- --R4Y234D0F3ccSCI3--