Re: passt & mbuto

[Stefano Brivio <sbrivio@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 5804 bytes --]

On Tue, 14 Jun 2022 12:57:20 +1000
David Gibson <david(a)gibson.dropbear.id.au> wrote:

> On Tue, Jun 14, 2022 at 03:33:13AM +0200, Stefano Brivio wrote:
> > On Fri, 10 Jun 2022 13:50:44 +1000
> > David Gibson <david(a)gibson.dropbear.id.au> wrote:
> >   
> > > Hi again,
> > > 
> > > I realized I wasn't quite right when I said that qrap problems where
> > > what was currently stopping me running the passt (not pasta) tests.  I
> > > did hit qrap issues somewhere, but the current stumbling block is that
> > > mbuto looks for udhcpc to put into the guest image, which I can't
> > > easily put onto my host system.
> > > 
> > > Now, in the short term, once my patch to remove usage of udhcpc from
> > > the passt/pasta tests is applied, we could just remove udhcpc from the
> > > mbuto profile as well.  However, that raises a wider scope issue:
> > > 
> > > The passt testing profile for mbuto appliances is in the mbuto tree,
> > > not the passt tree.  That doesn't realy make sense, since it means any
> > > change to what we need for the passt tests requires a synchronized
> > > change with mbuto.  Particularly for a pretty young and project like
> > > passt, that's not really tenable.  Plus, slurping an external tool
> > > from some random URL in the tests is just kinda ugly.  
> > 
> > Hmm, yes, in my ideal world mbuto would be already widely distributed
> > and we could drop the git clone. On the other hand, that's still one
> > long-term goal of mine, so:
> >   
> > > I'm not immediately sure how best to to address this:
> > > 
> > >  * We could make mbuto take the profiles as some sort of external
> > >    file, so they can be provided by the user, rather than built into
> > >    the mbuto repository.  
> > 
> > ...I would prefer this option. Even though if you look at mbuto's git  
> 
> Yeah, I think it looks the best option to me as well, though not
> necessarily the quickest to implement.

Kind of done actually, I'll share it tomorrow (Wednesday).

> > history, the last change to the passt profile was in September last
> > year, so quantitatively speaking this might be more of a theoretical
> > problem.  
> 
> Hmm.. I strongly suspect that's more a reflection that with just one
> person working on it, passt hasn't been moving that fast.  With
> another person (and maybe more in future) working on it, I think this
> will become a bigger problem.  It's also pretty clearly unsustainable
> once we start having proper passt releases: it's no good for a frozen
> released version to just pray that the latest mbuto downloaded is
> still good for it.

Might be, yeah.

> > Actually, mbuto already allows overriding every part of a profile with
> > environmental variables (this would be PROGS), but the resulting
> > command line wouldn't be that nice, especially for demos.  
> 
> Right, I saw that.  Maybe we can polish that up a bit and move the
> passt profile from the mbuto tree to the passt tree?

Yes, I'll post this as well.

> > I could implement an option there which sources a shell script file
> > with assignments, instead. Would that make sense?  
> 
> Yeah, I think that should do the job.
> 
> > >  * We could just fork a copy of mbuto into the passt tree, making
> > >    local modifications for the profile, and only manually updating it
> > >    to match upstream mbuto changes.  
> > 
> > Oh, you mean "vendoring"... :) this looks rather messy to me.  
> 
> Oh, it's definitely messy, but it nonetheless has some advantages.
> I'm also much more confortable vendoring something the size of mbuto
> than vendoring whole libraries and frameworks the way Go does by
> convention.  Even then, I'd definitely be considering that a stop-gap
> workaround.
> 
> > >  * We could use an entirely different and more established tool for
> > >    building our testing guest images in passt (e.g. supermin,
> > >    buildroot or just picking a standard distro guest image)  
> > 
> > supermin needs packages though: it only supports Debian and Fedora at
> > the moment, and we would also have an issue with neper's tcp_{,c}rr and
> > udp_rr.  
> 
> Yeah, I did notice that.  It also means we might still need host
> distro specific logic to get the right package names, which is pretty
> horrid.
> 
> > Buildroot would be somewhat slow in demos, same for a "standard" distro
> > image (which we would need to update and tweak before starting it, too).  
> 
> Right, I haven't worked with buildroot much so I'm not really familiar
> with it.
> 
> Hmm... one more option... could we use dracut for this?  IIRC it
> already has a plugin mechanism we could potentially use to do our
> specific bits.

The issue with dracut plug-ins is that they might also be distribution
specific to some extent, but I guess that's solvable. Still, for demos,
on my system:

--
$ time dracut -H dracut.img
[...]
dracut: *** Creating image file '/home/sbrivio/passt/test/dracut.img' ***
dracut: dracut: using auto-determined compression method 'gzip'
dracut: *** Creating initramfs image file '/home/sbrivio/passt/test/dracut.img' done ***

real	0m13.651s
user	0m13.135s
sys	0m0.817s
--

...we could probably skip a lot of "modules", but that looks like some
substantial work. For comparison:

--
$ time ~/mbuto/mbuto -c gzip
Applying profile base
Creating image: /tmp/tmp.sBPKY6xsHM
Size: bin  353k lib  4.1M kmod  213k total  4.6M compressed  1.9M
/tmp/tmp.sBPKY6xsHM

real	0m0.933s
user	0m0.860s
sys	0m0.079s
--

overall, I'm not sure switching to a more generic tool (less portable,
though, as it needs Bash >= 4.x), that's going to be slower and probably
with slightly harder to update "profiles", is worth the effort.

-- 
Stefano