Re: [PATCH] test: Add Podman system test with bats for pasta

[Stefano Brivio <sbrivio@redhat.com>]

On Wed, 23 Aug 2023 14:09:13 +0200
Paul Holzinger <pholzing@redhat.com> wrote:

> On 23/08/2023 11:35, Stefano Brivio wrote:
> > Ugly as hell, but we keep breaking things otherwise, and I keep
> > forgetting to run this manually (as long as it's based on my local
> > Podman setup, that's the only alternative).
> >
> > We need to clone the Podman repository as distribution packages don't
> > contain test scripts, typically. While at it, build the latest
> > version which is what really matters.  
>
> We actually ship a podman-tests rpm which contains `test/system` from 
> the podman repo. Of course this will contain the test version based on 
> the current shipped version and not the main branch.
> But if you want to the with latest you have to clone from github of course.

Not that I have a particular motivation... but podman-tests doesn't
exist on, say, Debian.

Still, I actually have a slight preference to test with the latest
HEAD, yes, so that we can unblock things quickly in case.

> > As we're planning anyway to revamp the test framework, I'd be
> > inclined to just add this without too many thoughts, and have it as
> > a nice-to-have requirement reminder for the new framework.
> >
> > Link: https://github.com/containers/podman/pull/19699
> > Suggested-by: Paul Holzinger <pholzing@redhat.com>
> > Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
> > ---
> >   test/README.md         |  4 ++--
> >   test/pasta_podman/bats | 22 ++++++++++++++++++++++
> >   test/run               |  4 ++++
> >   3 files changed, 28 insertions(+), 2 deletions(-)
> >   create mode 100644 test/pasta_podman/bats
> >
> > diff --git a/test/README.md b/test/README.md
> > index 03c7f57..0936b04 100644
> > --- a/test/README.md
> > +++ b/test/README.md
> > @@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
> >   
> >   Example for Debian, and possibly most Debian-based distributions:
> >   
> > -    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
> > -    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
> > +    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
> > +    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
> >       netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
> >       qemu-system-misc qemu-system-x86 valgrind
> >   
> > diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
> > new file mode 100644
> > index 0000000..f36da7c
> > --- /dev/null
> > +++ b/test/pasta_podman/bats
> > @@ -0,0 +1,22 @@
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +#
> > +# PASST - Plug A Simple Socket Transport
> > +#  for qemu/UNIX domain socket mode
> > +#
> > +# PASTA - Pack A Subtle Tap Abstraction
> > +#  for network namespace/tap device mode
> > +#
> > +# test/pasta_podman/bats - Build Podman, run pasta system test with bats
> > +#
> > +# Copyright (c) 2022 Red Hat GmbH
> > +# Author: Stefano Brivio <sbrivio@redhat.com>
> > +
> > +htools	git make go bats catatonit ip jq socat
> > +
> > +test	Podman system test with bats
> > +
> > +host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
> > +host	make -C __STATEDIR__/podman
> > +hout	WD pwd
> > +host	printf "[engine]\nhelper_binaries_dir=['__WD__']\n" > __STATEDIR__/containers.conf
> > +host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_CONF_OVERRIDE="__STATEDIR__/containers.conf" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats  
>
> I recommend to use the `CONTAINERS_HELPER_BINARY_DIR` to set the 
> directory over containers.conf, with your containers.conf change you 
> will unset all other helper directories.

Hah, thanks, that's how it's called, I couldn't find it. I'll post a
new version.

> This may be fine for the pasta specific tests for now as we do not use 
> other binaries such as netavark there but in the future it could change 
> and leaves you wondering what broke.

Sure, I didn't like that either.

-- 
Stefano