[PATCH] test: Add Podman system test with bats for pasta

[PATCH] test: Add Podman system test with bats for pasta

[Stefano Brivio]

Ugly as hell, but we keep breaking things otherwise, and I keep
forgetting to run this manually (as long as it's based on my local
Podman setup, that's the only alternative).

We need to clone the Podman repository as distribution packages don't
contain test scripts, typically. While at it, build the latest
version which is what really matters.

As we're planning anyway to revamp the test framework, I'd be
inclined to just add this without too many thoughts, and have it as
a nice-to-have requirement reminder for the new framework.

Link: https://github.com/containers/podman/pull/19699
Suggested-by: Paul Holzinger <pholzing@redhat.com>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
 test/README.md         |  4 ++--
 test/pasta_podman/bats | 22 ++++++++++++++++++++++
 test/run               |  4 ++++
 3 files changed, 28 insertions(+), 2 deletions(-)
 create mode 100644 test/pasta_podman/bats

diff --git a/test/README.md b/test/README.md
index 03c7f57..0936b04 100644
--- a/test/README.md
+++ b/test/README.md
@@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
 
 Example for Debian, and possibly most Debian-based distributions:
 
-    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
-    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
+    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
+    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
     netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
     qemu-system-misc qemu-system-x86 valgrind
 
diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
new file mode 100644
index 0000000..f36da7c
--- /dev/null
+++ b/test/pasta_podman/bats
@@ -0,0 +1,22 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/pasta_podman/bats - Build Podman, run pasta system test with bats
+#
+# Copyright (c) 2022 Red Hat GmbH
+# Author: Stefano Brivio <sbrivio@redhat.com>
+
+htools	git make go bats catatonit ip jq socat
+
+test	Podman system test with bats
+
+host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
+host	make -C __STATEDIR__/podman
+hout	WD pwd
+host	printf "[engine]\nhelper_binaries_dir=['__WD__']\n" > __STATEDIR__/containers.conf
+host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_CONF_OVERRIDE="__STATEDIR__/containers.conf" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats
diff --git a/test/run b/test/run
index 8f4f845..3b37663 100755
--- a/test/run
+++ b/test/run
@@ -82,6 +82,10 @@ run() {
 	test pasta_options/log_to_file
 	teardown pasta_options
 
+	setup build
+	test pasta_podman/bats
+	teardown build
+
 	setup memory
 	test memory/passt
 	teardown memory
-- 
@@ -82,6 +82,10 @@ run() {
 	test pasta_options/log_to_file
 	teardown pasta_options
 
+	setup build
+	test pasta_podman/bats
+	teardown build
+
 	setup memory
 	test memory/passt
 	teardown memory
-- 
2.39.2

Re: [PATCH] test: Add Podman system test with bats for pasta

[Paul Holzinger]

On 23/08/2023 11:35, Stefano Brivio wrote:
> Ugly as hell, but we keep breaking things otherwise, and I keep
> forgetting to run this manually (as long as it's based on my local
> Podman setup, that's the only alternative).
>
> We need to clone the Podman repository as distribution packages don't
> contain test scripts, typically. While at it, build the latest
> version which is what really matters.
We actually ship a podman-tests rpm which contains `test/system` from 
the podman repo. Of course this will contain the test version based on 
the current shipped version and not the main branch.
But if you want to the with latest you have to clone from github of course.
>
> As we're planning anyway to revamp the test framework, I'd be
> inclined to just add this without too many thoughts, and have it as
> a nice-to-have requirement reminder for the new framework.
>
> Link: https://github.com/containers/podman/pull/19699
> Suggested-by: Paul Holzinger <pholzing@redhat.com>
> Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
> ---
>   test/README.md         |  4 ++--
>   test/pasta_podman/bats | 22 ++++++++++++++++++++++
>   test/run               |  4 ++++
>   3 files changed, 28 insertions(+), 2 deletions(-)
>   create mode 100644 test/pasta_podman/bats
>
> diff --git a/test/README.md b/test/README.md
> index 03c7f57..0936b04 100644
> --- a/test/README.md
> +++ b/test/README.md
> @@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
>   
>   Example for Debian, and possibly most Debian-based distributions:
>   
> -    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
> -    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
> +    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
> +    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
>       netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
>       qemu-system-misc qemu-system-x86 valgrind
>   
> diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
> new file mode 100644
> index 0000000..f36da7c
> --- /dev/null
> +++ b/test/pasta_podman/bats
> @@ -0,0 +1,22 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# PASST - Plug A Simple Socket Transport
> +#  for qemu/UNIX domain socket mode
> +#
> +# PASTA - Pack A Subtle Tap Abstraction
> +#  for network namespace/tap device mode
> +#
> +# test/pasta_podman/bats - Build Podman, run pasta system test with bats
> +#
> +# Copyright (c) 2022 Red Hat GmbH
> +# Author: Stefano Brivio <sbrivio@redhat.com>
> +
> +htools	git make go bats catatonit ip jq socat
> +
> +test	Podman system test with bats
> +
> +host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
> +host	make -C __STATEDIR__/podman
> +hout	WD pwd
> +host	printf "[engine]\nhelper_binaries_dir=['__WD__']\n" > __STATEDIR__/containers.conf
> +host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_CONF_OVERRIDE="__STATEDIR__/containers.conf" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats
I recommend to use the `CONTAINERS_HELPER_BINARY_DIR` to set the 
directory over containers.conf, with your containers.conf change you 
will unset all other helper directories.
This may be fine for the pasta specific tests for now as we do not use 
other binaries such as netavark there but in the future it could change 
and leaves you wondering what broke.
> diff --git a/test/run b/test/run
> index 8f4f845..3b37663 100755
> --- a/test/run
> +++ b/test/run
> @@ -82,6 +82,10 @@ run() {
>   	test pasta_options/log_to_file
>   	teardown pasta_options
>   
> +	setup build
> +	test pasta_podman/bats
> +	teardown build
> +
>   	setup memory
>   	test memory/passt
>   	teardown memory

Re: [PATCH] test: Add Podman system test with bats for pasta

[Stefano Brivio]

On Wed, 23 Aug 2023 14:09:13 +0200
Paul Holzinger <pholzing@redhat.com> wrote:

> On 23/08/2023 11:35, Stefano Brivio wrote:
> > Ugly as hell, but we keep breaking things otherwise, and I keep
> > forgetting to run this manually (as long as it's based on my local
> > Podman setup, that's the only alternative).
> >
> > We need to clone the Podman repository as distribution packages don't
> > contain test scripts, typically. While at it, build the latest
> > version which is what really matters.  
>
> We actually ship a podman-tests rpm which contains `test/system` from 
> the podman repo. Of course this will contain the test version based on 
> the current shipped version and not the main branch.
> But if you want to the with latest you have to clone from github of course.

Not that I have a particular motivation... but podman-tests doesn't
exist on, say, Debian.

Still, I actually have a slight preference to test with the latest
HEAD, yes, so that we can unblock things quickly in case.

> > As we're planning anyway to revamp the test framework, I'd be
> > inclined to just add this without too many thoughts, and have it as
> > a nice-to-have requirement reminder for the new framework.
> >
> > Link: https://github.com/containers/podman/pull/19699
> > Suggested-by: Paul Holzinger <pholzing@redhat.com>
> > Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
> > ---
> >   test/README.md         |  4 ++--
> >   test/pasta_podman/bats | 22 ++++++++++++++++++++++
> >   test/run               |  4 ++++
> >   3 files changed, 28 insertions(+), 2 deletions(-)
> >   create mode 100644 test/pasta_podman/bats
> >
> > diff --git a/test/README.md b/test/README.md
> > index 03c7f57..0936b04 100644
> > --- a/test/README.md
> > +++ b/test/README.md
> > @@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
> >   
> >   Example for Debian, and possibly most Debian-based distributions:
> >   
> > -    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
> > -    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
> > +    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
> > +    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
> >       netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
> >       qemu-system-misc qemu-system-x86 valgrind
> >   
> > diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
> > new file mode 100644
> > index 0000000..f36da7c
> > --- /dev/null
> > +++ b/test/pasta_podman/bats
> > @@ -0,0 +1,22 @@
> > +# SPDX-License-Identifier: GPL-2.0-or-later
> > +#
> > +# PASST - Plug A Simple Socket Transport
> > +#  for qemu/UNIX domain socket mode
> > +#
> > +# PASTA - Pack A Subtle Tap Abstraction
> > +#  for network namespace/tap device mode
> > +#
> > +# test/pasta_podman/bats - Build Podman, run pasta system test with bats
> > +#
> > +# Copyright (c) 2022 Red Hat GmbH
> > +# Author: Stefano Brivio <sbrivio@redhat.com>
> > +
> > +htools	git make go bats catatonit ip jq socat
> > +
> > +test	Podman system test with bats
> > +
> > +host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
> > +host	make -C __STATEDIR__/podman
> > +hout	WD pwd
> > +host	printf "[engine]\nhelper_binaries_dir=['__WD__']\n" > __STATEDIR__/containers.conf
> > +host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_CONF_OVERRIDE="__STATEDIR__/containers.conf" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats  
>
> I recommend to use the `CONTAINERS_HELPER_BINARY_DIR` to set the 
> directory over containers.conf, with your containers.conf change you 
> will unset all other helper directories.

Hah, thanks, that's how it's called, I couldn't find it. I'll post a
new version.

> This may be fine for the pasta specific tests for now as we do not use 
> other binaries such as netavark there but in the future it could change 
> and leaves you wondering what broke.

Sure, I didn't like that either.

-- 
Stefano