[PATCH v2] test: Add Podman system test with bats for pasta

[PATCH v2] test: Add Podman system test with bats for pasta

[Stefano Brivio]

Ugly as hell, but we keep breaking things otherwise, and I keep
forgetting to run this manually (as long as it's based on my local
Podman setup, that's the only alternative).

We need to clone the Podman repository as distribution packages don't
contain test scripts, typically. While at it, build the latest
version which is what really matters.

As we're planning anyway to revamp the test framework, I'd be
inclined to just add this without too many thoughts, and have it as
a nice-to-have requirement reminder for the new framework.

Link: https://github.com/containers/podman/pull/19699
Suggested-by: Paul Holzinger <pholzing@redhat.com>
Signed-off-by: Stefano Brivio <sbrivio@redhat.com>
---
v2: Use CONTAINERS_HELPER_BINARY_DIR to override pasta's path (Paul Holzinger)

 test/README.md         |  4 ++--
 test/pasta_podman/bats | 21 +++++++++++++++++++++
 test/run               |  4 ++++
 3 files changed, 27 insertions(+), 2 deletions(-)
 create mode 100644 test/pasta_podman/bats

diff --git a/test/README.md b/test/README.md
index 03c7f57..0936b04 100644
--- a/test/README.md
+++ b/test/README.md
@@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
 
 Example for Debian, and possibly most Debian-based distributions:
 
-    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
-    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
+    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
+    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
     netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
     qemu-system-misc qemu-system-x86 valgrind
 
diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
new file mode 100644
index 0000000..21446f0
--- /dev/null
+++ b/test/pasta_podman/bats
@@ -0,0 +1,21 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/pasta_podman/bats - Build Podman, run pasta system test with bats
+#
+# Copyright (c) 2022 Red Hat GmbH
+# Author: Stefano Brivio <sbrivio@redhat.com>
+
+htools	git make go bats catatonit ip jq socat
+
+test	Podman system test with bats
+
+host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
+host	make -C __STATEDIR__/podman
+hout	WD pwd
+host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats
diff --git a/test/run b/test/run
index 8f4f845..3b37663 100755
--- a/test/run
+++ b/test/run
@@ -82,6 +82,10 @@ run() {
 	test pasta_options/log_to_file
 	teardown pasta_options
 
+	setup build
+	test pasta_podman/bats
+	teardown build
+
 	setup memory
 	test memory/passt
 	teardown memory
-- 
@@ -82,6 +82,10 @@ run() {
 	test pasta_options/log_to_file
 	teardown pasta_options
 
+	setup build
+	test pasta_podman/bats
+	teardown build
+
 	setup memory
 	test memory/passt
 	teardown memory
-- 
2.39.2

Re: [PATCH v2] test: Add Podman system test with bats for pasta

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3797 bytes --]

On Wed, Aug 23, 2023 at 03:51:49PM +0200, Stefano Brivio wrote:
> Ugly as hell, but we keep breaking things otherwise, and I keep
> forgetting to run this manually (as long as it's based on my local
> Podman setup, that's the only alternative).
> 
> We need to clone the Podman repository as distribution packages don't
> contain test scripts, typically. While at it, build the latest
> version which is what really matters.
> 
> As we're planning anyway to revamp the test framework, I'd be
> inclined to just add this without too many thoughts, and have it as
> a nice-to-have requirement reminder for the new framework.
> 
> Link: https://github.com/containers/podman/pull/19699
> Suggested-by: Paul Holzinger <pholzing@redhat.com>
> Signed-off-by: Stefano Brivio <sbrivio@redhat.com>

Clunky, but better than nothing, so

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

My biggest concern here is that we're unconditionally pulling the
latest podman tree.  That means that if future podman adds tests for
newer pasta features, then checking out an old pasta will get failing
tests even for things that were correct at the time of release, which
can suck for bisection.

> ---
> v2: Use CONTAINERS_HELPER_BINARY_DIR to override pasta's path (Paul Holzinger)
> 
>  test/README.md         |  4 ++--
>  test/pasta_podman/bats | 21 +++++++++++++++++++++
>  test/run               |  4 ++++
>  3 files changed, 27 insertions(+), 2 deletions(-)
>  create mode 100644 test/pasta_podman/bats
> 
> diff --git a/test/README.md b/test/README.md
> index 03c7f57..0936b04 100644
> --- a/test/README.md
> +++ b/test/README.md
> @@ -28,8 +28,8 @@ on a system, i.e. common utilities such as a shell are not included here.
>  
>  Example for Debian, and possibly most Debian-based distributions:
>  
> -    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bc
> -    clang-tidy cppcheck isc-dhcp-common psmisc linux-cpupower socat
> +    build-essential git jq strace iperf3 qemu-system-x86 tmux sipcalc bats bc
> +    catatonit clang-tidy cppcheck go isc-dhcp-common psmisc linux-cpupower socat
>      netcat-openbsd fakeroot lz4 lm-sensors qemu-system-arm qemu-system-ppc
>      qemu-system-misc qemu-system-x86 valgrind
>  
> diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats
> new file mode 100644
> index 0000000..21446f0
> --- /dev/null
> +++ b/test/pasta_podman/bats
> @@ -0,0 +1,21 @@
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# PASST - Plug A Simple Socket Transport
> +#  for qemu/UNIX domain socket mode
> +#
> +# PASTA - Pack A Subtle Tap Abstraction
> +#  for network namespace/tap device mode
> +#
> +# test/pasta_podman/bats - Build Podman, run pasta system test with bats
> +#
> +# Copyright (c) 2022 Red Hat GmbH
> +# Author: Stefano Brivio <sbrivio@redhat.com>
> +
> +htools	git make go bats catatonit ip jq socat
> +
> +test	Podman system test with bats
> +
> +host	git -C __STATEDIR__ clone https://github.com/containers/podman.git
> +host	make -C __STATEDIR__/podman
> +hout	WD pwd
> +host	PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats
> diff --git a/test/run b/test/run
> index 8f4f845..3b37663 100755
> --- a/test/run
> +++ b/test/run
> @@ -82,6 +82,10 @@ run() {
>  	test pasta_options/log_to_file
>  	teardown pasta_options
>  
> +	setup build
> +	test pasta_podman/bats
> +	teardown build
> +
>  	setup memory
>  	test memory/passt
>  	teardown memory

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]