[PATCH v8 0/5] Introduce first exeter based test infrastructure

[PATCH v8 0/5] Introduce first exeter based test infrastructure

[David Gibson]

Here's a new approach to building passt tests with exeter.  This new
one no longer uses Avocado in the default case, although it would
still be possible to manually run the exeter based tests with Avocado.

For now this only does simple tests, to show how the integration could
work.  It adds some new trivial "smoke tests" and converts the linter
and build checks to exeter.  More complex tests will require building
the tunbridge library we've discussed.  A lot of the work for that
already exists in my earlier exeter test series, but it will need some
rework to split it into a separate component.

v8:
 * Create video links for exeter tests too
 * Fix some bad indenting
v7:
 * Use "exetool list" rather than "--list" directly to get test manifest.
 * Update Makefiles to automatically update exeter sources as necessary.
 * Update series cover letter removing some outdated information.
v6:
 * Use exeter's new metadata support to print nicer test names
v5:
 * Updated according to Stefano's review
   - Fixed a number of whitespace errors
   - Improved many comments and variable names to make things clearer
 * New patch adding parallel test execution with BATS
 * Improved autodetection of exeter tests using "exetool probe"

David Gibson (5):
  test: Extend test scripts to allow running exeter tests.
  test: Run static checkers as exeter tests
  test: Convert build tests to exeter
  test: Allow exeter & podman tests to be parallel executed with BATS
  test: Explicit specify forwarding ports for pasta in log rotation
    tests

 test/.gitignore                |   2 +
 test/Makefile                  |  24 +++++++-
 test/build/all                 |  61 ------------------
 test/build/build.py            | 109 +++++++++++++++++++++++++++++++++
 test/build/clang_tidy          |  17 -----
 test/build/cppcheck            |  17 -----
 test/build/static_checkers.sh  |  26 ++++++++
 test/lib/exeter                |  58 ++++++++++++++++++
 test/pasta_options/log_to_file |  10 +--
 test/run                       |  18 ++++--
 test/smoke/smoke.sh            |  33 ++++++++++
 11 files changed, 267 insertions(+), 108 deletions(-)
 delete mode 100644 test/build/all
 create mode 100755 test/build/build.py
 delete mode 100644 test/build/clang_tidy
 delete mode 100644 test/build/cppcheck
 create mode 100755 test/build/static_checkers.sh
 create mode 100644 test/lib/exeter
 create mode 100755 test/smoke/smoke.sh

-- 
2.51.0

[PATCH v8 1/5] test: Extend test scripts to allow running exeter tests.

[David Gibson]

Introduce some trivial testcases based on the exeter library.  These run
passt and pasta with --help and --version options.  Extend our test
scripts to run these tests.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/.gitignore     |  1 +
 test/Makefile       | 10 ++++++--
 test/lib/exeter     | 58 +++++++++++++++++++++++++++++++++++++++++++++
 test/run            |  9 ++++++-
 test/smoke/smoke.sh | 33 ++++++++++++++++++++++++++
 5 files changed, 108 insertions(+), 3 deletions(-)
 create mode 100644 test/lib/exeter
 create mode 100755 test/smoke/smoke.sh

diff --git a/test/.gitignore b/test/.gitignore
index 3573444f..cf48b885 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -11,3 +11,4 @@ nstool
 rampstream
 guest-key
 guest-key.pub
+/exeter/
diff --git a/test/Makefile b/test/Makefile
index bf63db87..bfb58f27 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,6 +5,7 @@
 # Copyright Red Hat
 # Author: David Gibson <david@gibson.dropbear.id.au>
 
+EXETOOL = exeter/exetool/exetool
 WGET = wget -c
 
 DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \
@@ -50,14 +51,14 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \
 	jammy-server-cloudimg-s390x.img
 UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS)
 
-DOWNLOAD_ASSETS = mbuto podman \
+DOWNLOAD_ASSETS = exeter mbuto podman \
 	$(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS)
 TESTDATA_ASSETS = small.bin big.bin medium.bin \
 	rampstream
 LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 	$(DEBIAN_IMGS:%=prepared-%) $(FEDORA_IMGS:%=prepared-%) \
 	$(UBUNTU_NEW_IMGS:%=prepared-%) \
-	nstool guest-key guest-key.pub \
+	nstool guest-key guest-key.pub $(EXETOOL) \
 	$(TESTDATA_ASSETS)
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
@@ -70,6 +71,11 @@ assets: $(ASSETS)
 pull-%: %
 	git -C $* pull
 
+exeter:
+	git clone https://gitlab.com/dgibson/exeter.git
+
+exeter/exetool/exetool: pull-exeter
+
 mbuto:
 	git clone git://mbuto.sh/mbuto
 
diff --git a/test/lib/exeter b/test/lib/exeter
new file mode 100644
index 00000000..3b19beaa
--- /dev/null
+++ b/test/lib/exeter
@@ -0,0 +1,58 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/lib/exeter - Run exeter tests within the rest of passt's tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+EXETOOL="$BASEPATH/exeter/exetool/exetool"
+
+# is_exeter() - Determine if a test file is an exeter program
+# $@:	Command line to invoke test program
+is_exeter() {
+	$EXETOOL probe -- "$@"
+}
+
+# exeter() - Run each test in an exeter program, logging each test separately
+# $@:	Command line to invoke exeter test program
+exeter() {
+	STATESETUP="${STATEBASE}/$1"
+	mkdir -p "${STATESETUP}"
+
+	context_setup_host host
+	layout_host
+
+	cd test
+
+	__ntests=$($EXETOOL list -- "$@" | wc -l)
+	if [ $? != 0 ]; then
+		info "Failed to get exeter manifest for $@"
+		pause_continue \
+			"Press any key to pause test session"		\
+			"Resuming in "					\
+			"Paused, press any key to continue"		\
+			5
+		return
+	fi
+
+	status_file_start "$* (exeter)" ${__ntests}
+	[ ${CI} -eq 1 ] && video_link "${1}"
+
+	for __testid in $($EXETOOL list -- "$@"); do
+		__desc="$($EXETOOL desc -- "$@" -- ${__testid})"
+		status_test_start "${__desc}"
+		context_run host "$@" "${__testid}" && status_test_ok || status_test_fail
+	done
+
+	cd ..
+
+	teardown_context_watch ${PANE_HOST} host
+}
diff --git a/test/run b/test/run
index f73c3119..cf732b7f 100755
--- a/test/run
+++ b/test/run
@@ -53,6 +53,7 @@ COMMIT="$(git log --oneline --no-decorate -1)"
 . lib/layout_ugly
 . lib/test
 . lib/video
+. lib/exeter
 
 # cleanup() - Remove temporary files
 cleanup() {
@@ -67,6 +68,8 @@ run() {
 	perf_init
 	[ ${CI} -eq 1 ]   && video_start ci
 
+	exeter smoke/smoke.sh
+
 	setup build
 	test build/all
 	test build/cppcheck
@@ -223,6 +226,10 @@ run_selected() {
 
 	__setup=
 	for __test; do
+		if is_exeter "test/${__test}"; then
+			exeter "${__test}"
+			continue
+		fi
 		# HACK: the migrate tests need the setup repeated for
 		#       each test
 		if [ "${__test%%/*}" != "${__setup}" -o		\
@@ -234,7 +241,7 @@ run_selected() {
 
 		test "${__test}"
 	done
-	teardown "${__setup}"
+	[ -n "${__setup}" ] && teardown "${__setup}"
 
 	log "PASS: ${STATUS_PASS}, FAIL: ${STATUS_FAIL}, SKIPPED: ${STATUS_SKIPPED}"
 
diff --git a/test/smoke/smoke.sh b/test/smoke/smoke.sh
new file mode 100755
index 00000000..a642fb96
--- /dev/null
+++ b/test/smoke/smoke.sh
@@ -0,0 +1,33 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_set_description passt_version "Check passt --version works"
+
+exeter_register pasta_version $PASTA --version
+exeter_set_description pasta_version "Check pasta --version works"
+
+exeter_register passt_help $PASST --help
+exeter_set_description passt_help "Check passt --help works"
+
+exeter_register pasta_help $PASTA --help
+exeter_set_description pasta_help "Check pasta --help works"
+
+exeter_main "$@"
-- 
@@ -0,0 +1,33 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_set_description passt_version "Check passt --version works"
+
+exeter_register pasta_version $PASTA --version
+exeter_set_description pasta_version "Check pasta --version works"
+
+exeter_register passt_help $PASST --help
+exeter_set_description passt_help "Check passt --help works"
+
+exeter_register pasta_help $PASTA --help
+exeter_set_description pasta_help "Check pasta --help works"
+
+exeter_main "$@"
-- 
2.51.0

[PATCH v8 2/5] test: Run static checkers as exeter tests

[David Gibson]

Move the static checkers from the current DSL to exeter.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/build/clang_tidy         | 17 -----------------
 test/build/cppcheck           | 17 -----------------
 test/build/static_checkers.sh | 26 ++++++++++++++++++++++++++
 test/run                      |  3 +--
 4 files changed, 27 insertions(+), 36 deletions(-)
 delete mode 100644 test/build/clang_tidy
 delete mode 100644 test/build/cppcheck
 create mode 100755 test/build/static_checkers.sh

diff --git a/test/build/clang_tidy b/test/build/clang_tidy
deleted file mode 100644
index 40573bfd..00000000
--- a/test/build/clang_tidy
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/clang_tidy - Run source through clang-tidy(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	clang-tidy
-
-test	Run clang-tidy
-host	make clang-tidy
diff --git a/test/build/cppcheck b/test/build/cppcheck
deleted file mode 100644
index 0e1dbced..00000000
--- a/test/build/cppcheck
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/cppcheck - Run source through cppcheck(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	cppcheck
-
-test	Run cppcheck
-host	make cppcheck
diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
new file mode 100755
index 00000000..42806e79
--- /dev/null
+++ b/test/build/static_checkers.sh
@@ -0,0 +1,26 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/build/static_checkers.sh - Run static checkers
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+exeter_register cppcheck make -C .. cppcheck
+exeter_set_description cppcheck "passt sources pass cppcheck"
+
+exeter_register clang_tidy make -C .. clang-tidy
+exeter_set_description clang_tidy "passt sources pass clang-tidy"
+
+exeter_main "$@"
+
+
diff --git a/test/run b/test/run
index cf732b7f..652de1a0 100755
--- a/test/run
+++ b/test/run
@@ -69,11 +69,10 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/static_checkers.sh
 
 	setup build
 	test build/all
-	test build/cppcheck
-	test build/clang_tidy
 	teardown build
 
 	setup pasta
-- 
@@ -69,11 +69,10 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/static_checkers.sh
 
 	setup build
 	test build/all
-	test build/cppcheck
-	test build/clang_tidy
 	teardown build
 
 	setup pasta
-- 
2.51.0

[PATCH v8 3/5] test: Convert build tests to exeter

[David Gibson]

Convert the tests in build/all to be based on exeter.  The new version of
the tests is more robust than the original, since it makes a temporary copy
of the source tree so will not be affected by concurrent manual builds.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/build/all      |  61 -------------------------
 test/build/build.py | 109 ++++++++++++++++++++++++++++++++++++++++++++
 test/run            |   8 ++--
 3 files changed, 113 insertions(+), 65 deletions(-)
 delete mode 100644 test/build/all
 create mode 100755 test/build/build.py

diff --git a/test/build/all b/test/build/all
deleted file mode 100644
index 1f79e0d8..00000000
--- a/test/build/all
+++ /dev/null
@@ -1,61 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/all - Build targets, one by one, then all together, check output
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	make cc rm uname getconf mkdir cp rm man
-
-test	Build passt
-host	make clean
-check	! [ -e passt ]
-host	CFLAGS="-Werror" make passt
-check	[ -f passt ]
-
-test	Build pasta
-host	make clean
-check	! [ -e pasta ]
-host	CFLAGS="-Werror" make pasta
-check	[ -h pasta ]
-
-test	Build qrap
-host	make clean
-check	! [ -e qrap ]
-host	CFLAGS="-Werror" make qrap
-check	[ -f qrap ]
-
-test	Build all
-host	make clean
-check	! [ -e passt ]
-check	! [ -e pasta ]
-check	! [ -e qrap ]
-host	CFLAGS="-Werror" make
-check	[ -f passt ]
-check	[ -h pasta ]
-check	[ -f qrap ]
-
-test	Install
-host	mkdir __STATEDIR__/prefix
-host	prefix=__STATEDIR__/prefix make install
-check	[ -f __STATEDIR__/prefix/bin/passt ]
-check	[ -h __STATEDIR__/prefix/bin/pasta ]
-check	[ -f __STATEDIR__/prefix/bin/qrap ]
-check	man -M __STATEDIR__/prefix/share/man -W passt
-check	man -M __STATEDIR__/prefix/share/man -W pasta
-check	man -M __STATEDIR__/prefix/share/man -W qrap
-
-test	Uninstall
-host	prefix=__STATEDIR__/prefix make uninstall
-check	! [ -f __STATEDIR__/prefix/bin/passt ]
-check	! [ -h __STATEDIR__/prefix/bin/pasta ]
-check	! [ -f __STATEDIR__/prefix/bin/qrap ]
-check	! man -M __STATEDIR__/prefix/share/man -W passt 2>/dev/null
-check	! man -M __STATEDIR__/prefix/share/man -W pasta 2>/dev/null
-check	! man -M __STATEDIR__/prefix/share/man -W qrap 2>/dev/null
diff --git a/test/build/build.py b/test/build/build.py
new file mode 100755
index 00000000..e49287c9
--- /dev/null
+++ b/test/build/build.py
@@ -0,0 +1,109 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/build/build.py - Test build and install targets
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+import contextlib
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+from typing import Iterable, Iterator
+
+import exeter
+
+def sh(cmd):
+    """Run given command in a shell"""
+    subprocess.run(cmd, shell=True)
+
+
+@contextlib.contextmanager
+def clone_sources() -> Iterator[str]:
+    """Create a temporary copy of the passt sources.
+
+    When the context enters create a temporary directory and copy the
+    passt sources into it.  Clean it up when the context exits.
+    """
+
+    os.chdir('..')  # Move from test/ to repo base
+    with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+        sh(f"cp --parents -d $(git ls-files) {tmpdir}")
+        os.chdir(tmpdir)
+        yield tmpdir
+
+
+def test_make(target: str, expected_files: list[str]) -> None:
+    """Test `make {target}`
+
+    Arguments:
+    target -- make target to invoke
+    expected_files -- files make is expected to create
+
+    Verifies that
+      1) `make target` completes successfully
+      2) expected_files care created by `make target`
+      3) expected_files are removed by `make clean`
+    """
+
+    ex_paths = [Path(f) for f in expected_files]
+    with clone_sources():
+        for p in ex_paths:
+            assert not p.exists(), f"{p} existed before make"
+        sh(f'make {target} CFLAGS="-Werror"')
+        for p in ex_paths:
+            assert p.exists(), f"{p} wasn't made"
+        sh('make clean')
+        for p in ex_paths:
+            assert not p.exists(), f"{p} existed after make clean"
+
+
+exeter.register('make_passt', test_make, 'passt', ['passt'])
+exeter.register('make_pasta', test_make, 'pasta', ['pasta'])
+exeter.register('make_qrap', test_make, 'qrap', ['qrap'])
+exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_install_uninstall() -> None:
+    """Test `make install` and `make uninstall`
+
+    Tests that `make install` installs the expected files to the
+    install prefix, and that `make uninstall` removes them again.
+    """
+
+    with clone_sources():
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \
+             as prefix:
+            bindir = Path(prefix) / 'bin'
+            mandir = Path(prefix) / 'share/man'
+            progs = ['passt', 'pasta', 'qrap']
+
+            # Install
+            sh(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+            for prog in progs:
+                exe = bindir / prog
+                assert exe.is_file(), f"{exe} does not exist as a regular file"
+                sh(f'man -M {mandir} -W {prog}')
+
+            # Uninstall
+            sh(f'make uninstall prefix={prefix}')
+
+            for prog in progs:
+                exe = bindir / prog
+                assert not exe.exists(), f"{exe} exists after uninstall"
+                sh(f'! man -M {mandir} -W {prog}')
+
+
+if __name__ == '__main__':
+    exeter.main()
diff --git a/test/run b/test/run
index 652de1a0..f858e558 100755
--- a/test/run
+++ b/test/run
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
 
 COMMIT="$(git log --oneline --no-decorate -1)"
 
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
 . lib/util
 . lib/context
 . lib/setup
@@ -69,12 +72,9 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/build.py
 	exeter build/static_checkers.sh
 
-	setup build
-	test build/all
-	teardown build
-
 	setup pasta
 	test pasta/ndp
 	test pasta/dhcp
-- 
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
 
 COMMIT="$(git log --oneline --no-decorate -1)"
 
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
 . lib/util
 . lib/context
 . lib/setup
@@ -69,12 +72,9 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/build.py
 	exeter build/static_checkers.sh
 
-	setup build
-	test build/all
-	teardown build
-
 	setup pasta
 	test pasta/ndp
 	test pasta/dhcp
-- 
2.51.0

[PATCH v8 4/5] test: Allow exeter & podman tests to be parallel executed with BATS

[David Gibson]

As a proof-of-concept, this adds a "make bats" target which will execute
both the podman and exeter based testcases in parallel using BATS.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/.gitignore |  1 +
 test/Makefile   | 14 ++++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/test/.gitignore b/test/.gitignore
index cf48b885..9412f0d3 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -12,3 +12,4 @@ rampstream
 guest-key
 guest-key.pub
 /exeter/
+*.bats
diff --git a/test/Makefile b/test/Makefile
index bfb58f27..a774285e 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,6 +5,7 @@
 # Copyright Red Hat
 # Author: David Gibson <david@gibson.dropbear.id.au>
 
+BATS = bats -j $(shell nproc)
 EXETOOL = exeter/exetool/exetool
 WGET = wget -c
 
@@ -63,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+	build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+	podman/test/system/505-networking-pasta.bats
+
 CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
 
 assets: $(ASSETS)
@@ -121,6 +128,12 @@ medium.bin:
 big.bin:
 	dd if=/dev/urandom bs=1M count=10 of=$@
 
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+	PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+	PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
 check: assets
 	./run
 
@@ -130,6 +143,7 @@ debug: assets
 clean:
 	rm -f perf.js *~
 	rm -f $(LOCAL_ASSETS)
+	rm -f $(EXETER_BATS)
 	rm -rf test_logs
 	rm -f prepared-*.qcow2 prepared-*.img
 
-- 
@@ -5,6 +5,7 @@
 # Copyright Red Hat
 # Author: David Gibson <david@gibson.dropbear.id.au>
 
+BATS = bats -j $(shell nproc)
 EXETOOL = exeter/exetool/exetool
 WGET = wget -c
 
@@ -63,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+	build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+	podman/test/system/505-networking-pasta.bats
+
 CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
 
 assets: $(ASSETS)
@@ -121,6 +128,12 @@ medium.bin:
 big.bin:
 	dd if=/dev/urandom bs=1M count=10 of=$@
 
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+	PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+	PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
 check: assets
 	./run
 
@@ -130,6 +143,7 @@ debug: assets
 clean:
 	rm -f perf.js *~
 	rm -f $(LOCAL_ASSETS)
+	rm -f $(EXETER_BATS)
 	rm -rf test_logs
 	rm -f prepared-*.qcow2 prepared-*.img
 
-- 
2.51.0

[PATCH v8 5/5] test: Explicit specify forwarding ports for pasta in log rotation tests

[David Gibson]

test/pasta_options/log_to_file uses tcp_crr in a loop to generate log
messages for what it's actually testing.  For this to work, pasta needs to
forward two ports.  We set a PORTS variable with the options for this, but
never actually use it.  The test works, because without the options we use
automatic port forwarding.

That makes the test more fragile than it needs to be, in a way that's not
at all related to what we're actually trying to test.  Avoid this possible
point of failure by explicitly specifying the forwarding.

I encountered this because Fedora has now updated to a kernel with the
/proc/net lseek() bug.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/pasta_options/log_to_file | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/test/pasta_options/log_to_file b/test/pasta_options/log_to_file
index 3ead06c9..db78b042 100644
--- a/test/pasta_options/log_to_file
+++ b/test/pasta_options/log_to_file
@@ -30,19 +30,19 @@ endef
 
 test	Log creation
 
-set	PORTS -t 10001,10002 -u 10001,10002
+set	PORTS -t 10001,10002 -u 10001,10002 -T none -U none
 set	LOG_FILE __STATEDIR__/pasta.log
 
-passt	./pasta -l __LOG_FILE__ -- /bin/true
+passt	./pasta __PORTS__ -l __LOG_FILE__ -- /bin/true
 check	[ -s __LOG_FILE__ ]
 
 test	Log truncated on creation
-passt	./pasta -l __LOG_FILE__ -- /bin/true & wait
+passt	./pasta __PORTS__ -l __LOG_FILE__ -- /bin/true & wait
 pout	PID2 echo $!
 check	head -1 __LOG_FILE__ | grep '^pasta .* [(]__PID2__[)]$'
 
 test	Maximum log size
-passtb	./pasta --config-net -d -f -l __LOG_FILE__ --log-size $((100 * 1024)) -- sh -c 'while true; do tcp_crr --nolog -l1 -P 10001 -C 10002 -6; done'
+passtb	./pasta __PORTS__ --config-net -d -f -l __LOG_FILE__ --log-size $((100 * 1024)) -- sh -c 'while true; do tcp_crr --nolog -l1 -P 10001 -C 10002 -6; done'
 sleep	1
 
 flood_log_client
@@ -67,7 +67,7 @@ passt	unshare -rUm
 passt	mkdir __STATEDIR__/t
 passt	mount -t tmpfs none __STATEDIR__/t
 set	LOG_FILE __STATEDIR__/t/log
-passt	./pasta --config-net -d -l __LOG_FILE__ --log-size $((100 * 1024))
+passt	./pasta __PORTS__ --config-net -d -l __LOG_FILE__ --log-size $((100 * 1024))
 
 flood_log_server
 flood_log_client
-- 
@@ -30,19 +30,19 @@ endef
 
 test	Log creation
 
-set	PORTS -t 10001,10002 -u 10001,10002
+set	PORTS -t 10001,10002 -u 10001,10002 -T none -U none
 set	LOG_FILE __STATEDIR__/pasta.log
 
-passt	./pasta -l __LOG_FILE__ -- /bin/true
+passt	./pasta __PORTS__ -l __LOG_FILE__ -- /bin/true
 check	[ -s __LOG_FILE__ ]
 
 test	Log truncated on creation
-passt	./pasta -l __LOG_FILE__ -- /bin/true & wait
+passt	./pasta __PORTS__ -l __LOG_FILE__ -- /bin/true & wait
 pout	PID2 echo $!
 check	head -1 __LOG_FILE__ | grep '^pasta .* [(]__PID2__[)]$'
 
 test	Maximum log size
-passtb	./pasta --config-net -d -f -l __LOG_FILE__ --log-size $((100 * 1024)) -- sh -c 'while true; do tcp_crr --nolog -l1 -P 10001 -C 10002 -6; done'
+passtb	./pasta __PORTS__ --config-net -d -f -l __LOG_FILE__ --log-size $((100 * 1024)) -- sh -c 'while true; do tcp_crr --nolog -l1 -P 10001 -C 10002 -6; done'
 sleep	1
 
 flood_log_client
@@ -67,7 +67,7 @@ passt	unshare -rUm
 passt	mkdir __STATEDIR__/t
 passt	mount -t tmpfs none __STATEDIR__/t
 set	LOG_FILE __STATEDIR__/t/log
-passt	./pasta --config-net -d -l __LOG_FILE__ --log-size $((100 * 1024))
+passt	./pasta __PORTS__ --config-net -d -l __LOG_FILE__ --log-size $((100 * 1024))
 
 flood_log_server
 flood_log_client
-- 
2.51.0

Re: [PATCH v8 5/5] test: Explicit specify forwarding ports for pasta in log rotation tests

[Stefano Brivio]

On Fri,  5 Sep 2025 16:43:17 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> test/pasta_options/log_to_file uses tcp_crr in a loop to generate log
> messages for what it's actually testing.  For this to work, pasta needs to
> forward two ports.  We set a PORTS variable with the options for this, but
> never actually use it.  The test works, because without the options we use
> automatic port forwarding.

Oops.

> That makes the test more fragile than it needs to be, in a way that's not
> at all related to what we're actually trying to test.  Avoid this possible
> point of failure by explicitly specifying the forwarding.
> 
> I encountered this because Fedora has now updated to a kernel with the
> /proc/net lseek() bug.

By the way, the fix is now in mainline as 2ce3d282bd50 ("proc: fix
missing pde_set_flags() for net proc files"). I hoped the breakage would
never reach distributions but I guess the timing was unlucky enough.

-- 
Stefano

Re: [PATCH v8 0/5] Introduce first exeter based test infrastructure

[Stefano Brivio]

On Fri,  5 Sep 2025 16:43:12 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> Here's a new approach to building passt tests with exeter.  This new
> one no longer uses Avocado in the default case, although it would
> still be possible to manually run the exeter based tests with Avocado.
> 
> For now this only does simple tests, to show how the integration could
> work.  It adds some new trivial "smoke tests" and converts the linter
> and build checks to exeter.  More complex tests will require building
> the tunbridge library we've discussed.  A lot of the work for that
> already exists in my earlier exeter test series, but it will need some
> rework to split it into a separate component.
> 
> v8:
>  * Create video links for exeter tests too
>  * Fix some bad indenting

Applied!

-- 
Stefano

Re: [PATCH v8 5/5] test: Explicit specify forwarding ports for pasta in log rotation tests

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1338 bytes --]

On Fri, Sep 05, 2025 at 01:03:40PM +0200, Stefano Brivio wrote:
> On Fri,  5 Sep 2025 16:43:17 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > test/pasta_options/log_to_file uses tcp_crr in a loop to generate log
> > messages for what it's actually testing.  For this to work, pasta needs to
> > forward two ports.  We set a PORTS variable with the options for this, but
> > never actually use it.  The test works, because without the options we use
> > automatic port forwarding.
> 
> Oops.
> 
> > That makes the test more fragile than it needs to be, in a way that's not
> > at all related to what we're actually trying to test.  Avoid this possible
> > point of failure by explicitly specifying the forwarding.
> > 
> > I encountered this because Fedora has now updated to a kernel with the
> > /proc/net lseek() bug.
> 
> By the way, the fix is now in mainline as 2ce3d282bd50 ("proc: fix
> missing pde_set_flags() for net proc files"). I hoped the breakage would
> never reach distributions but I guess the timing was unlucky enough.

Apparently so.  Oh well, good to know the fix is upstream.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v8 0/5] Introduce first exeter based test infrastructure

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1128 bytes --]

On Fri, Sep 05, 2025 at 01:03:44PM +0200, Stefano Brivio wrote:
> On Fri,  5 Sep 2025 16:43:12 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > Here's a new approach to building passt tests with exeter.  This new
> > one no longer uses Avocado in the default case, although it would
> > still be possible to manually run the exeter based tests with Avocado.
> > 
> > For now this only does simple tests, to show how the integration could
> > work.  It adds some new trivial "smoke tests" and converts the linter
> > and build checks to exeter.  More complex tests will require building
> > the tunbridge library we've discussed.  A lot of the work for that
> > already exists in my earlier exeter test series, but it will need some
> > rework to split it into a separate component.
> > 
> > v8:
> >  * Create video links for exeter tests too
> >  * Fix some bad indenting
> 
> Applied!

Yay!

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v8 5/5] test: Explicit specify forwarding ports for pasta in log rotation tests

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1548 bytes --]

On Mon, Sep 08, 2025 at 11:53:58AM +1000, David Gibson wrote:
> On Fri, Sep 05, 2025 at 01:03:40PM +0200, Stefano Brivio wrote:
> > On Fri,  5 Sep 2025 16:43:17 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> > 
> > > test/pasta_options/log_to_file uses tcp_crr in a loop to generate log
> > > messages for what it's actually testing.  For this to work, pasta needs to
> > > forward two ports.  We set a PORTS variable with the options for this, but
> > > never actually use it.  The test works, because without the options we use
> > > automatic port forwarding.
> > 
> > Oops.
> > 
> > > That makes the test more fragile than it needs to be, in a way that's not
> > > at all related to what we're actually trying to test.  Avoid this possible
> > > point of failure by explicitly specifying the forwarding.
> > > 
> > > I encountered this because Fedora has now updated to a kernel with the
> > > /proc/net lseek() bug.
> > 
> > By the way, the fix is now in mainline as 2ce3d282bd50 ("proc: fix
> > missing pde_set_flags() for net proc files"). I hoped the breakage would
> > never reach distributions but I guess the timing was unlucky enough.
> 
> Apparently so.  Oh well, good to know the fix is upstream.

Hmm... Fedora is still on 6.16.4, so possibly they explicitly
backported the bad patch?  Oops.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]