[PATCH v4 0/3] RFC: New proof-of-concept based exeter tests

[PATCH v4 0/3] RFC: New proof-of-concept based exeter tests

[David Gibson]

Here's a new approach to building passt tests with exeter.  This new
one no longer uses Avocado in the default case, although it would
still be possible to manually run the exeter based tests with Avocado.

Here's another draft of my work on testing passt with Avocado and the
exeter library I recently created.  It includes Cleber's patch adding
some basic Avocado tests and builds on that.

For now this only does simple tests, to show how the integration could
work.  It adds some new trivial "smoke tests" and converts the linter
and build checks to exeter.  More complex tests will require building
the sinte/pesto library we've discussed.  A lot of the work for that
already exists in my earlier exeter test series, but it will need some
rework to split it into a separate component.

Changes since v3:
 * Remove reliance on Avocado
 * Build integration between exeter and existing test scripts
 * Drop most of the complex tests for the time being.
 
Changes since v2:
 * Added mypy type checking throughout
 * Use exeter "scenarios" to reduce boilerplate
 * Folded together a number of closely related patches (from 22
   patches down to 15)
 * Entirely avoid open-coded Python context managers in favour of
   contextlib.contextmanager
 * No longer accidentally run a lot of the "meta" tests in the "real"
   testsuite
 * So, so many assorted cleanups

David Gibson (3):
  test: Extend test scripts to allow running exeter tests.
  test: Run static checkers as exeter tests
  test: Convert build tests to exeter

 test/.gitignore               |  1 +
 test/Makefile                 |  5 ++-
 test/build/all                | 61 -------------------------
 test/build/build.py           | 84 +++++++++++++++++++++++++++++++++++
 test/build/clang_tidy         | 17 -------
 test/build/cppcheck           | 17 -------
 test/build/static_checkers.sh | 30 +++++++++++++
 test/lib/exeter               | 46 +++++++++++++++++++
 test/run                      | 18 +++++---
 test/smoke/smoke.sh           | 27 +++++++++++
 10 files changed, 204 insertions(+), 102 deletions(-)
 delete mode 100644 test/build/all
 create mode 100755 test/build/build.py
 delete mode 100644 test/build/clang_tidy
 delete mode 100644 test/build/cppcheck
 create mode 100755 test/build/static_checkers.sh
 create mode 100644 test/lib/exeter
 create mode 100755 test/smoke/smoke.sh

-- 
2.50.1

[PATCH v4 1/3] test: Extend test scripts to allow running exeter tests.

[David Gibson]

Introduce some trivial testcases based on the exeter library.  These run
passt and pasta with --help and --version options.  Extend our test
scripts to run these tests.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/.gitignore     |  1 +
 test/Makefile       |  5 ++++-
 test/lib/exeter     | 46 +++++++++++++++++++++++++++++++++++++++++++++
 test/run            |  9 ++++++++-
 test/smoke/smoke.sh | 27 ++++++++++++++++++++++++++
 5 files changed, 86 insertions(+), 2 deletions(-)
 create mode 100644 test/lib/exeter
 create mode 100755 test/smoke/smoke.sh

diff --git a/test/.gitignore b/test/.gitignore
index 3573444f..cf48b885 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -11,3 +11,4 @@ nstool
 rampstream
 guest-key
 guest-key.pub
+/exeter/
diff --git a/test/Makefile b/test/Makefile
index bf63db87..332f3f3e 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -50,7 +50,7 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \
 	jammy-server-cloudimg-s390x.img
 UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS)
 
-DOWNLOAD_ASSETS = mbuto podman \
+DOWNLOAD_ASSETS = exeter mbuto podman \
 	$(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS)
 TESTDATA_ASSETS = small.bin big.bin medium.bin \
 	rampstream
@@ -70,6 +70,9 @@ assets: $(ASSETS)
 pull-%: %
 	git -C $* pull
 
+exeter:
+	git clone https://gitlab.com/dgibson/exeter.git
+
 mbuto:
 	git clone git://mbuto.sh/mbuto
 
diff --git a/test/lib/exeter b/test/lib/exeter
new file mode 100644
index 00000000..4f7bcff9
--- /dev/null
+++ b/test/lib/exeter
@@ -0,0 +1,46 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/lib/exeter - Run exeter tests within the rest of passt's tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+exeter() {
+	STATESETUP="${STATEBASE}/$1"
+	mkdir -p "${STATESETUP}"
+
+	context_setup_host host
+        layout_host
+
+        cd test
+
+        __ntests=$("$@" --list | wc -l)
+        if [ $? != 0 ]; then
+                info "Failed to get exeter manifest for $@"
+	        pause_continue \
+		        "Press any key to pause test session"		\
+		        "Resuming in "					\
+		        "Paused, press any key to continue"		\
+		        5
+                return
+        fi
+
+        status_file_start "Exeter tests: $*" ${__ntests}
+
+        for __testid in $("$@" --list); do
+	        status_test_start "${__testid}"
+                context_run host "$@" "${__testid}" && status_test_ok || status_test_fail
+        done
+
+        cd ..
+
+	teardown_context_watch ${PANE_HOST} host
+}
diff --git a/test/run b/test/run
index f73c3119..9e183682 100755
--- a/test/run
+++ b/test/run
@@ -53,6 +53,7 @@ COMMIT="$(git log --oneline --no-decorate -1)"
 . lib/layout_ugly
 . lib/test
 . lib/video
+. lib/exeter
 
 # cleanup() - Remove temporary files
 cleanup() {
@@ -67,6 +68,8 @@ run() {
 	perf_init
 	[ ${CI} -eq 1 ]   && video_start ci
 
+	exeter smoke/smoke.sh
+
 	setup build
 	test build/all
 	test build/cppcheck
@@ -223,6 +226,10 @@ run_selected() {
 
 	__setup=
 	for __test; do
+                if "test/${__test}" --list; then
+                        exeter "${__test}"
+                        continue
+                fi
 		# HACK: the migrate tests need the setup repeated for
 		#       each test
 		if [ "${__test%%/*}" != "${__setup}" -o		\
@@ -234,7 +241,7 @@ run_selected() {
 
 		test "${__test}"
 	done
-	teardown "${__setup}"
+	[ -n "${__setup}" ] && teardown "${__setup}"
 
 	log "PASS: ${STATUS_PASS}, FAIL: ${STATUS_FAIL}, SKIPPED: ${STATUS_SKIPPED}"
 
diff --git a/test/smoke/smoke.sh b/test/smoke/smoke.sh
new file mode 100755
index 00000000..26a98d31
--- /dev/null
+++ b/test/smoke/smoke.sh
@@ -0,0 +1,27 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic snoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_register pasta_version $PASTA --version
+
+exeter_register passt_help $PASST --help
+exeter_register pasta_help $PASTA --help
+
+exeter_main "$@"
-- 
@@ -0,0 +1,27 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic snoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_register pasta_version $PASTA --version
+
+exeter_register passt_help $PASST --help
+exeter_register pasta_help $PASTA --help
+
+exeter_main "$@"
-- 
2.50.1

[PATCH v4 2/3] test: Run static checkers as exeter tests

[David Gibson]

Move the static checkers from the current DSL to exeter.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/build/clang_tidy         | 17 -----------------
 test/build/cppcheck           | 17 -----------------
 test/build/static_checkers.sh | 30 ++++++++++++++++++++++++++++++
 test/run                      |  3 +--
 4 files changed, 31 insertions(+), 36 deletions(-)
 delete mode 100644 test/build/clang_tidy
 delete mode 100644 test/build/cppcheck
 create mode 100755 test/build/static_checkers.sh

diff --git a/test/build/clang_tidy b/test/build/clang_tidy
deleted file mode 100644
index 40573bfd..00000000
--- a/test/build/clang_tidy
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/clang_tidy - Run source through clang-tidy(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	clang-tidy
-
-test	Run clang-tidy
-host	make clang-tidy
diff --git a/test/build/cppcheck b/test/build/cppcheck
deleted file mode 100644
index 0e1dbced..00000000
--- a/test/build/cppcheck
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/cppcheck - Run source through cppcheck(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	cppcheck
-
-test	Run cppcheck
-host	make cppcheck
diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
new file mode 100755
index 00000000..41152c25
--- /dev/null
+++ b/test/build/static_checkers.sh
@@ -0,0 +1,30 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/build/static_checkers.sh - Run static checkers
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+cppcheck () {
+        make -C .. cppcheck
+}
+exeter_register cppcheck
+
+clang_tidy () {
+        make -C .. clang-tidy
+}
+exeter_register clang_tidy
+
+exeter_main "$@"
+
+
diff --git a/test/run b/test/run
index 9e183682..5fd02d91 100755
--- a/test/run
+++ b/test/run
@@ -69,11 +69,10 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/static_checkers.sh
 
 	setup build
 	test build/all
-	test build/cppcheck
-	test build/clang_tidy
 	teardown build
 
 	setup pasta
-- 
@@ -69,11 +69,10 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/static_checkers.sh
 
 	setup build
 	test build/all
-	test build/cppcheck
-	test build/clang_tidy
 	teardown build
 
 	setup pasta
-- 
2.50.1

[PATCH v4 3/3] test: Convert build tests to exeter

[David Gibson]

Convert the tests in build/all to be based on exeter.  The new version of
the tests is more robust than the original, since it makes a temporary copy
of the source tree so will not be affected by concurrent manual builds.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/build/all      | 61 --------------------------------
 test/build/build.py | 84 +++++++++++++++++++++++++++++++++++++++++++++
 test/run            |  8 ++---
 3 files changed, 88 insertions(+), 65 deletions(-)
 delete mode 100644 test/build/all
 create mode 100755 test/build/build.py

diff --git a/test/build/all b/test/build/all
deleted file mode 100644
index 1f79e0d8..00000000
--- a/test/build/all
+++ /dev/null
@@ -1,61 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-#  for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-#  for network namespace/tap device mode
-#
-# test/build/all - Build targets, one by one, then all together, check output
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools	make cc rm uname getconf mkdir cp rm man
-
-test	Build passt
-host	make clean
-check	! [ -e passt ]
-host	CFLAGS="-Werror" make passt
-check	[ -f passt ]
-
-test	Build pasta
-host	make clean
-check	! [ -e pasta ]
-host	CFLAGS="-Werror" make pasta
-check	[ -h pasta ]
-
-test	Build qrap
-host	make clean
-check	! [ -e qrap ]
-host	CFLAGS="-Werror" make qrap
-check	[ -f qrap ]
-
-test	Build all
-host	make clean
-check	! [ -e passt ]
-check	! [ -e pasta ]
-check	! [ -e qrap ]
-host	CFLAGS="-Werror" make
-check	[ -f passt ]
-check	[ -h pasta ]
-check	[ -f qrap ]
-
-test	Install
-host	mkdir __STATEDIR__/prefix
-host	prefix=__STATEDIR__/prefix make install
-check	[ -f __STATEDIR__/prefix/bin/passt ]
-check	[ -h __STATEDIR__/prefix/bin/pasta ]
-check	[ -f __STATEDIR__/prefix/bin/qrap ]
-check	man -M __STATEDIR__/prefix/share/man -W passt
-check	man -M __STATEDIR__/prefix/share/man -W pasta
-check	man -M __STATEDIR__/prefix/share/man -W qrap
-
-test	Uninstall
-host	prefix=__STATEDIR__/prefix make uninstall
-check	! [ -f __STATEDIR__/prefix/bin/passt ]
-check	! [ -h __STATEDIR__/prefix/bin/pasta ]
-check	! [ -f __STATEDIR__/prefix/bin/qrap ]
-check	! man -M __STATEDIR__/prefix/share/man -W passt 2>/dev/null
-check	! man -M __STATEDIR__/prefix/share/man -W pasta 2>/dev/null
-check	! man -M __STATEDIR__/prefix/share/man -W qrap 2>/dev/null
diff --git a/test/build/build.py b/test/build/build.py
new file mode 100755
index 00000000..12bb82d8
--- /dev/null
+++ b/test/build/build.py
@@ -0,0 +1,84 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/build/build.py - Test build and install targets
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+import contextlib
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+from typing import Iterable, Iterator
+
+import exeter
+
+def sh(cmd):
+    subprocess.run(cmd, shell=True)
+
+
+@contextlib.contextmanager
+def clone_sources() -> Iterator[str]:
+    os.chdir('..')  # Move from test/ to repo base
+    with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+        sh(f"cp --parents -d $(git ls-files) {tmpdir}")
+        os.chdir(tmpdir)
+        yield tmpdir
+
+
+def test_make(target: str, outputs: Iterable[str]) -> None:
+    outpaths = [Path(o) for o in outputs]
+    with clone_sources():
+        for o in outpaths:
+            assert not o.exists(), f"{o} existed before make"
+        sh(f'make {target} CFLAGS="-Werror"')
+        for o in outpaths:
+            assert o.exists(), f"{o} wasn't made"
+        sh('make clean')
+        for o in outpaths:
+            assert not o.exists(), f"{o} existed after make clean"
+
+
+exeter.register('make_passt', test_make, 'passt', ['passt'])
+exeter.register('make_pasta', test_make, 'pasta', ['pasta'])
+exeter.register('make_qrap', test_make, 'qrap', ['qrap'])
+exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_install_uninstall() -> None:
+    with clone_sources():
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \
+             as prefix:
+            bindir = Path(prefix) / 'bin'
+            mandir = Path(prefix) / 'share/man'
+            progs = ['passt', 'pasta', 'qrap']
+
+            # Install
+            sh(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+            for prog in progs:
+                exe = bindir / prog
+                assert exe.is_file(), f"{exe} does not exist as a regular file"
+                sh(f'man -M {mandir} -W {prog}')
+
+            # Uninstall
+            sh(f'make uninstall prefix={prefix}')
+
+            for prog in progs:
+                exe = bindir / prog
+                assert not exe.exists(), f"{exe} exists after uninstall"
+                sh(f'! man -M {mandir} -W {prog}')
+
+
+if __name__ == '__main__':
+    exeter.main()
diff --git a/test/run b/test/run
index 5fd02d91..6a53e24b 100755
--- a/test/run
+++ b/test/run
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
 
 COMMIT="$(git log --oneline --no-decorate -1)"
 
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
 . lib/util
 . lib/context
 . lib/setup
@@ -69,12 +72,9 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/build.py
 	exeter build/static_checkers.sh
 
-	setup build
-	test build/all
-	teardown build
-
 	setup pasta
 	test pasta/ndp
 	test pasta/dhcp
-- 
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
 
 COMMIT="$(git log --oneline --no-decorate -1)"
 
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
 . lib/util
 . lib/context
 . lib/setup
@@ -69,12 +72,9 @@ run() {
 	[ ${CI} -eq 1 ]   && video_start ci
 
 	exeter smoke/smoke.sh
+	exeter build/build.py
 	exeter build/static_checkers.sh
 
-	setup build
-	test build/all
-	teardown build
-
 	setup pasta
 	test pasta/ndp
 	test pasta/dhcp
-- 
2.50.1