* [PATCH v7 1/4] test: Extend test scripts to allow running exeter tests.
2025-09-04 2:50 [PATCH v7 0/4] Introduce first exeter based test infrastructure David Gibson
@ 2025-09-04 2:50 ` David Gibson
2025-09-04 2:50 ` [PATCH v7 2/4] test: Run static checkers as " David Gibson
` (3 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: David Gibson @ 2025-09-04 2:50 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Introduce some trivial testcases based on the exeter library. These run
passt and pasta with --help and --version options. Extend our test
scripts to run these tests.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/.gitignore | 1 +
test/Makefile | 10 ++++++--
test/lib/exeter | 57 +++++++++++++++++++++++++++++++++++++++++++++
test/run | 9 ++++++-
test/smoke/smoke.sh | 33 ++++++++++++++++++++++++++
5 files changed, 107 insertions(+), 3 deletions(-)
create mode 100644 test/lib/exeter
create mode 100755 test/smoke/smoke.sh
diff --git a/test/.gitignore b/test/.gitignore
index 3573444f..cf48b885 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -11,3 +11,4 @@ nstool
rampstream
guest-key
guest-key.pub
+/exeter/
diff --git a/test/Makefile b/test/Makefile
index bf63db87..bfb58f27 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,6 +5,7 @@
# Copyright Red Hat
# Author: David Gibson <david@gibson.dropbear.id.au>
+EXETOOL = exeter/exetool/exetool
WGET = wget -c
DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \
@@ -50,14 +51,14 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \
jammy-server-cloudimg-s390x.img
UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS)
-DOWNLOAD_ASSETS = mbuto podman \
+DOWNLOAD_ASSETS = exeter mbuto podman \
$(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS)
TESTDATA_ASSETS = small.bin big.bin medium.bin \
rampstream
LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
$(DEBIAN_IMGS:%=prepared-%) $(FEDORA_IMGS:%=prepared-%) \
$(UBUNTU_NEW_IMGS:%=prepared-%) \
- nstool guest-key guest-key.pub \
+ nstool guest-key guest-key.pub $(EXETOOL) \
$(TESTDATA_ASSETS)
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
@@ -70,6 +71,11 @@ assets: $(ASSETS)
pull-%: %
git -C $* pull
+exeter:
+ git clone https://gitlab.com/dgibson/exeter.git
+
+exeter/exetool/exetool: pull-exeter
+
mbuto:
git clone git://mbuto.sh/mbuto
diff --git a/test/lib/exeter b/test/lib/exeter
new file mode 100644
index 00000000..dce4e49d
--- /dev/null
+++ b/test/lib/exeter
@@ -0,0 +1,57 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/lib/exeter - Run exeter tests within the rest of passt's tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+EXETOOL="$BASEPATH/exeter/exetool/exetool"
+
+# is_exeter() - Determine if a test file is an exeter program
+# $@: Command line to invoke test program
+is_exeter() {
+ $EXETOOL probe -- "$@"
+}
+
+# exeter() - Run each test in an exeter program, logging each test separately
+# $@: Command line to invoke exeter test program
+exeter() {
+ STATESETUP="${STATEBASE}/$1"
+ mkdir -p "${STATESETUP}"
+
+ context_setup_host host
+ layout_host
+
+ cd test
+
+ __ntests=$($EXETOOL list -- "$@" | wc -l)
+ if [ $? != 0 ]; then
+ info "Failed to get exeter manifest for $@"
+ pause_continue \
+ "Press any key to pause test session" \
+ "Resuming in " \
+ "Paused, press any key to continue" \
+ 5
+ return
+ fi
+
+ status_file_start "$* (exeter)" ${__ntests}
+
+ for __testid in $($EXETOOL list -- "$@"); do
+ __desc="$($EXETOOL desc -- "$@" -- ${__testid})"
+ status_test_start "${__desc}"
+ context_run host "$@" "${__testid}" && status_test_ok || status_test_fail
+ done
+
+ cd ..
+
+ teardown_context_watch ${PANE_HOST} host
+}
diff --git a/test/run b/test/run
index f73c3119..745a09fa 100755
--- a/test/run
+++ b/test/run
@@ -53,6 +53,7 @@ COMMIT="$(git log --oneline --no-decorate -1)"
. lib/layout_ugly
. lib/test
. lib/video
+. lib/exeter
# cleanup() - Remove temporary files
cleanup() {
@@ -67,6 +68,8 @@ run() {
perf_init
[ ${CI} -eq 1 ] && video_start ci
+ exeter smoke/smoke.sh
+
setup build
test build/all
test build/cppcheck
@@ -223,6 +226,10 @@ run_selected() {
__setup=
for __test; do
+ if is_exeter "test/${__test}"; then
+ exeter "${__test}"
+ continue
+ fi
# HACK: the migrate tests need the setup repeated for
# each test
if [ "${__test%%/*}" != "${__setup}" -o \
@@ -234,7 +241,7 @@ run_selected() {
test "${__test}"
done
- teardown "${__setup}"
+ [ -n "${__setup}" ] && teardown "${__setup}"
log "PASS: ${STATUS_PASS}, FAIL: ${STATUS_FAIL}, SKIPPED: ${STATUS_SKIPPED}"
diff --git a/test/smoke/smoke.sh b/test/smoke/smoke.sh
new file mode 100755
index 00000000..a642fb96
--- /dev/null
+++ b/test/smoke/smoke.sh
@@ -0,0 +1,33 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_set_description passt_version "Check passt --version works"
+
+exeter_register pasta_version $PASTA --version
+exeter_set_description pasta_version "Check pasta --version works"
+
+exeter_register passt_help $PASST --help
+exeter_set_description passt_help "Check passt --help works"
+
+exeter_register pasta_help $PASTA --help
+exeter_set_description pasta_help "Check pasta --help works"
+
+exeter_main "$@"
--
@@ -0,0 +1,33 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_set_description passt_version "Check passt --version works"
+
+exeter_register pasta_version $PASTA --version
+exeter_set_description pasta_version "Check pasta --version works"
+
+exeter_register passt_help $PASST --help
+exeter_set_description passt_help "Check passt --help works"
+
+exeter_register pasta_help $PASTA --help
+exeter_set_description pasta_help "Check pasta --help works"
+
+exeter_main "$@"
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 2/4] test: Run static checkers as exeter tests
2025-09-04 2:50 [PATCH v7 0/4] Introduce first exeter based test infrastructure David Gibson
2025-09-04 2:50 ` [PATCH v7 1/4] test: Extend test scripts to allow running exeter tests David Gibson
@ 2025-09-04 2:50 ` David Gibson
2025-09-04 2:50 ` [PATCH v7 3/4] test: Convert build tests to exeter David Gibson
` (2 subsequent siblings)
4 siblings, 0 replies; 10+ messages in thread
From: David Gibson @ 2025-09-04 2:50 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Move the static checkers from the current DSL to exeter.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/build/clang_tidy | 17 -----------------
test/build/cppcheck | 17 -----------------
test/build/static_checkers.sh | 26 ++++++++++++++++++++++++++
test/run | 3 +--
4 files changed, 27 insertions(+), 36 deletions(-)
delete mode 100644 test/build/clang_tidy
delete mode 100644 test/build/cppcheck
create mode 100755 test/build/static_checkers.sh
diff --git a/test/build/clang_tidy b/test/build/clang_tidy
deleted file mode 100644
index 40573bfd..00000000
--- a/test/build/clang_tidy
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/clang_tidy - Run source through clang-tidy(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools clang-tidy
-
-test Run clang-tidy
-host make clang-tidy
diff --git a/test/build/cppcheck b/test/build/cppcheck
deleted file mode 100644
index 0e1dbced..00000000
--- a/test/build/cppcheck
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/cppcheck - Run source through cppcheck(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools cppcheck
-
-test Run cppcheck
-host make cppcheck
diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
new file mode 100755
index 00000000..42806e79
--- /dev/null
+++ b/test/build/static_checkers.sh
@@ -0,0 +1,26 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/build/static_checkers.sh - Run static checkers
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+exeter_register cppcheck make -C .. cppcheck
+exeter_set_description cppcheck "passt sources pass cppcheck"
+
+exeter_register clang_tidy make -C .. clang-tidy
+exeter_set_description clang_tidy "passt sources pass clang-tidy"
+
+exeter_main "$@"
+
+
diff --git a/test/run b/test/run
index 745a09fa..9e26952c 100755
--- a/test/run
+++ b/test/run
@@ -69,11 +69,10 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/static_checkers.sh
setup build
test build/all
- test build/cppcheck
- test build/clang_tidy
teardown build
setup pasta
--
@@ -69,11 +69,10 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/static_checkers.sh
setup build
test build/all
- test build/cppcheck
- test build/clang_tidy
teardown build
setup pasta
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 3/4] test: Convert build tests to exeter
2025-09-04 2:50 [PATCH v7 0/4] Introduce first exeter based test infrastructure David Gibson
2025-09-04 2:50 ` [PATCH v7 1/4] test: Extend test scripts to allow running exeter tests David Gibson
2025-09-04 2:50 ` [PATCH v7 2/4] test: Run static checkers as " David Gibson
@ 2025-09-04 2:50 ` David Gibson
2025-09-04 2:50 ` [PATCH v7 4/4] test: Allow exeter & podman tests to be parallel executed with BATS David Gibson
2025-09-04 23:14 ` [PATCH v7 0/4] Introduce first exeter based test infrastructure Stefano Brivio
4 siblings, 0 replies; 10+ messages in thread
From: David Gibson @ 2025-09-04 2:50 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Convert the tests in build/all to be based on exeter. The new version of
the tests is more robust than the original, since it makes a temporary copy
of the source tree so will not be affected by concurrent manual builds.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/build/all | 61 -------------------------
test/build/build.py | 109 ++++++++++++++++++++++++++++++++++++++++++++
test/run | 8 ++--
3 files changed, 113 insertions(+), 65 deletions(-)
delete mode 100644 test/build/all
create mode 100755 test/build/build.py
diff --git a/test/build/all b/test/build/all
deleted file mode 100644
index 1f79e0d8..00000000
--- a/test/build/all
+++ /dev/null
@@ -1,61 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/all - Build targets, one by one, then all together, check output
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools make cc rm uname getconf mkdir cp rm man
-
-test Build passt
-host make clean
-check ! [ -e passt ]
-host CFLAGS="-Werror" make passt
-check [ -f passt ]
-
-test Build pasta
-host make clean
-check ! [ -e pasta ]
-host CFLAGS="-Werror" make pasta
-check [ -h pasta ]
-
-test Build qrap
-host make clean
-check ! [ -e qrap ]
-host CFLAGS="-Werror" make qrap
-check [ -f qrap ]
-
-test Build all
-host make clean
-check ! [ -e passt ]
-check ! [ -e pasta ]
-check ! [ -e qrap ]
-host CFLAGS="-Werror" make
-check [ -f passt ]
-check [ -h pasta ]
-check [ -f qrap ]
-
-test Install
-host mkdir __STATEDIR__/prefix
-host prefix=__STATEDIR__/prefix make install
-check [ -f __STATEDIR__/prefix/bin/passt ]
-check [ -h __STATEDIR__/prefix/bin/pasta ]
-check [ -f __STATEDIR__/prefix/bin/qrap ]
-check man -M __STATEDIR__/prefix/share/man -W passt
-check man -M __STATEDIR__/prefix/share/man -W pasta
-check man -M __STATEDIR__/prefix/share/man -W qrap
-
-test Uninstall
-host prefix=__STATEDIR__/prefix make uninstall
-check ! [ -f __STATEDIR__/prefix/bin/passt ]
-check ! [ -h __STATEDIR__/prefix/bin/pasta ]
-check ! [ -f __STATEDIR__/prefix/bin/qrap ]
-check ! man -M __STATEDIR__/prefix/share/man -W passt 2>/dev/null
-check ! man -M __STATEDIR__/prefix/share/man -W pasta 2>/dev/null
-check ! man -M __STATEDIR__/prefix/share/man -W qrap 2>/dev/null
diff --git a/test/build/build.py b/test/build/build.py
new file mode 100755
index 00000000..e49287c9
--- /dev/null
+++ b/test/build/build.py
@@ -0,0 +1,109 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/build/build.py - Test build and install targets
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+import contextlib
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+from typing import Iterable, Iterator
+
+import exeter
+
+def sh(cmd):
+ """Run given command in a shell"""
+ subprocess.run(cmd, shell=True)
+
+
+@contextlib.contextmanager
+def clone_sources() -> Iterator[str]:
+ """Create a temporary copy of the passt sources.
+
+ When the context enters create a temporary directory and copy the
+ passt sources into it. Clean it up when the context exits.
+ """
+
+ os.chdir('..') # Move from test/ to repo base
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+ sh(f"cp --parents -d $(git ls-files) {tmpdir}")
+ os.chdir(tmpdir)
+ yield tmpdir
+
+
+def test_make(target: str, expected_files: list[str]) -> None:
+ """Test `make {target}`
+
+ Arguments:
+ target -- make target to invoke
+ expected_files -- files make is expected to create
+
+ Verifies that
+ 1) `make target` completes successfully
+ 2) expected_files care created by `make target`
+ 3) expected_files are removed by `make clean`
+ """
+
+ ex_paths = [Path(f) for f in expected_files]
+ with clone_sources():
+ for p in ex_paths:
+ assert not p.exists(), f"{p} existed before make"
+ sh(f'make {target} CFLAGS="-Werror"')
+ for p in ex_paths:
+ assert p.exists(), f"{p} wasn't made"
+ sh('make clean')
+ for p in ex_paths:
+ assert not p.exists(), f"{p} existed after make clean"
+
+
+exeter.register('make_passt', test_make, 'passt', ['passt'])
+exeter.register('make_pasta', test_make, 'pasta', ['pasta'])
+exeter.register('make_qrap', test_make, 'qrap', ['qrap'])
+exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_install_uninstall() -> None:
+ """Test `make install` and `make uninstall`
+
+ Tests that `make install` installs the expected files to the
+ install prefix, and that `make uninstall` removes them again.
+ """
+
+ with clone_sources():
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \
+ as prefix:
+ bindir = Path(prefix) / 'bin'
+ mandir = Path(prefix) / 'share/man'
+ progs = ['passt', 'pasta', 'qrap']
+
+ # Install
+ sh(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+ for prog in progs:
+ exe = bindir / prog
+ assert exe.is_file(), f"{exe} does not exist as a regular file"
+ sh(f'man -M {mandir} -W {prog}')
+
+ # Uninstall
+ sh(f'make uninstall prefix={prefix}')
+
+ for prog in progs:
+ exe = bindir / prog
+ assert not exe.exists(), f"{exe} exists after uninstall"
+ sh(f'! man -M {mandir} -W {prog}')
+
+
+if __name__ == '__main__':
+ exeter.main()
diff --git a/test/run b/test/run
index 9e26952c..52507f1d 100755
--- a/test/run
+++ b/test/run
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
COMMIT="$(git log --oneline --no-decorate -1)"
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
. lib/util
. lib/context
. lib/setup
@@ -69,12 +72,9 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/build.py
exeter build/static_checkers.sh
- setup build
- test build/all
- teardown build
-
setup pasta
test pasta/ndp
test pasta/dhcp
--
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
COMMIT="$(git log --oneline --no-decorate -1)"
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
. lib/util
. lib/context
. lib/setup
@@ -69,12 +72,9 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/build.py
exeter build/static_checkers.sh
- setup build
- test build/all
- teardown build
-
setup pasta
test pasta/ndp
test pasta/dhcp
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v7 4/4] test: Allow exeter & podman tests to be parallel executed with BATS
2025-09-04 2:50 [PATCH v7 0/4] Introduce first exeter based test infrastructure David Gibson
` (2 preceding siblings ...)
2025-09-04 2:50 ` [PATCH v7 3/4] test: Convert build tests to exeter David Gibson
@ 2025-09-04 2:50 ` David Gibson
2025-09-04 23:14 ` [PATCH v7 0/4] Introduce first exeter based test infrastructure Stefano Brivio
4 siblings, 0 replies; 10+ messages in thread
From: David Gibson @ 2025-09-04 2:50 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
As a proof-of-concept, this adds a "make bats" target which will execute
both the podman and exeter based testcases in parallel using BATS.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/.gitignore | 1 +
test/Makefile | 14 ++++++++++++++
2 files changed, 15 insertions(+)
diff --git a/test/.gitignore b/test/.gitignore
index cf48b885..9412f0d3 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -12,3 +12,4 @@ rampstream
guest-key
guest-key.pub
/exeter/
+*.bats
diff --git a/test/Makefile b/test/Makefile
index bfb58f27..a774285e 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,6 +5,7 @@
# Copyright Red Hat
# Author: David Gibson <david@gibson.dropbear.id.au>
+BATS = bats -j $(shell nproc)
EXETOOL = exeter/exetool/exetool
WGET = wget -c
@@ -63,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+ build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+ podman/test/system/505-networking-pasta.bats
+
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
assets: $(ASSETS)
@@ -121,6 +128,12 @@ medium.bin:
big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+ PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+ PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
check: assets
./run
@@ -130,6 +143,7 @@ debug: assets
clean:
rm -f perf.js *~
rm -f $(LOCAL_ASSETS)
+ rm -f $(EXETER_BATS)
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
--
@@ -5,6 +5,7 @@
# Copyright Red Hat
# Author: David Gibson <david@gibson.dropbear.id.au>
+BATS = bats -j $(shell nproc)
EXETOOL = exeter/exetool/exetool
WGET = wget -c
@@ -63,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+ build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+ podman/test/system/505-networking-pasta.bats
+
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
assets: $(ASSETS)
@@ -121,6 +128,12 @@ medium.bin:
big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+ PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+ PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
check: assets
./run
@@ -130,6 +143,7 @@ debug: assets
clean:
rm -f perf.js *~
rm -f $(LOCAL_ASSETS)
+ rm -f $(EXETER_BATS)
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
--
2.51.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] Introduce first exeter based test infrastructure
2025-09-04 2:50 [PATCH v7 0/4] Introduce first exeter based test infrastructure David Gibson
` (3 preceding siblings ...)
2025-09-04 2:50 ` [PATCH v7 4/4] test: Allow exeter & podman tests to be parallel executed with BATS David Gibson
@ 2025-09-04 23:14 ` Stefano Brivio
2025-09-05 1:35 ` David Gibson
4 siblings, 1 reply; 10+ messages in thread
From: Stefano Brivio @ 2025-09-04 23:14 UTC (permalink / raw)
To: David Gibson; +Cc: passt-dev
On Thu, 4 Sep 2025 12:50:00 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:
> Here's a new approach to building passt tests with exeter. This new
> one no longer uses Avocado in the default case, although it would
> still be possible to manually run the exeter based tests with Avocado.
>
> For now this only does simple tests, to show how the integration could
> work. It adds some new trivial "smoke tests" and converts the linter
> and build checks to exeter. More complex tests will require building
> the tunbridge library we've discussed. A lot of the work for that
> already exists in my earlier exeter test series, but it will need some
> rework to split it into a separate component.
>
> v7:
> * Use "exetool list" rather than "--list" directly to get test manifest.
> * Update Makefiles to automatically update exeter sources as necessary.
> * Update series cover letter removing some outdated information.
I was about to apply this, but then realised that the usual build test
links at https://passt.top/#ci disappeared, see also
https://passt.top/builds/latest/web/ci.js. I didn't really investigate
yet, we're missing something equivalent to the "test" directive. If I
recall correctly, that was working in v5, but I'm not sure why. I still
need to look into this, unless you figure it out meanwhile.
Something minor I noticed is that indentation of some lines in
run_selected() is broken now (spaces instead of tabs).
--
Stefano
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] Introduce first exeter based test infrastructure
2025-09-04 23:14 ` [PATCH v7 0/4] Introduce first exeter based test infrastructure Stefano Brivio
@ 2025-09-05 1:35 ` David Gibson
2025-09-05 6:42 ` David Gibson
0 siblings, 1 reply; 10+ messages in thread
From: David Gibson @ 2025-09-05 1:35 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-dev
[-- Attachment #1: Type: text/plain, Size: 2381 bytes --]
On Fri, Sep 05, 2025 at 01:14:33AM +0200, Stefano Brivio wrote:
> On Thu, 4 Sep 2025 12:50:00 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
>
> > Here's a new approach to building passt tests with exeter. This new
> > one no longer uses Avocado in the default case, although it would
> > still be possible to manually run the exeter based tests with Avocado.
> >
> > For now this only does simple tests, to show how the integration could
> > work. It adds some new trivial "smoke tests" and converts the linter
> > and build checks to exeter. More complex tests will require building
> > the tunbridge library we've discussed. A lot of the work for that
> > already exists in my earlier exeter test series, but it will need some
> > rework to split it into a separate component.
> >
> > v7:
> > * Use "exetool list" rather than "--list" directly to get test manifest.
> > * Update Makefiles to automatically update exeter sources as necessary.
> > * Update series cover letter removing some outdated information.
>
> I was about to apply this, but then realised that the usual build test
> links at https://passt.top/#ci disappeared, see also
> https://passt.top/builds/latest/web/ci.js.
Huh, that's weird.
> I didn't really investigate
> yet, we're missing something equivalent to the "test" directive.
We shouldn't be. exeter() calls status_test_{start,ok,fail} much like
test().
> If I
> recall correctly, that was working in v5, but I'm not sure why. I still
> need to look into this, unless you figure it out meanwhile.
Right, you said v5 worked. Hrm.... one of the main differences since
v5 is getting the descriptions, which *are* used as the parameter to
status_test_start. Could the descriptions be using some character we
weren't before and we're missing an escape somewhere? Not seeing an
obvious spot.
> Something minor I noticed is that indentation of some lines in
> run_selected() is broken now (spaces instead of tabs).
Drat, I thought I fixed my editor config for that. I've fixed it in
my branch, I'll hold of on another spin until we figure out what's
going on with the CI links.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] Introduce first exeter based test infrastructure
2025-09-05 1:35 ` David Gibson
@ 2025-09-05 6:42 ` David Gibson
2025-09-05 11:01 ` Stefano Brivio
0 siblings, 1 reply; 10+ messages in thread
From: David Gibson @ 2025-09-05 6:42 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-dev
[-- Attachment #1: Type: text/plain, Size: 3035 bytes --]
On Fri, Sep 05, 2025 at 11:35:53AM +1000, David Gibson wrote:
> On Fri, Sep 05, 2025 at 01:14:33AM +0200, Stefano Brivio wrote:
> > On Thu, 4 Sep 2025 12:50:00 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >
> > > Here's a new approach to building passt tests with exeter. This new
> > > one no longer uses Avocado in the default case, although it would
> > > still be possible to manually run the exeter based tests with Avocado.
> > >
> > > For now this only does simple tests, to show how the integration could
> > > work. It adds some new trivial "smoke tests" and converts the linter
> > > and build checks to exeter. More complex tests will require building
> > > the tunbridge library we've discussed. A lot of the work for that
> > > already exists in my earlier exeter test series, but it will need some
> > > rework to split it into a separate component.
> > >
> > > v7:
> > > * Use "exetool list" rather than "--list" directly to get test manifest.
> > > * Update Makefiles to automatically update exeter sources as necessary.
> > > * Update series cover letter removing some outdated information.
> >
> > I was about to apply this, but then realised that the usual build test
> > links at https://passt.top/#ci disappeared, see also
> > https://passt.top/builds/latest/web/ci.js.
>
> Huh, that's weird.
>
> > I didn't really investigate
> > yet, we're missing something equivalent to the "test" directive.
>
> We shouldn't be. exeter() calls status_test_{start,ok,fail} much like
> test().
Ah.. but I didn't call video_link, like test_one() did. Which
explains why the new tests weren't getting links... but not why it
broke the existing ones.
Well, I've put that in, and ci.js looks sane to me, but I don't really
know how that video stuff works. I'll send v8.
> > If I
> > recall correctly, that was working in v5, but I'm not sure why. I still
> > need to look into this, unless you figure it out meanwhile.
>
> Right, you said v5 worked. Hrm.... one of the main differences since
> v5 is getting the descriptions, which *are* used as the parameter to
> status_test_start. Could the descriptions be using some character we
> weren't before and we're missing an escape somewhere? Not seeing an
> obvious spot.
>
> > Something minor I noticed is that indentation of some lines in
> > run_selected() is broken now (spaces instead of tabs).
>
> Drat, I thought I fixed my editor config for that. I've fixed it in
> my branch, I'll hold of on another spin until we figure out what's
> going on with the CI links.
>
> --
> David Gibson (he or they) | I'll have my music baroque, and my code
> david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
> | around.
> http://www.ozlabs.org/~dgibson
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] Introduce first exeter based test infrastructure
2025-09-05 6:42 ` David Gibson
@ 2025-09-05 11:01 ` Stefano Brivio
2025-09-08 1:52 ` David Gibson
0 siblings, 1 reply; 10+ messages in thread
From: Stefano Brivio @ 2025-09-05 11:01 UTC (permalink / raw)
To: David Gibson; +Cc: passt-dev
On Fri, 5 Sep 2025 16:42:58 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:
> On Fri, Sep 05, 2025 at 11:35:53AM +1000, David Gibson wrote:
> > On Fri, Sep 05, 2025 at 01:14:33AM +0200, Stefano Brivio wrote:
> > > On Thu, 4 Sep 2025 12:50:00 +1000
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >
> > > > Here's a new approach to building passt tests with exeter. This new
> > > > one no longer uses Avocado in the default case, although it would
> > > > still be possible to manually run the exeter based tests with Avocado.
> > > >
> > > > For now this only does simple tests, to show how the integration could
> > > > work. It adds some new trivial "smoke tests" and converts the linter
> > > > and build checks to exeter. More complex tests will require building
> > > > the tunbridge library we've discussed. A lot of the work for that
> > > > already exists in my earlier exeter test series, but it will need some
> > > > rework to split it into a separate component.
> > > >
> > > > v7:
> > > > * Use "exetool list" rather than "--list" directly to get test manifest.
> > > > * Update Makefiles to automatically update exeter sources as necessary.
> > > > * Update series cover letter removing some outdated information.
> > >
> > > I was about to apply this, but then realised that the usual build test
> > > links at https://passt.top/#ci disappeared, see also
> > > https://passt.top/builds/latest/web/ci.js.
> >
> > Huh, that's weird.
> >
> > > I didn't really investigate
> > > yet, we're missing something equivalent to the "test" directive.
> >
> > We shouldn't be. exeter() calls status_test_{start,ok,fail} much like
> > test().
>
> Ah.. but I didn't call video_link, like test_one() did. Which
> explains why the new tests weren't getting links... but not why it
> broke the existing ones.
No, no, existing ones are fine. Just the *build* test links were
missing. It all works now (with v8).
> Well, I've put that in, and ci.js looks sane to me, but I don't really
> know how that video stuff works. I'll send v8.
It's write-only code but it's conceptually simple: we record the start
timestamp (based on /proc/uptime) of each test file and add that,
together with a key, to a JavaScript array, video_cilinks.
One 'a' HTML element (link) is also generated for each test file, with
an 'id' attribute matching that key, and when you click on that link, a
listener will look into video_cilink and use the matching start
timestamp as seek time for AsciinemaPlayer.
> > > If I
> > > recall correctly, that was working in v5, but I'm not sure why. I still
> > > need to look into this, unless you figure it out meanwhile.
> >
> > Right, you said v5 worked. Hrm.... one of the main differences since
> > v5 is getting the descriptions, which *are* used as the parameter to
> > status_test_start. Could the descriptions be using some character we
> > weren't before and we're missing an escape somewhere? Not seeing an
> > obvious spot.
I guess I didn't actually recall correctly, or didn't check those
links, because it couldn't have worked in v5 either.
--
Stefano
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: [PATCH v7 0/4] Introduce first exeter based test infrastructure
2025-09-05 11:01 ` Stefano Brivio
@ 2025-09-08 1:52 ` David Gibson
0 siblings, 0 replies; 10+ messages in thread
From: David Gibson @ 2025-09-08 1:52 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-dev
[-- Attachment #1: Type: text/plain, Size: 3728 bytes --]
On Fri, Sep 05, 2025 at 01:01:54PM +0200, Stefano Brivio wrote:
> On Fri, 5 Sep 2025 16:42:58 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
>
> > On Fri, Sep 05, 2025 at 11:35:53AM +1000, David Gibson wrote:
> > > On Fri, Sep 05, 2025 at 01:14:33AM +0200, Stefano Brivio wrote:
> > > > On Thu, 4 Sep 2025 12:50:00 +1000
> > > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > > >
> > > > > Here's a new approach to building passt tests with exeter. This new
> > > > > one no longer uses Avocado in the default case, although it would
> > > > > still be possible to manually run the exeter based tests with Avocado.
> > > > >
> > > > > For now this only does simple tests, to show how the integration could
> > > > > work. It adds some new trivial "smoke tests" and converts the linter
> > > > > and build checks to exeter. More complex tests will require building
> > > > > the tunbridge library we've discussed. A lot of the work for that
> > > > > already exists in my earlier exeter test series, but it will need some
> > > > > rework to split it into a separate component.
> > > > >
> > > > > v7:
> > > > > * Use "exetool list" rather than "--list" directly to get test manifest.
> > > > > * Update Makefiles to automatically update exeter sources as necessary.
> > > > > * Update series cover letter removing some outdated information.
> > > >
> > > > I was about to apply this, but then realised that the usual build test
> > > > links at https://passt.top/#ci disappeared, see also
> > > > https://passt.top/builds/latest/web/ci.js.
> > >
> > > Huh, that's weird.
> > >
> > > > I didn't really investigate
> > > > yet, we're missing something equivalent to the "test" directive.
> > >
> > > We shouldn't be. exeter() calls status_test_{start,ok,fail} much like
> > > test().
> >
> > Ah.. but I didn't call video_link, like test_one() did. Which
> > explains why the new tests weren't getting links... but not why it
> > broke the existing ones.
>
> No, no, existing ones are fine. Just the *build* test links were
> missing. It all works now (with v8).
Ah, ok.
> > Well, I've put that in, and ci.js looks sane to me, but I don't really
> > know how that video stuff works. I'll send v8.
>
> It's write-only code but it's conceptually simple: we record the start
> timestamp (based on /proc/uptime) of each test file and add that,
> together with a key, to a JavaScript array, video_cilinks.
>
> One 'a' HTML element (link) is also generated for each test file, with
> an 'id' attribute matching that key, and when you click on that link, a
> listener will look into video_cilink and use the matching start
> timestamp as seek time for AsciinemaPlayer.
Right. I pretty much figured that out as I was looking at it.
> > > > If I
> > > > recall correctly, that was working in v5, but I'm not sure why. I still
> > > > need to look into this, unless you figure it out meanwhile.
> > >
> > > Right, you said v5 worked. Hrm.... one of the main differences since
> > > v5 is getting the descriptions, which *are* used as the parameter to
> > > status_test_start. Could the descriptions be using some character we
> > > weren't before and we're missing an escape somewhere? Not seeing an
> > > obvious spot.
>
> I guess I didn't actually recall correctly, or didn't check those
> links, because it couldn't have worked in v5 either.
Right. I'd also discarded this theory in the meantime.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread