* [PATCH v5 1/4] test: Extend test scripts to allow running exeter tests.
2025-08-20 10:54 [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests David Gibson
@ 2025-08-20 10:54 ` David Gibson
2025-08-20 10:54 ` [PATCH v5 2/4] test: Run static checkers as " David Gibson
` (3 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: David Gibson @ 2025-08-20 10:54 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Introduce some trivial testcases based on the exeter library. These run
passt and pasta with --help and --version options. Extend our test
scripts to run these tests.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/.gitignore | 1 +
test/Makefile | 5 +++-
test/lib/exeter | 56 +++++++++++++++++++++++++++++++++++++++++++++
test/run | 9 +++++++-
test/smoke/smoke.sh | 27 ++++++++++++++++++++++
5 files changed, 96 insertions(+), 2 deletions(-)
create mode 100644 test/lib/exeter
create mode 100755 test/smoke/smoke.sh
diff --git a/test/.gitignore b/test/.gitignore
index 3573444f..cf48b885 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -11,3 +11,4 @@ nstool
rampstream
guest-key
guest-key.pub
+/exeter/
diff --git a/test/Makefile b/test/Makefile
index bf63db87..332f3f3e 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -50,7 +50,7 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \
jammy-server-cloudimg-s390x.img
UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS)
-DOWNLOAD_ASSETS = mbuto podman \
+DOWNLOAD_ASSETS = exeter mbuto podman \
$(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS)
TESTDATA_ASSETS = small.bin big.bin medium.bin \
rampstream
@@ -70,6 +70,9 @@ assets: $(ASSETS)
pull-%: %
git -C $* pull
+exeter:
+ git clone https://gitlab.com/dgibson/exeter.git
+
mbuto:
git clone git://mbuto.sh/mbuto
diff --git a/test/lib/exeter b/test/lib/exeter
new file mode 100644
index 00000000..e381cc2d
--- /dev/null
+++ b/test/lib/exeter
@@ -0,0 +1,56 @@
+#!/bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/lib/exeter - Run exeter tests within the rest of passt's tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+EXETOOL="$BASEPATH/exeter/exetool/exetool"
+
+# is_exeter() - Determine if a test file is an exeter program
+# $@: Command line to invoke test program
+is_exeter() {
+ $EXETOOL probe -- "$@"
+}
+
+# exeter() - Run each test in an exeter program, logging each test separately
+# $@: Command line to invoke exeter test program
+exeter() {
+ STATESETUP="${STATEBASE}/$1"
+ mkdir -p "${STATESETUP}"
+
+ context_setup_host host
+ layout_host
+
+ cd test
+
+ __ntests=$("$@" --list | wc -l)
+ if [ $? != 0 ]; then
+ info "Failed to get exeter manifest for $@"
+ pause_continue \
+ "Press any key to pause test session" \
+ "Resuming in " \
+ "Paused, press any key to continue" \
+ 5
+ return
+ fi
+
+ status_file_start "$* (exeter)" ${__ntests}
+
+ for __testid in $("$@" --list); do
+ status_test_start "${__testid}"
+ context_run host "$@" "${__testid}" && status_test_ok || status_test_fail
+ done
+
+ cd ..
+
+ teardown_context_watch ${PANE_HOST} host
+}
diff --git a/test/run b/test/run
index f73c3119..745a09fa 100755
--- a/test/run
+++ b/test/run
@@ -53,6 +53,7 @@ COMMIT="$(git log --oneline --no-decorate -1)"
. lib/layout_ugly
. lib/test
. lib/video
+. lib/exeter
# cleanup() - Remove temporary files
cleanup() {
@@ -67,6 +68,8 @@ run() {
perf_init
[ ${CI} -eq 1 ] && video_start ci
+ exeter smoke/smoke.sh
+
setup build
test build/all
test build/cppcheck
@@ -223,6 +226,10 @@ run_selected() {
__setup=
for __test; do
+ if is_exeter "test/${__test}"; then
+ exeter "${__test}"
+ continue
+ fi
# HACK: the migrate tests need the setup repeated for
# each test
if [ "${__test%%/*}" != "${__setup}" -o \
@@ -234,7 +241,7 @@ run_selected() {
test "${__test}"
done
- teardown "${__setup}"
+ [ -n "${__setup}" ] && teardown "${__setup}"
log "PASS: ${STATUS_PASS}, FAIL: ${STATUS_FAIL}, SKIPPED: ${STATUS_SKIPPED}"
diff --git a/test/smoke/smoke.sh b/test/smoke/smoke.sh
new file mode 100755
index 00000000..b01c7c89
--- /dev/null
+++ b/test/smoke/smoke.sh
@@ -0,0 +1,27 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_register pasta_version $PASTA --version
+
+exeter_register passt_help $PASST --help
+exeter_register pasta_help $PASTA --help
+
+exeter_main "$@"
--
@@ -0,0 +1,27 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/smoke/smoke.sh - Basic smoke tests
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+PASST=$(dirname $0)/../../passt
+PASTA=$(dirname $0)/../../pasta
+
+exeter_register passt_version $PASST --version
+exeter_register pasta_version $PASTA --version
+
+exeter_register passt_help $PASST --help
+exeter_register pasta_help $PASTA --help
+
+exeter_main "$@"
--
2.50.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v5 2/4] test: Run static checkers as exeter tests
2025-08-20 10:54 [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests David Gibson
2025-08-20 10:54 ` [PATCH v5 1/4] test: Extend test scripts to allow running " David Gibson
@ 2025-08-20 10:54 ` David Gibson
2025-08-20 10:54 ` [PATCH v5 3/4] test: Convert build tests to exeter David Gibson
` (2 subsequent siblings)
4 siblings, 0 replies; 9+ messages in thread
From: David Gibson @ 2025-08-20 10:54 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Move the static checkers from the current DSL to exeter.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/build/clang_tidy | 17 -----------------
test/build/cppcheck | 17 -----------------
test/build/static_checkers.sh | 30 ++++++++++++++++++++++++++++++
test/run | 3 +--
4 files changed, 31 insertions(+), 36 deletions(-)
delete mode 100644 test/build/clang_tidy
delete mode 100644 test/build/cppcheck
create mode 100755 test/build/static_checkers.sh
diff --git a/test/build/clang_tidy b/test/build/clang_tidy
deleted file mode 100644
index 40573bfd..00000000
--- a/test/build/clang_tidy
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/clang_tidy - Run source through clang-tidy(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools clang-tidy
-
-test Run clang-tidy
-host make clang-tidy
diff --git a/test/build/cppcheck b/test/build/cppcheck
deleted file mode 100644
index 0e1dbced..00000000
--- a/test/build/cppcheck
+++ /dev/null
@@ -1,17 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/cppcheck - Run source through cppcheck(1) linter
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools cppcheck
-
-test Run cppcheck
-host make cppcheck
diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
new file mode 100755
index 00000000..e6e1e729
--- /dev/null
+++ b/test/build/static_checkers.sh
@@ -0,0 +1,30 @@
+#! /bin/sh
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/build/static_checkers.sh - Run static checkers
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+. $(dirname $0)/../exeter/sh/exeter.sh
+
+cppcheck () {
+ make -C .. cppcheck
+}
+exeter_register cppcheck
+
+clang_tidy () {
+ make -C .. clang-tidy
+}
+exeter_register clang_tidy
+
+exeter_main "$@"
+
+
diff --git a/test/run b/test/run
index 745a09fa..9e26952c 100755
--- a/test/run
+++ b/test/run
@@ -69,11 +69,10 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/static_checkers.sh
setup build
test build/all
- test build/cppcheck
- test build/clang_tidy
teardown build
setup pasta
--
@@ -69,11 +69,10 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/static_checkers.sh
setup build
test build/all
- test build/cppcheck
- test build/clang_tidy
teardown build
setup pasta
--
2.50.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v5 3/4] test: Convert build tests to exeter
2025-08-20 10:54 [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests David Gibson
2025-08-20 10:54 ` [PATCH v5 1/4] test: Extend test scripts to allow running " David Gibson
2025-08-20 10:54 ` [PATCH v5 2/4] test: Run static checkers as " David Gibson
@ 2025-08-20 10:54 ` David Gibson
2025-08-20 10:54 ` [PATCH v5 4/4] test: Allow exeter & podman tests to be parallel executed with BATS David Gibson
2025-08-20 20:40 ` [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests Stefano Brivio
4 siblings, 0 replies; 9+ messages in thread
From: David Gibson @ 2025-08-20 10:54 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
Convert the tests in build/all to be based on exeter. The new version of
the tests is more robust than the original, since it makes a temporary copy
of the source tree so will not be affected by concurrent manual builds.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/build/all | 61 -------------------------
test/build/build.py | 109 ++++++++++++++++++++++++++++++++++++++++++++
test/run | 8 ++--
3 files changed, 113 insertions(+), 65 deletions(-)
delete mode 100644 test/build/all
create mode 100755 test/build/build.py
diff --git a/test/build/all b/test/build/all
deleted file mode 100644
index 1f79e0d8..00000000
--- a/test/build/all
+++ /dev/null
@@ -1,61 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-or-later
-#
-# PASST - Plug A Simple Socket Transport
-# for qemu/UNIX domain socket mode
-#
-# PASTA - Pack A Subtle Tap Abstraction
-# for network namespace/tap device mode
-#
-# test/build/all - Build targets, one by one, then all together, check output
-#
-# Copyright (c) 2021 Red Hat GmbH
-# Author: Stefano Brivio <sbrivio@redhat.com>
-
-htools make cc rm uname getconf mkdir cp rm man
-
-test Build passt
-host make clean
-check ! [ -e passt ]
-host CFLAGS="-Werror" make passt
-check [ -f passt ]
-
-test Build pasta
-host make clean
-check ! [ -e pasta ]
-host CFLAGS="-Werror" make pasta
-check [ -h pasta ]
-
-test Build qrap
-host make clean
-check ! [ -e qrap ]
-host CFLAGS="-Werror" make qrap
-check [ -f qrap ]
-
-test Build all
-host make clean
-check ! [ -e passt ]
-check ! [ -e pasta ]
-check ! [ -e qrap ]
-host CFLAGS="-Werror" make
-check [ -f passt ]
-check [ -h pasta ]
-check [ -f qrap ]
-
-test Install
-host mkdir __STATEDIR__/prefix
-host prefix=__STATEDIR__/prefix make install
-check [ -f __STATEDIR__/prefix/bin/passt ]
-check [ -h __STATEDIR__/prefix/bin/pasta ]
-check [ -f __STATEDIR__/prefix/bin/qrap ]
-check man -M __STATEDIR__/prefix/share/man -W passt
-check man -M __STATEDIR__/prefix/share/man -W pasta
-check man -M __STATEDIR__/prefix/share/man -W qrap
-
-test Uninstall
-host prefix=__STATEDIR__/prefix make uninstall
-check ! [ -f __STATEDIR__/prefix/bin/passt ]
-check ! [ -h __STATEDIR__/prefix/bin/pasta ]
-check ! [ -f __STATEDIR__/prefix/bin/qrap ]
-check ! man -M __STATEDIR__/prefix/share/man -W passt 2>/dev/null
-check ! man -M __STATEDIR__/prefix/share/man -W pasta 2>/dev/null
-check ! man -M __STATEDIR__/prefix/share/man -W qrap 2>/dev/null
diff --git a/test/build/build.py b/test/build/build.py
new file mode 100755
index 00000000..9c7d9c23
--- /dev/null
+++ b/test/build/build.py
@@ -0,0 +1,109 @@
+#! /usr/bin/env python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+# for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+# for network namespace/tap device mode
+#
+# test/build/build.py - Test build and install targets
+#
+# Copyright Red Hat
+# Author: David Gibson <david@gibson.dropbear.id.au>
+
+import contextlib
+import os
+from pathlib import Path
+import subprocess
+import tempfile
+from typing import Iterable, Iterator
+
+import exeter
+
+def sh(cmd):
+ """Run given command in a shell"""
+ subprocess.run(cmd, shell=True)
+
+
+@contextlib.contextmanager
+def clone_sources() -> Iterator[str]:
+ """Create a temporary copy of the passt sources.
+
+ When the context enters create a temporary directory and copy the
+ passt sources into it. Clean it up when the context exits.
+ """
+
+ os.chdir('..') # Move from test/ to repo base
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+ sh(f"cp --parents -d $(git ls-files) {tmpdir}")
+ os.chdir(tmpdir)
+ yield tmpdir
+
+
+def test_make(target: str, expected_files: list[str]) -> None:
+ """Check a make target works as expected.
+
+ Arguments:
+ target -- make target to invoke
+ expected_files -- files make is expected to create
+
+ Verifies that
+ 1) `make target` completes successfully
+ 2) expected_files care created by `make target`
+ 3) expected_files are removed by `make clean`
+ """
+
+ ex_paths = [Path(f) for f in expected_files]
+ with clone_sources():
+ for p in ex_paths:
+ assert not p.exists(), f"{p} existed before make"
+ sh(f'make {target} CFLAGS="-Werror"')
+ for p in ex_paths:
+ assert p.exists(), f"{p} wasn't made"
+ sh('make clean')
+ for p in ex_paths:
+ assert not p.exists(), f"{p} existed after make clean"
+
+
+exeter.register('make_passt', test_make, 'passt', ['passt'])
+exeter.register('make_pasta', test_make, 'pasta', ['pasta'])
+exeter.register('make_qrap', test_make, 'qrap', ['qrap'])
+exeter.register('make_all', test_make, 'all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_install_uninstall() -> None:
+ """Test `make install` and `make uninstall work as expected
+
+ Tests that `make install` installs the expected files to the
+ install prefix, and that `make uninstall` removes them again.
+ """
+
+ with clone_sources():
+ with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) \
+ as prefix:
+ bindir = Path(prefix) / 'bin'
+ mandir = Path(prefix) / 'share/man'
+ progs = ['passt', 'pasta', 'qrap']
+
+ # Install
+ sh(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+ for prog in progs:
+ exe = bindir / prog
+ assert exe.is_file(), f"{exe} does not exist as a regular file"
+ sh(f'man -M {mandir} -W {prog}')
+
+ # Uninstall
+ sh(f'make uninstall prefix={prefix}')
+
+ for prog in progs:
+ exe = bindir / prog
+ assert not exe.exists(), f"{exe} exists after uninstall"
+ sh(f'! man -M {mandir} -W {prog}')
+
+
+if __name__ == '__main__':
+ exeter.main()
diff --git a/test/run b/test/run
index 9e26952c..52507f1d 100755
--- a/test/run
+++ b/test/run
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
COMMIT="$(git log --oneline --no-decorate -1)"
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
. lib/util
. lib/context
. lib/setup
@@ -69,12 +72,9 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/build.py
exeter build/static_checkers.sh
- setup build
- test build/all
- teardown build
-
setup pasta
test pasta/ndp
test pasta/dhcp
--
@@ -43,6 +43,9 @@ KERNEL=${KERNEL:-"/boot/vmlinuz-$(uname -r)"}
COMMIT="$(git log --oneline --no-decorate -1)"
+# Let exeter tests written in Python find their modules
+export PYTHONPATH=${BASEPATH}/exeter/py3
+
. lib/util
. lib/context
. lib/setup
@@ -69,12 +72,9 @@ run() {
[ ${CI} -eq 1 ] && video_start ci
exeter smoke/smoke.sh
+ exeter build/build.py
exeter build/static_checkers.sh
- setup build
- test build/all
- teardown build
-
setup pasta
test pasta/ndp
test pasta/dhcp
--
2.50.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [PATCH v5 4/4] test: Allow exeter & podman tests to be parallel executed with BATS
2025-08-20 10:54 [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests David Gibson
` (2 preceding siblings ...)
2025-08-20 10:54 ` [PATCH v5 3/4] test: Convert build tests to exeter David Gibson
@ 2025-08-20 10:54 ` David Gibson
2025-08-20 20:40 ` [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests Stefano Brivio
4 siblings, 0 replies; 9+ messages in thread
From: David Gibson @ 2025-08-20 10:54 UTC (permalink / raw)
To: Stefano Brivio, passt-dev; +Cc: David Gibson
As a proof-of-concept, this adds a "make bats" target which will execute
both the podman and exeter based testcases in parallel using BATS.
Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
test/.gitignore | 1 +
test/Makefile | 17 +++++++++++++++++
2 files changed, 18 insertions(+)
diff --git a/test/.gitignore b/test/.gitignore
index cf48b885..9412f0d3 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -12,3 +12,4 @@ rampstream
guest-key
guest-key.pub
/exeter/
+*.bats
diff --git a/test/Makefile b/test/Makefile
index 332f3f3e..1853fea3 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -5,7 +5,9 @@
# Copyright Red Hat
# Author: David Gibson <david@gibson.dropbear.id.au>
+BATS = bats -j $(shell nproc)
WGET = wget -c
+EXETOOL = ./exeter/exetool/exetool
DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \
debian-10-nocloud-amd64.qcow2 \
@@ -62,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+ build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+ podman/test/system/505-networking-pasta.bats
+
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
assets: $(ASSETS)
@@ -73,6 +81,8 @@ pull-%: %
exeter:
git clone https://gitlab.com/dgibson/exeter.git
+$(EXETOOL): pull-exeter
+
mbuto:
git clone git://mbuto.sh/mbuto
@@ -118,6 +128,12 @@ medium.bin:
big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+ PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+ PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
check: assets
./run
@@ -127,6 +143,7 @@ debug: assets
clean:
rm -f perf.js *~
rm -f $(LOCAL_ASSETS)
+ rm -f $(EXETER_BATS)
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
--
@@ -5,7 +5,9 @@
# Copyright Red Hat
# Author: David Gibson <david@gibson.dropbear.id.au>
+BATS = bats -j $(shell nproc)
WGET = wget -c
+EXETOOL = ./exeter/exetool/exetool
DEBIAN_IMGS = debian-8.11.0-openstack-amd64.qcow2 \
debian-10-nocloud-amd64.qcow2 \
@@ -62,6 +64,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
+EXETER_PYPATH = exeter/py3
+EXETER_BATS = smoke/smoke.sh.bats \
+ build/build.py.bats build/static_checkers.sh.bats
+BATS_FILES = $(EXETER_BATS) \
+ podman/test/system/505-networking-pasta.bats
+
CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
assets: $(ASSETS)
@@ -73,6 +81,8 @@ pull-%: %
exeter:
git clone https://gitlab.com/dgibson/exeter.git
+$(EXETOOL): pull-exeter
+
mbuto:
git clone git://mbuto.sh/mbuto
@@ -118,6 +128,12 @@ medium.bin:
big.bin:
dd if=/dev/urandom bs=1M count=10 of=$@
+$(EXETER_BATS): %.bats: % $(EXETOOL)
+ PYTHONPATH=$(EXETER_PYPATH) $(EXETOOL) bats -- $< > $@
+
+bats: $(BATS_FILES) pull-podman
+ PYTHONPATH=$(EXETER_PYPATH) CONTAINERS_HELPER_BINARY_DIR=.. $(BATS) $(BATS_FILES)
+
check: assets
./run
@@ -127,6 +143,7 @@ debug: assets
clean:
rm -f perf.js *~
rm -f $(LOCAL_ASSETS)
+ rm -f $(EXETER_BATS)
rm -rf test_logs
rm -f prepared-*.qcow2 prepared-*.img
--
2.50.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests
2025-08-20 10:54 [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests David Gibson
` (3 preceding siblings ...)
2025-08-20 10:54 ` [PATCH v5 4/4] test: Allow exeter & podman tests to be parallel executed with BATS David Gibson
@ 2025-08-20 20:40 ` Stefano Brivio
2025-08-21 2:48 ` David Gibson
4 siblings, 1 reply; 9+ messages in thread
From: Stefano Brivio @ 2025-08-20 20:40 UTC (permalink / raw)
To: David Gibson; +Cc: passt-dev
On Wed, 20 Aug 2025 20:54:52 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:
> Here's a new approach to building passt tests with exeter. This new
> one no longer uses Avocado in the default case, although it would
> still be possible to manually run the exeter based tests with Avocado.
>
> Here's another draft of my work on testing passt with Avocado and the
> exeter library I recently created. It includes Cleber's patch adding
> some basic Avocado tests and builds on that.
>
> For now this only does simple tests, to show how the integration could
> work. It adds some new trivial "smoke tests" and converts the linter
> and build checks to exeter. More complex tests will require building
> the sinte/pesto library we've discussed. A lot of the work for that
> already exists in my earlier exeter test series, but it will need some
> rework to split it into a separate component.
>
> v5:
> * Updated according to Stefano's review
> - Fixed a number of whitespace errors
> - Improved many comments and variable names to make things clearer
> * New patch adding parallel test execution with BATS
> * Improved autodetection of exeter tests using "exetool probe"
This works on my setup and looks good to me, I just have two comments:
- test names are still the same as before (not exactly descriptive, say,
'make_passt'). I already reported this on v4, I'm not sure what was
your conclusion about it
- I didn't check (yet) what happens when I run this as ./ci (for
example, from the pre-push hook), if generated web links are still
okay. I'll do that soon unless you can have a look first
--
Stefano
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests
2025-08-20 20:40 ` [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests Stefano Brivio
@ 2025-08-21 2:48 ` David Gibson
2025-08-21 21:27 ` Stefano Brivio
0 siblings, 1 reply; 9+ messages in thread
From: David Gibson @ 2025-08-21 2:48 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-dev
[-- Attachment #1: Type: text/plain, Size: 3348 bytes --]
On Wed, Aug 20, 2025 at 10:40:48PM +0200, Stefano Brivio wrote:
> On Wed, 20 Aug 2025 20:54:52 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
>
> > Here's a new approach to building passt tests with exeter. This new
> > one no longer uses Avocado in the default case, although it would
> > still be possible to manually run the exeter based tests with Avocado.
> >
> > Here's another draft of my work on testing passt with Avocado and the
> > exeter library I recently created. It includes Cleber's patch adding
> > some basic Avocado tests and builds on that.
> >
> > For now this only does simple tests, to show how the integration could
> > work. It adds some new trivial "smoke tests" and converts the linter
> > and build checks to exeter. More complex tests will require building
> > the sinte/pesto library we've discussed. A lot of the work for that
> > already exists in my earlier exeter test series, but it will need some
> > rework to split it into a separate component.
> >
> > v5:
> > * Updated according to Stefano's review
> > - Fixed a number of whitespace errors
> > - Improved many comments and variable names to make things clearer
> > * New patch adding parallel test execution with BATS
> > * Improved autodetection of exeter tests using "exetool probe"
>
> This works on my setup and looks good to me, I just have two comments:
>
> - test names are still the same as before (not exactly descriptive, say,
> 'make_passt'). I already reported this on v4, I'm not sure what was
> your conclusion about it
Sorry, I missed that comment on v4.
exeter test ids are by design machine-friendly identifiers more than
they are human-friendly names or descriptions. There are a few
reasons for that:
* The ids need to be passed around between test and runner both on
the command line and via stdio. Limiting them to characters that
are identifier friendly in most languages significantly reduces the
chances of screwing up quoting.
* In some existing Python cases, and maybe more language cases in
future, the ids are auto-generated, e.g. for a matrix or
composition of tests. That works more naturally for
identifiers than names/descriptions.
* Identifiers are more amenable to structured formatting grouping
related tests together, which is useful for filtering out groups of
test by glob/regexp.
* I like having a succinct id to refer to tests by rather than a
waffly English description
I'm not opposed to having an (optional) human-readable name or
description for tests in addition to the id. It would complexify the
exeter protocol, of course, which I'm trying to keep super simple.
Then again, I have several other things in mind that would need
per-test metadata, so it's probably is worth it.
> - I didn't check (yet) what happens when I run this as ./ci (for
> example, from the pre-push hook), if generated web links are still
> okay. I'll do that soon unless you can have a look first
I don't really know how to check that. I don't think there's any
reason it wouldn't work.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests
2025-08-21 2:48 ` David Gibson
@ 2025-08-21 21:27 ` Stefano Brivio
2025-08-25 1:55 ` David Gibson
0 siblings, 1 reply; 9+ messages in thread
From: Stefano Brivio @ 2025-08-21 21:27 UTC (permalink / raw)
To: David Gibson; +Cc: passt-dev
On Thu, 21 Aug 2025 12:48:27 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:
> On Wed, Aug 20, 2025 at 10:40:48PM +0200, Stefano Brivio wrote:
> > On Wed, 20 Aug 2025 20:54:52 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >
> > > Here's a new approach to building passt tests with exeter. This new
> > > one no longer uses Avocado in the default case, although it would
> > > still be possible to manually run the exeter based tests with Avocado.
> > >
> > > Here's another draft of my work on testing passt with Avocado and the
> > > exeter library I recently created. It includes Cleber's patch adding
> > > some basic Avocado tests and builds on that.
> > >
> > > For now this only does simple tests, to show how the integration could
> > > work. It adds some new trivial "smoke tests" and converts the linter
> > > and build checks to exeter. More complex tests will require building
> > > the sinte/pesto library we've discussed. A lot of the work for that
> > > already exists in my earlier exeter test series, but it will need some
> > > rework to split it into a separate component.
> > >
> > > v5:
> > > * Updated according to Stefano's review
> > > - Fixed a number of whitespace errors
> > > - Improved many comments and variable names to make things clearer
> > > * New patch adding parallel test execution with BATS
> > > * Improved autodetection of exeter tests using "exetool probe"
> >
> > This works on my setup and looks good to me, I just have two comments:
> >
> > - test names are still the same as before (not exactly descriptive, say,
> > 'make_passt'). I already reported this on v4, I'm not sure what was
> > your conclusion about it
>
> Sorry, I missed that comment on v4.
>
> exeter test ids are by design machine-friendly identifiers more than
> they are human-friendly names or descriptions. There are a few
> reasons for that:
>
> * The ids need to be passed around between test and runner both on
> the command line and via stdio. Limiting them to characters that
> are identifier friendly in most languages significantly reduces the
> chances of screwing up quoting.
>
> * In some existing Python cases, and maybe more language cases in
> future, the ids are auto-generated, e.g. for a matrix or
> composition of tests. That works more naturally for
> identifiers than names/descriptions.
>
> * Identifiers are more amenable to structured formatting grouping
> related tests together, which is useful for filtering out groups of
> test by glob/regexp.
It looks perhaps a bit awkward to filter Bats-based pasta tests from
Podman with, say, -f TCP, but I actually find it convenient. The test
name is human-friendly, and regexps are still easy.
> * I like having a succinct id to refer to tests by rather than a
> waffly English description
>
> I'm not opposed to having an (optional) human-readable name or
> description for tests in addition to the id. It would complexify the
> exeter protocol, of course, which I'm trying to keep super simple.
...but yes, I see. On the other hand, let's pick something like:
TCP/IPv4: host to ns (spliced): big transfer
would you call that... tcp_v4_host_to_ns_spliced_big? To me that would
look like an obvious regression.
It's very hard on eyes, and much less informative to newcomers (unless
you add "transfer", but then it gets quite long for a machine-friendly
identifier).
And we'll surely run into something worse than that...
> Then again, I have several other things in mind that would need
> per-test metadata, so it's probably is worth it.
I guess we might even want to have some attributes to categorise tests,
eventually. I'm rather clueless as to the amount of complexity it adds,
but it sounds like an obvious choice to me.
> > - I didn't check (yet) what happens when I run this as ./ci (for
> > example, from the pre-push hook), if generated web links are still
> > okay. I'll do that soon unless you can have a look first
>
> I don't really know how to check that. I don't think there's any
> reason it wouldn't work.
Run as ./ci, check that video_link_* links in web/ci.js make kind of
sense. Anyway, never mind, I just checked, they still work.
--
Stefano
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v5 0/4] RFC: New proof-of-concept based exeter tests
2025-08-21 21:27 ` Stefano Brivio
@ 2025-08-25 1:55 ` David Gibson
0 siblings, 0 replies; 9+ messages in thread
From: David Gibson @ 2025-08-25 1:55 UTC (permalink / raw)
To: Stefano Brivio; +Cc: passt-dev
[-- Attachment #1: Type: text/plain, Size: 5736 bytes --]
On Thu, Aug 21, 2025 at 11:27:02PM +0200, Stefano Brivio wrote:
> On Thu, 21 Aug 2025 12:48:27 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
>
> > On Wed, Aug 20, 2025 at 10:40:48PM +0200, Stefano Brivio wrote:
> > > On Wed, 20 Aug 2025 20:54:52 +1000
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
> > >
> > > > Here's a new approach to building passt tests with exeter. This new
> > > > one no longer uses Avocado in the default case, although it would
> > > > still be possible to manually run the exeter based tests with Avocado.
> > > >
> > > > Here's another draft of my work on testing passt with Avocado and the
> > > > exeter library I recently created. It includes Cleber's patch adding
> > > > some basic Avocado tests and builds on that.
> > > >
> > > > For now this only does simple tests, to show how the integration could
> > > > work. It adds some new trivial "smoke tests" and converts the linter
> > > > and build checks to exeter. More complex tests will require building
> > > > the sinte/pesto library we've discussed. A lot of the work for that
> > > > already exists in my earlier exeter test series, but it will need some
> > > > rework to split it into a separate component.
> > > >
> > > > v5:
> > > > * Updated according to Stefano's review
> > > > - Fixed a number of whitespace errors
> > > > - Improved many comments and variable names to make things clearer
> > > > * New patch adding parallel test execution with BATS
> > > > * Improved autodetection of exeter tests using "exetool probe"
> > >
> > > This works on my setup and looks good to me, I just have two comments:
> > >
> > > - test names are still the same as before (not exactly descriptive, say,
> > > 'make_passt'). I already reported this on v4, I'm not sure what was
> > > your conclusion about it
> >
> > Sorry, I missed that comment on v4.
> >
> > exeter test ids are by design machine-friendly identifiers more than
> > they are human-friendly names or descriptions. There are a few
> > reasons for that:
> >
> > * The ids need to be passed around between test and runner both on
> > the command line and via stdio. Limiting them to characters that
> > are identifier friendly in most languages significantly reduces the
> > chances of screwing up quoting.
> >
> > * In some existing Python cases, and maybe more language cases in
> > future, the ids are auto-generated, e.g. for a matrix or
> > composition of tests. That works more naturally for
> > identifiers than names/descriptions.
> >
> > * Identifiers are more amenable to structured formatting grouping
> > related tests together, which is useful for filtering out groups of
> > test by glob/regexp.
>
> It looks perhaps a bit awkward to filter Bats-based pasta tests from
> Podman with, say, -f TCP, but I actually find it convenient. The test
> name is human-friendly, and regexps are still easy.
Point taken. Consider human-readable names descriptions bumped
several notches up my priority table (and at least the basics of
metadata support with it).
> > * I like having a succinct id to refer to tests by rather than a
> > waffly English description
> >
> > I'm not opposed to having an (optional) human-readable name or
> > description for tests in addition to the id. It would complexify the
> > exeter protocol, of course, which I'm trying to keep super simple.
>
> ...but yes, I see. On the other hand, let's pick something like:
>
> TCP/IPv4: host to ns (spliced): big transfer
>
> would you call that... tcp_v4_host_to_ns_spliced_big? To me that would
> look like an obvious regression.
More likely I'd be looking at a name based on the grouping /
structure, maybe 'host_to_splice;tcp4_xfer_big'. Which makes sense
from the point of view of constructing those tests from compositions
of pieces (which should be possible for that case), but is even more
of a readability regression.
> It's very hard on eyes, and much less informative to newcomers (unless
> you add "transfer", but then it gets quite long for a machine-friendly
> identifier).
>
> And we'll surely run into something worse than that...
>
> > Then again, I have several other things in mind that would need
> > per-test metadata, so it's probably is worth it.
>
> I guess we might even want to have some attributes to categorise tests,
> eventually. I'm rather clueless as to the amount of complexity it adds,
> but it sounds like an obvious choice to me.
Right. 'tags' of this type are a very common feature for test
runners. Again, this is primarily just a matter of devising a means
of exposing the metadata. The only thing that makes that tricky is
the fact that every exeter implementation needs to do it, so it's
important to design it so as to keep the implementation complexity to
an absolute minimum.
> > > - I didn't check (yet) what happens when I run this as ./ci (for
> > > example, from the pre-push hook), if generated web links are still
> > > okay. I'll do that soon unless you can have a look first
> >
> > I don't really know how to check that. I don't think there's any
> > reason it wouldn't work.
>
> Run as ./ci, check that video_link_* links in web/ci.js make kind of
> sense. Anyway, never mind, I just checked, they still work.
I'm not sure I know enough about those videos or js to know what makes
sense. Anyway, thanks for checking it.
--
David Gibson (he or they) | I'll have my music baroque, and my code
david AT gibson.dropbear.id.au | minimalist, thank you, not the other way
| around.
http://www.ozlabs.org/~dgibson
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread