[PATCH 0/4] RFC: Proof-of-concept based exeter+Avocado tests

[PATCH 0/4] RFC: Proof-of-concept based exeter+Avocado tests

[David Gibson]

Here's a rough proof of concept showing how we could run tests for
passt with Avocado and the exeter library I recently created.  It
includes Cleber's patch adding some basic Avocado tests and builds on
that.

The current draft is pretty janky:
 * The build rules to download and install the necessary pieces are messy
 * We create the Avocado job files from the exeter sources in the
   Makefile.  Ideally Avocado would eventually be extended to handle
   this itself
 * We have more ugly make rules to mangle the right venv path into the
   exeter based python script
 * The names that Avocado sees for each test are terrible
   (nondescriptive and duplicated).  This is the point I made in an
   earlier reply to Cleber's patch - we need some way of setting the
   test name from the job files.
 * There's some hacks to make sure things are executed from the
   right working directory

But, it's a starting point.

Stefano,

I'm hoping most of the jank in the setup can be cleaned up.  The bit I
really want your opinion on is how the test scripts themslves look:
test/build/static_checkers.exeter (shell) and
test/build/build.exeter.in (Python).

Cleber,

I hope this concrete example makes it a bit clearer how I'm thinking
about running tests from Avocado, without making the test themselves
inherently tied to Avocado.

Cleber Rosa (1):
  test: run static checkers with Avocado and JSON definitions

David Gibson (3):
  test: Extend make targets to run Avocado tests
  test: Exeter based static tests
  test: Add exeter+Avocado based build tests

 test/.gitignore                   |  2 +
 test/Makefile                     | 33 +++++++++++-
 test/avocado/static_checkers.json | 16 ++++++
 test/build/.gitignore             |  2 +
 test/build/build.exeter.in        | 89 +++++++++++++++++++++++++++++++
 test/build/static_checkers.exeter | 17 ++++++
 test/run_avocado                  | 49 +++++++++++++++++
 7 files changed, 207 insertions(+), 1 deletion(-)
 create mode 100644 test/avocado/static_checkers.json
 create mode 100644 test/build/.gitignore
 create mode 100644 test/build/build.exeter.in
 create mode 100755 test/build/static_checkers.exeter
 create mode 100755 test/run_avocado

-- 
2.45.2

[PATCH 1/4] test: run static checkers with Avocado and JSON definitions

[David Gibson]

From: Cleber Rosa <crosa@redhat.com>

This adds a script and configuration to use the Avocado Testing
Framework to run, at this time, the static checkers.

The actual tests are defined using (JSON based) files, that are known
to Avocado as "recipes".  The JSON files are parsed and "resolved"
into tests by Avocado's "runnables-recipe" resolver.  The syntax
allows for any kind of test supported by Avocado to be defined there,
including a mix of different test types.

By the nature of Avocado's default configuration, those will run in
parallel in the host system.  For more complex tests or different use
cases, Avocado could help in future versions by running those in
different environments such as containers.

The entry point ("test/run_avocado") is intended to be an optional
tool at this point, coexisting with the current implementation to run
tests.  It uses Avocado's Job API to create a job with, at this point,
the static checkers suite.

The installation of Avocado itself is left to users, given that the
details on how to install it (virtual environments and specific
tooling) can be a very different and long discussion.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-ID: <20240629121342.3284907-1-crosa@redhat.com>
---
 test/avocado/static_checkers.json | 16 ++++++++++
 test/run_avocado                  | 49 +++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)
 create mode 100644 test/avocado/static_checkers.json
 create mode 100755 test/run_avocado

diff --git a/test/avocado/static_checkers.json b/test/avocado/static_checkers.json
new file mode 100644
index 00000000..5fae43ed
--- /dev/null
+++ b/test/avocado/static_checkers.json
@@ -0,0 +1,16 @@
+[
+    {
+        "kind": "exec-test",
+        "uri": "make",
+        "args": [
+            "clang-tidy"
+        ]
+    },
+    {
+        "kind": "exec-test",
+        "uri": "make",
+        "args": [
+            "cppcheck"
+        ]
+    }
+]
diff --git a/test/run_avocado b/test/run_avocado
new file mode 100755
index 00000000..37db17c3
--- /dev/null
+++ b/test/run_avocado
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+
+def check_avocado_version():
+    minimum_version = 106.0
+
+    def error_out():
+        print(
+            f"Avocado version {minimum_version} or later is required.\n"
+            f"You may install it with: \n"
+            f"   python3 -m pip install avocado-framework",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    try:
+        from avocado import VERSION
+
+        if (float(VERSION)) < minimum_version:
+            error_out()
+    except ImportError:
+        error_out()
+
+
+check_avocado_version()
+from avocado.core.job import Job
+from avocado.core.suite import TestSuite
+
+
+def main():
+    repo_root_path = os.path.abspath(
+        os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+    )
+    config = {
+        "resolver.references": [
+            os.path.join(repo_root_path, "test", "avocado", "static_checkers.json")
+        ],
+        "runner.identifier_format": "{args[0]}",
+    }
+    suite = TestSuite.from_config(config, name="static_checkers")
+    with Job(config, [suite]) as j:
+        return j.run()
+
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+
+def check_avocado_version():
+    minimum_version = 106.0
+
+    def error_out():
+        print(
+            f"Avocado version {minimum_version} or later is required.\n"
+            f"You may install it with: \n"
+            f"   python3 -m pip install avocado-framework",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    try:
+        from avocado import VERSION
+
+        if (float(VERSION)) < minimum_version:
+            error_out()
+    except ImportError:
+        error_out()
+
+
+check_avocado_version()
+from avocado.core.job import Job
+from avocado.core.suite import TestSuite
+
+
+def main():
+    repo_root_path = os.path.abspath(
+        os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+    )
+    config = {
+        "resolver.references": [
+            os.path.join(repo_root_path, "test", "avocado", "static_checkers.json")
+        ],
+        "runner.identifier_format": "{args[0]}",
+    }
+    suite = TestSuite.from_config(config, name="static_checkers")
+    with Job(config, [suite]) as j:
+        return j.run()
+
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
2.45.2

[PATCH 2/4] test: Extend make targets to run Avocado tests

[David Gibson]

Add a new 'avocado' target to the test/ Makefile, which will install
avocado into a Python venv, and run the Avocado based tests with it.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/.gitignore |  1 +
 test/Makefile   | 16 ++++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/test/.gitignore b/test/.gitignore
index 6dd4790b..a79d5b6f 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -10,3 +10,4 @@ QEMU_EFI.fd
 nstool
 guest-key
 guest-key.pub
+/venv/
diff --git a/test/Makefile b/test/Makefile
index 35a3b559..ff8dc63d 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -63,6 +63,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
+SYSTEM_PYTHON = python3
+VENV = venv
+PYTHON = $(VENV)/bin/python3
+PIP = $(VENV)/bin/pip3
+AVOCADO = $(VENV)/bin/avocado
+
 CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
 
 assets: $(ASSETS)
@@ -116,6 +122,15 @@ medium.bin:
 big.bin:
 	dd if=/dev/urandom bs=1M count=10 of=$@
 
+.PHONY: venv
+venv:
+	$(SYSTEM_PYTHON) -m venv $(VENV)
+	$(PIP) install avocado-framework
+
+.PHONY: avocado
+avocado: venv
+	cd .. && test/$(AVOCADO) run test/avocado
+
 check: assets
 	./run
 
@@ -127,6 +142,7 @@ clean:
 	rm -f $(LOCAL_ASSETS)
 	rm -rf test_logs
 	rm -f prepared-*.qcow2 prepared-*.img
+	rm -rf $(VENV)
 
 realclean: clean
 	rm -rf $(DOWNLOAD_ASSETS)
-- 
@@ -63,6 +63,12 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
+SYSTEM_PYTHON = python3
+VENV = venv
+PYTHON = $(VENV)/bin/python3
+PIP = $(VENV)/bin/pip3
+AVOCADO = $(VENV)/bin/avocado
+
 CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99
 
 assets: $(ASSETS)
@@ -116,6 +122,15 @@ medium.bin:
 big.bin:
 	dd if=/dev/urandom bs=1M count=10 of=$@
 
+.PHONY: venv
+venv:
+	$(SYSTEM_PYTHON) -m venv $(VENV)
+	$(PIP) install avocado-framework
+
+.PHONY: avocado
+avocado: venv
+	cd .. && test/$(AVOCADO) run test/avocado
+
 check: assets
 	./run
 
@@ -127,6 +142,7 @@ clean:
 	rm -f $(LOCAL_ASSETS)
 	rm -rf test_logs
 	rm -f prepared-*.qcow2 prepared-*.img
+	rm -rf $(VENV)
 
 realclean: clean
 	rm -rf $(DOWNLOAD_ASSETS)
-- 
2.45.2

[PATCH 3/4] test: Exeter based static tests

[David Gibson]

Introduce some trivial testcases based on the exeter library.  These run
the C static checkers, which is redundant with the included Avocado json
file, but are useful as an example.  We extend the make avocado target to
generate Avocado job files from the exeter tests and include them in the
test run.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/.gitignore                   |  1 +
 test/Makefile                     | 18 ++++++++++++++----
 test/build/.gitignore             |  1 +
 test/build/static_checkers.exeter | 17 +++++++++++++++++
 4 files changed, 33 insertions(+), 4 deletions(-)
 create mode 100644 test/build/.gitignore
 create mode 100755 test/build/static_checkers.exeter

diff --git a/test/.gitignore b/test/.gitignore
index a79d5b6f..bded349b 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -11,3 +11,4 @@ nstool
 guest-key
 guest-key.pub
 /venv/
+/exeter/
diff --git a/test/Makefile b/test/Makefile
index ff8dc63d..afc1334d 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -52,7 +52,7 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \
 	jammy-server-cloudimg-s390x.img
 UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS)
 
-DOWNLOAD_ASSETS = mbuto podman \
+DOWNLOAD_ASSETS = exeter mbuto podman \
 	$(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS)
 TESTDATA_ASSETS = small.bin big.bin medium.bin
 LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
@@ -63,6 +63,10 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
+EXETER_JOBS = build/static_checkers.json
+
+AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
+
 SYSTEM_PYTHON = python3
 VENV = venv
 PYTHON = $(VENV)/bin/python3
@@ -77,6 +81,9 @@ assets: $(ASSETS)
 pull-%: %
 	git -C $* pull
 
+exeter:
+	git clone https://gitlab.com/dgibson/exeter.git
+
 mbuto:
 	git clone git://mbuto.sh/mbuto
 
@@ -127,9 +134,12 @@ venv:
 	$(SYSTEM_PYTHON) -m venv $(VENV)
 	$(PIP) install avocado-framework
 
+%.json: %.exeter pull-exeter
+	cd ..; test/$< --avocado > test/$@
+
 .PHONY: avocado
-avocado: venv
-	cd .. && test/$(AVOCADO) run test/avocado
+avocado: venv $(AVOCADO_JOBS)
+	cd .. && test/$(AVOCADO) run $(AVOCADO_JOBS:%=test/%)
 
 check: assets
 	./run
@@ -142,7 +152,7 @@ clean:
 	rm -f $(LOCAL_ASSETS)
 	rm -rf test_logs
 	rm -f prepared-*.qcow2 prepared-*.img
-	rm -rf $(VENV)
+	rm -rf $(VENV) $(EXETER_JOBS)
 
 realclean: clean
 	rm -rf $(DOWNLOAD_ASSETS)
diff --git a/test/build/.gitignore b/test/build/.gitignore
new file mode 100644
index 00000000..a6c57f5f
--- /dev/null
+++ b/test/build/.gitignore
@@ -0,0 +1 @@
+*.json
diff --git a/test/build/static_checkers.exeter b/test/build/static_checkers.exeter
new file mode 100755
index 00000000..c929615f
--- /dev/null
+++ b/test/build/static_checkers.exeter
@@ -0,0 +1,17 @@
+#! /bin/sh
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+cppcheck () {
+        make cppcheck
+}
+exeter_register cppcheck
+
+clang_tidy () {
+        make clang-tidy
+}
+exeter_register clang_tidy
+
+exeter_main "$@"
+
+
-- 
@@ -0,0 +1,17 @@
+#! /bin/sh
+
+source $(dirname $0)/../exeter/sh/exeter.sh
+
+cppcheck () {
+        make cppcheck
+}
+exeter_register cppcheck
+
+clang_tidy () {
+        make clang-tidy
+}
+exeter_register clang_tidy
+
+exeter_main "$@"
+
+
-- 
2.45.2

[PATCH 4/4] test: Add exeter+Avocado based build tests

[David Gibson]

Add a new test script to run the equivalent of the tests in build/all
using exeter and Avocado.  This new version of the tests is more robust
than the original, since it makes a temporary copy of the source tree so
will not be affected by concurrent manual builds.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/Makefile              | 11 +++--
 test/build/.gitignore      |  1 +
 test/build/build.exeter.in | 89 ++++++++++++++++++++++++++++++++++++++
 3 files changed, 98 insertions(+), 3 deletions(-)
 create mode 100644 test/build/build.exeter.in

diff --git a/test/Makefile b/test/Makefile
index afc1334d..6c8a2820 100644
--- a/test/Makefile
+++ b/test/Makefile
@@ -63,13 +63,13 @@ LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \
 
 ASSETS = $(DOWNLOAD_ASSETS) $(LOCAL_ASSETS)
 
-EXETER_JOBS = build/static_checkers.json
+EXETER_JOBS = build/static_checkers.json build/build.json
 
 AVOCADO_JOBS = $(EXETER_JOBS) avocado/static_checkers.json
 
 SYSTEM_PYTHON = python3
 VENV = venv
-PYTHON = $(VENV)/bin/python3
+VENV_PYTHON = $(shell pwd)/$(VENV)/bin/python3
 PIP = $(VENV)/bin/pip3
 AVOCADO = $(VENV)/bin/avocado
 
@@ -130,13 +130,18 @@ big.bin:
 	dd if=/dev/urandom bs=1M count=10 of=$@
 
 .PHONY: venv
-venv:
+venv: pull-exeter
 	$(SYSTEM_PYTHON) -m venv $(VENV)
 	$(PIP) install avocado-framework
+	$(PIP) install exeter/
 
 %.json: %.exeter pull-exeter
 	cd ..; test/$< --avocado > test/$@
 
+%.exeter: %.exeter.in
+	sed 's!@PYTHON3@!$(VENV_PYTHON)!' < $< > $@
+	chmod +x $@
+
 .PHONY: avocado
 avocado: venv $(AVOCADO_JOBS)
 	cd .. && test/$(AVOCADO) run $(AVOCADO_JOBS:%=test/%)
diff --git a/test/build/.gitignore b/test/build/.gitignore
index a6c57f5f..4ef40dd0 100644
--- a/test/build/.gitignore
+++ b/test/build/.gitignore
@@ -1 +1,2 @@
 *.json
+build.exeter
diff --git a/test/build/build.exeter.in b/test/build/build.exeter.in
new file mode 100644
index 00000000..9b7da359
--- /dev/null
+++ b/test/build/build.exeter.in
@@ -0,0 +1,89 @@
+#! @PYTHON3@
+
+import contextlib
+import os.path
+import shutil
+import subprocess
+import tempfile
+
+import exeter
+
+def host_run(cmd, **kwargs):
+    return subprocess.run(cmd, shell=True, check=True, encoding='UTF-8', **kwargs)
+
+
+def host_out(cmd):
+    return host_run(cmd, capture_output=True).stdout
+
+
+@contextlib.contextmanager
+def clone_source_tree():
+    with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+        # Make a temporary copy of the sources
+        srcfiles = host_out('git ls-files').splitlines()
+        for src in srcfiles:
+            dst = os.path.join(tmpdir, src)
+            os.makedirs(os.path.dirname(dst), exist_ok=True)
+            shutil.copy(src, dst)
+        os.chdir(tmpdir)
+        yield tmpdir
+
+
+def build_target(target, outputs):
+    with clone_source_tree():
+        for o in outputs:
+            assert not os.path.exists(o)
+        host_run(f'make {target} CFLAGS="-Werror"')
+        for o in outputs:
+            assert os.path.exists(o)
+        host_run('make clean')
+        for o in outputs:
+            assert not os.path.exists(o)
+
+
+@exeter.test
+def test_make_passt():
+    build_target('passt', ['passt'])
+
+
+@exeter.test
+def test_make_pasta():
+    build_target('pasta', ['pasta'])
+
+
+@exeter.test
+def test_make_qrap():
+    build_target('qrap', ['qrap'])
+
+
+@exeter.test
+def test_make_all():
+    build_target('all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_make_install_uninstall():
+    with clone_source_tree():
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as prefix:
+            bindir = os.path.join(prefix, 'bin')
+            mandir = os.path.join(prefix, 'share', 'man')
+            exes = ['passt', 'pasta', 'qrap']
+
+            # Install
+            host_run(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+            for t in exes:
+                assert os.path.isfile(os.path.join(bindir, t))
+                host_run(f'man -M {mandir} -W passt')
+
+            # Uninstall
+            host_run(f'make uninstall prefix={prefix}')
+
+            for t in exes:
+                assert not os.path.exists(os.path.join(bindir, t))
+                exeter.assert_raises(subprocess.CalledProcessError,
+                                     host_run, f'man -M {mandir} -W passt')
+
+
+if __name__ == '__main__':
+    exeter.main()
-- 
@@ -0,0 +1,89 @@
+#! @PYTHON3@
+
+import contextlib
+import os.path
+import shutil
+import subprocess
+import tempfile
+
+import exeter
+
+def host_run(cmd, **kwargs):
+    return subprocess.run(cmd, shell=True, check=True, encoding='UTF-8', **kwargs)
+
+
+def host_out(cmd):
+    return host_run(cmd, capture_output=True).stdout
+
+
+@contextlib.contextmanager
+def clone_source_tree():
+    with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as tmpdir:
+        # Make a temporary copy of the sources
+        srcfiles = host_out('git ls-files').splitlines()
+        for src in srcfiles:
+            dst = os.path.join(tmpdir, src)
+            os.makedirs(os.path.dirname(dst), exist_ok=True)
+            shutil.copy(src, dst)
+        os.chdir(tmpdir)
+        yield tmpdir
+
+
+def build_target(target, outputs):
+    with clone_source_tree():
+        for o in outputs:
+            assert not os.path.exists(o)
+        host_run(f'make {target} CFLAGS="-Werror"')
+        for o in outputs:
+            assert os.path.exists(o)
+        host_run('make clean')
+        for o in outputs:
+            assert not os.path.exists(o)
+
+
+@exeter.test
+def test_make_passt():
+    build_target('passt', ['passt'])
+
+
+@exeter.test
+def test_make_pasta():
+    build_target('pasta', ['pasta'])
+
+
+@exeter.test
+def test_make_qrap():
+    build_target('qrap', ['qrap'])
+
+
+@exeter.test
+def test_make_all():
+    build_target('all', ['passt', 'pasta', 'qrap'])
+
+
+@exeter.test
+def test_make_install_uninstall():
+    with clone_source_tree():
+        with tempfile.TemporaryDirectory(ignore_cleanup_errors=False) as prefix:
+            bindir = os.path.join(prefix, 'bin')
+            mandir = os.path.join(prefix, 'share', 'man')
+            exes = ['passt', 'pasta', 'qrap']
+
+            # Install
+            host_run(f'make install CFLAGS="-Werror" prefix={prefix}')
+
+            for t in exes:
+                assert os.path.isfile(os.path.join(bindir, t))
+                host_run(f'man -M {mandir} -W passt')
+
+            # Uninstall
+            host_run(f'make uninstall prefix={prefix}')
+
+            for t in exes:
+                assert not os.path.exists(os.path.join(bindir, t))
+                exeter.assert_raises(subprocess.CalledProcessError,
+                                     host_run, f'man -M {mandir} -W passt')
+
+
+if __name__ == '__main__':
+    exeter.main()
-- 
2.45.2