[PATCH v2 0/3] Fixes to exeter test integration

[PATCH v2 0/3] Fixes to exeter test integration

[David Gibson]

In advance of the tunbridge work, here are some further small fixes to
the test logic, specifically how we interface with exeter tests.

v2:
 * Consistently use ${} style for shell variables
 * Assorted minor revisions based on Stefano's feedback

David Gibson (3):
  test: Use ${} consistently in lib/exeter
  test: Add some missing quoting in exeter runner
  test: For missing static checkers, skip rather than failing tests

 test/build/static_checkers.sh | 24 ++++++++++++++++++------
 test/lib/exeter               | 28 ++++++++++++++++++----------
 2 files changed, 36 insertions(+), 16 deletions(-)

-- 
2.51.0

[PATCH v2 1/3] test: Use ${} consistently in lib/exeter

[David Gibson]

Most of the test shell scripts use {} around variable names even when it's
not required, for consistency.  lib/exeter didn't do so consistently,
however.  Correct that.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/lib/exeter | 20 ++++++++++----------
 1 file changed, 10 insertions(+), 10 deletions(-)

diff --git a/test/lib/exeter b/test/lib/exeter
index 3b19beaa..0b361599 100644
--- a/test/lib/exeter
+++ b/test/lib/exeter
@@ -13,18 +13,18 @@
 # Copyright Red Hat
 # Author: David Gibson <david@gibson.dropbear.id.au>
 
-EXETOOL="$BASEPATH/exeter/exetool/exetool"
+EXETOOL="${BASEPATH}/exeter/exetool/exetool"
 
 # is_exeter() - Determine if a test file is an exeter program
 # $@:	Command line to invoke test program
 is_exeter() {
-	$EXETOOL probe -- "$@"
+	${EXETOOL} probe -- "${@}"
 }
 
 # exeter() - Run each test in an exeter program, logging each test separately
 # $@:	Command line to invoke exeter test program
 exeter() {
-	STATESETUP="${STATEBASE}/$1"
+	STATESETUP="${STATEBASE}/${1}"
 	mkdir -p "${STATESETUP}"
 
 	context_setup_host host
@@ -32,9 +32,9 @@ exeter() {
 
 	cd test
 
-	__ntests=$($EXETOOL list -- "$@" | wc -l)
-	if [ $? != 0 ]; then
-		info "Failed to get exeter manifest for $@"
+	__ntests=$(${EXETOOL} list -- "${@}" | wc -l)
+	if [ ${?} != 0 ]; then
+		info "Failed to get exeter manifest for ${@}"
 		pause_continue \
 			"Press any key to pause test session"		\
 			"Resuming in "					\
@@ -43,13 +43,13 @@ exeter() {
 		return
 	fi
 
-	status_file_start "$* (exeter)" ${__ntests}
+	status_file_start "${*} (exeter)" ${__ntests}
 	[ ${CI} -eq 1 ] && video_link "${1}"
 
-	for __testid in $($EXETOOL list -- "$@"); do
-		__desc="$($EXETOOL desc -- "$@" -- ${__testid})"
+	for __testid in $(${EXETOOL} list -- "${@}"); do
+		__desc="$(${EXETOOL} desc -- "${@}" -- ${__testid})"
 		status_test_start "${__desc}"
-		context_run host "$@" "${__testid}" && status_test_ok || status_test_fail
+		context_run host "${@}" "${__testid}" && status_test_ok || status_test_fail
 	done
 
 	cd ..
-- 
2.51.0

[PATCH v2 2/3] test: Add some missing quoting in exeter runner

[David Gibson]

exeter() quoted ${__testid}, but in some places we use it there's an
extra level of shell, which needs another layer of quoting.  This breaks
if testids include ';', which is quite common in exeter tests created as
a composition/pipeline of two functions.  Add the required extra quoting.

While we're there, improve consistency with other scripts by always using
the optional {} around variable names.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/lib/exeter | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/lib/exeter b/test/lib/exeter
index 0b361599..7ea084a4 100644
--- a/test/lib/exeter
+++ b/test/lib/exeter
@@ -47,9 +47,9 @@ exeter() {
 	[ ${CI} -eq 1 ] && video_link "${1}"
 
 	for __testid in $(${EXETOOL} list -- "${@}"); do
-		__desc="$(${EXETOOL} desc -- "${@}" -- ${__testid})"
+		__desc="$(${EXETOOL} desc -- "${@}" -- "${__testid}")"
 		status_test_start "${__desc}"
-		context_run host "${@}" "${__testid}" && status_test_ok || status_test_fail
+		context_run host "${*} '${__testid}'" && status_test_ok || status_test_fail
 	done
 
 	cd ..
-- 
2.51.0

[PATCH v2 3/3] test: For missing static checkers, skip rather than failing tests

[David Gibson]

We run a bunch of static checkers as part of our testsuite.  That's useful,
but it means that if a user doesn't have one of them installed, it fails
the entire testsuite.  Alter our scripts to skip the test, rather than
failing outright if the checker tool is not installed.

This requires exeter v0.4.4 or later.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
---
 test/build/static_checkers.sh | 24 ++++++++++++++++++------
 test/lib/exeter               | 10 +++++++++-
 2 files changed, 27 insertions(+), 7 deletions(-)

diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
index 228b99ae..96679fb2 100755
--- a/test/build/static_checkers.sh
+++ b/test/build/static_checkers.sh
@@ -13,18 +13,30 @@
 # Copyright Red Hat
 # Author: David Gibson <david@gibson.dropbear.id.au>
 
-. $(dirname $0)/../exeter/sh/exeter.sh
+. $(dirname ${0})/../exeter/sh/exeter.sh
 
-exeter_register cppcheck make -C .. cppcheck
+# do_check() - Run static checker as a test if the binary is available
+# $1:	Static checker (uased as both executable name and make target)
+# $@:	Any additional arguments required to make
+do_check() {
+	checker="${1}"
+	shift
+	if ! which "${checker}" >/dev/null 2>/dev/null; then
+		exeter_skip "${checker} not available"
+	fi
+	make "${@}" "${checker}"
+}
+
+exeter_register cppcheck do_check cppcheck -C ..
 exeter_set_description cppcheck "passt sources pass cppcheck"
 
-exeter_register clang_tidy make -C .. clang-tidy
+exeter_register clang_tidy do_check clang-tidy -C ..
 exeter_set_description clang_tidy "passt sources pass clang-tidy"
 
-exeter_register flake8 make flake8
+exeter_register flake8 do_check flake8
 exeter_set_description flake8 "passt tests in Python pass flake8"
 
-exeter_register mypy make mypy
+exeter_register mypy do_check mypy
 exeter_set_description mypy "passt tests in Python pass mypy --strict"
 
-exeter_main "$@"
+exeter_main "${@}"
diff --git a/test/lib/exeter b/test/lib/exeter
index 7ea084a4..ccdb19c2 100644
--- a/test/lib/exeter
+++ b/test/lib/exeter
@@ -49,7 +49,15 @@ exeter() {
 	for __testid in $(${EXETOOL} list -- "${@}"); do
 		__desc="$(${EXETOOL} desc -- "${@}" -- "${__testid}")"
 		status_test_start "${__desc}"
-		context_run host "${*} '${__testid}'" && status_test_ok || status_test_fail
+		status=0
+		context_run host "${*} '${__testid}'" || status="${?}"
+		if [ "${status}" = 0 ]; then
+			status_test_ok
+		elif [ "${status}" = 77 ]; then
+			status_test_skip
+		else
+			status_test_fail
+		fi
 	done
 
 	cd ..
-- 
2.51.0