Re: [PATCH 2/2] test: For missing static checkers, skip rather than failing tests

[Stefano Brivio <sbrivio@redhat.com>]

On Thu,  9 Oct 2025 14:43:58 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> We run a bunch of static checkers as part of our testsuite.  That's useful,
> but it means that if a user doesn't have one of them installed, it fails
> the entire testsuite.  Alter our scripts to skip the test, rather than
> failing outright if the checker tool is not installed.
> 
> This requires exeter v0.4.4 or later.
> 
> Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
> ---
>  test/build/static_checkers.sh | 17 +++++++++++++----
>  test/lib/exeter               | 10 +++++++++-
>  2 files changed, 22 insertions(+), 5 deletions(-)
> 
> diff --git a/test/build/static_checkers.sh b/test/build/static_checkers.sh
> index 228b99ae..caccd183 100755
> --- a/test/build/static_checkers.sh
> +++ b/test/build/static_checkers.sh
> @@ -15,16 +15,25 @@
>  
>  . $(dirname $0)/../exeter/sh/exeter.sh
>  
> -exeter_register cppcheck make -C .. cppcheck
> +do_check() {

Nit: everywhere else in the test suite, we have kerneldoc-style
comments (see lib/test).

> +	checker="$1"
> +	shift
> +	if ! which "${checker}"; then

I think we should hide standard output here, just like we do for the
*tools directives in test_one_line(), because if the check fails, it's
a failure we already handled.

> +		exeter_skip "${checker} not available"
> +	fi
> +	make "${@}" "${checker}"
> +}
> +
> +exeter_register cppcheck do_check cppcheck -C ..
>  exeter_set_description cppcheck "passt sources pass cppcheck"
>  
> -exeter_register clang_tidy make -C .. clang-tidy
> +exeter_register clang_tidy do_check clang-tody -C ..
>  exeter_set_description clang_tidy "passt sources pass clang-tidy"
>  
> -exeter_register flake8 make flake8
> +exeter_register flake8 do_check flake8
>  exeter_set_description flake8 "passt tests in Python pass flake8"
>  
> -exeter_register mypy make mypy
> +exeter_register mypy do_check mypy
>  exeter_set_description mypy "passt tests in Python pass mypy --strict"
>  
>  exeter_main "$@"
> diff --git a/test/lib/exeter b/test/lib/exeter
> index 530c6909..6ed92a16 100644
> --- a/test/lib/exeter
> +++ b/test/lib/exeter
> @@ -49,7 +49,15 @@ exeter() {
>  	for __testid in $($EXETOOL list -- "$@"); do
>  		__desc="$($EXETOOL desc -- "$@" -- "${__testid}")"
>  		status_test_start "${__desc}"
> -		context_run host "$* '${__testid}'" && status_test_ok || status_test_fail
> +		status=0
> +		context_run host "$* '${__testid}'" || status="$?"

It would be nice to keep all variable names enclosed in curly brackets,
for consistency, but also to avoid surprises.

> +		if [ "$status" = 0 ]; then
> +			status_test_ok
> +		elif [ "$status" = 77 ]; then

I couldn't quite find out where 77 comes from. It's not specified in
POSIX.1-2024 2.8.2 "Exit Status for Commands":

  https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.html#tag_19_08_02

and my version of which(1) just returns 1. Is this something from Bash?

> +			status_test_skip
> +		else
> +			status_test_fail
> +		fi
>  	done
>  
>  	cd ..

-- 
Stefano