From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202510 header.b=yVULsm39; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 922D55A026F for ; Fri, 10 Oct 2025 05:40:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202510; t=1760067653; bh=KcHTowSsBYWTTZU5RYXn8vjdnaxHE7Ms3hbiDbc7LGs=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=yVULsm396J35jDpv1iYsRMYbnlJOoFitTN9JM6mqWb8b0euv27cj4hoA1owUywohn dg8hyshqaNHsqE44hWtCxUz5HKFXrJOCGH2fOIM+9YuMXAtudveNpObBXWhf80RiJX rx98oRximVEe2HHIZ/B2FDACOhhJhHAu41jUXvm5qy8Nwb8ff+dOijt6aITawLphLe 8EgfNhc87ikEXUye5eWC3jz+sH+5er5funoOd7NA2a5LMpAQwWW/x07cWiiphKF44Z gW/f3dNPG3fxX0uqe24C9233fU98wOOYoQ74G1RGSOuw1qtV+IqNXpjZZN/dWQYll0 kZJNR0KBjbG1Q== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4cjXZs51Lkz4wCw; Fri, 10 Oct 2025 14:40:53 +1100 (AEDT) Date: Fri, 10 Oct 2025 14:11:16 +1100 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH 2/2] test: For missing static checkers, skip rather than failing tests Message-ID: References: <20251009034358.1256908-1-david@gibson.dropbear.id.au> <20251009034358.1256908-3-david@gibson.dropbear.id.au> <20251009212902.1b75082c@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="UAtNGcjCJuKMzZtG" Content-Disposition: inline In-Reply-To: <20251009212902.1b75082c@elisabeth> Message-ID-Hash: REFJYLBIAR7MZQ3SUJQW3NFY73JCXVQG X-Message-ID-Hash: REFJYLBIAR7MZQ3SUJQW3NFY73JCXVQG X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --UAtNGcjCJuKMzZtG Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Oct 09, 2025 at 09:29:02PM +0200, Stefano Brivio wrote: > On Thu, 9 Oct 2025 14:43:58 +1100 > David Gibson wrote: >=20 > > We run a bunch of static checkers as part of our testsuite. That's use= ful, > > but it means that if a user doesn't have one of them installed, it fails > > the entire testsuite. Alter our scripts to skip the test, rather than > > failing outright if the checker tool is not installed. > >=20 > > This requires exeter v0.4.4 or later. > >=20 > > Signed-off-by: David Gibson > > --- > > test/build/static_checkers.sh | 17 +++++++++++++---- > > test/lib/exeter | 10 +++++++++- > > 2 files changed, 22 insertions(+), 5 deletions(-) > >=20 > > diff --git a/test/build/static_checkers.sh b/test/build/static_checkers= =2Esh > > index 228b99ae..caccd183 100755 > > --- a/test/build/static_checkers.sh > > +++ b/test/build/static_checkers.sh > > @@ -15,16 +15,25 @@ > > =20 > > . $(dirname $0)/../exeter/sh/exeter.sh > > =20 > > -exeter_register cppcheck make -C .. cppcheck > > +do_check() { >=20 > Nit: everywhere else in the test suite, we have kerneldoc-style > comments (see lib/test). Good point, added. > > + checker=3D"$1" > > + shift > > + if ! which "${checker}"; then >=20 > I think we should hide standard output here, just like we do for the > *tools directives in test_one_line(), because if the check fails, it's > a failure we already handled. Good idea. I hid both stdout - it just generates noise on success - and stderr - the error is already handled on failure. >=20 > > + exeter_skip "${checker} not available" > > + fi > > + make "${@}" "${checker}" > > +} > > + > > +exeter_register cppcheck do_check cppcheck -C .. > > exeter_set_description cppcheck "passt sources pass cppcheck" > > =20 > > -exeter_register clang_tidy make -C .. clang-tidy > > +exeter_register clang_tidy do_check clang-tody -C .. > > exeter_set_description clang_tidy "passt sources pass clang-tidy" > > =20 > > -exeter_register flake8 make flake8 > > +exeter_register flake8 do_check flake8 > > exeter_set_description flake8 "passt tests in Python pass flake8" > > =20 > > -exeter_register mypy make mypy > > +exeter_register mypy do_check mypy > > exeter_set_description mypy "passt tests in Python pass mypy --strict" > > =20 > > exeter_main "$@" > > diff --git a/test/lib/exeter b/test/lib/exeter > > index 530c6909..6ed92a16 100644 > > --- a/test/lib/exeter > > +++ b/test/lib/exeter > > @@ -49,7 +49,15 @@ exeter() { > > for __testid in $($EXETOOL list -- "$@"); do > > __desc=3D"$($EXETOOL desc -- "$@" -- "${__testid}")" > > status_test_start "${__desc}" > > - context_run host "$* '${__testid}'" && status_test_ok || status_test= _fail > > + status=3D0 > > + context_run host "$* '${__testid}'" || status=3D"$?" >=20 > It would be nice to keep all variable names enclosed in curly brackets, > for consistency, but also to avoid surprises. Done. > > + if [ "$status" =3D 0 ]; then > > + status_test_ok > > + elif [ "$status" =3D 77 ]; then >=20 > I couldn't quite find out where 77 comes from. It's not specified in > POSIX.1-2024 2.8.2 "Exit Status for Commands": >=20 > https://pubs.opengroup.org/onlinepubs/9799919799/utilities/V3_chap02.ht= ml#tag_19_08_02 >=20 > and my version of which(1) just returns 1. Is this something from Bash? No, it's exeter_skip that generates code 77. That choice is made on what seems to be a loose convention amongst some existing test tools. Meson uses it, for example: https://mesonbuild.com/Unit-tests.html#skipped-tests-and-hard-errors I believe they got it from some GNU tools, like automake: https://www.gnu.org/software/automake/manual/automake.html#Scripts_002dbase= d-Testsuites-1 Based on those examples, I wrote it into the exeter protocol. https://gitlab.com/dgibson/exeter/-/blob/main/PROTOCOL.md?ref_type=3Dheads#= exit-codes BATS and Avocado recognize it by default. For BATS, I handled that by putting explicit handling in the .bats files generated by exetool. I didn't originally handle Avocado, but your question prompted me to figure it out, and I just committed a change which handles it (by configuring Avocado via the job file to recognize this code to mean skip). In any case, it's not perfect, but seemed as good a convention as any to pick. --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --UAtNGcjCJuKMzZtG Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmjoeVMACgkQzQJF27ox 2Gft2A//b3iLkHS+qYTLCkVYCdoZWgwFVDoXWW/IS7Y9ZtzqErnW1A57OGLFzwWH HTj3ci/Pffl1Slos0Jlm7WfmQYvH/51ju3SA4v14suo7CQX/W87LFf4EYi1zWBfa PB9Ze8SxnCnYk3OtZx42uEKzdihUcWHC4FSAwbA/zI/j+BlgGT9kqGZ5qws7quPV qqCrDeKnvNka4d6UxmX64fSe6pwPHodw3k4nFKzALedddjpsZF7MRSom2xgURhTb RB7LMRBehJcygbiiLoJphz79dAdvSTqU81C9v9dkmVhxLm3sz29/f2FbBz4rpFr0 nuSsJFDbqDjpN1qCCojV0TkL/FqRYFbxTJ47hcNOGY2Oge8AjY/J/kRht7NDaTq0 hKr+ZcpcBbkK9oEswH7/uVN1G8mtBpRP7RsXGhhBq4Y2wvfHpD28oun4P3TQ7rFB Dd20EirU75Je8l099ycgR5mw/RTDZTuviQ1hxYbVtIMt/0GVIOfjmuCOE+bnEPjG Qwi28wdaMDmkqUo5mJDI8ZHeaO9xSdbDSngOWZnBgRo0ugL7NJCRXZQ/yhMKM7l+ nEIMCiZkyQ4eqBcOj8hb44hfWyfXsWxVSGa7Ms1iqKmnDTPwatKnLQdyAeH01VYj BUT365IHr0dPm+cfVfz39gcTLGzJigAClpcUiJz3nabLWdsvplo= =XRIo -----END PGP SIGNATURE----- --UAtNGcjCJuKMzZtG--