From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 546845A004E for ; Fri, 05 Jul 2024 13:46:45 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202312; t=1720180000; bh=Omo2Uy3CdytCEZHupywn77nom0q/J1WAMtN22FyI3v8=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=ptRrFbWUkt6JkVXH1lo5cF0BKMPllpaXSs3C5iav7hWhLj2QL5y01UG8aJrnsxVQE bH9oPV+eTq0Drx+DSU7c/CLGzddtdcHl40cfTOnwgtDJI+mdD1JurcKo2LbO8Tg0T0 zgnGDuGq9e0u3Ih3F+QUfBeH6W+BVTAkGYS3skOjrHKxWvkKfEavGrxxuqCzpbtgWq VhvLA3qG3IAKCg2CQNBXAy6OP6tAOH3gyLK7sYhjbB7fCHw1Z26AuFOku3SL+/Mkhy hMhbtpcea7V2RTSHSM5jcQgK84AWrIzaUWJgMJsZeJmXw0t18aM5tq6JRlE/DZBkMQ dGmlSvNsLZUug== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4WFsDc36rzz4wbp; Fri, 5 Jul 2024 21:46:40 +1000 (AEST) Date: Fri, 5 Jul 2024 21:35:28 +1000 From: David Gibson To: Cleber Rosa Subject: Re: [PATCH/RFC] test: run static checkers with Avocado and JSON definitions Message-ID: References: <20240629121342.3284907-1-crosa@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Jt/iWiVP8+WJxVJa" Content-Disposition: inline In-Reply-To: <20240629121342.3284907-1-crosa@redhat.com> Message-ID-Hash: EUFNARDJF35UCJRQZH2XU6G3JLPWNGSS X-Message-ID-Hash: EUFNARDJF35UCJRQZH2XU6G3JLPWNGSS X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, David Gibson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --Jt/iWiVP8+WJxVJa Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 29, 2024 at 08:13:42AM -0400, Cleber Rosa wrote: Hi, sorry I've taken a while to reply to this. I've been sick again, which has been no fun, plus I've been kind of preoccupied with getting the flow table ready. > This adds a script and configuration to use the Avocado Testing > Framework to run, at this time, the static checkers. >=20 > The actual tests are defined using (JSON based) files, that are known > to Avocado as "recipes". The JSON files are parsed and "resolved" > into tests by Avocado's "runnables-recipe" resolver. The syntax > allows for any kind of test supported by Avocado to be defined there, > including a mix of different test types. I'm not entirely clear. Is this relying on the various extensions we've discussed and you've been working on? Or is this using the older job API, but will become neater with the recent/coming extensions? > By the nature of Avocado's default configuration, those will run in > parallel in the host system. For more complex tests or different use > cases, Avocado could help in future versions by running those in > different environments such as containers. Probably not super useful for the static checkers, but could be handy for other things. Then again... doing that could allow us to "lock" to a specific static checker version which could have some advantages. > The entry point ("test/run_avocado") is intended to be an optional > tool at this point, coexisting with the current implementation to run > tests. It uses Avocado's Job API to create a job with, at this point, > the static checkers suite. So, you can run the tests either with run_avocado, or running Avocado directly? > The installation of Avocado itself is left to users, given that the > details on how to install it (virtual environments and specific > tooling) can be a very different and long discussion. Right... if the required Avocado version was already packaged in latest Fedora and/or Debian, I'd have no qualms about that. It looks like you require a very new one though, which makes me a little nervous about requiring that complex installation. I guess if it's pip installable that helps. Though if I recall on Debian even that requires some venv setup. Eventually I think we'll want a README for Avocado installation, or better yet make targets that will install it automatically in a local venv. > Signed-off-by: Cleber Rosa > --- > test/avocado/static_checkers.json | 16 ++++++++++ > test/run_avocado | 49 +++++++++++++++++++++++++++++++ > 2 files changed, 65 insertions(+) > create mode 100644 test/avocado/static_checkers.json > create mode 100755 test/run_avocado >=20 > diff --git a/test/avocado/static_checkers.json b/test/avocado/static_chec= kers.json > new file mode 100644 > index 0000000..5fae43e > --- /dev/null > +++ b/test/avocado/static_checkers.json > @@ -0,0 +1,16 @@ > +[ > + { > + "kind": "exec-test", > + "uri": "make", > + "args": [ > + "clang-tidy" > + ] > + }, > + { > + "kind": "exec-test", > + "uri": "make", > + "args": [ > + "cppcheck" > + ] > + } > +] Looks pretty reasonable to me, at least for these simple cases. It would be nice to have comments, but IIUC that's one of the things notably missing from json :/, > diff --git a/test/run_avocado b/test/run_avocado > new file mode 100755 > index 0000000..37db17c > --- /dev/null > +++ b/test/run_avocado > @@ -0,0 +1,49 @@ > +#!/usr/bin/env python3 > + > +import os > +import sys > + > + > +def check_avocado_version(): > + minimum_version =3D 106.0 > + > + def error_out(): > + print( > + f"Avocado version {minimum_version} or later is required.\n" > + f"You may install it with: \n" > + f" python3 -m pip install avocado-framework", > + file=3Dsys.stderr, > + ) > + sys.exit(1) > + > + try: > + from avocado import VERSION > + > + if (float(VERSION)) < minimum_version: > + error_out() > + except ImportError: > + error_out() > + > + > +check_avocado_version() > +from avocado.core.job import Job > +from avocado.core.suite import TestSuite > + > + > +def main(): > + repo_root_path =3D os.path.abspath( > + os.path.dirname(os.path.dirname(os.path.dirname(__file__))) > + ) > + config =3D { > + "resolver.references": [ > + os.path.join(repo_root_path, "test", "avocado", "static_chec= kers.json") > + ], > + "runner.identifier_format": "{args[0]}", > + } > + suite =3D TestSuite.from_config(config, name=3D"static_checkers") > + with Job(config, [suite]) as j: > + return j.run() Do you expect to need run_avocado indefinitely, or is it an interim thing? =20 I find most of main() a bit cryptic, but it's small enough that that might be fine - as long as it's not expected to grow significantly. IIUC, it should usually be possible to add tests without changing anything here, yes? > + > + > +if __name__ =3D=3D "__main__": > + sys.exit(main()) --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --Jt/iWiVP8+WJxVJa Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmaH2nQACgkQzQJF27ox 2GeexA/8Do74be4eHQxdLGY3QQtdLIXzHGwLObnwy+//Np3F/8Sj3/vdQTSk51f6 m3y5eKX+TAZXtOGPAp03B7V24p0+tD8LNPQF66I7j99NEaZMMbbQXGtKWFWCBjsr 6kQzD0QiB78yFld+x5vDMpEYpM/kJqnNzWs6m4UhDusU+xorNA6U21VSLrFCnZDY qha9TvnFjUY23FFnYtnyK7VTcFjugQQbxsZhlXbVqDdPaNsLLHGB3/glrKrv/AZM AFCIJTafMnnMCZOzJCpI6qGNmUBW8jP71lDmDZFAqq46wcFH4bhp5kRFvcUEH5Fm RTGxY5oIVtlcLlbMAiWXwoN+nmUjPk6MB2wOEKO/IunbqvLmEUsxKp6iGdVP/mVT 4lHdrqS0eqpYjLWNWte2WSo9hB7ONfSCnrJSez/BoVGr+Nb9j3HX/dF9WpA/vRDM 4UyksiM5QAU5eMd04jlZXLlKOQd6sZVeRlWxT0b3dQOUuBnJn807vPIZ47K3xMCl aHr62TcwllupoowUyDdJ/VH26p8yw7vbZ/LThyaGsjXxGLoGLP3j5kUfE5UlsOFN vTmFyRMLeWQXBqC1iUpp24tBtc3vzZLpxlkyIdHiRHyYpxOopWBBALb24/sN6xg6 hAOcpdDf3qvbUbuo5nF919ggMyxHU6Sh/7iOjND4DgdiTjn4dZc= =soC/ -----END PGP SIGNATURE----- --Jt/iWiVP8+WJxVJa--