[PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[Cleber Rosa]

This adds a script and configuration to use the Avocado Testing
Framework to run, at this time, the static checkers.

The actual tests are defined using (JSON based) files, that are known
to Avocado as "recipes".  The JSON files are parsed and "resolved"
into tests by Avocado's "runnables-recipe" resolver.  The syntax
allows for any kind of test supported by Avocado to be defined there,
including a mix of different test types.

By the nature of Avocado's default configuration, those will run in
parallel in the host system.  For more complex tests or different use
cases, Avocado could help in future versions by running those in
different environments such as containers.

The entry point ("test/run_avocado") is intended to be an optional
tool at this point, coexisting with the current implementation to run
tests.  It uses Avocado's Job API to create a job with, at this point,
the static checkers suite.

The installation of Avocado itself is left to users, given that the
details on how to install it (virtual environments and specific
tooling) can be a very different and long discussion.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
---
 test/avocado/static_checkers.json | 16 ++++++++++
 test/run_avocado                  | 49 +++++++++++++++++++++++++++++++
 2 files changed, 65 insertions(+)
 create mode 100644 test/avocado/static_checkers.json
 create mode 100755 test/run_avocado

diff --git a/test/avocado/static_checkers.json b/test/avocado/static_checkers.json
new file mode 100644
index 0000000..5fae43e
--- /dev/null
+++ b/test/avocado/static_checkers.json
@@ -0,0 +1,16 @@
+[
+    {
+        "kind": "exec-test",
+        "uri": "make",
+        "args": [
+            "clang-tidy"
+        ]
+    },
+    {
+        "kind": "exec-test",
+        "uri": "make",
+        "args": [
+            "cppcheck"
+        ]
+    }
+]
diff --git a/test/run_avocado b/test/run_avocado
new file mode 100755
index 0000000..37db17c
--- /dev/null
+++ b/test/run_avocado
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+
+def check_avocado_version():
+    minimum_version = 106.0
+
+    def error_out():
+        print(
+            f"Avocado version {minimum_version} or later is required.\n"
+            f"You may install it with: \n"
+            f"   python3 -m pip install avocado-framework",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    try:
+        from avocado import VERSION
+
+        if (float(VERSION)) < minimum_version:
+            error_out()
+    except ImportError:
+        error_out()
+
+
+check_avocado_version()
+from avocado.core.job import Job
+from avocado.core.suite import TestSuite
+
+
+def main():
+    repo_root_path = os.path.abspath(
+        os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+    )
+    config = {
+        "resolver.references": [
+            os.path.join(repo_root_path, "test", "avocado", "static_checkers.json")
+        ],
+        "runner.identifier_format": "{args[0]}",
+    }
+    suite = TestSuite.from_config(config, name="static_checkers")
+    with Job(config, [suite]) as j:
+        return j.run()
+
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+import os
+import sys
+
+
+def check_avocado_version():
+    minimum_version = 106.0
+
+    def error_out():
+        print(
+            f"Avocado version {minimum_version} or later is required.\n"
+            f"You may install it with: \n"
+            f"   python3 -m pip install avocado-framework",
+            file=sys.stderr,
+        )
+        sys.exit(1)
+
+    try:
+        from avocado import VERSION
+
+        if (float(VERSION)) < minimum_version:
+            error_out()
+    except ImportError:
+        error_out()
+
+
+check_avocado_version()
+from avocado.core.job import Job
+from avocado.core.suite import TestSuite
+
+
+def main():
+    repo_root_path = os.path.abspath(
+        os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
+    )
+    config = {
+        "resolver.references": [
+            os.path.join(repo_root_path, "test", "avocado", "static_checkers.json")
+        ],
+        "runner.identifier_format": "{args[0]}",
+    }
+    suite = TestSuite.from_config(config, name="static_checkers")
+    with Job(config, [suite]) as j:
+        return j.run()
+
+
+if __name__ == "__main__":
+    sys.exit(main())
-- 
2.45.2

Re: [PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5442 bytes --]

On Sat, Jun 29, 2024 at 08:13:42AM -0400, Cleber Rosa wrote:

Hi, sorry I've taken a while to reply to this.  I've been sick again,
which has been no fun, plus I've been kind of preoccupied with getting
the flow table ready.

> This adds a script and configuration to use the Avocado Testing
> Framework to run, at this time, the static checkers.
> 
> The actual tests are defined using (JSON based) files, that are known
> to Avocado as "recipes".  The JSON files are parsed and "resolved"
> into tests by Avocado's "runnables-recipe" resolver.  The syntax
> allows for any kind of test supported by Avocado to be defined there,
> including a mix of different test types.

I'm not entirely clear.  Is this relying on the various extensions
we've discussed and you've been working on?  Or is this using the
older job API, but will become neater with the recent/coming
extensions?

> By the nature of Avocado's default configuration, those will run in
> parallel in the host system.  For more complex tests or different use
> cases, Avocado could help in future versions by running those in
> different environments such as containers.

Probably not super useful for the static checkers, but could be handy
for other things.  Then again... doing that could allow us to "lock"
to a specific static checker version which could have some advantages.

> The entry point ("test/run_avocado") is intended to be an optional
> tool at this point, coexisting with the current implementation to run
> tests.  It uses Avocado's Job API to create a job with, at this point,
> the static checkers suite.

So, you can run the tests either with run_avocado, or running Avocado
directly?

> The installation of Avocado itself is left to users, given that the
> details on how to install it (virtual environments and specific
> tooling) can be a very different and long discussion.

Right... if the required Avocado version was already packaged in
latest Fedora and/or Debian, I'd have no qualms about that.  It looks
like you require a very new one though, which makes me a little
nervous about requiring that complex installation.  I guess if it's
pip installable that helps.  Though if I recall on Debian even that
requires some venv setup.  Eventually I think we'll want a README for
Avocado installation, or better yet make targets that will install it
automatically in a local venv.

> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> ---
>  test/avocado/static_checkers.json | 16 ++++++++++
>  test/run_avocado                  | 49 +++++++++++++++++++++++++++++++
>  2 files changed, 65 insertions(+)
>  create mode 100644 test/avocado/static_checkers.json
>  create mode 100755 test/run_avocado
> 
> diff --git a/test/avocado/static_checkers.json b/test/avocado/static_checkers.json
> new file mode 100644
> index 0000000..5fae43e
> --- /dev/null
> +++ b/test/avocado/static_checkers.json
> @@ -0,0 +1,16 @@
> +[
> +    {
> +        "kind": "exec-test",
> +        "uri": "make",
> +        "args": [
> +            "clang-tidy"
> +        ]
> +    },
> +    {
> +        "kind": "exec-test",
> +        "uri": "make",
> +        "args": [
> +            "cppcheck"
> +        ]
> +    }
> +]

Looks pretty reasonable to me, at least for these simple cases.  It
would be nice to have comments, but IIUC that's one of the things
notably missing from json :/,

> diff --git a/test/run_avocado b/test/run_avocado
> new file mode 100755
> index 0000000..37db17c
> --- /dev/null
> +++ b/test/run_avocado
> @@ -0,0 +1,49 @@
> +#!/usr/bin/env python3
> +
> +import os
> +import sys
> +
> +
> +def check_avocado_version():
> +    minimum_version = 106.0
> +
> +    def error_out():
> +        print(
> +            f"Avocado version {minimum_version} or later is required.\n"
> +            f"You may install it with: \n"
> +            f"   python3 -m pip install avocado-framework",
> +            file=sys.stderr,
> +        )
> +        sys.exit(1)
> +
> +    try:
> +        from avocado import VERSION
> +
> +        if (float(VERSION)) < minimum_version:
> +            error_out()
> +    except ImportError:
> +        error_out()
> +
> +
> +check_avocado_version()
> +from avocado.core.job import Job
> +from avocado.core.suite import TestSuite
> +
> +
> +def main():
> +    repo_root_path = os.path.abspath(
> +        os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
> +    )
> +    config = {
> +        "resolver.references": [
> +            os.path.join(repo_root_path, "test", "avocado", "static_checkers.json")
> +        ],
> +        "runner.identifier_format": "{args[0]}",
> +    }
> +    suite = TestSuite.from_config(config, name="static_checkers")
> +    with Job(config, [suite]) as j:
> +        return j.run()

Do you expect to need run_avocado indefinitely, or is it an interim
thing?  

I find most of main() a bit cryptic, but it's small enough that that
might be fine - as long as it's not expected to grow significantly.
IIUC, it should usually be possible to add tests without changing
anything here, yes?

> +
> +
> +if __name__ == "__main__":
> +    sys.exit(main())

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[Stefano Brivio]

Not a review, just two remarks:

On Fri, 5 Jul 2024 21:35:28 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> > [...]
> 
> So, you can run the tests either with run_avocado, or running Avocado
> directly?
> 
> > The installation of Avocado itself is left to users, given that the
> > details on how to install it (virtual environments and specific
> > tooling) can be a very different and long discussion.  
> 
> Right... if the required Avocado version was already packaged in
> latest Fedora and/or Debian, I'd have no qualms about that.  It looks
> like you require a very new one though, which makes me a little
> nervous about requiring that complex installation.  I guess if it's
> pip installable that helps.  Though if I recall on Debian even that
> requires some venv setup.  Eventually I think we'll want a README for
> Avocado installation, or better yet make targets that will install it
> automatically in a local venv.

I plan to try this out on Debian and report back -- I didn't have time
yet. Indeed, it would help to have all the dependencies packaged because
it would be really beneficial if we could run the upstream tests also as
automated package testing, e.g. for Debian it's autopkgtest:
  https://wiki.debian.org/ContinuousIntegration/autopkgtest

The current test suite has all the dependencies packaged on (at least)
Debian and Fedora, minus neper (https://github.com/google/neper), but
tests will be skipped if those tools are not available.

> > [...]
> >
> > +++ b/test/avocado/static_checkers.json
> > @@ -0,0 +1,16 @@
> > +[
> > +    {
> > +        "kind": "exec-test",
> > +        "uri": "make",
> > +        "args": [
> > +            "clang-tidy"
> > +        ]
> > +    },
> > +    {
> > +        "kind": "exec-test",
> > +        "uri": "make",
> > +        "args": [
> > +            "cppcheck"
> > +        ]
> > +    }
> > +]  
> 
> Looks pretty reasonable to me, at least for these simple cases.  It
> would be nice to have comments, but IIUC that's one of the things
> notably missing from json :/,

You could use HJSON (https://hjson.github.io/), or a subset of it, for
example with just comments. We're using it like that in seitan, because
that's what Parson supports, example:
  https://seitan.rocks/seitan/tree/demo/routes.hjson

-- 
Stefano

Re: [PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2909 bytes --]

On Mon, Jul 08, 2024 at 12:00:32PM +0200, Stefano Brivio wrote:
> Not a review, just two remarks:
> 
> On Fri, 5 Jul 2024 21:35:28 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> 
> > > [...]
> > 
> > So, you can run the tests either with run_avocado, or running Avocado
> > directly?
> > 
> > > The installation of Avocado itself is left to users, given that the
> > > details on how to install it (virtual environments and specific
> > > tooling) can be a very different and long discussion.  
> > 
> > Right... if the required Avocado version was already packaged in
> > latest Fedora and/or Debian, I'd have no qualms about that.  It looks
> > like you require a very new one though, which makes me a little
> > nervous about requiring that complex installation.  I guess if it's
> > pip installable that helps.  Though if I recall on Debian even that
> > requires some venv setup.  Eventually I think we'll want a README for
> > Avocado installation, or better yet make targets that will install it
> > automatically in a local venv.
> 
> I plan to try this out on Debian and report back -- I didn't have time
> yet. Indeed, it would help to have all the dependencies packaged because
> it would be really beneficial if we could run the upstream tests also as
> automated package testing, e.g. for Debian it's autopkgtest:
>   https://wiki.debian.org/ContinuousIntegration/autopkgtest
> 
> The current test suite has all the dependencies packaged on (at least)
> Debian and Fedora, minus neper (https://github.com/google/neper), but
> tests will be skipped if those tools are not available.
> 
> > > [...]
> > >
> > > +++ b/test/avocado/static_checkers.json
> > > @@ -0,0 +1,16 @@
> > > +[
> > > +    {
> > > +        "kind": "exec-test",
> > > +        "uri": "make",
> > > +        "args": [
> > > +            "clang-tidy"
> > > +        ]
> > > +    },
> > > +    {
> > > +        "kind": "exec-test",
> > > +        "uri": "make",
> > > +        "args": [
> > > +            "cppcheck"
> > > +        ]
> > > +    }
> > > +]  
> > 
> > Looks pretty reasonable to me, at least for these simple cases.  It
> > would be nice to have comments, but IIUC that's one of the things
> > notably missing from json :/,
> 
> You could use HJSON (https://hjson.github.io/), or a subset of it, for
> example with just comments. We're using it like that in seitan, because
> that's what Parson supports, example:
>   https://seitan.rocks/seitan/tree/demo/routes.hjson

Yeah, there are bunch of json variants and extensions that allow for
comments.  Which is kind of the problem: there are a bunch of them,
making the choice non-obvious.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH/RFC] test: run static checkers with Avocado and JSON definitions

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2146 bytes --]


I've now had a chance to play around with the patch, and I have some
further thoughts on that basis.

On Fri, Jul 05, 2024 at 09:35:28PM +1000, David Gibson wrote:

> On Sat, Jun 29, 2024 at 08:13:42AM -0400, Cleber Rosa wrote:
> Hi, sorry I've taken a while to reply to this.  I've been sick again,
> which has been no fun, plus I've been kind of preoccupied with getting
> the flow table ready.
> 
> > This adds a script and configuration to use the Avocado Testing
> > Framework to run, at this time, the static checkers.
> > 
> > The actual tests are defined using (JSON based) files, that are known
> > to Avocado as "recipes".  The JSON files are parsed and "resolved"
> > into tests by Avocado's "runnables-recipe" resolver.  The syntax
> > allows for any kind of test supported by Avocado to be defined there,
> > including a mix of different test types.
> 
> I'm not entirely clear.  Is this relying on the various extensions
> we've discussed and you've been working on?  Or is this using the
> older job API, but will become neater with the recent/coming
> extensions?

I see from experimenting that I can run the tests either with the
'run_avocado' script, or just by running
	avocado run test/avocado/static_checkers.json
or
	avocado run test/avocado

That's great! But.. without the config override in 'run_avocado' the
identifiers for each case aren't right: they just get called 'make' as
the name of the executable that's being run.  We really need a way to
set the identifier from the json file so that it's consistent.

The other thing I noticed is that it only works if you invoke either
avocado or run_avocado from the base of the passt tree.  I can see
why, given the implementation, but that's pretty surprising behaviour.

So, I guess we also need a way to specify the initial pwd in the json
- I guess it would need to be relative to where the json file itself
is to make sense.

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]