public inbox for passt-dev@passt.top
 help / color / mirror / code / Atom feed
From: David Gibson <david@gibson.dropbear.id.au>
To: passt-dev@passt.top, Stefano Brivio <sbrivio@redhat.com>
Cc: Cleber Rosa <crosa@redhat.com>,
	jarichte@redhat.com, David Gibson <david@gibson.dropbear.id.au>
Subject: [PATCH v3 00/20] RFCv3: Proof-of-concept conversion of some tests to Avocado framework
Date: Wed, 31 May 2023 11:58:29 +1000	[thread overview]
Message-ID: <20230531015849.3229596-1-david@gibson.dropbear.id.au> (raw)

As discussed on calls, I've been investigating Avocado as a possible
replacement for our hand-rolled shell test framework.  This is a third
draft proof-of-concept conversion of a few of the tests to the Avocado
framework.  Specifically, the static checkers, the build tests and the
pasta transfer tests are converted.

This third version completely avoids the avocado/unittest built in
setUp() and tearDown() functions.  Instead it uses Python's with
statement and context managers.  This is more Pythonic, and avoids
some really cryptic multiple inheritance confusion when we want to mix
and match tests with different environment setups.

Promising things:
  * I remain happy with the speed improvement (under half the time of
    the equivalent old tests)
  * The fact that it's no longer dependent on the host network is great
  * I'm cautiously optimistic about the helper library making it
    relatively easy to add a bunch more tests with further coverage

Concerning things:
  * Using context managers instead of setUp/tearDown significantly
    mitigates it, but the OO/jUnit patterns still make things uglier.

Changes since RFCv2:
  * Remove all setUp() and tearDown() implementations
    * Remove the weird 'subsetup' stuff we were using along with those

Changes since RFCv1:
  * Sweeping rework of the test library
  * Started using tags to split between the "real" tests (run with
    "make avocado") and "meta" tests (run with "make avocado-meta").
    The latter are tests for the test infrastructure and helpers
    themselves, rather than testing anything in passt or pasta

David Gibson (20):
  avocado: Make a duplicate copy of testsuite for comparison purposes
  avocado: Don't double download assets for test/ and oldtest/
  avocado: Move static checkers to avocado
  avocado/tasst: Helper functions for executing commands in different
    places
  avocado/tasst: Type checking helpers
  avocado: Convert build tests to avocado
  avocado/tasst: Add helpers for running background commands on sites
  avocado/tasst: Add helper to get network interface names for a site
  avocado/tasst: Add helpers to run commands with nstool
  avocado/tasst: Add ifup and network address helpers to Site
  avocado/tasst: Helper for creating veth devices between namespaces
  avocado/tasst: Add helper for getting MTU of a network interface
  avocado/tasst: Add helper to wait for IP address to appear
  avocado/tasst: Add helpers for getting a site's routes
  avocado/tasst: Helpers for test transferring data between sites
  avocado/tasst: IP address allocation helpers
  avocado/tasst: Helpers for testing NDP behaviour
  avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour
  avocado/tasst: Helpers to construct a simple network environment for
    tests
  avocado: Convert basic pasta tests

 Makefile                              |  19 +
 avocado/.gitignore                    |   1 +
 avocado/build.py                      |  81 +++
 avocado/pasta.py                      | 195 ++++++
 avocado/static_checkers/clang-tidy.sh |   3 +
 avocado/static_checkers/cppcheck.sh   |   3 +
 avocado/tasst/__init__.py             |  22 +
 avocado/tasst/address.py              |  89 +++
 avocado/tasst/dhcp.py                 | 152 +++++
 avocado/tasst/dhcpv6.py               | 135 +++++
 avocado/tasst/meta/__init__.py        |  16 +
 avocado/tasst/meta/static_ifup.py     |  59 ++
 avocado/tasst/meta/veth.py            |  95 +++
 avocado/tasst/ndp.py                  | 137 +++++
 avocado/tasst/nstool.py               | 199 +++++++
 avocado/tasst/scenario/__init__.py    |  11 +
 avocado/tasst/scenario/simple.py      |  98 +++
 avocado/tasst/site.py                 | 226 +++++++
 avocado/tasst/transfer.py             | 224 +++++++
 avocado/tasst/typing.py               |  23 +
 oldtest/.gitignore                    |  11 +
 oldtest/Makefile                      | 119 ++++
 oldtest/README.md                     | 137 +++++
 {test => oldtest}/build/all           |   0
 {test => oldtest}/build/clang_tidy    |   0
 {test => oldtest}/build/cppcheck      |   0
 oldtest/ci                            |   1 +
 oldtest/demo/passt                    | 245 ++++++++
 oldtest/demo/pasta                    | 274 +++++++++
 oldtest/demo/podman                   | 819 ++++++++++++++++++++++++++
 oldtest/distro/debian                 | 252 ++++++++
 oldtest/distro/fedora                 | 396 +++++++++++++
 oldtest/distro/opensuse               | 208 +++++++
 oldtest/distro/ubuntu                 | 216 +++++++
 oldtest/env/mate-terminal.profile     |  42 ++
 oldtest/find-arm64-firmware.sh        |  13 +
 oldtest/lib/context                   | 130 ++++
 oldtest/lib/layout                    | 259 ++++++++
 oldtest/lib/layout_ugly               | 113 ++++
 oldtest/lib/perf_report               | 272 +++++++++
 oldtest/lib/setup                     | 385 ++++++++++++
 oldtest/lib/setup_ugly                |  58 ++
 oldtest/lib/term                      | 750 +++++++++++++++++++++++
 oldtest/lib/test                      | 398 +++++++++++++
 oldtest/lib/util                      | 133 +++++
 oldtest/lib/video                     | 152 +++++
 oldtest/memory/passt                  | 187 ++++++
 oldtest/nstool.c                      | 565 ++++++++++++++++++
 oldtest/passt.mbuto                   |  83 +++
 oldtest/passt.mem.mbuto               |  44 ++
 oldtest/passt/dhcp                    |  70 +++
 oldtest/passt/ndp                     |  33 ++
 oldtest/passt/shutdown                |  19 +
 oldtest/passt/tcp                     |  76 +++
 oldtest/passt/udp                     |  46 ++
 oldtest/passt_in_ns/icmp              |  32 +
 oldtest/passt_in_ns/shutdown          |  19 +
 oldtest/passt_in_ns/tcp               | 256 ++++++++
 oldtest/passt_in_ns/udp               | 138 +++++
 {test => oldtest}/pasta/dhcp          |   0
 {test => oldtest}/pasta/ndp           |   0
 {test => oldtest}/pasta/tcp           |   0
 {test => oldtest}/pasta/udp           |   0
 oldtest/pasta_options/log_to_file     |  93 +++
 oldtest/perf/passt_tcp                | 215 +++++++
 oldtest/perf/passt_udp                | 165 ++++++
 oldtest/perf/pasta_tcp                | 300 ++++++++++
 oldtest/perf/pasta_udp                | 219 +++++++
 oldtest/prepare-distro-img.sh         |  18 +
 oldtest/run                           | 238 ++++++++
 oldtest/run_demo                      |   1 +
 oldtest/two_guests/basic              |  80 +++
 oldtest/valgrind.supp                 |   9 +
 test/lib/layout                       |  31 -
 test/lib/setup                        |  47 --
 test/run                              |  14 -
 76 files changed, 10077 insertions(+), 92 deletions(-)
 create mode 100644 avocado/.gitignore
 create mode 100644 avocado/build.py
 create mode 100644 avocado/pasta.py
 create mode 100755 avocado/static_checkers/clang-tidy.sh
 create mode 100755 avocado/static_checkers/cppcheck.sh
 create mode 100644 avocado/tasst/__init__.py
 create mode 100644 avocado/tasst/address.py
 create mode 100644 avocado/tasst/dhcp.py
 create mode 100644 avocado/tasst/dhcpv6.py
 create mode 100644 avocado/tasst/meta/__init__.py
 create mode 100644 avocado/tasst/meta/static_ifup.py
 create mode 100644 avocado/tasst/meta/veth.py
 create mode 100644 avocado/tasst/ndp.py
 create mode 100644 avocado/tasst/nstool.py
 create mode 100644 avocado/tasst/scenario/__init__.py
 create mode 100644 avocado/tasst/scenario/simple.py
 create mode 100644 avocado/tasst/site.py
 create mode 100644 avocado/tasst/transfer.py
 create mode 100644 avocado/tasst/typing.py
 create mode 100644 oldtest/.gitignore
 create mode 100644 oldtest/Makefile
 create mode 100644 oldtest/README.md
 rename {test => oldtest}/build/all (100%)
 rename {test => oldtest}/build/clang_tidy (100%)
 rename {test => oldtest}/build/cppcheck (100%)
 create mode 120000 oldtest/ci
 create mode 100644 oldtest/demo/passt
 create mode 100644 oldtest/demo/pasta
 create mode 100644 oldtest/demo/podman
 create mode 100644 oldtest/distro/debian
 create mode 100644 oldtest/distro/fedora
 create mode 100644 oldtest/distro/opensuse
 create mode 100644 oldtest/distro/ubuntu
 create mode 100644 oldtest/env/mate-terminal.profile
 create mode 100755 oldtest/find-arm64-firmware.sh
 create mode 100644 oldtest/lib/context
 create mode 100644 oldtest/lib/layout
 create mode 100644 oldtest/lib/layout_ugly
 create mode 100755 oldtest/lib/perf_report
 create mode 100755 oldtest/lib/setup
 create mode 100755 oldtest/lib/setup_ugly
 create mode 100755 oldtest/lib/term
 create mode 100755 oldtest/lib/test
 create mode 100755 oldtest/lib/util
 create mode 100755 oldtest/lib/video
 create mode 100644 oldtest/memory/passt
 create mode 100644 oldtest/nstool.c
 create mode 100755 oldtest/passt.mbuto
 create mode 100755 oldtest/passt.mem.mbuto
 create mode 100644 oldtest/passt/dhcp
 create mode 100644 oldtest/passt/ndp
 create mode 100644 oldtest/passt/shutdown
 create mode 100644 oldtest/passt/tcp
 create mode 100644 oldtest/passt/udp
 create mode 100644 oldtest/passt_in_ns/icmp
 create mode 100644 oldtest/passt_in_ns/shutdown
 create mode 100644 oldtest/passt_in_ns/tcp
 create mode 100644 oldtest/passt_in_ns/udp
 rename {test => oldtest}/pasta/dhcp (100%)
 rename {test => oldtest}/pasta/ndp (100%)
 rename {test => oldtest}/pasta/tcp (100%)
 rename {test => oldtest}/pasta/udp (100%)
 create mode 100644 oldtest/pasta_options/log_to_file
 create mode 100644 oldtest/perf/passt_tcp
 create mode 100644 oldtest/perf/passt_udp
 create mode 100644 oldtest/perf/pasta_tcp
 create mode 100644 oldtest/perf/pasta_udp
 create mode 100755 oldtest/prepare-distro-img.sh
 create mode 100755 oldtest/run
 create mode 120000 oldtest/run_demo
 create mode 100644 oldtest/two_guests/basic
 create mode 100644 oldtest/valgrind.supp

-- 
2.40.1


             reply	other threads:[~2023-05-31  1:58 UTC|newest]

Thread overview: 21+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-05-31  1:58 David Gibson [this message]
2023-05-30 23:59 ` [PATCH v3 01/20] avocado: Make a duplicate copy of testsuite for comparison purposes David Gibson
2023-05-31  1:58 ` [PATCH v3 02/20] avocado: Don't double download assets for test/ and oldtest/ David Gibson
2023-05-31  1:58 ` [PATCH v3 03/20] avocado: Move static checkers to avocado David Gibson
2023-05-31  1:58 ` [PATCH v3 04/20] avocado/tasst: Helper functions for executing commands in different places David Gibson
2023-05-31  1:58 ` [PATCH v3 05/20] avocado/tasst: Type checking helpers David Gibson
2023-05-31  1:58 ` [PATCH v3 06/20] avocado: Convert build tests to avocado David Gibson
2023-05-31  1:58 ` [PATCH v3 07/20] avocado/tasst: Add helpers for running background commands on sites David Gibson
2023-05-31  1:58 ` [PATCH v3 08/20] avocado/tasst: Add helper to get network interface names for a site David Gibson
2023-05-31  1:58 ` [PATCH v3 09/20] avocado/tasst: Add helpers to run commands with nstool David Gibson
2023-05-31  1:58 ` [PATCH v3 10/20] avocado/tasst: Add ifup and network address helpers to Site David Gibson
2023-05-31  1:58 ` [PATCH v3 11/20] avocado/tasst: Helper for creating veth devices between namespaces David Gibson
2023-05-31  1:58 ` [PATCH v3 12/20] avocado/tasst: Add helper for getting MTU of a network interface David Gibson
2023-05-31  1:58 ` [PATCH v3 13/20] avocado/tasst: Add helper to wait for IP address to appear David Gibson
2023-05-31  1:58 ` [PATCH v3 14/20] avocado/tasst: Add helpers for getting a site's routes David Gibson
2023-05-31  1:58 ` [PATCH v3 15/20] avocado/tasst: Helpers for test transferring data between sites David Gibson
2023-05-31  1:58 ` [PATCH v3 16/20] avocado/tasst: IP address allocation helpers David Gibson
2023-05-31  1:58 ` [PATCH v3 17/20] avocado/tasst: Helpers for testing NDP behaviour David Gibson
2023-05-31  1:58 ` [PATCH v3 18/20] avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour David Gibson
2023-05-31  1:58 ` [PATCH v3 19/20] avocado/tasst: Helpers to construct a simple network environment for tests David Gibson
2023-05-31  1:58 ` [PATCH v3 20/20] avocado: Convert basic pasta tests David Gibson

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230531015849.3229596-1-david@gibson.dropbear.id.au \
    --to=david@gibson.dropbear.id.au \
    --cc=crosa@redhat.com \
    --cc=jarichte@redhat.com \
    --cc=passt-dev@passt.top \
    --cc=sbrivio@redhat.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
Code repositories for project(s) associated with this public inbox

	https://passt.top/passt

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).