From: David Gibson <david@gibson.dropbear.id.au>
To: passt-dev@passt.top, Stefano Brivio <sbrivio@redhat.com>
Cc: jarichte@redhat.com, Cleber Rosa <crosa@redhat.com>,
David Gibson <david@gibson.dropbear.id.au>
Subject: [PATCH v2 00/21] RFCv2: Proof-of-concept conversion of some tests to Avocado framework
Date: Tue, 16 May 2023 12:01:14 +1000 [thread overview]
Message-ID: <20230516020135.1901256-1-david@gibson.dropbear.id.au> (raw)
As discussed on calls, I've been investigating Avocado as a possible
replacement for our hand-rolled shell test framework. This is a
second draft proof-of-concept conversion of a few of the tests to the
Avocado framework. Specifically, the static checkers, the build tests
and the pasta transfer tests are converted.
In this second version, I've done a lot of rework (maybe too much) on
the helper library. It's got a lot more capabilities and is more
consistent.
The new tests will get failures if you don't also apply the fix I send
recently to prevent updating ip6.addr_seen to ::.
Promising things:
* I remain happy with the speed improvement (under half the time of
the equivalent old tests)
* The fact that it's no longer dependent on the host network is great
* I'm cautiously optimistic about the helper library making it
relatively easy to add a bunch more tests with further coverage
Concerning things:
* I'm coming to hate OO and jUnit patterns more and more, these can
make it pretty non-obvious what each test relies on in terms of setup
Changes since RFCv1:
* Sweeping rework of the test library
* Started using tags to split between the "real" tests (run with
"make avocado") and "meta" tests (run with "make avocado-meta").
The latter are tests for the test infrastructure and helpers
themselves, rather than testing anything in passt or pasta
David Gibson (21):
test/nstool: Provide useful error if given a path that's too long
test/nstool: Fix fd leak in accept() loop
avocado: Make a duplicate copy of testsuite for comparison purposes
avocado: Don't double download assets for test/ and oldtest/
avocado: Move static checkers to avocado
avocado/tasst: Helper functions for executing commands in different
places
avocado: Convert build tests to avocado
avocado/tasst: Add helpers for running background commands on sites
avocado/tasst: Add helper to get network interface names for a site
avocado/tasst: Add helpers to run commands with nstool
avocado/tasst: Add ifup and network address helpers to Site
avocado/tasst: Helper for creating veth devices between namespaces
avocado/tasst: Add helper for getting MTU of a network interface
avocado/tasst: Add helper to wait for IP address to appear
avocado/tasst: Add helpers for getting a site's routes
avocado/tasst: Helpers for test transferring data between sites
avocado/tasst: IP address allocation helpers
avocado/tasst: Helpers for testing NDP behaviour
avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour
avocado/tasst: Helpers to construct a simple network environment for
tests
avocado: Convert basic pasta tests
Makefile | 19 +
avocado/.gitignore | 1 +
avocado/build.py | 75 +++
avocado/pasta.py | 183 ++++++
avocado/static_checkers/clang-tidy.sh | 3 +
avocado/static_checkers/cppcheck.sh | 3 +
avocado/tasst/__init__.py | 32 +
avocado/tasst/address.py | 89 +++
avocado/tasst/dhcp.py | 133 +++++
avocado/tasst/dhcpv6.py | 123 ++++
avocado/tasst/meta/__init__.py | 16 +
avocado/tasst/meta/static_ifup.py | 60 ++
avocado/tasst/meta/veth.py | 95 +++
avocado/tasst/ndp.py | 129 ++++
avocado/tasst/nstool.py | 214 +++++++
avocado/tasst/scenario/__init__.py | 11 +
avocado/tasst/scenario/simple.py | 96 +++
avocado/tasst/site.py | 207 +++++++
avocado/tasst/transfer.py | 228 +++++++
oldtest/.gitignore | 11 +
oldtest/Makefile | 119 ++++
oldtest/README.md | 137 +++++
{test => oldtest}/build/all | 0
{test => oldtest}/build/clang_tidy | 0
{test => oldtest}/build/cppcheck | 0
oldtest/ci | 1 +
oldtest/demo/passt | 245 ++++++++
oldtest/demo/pasta | 274 +++++++++
oldtest/demo/podman | 819 ++++++++++++++++++++++++++
oldtest/distro/debian | 252 ++++++++
oldtest/distro/fedora | 396 +++++++++++++
oldtest/distro/opensuse | 208 +++++++
oldtest/distro/ubuntu | 216 +++++++
oldtest/env/mate-terminal.profile | 42 ++
oldtest/find-arm64-firmware.sh | 13 +
oldtest/lib/context | 130 ++++
oldtest/lib/layout | 259 ++++++++
oldtest/lib/layout_ugly | 113 ++++
oldtest/lib/perf_report | 272 +++++++++
oldtest/lib/setup | 385 ++++++++++++
oldtest/lib/setup_ugly | 58 ++
oldtest/lib/term | 750 +++++++++++++++++++++++
oldtest/lib/test | 398 +++++++++++++
oldtest/lib/util | 133 +++++
oldtest/lib/video | 152 +++++
oldtest/memory/passt | 187 ++++++
oldtest/nstool.c | 565 ++++++++++++++++++
oldtest/passt.mbuto | 83 +++
oldtest/passt.mem.mbuto | 44 ++
oldtest/passt/dhcp | 70 +++
oldtest/passt/ndp | 33 ++
oldtest/passt/shutdown | 19 +
oldtest/passt/tcp | 76 +++
oldtest/passt/udp | 46 ++
oldtest/passt_in_ns/icmp | 32 +
oldtest/passt_in_ns/shutdown | 19 +
oldtest/passt_in_ns/tcp | 256 ++++++++
oldtest/passt_in_ns/udp | 138 +++++
{test => oldtest}/pasta/dhcp | 0
{test => oldtest}/pasta/ndp | 0
{test => oldtest}/pasta/tcp | 0
{test => oldtest}/pasta/udp | 0
oldtest/pasta_options/log_to_file | 93 +++
oldtest/perf/passt_tcp | 215 +++++++
oldtest/perf/passt_udp | 165 ++++++
oldtest/perf/pasta_tcp | 300 ++++++++++
oldtest/perf/pasta_udp | 219 +++++++
oldtest/prepare-distro-img.sh | 18 +
oldtest/run | 238 ++++++++
oldtest/run_demo | 1 +
oldtest/two_guests/basic | 80 +++
oldtest/valgrind.supp | 9 +
test/lib/layout | 31 -
test/lib/setup | 47 --
test/nstool.c | 24 +-
test/run | 14 -
76 files changed, 10022 insertions(+), 100 deletions(-)
create mode 100644 avocado/.gitignore
create mode 100644 avocado/build.py
create mode 100644 avocado/pasta.py
create mode 100755 avocado/static_checkers/clang-tidy.sh
create mode 100755 avocado/static_checkers/cppcheck.sh
create mode 100644 avocado/tasst/__init__.py
create mode 100644 avocado/tasst/address.py
create mode 100644 avocado/tasst/dhcp.py
create mode 100644 avocado/tasst/dhcpv6.py
create mode 100644 avocado/tasst/meta/__init__.py
create mode 100644 avocado/tasst/meta/static_ifup.py
create mode 100644 avocado/tasst/meta/veth.py
create mode 100644 avocado/tasst/ndp.py
create mode 100644 avocado/tasst/nstool.py
create mode 100644 avocado/tasst/scenario/__init__.py
create mode 100644 avocado/tasst/scenario/simple.py
create mode 100644 avocado/tasst/site.py
create mode 100644 avocado/tasst/transfer.py
create mode 100644 oldtest/.gitignore
create mode 100644 oldtest/Makefile
create mode 100644 oldtest/README.md
rename {test => oldtest}/build/all (100%)
rename {test => oldtest}/build/clang_tidy (100%)
rename {test => oldtest}/build/cppcheck (100%)
create mode 120000 oldtest/ci
create mode 100644 oldtest/demo/passt
create mode 100644 oldtest/demo/pasta
create mode 100644 oldtest/demo/podman
create mode 100644 oldtest/distro/debian
create mode 100644 oldtest/distro/fedora
create mode 100644 oldtest/distro/opensuse
create mode 100644 oldtest/distro/ubuntu
create mode 100644 oldtest/env/mate-terminal.profile
create mode 100755 oldtest/find-arm64-firmware.sh
create mode 100644 oldtest/lib/context
create mode 100644 oldtest/lib/layout
create mode 100644 oldtest/lib/layout_ugly
create mode 100755 oldtest/lib/perf_report
create mode 100755 oldtest/lib/setup
create mode 100755 oldtest/lib/setup_ugly
create mode 100755 oldtest/lib/term
create mode 100755 oldtest/lib/test
create mode 100755 oldtest/lib/util
create mode 100755 oldtest/lib/video
create mode 100644 oldtest/memory/passt
create mode 100644 oldtest/nstool.c
create mode 100755 oldtest/passt.mbuto
create mode 100755 oldtest/passt.mem.mbuto
create mode 100644 oldtest/passt/dhcp
create mode 100644 oldtest/passt/ndp
create mode 100644 oldtest/passt/shutdown
create mode 100644 oldtest/passt/tcp
create mode 100644 oldtest/passt/udp
create mode 100644 oldtest/passt_in_ns/icmp
create mode 100644 oldtest/passt_in_ns/shutdown
create mode 100644 oldtest/passt_in_ns/tcp
create mode 100644 oldtest/passt_in_ns/udp
rename {test => oldtest}/pasta/dhcp (100%)
rename {test => oldtest}/pasta/ndp (100%)
rename {test => oldtest}/pasta/tcp (100%)
rename {test => oldtest}/pasta/udp (100%)
create mode 100644 oldtest/pasta_options/log_to_file
create mode 100644 oldtest/perf/passt_tcp
create mode 100644 oldtest/perf/passt_udp
create mode 100644 oldtest/perf/pasta_tcp
create mode 100644 oldtest/perf/pasta_udp
create mode 100755 oldtest/prepare-distro-img.sh
create mode 100755 oldtest/run
create mode 120000 oldtest/run_demo
create mode 100644 oldtest/two_guests/basic
create mode 100644 oldtest/valgrind.supp
--
2.40.1
next reply other threads:[~2023-05-16 2:01 UTC|newest]
Thread overview: 22+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-16 2:01 David Gibson [this message]
2023-05-16 2:01 ` [PATCH v2 01/21] test/nstool: Provide useful error if given a path that's too long David Gibson
2023-05-16 2:01 ` [PATCH v2 02/21] test/nstool: Fix fd leak in accept() loop David Gibson
2023-05-16 2:01 ` [PATCH v2 03/21] avocado: Make a duplicate copy of testsuite for comparison purposes David Gibson
2023-05-16 2:01 ` [PATCH v2 04/21] avocado: Don't double download assets for test/ and oldtest/ David Gibson
2023-05-16 2:01 ` [PATCH v2 05/21] avocado: Move static checkers to avocado David Gibson
2023-05-16 2:01 ` [PATCH v2 06/21] avocado/tasst: Helper functions for executing commands in different places David Gibson
2023-05-16 2:01 ` [PATCH v2 07/21] avocado: Convert build tests to avocado David Gibson
2023-05-16 2:01 ` [PATCH v2 08/21] avocado/tasst: Add helpers for running background commands on sites David Gibson
2023-05-16 2:01 ` [PATCH v2 09/21] avocado/tasst: Add helper to get network interface names for a site David Gibson
2023-05-16 2:01 ` [PATCH v2 10/21] avocado/tasst: Add helpers to run commands with nstool David Gibson
2023-05-16 2:01 ` [PATCH v2 11/21] avocado/tasst: Add ifup and network address helpers to Site David Gibson
2023-05-16 2:01 ` [PATCH v2 12/21] avocado/tasst: Helper for creating veth devices between namespaces David Gibson
2023-05-16 2:01 ` [PATCH v2 13/21] avocado/tasst: Add helper for getting MTU of a network interface David Gibson
2023-05-16 2:01 ` [PATCH v2 14/21] avocado/tasst: Add helper to wait for IP address to appear David Gibson
2023-05-16 2:01 ` [PATCH v2 15/21] avocado/tasst: Add helpers for getting a site's routes David Gibson
2023-05-16 2:01 ` [PATCH v2 16/21] avocado/tasst: Helpers for test transferring data between sites David Gibson
2023-05-16 2:01 ` [PATCH v2 17/21] avocado/tasst: IP address allocation helpers David Gibson
2023-05-16 2:01 ` [PATCH v2 18/21] avocado/tasst: Helpers for testing NDP behaviour David Gibson
2023-05-16 2:01 ` [PATCH v2 19/21] avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour David Gibson
2023-05-16 2:01 ` [PATCH v2 20/21] avocado/tasst: Helpers to construct a simple network environment for tests David Gibson
2023-05-16 2:01 ` [PATCH v2 21/21] avocado: Convert basic pasta tests David Gibson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230516020135.1901256-1-david@gibson.dropbear.id.au \
--to=david@gibson.dropbear.id.au \
--cc=crosa@redhat.com \
--cc=jarichte@redhat.com \
--cc=passt-dev@passt.top \
--cc=sbrivio@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
Code repositories for project(s) associated with this public inbox
https://passt.top/passt
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for IMAP folder(s).