From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gandalf.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 780DC5A026D for ; Wed, 31 May 2023 03:58:58 +0200 (CEST) Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4QWC9S25NLz4x3g; Wed, 31 May 2023 11:58:52 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=201602; t=1685498332; bh=kWVERHTdnJ9OUdE5J0UOnX3Pw/G3wXE0PTYK7i4purw=; h=From:To:Cc:Subject:Date:From; b=P/yZ1hG3nNiXhkj7r9712imQlVL/6onw8u56reTyVYsgdX5fMfWnatFHQMbGca7oC YmjPWUsLg7s6q9eeViIiqvpSevuekgG5CbxyKviqNLt1E8w0l2f1Qh7c42RYusBj6C k4GqhHIOFWG0XwqdnTThn9s1ZTuEsBce06F83JKM= From: David Gibson To: passt-dev@passt.top, Stefano Brivio Subject: [PATCH v3 00/20] RFCv3: Proof-of-concept conversion of some tests to Avocado framework Date: Wed, 31 May 2023 11:58:29 +1000 Message-Id: <20230531015849.3229596-1-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.40.1 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: Q4EUIZ3AIFC2HCRLJBJWDM4U2CDUHDQV X-Message-ID-Hash: Q4EUIZ3AIFC2HCRLJBJWDM4U2CDUHDQV X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Cleber Rosa , jarichte@redhat.com, David Gibson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: As discussed on calls, I've been investigating Avocado as a possible replacement for our hand-rolled shell test framework. This is a third draft proof-of-concept conversion of a few of the tests to the Avocado framework. Specifically, the static checkers, the build tests and the pasta transfer tests are converted. This third version completely avoids the avocado/unittest built in setUp() and tearDown() functions. Instead it uses Python's with statement and context managers. This is more Pythonic, and avoids some really cryptic multiple inheritance confusion when we want to mix and match tests with different environment setups. Promising things: * I remain happy with the speed improvement (under half the time of the equivalent old tests) * The fact that it's no longer dependent on the host network is great * I'm cautiously optimistic about the helper library making it relatively easy to add a bunch more tests with further coverage Concerning things: * Using context managers instead of setUp/tearDown significantly mitigates it, but the OO/jUnit patterns still make things uglier. Changes since RFCv2: * Remove all setUp() and tearDown() implementations * Remove the weird 'subsetup' stuff we were using along with those Changes since RFCv1: * Sweeping rework of the test library * Started using tags to split between the "real" tests (run with "make avocado") and "meta" tests (run with "make avocado-meta"). The latter are tests for the test infrastructure and helpers themselves, rather than testing anything in passt or pasta David Gibson (20): avocado: Make a duplicate copy of testsuite for comparison purposes avocado: Don't double download assets for test/ and oldtest/ avocado: Move static checkers to avocado avocado/tasst: Helper functions for executing commands in different places avocado/tasst: Type checking helpers avocado: Convert build tests to avocado avocado/tasst: Add helpers for running background commands on sites avocado/tasst: Add helper to get network interface names for a site avocado/tasst: Add helpers to run commands with nstool avocado/tasst: Add ifup and network address helpers to Site avocado/tasst: Helper for creating veth devices between namespaces avocado/tasst: Add helper for getting MTU of a network interface avocado/tasst: Add helper to wait for IP address to appear avocado/tasst: Add helpers for getting a site's routes avocado/tasst: Helpers for test transferring data between sites avocado/tasst: IP address allocation helpers avocado/tasst: Helpers for testing NDP behaviour avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour avocado/tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests Makefile | 19 + avocado/.gitignore | 1 + avocado/build.py | 81 +++ avocado/pasta.py | 195 ++++++ avocado/static_checkers/clang-tidy.sh | 3 + avocado/static_checkers/cppcheck.sh | 3 + avocado/tasst/__init__.py | 22 + avocado/tasst/address.py | 89 +++ avocado/tasst/dhcp.py | 152 +++++ avocado/tasst/dhcpv6.py | 135 +++++ avocado/tasst/meta/__init__.py | 16 + avocado/tasst/meta/static_ifup.py | 59 ++ avocado/tasst/meta/veth.py | 95 +++ avocado/tasst/ndp.py | 137 +++++ avocado/tasst/nstool.py | 199 +++++++ avocado/tasst/scenario/__init__.py | 11 + avocado/tasst/scenario/simple.py | 98 +++ avocado/tasst/site.py | 226 +++++++ avocado/tasst/transfer.py | 224 +++++++ avocado/tasst/typing.py | 23 + oldtest/.gitignore | 11 + oldtest/Makefile | 119 ++++ oldtest/README.md | 137 +++++ {test => oldtest}/build/all | 0 {test => oldtest}/build/clang_tidy | 0 {test => oldtest}/build/cppcheck | 0 oldtest/ci | 1 + oldtest/demo/passt | 245 ++++++++ oldtest/demo/pasta | 274 +++++++++ oldtest/demo/podman | 819 ++++++++++++++++++++++++++ oldtest/distro/debian | 252 ++++++++ oldtest/distro/fedora | 396 +++++++++++++ oldtest/distro/opensuse | 208 +++++++ oldtest/distro/ubuntu | 216 +++++++ oldtest/env/mate-terminal.profile | 42 ++ oldtest/find-arm64-firmware.sh | 13 + oldtest/lib/context | 130 ++++ oldtest/lib/layout | 259 ++++++++ oldtest/lib/layout_ugly | 113 ++++ oldtest/lib/perf_report | 272 +++++++++ oldtest/lib/setup | 385 ++++++++++++ oldtest/lib/setup_ugly | 58 ++ oldtest/lib/term | 750 +++++++++++++++++++++++ oldtest/lib/test | 398 +++++++++++++ oldtest/lib/util | 133 +++++ oldtest/lib/video | 152 +++++ oldtest/memory/passt | 187 ++++++ oldtest/nstool.c | 565 ++++++++++++++++++ oldtest/passt.mbuto | 83 +++ oldtest/passt.mem.mbuto | 44 ++ oldtest/passt/dhcp | 70 +++ oldtest/passt/ndp | 33 ++ oldtest/passt/shutdown | 19 + oldtest/passt/tcp | 76 +++ oldtest/passt/udp | 46 ++ oldtest/passt_in_ns/icmp | 32 + oldtest/passt_in_ns/shutdown | 19 + oldtest/passt_in_ns/tcp | 256 ++++++++ oldtest/passt_in_ns/udp | 138 +++++ {test => oldtest}/pasta/dhcp | 0 {test => oldtest}/pasta/ndp | 0 {test => oldtest}/pasta/tcp | 0 {test => oldtest}/pasta/udp | 0 oldtest/pasta_options/log_to_file | 93 +++ oldtest/perf/passt_tcp | 215 +++++++ oldtest/perf/passt_udp | 165 ++++++ oldtest/perf/pasta_tcp | 300 ++++++++++ oldtest/perf/pasta_udp | 219 +++++++ oldtest/prepare-distro-img.sh | 18 + oldtest/run | 238 ++++++++ oldtest/run_demo | 1 + oldtest/two_guests/basic | 80 +++ oldtest/valgrind.supp | 9 + test/lib/layout | 31 - test/lib/setup | 47 -- test/run | 14 - 76 files changed, 10077 insertions(+), 92 deletions(-) create mode 100644 avocado/.gitignore create mode 100644 avocado/build.py create mode 100644 avocado/pasta.py create mode 100755 avocado/static_checkers/clang-tidy.sh create mode 100755 avocado/static_checkers/cppcheck.sh create mode 100644 avocado/tasst/__init__.py create mode 100644 avocado/tasst/address.py create mode 100644 avocado/tasst/dhcp.py create mode 100644 avocado/tasst/dhcpv6.py create mode 100644 avocado/tasst/meta/__init__.py create mode 100644 avocado/tasst/meta/static_ifup.py create mode 100644 avocado/tasst/meta/veth.py create mode 100644 avocado/tasst/ndp.py create mode 100644 avocado/tasst/nstool.py create mode 100644 avocado/tasst/scenario/__init__.py create mode 100644 avocado/tasst/scenario/simple.py create mode 100644 avocado/tasst/site.py create mode 100644 avocado/tasst/transfer.py create mode 100644 avocado/tasst/typing.py create mode 100644 oldtest/.gitignore create mode 100644 oldtest/Makefile create mode 100644 oldtest/README.md rename {test => oldtest}/build/all (100%) rename {test => oldtest}/build/clang_tidy (100%) rename {test => oldtest}/build/cppcheck (100%) create mode 120000 oldtest/ci create mode 100644 oldtest/demo/passt create mode 100644 oldtest/demo/pasta create mode 100644 oldtest/demo/podman create mode 100644 oldtest/distro/debian create mode 100644 oldtest/distro/fedora create mode 100644 oldtest/distro/opensuse create mode 100644 oldtest/distro/ubuntu create mode 100644 oldtest/env/mate-terminal.profile create mode 100755 oldtest/find-arm64-firmware.sh create mode 100644 oldtest/lib/context create mode 100644 oldtest/lib/layout create mode 100644 oldtest/lib/layout_ugly create mode 100755 oldtest/lib/perf_report create mode 100755 oldtest/lib/setup create mode 100755 oldtest/lib/setup_ugly create mode 100755 oldtest/lib/term create mode 100755 oldtest/lib/test create mode 100755 oldtest/lib/util create mode 100755 oldtest/lib/video create mode 100644 oldtest/memory/passt create mode 100644 oldtest/nstool.c create mode 100755 oldtest/passt.mbuto create mode 100755 oldtest/passt.mem.mbuto create mode 100644 oldtest/passt/dhcp create mode 100644 oldtest/passt/ndp create mode 100644 oldtest/passt/shutdown create mode 100644 oldtest/passt/tcp create mode 100644 oldtest/passt/udp create mode 100644 oldtest/passt_in_ns/icmp create mode 100644 oldtest/passt_in_ns/shutdown create mode 100644 oldtest/passt_in_ns/tcp create mode 100644 oldtest/passt_in_ns/udp rename {test => oldtest}/pasta/dhcp (100%) rename {test => oldtest}/pasta/ndp (100%) rename {test => oldtest}/pasta/tcp (100%) rename {test => oldtest}/pasta/udp (100%) create mode 100644 oldtest/pasta_options/log_to_file create mode 100644 oldtest/perf/passt_tcp create mode 100644 oldtest/perf/passt_udp create mode 100644 oldtest/perf/pasta_tcp create mode 100644 oldtest/perf/pasta_udp create mode 100755 oldtest/prepare-distro-img.sh create mode 100755 oldtest/run create mode 120000 oldtest/run_demo create mode 100644 oldtest/two_guests/basic create mode 100644 oldtest/valgrind.supp -- 2.40.1