From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id E19585A0281 for ; Tue, 16 May 2023 04:01:46 +0200 (CEST) Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4QKzxc709sz4x3x; Tue, 16 May 2023 12:01:40 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=201602; t=1684202500; bh=NL4AniOFosN1BNFPUV09Fmq+68igV7FNOSF7AN8fH6c=; h=From:To:Cc:Subject:Date:From; b=VjHpr0XDZOrGGiCCQuuqy+jSVt5e+BoQtjrqKZt/iIQ8sSpMfkeuXtR9gcfV1e9Oh cX8yfhEjagLZLhPrtoFh4rl641U7+k42iuw3AF7yxpbEN0gXNGgG7F8PHdZvxFo6PF oNT9aPjQ8XqhfyjVYpWV3uPZcmiaVioHHBf12bk8= From: David Gibson To: passt-dev@passt.top, Stefano Brivio Subject: [PATCH v2 00/21] RFCv2: Proof-of-concept conversion of some tests to Avocado framework Date: Tue, 16 May 2023 12:01:14 +1000 Message-Id: <20230516020135.1901256-1-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.40.1 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: KA7BJMOBDGRH4CZRDTKSED6UFRFMIDTG X-Message-ID-Hash: KA7BJMOBDGRH4CZRDTKSED6UFRFMIDTG X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: jarichte@redhat.com, Cleber Rosa , David Gibson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: As discussed on calls, I've been investigating Avocado as a possible replacement for our hand-rolled shell test framework. This is a second draft proof-of-concept conversion of a few of the tests to the Avocado framework. Specifically, the static checkers, the build tests and the pasta transfer tests are converted. In this second version, I've done a lot of rework (maybe too much) on the helper library. It's got a lot more capabilities and is more consistent. The new tests will get failures if you don't also apply the fix I send recently to prevent updating ip6.addr_seen to ::. Promising things: * I remain happy with the speed improvement (under half the time of the equivalent old tests) * The fact that it's no longer dependent on the host network is great * I'm cautiously optimistic about the helper library making it relatively easy to add a bunch more tests with further coverage Concerning things: * I'm coming to hate OO and jUnit patterns more and more, these can make it pretty non-obvious what each test relies on in terms of setup Changes since RFCv1: * Sweeping rework of the test library * Started using tags to split between the "real" tests (run with "make avocado") and "meta" tests (run with "make avocado-meta"). The latter are tests for the test infrastructure and helpers themselves, rather than testing anything in passt or pasta David Gibson (21): test/nstool: Provide useful error if given a path that's too long test/nstool: Fix fd leak in accept() loop avocado: Make a duplicate copy of testsuite for comparison purposes avocado: Don't double download assets for test/ and oldtest/ avocado: Move static checkers to avocado avocado/tasst: Helper functions for executing commands in different places avocado: Convert build tests to avocado avocado/tasst: Add helpers for running background commands on sites avocado/tasst: Add helper to get network interface names for a site avocado/tasst: Add helpers to run commands with nstool avocado/tasst: Add ifup and network address helpers to Site avocado/tasst: Helper for creating veth devices between namespaces avocado/tasst: Add helper for getting MTU of a network interface avocado/tasst: Add helper to wait for IP address to appear avocado/tasst: Add helpers for getting a site's routes avocado/tasst: Helpers for test transferring data between sites avocado/tasst: IP address allocation helpers avocado/tasst: Helpers for testing NDP behaviour avocado/tasst: Helpers for testing DHCP & DHCPv6 behaviour avocado/tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests Makefile | 19 + avocado/.gitignore | 1 + avocado/build.py | 75 +++ avocado/pasta.py | 183 ++++++ avocado/static_checkers/clang-tidy.sh | 3 + avocado/static_checkers/cppcheck.sh | 3 + avocado/tasst/__init__.py | 32 + avocado/tasst/address.py | 89 +++ avocado/tasst/dhcp.py | 133 +++++ avocado/tasst/dhcpv6.py | 123 ++++ avocado/tasst/meta/__init__.py | 16 + avocado/tasst/meta/static_ifup.py | 60 ++ avocado/tasst/meta/veth.py | 95 +++ avocado/tasst/ndp.py | 129 ++++ avocado/tasst/nstool.py | 214 +++++++ avocado/tasst/scenario/__init__.py | 11 + avocado/tasst/scenario/simple.py | 96 +++ avocado/tasst/site.py | 207 +++++++ avocado/tasst/transfer.py | 228 +++++++ oldtest/.gitignore | 11 + oldtest/Makefile | 119 ++++ oldtest/README.md | 137 +++++ {test => oldtest}/build/all | 0 {test => oldtest}/build/clang_tidy | 0 {test => oldtest}/build/cppcheck | 0 oldtest/ci | 1 + oldtest/demo/passt | 245 ++++++++ oldtest/demo/pasta | 274 +++++++++ oldtest/demo/podman | 819 ++++++++++++++++++++++++++ oldtest/distro/debian | 252 ++++++++ oldtest/distro/fedora | 396 +++++++++++++ oldtest/distro/opensuse | 208 +++++++ oldtest/distro/ubuntu | 216 +++++++ oldtest/env/mate-terminal.profile | 42 ++ oldtest/find-arm64-firmware.sh | 13 + oldtest/lib/context | 130 ++++ oldtest/lib/layout | 259 ++++++++ oldtest/lib/layout_ugly | 113 ++++ oldtest/lib/perf_report | 272 +++++++++ oldtest/lib/setup | 385 ++++++++++++ oldtest/lib/setup_ugly | 58 ++ oldtest/lib/term | 750 +++++++++++++++++++++++ oldtest/lib/test | 398 +++++++++++++ oldtest/lib/util | 133 +++++ oldtest/lib/video | 152 +++++ oldtest/memory/passt | 187 ++++++ oldtest/nstool.c | 565 ++++++++++++++++++ oldtest/passt.mbuto | 83 +++ oldtest/passt.mem.mbuto | 44 ++ oldtest/passt/dhcp | 70 +++ oldtest/passt/ndp | 33 ++ oldtest/passt/shutdown | 19 + oldtest/passt/tcp | 76 +++ oldtest/passt/udp | 46 ++ oldtest/passt_in_ns/icmp | 32 + oldtest/passt_in_ns/shutdown | 19 + oldtest/passt_in_ns/tcp | 256 ++++++++ oldtest/passt_in_ns/udp | 138 +++++ {test => oldtest}/pasta/dhcp | 0 {test => oldtest}/pasta/ndp | 0 {test => oldtest}/pasta/tcp | 0 {test => oldtest}/pasta/udp | 0 oldtest/pasta_options/log_to_file | 93 +++ oldtest/perf/passt_tcp | 215 +++++++ oldtest/perf/passt_udp | 165 ++++++ oldtest/perf/pasta_tcp | 300 ++++++++++ oldtest/perf/pasta_udp | 219 +++++++ oldtest/prepare-distro-img.sh | 18 + oldtest/run | 238 ++++++++ oldtest/run_demo | 1 + oldtest/two_guests/basic | 80 +++ oldtest/valgrind.supp | 9 + test/lib/layout | 31 - test/lib/setup | 47 -- test/nstool.c | 24 +- test/run | 14 - 76 files changed, 10022 insertions(+), 100 deletions(-) create mode 100644 avocado/.gitignore create mode 100644 avocado/build.py create mode 100644 avocado/pasta.py create mode 100755 avocado/static_checkers/clang-tidy.sh create mode 100755 avocado/static_checkers/cppcheck.sh create mode 100644 avocado/tasst/__init__.py create mode 100644 avocado/tasst/address.py create mode 100644 avocado/tasst/dhcp.py create mode 100644 avocado/tasst/dhcpv6.py create mode 100644 avocado/tasst/meta/__init__.py create mode 100644 avocado/tasst/meta/static_ifup.py create mode 100644 avocado/tasst/meta/veth.py create mode 100644 avocado/tasst/ndp.py create mode 100644 avocado/tasst/nstool.py create mode 100644 avocado/tasst/scenario/__init__.py create mode 100644 avocado/tasst/scenario/simple.py create mode 100644 avocado/tasst/site.py create mode 100644 avocado/tasst/transfer.py create mode 100644 oldtest/.gitignore create mode 100644 oldtest/Makefile create mode 100644 oldtest/README.md rename {test => oldtest}/build/all (100%) rename {test => oldtest}/build/clang_tidy (100%) rename {test => oldtest}/build/cppcheck (100%) create mode 120000 oldtest/ci create mode 100644 oldtest/demo/passt create mode 100644 oldtest/demo/pasta create mode 100644 oldtest/demo/podman create mode 100644 oldtest/distro/debian create mode 100644 oldtest/distro/fedora create mode 100644 oldtest/distro/opensuse create mode 100644 oldtest/distro/ubuntu create mode 100644 oldtest/env/mate-terminal.profile create mode 100755 oldtest/find-arm64-firmware.sh create mode 100644 oldtest/lib/context create mode 100644 oldtest/lib/layout create mode 100644 oldtest/lib/layout_ugly create mode 100755 oldtest/lib/perf_report create mode 100755 oldtest/lib/setup create mode 100755 oldtest/lib/setup_ugly create mode 100755 oldtest/lib/term create mode 100755 oldtest/lib/test create mode 100755 oldtest/lib/util create mode 100755 oldtest/lib/video create mode 100644 oldtest/memory/passt create mode 100644 oldtest/nstool.c create mode 100755 oldtest/passt.mbuto create mode 100755 oldtest/passt.mem.mbuto create mode 100644 oldtest/passt/dhcp create mode 100644 oldtest/passt/ndp create mode 100644 oldtest/passt/shutdown create mode 100644 oldtest/passt/tcp create mode 100644 oldtest/passt/udp create mode 100644 oldtest/passt_in_ns/icmp create mode 100644 oldtest/passt_in_ns/shutdown create mode 100644 oldtest/passt_in_ns/tcp create mode 100644 oldtest/passt_in_ns/udp rename {test => oldtest}/pasta/dhcp (100%) rename {test => oldtest}/pasta/ndp (100%) rename {test => oldtest}/pasta/tcp (100%) rename {test => oldtest}/pasta/udp (100%) create mode 100644 oldtest/pasta_options/log_to_file create mode 100644 oldtest/perf/passt_tcp create mode 100644 oldtest/perf/passt_udp create mode 100644 oldtest/perf/pasta_tcp create mode 100644 oldtest/perf/pasta_udp create mode 100755 oldtest/prepare-distro-img.sh create mode 100755 oldtest/run create mode 120000 oldtest/run_demo create mode 100644 oldtest/two_guests/basic create mode 100644 oldtest/valgrind.supp -- 2.40.1