From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from gandalf.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id B9ADA5A0271 for ; Tue, 27 Jun 2023 04:54:40 +0200 (CEST) Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4Qqq7J5WKRz4wZy; Tue, 27 Jun 2023 12:54:36 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=201602; t=1687834476; bh=MlQ73GU3yDXyAmWsj868Q8wEME3+PZ8cPdCJC3Bd350=; h=From:To:Cc:Subject:Date:From; b=UfJPofq/i4GD2rsbY5IcBqK5/IUXfbIfSsbZvRyuQZsPxOyXpqGZ9BWmkWwy8ILAr 3t9+6DidLBD0I323ak67QCboIPEY/1hsnbJ2UPBbEaev8L7efy9kYXY/+wz0j0g978 DxnxCUUoZG1x/f482QDICzlnpjDvcp0cA84ll6LQ= From: David Gibson To: passt-dev@passt.top, Stefano Brivio Subject: [PATCH 00/27] RFC: Start converting passt & pasta tests to Avocado using special plugin Date: Tue, 27 Jun 2023 12:54:01 +1000 Message-ID: <20230627025429.2209702-1-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.41.0 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: 66YAHBG5F3REOG57SEM4QTHTSCWLUAOL X-Message-ID-Hash: 66YAHBG5F3REOG57SEM4QTHTSCWLUAOL X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: crosa@redhat.com, jarichte@redhat.com, David Gibson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: I've sent previous series converting some of the passt & pasta testsuite to use Avocado. However the jUnit-like structure of avocado tests wasn't a great fit for our needs (plus Stefano and I just dislike the style). Here's another attempt, where instead we use a new "avocado-classless" plugin to avoid jUnit-ism. This is definitely a work-in-progress. Some early observations: * It is noticeably less verbose than jUnit style * It does avoid some of the confusing stuff that arises from jUnit style * The present draft has some different pretty confusing stuff, involving nested and stacked function decorators, and some other non-entirely-obvious use of higher order functions * I hope I can improve this at least a bit, but it will need some thought * I was pleasantly surprised at how simple it was to write the plugin itself David Gibson (27): avocado: Make a duplicate copy of testsuite for comparison purposes avocado: Don't double download assets for test/ and oldtest/ avocado: Move static checkers to avocado avocado: Introduce "avocado-classless" plugin, runner and outline avocado, test: Add static checkers for Python code avocado: Resolver implementation for avocado-classless plugin avocado: Add basic assertion helpers to avocado-classless plugin tasst, avocado: Introduce library of common test helpers avocado-classless: Test matrices by composition tasst: Helper functions for executing commands in different places avocado-classless: Allow overriding default timeout avocado: Convert build tests to avocado tasst: Add helpers for running background commands on sites tasst: Add helper to get network interface names for a site tasst: Add helpers to run commands with nstool tasst: Add ifup and network address helpers to Site tasst: Helper for creating veth devices between namespaces tasst: Add helper for getting MTU of a network interface tasst: Add helper to wait for IP address to appear tasst: Add helpers for getting a site's routes tasst: Helpers to test transferring data between sites tasst: IP address allocation helpers tasst: Helpers for running daemons with a pidfile tasst: Helpers for testing NDP behaviour tasst: Helpers for testing DHCP & DHCPv6 behaviour tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests Makefile | 9 + oldtest/.gitignore | 11 + oldtest/Makefile | 119 +++ oldtest/README.md | 137 +++ {test => oldtest}/build/all | 0 {test => oldtest}/build/clang_tidy | 0 {test => oldtest}/build/cppcheck | 0 oldtest/ci | 1 + oldtest/demo/passt | 245 ++++++ oldtest/demo/pasta | 274 ++++++ oldtest/demo/podman | 819 ++++++++++++++++++ oldtest/distro/debian | 252 ++++++ oldtest/distro/fedora | 396 +++++++++ oldtest/distro/opensuse | 208 +++++ oldtest/distro/ubuntu | 216 +++++ oldtest/env/mate-terminal.profile | 42 + oldtest/find-arm64-firmware.sh | 13 + oldtest/lib/context | 130 +++ oldtest/lib/layout | 259 ++++++ oldtest/lib/layout_ugly | 113 +++ oldtest/lib/perf_report | 272 ++++++ oldtest/lib/setup | 385 ++++++++ oldtest/lib/setup_ugly | 58 ++ oldtest/lib/term | 750 ++++++++++++++++ oldtest/lib/test | 398 +++++++++ oldtest/lib/util | 133 +++ oldtest/lib/video | 152 ++++ oldtest/memory/passt | 187 ++++ oldtest/nstool.c | 565 ++++++++++++ oldtest/passt.mbuto | 83 ++ oldtest/passt.mem.mbuto | 44 + oldtest/passt/dhcp | 70 ++ oldtest/passt/ndp | 33 + oldtest/passt/shutdown | 19 + oldtest/passt/tcp | 76 ++ oldtest/passt/udp | 46 + oldtest/passt_in_ns/icmp | 32 + oldtest/passt_in_ns/shutdown | 19 + oldtest/passt_in_ns/tcp | 256 ++++++ oldtest/passt_in_ns/udp | 138 +++ {test => oldtest}/pasta/dhcp | 0 {test => oldtest}/pasta/ndp | 0 {test => oldtest}/pasta/tcp | 0 {test => oldtest}/pasta/udp | 0 oldtest/pasta_options/log_to_file | 93 ++ oldtest/perf/passt_tcp | 215 +++++ oldtest/perf/passt_udp | 165 ++++ oldtest/perf/pasta_tcp | 300 +++++++ oldtest/perf/pasta_udp | 219 +++++ oldtest/prepare-distro-img.sh | 18 + oldtest/run | 238 +++++ oldtest/run_demo | 1 + oldtest/two_guests/basic | 80 ++ oldtest/valgrind.supp | 9 + test/.gitignore | 2 + test/Makefile | 69 +- test/avocado/build.py | 94 ++ test/avocado/pasta.py | 129 +++ test/avocado/static_checkers/clang-tidy.sh | 3 + test/avocado/static_checkers/cppcheck.sh | 3 + test/avocado/static_checkers/flake8.sh | 3 + test/avocado/static_checkers/pylint.sh | 3 + test/avocado_classless/.gitignore | 1 + .../avocado_classless/__init__.py | 11 + .../avocado_classless/manifest.py | 48 + .../avocado_classless/plugin.py | 231 +++++ .../avocado_classless/test.py | 80 ++ test/avocado_classless/examples.py | 57 ++ test/avocado_classless/selftests.py | 76 ++ test/avocado_classless/setup.py | 32 + test/lib/layout | 31 - test/lib/setup | 47 - test/run | 14 - test/tasst/__init__.py | 11 + test/tasst/address.py | 80 ++ test/tasst/dhcp.py | 114 +++ test/tasst/dhcpv6.py | 74 ++ test/tasst/exesite.py | 285 ++++++ test/tasst/meta/__init__.py | 16 + test/tasst/meta/static_ifup.py | 60 ++ test/tasst/meta/veth.py | 86 ++ test/tasst/ndp.py | 99 +++ test/tasst/nstool.py | 184 ++++ test/tasst/pasta.py | 42 + test/tasst/scenario/__init__.py | 12 + test/tasst/scenario/simple.py | 98 +++ test/tasst/transfer.py | 174 ++++ test/tasst/typecheck.py | 47 + 88 files changed, 10518 insertions(+), 96 deletions(-) create mode 100644 oldtest/.gitignore create mode 100644 oldtest/Makefile create mode 100644 oldtest/README.md rename {test => oldtest}/build/all (100%) rename {test => oldtest}/build/clang_tidy (100%) rename {test => oldtest}/build/cppcheck (100%) create mode 120000 oldtest/ci create mode 100644 oldtest/demo/passt create mode 100644 oldtest/demo/pasta create mode 100644 oldtest/demo/podman create mode 100644 oldtest/distro/debian create mode 100644 oldtest/distro/fedora create mode 100644 oldtest/distro/opensuse create mode 100644 oldtest/distro/ubuntu create mode 100644 oldtest/env/mate-terminal.profile create mode 100755 oldtest/find-arm64-firmware.sh create mode 100644 oldtest/lib/context create mode 100644 oldtest/lib/layout create mode 100644 oldtest/lib/layout_ugly create mode 100755 oldtest/lib/perf_report create mode 100755 oldtest/lib/setup create mode 100755 oldtest/lib/setup_ugly create mode 100755 oldtest/lib/term create mode 100755 oldtest/lib/test create mode 100755 oldtest/lib/util create mode 100755 oldtest/lib/video create mode 100644 oldtest/memory/passt create mode 100644 oldtest/nstool.c create mode 100755 oldtest/passt.mbuto create mode 100755 oldtest/passt.mem.mbuto create mode 100644 oldtest/passt/dhcp create mode 100644 oldtest/passt/ndp create mode 100644 oldtest/passt/shutdown create mode 100644 oldtest/passt/tcp create mode 100644 oldtest/passt/udp create mode 100644 oldtest/passt_in_ns/icmp create mode 100644 oldtest/passt_in_ns/shutdown create mode 100644 oldtest/passt_in_ns/tcp create mode 100644 oldtest/passt_in_ns/udp rename {test => oldtest}/pasta/dhcp (100%) rename {test => oldtest}/pasta/ndp (100%) rename {test => oldtest}/pasta/tcp (100%) rename {test => oldtest}/pasta/udp (100%) create mode 100644 oldtest/pasta_options/log_to_file create mode 100644 oldtest/perf/passt_tcp create mode 100644 oldtest/perf/passt_udp create mode 100644 oldtest/perf/pasta_tcp create mode 100644 oldtest/perf/pasta_udp create mode 100755 oldtest/prepare-distro-img.sh create mode 100755 oldtest/run create mode 120000 oldtest/run_demo create mode 100644 oldtest/two_guests/basic create mode 100644 oldtest/valgrind.supp create mode 100644 test/avocado/build.py create mode 100644 test/avocado/pasta.py create mode 100755 test/avocado/static_checkers/clang-tidy.sh create mode 100755 test/avocado/static_checkers/cppcheck.sh create mode 100755 test/avocado/static_checkers/flake8.sh create mode 100755 test/avocado/static_checkers/pylint.sh create mode 100644 test/avocado_classless/.gitignore create mode 100644 test/avocado_classless/avocado_classless/__init__.py create mode 100644 test/avocado_classless/avocado_classless/manifest.py create mode 100644 test/avocado_classless/avocado_classless/plugin.py create mode 100644 test/avocado_classless/avocado_classless/test.py create mode 100644 test/avocado_classless/examples.py create mode 100644 test/avocado_classless/selftests.py create mode 100644 test/avocado_classless/setup.py create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/address.py create mode 100644 test/tasst/dhcp.py create mode 100644 test/tasst/dhcpv6.py create mode 100644 test/tasst/exesite.py create mode 100644 test/tasst/meta/__init__.py create mode 100644 test/tasst/meta/static_ifup.py create mode 100644 test/tasst/meta/veth.py create mode 100644 test/tasst/ndp.py create mode 100644 test/tasst/nstool.py create mode 100644 test/tasst/pasta.py create mode 100644 test/tasst/scenario/__init__.py create mode 100644 test/tasst/scenario/simple.py create mode 100644 test/tasst/transfer.py create mode 100644 test/tasst/typecheck.py -- 2.41.0