From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202408 header.b=ZP1wOBoR; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id BA14A5A004E for ; Mon, 26 Aug 2024 04:09:48 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202408; t=1724638184; bh=0YjEC1sjGUmQ7UwnrrLinb/7adobPOZvWMcxbm8w/rE=; h=From:To:Cc:Subject:Date:From; b=ZP1wOBoRC71whonUViQUirwe4zba1K8nXfyIHcrjE24IEmwqY/iv9ePFOTyZsAMOf BA6IQlHkO/nvWnrL2U3anxDzac1MD0QmiBVcVvhhw1nJnftIg4pnNBJOWJkWpes/HW 52hvYPmpNnH5r1pf+Hox6DO/B5iWhXDRNv2I91U4MNlVZmFJjyXeJ3MWhtb6wVHPZk b8ImGo2gPpwbcEfemAMvIV4L4MRr+0e03jXYVGBmFBC1y00r307gHUPqdAB7XDyLNz n3iYKmha1MxnmoEg0aTUcv6VJ2jk5+Tm4pm/OWNYe71XekOYHg3LMuWesit/nLmX5i jPhobDQuomLzw== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4WsYyw4zBNz4x6l; Mon, 26 Aug 2024 12:09:44 +1000 (AEST) From: David Gibson To: Stefano Brivio , passt-dev@passt.top Subject: [PATCH v3 00/15] RFC: Proof-of-concept based exeter+Avocado tests Date: Mon, 26 Aug 2024 12:09:27 +1000 Message-ID: <20240826020942.545155-1-david@gibson.dropbear.id.au> X-Mailer: git-send-email 2.46.0 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Message-ID-Hash: J7C3755AIUWSJKR7SPJWCO2BENFL2BSE X-Message-ID-Hash: J7C3755AIUWSJKR7SPJWCO2BENFL2BSE X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Cleber Rosa , David Gibson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Here's another draft of my work on testing passt with Avocado and the exeter library I recently created. It includes Cleber's patch adding some basic Avocado tests and builds on that. This draft is IMO significantly less janky than the last one, but certainly not jank-free: * We create the Avocado job files from the exeter sources in the Makefile. Ideally Avocado would eventually be extended to handle this itself * The names that Avocado sees for each test aren't great Stefano, If you could look particularly at 7/15 and 15/15 which add the real tests for passt/pasta, that would be great. The more specific you can be about what you find ugly about how the tests are written, then better I can try to address that. I suspect it will be easier to actually apply the series, then look at the new test files (test/build/build.py, and test/pasta/pasta.py particularly). From there you can look at as much of the support library as you need to, rather than digging through the actual patches to look for that. Cleber, If you could look at 4..7/22 particularly to review how I'm connecting the actual tests to the Avocado runner, that would be helpful. Changes since v2: * Added mypy type checking throughout * Use exeter "scenarios" to reduce boilerplate * Folded together a number of closely related patches (from 22 patches down to 15) * Entirely avoid open-coded Python context managers in favour of contextlib.contextmanager * No longer accidentally run a lot of the "meta" tests in the "real" testsuite * So, so many assorted cleanups Cleber Rosa (1): test: run static checkers with Avocado and JSON definitions David Gibson (14): test: Adjust how we invoke tests with run_avocado test: Extend make targets to run Avocado tests test: Exeter based static tests tasst: Support library and linters for tests in Python tasst/cmdsite: Base helpers for running shell commands in various places test: Add exeter+Avocado based build tests tasst/unshare: Add helpers to run commands in Linux unshared namespaces tasst/ip: Helpers for configuring IPv4 and IPv6 tasst/veth: Helpers for constructing veth devices between namespaces tasst: Helpers to test transferring data between sites tasst: Helpers for testing NDP behaviour tasst: Helpers for testing DHCP & DHCPv6 behaviour tasst: Helpers to construct a simple network environment for tests avocado: Convert basic pasta tests test/.gitignore | 2 + test/Makefile | 59 +++++++- test/avocado/static_checkers.json | 12 ++ test/build/.gitignore | 2 + test/build/build.py | 86 +++++++++++ test/build/static_checkers.sh | 28 ++++ test/meta/.gitignore | 1 + test/meta/lint.sh | 28 ++++ test/pasta/.gitignore | 1 + test/pasta/pasta.py | 130 +++++++++++++++++ test/run_avocado | 52 +++++++ test/tasst/.gitignore | 1 + test/tasst/__init__.py | 11 ++ test/tasst/__main__.py | 40 +++++ test/tasst/cmdsite.py | 193 ++++++++++++++++++++++++ test/tasst/dhcp.py | 190 ++++++++++++++++++++++++ test/tasst/ip.py | 234 ++++++++++++++++++++++++++++++ test/tasst/ndp.py | 106 ++++++++++++++ test/tasst/pasta.py | 48 ++++++ test/tasst/scenario/__init__.py | 12 ++ test/tasst/scenario/simple.py | 108 ++++++++++++++ test/tasst/selftest/__init__.py | 16 ++ test/tasst/transfer.py | 193 ++++++++++++++++++++++++ test/tasst/unshare.py | 166 +++++++++++++++++++++ test/tasst/veth.py | 104 +++++++++++++ 25 files changed, 1822 insertions(+), 1 deletion(-) create mode 100644 test/avocado/static_checkers.json create mode 100644 test/build/.gitignore create mode 100644 test/build/build.py create mode 100644 test/build/static_checkers.sh create mode 100644 test/meta/.gitignore create mode 100644 test/meta/lint.sh create mode 100644 test/pasta/.gitignore create mode 100644 test/pasta/pasta.py create mode 100755 test/run_avocado create mode 100644 test/tasst/.gitignore create mode 100644 test/tasst/__init__.py create mode 100644 test/tasst/__main__.py create mode 100644 test/tasst/cmdsite.py create mode 100644 test/tasst/dhcp.py create mode 100644 test/tasst/ip.py create mode 100644 test/tasst/ndp.py create mode 100644 test/tasst/pasta.py create mode 100644 test/tasst/scenario/__init__.py create mode 100644 test/tasst/scenario/simple.py create mode 100644 test/tasst/selftest/__init__.py create mode 100644 test/tasst/transfer.py create mode 100644 test/tasst/unshare.py create mode 100644 test/tasst/veth.py -- 2.46.0