* [PATCH 0/3] Make podman tests in testsuite more robust
@ 2024-03-19 5:07 David Gibson
2024-03-19 5:07 ` [PATCH 1/3] test: Make sure to update mbuto repository David Gibson
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: David Gibson @ 2024-03-19 5:07 UTC (permalink / raw)
To: passt-dev, Stefano Brivio; +Cc: David Gibson
I noticed these weren't actually running on my Fedora host. This
turns out to be because cataonit is not in the path on Fedora (it's in
/usr/libexec).
The podman tests also downloaded and build podman in the test proper,
which requires external network access. That's awkward in some cases,
and in particular precludes running the tests in an isolated network
environment. Move the download and build of podman to the "make
assets" stage instead.
David Gibson (3):
test: Make sure to update mbuto repository
test: Build and download podman as a test asset
test: catatonit may not be in $PATH
test/.gitignore | 1 +
test/Makefile | 20 +++++++++++++++++---
test/pasta_podman/bats | 6 ++----
3 files changed, 20 insertions(+), 7 deletions(-)
--
2.44.0
^ permalink raw reply [flat|nested] 6+ messages in thread* [PATCH 1/3] test: Make sure to update mbuto repository 2024-03-19 5:07 [PATCH 0/3] Make podman tests in testsuite more robust David Gibson @ 2024-03-19 5:07 ` David Gibson 2024-03-19 5:07 ` [PATCH 2/3] test: Build and download podman as a test asset David Gibson 2024-03-19 5:07 ` [PATCH 3/3] test: catatonit may not be in $PATH David Gibson 2 siblings, 0 replies; 6+ messages in thread From: David Gibson @ 2024-03-19 5:07 UTC (permalink / raw) To: passt-dev, Stefano Brivio; +Cc: David Gibson We download and use mbuto to build trivial boot images for our VM tests. However, if mbuto is already cloned, we won't update it to the current version. Add some make logic to ensure that we do this. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> --- test/Makefile | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/test/Makefile b/test/Makefile index 7b00bef4..711c61c1 100644 --- a/test/Makefile +++ b/test/Makefile @@ -67,13 +67,19 @@ CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 assets: $(ASSETS) +.PHONY: pull-% +pull-%: % + git -C $* pull + mbuto: git clone git://mbuto.sh/mbuto +mbuto/mbuto: pull-mbuto + guest-key guest-key.pub: ssh-keygen -f guest-key -N '' -mbuto.img: passt.mbuto mbuto guest-key.pub $(TESTDATA_ASSETS) +mbuto.img: passt.mbuto mbuto/mbuto guest-key.pub $(TESTDATA_ASSETS) ./mbuto/mbuto -p ./$< -c lz4 -f $@ mbuto.mem.img: passt.mem.mbuto mbuto ../passt.avx2 -- @@ -67,13 +67,19 @@ CFLAGS = -Wall -Werror -Wextra -pedantic -std=c99 assets: $(ASSETS) +.PHONY: pull-% +pull-%: % + git -C $* pull + mbuto: git clone git://mbuto.sh/mbuto +mbuto/mbuto: pull-mbuto + guest-key guest-key.pub: ssh-keygen -f guest-key -N '' -mbuto.img: passt.mbuto mbuto guest-key.pub $(TESTDATA_ASSETS) +mbuto.img: passt.mbuto mbuto/mbuto guest-key.pub $(TESTDATA_ASSETS) ./mbuto/mbuto -p ./$< -c lz4 -f $@ mbuto.mem.img: passt.mem.mbuto mbuto ../passt.avx2 -- 2.44.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/3] test: Build and download podman as a test asset 2024-03-19 5:07 [PATCH 0/3] Make podman tests in testsuite more robust David Gibson 2024-03-19 5:07 ` [PATCH 1/3] test: Make sure to update mbuto repository David Gibson @ 2024-03-19 5:07 ` David Gibson 2024-03-19 10:07 ` Paul Holzinger 2024-03-19 5:07 ` [PATCH 3/3] test: catatonit may not be in $PATH David Gibson 2 siblings, 1 reply; 6+ messages in thread From: David Gibson @ 2024-03-19 5:07 UTC (permalink / raw) To: passt-dev, Stefano Brivio; +Cc: David Gibson The pasta_podman/bats test scrpt downloads and builds podman, then runs its pasta specific tests. Downloading from within a test case has some drawbacks: * It can be very tedious if you have poor connectivity to the server * It makes a test that's ostensibly for pasta itself dependent on the state of the github server * It precludes runnning the tests in an isolated network environment The same concerns largely apply to building podman too, because it's pretty common for Go builds to download dependencies themselves. Therefore move the download and build of podman from the test itself, to the Makefile where we prepare other test assets. To avoid cryptic failures if something went wrong with the build, make running the test dependent on having the built podman binary. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> --- test/.gitignore | 1 + test/Makefile | 12 ++++++++++-- test/pasta_podman/bats | 6 ++---- 3 files changed, 13 insertions(+), 6 deletions(-) diff --git a/test/.gitignore b/test/.gitignore index 48374028..6dd4790b 100644 --- a/test/.gitignore +++ b/test/.gitignore @@ -1,5 +1,6 @@ test_logs/ mbuto/ +podman/ *.img QEMU_EFI.fd *.qcow2 diff --git a/test/Makefile b/test/Makefile index 711c61c1..35a3b559 100644 --- a/test/Makefile +++ b/test/Makefile @@ -52,10 +52,10 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \ jammy-server-cloudimg-s390x.img UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS) -DOWNLOAD_ASSETS = mbuto \ +DOWNLOAD_ASSETS = mbuto podman \ $(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS) TESTDATA_ASSETS = small.bin big.bin medium.bin -LOCAL_ASSETS = mbuto.img mbuto.mem.img QEMU_EFI.fd \ +LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ $(DEBIAN_IMGS:%=prepared-%) $(FEDORA_IMGS:%=prepared-%) \ $(UBUNTU_NEW_IMGS:%=prepared-%) \ nstool guest-key guest-key.pub \ @@ -76,6 +76,14 @@ mbuto: mbuto/mbuto: pull-mbuto +podman: + git clone https://github.com/containers/podman.git + +# To succesfully build podman, you will need gpgme and systemd +# development packages +podman/bin/podman: pull-podman + $(MAKE) -C podman + guest-key guest-key.pub: ssh-keygen -f guest-key -N '' diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats index 21446f08..cb88aa41 100644 --- a/test/pasta_podman/bats +++ b/test/pasta_podman/bats @@ -11,11 +11,9 @@ # Copyright (c) 2022 Red Hat GmbH # Author: Stefano Brivio <sbrivio@redhat.com> -htools git make go bats catatonit ip jq socat +htools git make go bats catatonit ip jq socat ./test/podman/bin/podman test Podman system test with bats -host git -C __STATEDIR__ clone https://github.com/containers/podman.git -host make -C __STATEDIR__/podman hout WD pwd -host PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats +host PODMAN="test/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats test/podman/test/system/505-networking-pasta.bats -- @@ -11,11 +11,9 @@ # Copyright (c) 2022 Red Hat GmbH # Author: Stefano Brivio <sbrivio@redhat.com> -htools git make go bats catatonit ip jq socat +htools git make go bats catatonit ip jq socat ./test/podman/bin/podman test Podman system test with bats -host git -C __STATEDIR__ clone https://github.com/containers/podman.git -host make -C __STATEDIR__/podman hout WD pwd -host PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats +host PODMAN="test/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats test/podman/test/system/505-networking-pasta.bats -- 2.44.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 2/3] test: Build and download podman as a test asset 2024-03-19 5:07 ` [PATCH 2/3] test: Build and download podman as a test asset David Gibson @ 2024-03-19 10:07 ` Paul Holzinger 2024-03-20 13:08 ` David Gibson 0 siblings, 1 reply; 6+ messages in thread From: Paul Holzinger @ 2024-03-19 10:07 UTC (permalink / raw) To: David Gibson, passt-dev, Stefano Brivio On 19/03/2024 06:07, David Gibson wrote: > The pasta_podman/bats test scrpt downloads and builds podman, then runs its > pasta specific tests. Downloading from within a test case has some > drawbacks: > * It can be very tedious if you have poor connectivity to the server > * It makes a test that's ostensibly for pasta itself dependent on the > state of the github server > * It precludes runnning the tests in an isolated network environment > > The same concerns largely apply to building podman too, because it's pretty > common for Go builds to download dependencies themselves. Therefore move > the download and build of podman from the test itself, to the Makefile > where we prepare other test assets. FYI, we use vendoring in podman and most of our other repos. So running something like `make podman` should never download anything. The only target that downloads the deps is `make vendor` which is not needed unless you update the dependencies. Anyhow it doesn't change the fact that your other reasons for this change are valid. > To avoid cryptic failures if something went wrong with the build, make > running the test dependent on having the built podman binary. > > Signed-off-by: David Gibson <david@gibson.dropbear.id.au> > --- > test/.gitignore | 1 + > test/Makefile | 12 ++++++++++-- > test/pasta_podman/bats | 6 ++---- > 3 files changed, 13 insertions(+), 6 deletions(-) > > diff --git a/test/.gitignore b/test/.gitignore > index 48374028..6dd4790b 100644 > --- a/test/.gitignore > +++ b/test/.gitignore > @@ -1,5 +1,6 @@ > test_logs/ > mbuto/ > +podman/ > *.img > QEMU_EFI.fd > *.qcow2 > diff --git a/test/Makefile b/test/Makefile > index 711c61c1..35a3b559 100644 > --- a/test/Makefile > +++ b/test/Makefile > @@ -52,10 +52,10 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \ > jammy-server-cloudimg-s390x.img > UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS) > > -DOWNLOAD_ASSETS = mbuto \ > +DOWNLOAD_ASSETS = mbuto podman \ > $(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS) > TESTDATA_ASSETS = small.bin big.bin medium.bin > -LOCAL_ASSETS = mbuto.img mbuto.mem.img QEMU_EFI.fd \ > +LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ > $(DEBIAN_IMGS:%=prepared-%) $(FEDORA_IMGS:%=prepared-%) \ > $(UBUNTU_NEW_IMGS:%=prepared-%) \ > nstool guest-key guest-key.pub \ > @@ -76,6 +76,14 @@ mbuto: > > mbuto/mbuto: pull-mbuto > > +podman: > + git clone https://github.com/containers/podman.git > + > +# To succesfully build podman, you will need gpgme and systemd > +# development packages > +podman/bin/podman: pull-podman > + $(MAKE) -C podman > + > guest-key guest-key.pub: > ssh-keygen -f guest-key -N '' > > diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats > index 21446f08..cb88aa41 100644 > --- a/test/pasta_podman/bats > +++ b/test/pasta_podman/bats > @@ -11,11 +11,9 @@ > # Copyright (c) 2022 Red Hat GmbH > # Author: Stefano Brivio <sbrivio@redhat.com> > > -htools git make go bats catatonit ip jq socat > +htools git make go bats catatonit ip jq socat ./test/podman/bin/podman > > test Podman system test with bats > > -host git -C __STATEDIR__ clone https://github.com/containers/podman.git > -host make -C __STATEDIR__/podman > hout WD pwd > -host PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats > +host PODMAN="test/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats test/podman/test/system/505-networking-pasta.bats Note while these env vars are stable and should not change I would strongly recommend to add some kind of smoke test here in the form of running `podman info --format "{{.Host.Pasta.Executable}}"`, this will return the full path to the binary so you should make sure this is the path to your test binary and not the system pasta version to ensure you are actually testing the right thing. ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/3] test: Build and download podman as a test asset 2024-03-19 10:07 ` Paul Holzinger @ 2024-03-20 13:08 ` David Gibson 0 siblings, 0 replies; 6+ messages in thread From: David Gibson @ 2024-03-20 13:08 UTC (permalink / raw) To: Paul Holzinger; +Cc: passt-dev, Stefano Brivio [-- Attachment #1: Type: text/plain, Size: 4570 bytes --] On Tue, Mar 19, 2024 at 11:07:40AM +0100, Paul Holzinger wrote: > > On 19/03/2024 06:07, David Gibson wrote: > > The pasta_podman/bats test scrpt downloads and builds podman, then runs its > > pasta specific tests. Downloading from within a test case has some > > drawbacks: > > * It can be very tedious if you have poor connectivity to the server > > * It makes a test that's ostensibly for pasta itself dependent on the > > state of the github server > > * It precludes runnning the tests in an isolated network environment > > > > The same concerns largely apply to building podman too, because it's pretty > > common for Go builds to download dependencies themselves. Therefore move > > the download and build of podman from the test itself, to the Makefile > > where we prepare other test assets. > FYI, we use vendoring in podman and most of our other repos. So running > something like `make podman` should never download anything. The only target > that downloads the deps is `make vendor` which is not needed unless you > update the dependencies. Anyhow it doesn't change the fact that your other > reasons for this change are valid. Ok, good to know. > > To avoid cryptic failures if something went wrong with the build, make > > running the test dependent on having the built podman binary. > > > > Signed-off-by: David Gibson <david@gibson.dropbear.id.au> > > --- > > test/.gitignore | 1 + > > test/Makefile | 12 ++++++++++-- > > test/pasta_podman/bats | 6 ++---- > > 3 files changed, 13 insertions(+), 6 deletions(-) > > > > diff --git a/test/.gitignore b/test/.gitignore > > index 48374028..6dd4790b 100644 > > --- a/test/.gitignore > > +++ b/test/.gitignore > > @@ -1,5 +1,6 @@ > > test_logs/ > > mbuto/ > > +podman/ > > *.img > > QEMU_EFI.fd > > *.qcow2 > > diff --git a/test/Makefile b/test/Makefile > > index 711c61c1..35a3b559 100644 > > --- a/test/Makefile > > +++ b/test/Makefile > > @@ -52,10 +52,10 @@ UBUNTU_NEW_IMGS = xenial-server-cloudimg-powerpc-disk1.img \ > > jammy-server-cloudimg-s390x.img > > UBUNTU_IMGS = $(UBUNTU_OLD_IMGS) $(UBUNTU_NEW_IMGS) > > -DOWNLOAD_ASSETS = mbuto \ > > +DOWNLOAD_ASSETS = mbuto podman \ > > $(DEBIAN_IMGS) $(FEDORA_IMGS) $(OPENSUSE_IMGS) $(UBUNTU_IMGS) > > TESTDATA_ASSETS = small.bin big.bin medium.bin > > -LOCAL_ASSETS = mbuto.img mbuto.mem.img QEMU_EFI.fd \ > > +LOCAL_ASSETS = mbuto.img mbuto.mem.img podman/bin/podman QEMU_EFI.fd \ > > $(DEBIAN_IMGS:%=prepared-%) $(FEDORA_IMGS:%=prepared-%) \ > > $(UBUNTU_NEW_IMGS:%=prepared-%) \ > > nstool guest-key guest-key.pub \ > > @@ -76,6 +76,14 @@ mbuto: > > mbuto/mbuto: pull-mbuto > > +podman: > > + git clone https://github.com/containers/podman.git > > + > > +# To succesfully build podman, you will need gpgme and systemd > > +# development packages > > +podman/bin/podman: pull-podman > > + $(MAKE) -C podman > > + > > guest-key guest-key.pub: > > ssh-keygen -f guest-key -N '' > > diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats > > index 21446f08..cb88aa41 100644 > > --- a/test/pasta_podman/bats > > +++ b/test/pasta_podman/bats > > @@ -11,11 +11,9 @@ > > # Copyright (c) 2022 Red Hat GmbH > > # Author: Stefano Brivio <sbrivio@redhat.com> > > -htools git make go bats catatonit ip jq socat > > +htools git make go bats catatonit ip jq socat ./test/podman/bin/podman > > test Podman system test with bats > > -host git -C __STATEDIR__ clone https://github.com/containers/podman.git > > -host make -C __STATEDIR__/podman > > hout WD pwd > > -host PODMAN="__STATEDIR__/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats __STATEDIR__/podman/test/system/505-networking-pasta.bats > > +host PODMAN="test/podman/bin/podman" CONTAINERS_HELPER_BINARY_DIR="__WD__" bats test/podman/test/system/505-networking-pasta.bats > Note while these env vars are stable and should not change I would strongly > recommend to add some kind of smoke test here in the form of running `podman > info --format "{{.Host.Pasta.Executable}}"`, this will return the full path > to the binary so you should make sure this is the path to your test binary > and not the system pasta version to ensure you are actually testing the > right thing. Good idea, I've added that for v2. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 3/3] test: catatonit may not be in $PATH 2024-03-19 5:07 [PATCH 0/3] Make podman tests in testsuite more robust David Gibson 2024-03-19 5:07 ` [PATCH 1/3] test: Make sure to update mbuto repository David Gibson 2024-03-19 5:07 ` [PATCH 2/3] test: Build and download podman as a test asset David Gibson @ 2024-03-19 5:07 ` David Gibson 2 siblings, 0 replies; 6+ messages in thread From: David Gibson @ 2024-03-19 5:07 UTC (permalink / raw) To: passt-dev, Stefano Brivio; +Cc: David Gibson The pasta_podman/bats test script looks for 'catatonit' amongst other tools to be avaiiliable on the host. However, while the podman tests do require catatonit, it doesn't necessarily need to be in the regular path. For example Fedora and RHEL place catatonit in /usr/libexec and podman finds it there fine. Therefore, remove it as an htools dependency. Signed-off-by: David Gibson <david@gibson.dropbear.id.au> --- test/pasta_podman/bats | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/test/pasta_podman/bats b/test/pasta_podman/bats index cb88aa41..46a958a9 100644 --- a/test/pasta_podman/bats +++ b/test/pasta_podman/bats @@ -11,7 +11,7 @@ # Copyright (c) 2022 Red Hat GmbH # Author: Stefano Brivio <sbrivio@redhat.com> -htools git make go bats catatonit ip jq socat ./test/podman/bin/podman +htools git make go bats ip jq socat ./test/podman/bin/podman test Podman system test with bats -- @@ -11,7 +11,7 @@ # Copyright (c) 2022 Red Hat GmbH # Author: Stefano Brivio <sbrivio@redhat.com> -htools git make go bats catatonit ip jq socat ./test/podman/bin/podman +htools git make go bats ip jq socat ./test/podman/bin/podman test Podman system test with bats -- 2.44.0 ^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2024-03-20 13:13 UTC | newest] Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2024-03-19 5:07 [PATCH 0/3] Make podman tests in testsuite more robust David Gibson 2024-03-19 5:07 ` [PATCH 1/3] test: Make sure to update mbuto repository David Gibson 2024-03-19 5:07 ` [PATCH 2/3] test: Build and download podman as a test asset David Gibson 2024-03-19 10:07 ` Paul Holzinger 2024-03-20 13:08 ` David Gibson 2024-03-19 5:07 ` [PATCH 3/3] test: catatonit may not be in $PATH David Gibson
Code repositories for project(s) associated with this public inbox https://passt.top/passt This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for IMAP folder(s).