[PATCH 00/10] passt test fixes, batch 6

[PATCH 00/10] passt test fixes, batch 6

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2003 bytes --]

Here's get another batch of fixes and cleanups for passt/pasta tests.
Some of these are preliminary fixes before an upcoming series changing
to use more reliable (less tmux screen-scraping dependent) methods for
dispatching commands in various contexts.  Others are just
miscellaneous fixes.

David Gibson (10):
  test: Convert distro tests to use socat instead of nc/ncat
  test: Split cppcheck and clang-tidy tests into different files
  test: Remove unused *_XTERM variables
  test: Ignore video processing temporary files
  test: Split setup/teardown functions for build and distro tests
  test: Only select a single interface or gateway in tests
  test: Rename slightly misleading "valgrind" tests
  test: Use shutdown test for pasta
  test: Log debugging output from test script
  test: Kill qemu by pidfile rather than ^C

 test/.gitignore                            |  4 ++
 test/build/{static_checkers => clang_tidy} |  5 +--
 test/build/cppcheck                        | 17 ++++++++
 test/dhcp/passt                            |  4 +-
 test/dhcp/pasta                            |  4 +-
 test/distro/debian                         | 28 ++++++-------
 test/distro/fedora                         | 48 +++++++++++-----------
 test/distro/opensuse                       | 18 ++++----
 test/distro/ubuntu                         | 16 ++++----
 test/lib/setup                             | 41 ++++++++++++------
 test/lib/term                              |  4 --
 test/ndp/passt                             |  4 +-
 test/ndp/pasta                             |  2 +-
 test/run                                   | 15 +++++--
 test/{valgrind => shutdown}/passt          |  9 ++--
 test/{valgrind => shutdown}/passt_in_ns    |  9 ++--
 16 files changed, 134 insertions(+), 94 deletions(-)
 rename test/build/{static_checkers => clang_tidy} (86%)
 create mode 100644 test/build/cppcheck
 rename test/{valgrind => shutdown}/passt (62%)
 rename test/{valgrind => shutdown}/passt_in_ns (62%)

-- 
2.37.2

[PATCH 01/10] test: Convert distro tests to use socat instead of nc/ncat

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 21331 bytes --]

We've recently converted most of our tests to use socat instead of
nc/netcat/ncat, because socat is more powerful and we don't need to deal
with the several possible variants of netcat.

We still use nc or ncat for the distro tests.  Because there we control
the guest environment and can pick our tools, there isn't the same reason
to switch to socat.  However, using socat here as well makes the tests
a bit easier to read, and doesn't require people reading or modifying them
to become familiar with an additional tool.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/distro/debian   | 28 +++++++++++++-------------
 test/distro/fedora   | 48 ++++++++++++++++++++++----------------------
 test/distro/opensuse | 18 ++++++++---------
 test/distro/ubuntu   | 16 +++++++--------
 4 files changed, 55 insertions(+), 55 deletions(-)

diff --git a/test/distro/debian b/test/distro/debian
index 1548761..0fac983 100644
--- a/test/distro/debian
+++ b/test/distro/debian
@@ -18,12 +18,12 @@ htools	cat kill qemu-system-x86_64 qemu-system-aarch64 qemu-system-ppc64
 def	distro_quick_pasta_test
 host	export SHELL="/bin/dash"
 host	dash
-host	(nc -w1 -6 -l -p 10000 > /tmp/init_msg; echo "from_init" | nc -q0 ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000 OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
-host	nc -w1 -6 -l -p 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999 OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | nc -q0 ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 sleep	1
@@ -47,7 +47,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make
@@ -68,7 +68,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -86,7 +86,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -104,7 +104,7 @@ host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios _
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -122,7 +122,7 @@ host	./qrap 5 qemu-system-ppc64 -m 2048 -smp 2 -nographic -serial stdio -nodefau
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -140,7 +140,7 @@ test	Debian GNU/Linux 11 (bullseye, stable), amd64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-debian-11-nocloud-amd64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -156,7 +156,7 @@ test	Debian GNU/Linux 11 (bullseye, stable), aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-debian-11-generic-arm64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -172,7 +172,7 @@ test	Debian GNU/Linux 11 (bullseye, stable), ppc64le
 host	./qrap 5 qemu-system-ppc64 -m 2048 -smp 2 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-debian-11-generic-ppc64el.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -198,7 +198,7 @@ sleep	2
 host	growpart /dev/sda 1
 host	resize2fs -p /dev/sda1
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -217,7 +217,7 @@ sleep	2
 host	growpart /dev/vda 1
 host	resize2fs -p /dev/vda1
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -233,7 +233,7 @@ test	Debian GNU/Linux sid (experimental), ppc64le
 host	./qrap 5 qemu-system-ppc64 -m 2048 -smp 2 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-debian-sid-nocloud-ppc64el-daily.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
diff --git a/test/distro/fedora b/test/distro/fedora
index 56805c5..daad6a4 100644
--- a/test/distro/fedora
+++ b/test/distro/fedora
@@ -16,13 +16,13 @@ htools	cat kill qemu-system-x86_64
 
 # Quick pasta test: send message from init to ns, and from ns to init
 def	distro_quick_pasta_test
-host	(ncat -6 -l ::1 10000 > /tmp/init_msg; echo "from_init" | ncat ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000,bind=[::1] OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
 host	PS1='$ '
-host	ncat -6 -l ::1 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999,bind=[::1] OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | ncat ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 sleep	1
@@ -37,14 +37,14 @@ endef
 # Bracketed paste mode off, needed from Fedora 34
 def	distro_quick_pasta_test_fedora34
 host	bind 'set enable-bracketed-paste off'
-host	(ncat -6 -l ::1 10000 > /tmp/init_msg; echo "from_init" | ncat -6 ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000,bind=[::1] OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
 host	PS1='$ '
 host	bind 'set enable-bracketed-paste off'
-host	ncat -6 -l ::1 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999,bind=[::1] OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | ncat -6 ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 hout	NS_MSG cat /tmp/ns_msg
@@ -65,7 +65,7 @@ test	Fedora 26, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-26-1.5.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -85,7 +85,7 @@ test	Fedora 27, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-27-1.6.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -102,7 +102,7 @@ test	Fedora 28, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-28-1.1.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -119,7 +119,7 @@ test	Fedora 28, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-28-1.1.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -139,7 +139,7 @@ test	Fedora 29, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-29-1.2.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -156,7 +156,7 @@ test	Fedora 29, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-29-1.2.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -176,7 +176,7 @@ test	Fedora 30, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-30-1.2.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -193,7 +193,7 @@ test	Fedora 30, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-30-1.2.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -213,7 +213,7 @@ test	Fedora 31, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-31-1.9.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -230,7 +230,7 @@ test	Fedora 31, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-31-1.9.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -250,7 +250,7 @@ test	Fedora 32, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-32-1.6.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -267,7 +267,7 @@ test	Fedora 32, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-32-1.6.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -287,7 +287,7 @@ test	Fedora 33, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-33-1.2.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -304,7 +304,7 @@ test	Fedora 33, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-33-1.2.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -324,7 +324,7 @@ test	Fedora 34, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-34-1.2.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -341,7 +341,7 @@ test	Fedora 34, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-34-1.2.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -361,7 +361,7 @@ test	Fedora 35, x86_64
 host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial stdio -nodefaults -no-reboot -nographic -vga none __BASEPATH__/prepared-Fedora-Cloud-Base-35-1.2.x86_64.qcow2 -net socket,fd=5 -net nic,model=virtio -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
@@ -378,7 +378,7 @@ test	Fedora 35, aarch64
 host	./qrap 5 qemu-system-aarch64 -m 2048 -cpu cortex-a57 -smp 2 -M virt -bios __BASEPATH__/QEMU_EFI.fd -nodefaults -nographic -vga none -serial stdio __BASEPATH__/prepared-Fedora-Cloud-Base-35-1.2.aarch64.qcow2 -net socket,fd=5 -net nic,model=virtio -device virtio-rng-pci -snapshot
 host	PS1='$ '
 sleep	2
-host	yum -y install make gcc nmap-ncat
+host	yum -y install make gcc socat
 
 host	make clean
 hout	RET CFLAGS="-Werror" make; echo $?
diff --git a/test/distro/opensuse b/test/distro/opensuse
index ad2108f..75b49c8 100644
--- a/test/distro/opensuse
+++ b/test/distro/opensuse
@@ -16,13 +16,13 @@ htools	qemu-img virt-edit guestfish head sed cat kill qemu-system-x86_64 qemu-sy
 
 # Quick pasta test: send message from init to ns, and from ns to init
 def	distro_quick_pasta_test
-host	(nc -6 -l -p 10000 > /tmp/init_msg; echo "from_init" | nc -N ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000 OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
 host	PS1='$ '
-host	nc -6 -l -p 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999 OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | nc -N ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 sleep	1
@@ -57,7 +57,7 @@ host	ip link set eth0 up
 sleep	2
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 # zypper sometimes segfaults, hence the retries
-host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
+host	for i in $(seq 1 10); do zypper install -y gcc make socat && break; done; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -85,7 +85,7 @@ host	ip link set eth0 up
 sleep	2
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 # zypper sometimes segfaults, hence the retries
-host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
+host	for i in $(seq 1 10); do zypper install -y gcc make socat && break; done; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -112,7 +112,7 @@ host	ip link set eth0 up
 sleep	2
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 # zypper sometimes segfaults, hence the retries
-host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
+host	for i in $(seq 1 10); do zypper install -y gcc make socat && break; done; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -137,7 +137,7 @@ sleep	10
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i enp0s1
 sleep	10
 # No segfaults ever seen with this
-host	zypper install -y gcc make netcat-openbsd; echo
+host	zypper install -y gcc make socat; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -165,7 +165,7 @@ host	ip link set eth0 up
 sleep	10
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 sleep	10
-host	zypper install -y gcc make netcat-openbsd; echo
+host	zypper install -y gcc make socat; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -192,7 +192,7 @@ host	ip link set ens2 up
 sleep	2
 host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i ens2
 # zypper sometimes segfaults, hence the retries
-host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
+host	for i in $(seq 1 10); do zypper install -y gcc make socat && break; done; echo
 
 host	make clean
 host	CFLAGS="-Werror" make
diff --git a/test/distro/ubuntu b/test/distro/ubuntu
index b9a0446..b85a4e2 100644
--- a/test/distro/ubuntu
+++ b/test/distro/ubuntu
@@ -16,13 +16,13 @@ htools	qemu-img virt-edit guestfish cat kill qemu-system-x86_64 qemu-system-ppc6
 
 # Quick pasta test: send message from init to ns, and from ns to init
 def	distro_quick_pasta_test
-host	(nc -w1 -6 -l -p 10000 > /tmp/init_msg; echo "from_init" | nc -q0 ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000 OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
 host	PS1='$ '
-host	nc -w1 -6 -l -p 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999 OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | nc -q0 ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 sleep	1
@@ -59,7 +59,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make
@@ -92,7 +92,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
@@ -122,7 +122,7 @@ host	./qrap 5 qemu-system-ppc64 -m 2048 -smp 2 -nographic -serial stdio -nodefau
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
@@ -145,7 +145,7 @@ host	dhclient -4
 # Skip apt-get update here: some updates to xenial-updates around 2022-01-30
 # broke dependencies for libc6 and gcc-5 -- note that powerpc is not officially
 # supported on this version
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -169,7 +169,7 @@ host	apt-get -y remove needrestart snapd
 host	dhclient -4
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
-- 
@@ -16,13 +16,13 @@ htools	qemu-img virt-edit guestfish cat kill qemu-system-x86_64 qemu-system-ppc6
 
 # Quick pasta test: send message from init to ns, and from ns to init
 def	distro_quick_pasta_test
-host	(nc -w1 -6 -l -p 10000 > /tmp/init_msg; echo "from_init" | nc -q0 ::1 9999) &
+host	(socat -u TCP6-LISTEN:10000 OPEN:/tmp/init_msg,create,trunc; echo "from_init" | socat -u STDIN TCP6:[::1]:9999) &
 hostb	./pasta
 sleep	1
 host	PS1='$ '
-host	nc -w1 -6 -l -p 9999 > /tmp/ns_msg &
+host	socat -u TCP6-LISTEN:9999 OPEN:/tmp/ns_msg,create,trunc &
 sleep	2
-host	echo "from_ns" | nc -q0 ::1 10000
+host	echo "from_ns" | socat -u STDIN TCP6:[::1]:10000
 sleep	2
 host	echo
 sleep	1
@@ -59,7 +59,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make
@@ -92,7 +92,7 @@ host	./qrap 5 qemu-system-x86_64 -M pc,accel=kvm:tcg -m 1024 -nographic -serial
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
@@ -122,7 +122,7 @@ host	./qrap 5 qemu-system-ppc64 -m 2048 -smp 2 -nographic -serial stdio -nodefau
 host	PS1='$ '
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
@@ -145,7 +145,7 @@ host	dhclient -4
 # Skip apt-get update here: some updates to xenial-updates around 2022-01-30
 # broke dependencies for libc6 and gcc-5 -- note that powerpc is not officially
 # supported on this version
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
@@ -169,7 +169,7 @@ host	apt-get -y remove needrestart snapd
 host	dhclient -4
 sleep	2
 host	apt-get update
-host	apt-get -y install make gcc netcat-openbsd
+host	apt-get -y install make gcc socat
 
 host	make clean
 host	CFLAGS="-Werror" make
-- 
2.37.2

[PATCH 02/10] test: Split cppcheck and clang-tidy tests into different files

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2066 bytes --]

Both clang-tidy and cppcheck linting are handled by the same test file,
test/build/static_checkers.  The two linters are independent of each other
though, and each one takes quite a long time.  Split them into separate
files to make it easier to control which are executed from the top level
test script.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/build/{static_checkers => clang_tidy} |  5 +----
 test/build/cppcheck                        | 17 +++++++++++++++++
 test/run                                   |  3 ++-
 3 files changed, 20 insertions(+), 5 deletions(-)
 rename test/build/{static_checkers => clang_tidy} (86%)
 create mode 100644 test/build/cppcheck

diff --git a/test/build/static_checkers b/test/build/clang_tidy
similarity index 86%
rename from test/build/static_checkers
rename to test/build/clang_tidy
index 34c8dff..0ead9e8 100644
--- a/test/build/static_checkers
+++ b/test/build/clang_tidy
@@ -11,10 +11,7 @@
 # Copyright (c) 2021 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-htools	clang-tidy cppcheck
+htools	clang-tidy
 
 test	Run clang-tidy
 host	make clang-tidy
-
-test	Run cppcheck
-host	make cppcheck
diff --git a/test/build/cppcheck b/test/build/cppcheck
new file mode 100644
index 0000000..05166dd
--- /dev/null
+++ b/test/build/cppcheck
@@ -0,0 +1,17 @@
+# SPDX-License-Identifier: AGPL-3.0-or-later
+#
+# PASST - Plug A Simple Socket Transport
+#  for qemu/UNIX domain socket mode
+#
+# PASTA - Pack A Subtle Tap Abstraction
+#  for network namespace/tap device mode
+#
+# test/build/cppcheck - Run source through cppcheck(1) linter
+#
+# Copyright (c) 2021 Red Hat GmbH
+# Author: Stefano Brivio <sbrivio(a)redhat.com>
+
+htools	cppcheck
+
+test	Run cppcheck
+host	make cppcheck
diff --git a/test/run b/test/run
index 3e5d56b..258a89b 100755
--- a/test/run
+++ b/test/run
@@ -61,7 +61,8 @@ run() {
 
 	setup build
 	test build/all
-	test build/static_checkers
+	test build/cppcheck
+	test build/clang_tidy
 	test distro/debian
 	test distro/fedora
 	test distro/opensuse
-- 
@@ -61,7 +61,8 @@ run() {
 
 	setup build
 	test build/all
-	test build/static_checkers
+	test build/cppcheck
+	test build/clang_tidy
 	test distro/debian
 	test distro/fedora
 	test distro/opensuse
-- 
2.37.2

[PATCH 03/10] test: Remove unused *_XTERM variables

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

The DEMO_XTERM and CI_XTERM variables defined in test/lib/term aren't used
anywhere.  Remove them.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/term | 4 ----
 1 file changed, 4 deletions(-)

diff --git a/test/lib/term b/test/lib/term
index 0f581de..ddabf8d 100755
--- a/test/lib/term
+++ b/test/lib/term
@@ -13,10 +13,6 @@
 # Copyright (c) 2021 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-# Commands of X terminals for "CI" and "demo" runs
-DEMO_XTERM="cool-retro-term --verbose --workdir"
-CI_XTERM="mate-terminal --hide-menubar --profile=passt_ci --working-directory"
-
 STATUS_FILE=
 STATUS_FILE_NTESTS=
 STATUS_FILE_INDEX=0
-- 
@@ -13,10 +13,6 @@
 # Copyright (c) 2021 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-# Commands of X terminals for "CI" and "demo" runs
-DEMO_XTERM="cool-retro-term --verbose --workdir"
-CI_XTERM="mate-terminal --hide-menubar --profile=passt_ci --working-directory"
-
 STATUS_FILE=
 STATUS_FILE_NTESTS=
 STATUS_FILE_INDEX=0
-- 
2.37.2

[PATCH 04/10] test: Ignore video processing temporary files

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 521 bytes --]

When recording tests or demos with asciinema we generate several temporary
files during post-processing.  Add these to the .gitignore file so they're
not accidentally comitted.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/.gitignore | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/test/.gitignore b/test/.gitignore
index f81126e..129ddc0 100644
--- a/test/.gitignore
+++ b/test/.gitignore
@@ -6,3 +6,7 @@ QEMU_EFI.fd
 *.qcow2
 *.raw
 *.raw.xz
+*.uncut
+*.start
+*.stop
+*.js
-- 
@@ -6,3 +6,7 @@ QEMU_EFI.fd
 *.qcow2
 *.raw
 *.raw.xz
+*.uncut
+*.start
+*.stop
+*.js
-- 
2.37.2

[PATCH 05/10] test: Split setup/teardown functions for build and distro tests

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1616 bytes --]

Currently the build tests and distro tests share a common setup function.
That works for now, but changes we want to make will mean they need
slightly different setup, so split the setup functions in preparation.

Currently, neither build nor distro tests have any teardown function.
Again, future changes are going to mean we need to do some teardown, so
create some empty for now teardown functions in preparation.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/setup | 15 +++++++++++++++
 test/run       |  4 ++++
 2 files changed, 19 insertions(+)

diff --git a/test/lib/setup b/test/lib/setup
index 69eb898..d858c71 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -23,6 +23,11 @@ setup_build() {
 	layout_host
 }
 
+# setup_distro() - Set up pane layout for distro tests
+setup_distro() {
+	layout_host
+}
+
 # setup_passt() - Start qemu and passt
 setup_passt() {
 	layout_passt
@@ -281,6 +286,16 @@ setup_two_guests() {
 	pane_status GUEST_2
 }
 
+# teardown_build() - Nothing to do, yet
+teardown_build() {
+    :
+}
+
+# teardown_distro() - Nothing to do, yet
+teardown_distro() {
+    :
+}
+
 # teardown_passt() - Kill qemu, remove passt PID file
 teardown_passt() {
 	tmux send-keys -t ${PANE_GUEST} "C-c"
diff --git a/test/run b/test/run
index 258a89b..efec955 100755
--- a/test/run
+++ b/test/run
@@ -63,10 +63,14 @@ run() {
 	test build/all
 	test build/cppcheck
 	test build/clang_tidy
+	teardown build
+
+	setup distro
 	test distro/debian
 	test distro/fedora
 	test distro/opensuse
 	test distro/ubuntu
+	teardown distro
 
 	setup pasta
 	test ndp/pasta
-- 
@@ -63,10 +63,14 @@ run() {
 	test build/all
 	test build/cppcheck
 	test build/clang_tidy
+	teardown build
+
+	setup distro
 	test distro/debian
 	test distro/fedora
 	test distro/opensuse
 	test distro/ubuntu
+	teardown distro
 
 	setup pasta
 	test ndp/pasta
-- 
2.37.2

[PATCH 06/10] test: Only select a single interface or gateway in tests

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3869 bytes --]

The queries we use in the test scripts to locate the external interface
or gateway can return multiple results.  We get away with this because the
way we parse command output only looks at the last line.  It's not really
correct, though, and improvements to our handling of command output will
mean it breaks.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/dhcp/passt | 4 ++--
 test/dhcp/pasta | 4 ++--
 test/ndp/passt  | 4 ++--
 test/ndp/pasta  | 2 +-
 4 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/test/dhcp/passt b/test/dhcp/passt
index 11e0eb3..37bf6b5 100644
--- a/test/dhcp/passt
+++ b/test/dhcp/passt
@@ -28,7 +28,7 @@ check	[ "__ADDR__" = "__HOST_ADDR__" ]
 
 test	DHCP: route
 gout	GW ip -j -4 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW ip -j -4 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW ip -j -4 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ "__GW__" = "__HOST_GW__" ]
 
 test	DHCP: MTU
@@ -55,7 +55,7 @@ check	[ "__ADDR6__" = "__HOST_ADDR6__" ]
 
 test	DHCPv6: route
 gout	GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ "__GW6__" = "__HOST_GW6__" ]
 
 # Strip interface specifier: interface names might differ between host and guest
diff --git a/test/dhcp/pasta b/test/dhcp/pasta
index 076ec8d..d1e9611 100644
--- a/test/dhcp/pasta
+++ b/test/dhcp/pasta
@@ -26,7 +26,7 @@ check	[ __ADDR__ = __HOST_ADDR__ ]
 
 test	DHCP: route
 nsout	GW ip -j -4 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW ip -j -4 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW ip -j -4 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ __GW__ = __HOST_GW__ ]
 
 test	DHCP: MTU
@@ -42,5 +42,5 @@ check	[ __ADDR6__ = __HOST_ADDR6__ ]
 
 test	DHCPv6: route
 nsout	GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ __GW6__ = __HOST_GW6__ ]
diff --git a/test/ndp/passt b/test/ndp/passt
index 8ef15e7..c73fd4d 100644
--- a/test/ndp/passt
+++ b/test/ndp/passt
@@ -17,7 +17,7 @@ htools	ip jq sipcalc grep cut
 test	Interface name
 gout	IFNAME ip -j link show | jq -rM '.[] | select(.link_type == "ether").ifname'
 guest	ip link set dev __IFNAME__ up && sleep 2
-hout	HOST_IFNAME6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").dev'
+hout	HOST_IFNAME6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").dev] | .[0]'
 check	[ -n "__IFNAME__" ]
 
 test	SLAAC: prefix
@@ -29,5 +29,5 @@ check	[ "__PREFIX6__" = "__HOST_PREFIX6__" ]
 
 test	SLAAC: route
 gout	GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ __GW6__ = __HOST_GW6__ ]
diff --git a/test/ndp/pasta b/test/ndp/pasta
index 28fdd72..d776055 100644
--- a/test/ndp/pasta
+++ b/test/ndp/pasta
@@ -29,5 +29,5 @@ check	[ "__PREFIX6__" = "__HOST_PREFIX6__" ]
 
 test	SLAAC: route
 nsout	GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ __GW6__ = __HOST_GW6__ ]
-- 
@@ -29,5 +29,5 @@ check	[ "__PREFIX6__" = "__HOST_PREFIX6__" ]
 
 test	SLAAC: route
 nsout	GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
-hout	HOST_GW6 ip -j -6 route show|jq -rM '.[] | select(.dst == "default").gateway'
+hout	HOST_GW6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").gateway] | .[0]'
 check	[ __GW6__ = __HOST_GW6__ ]
-- 
2.37.2

[PATCH 07/10] test: Rename slightly misleading "valgrind" tests

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4101 bytes --]

The "valgrind" test cases are designed to pick up errors reported when
passt is running under valgrind.  But what it actually does is just kill
the passt process, then see if it had a non-zero exit code.  That means it
will equally well pick up any other problems which caused passt to exit
with an error status: either something detected within passt or as a result
of passt being killed by an unexpected signal.

The fact that the "valgrind" test is actually responsible for shutting down
the passt process is non-obvious and can lead to problems when selectively
running tests during debugging.

Rename the "valgrind" tests to "shutdown" tests and run it regardless of
whether we're using valgrind or not.  This allows us to remove an ugly
speacial case in the passt_in_ns teardown code.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/setup                          | 2 --
 test/run                                | 5 +++--
 test/{valgrind => shutdown}/passt       | 9 +++++----
 test/{valgrind => shutdown}/passt_in_ns | 9 +++++----
 4 files changed, 13 insertions(+), 12 deletions(-)
 rename test/{valgrind => shutdown}/passt (64%)
 rename test/{valgrind => shutdown}/passt_in_ns (62%)

diff --git a/test/lib/setup b/test/lib/setup
index d858c71..19ddd37 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -317,8 +317,6 @@ teardown_passt_in_ns() {
 	pane_wait GUEST
 	tmux send-keys -t ${PANE_GUEST} "C-d"
 
-	[ ${VALGRIND} -eq 0 ] && tmux send-keys -t ${PANE_PASST} "C-c"
-	[ ${VALGRIND} -eq 0 ] && pane_status GUEST
 	tmux send-keys -t ${PANE_PASST} "C-d"
 
 	pane_wait GUEST
diff --git a/test/run b/test/run
index efec955..f9792bd 100755
--- a/test/run
+++ b/test/run
@@ -84,7 +84,7 @@ run() {
 	test dhcp/passt
 	test tcp/passt
 	test udp/passt
-	test valgrind/passt
+	test shutdown/passt
 	teardown passt
 
 	VALGRIND=1
@@ -94,7 +94,7 @@ run() {
 	test icmp/passt_in_ns
 	test tcp/passt_in_ns
 	test udp/passt_in_ns
-	test valgrind/passt_in_ns
+	test shutdown/passt_in_ns
 	teardown passt_in_ns
 
 	VALGRIND=0
@@ -105,6 +105,7 @@ run() {
 	test perf/passt_udp
 	test perf/pasta_tcp
 	test perf/pasta_udp
+	test shutdown/passt_in_ns
 	teardown passt_in_ns
 
 	setup two_guests
diff --git a/test/valgrind/passt b/test/shutdown/passt
similarity index 64%
rename from test/valgrind/passt
rename to test/shutdown/passt
index f8bcf00..00ced61 100644
--- a/test/valgrind/passt
+++ b/test/shutdown/passt
@@ -6,16 +6,17 @@
 # PASTA - Pack A Subtle Tap Abstraction
 #  for network namespace/tap device mode
 #
-# test/valgrind/passt - Terminate passt and check valgrind exit code
+# test/shutdown/passt - Shut down passt and check exit code (will detect
+#                       valgrind errors amongst others)
 #
 # Copyright (c) 2022 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-test	valgrind: exit code
+test	shutdown: exit code
 
 hout	PASST_PID cat passt.pid
 host	kill __PASST_PID__
 sleep	1
 
-pout	VALGRIND_EXIT echo $?
-check	[ "__VALGRIND_EXIT__" = "0" ]
+pout	EXIT echo $?
+check	[ "__EXIT__" = "0" ]
diff --git a/test/valgrind/passt_in_ns b/test/shutdown/passt_in_ns
similarity index 62%
rename from test/valgrind/passt_in_ns
rename to test/shutdown/passt_in_ns
index d28e251..a427bf4 100644
--- a/test/valgrind/passt_in_ns
+++ b/test/shutdown/passt_in_ns
@@ -6,16 +6,17 @@
 # PASTA - Pack A Subtle Tap Abstraction
 #  for network namespace/tap device mode
 #
-# test/valgrind/passt_in_ns - Terminate passt and check valgrind exit code
+# test/shutdown/passt_in_ns - Shut down passt and check exit code (will detect
+#                             valgrind errors amongst others)
 #
 # Copyright (c) 2022 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-test	valgrind: exit code
+test	shutdown: exit code
 
 nsout	PASST_PID cat passt.pid
 ns	kill __PASST_PID__
 sleep	1
 
-pout	VALGRIND_EXIT echo $?
-check	[ "__VALGRIND_EXIT__" = "0" ]
+pout	EXIT echo $?
+check	[ "__EXIT__" = "0" ]
-- 
@@ -6,16 +6,17 @@
 # PASTA - Pack A Subtle Tap Abstraction
 #  for network namespace/tap device mode
 #
-# test/valgrind/passt_in_ns - Terminate passt and check valgrind exit code
+# test/shutdown/passt_in_ns - Shut down passt and check exit code (will detect
+#                             valgrind errors amongst others)
 #
 # Copyright (c) 2022 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
 
-test	valgrind: exit code
+test	shutdown: exit code
 
 nsout	PASST_PID cat passt.pid
 ns	kill __PASST_PID__
 sleep	1
 
-pout	VALGRIND_EXIT echo $?
-check	[ "__VALGRIND_EXIT__" = "0" ]
+pout	EXIT echo $?
+check	[ "__EXIT__" = "0" ]
-- 
2.37.2

[PATCH 08/10] test: Use shutdown test for pasta

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 2045 bytes --]

For the passt and passt_in_ns tests we have a "shutdown" testcase that
checks for any errors from the passt process we were using (including
valgrind warnings).  Do the same for pasta tests, so that we catch any
error codes from the pasta process.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/setup      | 4 +---
 test/run            | 1 +
 test/shutdown/passt | 4 ++--
 3 files changed, 4 insertions(+), 5 deletions(-)

diff --git a/test/lib/setup b/test/lib/setup
index 19ddd37..450b68a 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -98,7 +98,7 @@ setup_pasta() {
 	[ ${DEBUG} -eq 1 ] && __opts="${__opts} -d"
 	[ ${TRACE} -eq 1 ] && __opts="${__opts} --trace"
 
-	pane_run PASST "./pasta ${__opts} -f -t 10002 -T 10003 -u 10002 -U 10003 ${__target_pid}"
+	pane_run PASST "./pasta ${__opts} -f -t 10002 -T 10003 -u 10002 -U 10003 -P passt.pid ${__target_pid}"
 	sleep 1
 }
 
@@ -305,8 +305,6 @@ teardown_passt() {
 
 # teardown_passt() - Exit namespace, kill pasta process
 teardown_pasta() {
-	tmux send-keys -t ${PANE_PASST} "C-c"
-	pane_wait PASST
 	tmux send-keys -t ${PANE_NS} "C-d"
 	pane_wait NS
 }
diff --git a/test/run b/test/run
index f9792bd..609d91d 100755
--- a/test/run
+++ b/test/run
@@ -77,6 +77,7 @@ run() {
 	test dhcp/pasta
 	test tcp/pasta
 	test udp/pasta
+	test shutdown/passt
 	teardown pasta
 
 	setup passt
diff --git a/test/shutdown/passt b/test/shutdown/passt
index 00ced61..4a7b702 100644
--- a/test/shutdown/passt
+++ b/test/shutdown/passt
@@ -6,8 +6,8 @@
 # PASTA - Pack A Subtle Tap Abstraction
 #  for network namespace/tap device mode
 #
-# test/shutdown/passt - Shut down passt and check exit code (will detect
-#                       valgrind errors amongst others)
+# test/shutdown/passt - Shut down passt (or pasta) and check exit code (will
+#                       detect valgrind errors amongst others)
 #
 # Copyright (c) 2022 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
-- 
@@ -6,8 +6,8 @@
 # PASTA - Pack A Subtle Tap Abstraction
 #  for network namespace/tap device mode
 #
-# test/shutdown/passt - Shut down passt and check exit code (will detect
-#                       valgrind errors amongst others)
+# test/shutdown/passt - Shut down passt (or pasta) and check exit code (will
+#                       detect valgrind errors amongst others)
 #
 # Copyright (c) 2022 Red Hat GmbH
 # Author: Stefano Brivio <sbrivio(a)redhat.com>
-- 
2.37.2

[PATCH 09/10] test: Log debugging output from test script

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

The test scripts run with sh -e, which means they will stop if any commands
return an error.  That's generally desirable, because we won't continue
after things are hopeless due to an earlier step failing.

Unfortunately, the tmux setup we run the script in means it's not obvious
where any error messages related to such a failure will go.  Depending on
exactly where the error occurs they might go to the original terminal
hidden behind tmux, or they might go to a tmux panel that's not visible in
the normal layouts.

To make it easier to find such error message, redirect direct output and
errors from the test script itself to a 'script.log' file in the logs
directory.  When in DEBUG=1 mode, additionaly 'set -x' so we log all the
commands we execute to that file.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/run | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/run b/test/run
index 609d91d..aa9a380 100755
--- a/test/run
+++ b/test/run
@@ -161,6 +161,8 @@ demo() {
 [ "$(basename "${0}")" = "run_demo" ] && DEMO=1
 
 if [ "${1}" = "from_term" ]; then
+	exec > ${LOGDIR}/script.log 2>&1
+	[ ${DEBUG} -eq 1 ] && set -x
 	cd ..
 	if [ ${DEMO} -eq 1 ]; then
 		demo
-- 
@@ -161,6 +161,8 @@ demo() {
 [ "$(basename "${0}")" = "run_demo" ] && DEMO=1
 
 if [ "${1}" = "from_term" ]; then
+	exec > ${LOGDIR}/script.log 2>&1
+	[ ${DEBUG} -eq 1 ] && set -x
 	cd ..
 	if [ ${DEMO} -eq 1 ]; then
 		demo
-- 
2.37.2

[PATCH 10/10] test: Kill qemu by pidfile rather than ^C

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3477 bytes --]

Currently in at least some of the testcases we kill qemu processes we're
done with by issuing a Control-C to the tmux panel it's running in.  That
makes things harder as we try to move towards allowing "headless" testing
without tmux.

So, instead always use an explicit kill on a pid derived from a pidfile
for killing qemu.  Note that we don't need to remove the pidfiles
afterwards, because qemu does that itself when terminated.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/setup | 20 ++++++++++++--------
 1 file changed, 12 insertions(+), 8 deletions(-)

diff --git a/test/lib/setup b/test/lib/setup
index 450b68a..42f03c1 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -60,7 +60,8 @@ setup_passt() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	   \
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"  \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			   \
+		' -pidfile passt_qemu.pid'
 	pane_status GUEST
 
 	# Set things up to reduce problems parsing host command output
@@ -178,7 +179,8 @@ setup_passt_in_ns() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	   \
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=524288" \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			   \
+		' -pidfile passt_in_ns_qemu.pid'
 	pane_status GUEST
 
 	# Set things up to reduce problems parsing host command output
@@ -271,7 +273,8 @@ setup_two_guests() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	     \
 		'virtio-net.napi_tx=1"'					     \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"    \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			     \
+		' -pidfile two_guests_qemu_1.pid'
 	pane_run GUEST_2 './qrap 5 qemu-system-$(uname -m)'                  \
 		' -M accel=kvm:tcg'                                          \
 		' -m '${VMEM}' -cpu host -smp '${VCPUS}                      \
@@ -281,7 +284,8 @@ setup_two_guests() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	     \
 		'virtio-net.napi_tx=1"'					     \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"    \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			     \
+		' -pidfile two_guests_qemu_2.pid'
 	pane_status GUEST_1
 	pane_status GUEST_2
 }
@@ -298,7 +302,7 @@ teardown_distro() {
 
 # teardown_passt() - Kill qemu, remove passt PID file
 teardown_passt() {
-	tmux send-keys -t ${PANE_GUEST} "C-c"
+	kill $(cat passt_qemu.pid)
 	pane_wait GUEST
 	rm passt.pid
 }
@@ -311,7 +315,7 @@ teardown_pasta() {
 
 # teardown_passt_in_ns() - Exit namespace, kill qemu and pasta, remove pid file
 teardown_passt_in_ns() {
-	tmux send-keys -t ${PANE_GUEST} "C-c"
+	kill $(cat passt_in_ns_qemu.pid)
 	pane_wait GUEST
 	tmux send-keys -t ${PANE_GUEST} "C-d"
 
@@ -326,11 +330,11 @@ teardown_passt_in_ns() {
 
 # teardown_two_guests() - Exit namespaces, kill qemu processes, passt and pasta
 teardown_two_guests() {
-	tmux send-keys -t ${PANE_GUEST_1} "C-c"
+	kill $(cat two_guests_qemu_1.pid)
 	pane_wait GUEST_1
 	tmux send-keys -t ${PANE_GUEST_1} "C-d"
 
-	tmux send-keys -t ${PANE_GUEST_2} "C-c"
+	kill $(cat two_guests_qemu_2.pid)
 	pane_wait GUEST_2
 	tmux send-keys -t ${PANE_GUEST_2} "C-d"
 
-- 
@@ -60,7 +60,8 @@ setup_passt() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	   \
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"  \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			   \
+		' -pidfile passt_qemu.pid'
 	pane_status GUEST
 
 	# Set things up to reduce problems parsing host command output
@@ -178,7 +179,8 @@ setup_passt_in_ns() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	   \
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=524288" \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			   \
+		' -pidfile passt_in_ns_qemu.pid'
 	pane_status GUEST
 
 	# Set things up to reduce problems parsing host command output
@@ -271,7 +273,8 @@ setup_two_guests() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	     \
 		'virtio-net.napi_tx=1"'					     \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"    \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			     \
+		' -pidfile two_guests_qemu_1.pid'
 	pane_run GUEST_2 './qrap 5 qemu-system-$(uname -m)'                  \
 		' -M accel=kvm:tcg'                                          \
 		' -m '${VMEM}' -cpu host -smp '${VCPUS}                      \
@@ -281,7 +284,8 @@ setup_two_guests() {
 		' -append "console=ttyS0 mitigations=off apparmor=0 '	     \
 		'virtio-net.napi_tx=1"'					     \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"    \
-		" -netdev socket,fd=5,id=hostnet0"
+		" -netdev socket,fd=5,id=hostnet0"			     \
+		' -pidfile two_guests_qemu_2.pid'
 	pane_status GUEST_1
 	pane_status GUEST_2
 }
@@ -298,7 +302,7 @@ teardown_distro() {
 
 # teardown_passt() - Kill qemu, remove passt PID file
 teardown_passt() {
-	tmux send-keys -t ${PANE_GUEST} "C-c"
+	kill $(cat passt_qemu.pid)
 	pane_wait GUEST
 	rm passt.pid
 }
@@ -311,7 +315,7 @@ teardown_pasta() {
 
 # teardown_passt_in_ns() - Exit namespace, kill qemu and pasta, remove pid file
 teardown_passt_in_ns() {
-	tmux send-keys -t ${PANE_GUEST} "C-c"
+	kill $(cat passt_in_ns_qemu.pid)
 	pane_wait GUEST
 	tmux send-keys -t ${PANE_GUEST} "C-d"
 
@@ -326,11 +330,11 @@ teardown_passt_in_ns() {
 
 # teardown_two_guests() - Exit namespaces, kill qemu processes, passt and pasta
 teardown_two_guests() {
-	tmux send-keys -t ${PANE_GUEST_1} "C-c"
+	kill $(cat two_guests_qemu_1.pid)
 	pane_wait GUEST_1
 	tmux send-keys -t ${PANE_GUEST_1} "C-d"
 
-	tmux send-keys -t ${PANE_GUEST_2} "C-c"
+	kill $(cat two_guests_qemu_2.pid)
 	pane_wait GUEST_2
 	tmux send-keys -t ${PANE_GUEST_2} "C-d"
 
-- 
2.37.2

Re: [PATCH 00/10] passt test fixes, batch 6

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 1305 bytes --]

On Thu, 18 Aug 2022 16:13:48 +1000
David Gibson <david(a)gibson.dropbear.id.au> wrote:

> Here's get another batch of fixes and cleanups for passt/pasta tests.
> Some of these are preliminary fixes before an upcoming series changing
> to use more reliable (less tmux screen-scraping dependent) methods for
> dispatching commands in various contexts.  Others are just
> miscellaneous fixes.
> 
> David Gibson (10):
>   test: Convert distro tests to use socat instead of nc/ncat
>   test: Split cppcheck and clang-tidy tests into different files
>   test: Remove unused *_XTERM variables
>   test: Ignore video processing temporary files
>   test: Split setup/teardown functions for build and distro tests
>   test: Only select a single interface or gateway in tests
>   test: Rename slightly misleading "valgrind" tests
>   test: Use shutdown test for pasta
>   test: Log debugging output from test script
>   test: Kill qemu by pidfile rather than ^C

Applied, thanks! I just needed a minor change in 1/10: socat doesn't
appear to be available on Ubuntu 16.04 ppc64 -- which is not even an
officially supported architecture for that version, but it looked like
the most convenient way to add a big-endian PPC to the mix.

So, well, I added a special case for it, still using netcat-openbsd.

-- 
Stefano