[PATCH 00/12] Tests fixes and clean-ups

[PATCH 00/12] Tests fixes and clean-ups

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 1620 bytes --]

This series, mostly from David, cleans up a number of atrocities in
the test suite, makes it more observable by reporting test failures
whenever a command fails, and a bit more robust by improving filtering
in pane_parse().

Further, the whole script now fails if an unhandled failure occurs. A
few assorted fixes are included as well.

David Gibson (9):
  tests: Remove unused XVFB variable
  tests: Add some debugging output for the test scripts themselves
  tests: Don't use tmux update-environment
  tests: Don't globally set tmux default-shell
  tests: Improve control character filtering in pane_parse
  tests: Don't ignore errors during script
  tests: Add pane_status command to check for success of issued commands
  tests: Simplify *tools commands using pane_status
  tests: Simplify explicit checks for command success

Stefano Brivio (3):
  tests: Update mbuto git URLs
  test/perf/pasta_udp: Drop redundant assignment of ::1 to loopback
    interface
  test/run: Return 0 from run(), exit value already reflects failures

 test/build/static_checkers |  6 ++--
 test/demo/passt            |  2 +-
 test/distro/debian         | 33 +++++++------------
 test/distro/opensuse       | 18 ++++-------
 test/distro/ubuntu         | 15 +++------
 test/icmp/passt_in_ns      |  4 ---
 test/lib/setup             | 59 ++++++++++++++++------------------
 test/lib/term              | 30 ++++++++++++++----
 test/lib/test              | 65 ++++++++++++++++++--------------------
 test/perf/pasta_udp        |  1 -
 test/run                   |  7 ++--
 11 files changed, 108 insertions(+), 132 deletions(-)

-- 
2.35.1

[PATCH 01/12] tests: Update mbuto git URLs

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 1089 bytes --]

The project is now at mbuto.sh, and git transport is enabled.

Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/demo/passt | 2 +-
 test/lib/setup  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/test/demo/passt b/test/demo/passt
index 8838363..cf75c9b 100644
--- a/test/demo/passt
+++ b/test/demo/passt
@@ -54,7 +54,7 @@ nl
 nl
 say	Let's create a small initramfs image for the guest.
 guest	cd __TEMPDIR__
-guest	git clone https://mbuto.lameexcu.se/mbuto
+guest	git clone git://mbuto.sh/mbuto
 guest	cd mbuto
 guest	./mbuto -f passt.img -p passt -c lz4
 sleep	2
diff --git a/test/lib/setup b/test/lib/setup
index 823c706..b954596 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -31,7 +31,7 @@ setup_passt() {
 	layout_passt
 
 	__mbuto_dir="$(mktemp -d)"
-	pane_run GUEST "git -C ${__mbuto_dir} clone https://mbuto.lameexcu.se/mbuto/"
+	pane_run GUEST "git -C ${__mbuto_dir} clone git://mbuto.sh/mbuto"
 	pane_wait GUEST
 
 	pane_run GUEST "${__mbuto_dir}/mbuto/mbuto -p passt -c lz4 -f mbuto.img"
-- 
@@ -31,7 +31,7 @@ setup_passt() {
 	layout_passt
 
 	__mbuto_dir="$(mktemp -d)"
-	pane_run GUEST "git -C ${__mbuto_dir} clone https://mbuto.lameexcu.se/mbuto/"
+	pane_run GUEST "git -C ${__mbuto_dir} clone git://mbuto.sh/mbuto"
 	pane_wait GUEST
 
 	pane_run GUEST "${__mbuto_dir}/mbuto/mbuto -p passt -c lz4 -f mbuto.img"
-- 
2.35.1

[PATCH 02/12] tests: Remove unused XVFB variable

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 660 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

The XVFB variable is initialized at the beginning of test/run then never
used again.  I'm assuming it's a leftover from some ealier iteration.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/run | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/test/run b/test/run
index b2819ef..a8e1c98 100755
--- a/test/run
+++ b/test/run
@@ -35,9 +35,6 @@ DEBUG=${DEBUG:-0}
 # If set, tell passt and pasta to take packet captures
 PCAP=${PCAP:-0}
 
-# If set, run X terminals for CI and demos in Xvfb
-XVFB=${XVFB:-1}
-
 COMMIT="$(git log --oneline --no-decorate -1)"
 
 . lib/util
-- 
@@ -35,9 +35,6 @@ DEBUG=${DEBUG:-0}
 # If set, tell passt and pasta to take packet captures
 PCAP=${PCAP:-0}
 
-# If set, run X terminals for CI and demos in Xvfb
-XVFB=${XVFB:-1}
-
 COMMIT="$(git log --oneline --no-decorate -1)"
 
 . lib/util
-- 
2.35.1

[PATCH 03/12] tests: Add some debugging output for the test scripts themselves

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 786 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

The DEBUG option for tests/run enables debugging options to passt/pasta,
however that doesn't help with debugging the test scripts themselves, which
are fairly fragile.

Extend the DEBUG option so it also prints information on each command in
the test scripts to make it easier to work out where things are falling
over.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/test | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/test/lib/test b/test/lib/test
index 2854191..93c86e8 100755
--- a/test/lib/test
+++ b/test/lib/test
@@ -89,6 +89,8 @@ test_iperf3() {
 test_one_line() {
 	__line="${1}"
 
+	[ ${DEBUG} -eq 1 ] && info DEBUG: "${__line}"
+
 	# Strip comments
 	__line="${__line%%#*}"
 
-- 
@@ -89,6 +89,8 @@ test_iperf3() {
 test_one_line() {
 	__line="${1}"
 
+	[ ${DEBUG} -eq 1 ] && info DEBUG: "${__line}"
+
 	# Strip comments
 	__line="${__line%%#*}"
 
-- 
2.35.1

[PATCH 04/12] tests: Don't use tmux update-environment

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 1825 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

The semantics of tmux's update-environment option are a bit confusing.
It says it means the given variables are copied into the session
environment from the source environment, but it's not entirely clear
what the "source" environment means.

From my experimentation it appeast to be the environment from which
the tmux *server* is launched, not the one issuing the 'new-session'
command.  That makes it pretty much useles, certainly in our case where
we have no way of knowing if the user has pre-existing tmux sessions.

Instead use the new-session -e option to explicitly pass in the variables
we want to propagate.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/term | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/test/lib/term b/test/lib/term
index c9f5be3..42b0020 100755
--- a/test/lib/term
+++ b/test/lib/term
@@ -544,18 +544,19 @@ pause_continue() {
 run_term() {
 	export SHELL="/bin/sh"
 	tmux set-option -g default-shell "/bin/sh"
-	tmux set-option -g update-environment "PCAP DEBUG"
+
+	TMUX="tmux new-session -s passt_test -ePCAP=$PCAP -eDEBUG=$DEBUG"
 
 	if [ ${CI} -eq 1 ]; then
 		printf '\e[8;50;240t'
-		asciinema rec --overwrite ci.uncut -c 'tmux new-session -s passt_test ./ci from_term'
+		asciinema rec --overwrite ci.uncut -c "$TMUX ./ci from_term"
 		video_postprocess ci.uncut
 	elif [ ${DEMO} -eq 1 ]; then
 		printf '\e[8;40;130t'
-		asciinema rec --overwrite demo.uncut -c 'tmux new-session -s passt_test ./run_demo from_term'
+		asciinema rec --overwrite demo.uncut -c "$TMUX ./run_demo from_term"
 		video_postprocess demo.uncut
 	else
-		tmux new-session -s passt_test ./run from_term
+		$TMUX ./run from_term
 	fi
 }
 
-- 
@@ -544,18 +544,19 @@ pause_continue() {
 run_term() {
 	export SHELL="/bin/sh"
 	tmux set-option -g default-shell "/bin/sh"
-	tmux set-option -g update-environment "PCAP DEBUG"
+
+	TMUX="tmux new-session -s passt_test -ePCAP=$PCAP -eDEBUG=$DEBUG"
 
 	if [ ${CI} -eq 1 ]; then
 		printf '\e[8;50;240t'
-		asciinema rec --overwrite ci.uncut -c 'tmux new-session -s passt_test ./ci from_term'
+		asciinema rec --overwrite ci.uncut -c "$TMUX ./ci from_term"
 		video_postprocess ci.uncut
 	elif [ ${DEMO} -eq 1 ]; then
 		printf '\e[8;40;130t'
-		asciinema rec --overwrite demo.uncut -c 'tmux new-session -s passt_test ./run_demo from_term'
+		asciinema rec --overwrite demo.uncut -c "$TMUX ./run_demo from_term"
 		video_postprocess demo.uncut
 	else
-		tmux new-session -s passt_test ./run from_term
+		$TMUX ./run from_term
 	fi
 }
 
-- 
2.35.1

[PATCH 05/12] tests: Don't globally set tmux default-shell

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 2331 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

run_term() uses tmux set-option -g to globally set the default shell.
Unfortunately this hits a chicken-and-egg problem that's common with many
of tmux's session options.  If there isn't already a tmux server running,
we can't connect to set the option.  If we attempt this after starting our
session (and therefore the server), then the session will already be
started with the previous default shell.

In any case it's not a good idea to set tmux global options, since that
might interfere with whatever else the user is doing in tmux.  So, instead
set the default-shell option locally to the session after starting it.  To
make sure we get the right shell for our initial script, explicitly invoke
/bin/sh to interpret it.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/term | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/test/lib/term b/test/lib/term
index 42b0020..7dd9826 100755
--- a/test/lib/term
+++ b/test/lib/term
@@ -542,26 +542,25 @@ pause_continue() {
 
 # run_term() - Start tmux session, running entry point, with recording if needed
 run_term() {
-	export SHELL="/bin/sh"
-	tmux set-option -g default-shell "/bin/sh"
-
 	TMUX="tmux new-session -s passt_test -ePCAP=$PCAP -eDEBUG=$DEBUG"
 
 	if [ ${CI} -eq 1 ]; then
 		printf '\e[8;50;240t'
-		asciinema rec --overwrite ci.uncut -c "$TMUX ./ci from_term"
+		asciinema rec --overwrite ci.uncut -c "$TMUX /bin/sh -c './ci from_term'"
 		video_postprocess ci.uncut
 	elif [ ${DEMO} -eq 1 ]; then
 		printf '\e[8;40;130t'
-		asciinema rec --overwrite demo.uncut -c "$TMUX ./run_demo from_term"
+		asciinema rec --overwrite demo.uncut -c "$TMUX /bin/sh -c './run_demo from_term'"
 		video_postprocess demo.uncut
 	else
-		$TMUX ./run from_term
+		$TMUX /bin/sh -c './run from_term'
 	fi
 }
 
 # term() - Set up terminal window and panes for regular tests or CI
 term() {
+	tmux set-option default-shell "/bin/sh"
+
 	tmux set status-interval 1
 	tmux rename-window ''
 
@@ -596,6 +595,8 @@ term() {
 
 # term_demo() - Set up terminal window and panes for demo
 term_demo() {
+	tmux set-option default-shell "/bin/sh"
+
 	tmux set status-interval 1
 	tmux rename-window ''
 
-- 
@@ -542,26 +542,25 @@ pause_continue() {
 
 # run_term() - Start tmux session, running entry point, with recording if needed
 run_term() {
-	export SHELL="/bin/sh"
-	tmux set-option -g default-shell "/bin/sh"
-
 	TMUX="tmux new-session -s passt_test -ePCAP=$PCAP -eDEBUG=$DEBUG"
 
 	if [ ${CI} -eq 1 ]; then
 		printf '\e[8;50;240t'
-		asciinema rec --overwrite ci.uncut -c "$TMUX ./ci from_term"
+		asciinema rec --overwrite ci.uncut -c "$TMUX /bin/sh -c './ci from_term'"
 		video_postprocess ci.uncut
 	elif [ ${DEMO} -eq 1 ]; then
 		printf '\e[8;40;130t'
-		asciinema rec --overwrite demo.uncut -c "$TMUX ./run_demo from_term"
+		asciinema rec --overwrite demo.uncut -c "$TMUX /bin/sh -c './run_demo from_term'"
 		video_postprocess demo.uncut
 	else
-		$TMUX ./run from_term
+		$TMUX /bin/sh -c './run from_term'
 	fi
 }
 
 # term() - Set up terminal window and panes for regular tests or CI
 term() {
+	tmux set-option default-shell "/bin/sh"
+
 	tmux set status-interval 1
 	tmux rename-window ''
 
@@ -596,6 +595,8 @@ term() {
 
 # term_demo() - Set up terminal window and panes for demo
 term_demo() {
+	tmux set-option default-shell "/bin/sh"
+
 	tmux set status-interval 1
 	tmux rename-window ''
 
-- 
2.35.1

[PATCH 06/12] tests: Improve control character filtering in pane_parse

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 1904 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

pane_parse() attempts to grab the output from the last command issued
into a tmux pane.  It strips out control characters using tr, which in
particular includes the final \r\n.  However, this won't fully strip
out terminal escape sequences.  In particular this breaks if the shell
in the pane is bash, with enable-bracketed-paste enabled in readline.
That issues terminal sequences to enable and disable bracketed paste
mode around every shell prompt.

We can work around this because these escapes are followed by a \r
(CR).  More generally, it seems reasonable to assume that any terminal
shenanigans followed by a CR, but not an LF is supposed to be hidden.
So, use sed to strip everything before the second last CR.  We still
need the tr to remove the final \r\n from the string (sed processes a
line at a time, and doesn't consider the CRLF part of the buffer it's
processing).

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
[sbrivio: modify regexp to keep foo\r\r\n unchanged, by matching on at
 least one CR and a non-CR afterwards: that's the usual output pattern
 for bash on Debian 8 and Debian 9]
Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/lib/term | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/lib/term b/test/lib/term
index 7dd9826..0093f90 100755
--- a/test/lib/term
+++ b/test/lib/term
@@ -221,7 +221,7 @@ pane_wait() {
 pane_parse() {
 	__pane_lc="$(echo "${1}" | tr [A-Z] [a-z])"
 
-	__buf="$(tail -n2 ${LOGDIR}/pane_${__pane_lc}.log | head -n1 | tr -d -c [:print:])"
+	__buf="$(tail -n2 ${LOGDIR}/pane_${__pane_lc}.log | head -n1 | sed 's/^[^\r]*\r\([^\r]\)/\1/' | tr -d '\r\n')"
 
 	[ "# $(eval printf '%s' \"\$${1}_LAST_CMD\")" != "${__buf}" ] && \
 	[ "$ $(eval printf '%s' \"\$${1}_LAST_CMD\")" != "${__buf}" ] &&
-- 
@@ -221,7 +221,7 @@ pane_wait() {
 pane_parse() {
 	__pane_lc="$(echo "${1}" | tr [A-Z] [a-z])"
 
-	__buf="$(tail -n2 ${LOGDIR}/pane_${__pane_lc}.log | head -n1 | tr -d -c [:print:])"
+	__buf="$(tail -n2 ${LOGDIR}/pane_${__pane_lc}.log | head -n1 | sed 's/^[^\r]*\r\([^\r]\)/\1/' | tr -d '\r\n')"
 
 	[ "# $(eval printf '%s' \"\$${1}_LAST_CMD\")" != "${__buf}" ] && \
 	[ "$ $(eval printf '%s' \"\$${1}_LAST_CMD\")" != "${__buf}" ] &&
-- 
2.35.1

[PATCH 07/12] tests: Don't ignore errors during script

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 2506 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

Most commands issued during the testing scripts aren't explicitly checked
for errors.  Therefore, if they fail, the shell will just keep on
executing.  This makes it difficult to figure out where things started
going wrong if things fall over.

Run the whole script with the set -e mode so that it will exit in the case
of any (unchecked) failing command.  To make this work we do need to add
explicit checks / fallbacks for some commands which we expect to fail.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
[sbrivio: use sh -e instead of setting -e later, so that we don't miss
 anything before set -e is issued]
Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/lib/test | 10 +++++-----
 test/run      |  2 +-
 2 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/test/lib/test b/test/lib/test
index 93c86e8..2cfec9a 100755
--- a/test/lib/test
+++ b/test/lib/test
@@ -153,7 +153,7 @@ test_one_line() {
 	"htools")
 		pane_run HOST 'which '"${__arg}"' >/dev/null || echo skip'
 		pane_wait HOST
-		[ "$(pane_parse HOST)" = "skip" ] && TEST_ONE_skip=1
+		[ "$(pane_parse HOST)" = "skip" ] && TEST_ONE_skip=1 || true
 		;;
 	"passt")
 		pane_run PASST "${__arg}"
@@ -194,17 +194,17 @@ test_one_line() {
 	"gtools")
 		pane_run GUEST 'which '"${__arg}"' >/dev/null || echo skip'
 		pane_wait GUEST
-		[ "$(pane_parse GUEST)" = "skip" ] && TEST_ONE_skip=1
+		[ "$(pane_parse GUEST)" = "skip" ] && TEST_ONE_skip=1 || true
 		;;
 	"g1tools")
 		pane_run GUEST_1 'which '"${__arg}"' >/dev/null || echo skip'
 		pane_wait GUEST_1
-		[ "$(pane_parse GUEST_1)" = "skip" ] && TEST_ONE_skip=1
+		[ "$(pane_parse GUEST_1)" = "skip" ] && TEST_ONE_skip=1 || true
 		;;
 	"g2tools")
 		pane_run GUEST_2 'which '"${__arg}"' >/dev/null || echo skip'
 		pane_wait GUEST_2
-		[ "$(pane_parse GUEST_2)" = "skip" ] && TEST_ONE_skip=1
+		[ "$(pane_parse GUEST_2)" = "skip" ] && TEST_ONE_skip=1 || true
 		;;
 	"guest2")
 		pane_run GUEST_2 "${__arg}"
@@ -249,7 +249,7 @@ test_one_line() {
 	"nstools")
 		pane_run NS 'which '"${__arg}"' >/dev/null || echo skip'
 		pane_wait NS
-		[ "$(pane_parse NS)" = "skip" ] && TEST_ONE_skip=1
+		[ "$(pane_parse NS)" = "skip" ] && TEST_ONE_skip=1 || true
 		;;
 	"gout")
 		__varname="${__arg%% *}"
diff --git a/test/run b/test/run
index a8e1c98..737fa81 100755
--- a/test/run
+++ b/test/run
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/sh -e
 #
 # SPDX-License-Identifier: AGPL-3.0-or-later
 #
-- 
@@ -1,4 +1,4 @@
-#!/bin/sh
+#!/bin/sh -e
 #
 # SPDX-License-Identifier: AGPL-3.0-or-later
 #
-- 
2.35.1

[PATCH 08/12] tests: Add pane_status command to check for success of issued commands

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 11060 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

When we use pane_wait to wait for a command issued to a tmux pane to finish
we have no idea whether the command succeeded or not.  This means that the
test scripts can keep running long after the point something vital has
failed, making it difficult to work out what went wrong.

Add a new pane_status command that checks for success of the issued command
and use it in most places instead of pane_wait.  We still need explicit
pane_wait where we're gathering explicit output with pane_parse, because
the way we check the status with 'echo $?' means we lose track of that
output.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
[sbrivio:
 - instead of quitting the script, make a test fail if a command
    issued in a pane fails during a test, and loop until the status code is
    numeric in pane_status() as a hack to make it a bit more robust
 - retain usage of pane_wait() in iperf3 and teardown functions as we
   interrupt iperf3, passt, and pasta, so a non-zero exit code is expected
 - drop bogus ns_{1,2}_wait() calls in teardown_two_guests(), those
   functions were never implemented
 - use pane_status() for "guest" test directives too
]
Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/lib/setup | 57 +++++++++++++++++++++++---------------------------
 test/lib/term  | 14 +++++++++++++
 test/lib/test  | 38 ++++++++++++++++-----------------
 3 files changed, 59 insertions(+), 50 deletions(-)

diff --git a/test/lib/setup b/test/lib/setup
index b954596..6139e8f 100755
--- a/test/lib/setup
+++ b/test/lib/setup
@@ -32,10 +32,10 @@ setup_passt() {
 
 	__mbuto_dir="$(mktemp -d)"
 	pane_run GUEST "git -C ${__mbuto_dir} clone git://mbuto.sh/mbuto"
-	pane_wait GUEST
+	pane_status GUEST
 
 	pane_run GUEST "${__mbuto_dir}/mbuto/mbuto -p passt -c lz4 -f mbuto.img"
-	pane_wait GUEST
+	pane_status GUEST
 
 	rm -rf "${__mbuto_dir}"
 
@@ -52,9 +52,9 @@ setup_passt() {
 	[ ${TRACE} -eq 1 ] && __opts="${__opts} --trace"
 
 	pane_run PASST "make clean"
-	pane_wait PASST
+	pane_status PASST
 	pane_run PASST "make valgrind"
-	pane_wait PASST
+	pane_status PASST
 	pane_run PASST "valgrind --max-stackframe=$((4 * 1024 * 1024)) --trace-children=yes --vgdb=no --error-exitcode=1 --suppressions=test/valgrind.supp ./passt ${__opts} -f -t 10001 -u 10001 -P passt.pid"
 	sleep 5
 
@@ -66,7 +66,7 @@ setup_passt() {
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"  \
 		" -netdev socket,fd=5,id=hostnet0"
-	pane_wait GUEST
+	pane_status GUEST
 }
 
 # setup_pasta() - Create a network and user namespace, connect pasta to it
@@ -80,7 +80,7 @@ setup_pasta() {
 	__tty_pid="$(pane_parse NS)"
 
 	pane_run NS "unshare -rUnpf /bin/sh"
-	pane_wait NS
+	pane_status NS
 
 	pane_run PASST "pstree -p ${__tty_pid} | sed -n 's/.*(\([0-9].*\))$/\1/p'"
 	pane_wait PASST
@@ -131,24 +131,24 @@ setup_passt_in_ns() {
 	pane_run PASST "./pasta ${__opts} -t 10001,10002,10011,10012 -T 10003,10013 -u 10001,10002,10011,10012 -U 10003,10013 -P ${__pid_file}"
 	sleep 1
 	pane_run PASST ''
-	pane_wait PASST
+	pane_status PASST
 	__pasta_pid="$(cat "${__pid_file}")"
 	__ns_pid="$(cat /proc/${__pasta_pid}/task/${__pasta_pid}/children | cut -f1 -d' ')"
 	rm "${__pid_file}"
 
 	pane_run GUEST "nsenter -t ${__ns_pid} -U -n --preserve-credentials"
 	pane_run NS "nsenter -t ${__ns_pid} -U -n -p --preserve-credentials"
-	pane_wait GUEST
-	pane_wait NS
+	pane_status GUEST
+	pane_status NS
 
 	pane_run NS "ip -j li sh | jq -rM '.[] | select(.link_type == \"ether\").ifname'"
 	pane_wait NS
 	__ifname="$(pane_parse NS)"
 	pane_run NS "/sbin/udhcpc -i ${__ifname}"
-	pane_wait NS
+	pane_status NS
 	sleep 2
 	pane_run NS "/sbin/dhclient -6 ${__ifname}"
-	pane_wait NS
+	pane_status NS
 
 	__opts=
 	[ ${PCAP} -eq 1 ] && __opts="${__opts} -p /tmp/passt_in_pasta.pcap"
@@ -157,15 +157,15 @@ setup_passt_in_ns() {
 
 	if [ ${VALGRIND} -eq 1 ]; then
 		pane_run PASST "make clean"
-		pane_wait PASST
+		pane_status PASST
 		pane_run PASST "make valgrind"
-		pane_wait PASST
+		pane_status PASST
 		pane_run PASST "valgrind --max-stackframe=$((4 * 1024 * 1024)) --trace-children=yes --vgdb=no --error-exitcode=1 --suppressions=test/valgrind.supp ./passt -f ${__opts} -t 10001,10011,10021,10031 -u 10001,10011,10021,10031 -P passt.pid"
 	else
 		pane_run PASST "make clean"
-		pane_wait PASST
+		pane_status PASST
 		pane_run PASST "make"
-		pane_wait PASST
+		pane_status PASST
 		pane_run PASST "./passt -f ${__opts} -t 10001,10011,10021,10031 -u 10001,10011,10021,10031 -P passt.pid"
 	fi
 	sleep 5
@@ -178,7 +178,7 @@ setup_passt_in_ns() {
 		'virtio-net.napi_tx=1"'					   \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=524288" \
 		" -netdev socket,fd=5,id=hostnet0"
-	pane_wait GUEST
+	pane_status GUEST
 }
 
 # setup_two_guests() - Set up two namespace, run qemu and passt in both of them
@@ -216,8 +216,8 @@ setup_two_guests() {
 	pane_run PASST_1 ''
 	pane_run PASST_2 ''
 
-	pane_wait PASST_1
-	pane_wait PASST_2
+	pane_status PASST_1
+	pane_status PASST_2
 	__pasta1_pid="$(cat "${__pid1_file}")"
 	__ns1_pid="$(cat /proc/${__pasta1_pid}/task/${__pasta1_pid}/children | cut -f1 -d' ')"
 	rm "${__pid1_file}"
@@ -234,13 +234,13 @@ setup_two_guests() {
 
 	pane_run GUEST_1 "/sbin/udhcpc -i ${__ifname}"
 	pane_run GUEST_2 "/sbin/udhcpc -i ${__ifname}"
-	pane_wait GUEST_1
-	pane_wait GUEST_2
+	pane_status GUEST_1
+	pane_status GUEST_2
 	sleep 2
 	pane_run GUEST_1 "/sbin/dhclient -6 ${__ifname}"
 	pane_run GUEST_2 "/sbin/dhclient -6 ${__ifname}"
-	pane_wait GUEST_1
-	pane_wait GUEST_2
+	pane_status GUEST_1
+	pane_status GUEST_2
 
 	__opts=
 	[ ${PCAP} -eq 1 ] && __opts="${__opts} -p /tmp/passt_1.pcap"
@@ -257,7 +257,7 @@ setup_two_guests() {
 	pane_run PASST_2 "./passt -f ${__opts} -t 10004 -u 10004"
 
 	pane_run GUEST_2 'cp mbuto.img mbuto_2.img'
-	pane_wait GUEST_2
+	pane_status GUEST_2
 
 	pane_run GUEST_1 './qrap 5 kvm -m '${VMEM}' -cpu host -smp '${VCPUS} \
 		' -kernel ' "/boot/vmlinuz-$(uname -r)"			     \
@@ -275,8 +275,8 @@ setup_two_guests() {
 		'virtio-net.napi_tx=1"'					     \
 		" -device virtio-net-pci,netdev=hostnet0,x-txburst=16384"    \
 		" -netdev socket,fd=5,id=hostnet0"
-	pane_wait GUEST_1
-	pane_wait GUEST_2
+	pane_status GUEST_1
+	pane_status GUEST_2
 }
 
 # teardown_passt() - Kill qemu, remove passt PID file
@@ -301,7 +301,7 @@ teardown_passt_in_ns() {
 	tmux send-keys -t ${PANE_GUEST} "C-d"
 
 	[ ${VALGRIND} -eq 0 ] && tmux send-keys -t ${PANE_PASST} "C-c"
-	[ ${VALGRIND} -eq 0 ] && pane_wait GUEST
+	[ ${VALGRIND} -eq 0 ] && pane_status GUEST
 	tmux send-keys -t ${PANE_PASST} "C-d"
 
 	tmux send-keys -t ${PANE_NS} "C-d"
@@ -331,13 +331,8 @@ teardown_two_guests() {
 	pane_wait PASST_2
 	tmux send-keys -t ${PANE_PASST_2} "C-d"
 
-	tmux send-keys -t ${PANE_NS_1} "C-d"
-	tmux send-keys -t ${PANE_NS_2} "C-d"
-
 	pane_wait GUEST_1
 	pane_wait GUEST_2
-	ns_1_wait
-	ns_2_wait
 	pane_wait PASST_1
 	pane_wait PASST_2
 }
diff --git a/test/lib/term b/test/lib/term
index 0093f90..004c2a2 100755
--- a/test/lib/term
+++ b/test/lib/term
@@ -228,6 +228,20 @@ pane_parse() {
 		printf '%s' "${__buf}" || printf '@EMPTY@'
 }
 
+# pane_status() - Wait for command to complete and return its exit status
+# $1:	Pane name
+pane_status() {
+	pane_wait "${1}"
+	__status="$(pane_parse "${1}")"
+	while ! [ "${__status}" -eq "${__status}" ]; do
+		sleep 1
+		pane_run "${1}" 'echo $?'
+		pane_wait "${1}"
+		__status="$(pane_parse "${1}")"
+	done
+	return ${__status}
+}
+
 # status_file_end() - Display and log messages when tests from one file are done
 status_file_end() {
 	[ -z "${STATUS_FILE}" ] && return
diff --git a/test/lib/test b/test/lib/test
index 2cfec9a..4262343 100755
--- a/test/lib/test
+++ b/test/lib/test
@@ -52,7 +52,7 @@ test_iperf3() {
 
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		':> s${i}.bw; done'
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 
 	if [ ${UDP_CLIENT} -eq 0 ]; then
 		pane_run "${__pane}" 'for i in $(seq 0 '${__procs}');'	\
@@ -70,7 +70,7 @@ test_iperf3() {
 			'done'
 	fi
 
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 	sleep 45
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		 'kill -INT $(cat s${i}.pid) 2>/dev/null; done'
@@ -83,7 +83,7 @@ test_iperf3() {
 	pane_parse "${__pane}"
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		'rm -f s${i}.bw; done'
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 }
 
 test_one_line() {
@@ -139,13 +139,13 @@ test_one_line() {
 		;;
 	"host")
 		pane_run HOST "${__arg}"
-		pane_wait HOST
+		pane_status HOST || TEST_ONE_nok=1
 		;;
 	"hostb")
 		pane_run HOST "${__arg}"
 		;;
 	"hostw")
-		pane_wait HOST
+		pane_status HOST || TEST_ONE_nok=1
 		;;
 	"hint")
 		tmux send-keys -t ${PANE_HOST} "C-c"
@@ -157,13 +157,13 @@ test_one_line() {
 		;;
 	"passt")
 		pane_run PASST "${__arg}"
-		pane_wait PASST
+		pane_status PASST || TEST_ONE_nok=1
 		;;
 	"passtb")
 		pane_run PASST "${__arg}"
 		;;
 	"passtw")
-		pane_wait PASST
+		pane_status PASST || TEST_ONE_nok=1
 		;;
 	"pout")
 		__varname="${__arg%% *}"
@@ -173,23 +173,23 @@ test_one_line() {
 		;;
 	"guest")
 		pane_run GUEST "${__arg}"
-		pane_wait GUEST
+		pane_status GUEST || TEST_ONE_nok=1
 		;;
 	"guestb")
 		pane_run GUEST "${__arg}"
 		;;
 	"guestw")
-		pane_wait GUEST
+		pane_status GUEST || TEST_ONE_nok=1
 		;;
 	"guest1")
 		pane_run GUEST_1 "${__arg}"
-		pane_wait GUEST_1
+		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"guest1b")
 		pane_run GUEST_1 "${__arg}"
 		;;
 	"guest1w")
-		pane_wait GUEST_1
+		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"gtools")
 		pane_run GUEST 'which '"${__arg}"' >/dev/null || echo skip'
@@ -208,25 +208,25 @@ test_one_line() {
 		;;
 	"guest2")
 		pane_run GUEST_2 "${__arg}"
-		pane_wait GUEST_2
+		pane_status GUEST_2 || TEST_ONE_nok=1
 		;;
 	"guest2b")
 		pane_run GUEST_2 "${__arg}"
 		;;
 	"guest2w")
-		pane_wait GUEST_2
+		pane_status GUEST_2 || TEST_ONE_nok=1
 		;;
 	"ns")
 		pane_run NS "${__arg}"
-		pane_wait NS
+		pane_status NS || TEST_ONE_nok=1
 		;;
 	"ns1")
 		pane_run NS1 "${__arg}"
-		pane_wait NS1
+		pane_status NS1 || TEST_ONE_nok=1
 		;;
 	"ns2")
 		pane_run NS2 "${__arg}"
-		pane_wait NS2
+		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nsb")
 		pane_run NS "${__arg}"
@@ -238,13 +238,13 @@ test_one_line() {
 		pane_run NS2 "${__arg}"
 		;;
 	"nsw")
-		pane_wait NS
+		pane_status NS || TEST_ONE_nok=1
 		;;
 	"ns1w")
-		pane_wait NS1
+		pane_status NS1 || TEST_ONE_nok=1
 		;;
 	"ns2w")
-		pane_wait NS2
+		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nstools")
 		pane_run NS 'which '"${__arg}"' >/dev/null || echo skip'
-- 
@@ -52,7 +52,7 @@ test_iperf3() {
 
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		':> s${i}.bw; done'
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 
 	if [ ${UDP_CLIENT} -eq 0 ]; then
 		pane_run "${__pane}" 'for i in $(seq 0 '${__procs}');'	\
@@ -70,7 +70,7 @@ test_iperf3() {
 			'done'
 	fi
 
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 	sleep 45
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		 'kill -INT $(cat s${i}.pid) 2>/dev/null; done'
@@ -83,7 +83,7 @@ test_iperf3() {
 	pane_parse "${__pane}"
 	pane_run "${__pane}" 'for i in $(seq 0 '${__procs}'); do'	\
 		'rm -f s${i}.bw; done'
-	pane_wait "${__pane}"
+	pane_status "${__pane}"
 }
 
 test_one_line() {
@@ -139,13 +139,13 @@ test_one_line() {
 		;;
 	"host")
 		pane_run HOST "${__arg}"
-		pane_wait HOST
+		pane_status HOST || TEST_ONE_nok=1
 		;;
 	"hostb")
 		pane_run HOST "${__arg}"
 		;;
 	"hostw")
-		pane_wait HOST
+		pane_status HOST || TEST_ONE_nok=1
 		;;
 	"hint")
 		tmux send-keys -t ${PANE_HOST} "C-c"
@@ -157,13 +157,13 @@ test_one_line() {
 		;;
 	"passt")
 		pane_run PASST "${__arg}"
-		pane_wait PASST
+		pane_status PASST || TEST_ONE_nok=1
 		;;
 	"passtb")
 		pane_run PASST "${__arg}"
 		;;
 	"passtw")
-		pane_wait PASST
+		pane_status PASST || TEST_ONE_nok=1
 		;;
 	"pout")
 		__varname="${__arg%% *}"
@@ -173,23 +173,23 @@ test_one_line() {
 		;;
 	"guest")
 		pane_run GUEST "${__arg}"
-		pane_wait GUEST
+		pane_status GUEST || TEST_ONE_nok=1
 		;;
 	"guestb")
 		pane_run GUEST "${__arg}"
 		;;
 	"guestw")
-		pane_wait GUEST
+		pane_status GUEST || TEST_ONE_nok=1
 		;;
 	"guest1")
 		pane_run GUEST_1 "${__arg}"
-		pane_wait GUEST_1
+		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"guest1b")
 		pane_run GUEST_1 "${__arg}"
 		;;
 	"guest1w")
-		pane_wait GUEST_1
+		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"gtools")
 		pane_run GUEST 'which '"${__arg}"' >/dev/null || echo skip'
@@ -208,25 +208,25 @@ test_one_line() {
 		;;
 	"guest2")
 		pane_run GUEST_2 "${__arg}"
-		pane_wait GUEST_2
+		pane_status GUEST_2 || TEST_ONE_nok=1
 		;;
 	"guest2b")
 		pane_run GUEST_2 "${__arg}"
 		;;
 	"guest2w")
-		pane_wait GUEST_2
+		pane_status GUEST_2 || TEST_ONE_nok=1
 		;;
 	"ns")
 		pane_run NS "${__arg}"
-		pane_wait NS
+		pane_status NS || TEST_ONE_nok=1
 		;;
 	"ns1")
 		pane_run NS1 "${__arg}"
-		pane_wait NS1
+		pane_status NS1 || TEST_ONE_nok=1
 		;;
 	"ns2")
 		pane_run NS2 "${__arg}"
-		pane_wait NS2
+		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nsb")
 		pane_run NS "${__arg}"
@@ -238,13 +238,13 @@ test_one_line() {
 		pane_run NS2 "${__arg}"
 		;;
 	"nsw")
-		pane_wait NS
+		pane_status NS || TEST_ONE_nok=1
 		;;
 	"ns1w")
-		pane_wait NS1
+		pane_status NS1 || TEST_ONE_nok=1
 		;;
 	"ns2w")
-		pane_wait NS2
+		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nstools")
 		pane_run NS 'which '"${__arg}"' >/dev/null || echo skip'
-- 
2.35.1

[PATCH 09/12] tests: Simplify *tools commands using pane_status

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 2230 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

Now that we have pane_status to check the success of commands issued to
panes, we can more easily check for the success of the 'which' commands
used to check tool availability, rather than constructing, then parsing
special "skip" output.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/lib/test | 25 ++++++++++---------------
 1 file changed, 10 insertions(+), 15 deletions(-)

diff --git a/test/lib/test b/test/lib/test
index 4262343..a5583e3 100755
--- a/test/lib/test
+++ b/test/lib/test
@@ -151,9 +151,8 @@ test_one_line() {
 		tmux send-keys -t ${PANE_HOST} "C-c"
 		;;
 	"htools")
-		pane_run HOST 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait HOST
-		[ "$(pane_parse HOST)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run HOST 'which '"${__arg}"' >/dev/null'
+		pane_status HOST || TEST_ONE_skip=1
 		;;
 	"passt")
 		pane_run PASST "${__arg}"
@@ -192,19 +191,16 @@ test_one_line() {
 		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"gtools")
-		pane_run GUEST 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST
-		[ "$(pane_parse GUEST)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST || TEST_ONE_skip=1
 		;;
 	"g1tools")
-		pane_run GUEST_1 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST_1
-		[ "$(pane_parse GUEST_1)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST_1 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST_1 || TEST_ONE_skip=1
 		;;
 	"g2tools")
-		pane_run GUEST_2 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST_2
-		[ "$(pane_parse GUEST_2)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST_2 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST_2 || TEST_ONE_skip=1
 		;;
 	"guest2")
 		pane_run GUEST_2 "${__arg}"
@@ -247,9 +243,8 @@ test_one_line() {
 		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nstools")
-		pane_run NS 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait NS
-		[ "$(pane_parse NS)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run NS 'which '"${__arg}"' >/dev/null'
+		pane_status NS || TEST_ONE_skip=1
 		;;
 	"gout")
 		__varname="${__arg%% *}"
-- 
@@ -151,9 +151,8 @@ test_one_line() {
 		tmux send-keys -t ${PANE_HOST} "C-c"
 		;;
 	"htools")
-		pane_run HOST 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait HOST
-		[ "$(pane_parse HOST)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run HOST 'which '"${__arg}"' >/dev/null'
+		pane_status HOST || TEST_ONE_skip=1
 		;;
 	"passt")
 		pane_run PASST "${__arg}"
@@ -192,19 +191,16 @@ test_one_line() {
 		pane_status GUEST_1 || TEST_ONE_nok=1
 		;;
 	"gtools")
-		pane_run GUEST 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST
-		[ "$(pane_parse GUEST)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST || TEST_ONE_skip=1
 		;;
 	"g1tools")
-		pane_run GUEST_1 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST_1
-		[ "$(pane_parse GUEST_1)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST_1 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST_1 || TEST_ONE_skip=1
 		;;
 	"g2tools")
-		pane_run GUEST_2 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait GUEST_2
-		[ "$(pane_parse GUEST_2)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run GUEST_2 'which '"${__arg}"' >/dev/null'
+		pane_status GUEST_2 || TEST_ONE_skip=1
 		;;
 	"guest2")
 		pane_run GUEST_2 "${__arg}"
@@ -247,9 +243,8 @@ test_one_line() {
 		pane_status NS2 || TEST_ONE_nok=1
 		;;
 	"nstools")
-		pane_run NS 'which '"${__arg}"' >/dev/null || echo skip'
-		pane_wait NS
-		[ "$(pane_parse NS)" = "skip" ] && TEST_ONE_skip=1 || true
+		pane_run NS 'which '"${__arg}"' >/dev/null'
+		pane_status NS || TEST_ONE_skip=1
 		;;
 	"gout")
 		__varname="${__arg%% *}"
-- 
2.35.1

[PATCH 10/12] tests: Simplify explicit checks for command success

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 9236 bytes --]

From: David Gibson <david(a)gibson.dropbear.id.au>

A number of individual test cases use '*out' commands to check for success
of specific commands they've issued.  Now that the test harness is testing
for success of all issued commands as a matter of course, we no longer need
to do this.

Signed-off-by: David Gibson <david(a)gibson.dropbear.id.au>
---
 test/build/static_checkers |  6 ++----
 test/distro/debian         | 33 +++++++++++----------------------
 test/distro/opensuse       | 18 ++++++------------
 test/distro/ubuntu         | 15 +++++----------
 test/icmp/passt_in_ns      |  4 ----
 5 files changed, 24 insertions(+), 52 deletions(-)

diff --git a/test/build/static_checkers b/test/build/static_checkers
index d18dea8..34c8dff 100644
--- a/test/build/static_checkers
+++ b/test/build/static_checkers
@@ -14,9 +14,7 @@
 htools	clang-tidy cppcheck
 
 test	Run clang-tidy
-hout	RET make clang-tidy; echo $?
-check	[ __RET__ -eq 0 ]
+host	make clang-tidy
 
 test	Run cppcheck
-hout	RET make cppcheck; echo $?
-check	[ __RET__ -eq 0 ]
+host	make cppcheck
diff --git a/test/distro/debian b/test/distro/debian
index 344f453..f748dea 100644
--- a/test/distro/debian
+++ b/test/distro/debian
@@ -61,8 +61,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make
 
 # TODO: pasta test skipped for the moment: clone() as called by NS_CALL hangs
 # with wrapper provided by glibc 2.19, probably wrong argument order.
@@ -89,8 +88,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 host	sysctl -w kernel.unprivileged_userns_clone=1
 distro_quick_pasta_test
@@ -114,8 +112,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 host	sysctl -w kernel.unprivileged_userns_clone=1
 distro_quick_pasta_test
@@ -146,8 +143,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 host	sysctl -w kernel.unprivileged_userns_clone=1
 distro_quick_pasta_test
@@ -179,8 +175,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 host	sysctl -w kernel.unprivileged_userns_clone=1
 distro_quick_pasta_test
@@ -204,8 +199,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -234,8 +228,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -265,8 +258,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -289,8 +281,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -311,8 +302,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -334,8 +324,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
diff --git a/test/distro/opensuse b/test/distro/opensuse
index a2c0944..39f059a 100644
--- a/test/distro/opensuse
+++ b/test/distro/opensuse
@@ -60,8 +60,7 @@ host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -89,8 +88,7 @@ host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -117,8 +115,7 @@ host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i eth0
 host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -143,8 +140,7 @@ sleep	10
 host	zypper install -y gcc make netcat-openbsd; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -172,8 +168,7 @@ sleep	10
 host	zypper install -y gcc make netcat-openbsd; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -200,8 +195,7 @@ host	echo "DNSSERVERS='__DNS6__'" | netconfig modify -s dns_resolver -i ens2
 host	for i in $(seq 1 10); do zypper install -y gcc make netcat-openbsd && break; done; echo
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
diff --git a/test/distro/ubuntu b/test/distro/ubuntu
index 781daab..3ce156d 100644
--- a/test/distro/ubuntu
+++ b/test/distro/ubuntu
@@ -62,8 +62,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits" make
 
 # TODO: pasta test skipped for the moment: clone() as called by NS_CALL hangs
 # with wrapper provided by glibc 2.19, probably wrong argument order.
@@ -96,8 +95,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
 
 # TODO: pasta test skipped for the moment: clone() as called by NS_CALL hangs
 # with wrapper provided by glibc 2.19, probably wrong argument order.
@@ -127,8 +125,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Wno-missing-field-initializers -Wno-missing-braces -Wno-type-limits -Wno-sign-compare" make
 
 # TODO: pasta test skipped for the moment: clone() as called by NS_CALL hangs
 # with wrapper provided by glibc 2.19, probably wrong argument order.
@@ -160,8 +157,7 @@ host	dhclient
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 distro_quick_pasta_test
 
@@ -195,8 +191,7 @@ host	apt-get update
 host	apt-get -y install make gcc netcat-openbsd
 
 host	make clean
-hout	RET CFLAGS="-Werror" make; echo $?
-check	[ __RET__ -eq 0 ]
+host	CFLAGS="-Werror" make
 
 host	export SHELL="/bin/dash"
 host	dash
diff --git a/test/icmp/passt_in_ns b/test/icmp/passt_in_ns
index be5be29..38a9a4f 100644
--- a/test/icmp/passt_in_ns
+++ b/test/icmp/passt_in_ns
@@ -24,14 +24,10 @@ test	ICMP echo: guest to ns
 nsout	IFNAME_NS ip -j li sh | jq -rM '.[] | select(.link_type == "ether").ifname'
 ns	ip addr add 192.0.2.1/32 dev __IFNAME_NS__
 guest	ping -c1 -w1 192.0.2.1
-gout	RET echo $?
 ns	ip addr del 192.0.2.1/32 dev __IFNAME_NS__
-check	[ __RET__ -eq 0 ]
 
 test	ICMPv6 echo: guest to ns
 ns	ip addr add 2001:db8::1 dev __IFNAME_NS__ && sleep 2 # DAD
 gout	IFNAME ip -j li sh | jq -rM '.[] | select(.link_type == "ether").ifname'
 guest	ping -c1 -w1 2001:db8::1
-gout	RET echo $?
 ns	ip addr del 2001:db8::1 dev __IFNAME_NS__
-check	[ __RET__ -eq 0 ]
-- 
2.35.1

[PATCH 11/12] test/perf/pasta_udp: Drop redundant assignment of ::1 to loopback interface

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

There are a few occurrences of this assignment, which are needed to
re-add ::1 as loopback address after the MTU has been increased
back from a value below 1280 bytes.

This one, however, is redundant, and causes an error in the
execution.

Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/perf/pasta_udp | 1 -
 1 file changed, 1 deletion(-)

diff --git a/test/perf/pasta_udp b/test/perf/pasta_udp
index 39a6e24..8b68b78 100644
--- a/test/perf/pasta_udp
+++ b/test/perf/pasta_udp
@@ -62,7 +62,6 @@ lat	__LAT__ 200 150
 
 tr	UDP throughput over IPv4: ns to host
 ns	ip link set dev lo mtu 1500
-ns	ip addr add ::1 dev lo
 iperf3c	ns 127.0.0.1 100${i}3 __THREADS__ __OPTS__ -b 3G
 iperf3s	BW host 100${i}3 __THREADS__
 bw	__BW__ 1.0 1.5
-- 
@@ -62,7 +62,6 @@ lat	__LAT__ 200 150
 
 tr	UDP throughput over IPv4: ns to host
 ns	ip link set dev lo mtu 1500
-ns	ip addr add ::1 dev lo
 iperf3c	ns 127.0.0.1 100${i}3 __THREADS__ __OPTS__ -b 3G
 iperf3s	BW host 100${i}3 __THREADS__
 bw	__BW__ 1.0 1.5
-- 
2.35.1

[PATCH 12/12] test/run: Return 0 from run(), exit value already reflects failures

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 651 bytes --]

There's no need to return non-zero if there have been failures in
run(), because the exit value is already determined from the number
of failures reported in the log file.

Return zero, so that this doesn't cause the script to fail, given we
now run it with -e.

Signed-off-by: Stefano Brivio <sbrivio(a)redhat.com>
---
 test/run | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/test/run b/test/run
index 737fa81..d9e5107 100755
--- a/test/run
+++ b/test/run
@@ -110,7 +110,7 @@ run() {
 		"Interrupted, press any key to quit"		\
 		9
 
-	return ${STATUS_FAIL}
+	return 0
 }
 
 # demo() - Simpler path for demo purposes
-- 
@@ -110,7 +110,7 @@ run() {
 		"Interrupted, press any key to quit"		\
 		9
 
-	return ${STATUS_FAIL}
+	return 0
 }
 
 # demo() - Simpler path for demo purposes
-- 
2.35.1

Re: [PATCH 00/12] Tests fixes and clean-ups

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 706 bytes --]

On Tue, May 17, 2022 at 02:36:02PM +0200, Stefano Brivio wrote:
> This series, mostly from David, cleans up a number of atrocities in
> the test suite, makes it more observable by reporting test failures
> whenever a command fails, and a bit more robust by improving filtering
> in pane_parse().
> 
> Further, the whole script now fails if an unhandled failure occurs. A
> few assorted fixes are included as well.

Unsurprisingly, these all LGTM.  I notice they're not in the main git
tree yet; is that intentional?

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 00/12] Tests fixes and clean-ups

[Stefano Brivio]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

On Wed, 18 May 2022 13:17:31 +1000
David Gibson <david(a)gibson.dropbear.id.au> wrote:

> On Tue, May 17, 2022 at 02:36:02PM +0200, Stefano Brivio wrote:
> > This series, mostly from David, cleans up a number of atrocities in
> > the test suite, makes it more observable by reporting test failures
> > whenever a command fails, and a bit more robust by improving filtering
> > in pane_parse().
> > 
> > Further, the whole script now fails if an unhandled failure occurs. A
> > few assorted fixes are included as well.  
> 
> Unsurprisingly, these all LGTM.  I notice they're not in the main git
> tree yet; is that intentional?

Yes, I post patches to the list and leave a bit of time in case anybody
wants to review or comment. Also, the pre-push hook runs CI and demos
so that takes a bit as well.

-- 
Stefano