From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=R1PadN19; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 2890B5A0278 for ; Fri, 05 Sep 2025 01:14:36 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1757027675; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/4rGxxtdW+w+HJNFYsUZGrOzt30KbFyFAi7mB33rk/Q=; b=R1PadN19PQR6rlrOCgGsRINRVnV2e2/rApz1nGP7WqO94o6R6+5lHVxTdVl53S1a4t/Tti JRku4noizw9oiJDOS883fthdUW0rVSgdEu+eBdRJQZt3BiiL5CZrs9C1cmWxfnTjd8kINI OVTgS0oPQzQqDf0JE6rs+0mF3ewLIOI= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-222-snCrdBxFM7SwrgLCk3XPdw-1; Thu, 04 Sep 2025 19:14:33 -0400 X-MC-Unique: snCrdBxFM7SwrgLCk3XPdw-1 X-Mimecast-MFC-AGG-ID: snCrdBxFM7SwrgLCk3XPdw_1757027672 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-45b9912a07dso8242745e9.3 for ; Thu, 04 Sep 2025 16:14:33 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1757027671; x=1757632471; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/4rGxxtdW+w+HJNFYsUZGrOzt30KbFyFAi7mB33rk/Q=; b=qMNV0XNzQYRdC09P89QzkLUdFmJHhmYFqF0zGZ+LU/qJ7UAKA9kELzk5Dix4AxbSs4 rJGkro7IGQ0Q4jpJS1xtylIZsl+cbBkFT+5uHLu3l4Z4jcwuqthv14TixP+g+l2GHdV5 mNORML39AHn9rB/TsMrDNuZjCxXccRwQkeSPpnsbX68H36f/4txQRKY+9yreTUXfpo7W fHo36FWZ81c95MlYKTikQGanl9pGG7012QKUN1B18Bj4DoNYpVERBJQlUziqNO6Z7VVj K8B6BKfyPGHDPcVOFyEU87nCpzgB8crtCyADvrjGBlCkNlhMdxYyLyWdKKVs0UKvZGwH /2hA== X-Gm-Message-State: AOJu0Yy5kEecRuRqKDYXB2vfzRM0bWP247qrVc7V1TWMh7jqgCkelfdw EXD/qrNjpzIemdvRxnmS1S6iN6V38BNhqYpVOHjkhLOWknG9dSpCYlQK1Oeh3gzwqJJ4BA8oXgQ IZL5hztLQORg8+SvJQbFgM8PcG61HcphI3Aqg3S8KmLIpsflcZ9w185cn4k+gPA== X-Gm-Gg: ASbGncuE8AppSyL6PQubMAmtzX/Us3BW5529YsWT9Rt6zF2+JXiBztYZMIaumJJXaFK ghJcdg0LCq+pBiqyDUPiI4Ur/2wiBlfGdR/JxXvModxGrsNa8ri5hSOnWxkjrOWEDFIja2HzQRO 1wm+DFnLpjHlhsxP2KS/mpqJMg/eLnIg8vFA1lrdIxldnxK5qTB9vtumsjGXtshbMqfAmoRziX+ wsNN2jOS/mVv5bYlDh0kE3r5LumUT8SgPuF5iXstR+WNGmJK+XPCIx6RVX9u9ACZfpuplVNnHRj cZTXW0M73ORwatsh/fE4nO2sa2VxYdo/EuZmQwhiRBacEJlw3wY= X-Received: by 2002:a05:600c:3ba3:b0:45c:b66d:352c with SMTP id 5b1f17b1804b1-45cb66d3816mr71777515e9.5.1757027671521; Thu, 04 Sep 2025 16:14:31 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFbd9jigHxBSXEVzEjUZqTfjrnazo1XnjEUktEEdEUhnrQY+w8qeML0Y0ghNRnBEvxkXKcLxg== X-Received: by 2002:a05:600c:3ba3:b0:45c:b66d:352c with SMTP id 5b1f17b1804b1-45cb66d3816mr71777365e9.5.1757027671112; Thu, 04 Sep 2025 16:14:31 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-45dcfc414f4sm30061335e9.0.2025.09.04.16.14.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 04 Sep 2025 16:14:30 -0700 (PDT) Date: Fri, 5 Sep 2025 01:14:28 +0200 From: Stefano Brivio To: David Gibson Subject: Re: [PATCH v6 0/4] New proof-of-concept based exeter tests Message-ID: <20250905011428.08542374@elisabeth> In-Reply-To: References: <20250901042515.138861-1-david@gibson.dropbear.id.au> <20250903231435.11a1bdab@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 0mBaEXoJEIEgHfmUrLfqtfse4ipJxtGFsDjTUXQf_MU_1757027672 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: DNIGJAODAX367LQC7CJ76HEKFAQQK5U5 X-Message-ID-Hash: DNIGJAODAX367LQC7CJ76HEKFAQQK5U5 X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 4 Sep 2025 12:42:57 +1000 David Gibson wrote: > On Wed, Sep 03, 2025 at 11:14:35PM +0200, Stefano Brivio wrote: > > On Mon, 1 Sep 2025 14:25:11 +1000 > > David Gibson wrote: > > > > > Here's a new approach to building passt tests with exeter. This new > > > one no longer uses Avocado in the default case, although it would > > > still be possible to manually run the exeter based tests with Avocado. > > > > > > Here's another draft of my work on testing passt with Avocado and the > > > exeter library I recently created. It includes Cleber's patch adding > > > some basic Avocado tests and builds on that. > > > > > > For now this only does simple tests, to show how the integration could > > > work. It adds some new trivial "smoke tests" and converts the linter > > > and build checks to exeter. More complex tests will require building > > > the sinte/pesto library we've discussed. A lot of the work for that > > > already exists in my earlier exeter test series, but it will need some > > > rework to split it into a separate component. > > > > > > v6: > > > * Use exeter's new metadata support to print nicer test names > > > > Thanks, it looks much more readable now. > > > > And to me it looks ready to merge, but I hit something during testing > > that I'm not quite sure how to solve, assuming it can be considered an > > issue at all. > > > > Initially, I hadn't upgraded my local copy of exeter, so even smoke > > tests would fail, until it occurred to me that of course I needed to > > drop the 'exeter' directory and 'make exeter' again. It's an issue we > > conceptually already have with mbuto (even though it didn't get > > breaking changes for months now) and with Podman to some extent. > > Yes. > > > With exeter, I guess we're going to hit that kind of issue pretty > > frequently at least in the near future. So I was wondering: should we > > enforce some form of up-to-date check from test/Makefile? > > Good point. We actually already have a mechanism for this: the pull-% > targets, which we already use for mbuto and podman. I pull a > dependency on this for the "make bats" target, but not for regular > "make assets" so we didn't update exeter. Ah, right, of course. That actually looks good enough to me. > I'll respin for this, because I have one other trivial change to make. > > It does occur to me that there's another version of the problem too. > Once again we already conceptually have it with mbuto and podman, but > it will be much worse with exeter due to pace of development: > > If (for debugging) we check out an old version of the passt source > tree, it will still pull the latest version of exeter, which it might > not be compatible with[0]. The obvious solution - but you might not > like it much - is git submodules. They (kind of rightly) have a > reputation as being a pain to work with, but they do solve exactly > this problem - qemu uses them heavily and pretty successfully for > this. To me it looks like a solution adding permanent complexity to solve a problem that's, in practice, temporary. Once exeter is stable enough, we'll just require a recent version. And before that day, I doubt we'll want to debug / bisect issues by running any of the exeter-based tests. Every time I'm debugging a possible regression I come up with some stand-alone reproducer or manual test. If the test suite had caught the regression at a given point, then we wouldn't have hit the regression in the first place. > > I'm personally fine without it and, given that I plan to update > > test/README.md soon anyway ('make' under test/ is not mentioned at > > all), I can mention this kind of problem as well, so it shouldn't be a > > big deal for others. But I wanted to know if you have thoughts or > > proposals on the matter, before applying this. > > > > -- > > Stefano > > > > [0] At some point, I do want to stabilise the exeter interfaces, and > be concerned with backwards compatibility, which would largely fix > this. However, at present I'm still finding stuff that wants to > change at the exeter level often enough that I think the effort of > maintaining compatibility would be prohibitive. Right, I wouldn't care, I really can't see any practical need. In practice, the kind of breakages we have right now are something much more basic: outdated test/README.md and broken test/Makefile which I couldn't find a moment to take care of, yet. And I still need to killall -9 nstool every time. -- Stefano