From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=TzoXQb5X; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id C086C5A0271 for ; Tue, 05 Aug 2025 09:52:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1754380368; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=qDKXg6LgthfoSkHXKa4/YVAl+IffIlTzGkpJpaanDLA=; b=TzoXQb5XImMnADCibTn+tp5Q6U9GdwgUsK1xvHl2pV264At573zEZ+/OSXir9chSsj+SY6 8/i7lpH51WvjBsWmKiko/tETQkBMjr2/tID3Nr+ZoSqaC4lu4Y0Xp22SPNjU5/oafMhNza gubteGQIN+4VGb1D2zz/me7CvYpZUSQ= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-463-hYS_RMisMpSP2UjBDorU8A-1; Tue, 05 Aug 2025 03:52:47 -0400 X-MC-Unique: hYS_RMisMpSP2UjBDorU8A-1 X-Mimecast-MFC-AGG-ID: hYS_RMisMpSP2UjBDorU8A_1754380366 Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-45359bfe631so21174215e9.0 for ; Tue, 05 Aug 2025 00:52:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1754380365; x=1754985165; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=qDKXg6LgthfoSkHXKa4/YVAl+IffIlTzGkpJpaanDLA=; b=ofrJl21FJjwU3+jJKWteoZDydf+yKbSdtsbRhNTSFAg/Cpzmy1dgK7Uli2J47hQ3DS mu7Op4LKjZTF1+bc6HBak4EEEGsAU/zERIA6BUt2+ASf11sh7M1clP1RhsyjBUo2SY0B BSt/pj+7tyDC6fRQiGsE/ofQizACAvPHM+ObyU0/LgVhSs8bJfN0a31kMirjBiLm4ltY ey1lP5eANEA3LlPrH8vJ9vxLy9AzrN/XytTBbts35yMrq3F7pFGdcwzUK7Z4A/5Hj04w z6gMz7b17ZpkOLR8LdXk3CnHdcdfuEz+PJTtWFZrlDwPUbuHsMqv6lr555NR20bQhFQc I5jw== X-Gm-Message-State: AOJu0Yy4WguSxgbYpDCP7kPLlIE9iEgYLNbZOWNkuFaVppXKrIUDNDBQ XSi63NbXfq5fNzublAmMBvwVIl7+C+hckDAgdNmDPXULITkqcbVAS5i3vNyr5lrjaAkCqvdS9Wx oRH6t+mhEeTKPJ8hReZRXP+SdOEzCG89NRJ7gbN1veRnYEZ/U4n8yc3yOvdHjwQ== X-Gm-Gg: ASbGnctQ8LpETCtPWmkjIZrNidjq0sSz8UD43fyT00WRL2/cyDRVfpPSwwIw2vu4Lv6 Q2K4nGumMR1+F0hliShdn7VWnqWB14MgojG/A514yvR0PY3S9+HlbbG51OPUrkPk0LcCvmCIa/j 9OyH3bps0uHcfI++Vw7zU24MVtcOg0AF/7BEdcMrhR5+TTa8gw1BdyF+i289/MchRLMgY1nDx7v s6afY1TTHR1VMBGSTyd+/z6wSh7dRKlGb56LWWZbMGM9nUkcju5hkyYdyneNK5JNC6CRHXlAdyk T2c/iIO+0kO/dbydz6+TIzq8r9DCZdglOa4VaN1x8+UDdgVytnow02S3j+0x8kIw8He8 X-Received: by 2002:a5d:5d0c:0:b0:3b7:8735:9469 with SMTP id ffacd0b85a97d-3b8d9469fcfmr8385495f8f.10.1754380365115; Tue, 05 Aug 2025 00:52:45 -0700 (PDT) X-Google-Smtp-Source: AGHT+IE8ru5FkPXy31xeZT6U2GsiLYnyEbuHozIr1Uwe7unuervdgoEvt6JaTZJM8WZedRBKmuRtvA== X-Received: by 2002:a5d:5d0c:0:b0:3b7:8735:9469 with SMTP id ffacd0b85a97d-3b8d9469fcfmr8385466f8f.10.1754380364365; Tue, 05 Aug 2025 00:52:44 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3b79c4a6f62sm17817453f8f.73.2025.08.05.00.52.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 05 Aug 2025 00:52:43 -0700 (PDT) Date: Tue, 5 Aug 2025 09:52:41 +0200 From: Stefano Brivio To: David Gibson Subject: Re: Exeter testing, next steps Message-ID: <20250805095241.6cb715d5@elisabeth> In-Reply-To: References: Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Y5uJqi1gp1OuFZ_VtY9isyN8Hf0Xa3h4B90UlRGiUfc_1754380366 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: UHBLKHS5OTYROHXQPBZMIE4W4X66YW4D X-Message-ID-Hash: UHBLKHS5OTYROHXQPBZMIE4W4X66YW4D X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, 5 Aug 2025 16:09:34 +1000 David Gibson wrote: > Today I finally had a good look at what the next steps need to be if > we are to use exeter based test "for real" in the passt repo. \o/ > I already have draft series which move a bunch of the simpler tests to > exeter. But in order to actually use these we need some sort of > runner. We're not tied to that runner, we can easily change - that's > the whole point of exeter - but we need to have something. I guess as an even earlier first step we could actually have a hardcoded shell script similar to today's test/run... or am I misrepresenting the problem? > AIUI, Stefano is not happy with the idea of using either Avocado or > Meson as the default which were the runners I initially focused on for > exeter. I'm concerned about compatibility. I don't actually see a problem with Meson *as a test runner* as it's widely packaged, but as a build system, it's arguably less compatible and more complicated than Make is (and we don't need all those features to build passt). > I now suggest two more options: > > 1) Add a tool as part of exeter that will generate a BATS test script > from an exeter test program. So you'd do something like: > > $ exetool --bats > foo.bats > $ bats foo.bats > > This should be pretty easy to do, it's basically what I already have > for Avocado support, indeed a little easier. For reasons internal to > exeter, Python is the obvious choice to implement exetool, but I could > do it in shell if you really don't like that. As long as it can reasonably run on all the distributions where we might want to run tests, I don't see an issue with Python. It depends a bit on what the required modules are, I would say. > 2) Hand-roll a minimal exeter runner as part of passt's existing test > scripts. I'm thinking you could essentially point our test stuff > at an exeter program as an alternative to pointing it at a file > with the existing test DSL. > > This is more work, but still not too bad. It has the advantage of > not adding another dependency, and means we could count exeter > results along with our existing test results in the final summary. It also has the advantage of being conceptually simpler in that it avoids one additional step and one additional "language" (Bats). > However, we'd lose parallel execution and filtering. Aren't those sort of trivial once you have a "real" programming language? > Stefano, would you be willing to merge patches which add some basic > exeter tests using one of these approaches? This would probably just > be static checked and build tests at this point, as a > proof-of-concept. Sure! As long as current tests keep working, of course. > If we can start introducing some exeter tests, the next step would be > to work on the support library stuff for constructing more complex > network environments from namespaces. I have draft series with this > as well, but I was looking at splitting it into another mini-project > (tentative name "sinte" - Simulated Inter Network Test Environment). Neat! That sounds like the juicy part and surely one part we really miss at this point. I would go as far as proposing PESTO (playground environment simplifies test orchestration / spurs test opportunities) but "sinte" sounds good to me as well. -- Stefano