From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202608 header.b=iBl4Jibv; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id 2C4255A0265 for ; Fri, 14 Aug 2026 12:03:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202608; t=1786701786; bh=Xb+585u3CLMOdomPtjJwodyGhVpUCYESKEtSbRn1MiE=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=iBl4JibvdEwiPdbiGXjAAYbaUHvBv9gtAUpplbvRBuSaJDNzQXDuf3mV8sbxSGcBV qw2wd6OxyNv6xZnv/zc1NmJR6qxjD7jnmPGESJQaiSdZWBEfVGOFWHYUzYVtdNP/p+ 4emNckTM6PKIeFrau+KYbD6Lm9Tk7R7MKIoIR+FoQV976PV2ldKb14RKYF1jjW4ycs astdRH233H/0EYkTv//+3UQnQaIV9RG9hADgWeWnqFrq3BKl/jqUP/odj2aTDBYnz6 1HhJ40ESRMEHrsfmuvrdcssecAr7bW4wT3JAL2wUqP4jaL5ma+M5+B6UmoYAISGuSi mqSN2eKKkBmWQ== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4hLyTk2NFmz4w2B; Fri, 14 Aug 2026 20:03:06 +1000 (AEST) Date: Fri, 14 Aug 2026 20:02:46 +1000 From: David Gibson To: Stefano Brivio Subject: Re: [PATCH 5/5] fuzz: Add test server for bidirectional protocol fuzzing Message-ID: References: <20260812072630.3235261-1-anskuma@redhat.com> <20260812072630.3235261-6-anskuma@redhat.com> <20260813095323.4c2719bd@elisabeth> <20260814093558.41626ef3@elisabeth> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="rrJNNN2oihKtkqdn" Content-Disposition: inline In-Reply-To: <20260814093558.41626ef3@elisabeth> Message-ID-Hash: CUW46Q452NYSLSQUDH5BHKCFLNWJL3T7 X-Message-ID-Hash: CUW46Q452NYSLSQUDH5BHKCFLNWJL3T7 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Anshu Kumari , passt-dev@passt.top, aerosound161@gmail.com, abdobngad@gmail.com, lvivier@redhat.com X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --rrJNNN2oihKtkqdn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Aug 14, 2026 at 09:35:59AM +0200, Stefano Brivio wrote: > On Fri, 14 Aug 2026 15:40:35 +1000 > David Gibson wrote: >=20 > > On Thu, Aug 13, 2026 at 09:53:24AM +0200, Stefano Brivio wrote: > > > On Wed, 12 Aug 2026 12:56:28 +0530 > > > Anshu Kumari wrote: > > > =20 > > > > Add fuzz-server that acts as passt's network peer > > > > during fuzzing. =20 > > >=20 > > > To me, this part makes sense. But this one: > > > =20 > > > > It connects to passt's UNIX socket =20 > > >=20 > > > much less, while: > > > =20 > > > > and listens on 127.0.0.1:9999 for TCP connections. =20 > > >=20 > > > this is the part that I expected instead. Otherwise it's not just > > > passt's network peer, it's the guest as well. > > > =20 > > > > UNIX socket path: responds to ARP requests and TCP SYNs with > > > > stateless replies (swapped addresses, fixed ISN). Responses > > > > are XOR'd with AFL++ shared memory data so the fuzzer can > > > > mutate server behavior. =20 > > >=20 > > > This looks rather complicated to me. =20 > >=20 > > It does. > >=20 > > > The approach I was suggesting with a test server is the following: > > >=20 > > >=20 > > > ,- exchanges guest-side data with ------------. > > > | ,---------|---------. > > > | ,--| passt | > > > | / '-.---------------^-' > > > ,---|---. / | connect(), | accept(), > > > | AFL++ |-- shares memory with ---| | send data, | reply with > > > '---|---' \ | etc. | data, etc. > > > | \ ,-v---------------'-. > > > | '--| test server | > > > | '---------|---------' > > > '- exchanges host-side data with -------------' > > >=20 > > > ...at least in its basic form. Eventually, the test server should be > > > able to connect to passt itself (and we could call it "test peer" at > > > that point). =20 > >=20 > > So, I agree that to meaningfully fuzz things, we want the fuzzer to be > > able to control data on both the guest and host side. I can see two > > basic approaches: > >=20 > > A) Alter passt/pasta so that instead of directly communicating with > > external entities (either guest or host side) we use mocked > > versions which retrieve data from AFL. We can do that either at > > the system call level, or at a higher helper function level, the > > lower level we go, the more of the "normal" passt code we're > > exercising, but doing at a slightly higher level might be easier > > to implement > >=20 > > B) Run passt/pasta in an environment where we can intercept the > > external transfers to a test server / test peer / test guest which > > in turn responds based on data from AFL. > >=20 > > Both the current draft and the sketch diagram Stefano has provided are > > a hybrid of both approaches, so far I'm not seeing a clear advantage > > to that over going all one way or the other. >=20 > Of course, B) would be cleaner (and not that complicated, see below), > but we don't want to do that guest-side (sending data from a test > guest) because we would lose the speed advantage of having shared > memory on the path that _really_ matters for fuzzing (the guest is > untrusted, the kernel isn't). I agree we lose the speed advantage, but I don't really see why that matters more on the guest side than the host. Yes, fuzzing the guest side matters more, but most guest side operations will induce passt to perform a host side operation, so the speed of the host side handling matters even if the guest side is what we care about fuzzing. > That's something we already established a while ago when AbdAlRahman > was working on it. We hadn't really looked into the host side yet, back > then. >=20 > So, host side: we can't do it (and it's much less important) because we > need to use those sockets in the same way passt uses them. I'm not sure what you mean. I outlined a way to do this below. > The guest side interface is a trivial recv(), the host side is something > complicated with iovecs and everything. I'm now not sure if you're saying this in relation to approach B, or approach A. > > (B) is quite easy to do guest side - we just connect a "test guest" to > > passt's socket. Approach B is much harder for host side. The current > > draft has a test server listening on a single address. >=20 > This is just to get something up and running though, it obviously needs > to be changed later. I don't see how we are "up and running" if fuzzed packets from the guest induce passt to forward them to random host side addresses that we're not controlling - we won't generate reproducible results. > > But that means > > the fuzzing is fundamentally incapable of finding bugs involving > > talking to multiple peers at once. Worse, we have to constrain the > > construction of guest side data so that we talk to the test server not > > something else, and that's one of the things we most want to fuzz. > >=20 > > To really take approach B for the host side we'd need to intercept > > *all* host side network traffic regardless of address. Probably > > easiest way to do that would be put the whole thing inside another > > netns. That outside netns would have a default route to a tap device, > > and on the other side of the tap device would be a test server serving > > up frames built from the fuzzer output. >=20 > As I was mentioning, this could be done in a network namespace without > any interface, by making the test server listen to all ports and all > addresses, with a non-local bind and a so-called AnyIP route. Tested: >=20 > $ pasta -- sh -c 'ip route add local default dev lo; nc -l 1 & { sleep 1;= echo x | nc -N 1.2.3.4 1; }' > x True, but having the fuzzer synthesize L2 frames seems easier to me than having it directly synthesize the various socket operations the host side peer might perform. > > It's probably easier to co-ordinate if the host side and guest side > > test server is the same, so we'd have: > >=20 > > ,-------. ,-------------. > > | AFL++ |-- shares memory with ---| test peer | > > '-------' '--v------v---' > > | | > > /-tap device-/ | > > | | > > /- test netns ----------^-------------------|-------------\ > > | | | > > | ,--------. | | > > | | passt >----------------/ | > > | '--------' | > > \---------------------------------------------------------/ > >=20 > > The test peer generates host side frames via the tap device, and guest > > side frames via the Unix socket. >=20 > The UNIX socket is something we want to avoid, it's really much slower > compared to shared memory (we tried something like that) on the path > where AFL++ is trying to mutate data fast (because it can hit a lot of > different code paths with small changes, compared to changing > socket-side payload). Right. That's why I conclude approch A is probably better further down. But even if we only care about fuzzing guest side, we'll usually incur the cost of operations on both sides, so I don't see that using shared memory is more important for guest side than host side interposition. > > It could also be done with pasta, like this: > >=20 > > ,-------. ,-------------. > > | AFL++ |-- shares memory with ---| test peer | > > '-------' '--v------v---' > > | | > > /-tap device-/ packet > > | socket > > /- test netns ----------^-------------------|-------------\ > > | | | > > | ,--------. ,----------------|---. | > > | | pasta >-tap device-< guest netns * | | > > | '--------' '--------------------' | > > \---------------------------------------------------------/ > >=20 > >=20 > > The order it generates host vs. guest frames should also come from the > > fuzzer, not be fixed. At least theoretically, this is non-invasive: > > it could run with an unmodified passt/pasta. Except that > > - AFL would still need coverage feedback from passt/pasta > > - It wouldn't allow us to simulate odd timings (except by actually > > expending real time) > > - The various interposing layers will probably slow down fuzzing. > > =20 > > So, I rather suspect it will work better to go fully to approach A: no > > test peer at all, instead passt itself is modified to use mocked > > versions of all the external syscalls to slurp data from AFL. The > > draft series already does this for epoll_wait() and recv*(), but we'd > > need to also do that for recv*() on the tap socket, connect(), > > accept(), TCP_INFO and probably others. We'd also need to mock > > "sending" calls, send(), write() and shutdown() at least - but those > > could probably be no-ops. >=20 > ...except that by mocking all those we lose a lot of complexity where > historically we had a ton of bugs. If we just mock recv() it's much > less (well yes we had bugs there as well but it was like 3 or 4 over the > entire project history). If we mock as close as possible to the syscall level, I don't see that bypass much of our complexity. To be clear, I'm suggesting mocks where both returned data and error codes are derived from the fuzzer, not just no-op stubs. > > This is more invasive, of course. It also means we need to deal with > > the case where AFL generates a syscall results that should be > > impossible - that should move onto the next case ASAP, but not be > > flagged as a passt bug. On the other hand, this approach should be > > fast, and since we can also mock clock_gettime(), the fuzzer can > > potentially find timer logic bugs that would only occur after hours or > > days in real time. > > > > > As far as I understood, it's not trivial to make the same instance > > > of AFL++ share memory with two processes at the same time, so the > > > memory-sharing path might need to take a more complicated turn, for > > > example there could be a wrapper starting both passt and the test > > > server and sharing memory with them, or passt could _additionally_ > > > (using a special out-of-band fuzzing channel) share data from AFL++ > > > with the test server. > > >=20 > > > An example of communication below (but events don't necessarily need > > > to be in this order, this is just an example). For simplicity, let's > > > ignore the fact that AFL++ might not directly share memory with passt > > > and test server, and assume there are three areas of memory that > > > AFL++ directly controls: > > >=20 > > > a. shared with passt: an array of struct epoll_event, 'ev' > > >=20 > > > b. shared with passt: the kind of tap-side buffer you implemented in > > > 4/5, 'buf' > > >=20 > > > c. shared with the test server: a separate buffer, 'test_buf' > > >=20 > > > Example: > > >=20 > > > 1. AFL++ writes an EPOLLIN event in 'ev' with type > > > EPOLL_TYPE_TAP_PASST, of some data in 'buf', and some data in > > > 'test_buf' > > >=20 > > > 2. AFL++ starts passt and the test server > > >=20 > > > 3. passt reads the EPOLL_TYPE_TAP_PASST event from 'ev', reads data > > > from 'buf' and hands it to passt_tap_handler() > > >=20 > > > 4. this happens to be have Ethernet, IP, and TCP headers, with the > > > SYN flag set, and destination address set to the address of the > > > test server (we might want to force all this, at least initially, > > > or give it as a hint to AFL++ somehow), so passt connects to > > > the test server > > >=20 > > > 5. the test server accepts the connection, and sends the contents > > > of 'test_buf' on it (for the test server, this is directly > > > payload, without headers, as they don't make sense there). I'm > > > not sure if we should have a different set of events (maybe we > > > need a "play script" for the server, in case?) > > >=20 > > > 6. this generates an EPOLLOUT event for passt. It's not in 'ev', > > > it's a regular epoll_wait() (I think we could have an > > > epoll_wait() loop where we additionally read one event from > > > 'ev' for every iteration, or something like that) > > >=20 > > > 7. passt marks the connection as established and inserts it in the > > > flow table > > >=20 > > > 8. passt reads the data sent from the test server and generates > > > whatever TCP data packet to the "guest" (it might simply be > > > a sink) > > >=20 > > > ...and this attempt ends here because AFL++ generated a single > > > event for passt, but there could be more (this should also be > > > decided by AFL++). > > >=20 > > > Would something like this make sense? =20 >=20 > --=20 > Stefano >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --rrJNNN2oihKtkqdn Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmp+57gACgkQzQJF27ox 2Gfitw//WAgWzlyacime6zHsGre4y5fth5MTPFmL9u+gavxjRVtd70xOaWsgLXdO kzObs+teoFJ/EtBajsmUPwuj1RiMeh0eLna8VjZbASvhmsVVfqJ28O499xwO1qat nbC8am/yP63rFfG15/q7y90LxBcV128TDAlvdEU65NPG2f+WyC4MCfKjTYmMbSSX eiV3fU6WXNjhWzAx1+Q1X5wuRL5tDHJrhy/zpllspDqOYgmclyBmWBGgRbm+FBk+ 34Tk/BZzhv+zmaJAaczidSFCRETYSLrqNXISUgXmK2dLoh9uxsLFDhemyJ27E99g 0C87BzIasRctE2xHSXZA7+MgOIpeSc+dRM6IKTOuJV2HgzFusN/D1Cs6xjRl46mb BlDTsCzJBqaPOIxOR9TJ/aDPeeCrnPDoSh8UryVt/vxE23wifuDOyvXWmEiXpXHP GIPmaW53CHMa6kW4oL62qCdP5udyx/AIMb6SDVe4puuAlBiw3C4JGV+kO8uu+R07 8FSNMI/6aSfNoNxzHlThOBB7Jrt2VwvCp7LAqI5C4fyt3xAw2feQGiX+7kww5DdE 233iFE25Rux+dYBxpQJUFA8GZGxBWVtRY2570NSh9R/7Qx59w3G8xuu+icHm2MhW 4xG6zm/RbkZz46MbgCjYBpyw0tfu0AckzpeY4ROyk6YhG+G5C8Q= =7cuS -----END PGP SIGNATURE----- --rrJNNN2oihKtkqdn--