Re: [PATCH 5/5] fuzz: Add test server for bidirectional protocol fuzzing

[Stefano Brivio <sbrivio@redhat.com>]

On Fri, 14 Aug 2026 20:02:46 +1000
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Fri, Aug 14, 2026 at 09:35:59AM +0200, Stefano Brivio wrote:
> > On Fri, 14 Aug 2026 15:40:35 +1000
> > David Gibson <david@gibson.dropbear.id.au> wrote:
> >   
> > > On Thu, Aug 13, 2026 at 09:53:24AM +0200, Stefano Brivio wrote:  
> > > > On Wed, 12 Aug 2026 12:56:28 +0530
> > > > Anshu Kumari <anskuma@redhat.com> wrote:
> > > >     
> > > > > Add fuzz-server that acts as passt's network peer
> > > > > during fuzzing.    
> > > > 
> > > > To me, this part makes sense. But this one:
> > > >     
> > > > > It connects to passt's UNIX socket    
> > > > 
> > > > much less, while:
> > > >     
> > > > > and listens on 127.0.0.1:9999 for TCP connections.    
> > > > 
> > > > this is the part that I expected instead. Otherwise it's not just
> > > > passt's network peer, it's the guest as well.
> > > >     
> > > > > UNIX socket path: responds to ARP requests and TCP SYNs with
> > > > >   stateless replies (swapped addresses, fixed ISN). Responses
> > > > >   are XOR'd with AFL++ shared memory data so the fuzzer can
> > > > >   mutate server behavior.    
> > > > 
> > > > This looks rather complicated to me.    
> > > 
> > > It does.
> > >   
> > > > The approach I was suggesting with a test server is the following:
> > > > 
> > > > 
> > > >     ,- exchanges guest-side data with ------------.
> > > >     |                                   ,---------|---------.
> > > >     |                                ,--|      passt        |
> > > >     |                               /   '-.---------------^-'
> > > > ,---|---.                          /      | connect(),    | accept(),
> > > > | AFL++ |-- shares memory with ---|       | send data,    | reply with
> > > > '---|---'                          \      | etc.          | data, etc.
> > > >     |                               \   ,-v---------------'-.
> > > >     |                                '--|    test server    |
> > > >     |                                   '---------|---------'
> > > >     '- exchanges host-side data with -------------'
> > > > 
> > > > ...at least in its basic form. Eventually, the test server should be
> > > > able to connect to passt itself (and we could call it "test peer" at
> > > > that point).    
> > > 
> > > So, I agree that to meaningfully fuzz things, we want the fuzzer to be
> > > able to control data on both the guest and host side.  I can see two
> > > basic approaches:
> > > 
> > >  A) Alter passt/pasta so that instead of directly communicating with
> > >     external entities (either guest or host side) we use mocked
> > >     versions which retrieve data from AFL.  We can do that either at
> > >     the system call level, or at a higher helper function level, the
> > >     lower level we go, the more of the "normal" passt code we're
> > >     exercising, but doing at a slightly higher level might be easier
> > >     to implement
> > > 
> > >  B) Run passt/pasta in an environment where we can intercept the
> > >     external transfers to a test server / test peer / test guest which
> > >     in turn responds based on data from AFL.
> > > 
> > > Both the current draft and the sketch diagram Stefano has provided are
> > > a hybrid of both approaches, so far I'm not seeing a clear advantage
> > > to that over going all one way or the other.  
> > 
> > Of course, B) would be cleaner (and not that complicated, see below),
> > but we don't want to do that guest-side (sending data from a test
> > guest) because we would lose the speed advantage of having shared
> > memory on the path that _really_ matters for fuzzing (the guest is
> > untrusted, the kernel isn't).  
> 
> I agree we lose the speed advantage, but I don't really see why that
> matters more on the guest side than the host.

Because AFL++ tries to vary the input to discover new code paths, but
if it's not necessary (and in general it's not for host-side payload),
the input might remain relatively constant, and memory content that
isn't changed it's cache hot.

I have only profiled the "guest" side of things so far, though.

In any case, this is minor. If we can have shared memory and no further
transport on one path, it's better than having it on zero paths.

> Yes, fuzzing the guest
> side matters more, but most guest side operations will induce passt to
> perform a host side operation, so the speed of the host side handling
> matters even if the guest side is what we care about fuzzing.
> 
> > That's something we already established a while ago when AbdAlRahman
> > was working on it. We hadn't really looked into the host side yet, back
> > then.
> > 
> > So, host side: we can't do it (and it's much less important) because we
> > need to use those sockets in the same way passt uses them.  
> 
> I'm not sure what you mean.  I outlined a way to do this below.

Sure, strictly speaking, we can, but we can't if we want to obtain a
realistic approximation of what we would be normally doing on sockets.

We would wrap all the host-side socket operations, in that case, which
is really not ideal, to the point of questioning the whole
effectiveness.

> > The guest side interface is a trivial recv(), the host side is something
> > complicated with iovecs and everything.  
> 
> I'm now not sure if you're saying this in relation to approach B, or
> approach A.

That's independent from the chosen approach: it's like that without
fuzzing.

If we wrap / skip recv() on the guest side, that's not a big loss. If
we wrap host-side socket operations, that's a significant deviation and
we'll miss bugs.

> > > (B) is quite easy to do guest side - we just connect a "test guest" to
> > > passt's socket.  Approach B is much harder for host side.  The current
> > > draft has a test server listening on a single address.  
> > 
> > This is just to get something up and running though, it obviously needs
> > to be changed later.  
> 
> I don't see how we are "up and running" if fuzzed packets from the
> guest induce passt to forward them to random host side addresses that
> we're not controlling - we won't generate reproducible results.

If you look at patch 4/5, it hardcodes the destination port and uses a
known destination address (again, for the moment). It's not random. It
will need to be random of course.

> > > But that means
> > > the fuzzing is fundamentally incapable of finding bugs involving
> > > talking to multiple peers at once.  Worse, we have to constrain the
> > > construction of guest side data so that we talk to the test server not
> > > something else, and that's one of the things we most want to fuzz.
> > > 
> > > To really take approach B for the host side we'd need to intercept
> > > *all* host side network traffic regardless of address.  Probably
> > > easiest way to do that would be put the whole thing inside another
> > > netns.  That outside netns would have a default route to a tap device,
> > > and on the other side of the tap device would be a test server serving
> > > up frames built from the fuzzer output.  
> > 
> > As I was mentioning, this could be done in a network namespace without
> > any interface, by making the test server listen to all ports and all
> > addresses, with a non-local bind and a so-called AnyIP route. Tested:
> > 
> > $ pasta -- sh -c 'ip route add local default dev lo; nc -l 1 & { sleep 1; echo x | nc -N 1.2.3.4 1; }'
> > x  
> 
> True, but having the fuzzer synthesize L2 frames

Slightly easier, perhaps, but that's beyond the scope of fuzzing we
need (AFL++ will start trying to build malformed frames which we won't
see anyway and effectively fuzz the kernel instead).

> seems easier to me
> than having it directly synthesize the various socket operations the
> host side peer might perform.

I don't see this as complicated. There just needs to be a way for AFL++
to say how much payload the test server might need to send in a given
round, and that can be used as argument to send() or sendmsg(). That's
what I meant by "play script."

> > > It's probably easier to co-ordinate if the host side and guest side
> > > test server is the same, so we'd have:
> > > 
> > > ,-------.                         ,-------------.
> > > | AFL++ |-- shares memory with ---|  test peer  |
> > > '-------'                         '--v------v---'
> > >                                      |      |
> > >                         /-tap device-/      |
> > >                         |                   |
> > > /- test netns ----------^-------------------|-------------\
> > > |                                           |             |
> > > |    ,--------.                             |             |
> > > |    | passt  >-----------<unix socket>-----/             |
> > > |    '--------'                                           |
> > > \---------------------------------------------------------/
> > > 
> > > The test peer generates host side frames via the tap device, and guest
> > > side frames via the Unix socket.  
> > 
> > The UNIX socket is something we want to avoid, it's really much slower
> > compared to shared memory (we tried something like that) on the path
> > where AFL++ is trying to mutate data fast (because it can hit a lot of
> > different code paths with small changes, compared to changing
> > socket-side payload).  
> 
> Right.  That's why I conclude approch A is probably better further
> down.  But even if we only care about fuzzing guest side, we'll
> usually incur the cost of operations on both sides, so I don't see
> that using shared memory is more important for guest side than host
> side interposition.
>
> > > It could also be done with pasta, like this:
> > > 
> > > ,-------.                         ,-------------.
> > > | AFL++ |-- shares memory with ---|  test peer  |
> > > '-------'                         '--v------v---'
> > >                                      |      |
> > >                         /-tap device-/      packet
> > >                         |                   socket
> > > /- test netns ----------^-------------------|-------------\
> > > |                                           |             |
> > > |    ,--------.            ,----------------|---.         |
> > > |    | pasta  >-tap device-< guest netns    *   |         |
> > > |    '--------'            '--------------------'         |
> > > \---------------------------------------------------------/
> > > 
> > > 
> > > The order it generates host vs. guest frames should also come from the
> > > fuzzer, not be fixed.  At least theoretically, this is non-invasive:
> > > it could run with an unmodified passt/pasta.  Except that
> > >   - AFL would still need coverage feedback from passt/pasta
> > >   - It wouldn't allow us to simulate odd timings (except by actually
> > >      expending real time)
> > >   - The various interposing layers will probably slow down fuzzing.
> > >   
> > > So, I rather suspect it will work better to go fully to approach A: no
> > > test peer at all, instead passt itself is modified to use mocked
> > > versions of all the external syscalls to slurp data from AFL.  The
> > > draft series already does this for epoll_wait() and recv*(), but we'd
> > > need to also do that for recv*() on the tap socket, connect(),
> > > accept(), TCP_INFO and probably others.  We'd also need to mock
> > > "sending" calls, send(), write() and shutdown() at least - but those
> > > could probably be no-ops.  
> > 
> > ...except that by mocking all those we lose a lot of complexity where
> > historically we had a ton of bugs. If we just mock recv() it's much
> > less (well yes we had bugs there as well but it was like 3 or 4 over the
> > entire project history).  
> 
> If we mock as close as possible to the syscall level, I don't see that
> bypass much of our complexity.  To be clear, I'm suggesting mocks
> where both returned data and error codes are derived from the fuzzer,
> not just no-op stubs.

Doing that is not a realistic test though, because the kernel won't
return random error codes.

Indeed it would be nice to be robust to kernel issues, but I don't see
it as a priority (and I guess we would waste the whole time on those
"issues" if we do that).

> > > This is more invasive, of course.  It also means we need to deal with
> > > the case where AFL generates a syscall results that should be
> > > impossible - that should move onto the next case ASAP, but not be
> > > flagged as a passt bug.  On the other hand, this approach should be
> > > fast, and since we can also mock clock_gettime(), the fuzzer can
> > > potentially find timer logic bugs that would only occur after hours or
> > > days in real time.
> > >  
> > > > As far as I understood, it's not trivial to make the same instance
> > > > of AFL++ share memory with two processes at the same time, so the
> > > > memory-sharing path might need to take a more complicated turn, for
> > > > example there could be a wrapper starting both passt and the test
> > > > server and sharing memory with them, or passt could _additionally_
> > > > (using a special out-of-band fuzzing channel) share data from AFL++
> > > > with the test server.
> > > > 
> > > > An example of communication below (but events don't necessarily need
> > > > to be in this order, this is just an example). For simplicity, let's
> > > > ignore the fact that AFL++ might not directly share memory with passt
> > > > and test server, and assume there are three areas of memory that
> > > > AFL++ directly controls:
> > > > 
> > > > a. shared with passt: an array of struct epoll_event, 'ev'
> > > > 
> > > > b. shared with passt: the kind of tap-side buffer you implemented in
> > > >    4/5, 'buf'
> > > > 
> > > > c. shared with the test server: a separate buffer, 'test_buf'
> > > > 
> > > > Example:
> > > > 
> > > > 1. AFL++ writes an EPOLLIN event in 'ev' with type
> > > >    EPOLL_TYPE_TAP_PASST, of some data in 'buf', and some data in
> > > >    'test_buf'
> > > > 
> > > > 2. AFL++ starts passt and the test server
> > > > 
> > > > 3. passt reads the EPOLL_TYPE_TAP_PASST event from 'ev', reads data
> > > >    from 'buf' and hands it to passt_tap_handler()
> > > > 
> > > > 4. this happens to be have Ethernet, IP, and TCP headers, with the
> > > >    SYN flag set, and destination address set to the address of the
> > > >    test server (we might want to force all this, at least initially,
> > > >    or give it as a hint to AFL++ somehow), so passt connects to
> > > >    the test server
> > > > 
> > > > 5. the test server accepts the connection, and sends the contents
> > > >    of 'test_buf' on it (for the test server, this is directly
> > > >    payload, without headers, as they don't make sense there). I'm
> > > >    not sure if we should have a different set of events (maybe we
> > > >    need a "play script" for the server, in case?)
> > > > 
> > > > 6. this generates an EPOLLOUT event for passt. It's not in 'ev',
> > > >    it's a regular epoll_wait() (I think we could have an
> > > >    epoll_wait() loop where we additionally read one event from
> > > >    'ev' for every iteration, or something like that)
> > > > 
> > > > 7. passt marks the connection as established and inserts it in the
> > > >    flow table
> > > > 
> > > > 8. passt reads the data sent from the test server and generates
> > > >    whatever TCP data packet to the "guest" (it might simply be
> > > >    a sink)
> > > > 
> > > > ...and this attempt ends here because AFL++ generated a single
> > > > event for passt, but there could be more (this should also be
> > > > decided by AFL++).
> > > > 
> > > > Would something like this make sense?    

-- 
Stefano