Re: [PATCH v4 00/10] vhost-user: Preparatory series for multiple iovec entries per virtqueue element

[Stefano Brivio <sbrivio@redhat.com>]

On Tue, 26 May 2026 09:31:51 +0200
Laurent Vivier <lvivier@redhat.com> wrote:

> On 5/22/26 14:04, Stefano Brivio wrote:
> > On Fri, 22 May 2026 07:44:55 +0200
> > Stefano Brivio <sbrivio@redhat.com> wrote:
> >   
> >> On Fri, 22 May 2026 06:22:39 +0200
> >> Stefano Brivio <sbrivio@redhat.com> wrote:
> >>  
> >>> On Fri, 22 May 2026 01:13:33 +0200
> >>> Laurent Vivier <lvivier@redhat.com> wrote:
> >>>      
> >>>> On 5/21/26 10:30, Laurent Vivier wrote:  
> >>>>> On 5/20/26 22:53, Stefano Brivio wrote:  
> >>>>>> On Wed, 20 May 2026 18:18:52 +0200
> >>>>>> Stefano Brivio <sbrivio@redhat.com> wrote:
> >>>>>>         
> >>>>>>> On Wed, 20 May 2026 18:07:08 +0200
> >>>>>>> Stefano Brivio <sbrivio@redhat.com> wrote:
> >>>>>>>         
> >>>>>>>> On Wed, 20 May 2026 17:34:45 +0200
> >>>>>>>> Stefano Brivio <sbrivio@redhat.com> wrote:  
> >>>>>>>>> On Wed, 13 May 2026 13:52:08 +0200
> >>>>>>>>> Laurent Vivier <lvivier@redhat.com> wrote:  
> >>>>>>>>>> Currently, the vhost-user path assumes each virtqueue element contains
> >>>>>>>>>> exactly one iovec entry covering the entire frame.  This assumption
> >>>>>>>>>> breaks as some virtio-net drivers (notably iPXE) provide descriptors where the
> >>>>>>>>>> vnet header and the frame payload are in separate buffers, resulting in
> >>>>>>>>>> two iovec entries per virtqueue element.
> >>>>>>>>>>
> >>>>>>>>>> This series refactors the vhost-user data path so that frame lengths,
> >>>>>>>>>> header sizes, and padding are tracked and passed explicitly rather than
> >>>>>>>>>> being derived from iovec sizes.  This decoupling is a prerequisite for
> >>>>>>>>>> correctly handling padding of multi-buffer frames.  
> >>>>>>>>>
> >>>>>>>>> Sorry to bring (likely) bad news, but this series seems to introduce a
> >>>>>>>>> regression: I got the migration/rampstream_in tests fail twice in a
> >>>>>>>>> row, which I've never saw happening (I think I saw a single failure a
> >>>>>>>>> long time ago when the machine had a high CPU load, but nothing else).
> >>>>>>>>>
> >>>>>>>>> I'm currently bisecting and the bisect seems to point towards the end
> >>>>>>>>> of the series (probably 10/10), but I haven't finished yet. I'll keep
> >>>>>>>>> you posted. I haven't spotted anything that might cause issues there.  
> >>>>>>>>
> >>>>>>>> Yeah, that's the one :(
> >>>>>>>>
> >>>>>>>> $ git bisect bad
> >>>>>>>> db798fc60f4c5869cb53168354e068fb4dabd91a is the first bad commit
> >>>>>>>> commit db798fc60f4c5869cb53168354e068fb4dabd91a
> >>>>>>>> Author: Laurent Vivier <lvivier@redhat.com>
> >>>>>>>> Date:   Wed May 13 13:52:18 2026 +0200
> >>>>>>>>
> >>>>>>>>       vhost-user: Centralise Ethernet frame padding in vu_collect() and vu_pad()  
> >>>>>
> >>>>> I checked on my system with the commit previous to this series,
> >>>>> bcc3d37a6e01 ("util: Fix changes to assert_with_msg()") and rampstream_in fails too (not
> >>>>> everytime).
> >>>>>          
> >>>>>   > TCP/IPv4: sequence check, ramps, inbound  
> >>>>> ...failed.
> >>>>>
> >>>>> and rampstream_out hangs sometime too.
> >>>>>
> >>>>> I'm going to try with ealier commits.  
> >>>>
> >>>> For me the problem can happen with any commit...
> >>>>
> >>>> As it depends on the execution path and on the load and speed of the system it looks like
> >>>> a race condition.  
> >>>
> >>> Hah, thanks for checking. Maybe...
> >>>      
> >>>> Did you try to test on a host with a kernel patched with
> >>>> "[PATCH net v2 0/2] Fix race condition between TCP_REPAIR dump and data receive" ?  
> >>>
> >>> Now I tried, and yes, the test doesn't hang anymore! I seem to have an
> >>> issue with teardown functions on recent kernels (current net.git HEAD
> >>> more or less):
> >>>
> >>> ---
> >>> [...]
> >>>
> >>> 2026/05/22 04:08:23 socat[73089] E connect(5, AF=40 cid:94558 port:22, 16): Connection timed out
> >>> Connection closed by UNKNOWN port 65535
> >>> ...
> >>> ---
> >>>
> >>> it looks like we stop QEMU a bit too early. But it should be unrelated.  
> > 
> > Oops, I forgot to upgrade QEMU on the virtual machine I was using to
> > test those kernel builds, I had a somewhat outdated 8.1 version and it
> > failed migration for unrelated reasons. It works with 11.0.
> > 
> > Back to kernel versions: the "problem" is that with a recent
> > net-next.git HEAD, with or without my fix, in a nested VM, the test
> > always passes (20/20). And I can't easily test things non-nested.
> > 
> > I guess could just skip that test for the moment from the set I run git
> > push, and run it manually in the virtual machine, for the moment.
> > 
> > But judging from captures (test_logs/pasta_1.pcap from PCAP=1 ./run)
> > I'm fairly sure it's not *that* issue:
> > 
> >    465  12.141763    192.0.2.1 → 88.198.0.164 58451 TCP [TCP Window Full] 34416 → 10001 [PSH, ACK] Seq=10002100 Ack=1 Win=65536 Len=58397
> >    466  12.187195 88.198.0.164 → 192.0.2.1    54 TCP [TCP ZeroWindow] 10001 → 34416 [ACK] Seq=1 Ack=10060497 Win=0 Len=0
> >    467  13.187281    192.0.2.1 → 88.198.0.164 4150 TCP 34416 → 10001 [PSH, ACK] Seq=10060497 Ack=1 Win=65536 Len=4096
> > 
> > last data transfer from client (rampstream):
> > 
> >    468  13.187358 88.198.0.164 → 192.0.2.1    54 TCP [TCP ZeroWindow] 10001 → 34416 [ACK] Seq=1 Ack=10060497 Win=0 Len=0
> > 
> > everything acknowledged, migration starts now:
> > 
> >    469  14.143217 fe80::f471:c3ff:fe10:4e45 → ff02::2      70 ICMPv6 Router Solicitation from f6:71:c3:10:4e:45
> >    470  14.687123 88.198.0.164 → 192.0.2.1    54 TCP [TCP ZeroWindow] [TCP Keep-Alive] 10001 → 34416 [ACK] Seq=0 Ack=10060497 Win=0 Len=0
> > 
> > migration completed: and we acknowledge the right sequence (10060497),
> > so it didn't jump forward.
> > 
> > But starting from this point:
> > 
> >    471  14.687265    192.0.2.1 → 88.198.0.164 60 TCP 34416 → 10001 [ACK] Seq=10060497 Ack=1 Win=65536 Len=0
> >    472  16.687412    192.0.2.1 → 88.198.0.164 4150 TCP [TCP Retransmission] 34416 → 10001 [PSH, ACK] Seq=10060497 Ack=1 Win=65536 Len=4096
> >    473  16.687450 88.198.0.164 → 192.0.2.1    54 TCP [TCP ZeroWindow] 10001 → 34416 [ACK] Seq=1 Ack=10060497 Win=0 Len=0
> >    474  20.687650    192.0.2.1 → 88.198.0.164 4150 TCP [TCP Retransmission] 34416 → 10001 [PSH, ACK] Seq=10060497 Ack=1 Win=65536 Len=4096
> >    475  20.687692 88.198.0.164 → 192.0.2.1    54 TCP [TCP ZeroWindow] 10001 → 34416 [ACK] Seq=1 Ack=10060497 Win=0 Len=0
> >    476  28.687817    192.0.2.1 → 88.198.0.164 4150 TCP [TCP Retransmission] 34416 → 10001 [PSH, ACK] Seq=10060497 Ack=1 Win=65536 Len=4096
> > 
> > we keep advertising a zero window (that's the kernel doing it really),
> > as if we were unable to dequeue data.
> > 
> > I enabled --trace just for the target instance of passt, and I don't
> > see anything suspicious there:
> > 
> > 13.0958: Receiving 1 flows
> > 13.0958: Flow 0 (NEW): FREE -> NEW
> > 13.0958: Flow 0 (TCP connection): TGT -> TYPED
> > 13.0958: Flow 0 (TCP connection): HOST [192.0.2.1]:49892 -> [88.198.0.164]:10001 => TAP [192.0.2.1]:49892 -> [88.198.0.164]:10001
> > 13.0958: Flow 0 (TCP connection): Side 1 hash table insert: bucket: 138154
> > 13.0958: Flow 0 (TCP connection): TYPED -> ACTIVE
> > 13.0958: Flow 0 (TCP connection): HOST [192.0.2.1]:49892 -> [88.198.0.164]:10001 => TAP [192.0.2.1]:49892 -> [88.198.0.164]:10001
> > 13.0959: Flow 0 (TCP connection): Extended migration data, socket 83 sequences send 3121929544 receive 1643895001
> > 13.0959: Flow 0 (TCP connection):   pending queues: send 0 not sent 0 receive 3500081
> > 13.0959: Flow 0 (TCP connection):   window: snd_wl1 1647395082 snd_wnd 65536 max 65536 rcv_wnd 0 rcv_wup 1647395082
> > 13.0959: Flow 0 (TCP connection):   SO_PEEK_OFF disabled  offset=0
> > 13.0985: Got packet, but RX virtqueue not usable yet
> > 13.0985: Closing migration channel, fd: 82
> > 13.0985: Closing TCP_REPAIR helper socket
> > 13.0985: passt: epoll event on vhost-user command socket 77 (events: 0x00000001)
> > 
> > then the usual VHOST_USER_CHECK_DEVICE_STATE and VHOST_USER_SET_VRING_ENABLE
> > commands. After that, a tight loop of:
> > 
> > 13.0986: passt: epoll event on connected TCP socket 83 (events: 0x00000001)
> > 13.0986: Got packet, but RX virtqueue not usable yet
> > 13.0986: passt: epoll event on connected TCP socket 83 (events: 0x00000001)
> > 13.0986: Got packet, but RX virtqueue not usable yet
> > 
> > until we go further with the vhost-user setup. I still see this message
> > which I had never noticed (but I didn't try to bisect around it):
> > 
> > 13.1006: ================ Vhost user message ================
> > 13.1006: Request: VHOST_USER_SET_VRING_ADDR (9)
> > [...]
> > 13.1006: Last avail index != used index: 3252 != 3027
> > 
> > and then after VHOST_USER_SET_VRING_CALL, and:
> > 
> > 13.1008: passt: epoll event on vhost-user kick socket 78 (events: 0x00000001)
> > 13.1008: vhost-user: got kick_data: 0000000000000001 idx: 1
> > 
> > it's just a tight loop of:
> > 
> > 13.1008: passt: epoll event on connected TCP socket 83 (events: 0x00000001)
> > 13.1008: passt: epoll event on connected TCP socket 83 (events: 0x00000001)
> > 13.1008: passt: epoll event on connected TCP socket 83 (events: 0x00000001)
> > 
> > as if we weren't dequeueing anything from there.
> > 
> > I start suspecting we might be hitting two different issues: perhaps
> > things fail on your setup because of the kernel bug with TCP_REPAIR not
> > freezing the queue, and they fail on my setup for some other reason.
> > 
> > For me it's very deterministic though: with patch 10/10 things always
> > fail, and without it they never fail.
> > 
> > I guess I'll add more prints and check for more messages before/after
> > that patch.
> >   
> 
> In fact there is a buffer leak because iov_skip_bytes() doesn't correctly compute the 
> number of used elements and then we don't release all the unused buffers.
> 
> I'm trying to fix that.
> 
> Please try with series "[PATCH v7 0/4] vhost-user,tcp: Handle multiple iovec entries per 
> virtqueue element" applied, it reworks this part.

I'm trying it now. If that totally reworks this part and it fixes
things and it's ready to be merged (sorry, I didn't manage to have a
look yet) I don't think it's strictly necessary to figure out the
leak.

-- 
Stefano