Re: vhost-kernel net on pasta: from 26 to 37Gbit/s

[Eugenio Perez Martin <eperezma@redhat.com>]

On Wed, May 21, 2025 at 12:35 PM Eugenio Perez Martin
<eperezma@redhat.com> wrote:
>
> On Wed, May 21, 2025 at 12:09 PM Stefano Brivio <sbrivio@redhat.com> wrote:
> >
> > On Tue, 20 May 2025 17:09:44 +0200
> > Eugenio Perez Martin <eperezma@redhat.com> wrote:
> >
> > > [...]
> > >
> > > Now if I isolate the vhost kernel thread [1] I get way more
> > > performance as expected:
> > > - - - - - - - - - - - - - - - - - - - - - - - - -
> > > [ ID] Interval           Transfer     Bitrate         Retr
> > > [  5]   0.00-10.00  sec  43.1 GBytes  37.1 Gbits/sec    0             sender
> > > [  5]   0.00-10.04  sec  43.1 GBytes  36.9 Gbits/sec                  receiver
> > >
> > > After analyzing perf output, rep_movs_alternative is the most called
> > > function in the three iperf3 (~20%Self), passt.avx2 (~15%Self) and
> > > vhost (~15%Self)
> >
> > Interesting... s/most called function/function using the most cycles/, I
> > suppose.
> >
>
> Right!
>
> > So it looks somewhat similar to
> >
> >   https://archives.passt.top/passt-dev/20241017021027.2ac9ea53@elisabeth/
> >
> > now?
> >
>
> Kind of. Below tcp_sendmsg_locked I don't see sk_page_frag_refill but
> skb_do_copy_data_nocache. Not sure if that means something, as it
> should not be affected by vhost.
>
> > > But I don't see any of them consuming 100% of CPU in
> > > top: pasta consumes ~85% %CPU, both iperf3 client and server consumes
> > > 60%, and vhost consumes ~53%.
> > >
> > > So... I have mixed feelings about this :). By "default" it seems to
> > > have less performance, but my test is maybe too synthetic.
> >
> > Well, surely we can't ask Podman users to pin specific stuff to given
> > CPU threads. :)
> >
>
> Yes but maybe the result changes under the right schedule? I'm
> isolating the CPUs entirely, which is not the usual case for pasta for
> sure :).
>
> > > There is room for improvement with the mentioned optimizations so I'd
> > > continue applying them, continuing with UDP and TCP zerocopy, and
> > > developing zerocopy vhost rx.
> >
> > That definitely makes sense to me.
> >
>
> Good!
>
> > > With these numbers I think the series should not be
> > > merged at the moment. I could send it as RFC if you want but I've not
> > > applied the comments the first one received, POC style :).
> >
> > I don't think it's really needed for you to spend time on
> > semi-polishing something just to have an RFC if you're still working on
> > it. I guess the implementation will change substantially anyway once
> > you factor in further optimisations.
> >
>
> Agree! I'll keep iterating on this then.
>

Actually, if I remove all the taskset etc, and trust the kernel
scheduler, vanilla pasta gives me:
[pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 -w 8M
Connecting to host 10.6.68.254, port 5201
[  5] local 10.6.68.20 port 40408 connected to 10.6.68.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  3.11 GBytes  26.7 Gbits/sec    0   25.4 MBytes
[  5]   1.00-2.00   sec  3.11 GBytes  26.7 Gbits/sec    0   25.4 MBytes
[  5]   2.00-3.00   sec  3.12 GBytes  26.8 Gbits/sec    0   25.4 MBytes
[  5]   3.00-4.00   sec  3.11 GBytes  26.7 Gbits/sec    0   25.4 MBytes
[  5]   4.00-5.00   sec  3.10 GBytes  26.6 Gbits/sec    0   25.4 MBytes
[  5]   5.00-6.00   sec  3.11 GBytes  26.7 Gbits/sec    0   25.4 MBytes
[  5]   6.00-7.00   sec  3.11 GBytes  26.7 Gbits/sec    0   25.4 MBytes
[  5]   7.00-8.00   sec  3.09 GBytes  26.6 Gbits/sec    0   25.4 MBytes
[  5]   8.00-9.00   sec  3.08 GBytes  26.5 Gbits/sec    0   25.4 MBytes
[  5]   9.00-10.00  sec  3.10 GBytes  26.6 Gbits/sec    0   25.4 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  31.0 GBytes  26.7 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  31.0 GBytes  26.5 Gbits/sec                  receiver

And with vhost-net :
[pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 -w 8M
...
Connecting to host 10.6.68.254, port 5201
[  5] local 10.6.68.20 port 46720 connected to 10.6.68.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.17 GBytes  35.8 Gbits/sec    0   11.9 MBytes
[  5]   1.00-2.00   sec  4.17 GBytes  35.9 Gbits/sec    0   11.9 MBytes
[  5]   2.00-3.00   sec  4.16 GBytes  35.7 Gbits/sec    0   11.9 MBytes
[  5]   3.00-4.00   sec  4.14 GBytes  35.6 Gbits/sec    0   11.9 MBytes
[  5]   4.00-5.00   sec  4.16 GBytes  35.7 Gbits/sec    0   11.9 MBytes
[  5]   5.00-6.00   sec  4.16 GBytes  35.8 Gbits/sec    0   11.9 MBytes
[  5]   6.00-7.00   sec  4.18 GBytes  35.9 Gbits/sec    0   11.9 MBytes
[  5]   7.00-8.00   sec  4.19 GBytes  35.9 Gbits/sec    0   11.9 MBytes
[  5]   8.00-9.00   sec  4.18 GBytes  35.9 Gbits/sec    0   11.9 MBytes
[  5]   9.00-10.00  sec  4.18 GBytes  35.9 Gbits/sec    0   11.9 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  41.7 GBytes  35.8 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  41.7 GBytes  35.7 Gbits/sec                  receiver

If I  go the extra mile and disable notifications (it might be just
noise, but...)
[pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 -w 8M
...
Connecting to host 10.6.68.254, port 5201
[  5] local 10.6.68.20 port 56590 connected to 10.6.68.254 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  4.19 GBytes  36.0 Gbits/sec    0   12.4 MBytes
[  5]   1.00-2.00   sec  4.18 GBytes  35.9 Gbits/sec    0   12.4 MBytes
[  5]   2.00-3.00   sec  4.18 GBytes  35.9 Gbits/sec    0   12.4 MBytes
[  5]   3.00-4.00   sec  4.20 GBytes  36.1 Gbits/sec    0   12.4 MBytes
[  5]   4.00-5.00   sec  4.21 GBytes  36.2 Gbits/sec    0   12.4 MBytes
[  5]   5.00-6.00   sec  4.21 GBytes  36.1 Gbits/sec    0   12.4 MBytes
[  5]   6.00-7.00   sec  4.20 GBytes  36.1 Gbits/sec    0   12.4 MBytes
[  5]   7.00-8.00   sec  4.23 GBytes  36.4 Gbits/sec    0   12.4 MBytes
[  5]   8.00-9.00   sec  4.24 GBytes  36.4 Gbits/sec    0   12.4 MBytes
[  5]   9.00-10.00  sec  4.21 GBytes  36.2 Gbits/sec    0   12.4 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  42.1 GBytes  36.1 Gbits/sec    0             sender
[  5]   0.00-10.04  sec  42.1 GBytes  36.0 Gbits/sec                  receiver

So I guess the best is to actually run performance tests closer to
real-world workload against the new version and see if it works
better?

Thanks!