From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UExJiu03; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 3462C5A0280 for ; Wed, 21 May 2025 02:57:27 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747789046; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=Mf5kqv1ThGB317/joKxN5cMyxBOofGTE7s+pmph4cPY=; b=UExJiu03+9pkgsSBKCCxU6M032SBNsKOGKqX1kO9z/qj5MZZwFwP47+WZihkx5RYDd83bL gA5scWyXDc2RUFq82s7nD1pmBodL/j2HzWXAjCorLVggSZ5gb9FyjYKChNkTGLkBeGB8m4 aPiLJDc84oaV2SV7Ju77eQjjGEhmT8U= Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-587-ZHSZPl6aPZ-5M9lILa02Sg-1; Tue, 20 May 2025 20:57:24 -0400 X-MC-Unique: ZHSZPl6aPZ-5M9lILa02Sg-1 X-Mimecast-MFC-AGG-ID: ZHSZPl6aPZ-5M9lILa02Sg_1747789043 Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-30e9338430eso3708290a91.3 for ; Tue, 20 May 2025 17:57:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747789043; x=1748393843; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Mf5kqv1ThGB317/joKxN5cMyxBOofGTE7s+pmph4cPY=; b=b+mmaMQMiYE2DGv/bwtTFHTTmLrBUNfp4Zlbuf+nrtsiRgf1HS9/HRpFbMUQG6vgEj VE6+t7QtzdzL7PPbx7B9vPX6bM0VX89rTDg5aZGa7gczUixnVzVxc/k26vI9rRdeN2Yd fb11k17dGh7ncFAbUsa0CLWHkrfFhj/gO09+gOvlNqnQqYQWGnGjcK/3WCPU17tnRzmB yJIl6BK4I/7FLaYRAqj55SOW1faEJkFDhhUXs1tQp0+IeZhARhgzKkzxT08iKX15RTKG DL0Nopjy00v5VBNEqwHAtR+dw063FK1Err8RQLqO2nLLpa4x6cWi+0BrDQkBIqvJcsU5 XZkw== X-Gm-Message-State: AOJu0Yw72vwxCCyViccPia+uGNPnjwE2LWZSqGCmlnK5rckjlR1cVJhg 1V0EmAoAXWKzTFAlM6uF+KS0V8/Y77wNotSVHr1JWA11o2pu9noQSPp4VLTRERn/vn+A1GcfYcn lTm5vPAGmx1e9wPqWTOyQF2cJwZEd8ScPoNfWKsBNJVFau82l2BD7SQ0stmwsJtl+lwhB9GJiwq PW1lflIjLA/4h7dPzep/+2wKROYA/o X-Gm-Gg: ASbGncvOZwRHsnVH8nZwWuQBo7780eHIQyiGhEaWmlhVXxFpdBeluIHPtAj8gWa9NDm ylXN9NjrRQ0c8IJDAnqxI6eP299I8z1BhNztIJ/zGQgO6UWHKjQdYZYj1bhDLwIMEDDCjSA== X-Received: by 2002:a17:90b:4b0f:b0:2fa:e9b:33b8 with SMTP id 98e67ed59e1d1-30e7d5565a9mr32609569a91.18.1747789043417; Tue, 20 May 2025 17:57:23 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEQPiDt/DgjIVbUlIyrgJTdf7IgwryXY7AQg95pVL2bkpt4JVSqJKprkfZSNI0Iob9qrmLnkPsNWL2PipiY5fk= X-Received: by 2002:a17:90b:4b0f:b0:2fa:e9b:33b8 with SMTP id 98e67ed59e1d1-30e7d5565a9mr32609539a91.18.1747789042943; Tue, 20 May 2025 17:57:22 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Jason Wang Date: Wed, 21 May 2025 08:57:11 +0800 X-Gm-Features: AX0GCFtsUomgBXlm7d8527ziWnbhichtI4Ls1r1gfIZ3A_uNNQxg3LFZsTuclNE Message-ID: Subject: Re: vhost-kernel net on pasta: from 26 to 37Gbit/s To: Eugenio Perez Martin X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: LSEDSPVbIsD7sfJPoeq_E4Wut9EepL7_Epw_BHq9_BE_1747789043 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-MailFrom: jasowang@redhat.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: VCFKS5MPVPONRZD6HC5RQ23P4EIGJGFU X-Message-ID-Hash: VCFKS5MPVPONRZD6HC5RQ23P4EIGJGFU X-Mailman-Approved-At: Wed, 21 May 2025 11:24:15 +0200 CC: passt-dev@passt.top, Jeff Nelson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, May 20, 2025 at 11:10=E2=80=AFPM Eugenio Perez Martin wrote: > > Hi! > > Some updates on the integration. The main culprit was to allow pasta > to keep reading packets using the regular read() on the tap device. I > thought that part was completely disabled, but I guess the kernel is > able to omit the notification on tap as long as the userspace does not > read it. > > My scenario: in different CPUs, all in the same NUMA. I run iperf > server on CPU 11 with "iperf3 -A 11 -s". All odds CPUs are isolated > with isolcpus=3D1,3,... nohz=3Don nohz_full=3D1,3,... > > With vanilla pasta isolated to CPUs 1,3 with taskset, and just > --config-net option, and running iperf with "iperf3 -A 5 -c > 10.6.68.254 -w 8M": > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 30.7 GBytes 26.4 Gbits/sec 0 sen= der > [ 5] 0.00-10.04 sec 30.7 GBytes 26.3 Gbits/sec rec= eiver > > Now trying with the vhost patches we get a slightly worse performance: > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 25.5 GBytes 21.9 Gbits/sec 0 sen= der > [ 5] 0.00-10.04 sec 25.5 GBytes 21.8 Gbits/sec rec= eiver > > Now vhost patch still lacks optimizations like disabling notifications > or batch more rx available buffer notifications. At the moment it > refills the rx buffers in each iteration, and does not set the > no_notify bit which makes the kernel skip the used buffer > notifications if pasta is actively checking the queue, which is not > optimal. > > Now if I isolate the vhost kernel thread [1] I get way more > performance as expected: > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bitrate Retr > [ 5] 0.00-10.00 sec 43.1 GBytes 37.1 Gbits/sec 0 sen= der > [ 5] 0.00-10.04 sec 43.1 GBytes 36.9 Gbits/sec rec= eiver > > After analyzing perf output, rep_movs_alternative is the most called > function in the three iperf3 (~20%Self), passt.avx2 (~15%Self) and > vhost (~15%Self), But I don't see any of them consuming 100% of CPU in > top: pasta consumes ~85% %CPU, both iperf3 client and server consumes > 60%, and vhost consumes ~53%. > > So... I have mixed feelings about this :). By "default" it seems to > have less performance, but my test is maybe too synthetic. There is > room for improvement with the mentioned optimizations so I'd continue > applying them, continuing with UDP and TCP zerocopy, and developing > zerocopy vhost rx. With these numbers I think the series should not be > merged at the moment. I could send it as RFC if you want but I've not > applied the comments the first one received, POC style :). Have you pinned pasta in a specific CPU? Note that vhost will inherit the affinity so there could be some contention if you do that. Thanks > > Thanks! > > [1] Notes to reproduce it, I'm able to see it with top -H and then set > with taskset. Either the latest changes on the module or the way pasta > behaves does not allow me to see in classical ps output. >