From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Z58n4Rdr; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 43EF95A027C for ; Wed, 21 May 2025 07:37:56 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747805875; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O0zqIr2CMkMf5e2EDyChB3PZsKv7p0Lm0NVMv4kxr/w=; b=Z58n4Rdr3qY/tJ4F1UHCfoS19dNPkfJfgMElHvbmdUwGVuvREEJpN3FJqdnJdoNp6duLAy TwQSV8zZBtgRPk+ONXcVyJVxF2Hyv/TGlpwFMHMzY9HIS+/k2HLwA7W/rDW0cG2nvw1Ds4 I9YJATmrai+yGMvUjyd2xW48fIdW4Iw= Received: from mail-pj1-f72.google.com (mail-pj1-f72.google.com [209.85.216.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-625-0KcXfQRqP26cpJhP0lSW1Q-1; Wed, 21 May 2025 01:37:53 -0400 X-MC-Unique: 0KcXfQRqP26cpJhP0lSW1Q-1 X-Mimecast-MFC-AGG-ID: 0KcXfQRqP26cpJhP0lSW1Q_1747805872 Received: by mail-pj1-f72.google.com with SMTP id 98e67ed59e1d1-30ea0e890ccso3953737a91.2 for ; Tue, 20 May 2025 22:37:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747805872; x=1748410672; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O0zqIr2CMkMf5e2EDyChB3PZsKv7p0Lm0NVMv4kxr/w=; b=BiYfxZAzMxIG6tDMP1kV/E5rUYnvlM7Qu/X1Ov318yDpEuybpPfLQLxG9iRVanG1uD /PzzJBvscO5bUgh5dBkTKzFTQx3VZK8Wl0l1I0UkF0lnW88crzqsvkUjbXf8xPdwp4tO v/hD2s/ph10ADuTRT4fx1Ip7Z0zaeT0H+tu2bhk0vkXPQXtM98i5eOTIGSrFvNThakK6 RHBjAGUu+QjHXuyOO5XB105oJtPbHn21FfWhPdbLbMS8CDVAFNofIDERx8YotjNeo4vK tMYMf42LJ0FJTjt21ASlvVl6iFmdbu5NAPrGhsQjyGjq7L/jcKZ183XbrMyTIBFdQJVb ts8g== X-Gm-Message-State: AOJu0Yx7Ih2Rq2ncrCEIGaCemWXIUvCfa7t/doi6A5ieJ6gG+vYou5P4 XT9Yyri278fPfgz4DCDXEygdaRP5u4TdepNNe5z6J+rrKNQSbkkGizR07tg0s1m33yfaWZn06HV DOSAIofWcbJVGpoudt07LJrf1h2orDgWLbW3hY/fOFq6mop/ub/O9qLv9zcLlY42XhzLZkyETAx 3rhW18PWJzfxD/tKL5ldPEAUtRxTeU X-Gm-Gg: ASbGncu3mkDjXb+yNCVyihq1US0kVVEjGVq8ek1L5ThSYRtidkVk4qptrzuUA3AKeVl kksi5KHgoZRuqB8srjtxRrflVptyRa/sU3zVrirSca9OTFFysj5U3y04xztF01iXbiPNy X-Received: by 2002:a17:90b:55cc:b0:2ee:d63f:d73 with SMTP id 98e67ed59e1d1-30e830ebe05mr34040549a91.11.1747805872545; Tue, 20 May 2025 22:37:52 -0700 (PDT) X-Google-Smtp-Source: AGHT+IEXP2fklf6e23xKVRQvwMgPT/9fBODTmL90ObLM3Nw+LKO9HvQaHwq/cFZvYB09LmZmUKu7/7XtQqsomgJ5MkA= X-Received: by 2002:a17:90b:55cc:b0:2ee:d63f:d73 with SMTP id 98e67ed59e1d1-30e830ebe05mr34040356a91.11.1747805869659; Tue, 20 May 2025 22:37:49 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Eugenio Perez Martin Date: Wed, 21 May 2025 07:37:13 +0200 X-Gm-Features: AX0GCFsgz2NNtH_lUwlN2sYPGKU_qZiAxcTww_iK01ju7mMq5edsedajiWUCdUQ Message-ID: Subject: Re: vhost-kernel net on pasta: from 26 to 37Gbit/s To: Jason Wang X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: MTAXONPDbJUTXGJFbY7L3f3coPiEaYc2lUiqZkXSMyA_1747805872 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: Z2BLTDCC7WQ3EYCKKE2Q4F2ZCVPXNULH X-Message-ID-Hash: Z2BLTDCC7WQ3EYCKKE2Q4F2ZCVPXNULH X-MailFrom: eperezma@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Jeff Nelson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, May 21, 2025 at 2:57=E2=80=AFAM Jason Wang wr= ote: > > On Tue, May 20, 2025 at 11:10=E2=80=AFPM Eugenio Perez Martin > wrote: > > > > Hi! > > > > Some updates on the integration. The main culprit was to allow pasta > > to keep reading packets using the regular read() on the tap device. I > > thought that part was completely disabled, but I guess the kernel is > > able to omit the notification on tap as long as the userspace does not > > read it. > > > > My scenario: in different CPUs, all in the same NUMA. I run iperf > > server on CPU 11 with "iperf3 -A 11 -s". All odds CPUs are isolated > > with isolcpus=3D1,3,... nohz=3Don nohz_full=3D1,3,... > > > > With vanilla pasta isolated to CPUs 1,3 with taskset, and just > > --config-net option, and running iperf with "iperf3 -A 5 -c > > 10.6.68.254 -w 8M": > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 30.7 GBytes 26.4 Gbits/sec 0 s= ender > > [ 5] 0.00-10.04 sec 30.7 GBytes 26.3 Gbits/sec r= eceiver > > > > Now trying with the vhost patches we get a slightly worse performance: > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 25.5 GBytes 21.9 Gbits/sec 0 s= ender > > [ 5] 0.00-10.04 sec 25.5 GBytes 21.8 Gbits/sec r= eceiver > > > > Now vhost patch still lacks optimizations like disabling notifications > > or batch more rx available buffer notifications. At the moment it > > refills the rx buffers in each iteration, and does not set the > > no_notify bit which makes the kernel skip the used buffer > > notifications if pasta is actively checking the queue, which is not > > optimal. > > > > Now if I isolate the vhost kernel thread [1] I get way more > > performance as expected: > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 43.1 GBytes 37.1 Gbits/sec 0 s= ender > > [ 5] 0.00-10.04 sec 43.1 GBytes 36.9 Gbits/sec r= eceiver > > > > After analyzing perf output, rep_movs_alternative is the most called > > function in the three iperf3 (~20%Self), passt.avx2 (~15%Self) and > > vhost (~15%Self), But I don't see any of them consuming 100% of CPU in > > top: pasta consumes ~85% %CPU, both iperf3 client and server consumes > > 60%, and vhost consumes ~53%. > > > > So... I have mixed feelings about this :). By "default" it seems to > > have less performance, but my test is maybe too synthetic. There is > > room for improvement with the mentioned optimizations so I'd continue > > applying them, continuing with UDP and TCP zerocopy, and developing > > zerocopy vhost rx. With these numbers I think the series should not be > > merged at the moment. I could send it as RFC if you want but I've not > > applied the comments the first one received, POC style :). > > Have you pinned pasta in a specific CPU? Note that vhost will inherit > the affinity so there could be some contention if you do that. > Yes, pasta was pinned to 1,3 and vhost to 7.