From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dE5Na6JZ; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 5B82D5A0280 for ; Fri, 06 Jun 2025 16:33:19 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1749220398; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iB2jK4y/2jBycEyA8datyo0IaJpLu9j9KZvBPQIqhuE=; b=dE5Na6JZuQQEo22zKp8Q0hS4t80l0O57jxwMlpJzTKVI1PJ4m7xT6SjwTWpZyfC39uSBAi SLFA6fpLfPRqPCSCN4MXD5OvlXSnjgL6f4nQSorrhRlME+JTmcAcDPZAeOl7L++TfPHshl rz9/8SkGcHcYiguTukZzftvzBCp2zGo= Received: from mail-pj1-f70.google.com (mail-pj1-f70.google.com [209.85.216.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-487-R5lfXdFsNMu9g5ZayQZWtQ-1; Fri, 06 Jun 2025 10:33:16 -0400 X-MC-Unique: R5lfXdFsNMu9g5ZayQZWtQ-1 X-Mimecast-MFC-AGG-ID: R5lfXdFsNMu9g5ZayQZWtQ_1749220396 Received: by mail-pj1-f70.google.com with SMTP id 98e67ed59e1d1-312eaf676b3so2810924a91.3 for ; Fri, 06 Jun 2025 07:33:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1749220396; x=1749825196; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=iB2jK4y/2jBycEyA8datyo0IaJpLu9j9KZvBPQIqhuE=; b=HvRnbKx+nXZN/82NZ3ZV5oIaP5Qb6vom3XYM7dEE/owd74fBU0bhsC3xcOq++7ILRN o8xhUADQVeNcE/tFCDwWlulS7TyvCoYNE9y1D98WDvVgJvb+KVaJQ05c5qK5nAXYMBPO vMLGHIpqRjMs/LdxOYaVOI61HF4Yv4DhgMUd9bxIDERo22jfFRodkVwDecu1p2axHzSy 38gYc0W3auaEE90R4lE6G+UXQwdoLJqaxQMWEfBd7eggkYCyqlL6YavgqJnBZzbG4wgb NOtij4yt3nQGfFw3iKXNr9QbCiuwDsyJWppQLTH6QFxjNHrjgaRZcp9de4/1r+Xu2iHz IIPA== X-Gm-Message-State: AOJu0Yz+3zLdFm8CxXLauJL02F5RkHK1OUm7/SS3qtpoRpxYvM30PdhC UNfS47XAyx2Ygf2gAAJiy6M4sELJqhVYY3h17GlyDK+tysGzCifoV18vlRirjPM6+JB18SsFJWN JaJSM7XOageTyJ7gAEDFWgV6JWOKLL1wNGGo5/Ex0v+n1MJpclnxx/HQVKRhqfA6X8yWwN/6T7V 2S+q7lKU9Nld9pKApbuZ2W4+wd4DRq X-Gm-Gg: ASbGncu+kUcbdxH2Hel7l7TvVHapwLJZKkdOhCDKzeciOsmgkatRxSUamsjXaVXc34J KnvHYqoAubXfMPEW+j142tik0Q4dRU7Yd6+VNj0iF2lu+Z084vF1S+n9wS4SltpvzHaEIK8Phhb CtxA== X-Received: by 2002:a17:90b:1b51:b0:311:b0ec:135b with SMTP id 98e67ed59e1d1-3134768152amr5664471a91.24.1749220395485; Fri, 06 Jun 2025 07:33:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFZXH3VL1U+PqydBKN7+m4C6t4NPluvYmksBBhnm7JuqaESeJBx++oYVuWFAEqJKVU4ZE370r/qvtVOyGFuTXM= X-Received: by 2002:a17:90b:1b51:b0:311:b0ec:135b with SMTP id 98e67ed59e1d1-3134768152amr5664413a91.24.1749220394868; Fri, 06 Jun 2025 07:33:14 -0700 (PDT) MIME-Version: 1.0 References: <20250521120855.5cdaeb04@elisabeth> In-Reply-To: From: Eugenio Perez Martin Date: Fri, 6 Jun 2025 16:32:38 +0200 X-Gm-Features: AX0GCFv-g7XT2PdJSf6QAA7fN0ijXCFqnk0glnxTforB-lvgRfzcRzEaDE0C-xE Message-ID: Subject: Re: vhost-kernel net on pasta: from 26 to 37Gbit/s To: Stefano Brivio X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: LTb3Uv5rJ6g3vo4EEW1hZnOfF4Y5lSrNFdxUMx98mVY_1749220396 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: FE7W2T6F4IAIPO6TSMZGFSRO25SB5EQB X-Message-ID-Hash: FE7W2T6F4IAIPO6TSMZGFSRO25SB5EQB X-MailFrom: eperezma@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Jason Wang , Jeff Nelson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, May 21, 2025 at 12:35=E2=80=AFPM Eugenio Perez Martin wrote: > > On Wed, May 21, 2025 at 12:09=E2=80=AFPM Stefano Brivio wrote: > > > > On Tue, 20 May 2025 17:09:44 +0200 > > Eugenio Perez Martin wrote: > > > > > [...] > > > > > > Now if I isolate the vhost kernel thread [1] I get way more > > > performance as expected: > > > - - - - - - - - - - - - - - - - - - - - - - - - - > > > [ ID] Interval Transfer Bitrate Retr > > > [ 5] 0.00-10.00 sec 43.1 GBytes 37.1 Gbits/sec 0 = sender > > > [ 5] 0.00-10.04 sec 43.1 GBytes 36.9 Gbits/sec = receiver > > > > > > After analyzing perf output, rep_movs_alternative is the most called > > > function in the three iperf3 (~20%Self), passt.avx2 (~15%Self) and > > > vhost (~15%Self) > > > > Interesting... s/most called function/function using the most cycles/, = I > > suppose. > > > > Right! > > > So it looks somewhat similar to > > > > https://archives.passt.top/passt-dev/20241017021027.2ac9ea53@elisabet= h/ > > > > now? > > > > Kind of. Below tcp_sendmsg_locked I don't see sk_page_frag_refill but > skb_do_copy_data_nocache. Not sure if that means something, as it > should not be affected by vhost. > > > > But I don't see any of them consuming 100% of CPU in > > > top: pasta consumes ~85% %CPU, both iperf3 client and server consumes > > > 60%, and vhost consumes ~53%. > > > > > > So... I have mixed feelings about this :). By "default" it seems to > > > have less performance, but my test is maybe too synthetic. > > > > Well, surely we can't ask Podman users to pin specific stuff to given > > CPU threads. :) > > > > Yes but maybe the result changes under the right schedule? I'm > isolating the CPUs entirely, which is not the usual case for pasta for > sure :). > > > > There is room for improvement with the mentioned optimizations so I'd > > > continue applying them, continuing with UDP and TCP zerocopy, and > > > developing zerocopy vhost rx. > > > > That definitely makes sense to me. > > > > Good! > > > > With these numbers I think the series should not be > > > merged at the moment. I could send it as RFC if you want but I've not > > > applied the comments the first one received, POC style :). > > > > I don't think it's really needed for you to spend time on > > semi-polishing something just to have an RFC if you're still working on > > it. I guess the implementation will change substantially anyway once > > you factor in further optimisations. > > > > Agree! I'll keep iterating on this then. > Actually, if I remove all the taskset etc, and trust the kernel scheduler, vanilla pasta gives me: [pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 = -w 8M Connecting to host 10.6.68.254, port 5201 [ 5] local 10.6.68.20 port 40408 connected to 10.6.68.254 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 3.11 GBytes 26.7 Gbits/sec 0 25.4 MBytes [ 5] 1.00-2.00 sec 3.11 GBytes 26.7 Gbits/sec 0 25.4 MBytes [ 5] 2.00-3.00 sec 3.12 GBytes 26.8 Gbits/sec 0 25.4 MBytes [ 5] 3.00-4.00 sec 3.11 GBytes 26.7 Gbits/sec 0 25.4 MBytes [ 5] 4.00-5.00 sec 3.10 GBytes 26.6 Gbits/sec 0 25.4 MBytes [ 5] 5.00-6.00 sec 3.11 GBytes 26.7 Gbits/sec 0 25.4 MBytes [ 5] 6.00-7.00 sec 3.11 GBytes 26.7 Gbits/sec 0 25.4 MBytes [ 5] 7.00-8.00 sec 3.09 GBytes 26.6 Gbits/sec 0 25.4 MBytes [ 5] 8.00-9.00 sec 3.08 GBytes 26.5 Gbits/sec 0 25.4 MBytes [ 5] 9.00-10.00 sec 3.10 GBytes 26.6 Gbits/sec 0 25.4 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 31.0 GBytes 26.7 Gbits/sec 0 sende= r [ 5] 0.00-10.04 sec 31.0 GBytes 26.5 Gbits/sec recei= ver And with vhost-net : [pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 = -w 8M ... Connecting to host 10.6.68.254, port 5201 [ 5] local 10.6.68.20 port 46720 connected to 10.6.68.254 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.17 GBytes 35.8 Gbits/sec 0 11.9 MBytes [ 5] 1.00-2.00 sec 4.17 GBytes 35.9 Gbits/sec 0 11.9 MBytes [ 5] 2.00-3.00 sec 4.16 GBytes 35.7 Gbits/sec 0 11.9 MBytes [ 5] 3.00-4.00 sec 4.14 GBytes 35.6 Gbits/sec 0 11.9 MBytes [ 5] 4.00-5.00 sec 4.16 GBytes 35.7 Gbits/sec 0 11.9 MBytes [ 5] 5.00-6.00 sec 4.16 GBytes 35.8 Gbits/sec 0 11.9 MBytes [ 5] 6.00-7.00 sec 4.18 GBytes 35.9 Gbits/sec 0 11.9 MBytes [ 5] 7.00-8.00 sec 4.19 GBytes 35.9 Gbits/sec 0 11.9 MBytes [ 5] 8.00-9.00 sec 4.18 GBytes 35.9 Gbits/sec 0 11.9 MBytes [ 5] 9.00-10.00 sec 4.18 GBytes 35.9 Gbits/sec 0 11.9 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 41.7 GBytes 35.8 Gbits/sec 0 sende= r [ 5] 0.00-10.04 sec 41.7 GBytes 35.7 Gbits/sec recei= ver If I go the extra mile and disable notifications (it might be just noise, but...) [pasta@virtlab716 ~]$ /home/passt/pasta --config-net iperf3 -c 10.6.68.254 = -w 8M ... Connecting to host 10.6.68.254, port 5201 [ 5] local 10.6.68.20 port 56590 connected to 10.6.68.254 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 4.19 GBytes 36.0 Gbits/sec 0 12.4 MBytes [ 5] 1.00-2.00 sec 4.18 GBytes 35.9 Gbits/sec 0 12.4 MBytes [ 5] 2.00-3.00 sec 4.18 GBytes 35.9 Gbits/sec 0 12.4 MBytes [ 5] 3.00-4.00 sec 4.20 GBytes 36.1 Gbits/sec 0 12.4 MBytes [ 5] 4.00-5.00 sec 4.21 GBytes 36.2 Gbits/sec 0 12.4 MBytes [ 5] 5.00-6.00 sec 4.21 GBytes 36.1 Gbits/sec 0 12.4 MBytes [ 5] 6.00-7.00 sec 4.20 GBytes 36.1 Gbits/sec 0 12.4 MBytes [ 5] 7.00-8.00 sec 4.23 GBytes 36.4 Gbits/sec 0 12.4 MBytes [ 5] 8.00-9.00 sec 4.24 GBytes 36.4 Gbits/sec 0 12.4 MBytes [ 5] 9.00-10.00 sec 4.21 GBytes 36.2 Gbits/sec 0 12.4 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 42.1 GBytes 36.1 Gbits/sec 0 sende= r [ 5] 0.00-10.04 sec 42.1 GBytes 36.0 Gbits/sec recei= ver So I guess the best is to actually run performance tests closer to real-world workload against the new version and see if it works better? Thanks!