From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=gOsHcORv; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 9803A5A027C for ; Wed, 21 May 2025 12:35:41 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1747823740; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WC+Nd9iGmA713iNn3TNzpMYZFj9KO0FyPMIiq80lSf4=; b=gOsHcORvHZHJVjx7gSOxvNNMQxzi9t2oeRXaDWK482WNuMo8Ik5NSMIEExypZarTa9KKpE jN6bO4ts0lNhzz+nlgpWi5df8sc/4+RVZ5kFZ+MVCYeeE9OUVTqpRZLt5rEEAh32ZZnYSi HGKNktl6XPkQq4ygPunU8yu7+x6KtTA= Received: from mail-pj1-f69.google.com (mail-pj1-f69.google.com [209.85.216.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-463-JLP4ErnuOFetWtdPRVl5ig-1; Wed, 21 May 2025 06:35:39 -0400 X-MC-Unique: JLP4ErnuOFetWtdPRVl5ig-1 X-Mimecast-MFC-AGG-ID: JLP4ErnuOFetWtdPRVl5ig_1747823738 Received: by mail-pj1-f69.google.com with SMTP id 98e67ed59e1d1-30ed0017688so3304333a91.2 for ; Wed, 21 May 2025 03:35:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1747823738; x=1748428538; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WC+Nd9iGmA713iNn3TNzpMYZFj9KO0FyPMIiq80lSf4=; b=XR2K2Vuvj5ceuM/DpuSKZZmK56xoEQTlr8yu0xFdw2Of0/TUbenm/7Sgtpgv5dbwFK EK8AkULN7hnT9Utt5khYKoykPAzAvr5KdZ53H8/8MrQy4jTH7+wWnxrB5Me+0ncCWSqd opGvO1JkBfEvqLSXLJScBLN7G0xRCgpj5bY9mfUCYD7okKdvJxIxSN0VSXQCJxdRRhKm 9S2ShD1O1KbtTROBNAk6TknGry5++Ua3LjGzw00dlq9zUswxAW2Q01oIV1KGuFW3/qw5 HrIcZoQrSoOwVlg6jFLPye0YiD3kN5Z6kG6Ot+XIzqBT+oKJReYjQTs7WN0UceYm27A/ 7XVg== X-Gm-Message-State: AOJu0YzOkoYiOsXjPr83rgPM9jgX+YuRFXjUdHBpBgVWluKb4Vj/QTnY uk10L+ME4AIMPp3V/29SufF6xknzgLs+3vduK6GZOSdkdeu1jIWptPDXQ9wr6WsJXK4SfIE0FMM JVI9gUCPi8Kjzo1YnlanZy5etKbLSkFy3P3wmcyg5WLteNRbUz6GvzPaQLLSmB2KzNJXKqeiFgQ YdsUCUeXmvnMxHukDMQ+xgAceCgQj+ X-Gm-Gg: ASbGncvWFvtw1PSxtqg1w4XOFImohovteV+wnafEBIw/1OuXU8oINRzvo0YXiVY44hb nYTx/dT1iwlcikUKqHv6hQfz2h4obLP2RoVH6JOGHknqVvzataljp/23Sqyl4H5u25jSC X-Received: by 2002:a17:90b:56cc:b0:2fe:8282:cb9d with SMTP id 98e67ed59e1d1-30e7d5a8ab9mr31087342a91.28.1747823738111; Wed, 21 May 2025 03:35:38 -0700 (PDT) X-Google-Smtp-Source: AGHT+IGYz+yNjHCxwevSmQNE4EArxG/pBekVLG9HEuD8jV/p9776t72YEny5VqCbOv/oPyEjTpPgQ8dfNseDN1jxCgo= X-Received: by 2002:a17:90b:56cc:b0:2fe:8282:cb9d with SMTP id 98e67ed59e1d1-30e7d5a8ab9mr31087321a91.28.1747823737695; Wed, 21 May 2025 03:35:37 -0700 (PDT) MIME-Version: 1.0 References: <20250521120855.5cdaeb04@elisabeth> In-Reply-To: <20250521120855.5cdaeb04@elisabeth> From: Eugenio Perez Martin Date: Wed, 21 May 2025 12:35:01 +0200 X-Gm-Features: AX0GCFtmNWGL4PsmT95JLcRHIRbwlS4oU85A_LdWGwTCFv2NNONxU50GAxQcLTs Message-ID: Subject: Re: vhost-kernel net on pasta: from 26 to 37Gbit/s To: Stefano Brivio X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: C7FYs2a0Gkx5A3idg6zYH6ZYE218UA6ICznSbay8_w8_1747823738 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Message-ID-Hash: IQJ3C47JW37WMN2AP2TGJTCVWRFKEQU3 X-Message-ID-Hash: IQJ3C47JW37WMN2AP2TGJTCVWRFKEQU3 X-MailFrom: eperezma@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Jason Wang , Jeff Nelson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, May 21, 2025 at 12:09=E2=80=AFPM Stefano Brivio wrote: > > On Tue, 20 May 2025 17:09:44 +0200 > Eugenio Perez Martin wrote: > > > [...] > > > > Now if I isolate the vhost kernel thread [1] I get way more > > performance as expected: > > - - - - - - - - - - - - - - - - - - - - - - - - - > > [ ID] Interval Transfer Bitrate Retr > > [ 5] 0.00-10.00 sec 43.1 GBytes 37.1 Gbits/sec 0 s= ender > > [ 5] 0.00-10.04 sec 43.1 GBytes 36.9 Gbits/sec r= eceiver > > > > After analyzing perf output, rep_movs_alternative is the most called > > function in the three iperf3 (~20%Self), passt.avx2 (~15%Self) and > > vhost (~15%Self) > > Interesting... s/most called function/function using the most cycles/, I > suppose. > Right! > So it looks somewhat similar to > > https://archives.passt.top/passt-dev/20241017021027.2ac9ea53@elisabeth/ > > now? > Kind of. Below tcp_sendmsg_locked I don't see sk_page_frag_refill but skb_do_copy_data_nocache. Not sure if that means something, as it should not be affected by vhost. > > But I don't see any of them consuming 100% of CPU in > > top: pasta consumes ~85% %CPU, both iperf3 client and server consumes > > 60%, and vhost consumes ~53%. > > > > So... I have mixed feelings about this :). By "default" it seems to > > have less performance, but my test is maybe too synthetic. > > Well, surely we can't ask Podman users to pin specific stuff to given > CPU threads. :) > Yes but maybe the result changes under the right schedule? I'm isolating the CPUs entirely, which is not the usual case for pasta for sure :). > > There is room for improvement with the mentioned optimizations so I'd > > continue applying them, continuing with UDP and TCP zerocopy, and > > developing zerocopy vhost rx. > > That definitely makes sense to me. > Good! > > With these numbers I think the series should not be > > merged at the moment. I could send it as RFC if you want but I've not > > applied the comments the first one received, POC style :). > > I don't think it's really needed for you to spend time on > semi-polishing something just to have an RFC if you're still working on > it. I guess the implementation will change substantially anyway once > you factor in further optimisations. > Agree! I'll keep iterating on this then. Thanks!