From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=F6hEDhCp; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id CBE485A0008 for ; Tue, 18 Mar 2025 12:07:19 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1742296038; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding:autocrypt:autocrypt; bh=Hpl5wGF2xTcwUG+DkinOmK/MWbOiHETf/aF0nFNXCWU=; b=F6hEDhCpnBZ/7rv+MBDJ/0i+eBqKHij+Cvg72zkdz2bH/pyiKgrONoKRacBRWSK5IK0hEI nzP2Iyr0FHZeCAsIVGgaUlmSgmI52z76K5PsXYLMCByOJ1D/YOSPBwsTY1dauYoGvS2l/t F+t1qL4432XA84pSWxFLesjXaSckwLQ= Received: from mail-wr1-f70.google.com (mail-wr1-f70.google.com [209.85.221.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-221-B6mEM0fcMyGXVgJhMRD0UQ-1; Tue, 18 Mar 2025 07:07:17 -0400 X-MC-Unique: B6mEM0fcMyGXVgJhMRD0UQ-1 X-Mimecast-MFC-AGG-ID: B6mEM0fcMyGXVgJhMRD0UQ_1742296036 Received: by mail-wr1-f70.google.com with SMTP id ffacd0b85a97d-39126c3469fso2511470f8f.3 for ; Tue, 18 Mar 2025 04:07:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1742296036; x=1742900836; h=content-transfer-encoding:autocrypt:subject:from:to :content-language:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Hpl5wGF2xTcwUG+DkinOmK/MWbOiHETf/aF0nFNXCWU=; b=SB85R0m23KPzMelbZbiLRIZ9+osYubUVrfkk88uz+eXIohQrl67jRiyqUqdvK+aGBo vui0Fm5d8IaiAeAfNCikwbzY8hwYEl3cMgFPzvE5V5uxGxjf5J2jvF9SB5gZddN+iSXI Li4iq2oTgq/ThcBldzMBPD9Ol39Of4171cg66grWZ2mMnqFwgScKgn+9wle2KhrPuoOF TG+1Q6MOl1xjsJPRHO6E20lk1sKYN7sEaX2GwR++EzpmY0136VEvNMH+pw8V5m7vgJJy uXxt2Skn7D1zD/HwBvmMJZGiSMQL5UVz6OZQxW2nneh5UQpIu0DSCgzUTj/G6gYkr7uE 28zg== X-Gm-Message-State: AOJu0YxI2gGBrhZAqrUS/DZIhxmC/q0F9I++n1BO4xtA4pl08JiVL8t8 vVjd2fDMhiwv7r/SlvWJYg0z2t6ki+UBBYN0IaVLKBcj3BA1cS2CzFKes2DLcetPtBNTg1FD86+ 4HMSJFJ20hLrlgKwwz3gamoBeKZQCEtusrfyDnDQM2YpBIwqT9/NKp3QqOV4qsfVyJvCbW24+2V Gquyd/XLTGGL/s7ULKw0YTlCY8fflSZu+L X-Gm-Gg: ASbGnctAMxKJtkK7CD+9Bdgt9qn//2kRlAwFAl+nipud1yu4oCnoZgQLTkEOFNrZXFc VnaZV2FZ2rlrt64TKKfK7161CtLH/5+WqBZ3f3upVfbj8mNAbil9KH5ZpT20AgDXxuDm9Kg4zXf ykMXzgA3xIIamgPkaLV6BY8L14LRAIgqoJslZBEtoMJo1GMR8SoNN58OzqeYgO4zcValmIKrRM1 7o1EzAVzMscWqvnD6OGF+MzR8m12HuH1ffLyQFlsYZNtw63EaQYN4MFLsH+mM9fuIlFmP/4VAG+ mCfssSpWC5rsGicZ1brT74gsEmNUmvWfDjwZQHyW/5HG6DQyOE82tA== X-Received: by 2002:a05:6000:184b:b0:391:4743:6db1 with SMTP id ffacd0b85a97d-3971d616f4bmr13866783f8f.20.1742296035728; Tue, 18 Mar 2025 04:07:15 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFYdaeJcrRhkRW6YC3TwP0a51Kqye1j/XtVIqPb6HT8jZrOhEiPXiPggbKPB6cKyeMNn5CTgw== X-Received: by 2002:a05:6000:184b:b0:391:4743:6db1 with SMTP id ffacd0b85a97d-3971d616f4bmr13866759f8f.20.1742296035194; Tue, 18 Mar 2025 04:07:15 -0700 (PDT) Received: from ?IPV6:2a01:e0a:e10:ef90:343a:68f:2e91:95c? ([2a01:e0a:e10:ef90:343a:68f:2e91:95c]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-395cb3189absm18096392f8f.71.2025.03.18.04.07.10 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 18 Mar 2025 04:07:12 -0700 (PDT) Message-ID: <2f3fa74f-523d-4c03-bffb-8d154c40261b@redhat.com> Date: Tue, 18 Mar 2025 12:07:09 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird To: passt-dev@passt.top From: Laurent Vivier Subject: Testing vhost-user with "virtio-net: tweak for better TX performance in NAPI mode" Autocrypt: addr=lvivier@redhat.com; keydata= xsFNBFYFJhkBEAC2me7w2+RizYOKZM+vZCx69GTewOwqzHrrHSG07MUAxJ6AY29/+HYf6EY2 WoeuLWDmXE7A3oJoIsRecD6BXHTb0OYS20lS608anr3B0xn5g0BX7es9Mw+hV/pL+63EOCVm SUVTEQwbGQN62guOKnJJJfphbbv82glIC/Ei4Ky8BwZkUuXd7d5NFJKC9/GDrbWdj75cDNQx UZ9XXbXEKY9MHX83Uy7JFoiFDMOVHn55HnncflUncO0zDzY7CxFeQFwYRbsCXOUL9yBtqLer Ky8/yjBskIlNrp0uQSt9LMoMsdSjYLYhvk1StsNPg74+s4u0Q6z45+l8RAsgLw5OLtTa+ePM JyS7OIGNYxAX6eZk1+91a6tnqfyPcMbduxyBaYXn94HUG162BeuyBkbNoIDkB7pCByed1A7q q9/FbuTDwgVGVLYthYSfTtN0Y60OgNkWCMtFwKxRaXt1WFA5ceqinN/XkgA+vf2Ch72zBkJL RBIhfOPFv5f2Hkkj0MvsUXpOWaOjatiu0fpPo6Hw14UEpywke1zN4NKubApQOlNKZZC4hu6/ 8pv2t4HRi7s0K88jQYBRPObjrN5+owtI51xMaYzvPitHQ2053LmgsOdN9EKOqZeHAYG2SmRW LOxYWKX14YkZI5j/TXfKlTpwSMvXho+efN4kgFvFmP6WT+tPnwARAQABzSNMYXVyZW50IFZp dmllciA8bHZpdmllckByZWRoYXQuY29tPsLBeAQTAQIAIgUCVgVQgAIbAwYLCQgHAwIGFQgC CQoLBBYCAwECHgECF4AACgkQ8ww4vT8vvjwpgg//fSGy0Rs/t8cPFuzoY1cex4limJQfReLr SJXCANg9NOWy/bFK5wunj+h/RCFxIFhZcyXveurkBwYikDPUrBoBRoOJY/BHK0iZo7/WQkur 6H5losVZtrotmKOGnP/lJYZ3H6OWvXzdz8LL5hb3TvGOP68K8Bn8UsIaZJoeiKhaNR0sOJyI YYbgFQPWMHfVwHD/U+/gqRhD7apVysxv5by/pKDln1I5v0cRRH6hd8M8oXgKhF2+rAOL7gvh jEHSSWKUlMjC7YwwjSZmUkL+TQyE18e2XBk85X8Da3FznrLiHZFHQ/NzETYxRjnOzD7/kOVy gKD/o7asyWQVU65mh/ECrtjfhtCBSYmIIVkopoLaVJ/kEbVJQegT2P6NgERC/31kmTF69vn8 uQyW11Hk8tyubicByL3/XVBrq4jZdJW3cePNJbTNaT0d/bjMg5zCWHbMErUib2Nellnbg6bc 2HLDe0NLVPuRZhHUHM9hO/JNnHfvgiRQDh6loNOUnm9Iw2YiVgZNnT4soUehMZ7au8PwSl4I KYE4ulJ8RRiydN7fES3IZWmOPlyskp1QMQBD/w16o+lEtY6HSFEzsK3o0vuBRBVp2WKnssVH qeeV01ZHw0bvWKjxVNOksP98eJfWLfV9l9e7s6TaAeySKRRubtJ+21PRuYAxKsaueBfUE7ZT 7zfOwU0EVgUmGQEQALxSQRbl/QOnmssVDxWhHM5TGxl7oLNJms2zmBpcmlrIsn8nNz0rRyxT 460k2niaTwowSRK8KWVDeAW6ZAaWiYjLlTunoKwvF8vP3JyWpBz0diTxL5o+xpvy/Q6YU3BN efdq8Vy3rFsxgW7mMSrI/CxJ667y8ot5DVugeS2NyHfmZlPGE0Nsy7hlebS4liisXOrN3jFz asKyUws3VXek4V65lHwB23BVzsnFMn/bw/rPliqXGcwl8CoJu8dSyrCcd1Ibs0/Inq9S9+t0 VmWiQWfQkz4rvEeTQkp/VfgZ6z98JRW7S6l6eophoWs0/ZyRfOm+QVSqRfFZdxdP2PlGeIFM C3fXJgygXJkFPyWkVElr76JTbtSHsGWbt6xUlYHKXWo+xf9WgtLeby3cfSkEchACrxDrQpj+ Jt/JFP+q997dybkyZ5IoHWuPkn7uZGBrKIHmBunTco1+cKSuRiSCYpBIXZMHCzPgVDjk4viP brV9NwRkmaOxVvye0vctJeWvJ6KA7NoAURplIGCqkCRwg0MmLrfoZnK/gRqVJ/f6adhU1oo6 z4p2/z3PemA0C0ANatgHgBb90cd16AUxpdEQmOCmdNnNJF/3Zt3inzF+NFzHoM5Vwq6rc1JP jfC3oqRLJzqAEHBDjQFlqNR3IFCIAo4SYQRBdAHBCzkM4rWyRhuVABEBAAHCwV8EGAECAAkF AlYFJhkCGwwACgkQ8ww4vT8vvjwg9w//VQrcnVg3TsjEybxDEUBm8dBmnKqcnTBFmxN5FFtI WlEuY8+YMiWRykd8Ln9RJ/98/ghABHz9TN8TRo2b6WimV64FmlVn17Ri6FgFU3xNt9TTEChq AcNg88eYryKsYpFwegGpwUlaUaaGh1m9OrTzcQy+klVfZWaVJ9Nw0keoGRGb8j4XjVpL8+2x OhXKrM1fzzb8JtAuSbuzZSQPDwQEI5CKKxp7zf76J21YeRrEW4WDznPyVcDTa+tz++q2S/Bp P4W98bXCBIuQgs2m+OflERv5c3Ojldp04/S4NEjXEYRWdiCxN7ca5iPml5gLtuvhJMSy36gl U6IW9kn30IWuSoBpTkgV7rLUEhh9Ms82VWW/h2TxL8enfx40PrfbDtWwqRID3WY8jLrjKfTd R3LW8BnUDNkG+c4FzvvGUs8AvuqxxyHbXAfDx9o/jXfPHVRmJVhSmd+hC3mcQ+4iX5bBPBPM oDqSoLt5w9GoQQ6gDVP2ZjTWqwSRMLzNr37rJjZ1pt0DCMMTbiYIUcrhX8eveCJtY7NGWNyx FCRkhxRuGcpwPmRVDwOl39MB3iTsRighiMnijkbLXiKoJ5CDVvX5yicNqYJPKh5MFXN1bvsB kmYiStMRbrD0HoY1kx5/VozBtc70OU0EB8Wrv9hZD+Ofp0T3KOr1RUHvCZoLURfFhSQ= X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: 0_rip_OxEHEC-57krzPl-PuwHVkNdoix-lJwixQr51s_1742296036 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: UNEGLSOME2SHFINP3KYSA4IQK4PNIP2Q X-Message-ID-Hash: UNEGLSOME2SHFINP3KYSA4IQK4PNIP2Q X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Hi, as reported by Stefano there is an asymmetry in the throughput between host and guest with vhost-user. I've tested the following kernel patch from Jason to see if it can improve the performance: ------------------------------------------------------------------------------ commit e13b6da7045f997e1a5a5efd61d40e63c4fc20e8 Author: Jason Wang Date: Tue Feb 18 10:39:08 2025 +0800 virtio-net: tweak for better TX performance in NAPI mode There are several issues existed in start_xmit(): - Transmitted packets need to be freed before sending a packet, this introduces delay and increases the average packets transmit time. This also increase the time that spent in holding the TX lock. - Notification is enabled after free_old_xmit_skbs() which will introduce unnecessary interrupts if TX notification happens on the same CPU that is doing the transmission now (actually, virtio-net driver are optimized for this case). So this patch tries to avoid those issues by not cleaning transmitted packets in start_xmit() when TX NAPI is enabled and disable notifications even more aggressively. Notification will be since the beginning of the start_xmit(). But we can't enable delayed notification after TX is stopped as we will lose the notifications. Instead, the delayed notification needs is enabled after the virtqueue is kicked for best performance. Performance numbers: 1) single queue 2 vcpus guest with pktgen_sample03_burst_single_flow.sh (burst 256) + testpmd (rxonly) on the host: - When pinning TX IRQ to pktgen VCPU: split virtqueue PPS were increased 55% from 6.89 Mpps to 10.7 Mpps and 32% TX interrupts were eliminated. Packed virtqueue PPS were increased 50% from 7.09 Mpps to 10.7 Mpps, 99% TX interrupts were eliminated. - When pinning TX IRQ to VCPU other than pktgen: split virtqueue PPS were increased 96% from 5.29 Mpps to 10.4 Mpps and 45% TX interrupts were eliminated; Packed virtqueue PPS were increased 78% from 6.12 Mpps to 10.9 Mpps and 99% TX interrupts were eliminated. 2) single queue 1 vcpu guest + vhost-net/TAP on the host: single session netperf from guest to host shows 82% improvement from 31Gb/s to 58Gb/s, %stddev were reduced from 34.5% to 1.9% and 88% of TX interrupts were eliminated. Signed-off-by: Jason Wang Acked-by: Michael S. Tsirkin Signed-off-by: David S. Miller ------------------------------------------------------------------------------ systemctl stop firewalld.service || service iptables stop || iptables -Ft /sbin/sysctl -w net.core.rmem_max=536870912 /sbin/sysctl -w net.core.wmem_max=536870912 ____ I made my test using 6.14-rc7 kernel: From guest: iperf3 -c 10.6.68.254 -P2 -Z -t5 -l 1M -w 16M [SUM] 0.00-5.00 sec 14.5 GBytes 24.9 Gbits/sec 0 sender [SUM] 0.00-5.00 sec 14.5 GBytes 24.9 Gbits/sec receiver From host: iperf3 -c localhost -P2 -Z -t5 -p 10001 -l 1M -w 16M [SUM] 0.00-5.00 sec 28.9 GBytes 49.6 Gbits/sec 0 sender [SUM] 0.00-5.03 sec 28.8 GBytes 49.2 Gbits/sec receiver ____ The results with a 6.14-rc7 + e13b6da7045f: From guest: iperf3 -c 10.6.68.254 -P2 -Z -t5 -l 1M -w 16M [SUM] 0.00-5.00 sec 14.8 GBytes 25.4 Gbits/sec 0 sender [SUM] 0.00-5.01 sec 14.8 GBytes 25.4 Gbits/sec receiver From host: iperf3 -c localhost -P2 -Z -t5 -p 10001 -l 1M -w 16M [SUM] 0.00-5.00 sec 28.5 GBytes 48.9 Gbits/sec 0 sender [SUM] 0.00-5.03 sec 28.4 GBytes 48.6 Gbits/sec receiver We have only a 2% improvement. Thanks, Laurent