From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=ILX6a36g; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTP id 4F7375A004C for ; Wed, 09 Oct 2024 19:37:37 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728495456; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=w3r3uC/BmE7aQV7rUxv1kya01n9RlyWawY9Te0KzZdQ=; b=ILX6a36gI1Z11BaeCCH/+3J0zlq1yLe0CoYOhaByNgWJKI83I5GiCkERcBD4kezth6WgC/ sO7HarvahcbYP/fSlSWZHrhNmV6oZmFOc03VEhp9FYgMJBs6ghUZbvsGt+4shMFqVNsOU0 5zngpdWa4HIig9vz7WLQOOmezIusRXg= Received: from mail-pf1-f197.google.com (mail-pf1-f197.google.com [209.85.210.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-601-5Pb4vGfINaSQ53Df0pdqtA-1; Wed, 09 Oct 2024 13:37:35 -0400 X-MC-Unique: 5Pb4vGfINaSQ53Df0pdqtA-1 Received: by mail-pf1-f197.google.com with SMTP id d2e1a72fcca58-71e123d3d22so52423b3a.3 for ; Wed, 09 Oct 2024 10:37:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728495453; x=1729100253; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=YE37jAPGZfBBQoPPSK8ZUAI9blxpuzi1edWqQiV0Gpo=; b=xKBc869T/NM/tgMrBcBfXUNuPrOhXC0oN85bSRAfsJNGVOtBIunnD/G+Wiytg7o18P nIRl0TrM46LVTAY6ZW+DHPXnWdwVmsvgqUf5QJI9Q1I2y6YF1nqQXr+rq/ePZoIqT0kL NsutFTnbS7DIsAcNRJPH0T0BPwd0kQN3Qmn6rA6rB+wFtaz+dAPQ60cQpXDeMyPQBg07 cAcnLo7tUpPrO9KGYSlOBLMFmnUXQbdnWLo/CYR29j+jVaZTydT1QdP8ShAMEjt6Gyf4 YM1UO0gfWFQKGBc2tN3GoApmr6pBOV/UH/Ecex3NclBL4FDTtQQMNnYr+pQ6BEfvQEIo 5OoA== X-Gm-Message-State: AOJu0YzfYr/HOGciKlo5nK5LHONWkheB9Cu/g0HyyanBke8aYHLxx9rs 66aZ0nTrrSeDmlaY+4ZH6Umw/G24sLDyWKqv5LNR5iB7DObpeAnxe7mnTlNBELB/NK+aK5ehRrK d0CIAC/3NRiw6I57/NR3vioyrTLszl5J0dCI1yA5JlPA/32W42cMaljF9DS5XmtlxmR/q995ABy KiwfuNGCW923LW94WU52XridUChxNutWuw X-Received: by 2002:a05:6a00:1ac6:b0:71d:f5fd:4de4 with SMTP id d2e1a72fcca58-71e1dbb5338mr4746914b3a.17.1728495453037; Wed, 09 Oct 2024 10:37:33 -0700 (PDT) X-Google-Smtp-Source: AGHT+IHNJZ2BJYzBLNAPyM/w4mYpc5XoDyFon2SLPEp5wabFlgylWisYqPb2DWARFG6gOaV+lWWt+g== X-Received: by 2002:a05:6a00:1ac6:b0:71d:f5fd:4de4 with SMTP id d2e1a72fcca58-71e1dbb5338mr4746871b3a.17.1728495452477; Wed, 09 Oct 2024 10:37:32 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71df0ccdbf7sm8021628b3a.56.2024.10.09.10.37.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Oct 2024 10:37:31 -0700 (PDT) Date: Wed, 9 Oct 2024 19:37:26 +0200 From: Stefano Brivio To: Laurent Vivier Subject: Re: [PATCH v7 0/8] Add vhost-user support to passt. (part 3) Message-ID: <20241009193726.7e2e5790@elisabeth> In-Reply-To: <20241009090716.691361-1-lvivier@redhat.com> References: <20241009090716.691361-1-lvivier@redhat.com> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: X2IYHBYL4EBB2XC43QP4S27ZUVLBGXMW X-Message-ID-Hash: X2IYHBYL4EBB2XC43QP4S27ZUVLBGXMW X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 9 Oct 2024 11:07:07 +0200 Laurent Vivier wrote: > This series of patches adds vhost-user support to passt > and then allows passt to connect to QEMU network backend using > virtqueue rather than a socket. >=20 > With QEMU, rather than using to connect: >=20 > -netdev stream,id=3Ds,server=3Doff,addr.type=3Dunix,addr.path=3D/tmp/pa= sst_1.socket >=20 > we will use: >=20 > -chardev socket,id=3Dchr0,path=3D/tmp/passt_1.socket > -netdev vhost-user,id=3Dnetdev0,chardev=3Dchr0 > -device virtio-net,netdev=3Dnetdev0 > -object memory-backend-memfd,id=3Dmemfd0,share=3Don,size=3D$RAMSIZE > -numa node,memdev=3Dmemfd0 >=20 > The memory backend is needed to share data between passt and QEMU. >=20 > Performance comparison between "-netdev stream" and "-netdev vhost-user": On my setup, with a few tweaks (don't ask me why... we should figure out eventually): -- diff --git a/test/perf/passt_vu_tcp b/test/perf/passt_vu_tcp index b434008..76bdd48 100644 --- a/test/perf/passt_vu_tcp +++ b/test/perf/passt_vu_tcp @@ -38,10 +38,10 @@ hout=09FREQ_PROCFS (echo "scale=3D1"; sed -n 's/cpu MHz= .*: \([0-9]*\)\..*$/(\1+10^2\ hout=09FREQ_CPUFREQ (echo "scale=3D1"; printf '( %i + 10^5 / 2 ) / 10^6\n'= $(cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq) ) | bc -l hout=09FREQ [ -n "__FREQ_CPUFREQ__" ] && echo __FREQ_CPUFREQ__ || echo __F= REQ_PROCFS__ =20 -set=09THREADS 4 +set=09THREADS 4-6 set=09TIME 5 set=09OMIT 0.1 -set=09OPTS -Z -P __THREADS__ -l 1M -O__OMIT__ -N +set=09OPTS -Z -O__OMIT__ -N =20 info=09Throughput in Gbps, latency in =C2=B5s, __THREADS__ threads at __FR= EQ__ GHz report=09passt_vu tcp __THREADS__ __FREQ__ @@ -55,16 +55,16 @@ iperf3s=09ns 10002 bw=09- bw=09- guest=09ip link set dev __IFNAME__ mtu 1280 -iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 16M +iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 16M -l 1M -P 4 bw=09__BW__ 1.2 1.5 guest=09ip link set dev __IFNAME__ mtu 1500 -iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 32M +iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 32M -l 1M -P 4 bw=09__BW__ 1.6 1.8 guest=09ip link set dev __IFNAME__ mtu 9000 -iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 64M +iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 64M -l 1M -P 4 bw=09__BW__ 4.0 5.0 guest=09ip link set dev __IFNAME__ mtu 65520 -iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 64M +iperf3=09BW guest __MAP_NS6__ 10002 __TIME__ __OPTS__ -w 256M -l 1M -P 4 bw=09__BW__ 7.0 8.0 =20 iperf3k=09ns @@ -93,22 +93,22 @@ tr=09TCP throughput over IPv4: guest to host iperf3s=09ns 10002 =20 guest=09ip link set dev __IFNAME__ mtu 256 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 2M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 2M -l 1M -P 4 bw=09__BW__ 0.2 0.3 guest=09ip link set dev __IFNAME__ mtu 576 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 4M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 4M -l 1M -P 4 bw=09__BW__ 0.5 0.8 guest=09ip link set dev __IFNAME__ mtu 1280 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 8M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 8M -l 1M -P 4 bw=09__BW__ 1.2 1.5 guest=09ip link set dev __IFNAME__ mtu 1500 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 16M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 16M -l 1M -P 4 bw=09__BW__ 1.6 1.8 guest=09ip link set dev __IFNAME__ mtu 9000 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 64M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 64M -l 1M -P 4 bw=09__BW__ 4.0 5.0 guest=09ip link set dev __IFNAME__ mtu 65520 -iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 64M +iperf3=09BW guest __MAP_NS4__ 10002 __TIME__ __OPTS__ -w 256M -l 1M -P 4 bw=09__BW__ 7.0 8.0 =20 iperf3k=09ns @@ -145,7 +145,7 @@ bw=09- bw=09- bw=09- bw=09- -iperf3=09BW ns ::1 10001 __TIME__ __OPTS__ -w 32M +iperf3=09BW ns ::1 10001 __TIME__ __OPTS__ -w 256M -l 16k -P 6 bw=09__BW__ 6.0 6.8 =20 iperf3k=09guest @@ -181,7 +181,7 @@ bw=09- bw=09- bw=09- bw=09- -iperf3=09BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -w 32M +iperf3=09BW ns 127.0.0.1 10001 __TIME__ __OPTS__ -w 256M -l 16k -P 6 bw=09__BW__ 6.0 6.8 =20 iperf3k=09guest -- I'm getting an even bigger improvement in throughput (and also significantly lower latency). Non-vhost-user first: -- =3D=3D=3D perf/passt_tcp > passt: throughput and latency Throughput in Gbps, latency in =C2=B5s, 4 threads at 3.6 GHz MTU: | 256B | 576B= | 1280B | 1500B | 9000B | 65520B | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv6: guest to host | - | = - | 6.3 | 6.8 | 18.4 | 21.4 | TCP RR latency over IPv6: guest to host | - | = - | - | - | - | 52 | TCP CRR latency over IPv6: guest to host | - | = - | - | - | - | 141 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv4: guest to host | 0.8 | 3.= 0 | 5.6 | 7.4 | 19.6 | 21.3 | TCP RR latency over IPv4: guest to host | - | = - | - | - | - | 58 | TCP CRR latency over IPv4: guest to host | - | = - | - | - | - | 132 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv6: host to guest | - | = - | - | - | - | 18.0 | TCP RR latency over IPv6: host to guest | - | = - | - | - | - | 50 | TCP CRR latency over IPv6: host to guest | - | = - | - | - | - | 115 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv4: host to guest | - | = - | - | - | - | 17.8 | TCP RR latency over IPv4: host to guest | - | = - | - | - | - | 60 | TCP CRR latency over IPv6: host to guest | - | = - | - | - | - | 94 | '--------'------= --'--------'--------'--------'--------' ...passed. =3D=3D=3D perf/passt_udp > passt: throughput and latency Throughput in Gbps, latency in =C2=B5s, 2 threads at 3.6 GHz pktlen: | 256B | 576B= | 1280B | 1500B | 9000B | 65520B | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv6: guest to host | - | = - | 3.4 | 4.1 | 12.3 | 18.2 | UDP RR latency over IPv6: guest to host | - | = - | - | - | - | 49 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv4: guest to host | 0.8 | 1.= 9 | 3.7 | 4.0 | 11.1 | 17.2 | UDP RR latency over IPv4: guest to host | - | = - | - | - | - | 52 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv6: host to guest | - | = - | 2.6 | 3.1 | 5.4 | 17.9 | UDP RR latency over IPv6: host to guest | - | = - | - | - | - | 48 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv4: host to guest | 0.9 | 2.= 3 | 5.6 | 7.4 | 12.8 | 16.6 | UDP RR latency over IPv4: host to guest | - | = - | - | - | - | 48 | '--------'------= --'--------'--------'--------'--------' ...passed. [...] =3D=3D=3D perf/passt_vu_tcp > passt: throughput and latency Throughput in Gbps, latency in =C2=B5s, 4-6 threads at 3.6 GHz MTU: | 256B | 576B= | 1280B | 1500B | 9000B | 65520B | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv6: guest to host | - | = - | 8.2 | 10.1 | 14.7 | 22.3 | TCP RR latency over IPv6: guest to host | - | = - | - | - | - | 30 | TCP CRR latency over IPv6: guest to host | - | = - | - | - | - | 88 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv4: guest to host | 1.2 | 5.= 3 | 9.2 | 10.1 | 18.5 | 23.7 | TCP RR latency over IPv4: guest to host | - | = - | - | - | - | 31 | TCP CRR latency over IPv4: guest to host | - | = - | - | - | - | 93 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv6: host to guest | - | = - | - | - | - | 42.1 | TCP RR latency over IPv6: host to guest | - | = - | - | - | - | 30 | TCP CRR latency over IPv6: host to guest | - | = - | - | - | - | 88 | |--------|------= --|--------|--------|--------|--------| TCP throughput over IPv4: host to guest | - | = - | - | - | - | 48.8 | TCP RR latency over IPv4: host to guest | - | = - | - | - | - | 35 | TCP CRR latency over IPv6: host to guest | - | = - | - | - | - | 79 | '--------'------= --'--------'--------'--------'--------' ...passed. =3D=3D=3D perf/passt_vu_udp > passt: throughput and latency Throughput in Gbps, latency in =C2=B5s, 2 threads at 3.6 GHz pktlen: | 256B | 576B= | 1280B | 1500B | 9000B | 65520B | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv6: guest to host | - | = - | 2.2 | 2.6 | 14.1 | 33.4 | UDP RR latency over IPv6: guest to host | - | = - | - | - | - | 32 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv4: guest to host | 0.4 | 1.= 1 | 2.6 | 2.3 | 13.6 | 28.9 | UDP RR latency over IPv4: guest to host | - | = - | - | - | - | 31 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv6: host to guest | - | = - | 3.1 | 3.7 | 18.7 | 29.4 | UDP RR latency over IPv6: host to guest | - | = - | - | - | - | 35 | |--------|------= --|--------|--------|--------|--------| UDP throughput over IPv4: host to guest | 0.5 | 1.= 3 | 3.3 | 3.8 | 18.6 | 37.5 | UDP RR latency over IPv4: host to guest | - | = - | - | - | - | 35 | '--------'------= --'--------'--------'--------'--------' ...passed. -- passt is CPU-bound only on host-to-guest tests. But there, iperf3 seems to actually use more CPU time than passt itself. --=20 Stefano