From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202510 header.b=tM/mEn8e; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id C198B5A0271 for ; Wed, 26 Nov 2025 04:17:03 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202510; t=1764127020; bh=8oTzlSmQklXemQONtOPxhBrv/dyNdGmLhojCSXp6sqg=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=tM/mEn8eVItkKks9n9GpwxH/OKHWsl1cH2PXmOEfJHSB8caZwV1Ra40DEKPJJ2XlI rrBX0iAPiCyVlQI2ALYq84ClqKUwNfntIJYdBzBHerkCGyN1cBthCphOxDFb4JRka6 5jSTHSzPec5DIGp5+DEtWUjdpO4zDAV0TkcIYNfqGrJy9gnoh7gXi5wBk3h20lCLur fiaRhmq1xYfRrfujwXjuaVZeUUNxGlglaobPRza7eTwHLEk4z17rh1Fd5qcuPMJD00 /arj/TCQtv9bpYJZ6y4slkMc1LWzObIn1vQSgUOj3LlPbstaIfj5gsWnAIqLdEPTzQ 3424FXAVq3l3w== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4dGPqc0bD7z4wGT; Wed, 26 Nov 2025 14:17:00 +1100 (AEDT) Date: Wed, 26 Nov 2025 13:20:41 +1100 From: David Gibson To: Laurent Vivier Subject: Re: [PATCH v2 2/6] vhost-user: Enable multiqueue Message-ID: References: <20251121165902.1014964-1-lvivier@redhat.com> <20251121165902.1014964-3-lvivier@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="IdGHezylb0OjrLow" Content-Disposition: inline In-Reply-To: <20251121165902.1014964-3-lvivier@redhat.com> Message-ID-Hash: IEJAK63YU2J74NYTB7SFCKH7V75Y33Q7 X-Message-ID-Hash: IEJAK63YU2J74NYTB7SFCKH7V75Y33Q7 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --IdGHezylb0OjrLow Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Nov 21, 2025 at 05:58:58PM +0100, Laurent Vivier wrote: > Add the --max-queues parameter to specify the maximum number of > queue Nit: you updated the option to be --max-qpairs, which I think is good, but now the commit message is out of date. One other query: what makes it "max qpairs" rather than just "qpairs" - are there (now or in planned work) circumstances where you'd end up with less qpairs than specified here? > pairs supported in vhost-user mode. This enables multi-queue support > by allowing configuration of up to 16 queue pairs (32 virtqueues). >=20 > For the moment, only the first RX queue is used, the TX queue is > selected by the guest kernel. IIUC, with this patch (but not the ones after) things will break if the guest uses a qpair other than 0, right? AFAICT vu_kick_cb() isn't updated so will ignore anything on the other qpairs. >=20 > Signed-off-by: Laurent Vivier > --- > conf.c | 31 ++++++++++++++++++++++++++++++- > passt.h | 2 ++ > tap.c | 15 +++++++++++++-- > vhost_user.c | 38 +++++++++++++++++++++----------------- > virtio.h | 2 +- > 5 files changed, 67 insertions(+), 21 deletions(-) >=20 > diff --git a/conf.c b/conf.c > index 66b9e63400ec..99a7995f8038 100644 > --- a/conf.c > +++ b/conf.c > @@ -862,7 +862,9 @@ static void usage(const char *name, FILE *f, int stat= us) > " --vhost-user Enable vhost-user mode\n" > " UNIX domain socket is provided by -s option\n" > " --print-capabilities print back-end capabilities in JSON format,\n" > - " only meaningful for vhost-user mode\n"); > + " only meaningful for vhost-user mode\n" > + " --max-qpairs Specify the maximum number of queue pairs\n" > + ); > FPRINTF(f, > " --repair-path PATH path for passt-repair(1)\n" > " default: append '.repair' to UNIX domain path\n"); > @@ -1483,6 +1485,7 @@ void conf(struct ctx *c, int argc, char **argv) > {"migrate-exit", no_argument, NULL, 29 }, > {"migrate-no-linger", no_argument, NULL, 30 }, > {"stats", required_argument, NULL, 31 }, > + {"max-qpairs", required_argument, NULL, 32 }, > { 0 }, > }; > const char *optstring =3D "+dqfel:hs:F:I:p:P:m:a:n:M:g:i:o:D:S:H:461t:u= :T:U:"; > @@ -1514,6 +1517,7 @@ void conf(struct ctx *c, int argc, char **argv) > c->tcp.fwd_in.mode =3D c->tcp.fwd_out.mode =3D FWD_UNSET; > c->udp.fwd_in.mode =3D c->udp.fwd_out.mode =3D FWD_UNSET; > memcpy(c->our_tap_mac, MAC_OUR_LAA, ETH_ALEN); > + c->max_qpairs =3D 1; > =20 > optind =3D 0; > do { > @@ -1717,6 +1721,31 @@ void conf(struct ctx *c, int argc, char **argv) > die("Can't display statistics if not running in foreground"); > c->stats =3D strtol(optarg, NULL, 0); > break; > + case 32: { > + unsigned long max_qpairs; > + char *e; > + > + if (c->mode !=3D MODE_VU) > + die("--max-qpairs is for vhost-user mode only"); > + > + errno =3D 0; > + max_qpairs =3D strtoul(optarg, &e, 0); > + > + if (errno || *e) > + die("Invalid max-qpairs: %s", optarg); > + > + if (max_qpairs < 1) { > + die("max-qpairs %lu too small (min 1)", > + max_qpairs); > + } > + > + if (max_qpairs * 2 > VHOST_USER_MAX_VQS) { > + die("max-qpairs %lu too big (maximum %u)", > + max_qpairs, VHOST_USER_MAX_VQS / 2); > + } > + c->max_qpairs =3D max_qpairs; > + break; > + } > case 'd': > c->debug =3D 1; > c->quiet =3D 0; > diff --git a/passt.h b/passt.h > index 15801b44bfa8..c7b6dad69190 100644 > --- a/passt.h > +++ b/passt.h > @@ -205,6 +205,7 @@ struct ip6_ctx { > * @low_wmem: Low probed net.core.wmem_max > * @low_rmem: Low probed net.core.rmem_max > * @vdev: vhost-user device > + * @max_qpairs: Maximum number of queue pairs > * @device_state_fd: Device state migration channel > * @device_state_result: Device state migration result > * @migrate_target: Are we the target, on the next migration request? > @@ -283,6 +284,7 @@ struct ctx { > int low_rmem; > =20 > struct vu_dev *vdev; > + unsigned int max_qpairs; > =20 > /* Migration */ > int device_state_fd; > diff --git a/tap.c b/tap.c > index d7f777fd0b19..d098061ed559 100644 > --- a/tap.c > +++ b/tap.c > @@ -1314,8 +1314,19 @@ static void tap_backend_show_hints(struct ctx *c) > break; > case MODE_VU: > info("You can start qemu with:"); > - info(" kvm ... -chardev socket,id=3Dchr0,path=3D%s -netdev vhost-us= er,id=3Dnetdev0,chardev=3Dchr0 -device virtio-net,netdev=3Dnetdev0 -object = memory-backend-memfd,id=3Dmemfd0,share=3Don,size=3D$RAMSIZE -numa node,memd= ev=3Dmemfd0\n", > - c->sock_path); > + if (c->max_qpairs > 1) { > + info(" kvm ... -chardev socket,id=3Dchr0,path=3D%s " > + "-netdev vhost-user,id=3Dnetdev0,chardev=3Dchr0,queues=3D%d " > + "-device virtio-net,netdev=3Dnetdev0,mq=3Dtrue " > + "-object memory-backend-memfd,id=3Dmemfd0,share=3Don,size=3D$RAM= SIZE " > + "-numa node,memdev=3Dmemfd0\n", c->sock_path, c->max_qpairs); > + } else { > + info(" kvm ... -chardev socket,id=3Dchr0,path=3D%s " > + "-netdev vhost-user,id=3Dnetdev0,chardev=3Dchr0 " > + "-device virtio-net,netdev=3Dnetdev0 " > + "-object memory-backend-memfd,id=3Dmemfd0,share=3Don,size=3D$RAM= SIZE " > + "-numa node,memdev=3Dmemfd0\n", c->sock_path); > + } > break; > } > } > diff --git a/vhost_user.c b/vhost_user.c > index aa7c869d9e56..6d3fa04d2119 100644 > --- a/vhost_user.c > +++ b/vhost_user.c > @@ -323,6 +323,7 @@ static bool vu_get_features_exec(struct vu_dev *vdev, > uint64_t features =3D > 1ULL << VIRTIO_F_VERSION_1 | > 1ULL << VIRTIO_NET_F_MRG_RXBUF | > + 1ULL << VIRTIO_NET_F_MQ | > 1ULL << VHOST_F_LOG_ALL | > 1ULL << VHOST_USER_F_PROTOCOL_FEATURES; > =20 > @@ -342,9 +343,9 @@ static bool vu_get_features_exec(struct vu_dev *vdev, > */ > static void vu_set_enable_all_rings(struct vu_dev *vdev, bool enable) > { > - uint16_t i; > + unsigned int i; > =20 > - for (i =3D 0; i < VHOST_USER_MAX_VQS; i++) > + for (i =3D 0; i < vdev->context->max_qpairs * 2; i++) > vdev->vq[i].enable =3D enable; > } > =20 > @@ -476,7 +477,7 @@ static bool vu_set_mem_table_exec(struct vu_dev *vdev, > close(vmsg->fds[i]); > } > =20 > - for (i =3D 0; i < VHOST_USER_MAX_VQS; i++) { > + for (i =3D 0; i < vdev->context->max_qpairs * 2; i++) { > if (vdev->vq[i].vring.desc) { > if (map_ring(vdev, &vdev->vq[i])) > die("remapping queue %d during setmemtable", i); > @@ -759,15 +760,18 @@ static void vu_set_watch(const struct vu_dev *vdev,= int idx) > /** > * vu_check_queue_msg_file() - Check if a message is valid, > * close fds if NOFD bit is set > + * @vdev: vhost-user device > * @vmsg: vhost-user message > */ > -static void vu_check_queue_msg_file(struct vhost_user_msg *vmsg) > +static void vu_check_queue_msg_file(const struct vu_dev *vdev, > + struct vhost_user_msg *vmsg) > { > bool nofd =3D vmsg->payload.u64 & VHOST_USER_VRING_NOFD_MASK; > - int idx =3D vmsg->payload.u64 & VHOST_USER_VRING_IDX_MASK; > + unsigned int idx =3D vmsg->payload.u64 & VHOST_USER_VRING_IDX_MASK; > =20 > - if (idx >=3D VHOST_USER_MAX_VQS) > - die("Invalid vhost-user queue index: %u", idx); > + if (idx >=3D vdev->context->max_qpairs * 2) > + die("Invalid vhost-user queue index: %u (maximum %u)", idx, > + vdev->context->max_qpairs * 2); > =20 > if (nofd) { > vmsg_close_fds(vmsg); > @@ -794,7 +798,7 @@ static bool vu_set_vring_kick_exec(struct vu_dev *vde= v, > =20 > debug("u64: 0x%016"PRIx64, vmsg->payload.u64); > =20 > - vu_check_queue_msg_file(vmsg); > + vu_check_queue_msg_file(vdev, vmsg); > =20 > if (vdev->vq[idx].kick_fd !=3D -1) { > epoll_del(vdev->context->epollfd, vdev->vq[idx].kick_fd); > @@ -834,7 +838,7 @@ static bool vu_set_vring_call_exec(struct vu_dev *vde= v, > =20 > debug("u64: 0x%016"PRIx64, vmsg->payload.u64); > =20 > - vu_check_queue_msg_file(vmsg); > + vu_check_queue_msg_file(vdev, vmsg); > =20 > if (vdev->vq[idx].call_fd !=3D -1) { > close(vdev->vq[idx].call_fd); > @@ -869,7 +873,7 @@ static bool vu_set_vring_err_exec(struct vu_dev *vdev, > =20 > debug("u64: 0x%016"PRIx64, vmsg->payload.u64); > =20 > - vu_check_queue_msg_file(vmsg); > + vu_check_queue_msg_file(vdev, vmsg); > =20 > if (vdev->vq[idx].err_fd !=3D -1) { > close(vdev->vq[idx].err_fd); > @@ -896,7 +900,8 @@ static bool vu_get_protocol_features_exec(struct vu_d= ev *vdev, > uint64_t features =3D 1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK | > 1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD | > 1ULL << VHOST_USER_PROTOCOL_F_DEVICE_STATE | > - 1ULL << VHOST_USER_PROTOCOL_F_RARP; > + 1ULL << VHOST_USER_PROTOCOL_F_RARP | > + 1ULL << VHOST_USER_PROTOCOL_F_MQ; > =20 > (void)vdev; > vmsg_set_reply_u64(vmsg, features); > @@ -935,10 +940,9 @@ static bool vu_get_queue_num_exec(struct vu_dev *vde= v, > { > (void)vdev; > =20 > - /* NOLINTNEXTLINE(misc-redundant-expression) */ > - vmsg_set_reply_u64(vmsg, VHOST_USER_MAX_VQS / 2); > + vmsg_set_reply_u64(vmsg, vdev->context->max_qpairs); > =20 > - debug("VHOST_USER_MAX_VQS %u", VHOST_USER_MAX_VQS / 2); > + debug("max_qpairs %u", vdev->context->max_qpairs); > =20 > return true; > } > @@ -959,7 +963,7 @@ static bool vu_set_vring_enable_exec(struct vu_dev *v= dev, > debug("State.index: %u", idx); > debug("State.enable: %u", enable); > =20 > - if (idx >=3D VHOST_USER_MAX_VQS) > + if (idx >=3D vdev->context->max_qpairs * 2) > die("Invalid vring_enable index: %u", idx); > =20 > vdev->vq[idx].enable =3D enable; > @@ -1047,7 +1051,7 @@ static bool vu_check_device_state_exec(struct vu_de= v *vdev, > */ > void vu_init(struct ctx *c) > { > - int i; > + unsigned int i; > =20 > c->vdev =3D &vdev_storage; > c->vdev->context =3D c; > @@ -1074,7 +1078,7 @@ void vu_cleanup(struct vu_dev *vdev) > { > unsigned int i; > =20 > - for (i =3D 0; i < VHOST_USER_MAX_VQS; i++) { > + for (i =3D 0; i < vdev->context->max_qpairs * 2; i++) { > struct vu_virtq *vq =3D &vdev->vq[i]; > =20 > vq->started =3D false; > diff --git a/virtio.h b/virtio.h > index 12caaa0b6def..176c935cecc7 100644 > --- a/virtio.h > +++ b/virtio.h > @@ -88,7 +88,7 @@ struct vu_dev_region { > uint64_t mmap_addr; > }; > =20 > -#define VHOST_USER_MAX_VQS 2 > +#define VHOST_USER_MAX_VQS 32 > =20 > /* > * Set a reasonable maximum number of ram slots, which will be supported= by > --=20 > 2.51.0 >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --IdGHezylb0OjrLow Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmkmY+gACgkQzQJF27ox 2GdHfg//SPf/E4ZG+/7pDm5IcLvf2JybWKJmYqte7JlycELmtdmLgSY448guIHy9 wQO/5MhBJLjaqw/0UHKcuMCVDBv2mnrFQy4CTfLJLX6UPx3+TOYC25Alg3rBZfPw ic2nURZYhxek81U5LLJ/ln4R1yo/s++ZcTd1FQREXIZxDfGYP1xiq/p6BgbqrJUz MuxhJlUcrK8JSaF0Aw4DrKARE+4Z4d7McXruK46oVX6sRL9XriMMAYB4Sbp5Qv40 x24ipnCFHTrZnufsiCNlARsuH9TrxHVPlR7FSlPtoV/i5hgN5wuLqL54M/OG7ZDq wKHgq8UUaYmzyVRBuHUZX1g93C+95vl7QL68M/k4omZAbeHnskBXDIKsnQqRcHvn 6BFncENntPgNpd4QGx47RrTyOPfx8GxOav4O5uo3pTc5XO/a/WVv0vRbhIGKqJ4C kBp8bUFFACimAh50DrHjakP2o2BuyLu/P1d3SJz2in0Kfhcmt4+5BSJhQYoA4Zsk MMTuL2o/QAuL6/fjQKbF4Tdo4Fx9jMMDHwVWeEq1N2Kv7gHc/oE7sOBCZ1G7nWtv nOaJi9Tj36Qdz5fDLZWPLgmwB2LRF5oSetar+O1YAvhMtMXD2FLaSetH59sPvajr gE8U9mMkFOWyl8m/9479MjjFCKfba6NYi9c7vzsMAs1jOUsuApE= =zKKo -----END PGP SIGNATURE----- --IdGHezylb0OjrLow--