From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202608 header.b=migPqAus; dkim-atps=neutral Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id DAE5D5A0265 for ; Mon, 03 Aug 2026 07:43:34 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202608; t=1785735811; bh=3Dr2h1mOKZcGWWvFRfiKjPhabn3I4mbf/CIp3Ab1dCU=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=migPqAusJRsrWjUFyuyKG2qp55m28edjIMXRl93QsYsdz/3ZmnGJi39ZAlTFNeWN+ bXDGNAcY2IIp1sacWNSPP5hKhbOEYBm4wth5ioNRrTNvcfMkj8ym5H6oJjvMnL8qDw ZT04XNtOb1tDQ29MPoGth2Duh6SBPuNqccJ4RAfNgL7dfQ6B70fj52dTuPGimfBUxc B+m9p4P3WdPvRUoPcvAE6OAgk7ydwpYwGFyDuS2AhVGqk3TAsLbzjami5QrW0/gV+k LZIgCSpETDAK6ohaNGd9WY6EKPbK/cFUok63k6iBqEf2XENL8idUit9iJChAD8/Z98 BhVDPR5opQN7A== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4hD5FH2nmjz4wCC; Mon, 03 Aug 2026 15:43:31 +1000 (AEST) Date: Mon, 3 Aug 2026 15:41:58 +1000 From: David Gibson To: Laurent Vivier Subject: Re: [PATCH v2 01/10] tap: Convert packet pools to per-queue-pair arrays for multiqueue Message-ID: References: <20260731162329.3552800-1-lvivier@redhat.com> <20260731162329.3552800-2-lvivier@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="nhv7sLor9sXJF0ID" Content-Disposition: inline In-Reply-To: <20260731162329.3552800-2-lvivier@redhat.com> Message-ID-Hash: XPTI533C7CWCAFLLSWJ4S7OW4LGHRIHY X-Message-ID-Hash: XPTI533C7CWCAFLLSWJ4S7OW4LGHRIHY X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --nhv7sLor9sXJF0ID Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 31, 2026 at 06:23:20PM +0200, Laurent Vivier wrote: > Convert the global pool_tap4 and pool_tap6 packet pools from single > pools to arrays of pools, one for each queue pair. This change is > necessary to support multiqueue operation in vhost-user mode, where > multiple queue pairs may be processing packets concurrently. >=20 > The pool storage structures (pool_tap4_storage and pool_tap6_storage) > are now arrays of VHOST_USER_MAX_VQS/2 elements, with corresponding > pointer arrays (pool_tap4 and pool_tap6) for accessing them. >=20 > Add a qpair parameter to tap_flush_pools() so it flushes the correct > pool. tap4_handler() and tap6_handler() now use the qpair they > already receive to index into the pool arrays. Add bounds checking > assertions in tap_handler() and tap_add_packet(). >=20 > In passt and pasta modes, all operations use QPAIR_DEFAULT. In > vhost-user mode, the queue pair is derived from the virtqueue index > via QPAIR_FROM_QUEUE(). >=20 > All pools within the array share the same buffer pointer: > - In vhost-user mode: Points to the vhost-user memory structure, which > is safe as packet data remains in guest memory and pools only track > iovecs > - In passt/pasta mode: Points to pkt_buf, which is safe as only queue > pair 0 is used >=20 > Signed-off-by: Laurent Vivier Reviewed-by: David Gibson > --- > tap.c | 77 ++++++++++++++++++++++++++++++----------------------- > tap.h | 2 +- > vu_common.c | 2 +- > 3 files changed, 46 insertions(+), 35 deletions(-) >=20 > diff --git a/tap.c b/tap.c > index 029888e32db5..6bba436f148f 100644 > --- a/tap.c > +++ b/tap.c > @@ -94,9 +94,13 @@ CHECK_FRAME_LEN(L2_MAX_LEN_VU); > DIV_ROUND_UP(sizeof(pkt_buf), \ > ETH_HLEN + sizeof(struct ipv6hdr) + sizeof(struct udphdr)) > =20 > -/* IPv4 (plus ARP) and IPv6 message batches from tap/guest to IP handler= s */ > -static PACKET_POOL_NOINIT(pool_tap4, TAP_MSGS_IP4); > -static PACKET_POOL_NOINIT(pool_tap6, TAP_MSGS_IP6); > +/* IPv4 (plus ARP) and IPv6 message batches from tap/guest to IP handlers > + * One pool per queue pair for multiqueue support > + */ > +static PACKET_POOL_DECL(pool_tap4, TAP_MSGS_IP4) pool_tap4_storage[VHOST= _USER_MAX_VQS / 2]; > +static struct pool *pool_tap4[VHOST_USER_MAX_VQS / 2]; > +static PACKET_POOL_DECL(pool_tap6, TAP_MSGS_IP6) pool_tap6_storage[VHOST= _USER_MAX_VQS / 2]; > +static struct pool *pool_tap6[VHOST_USER_MAX_VQS / 2]; > =20 > #define TAP_SEQS 128 /* Different L4 tuples in one batch */ > =20 > @@ -702,12 +706,12 @@ static int tap4_handler(struct ctx *c, unsigned int= qpair, > unsigned int i, j, seq_count; > struct tap4_l4_t *seq; > =20 > - if (!c->ifi4 || !pool_tap4->count) > - return pool_tap4->count; > + if (!c->ifi4 || !pool_tap4[qpair]->count) > + return pool_tap4[qpair]->count; > =20 > i =3D 0; > resume: > - for (seq_count =3D 0, seq =3D NULL; i < pool_tap4->count; i++) { > + for (seq_count =3D 0, seq =3D NULL; i < pool_tap4[qpair]->count; i++) { > struct iovec trim_iov[UIO_MAXIOV]; > size_t l3len, hlen, l4len, check; > struct ethhdr eh_storage; > @@ -718,7 +722,7 @@ resume: > struct iov_tail data; > struct iphdr *iph; > =20 > - if (!packet_get(pool_tap4, i, &data)) > + if (!packet_get(pool_tap4[qpair], i, &data)) > continue; > =20 > eh =3D IOV_PEEK_HEADER(&data, eh_storage); > @@ -801,7 +805,7 @@ resume: > if (iph->protocol =3D=3D IPPROTO_UDP) { > struct iov_tail eh_data; > =20 > - packet_get(pool_tap4, i, &eh_data); > + packet_get(pool_tap4[qpair], i, &eh_data); > if (dhcp(c, qpair, &eh_data)) > continue; > } > @@ -832,7 +836,7 @@ resume: > goto append; > =20 > if (seq_count =3D=3D TAP_SEQS) > - break; /* Resume after flushing if i < pool_tap4->count */ > + break; /* Resume after flushing if i < pool_tap4[qpair]->count */ > =20 > for (seq =3D tap4_l4 + seq_count - 1; seq >=3D tap4_l4; seq--) { > if (L4_MATCH(iph, uh, seq)) { > @@ -878,10 +882,10 @@ append: > } > } > =20 > - if (i < pool_tap4->count) > + if (i < pool_tap4[qpair]->count) > goto resume; > =20 > - return pool_tap4->count; > + return pool_tap4[qpair]->count; > } > =20 > #define IPV6_NH_OPT(nh) \ > @@ -950,12 +954,12 @@ static int tap6_handler(struct ctx *c, unsigned int= qpair, > unsigned int i, j, seq_count =3D 0; > struct tap6_l4_t *seq; > =20 > - if (!c->ifi6 || !pool_tap6->count) > - return pool_tap6->count; > + if (!c->ifi6 || !pool_tap6[qpair]->count) > + return pool_tap6[qpair]->count; > =20 > i =3D 0; > resume: > - for (seq_count =3D 0, seq =3D NULL; i < pool_tap6->count; i++) { > + for (seq_count =3D 0, seq =3D NULL; i < pool_tap6[qpair]->count; i++) { > size_t l4len, plen, check; > struct in6_addr *saddr, *daddr; > struct ipv6hdr ip6h_storage; > @@ -967,7 +971,7 @@ resume: > struct ipv6hdr *ip6h; > uint8_t proto; > =20 > - if (!packet_get(pool_tap6, i, &data)) > + if (!packet_get(pool_tap6[qpair], i, &data)) > return -1; > =20 > eh =3D IOV_REMOVE_HEADER(&data, eh_storage); > @@ -1082,7 +1086,7 @@ resume: > goto append; > =20 > if (seq_count =3D=3D TAP_SEQS) > - break; /* Resume after flushing if i < pool_tap6->count */ > + break; /* Resume after flushing if i < pool_tap6[qpair]->count */ > =20 > for (seq =3D tap6_l4 + seq_count - 1; seq >=3D tap6_l4; seq--) { > if (L4_MATCH(ip6h, proto, uh, seq)) { > @@ -1129,19 +1133,19 @@ append: > } > } > =20 > - if (i < pool_tap6->count) > + if (i < pool_tap6[qpair]->count) > goto resume; > =20 > - return pool_tap6->count; > + return pool_tap6[qpair]->count; > } > =20 > /** > - * tap_flush_pools() - Flush both IPv4 and IPv6 packet pools > + * tap_flush_pools() - Flush both IPv4 and IPv6 packet pools for a given= qpair > */ > -void tap_flush_pools(void) > +void tap_flush_pools(unsigned int qpair) > { > - pool_flush(pool_tap4); > - pool_flush(pool_tap6); > + pool_flush(pool_tap4[qpair]); > + pool_flush(pool_tap6[qpair]); > } > =20 > /** > @@ -1152,6 +1156,7 @@ void tap_flush_pools(void) > */ > void tap_handler(struct ctx *c, unsigned int qpair, const struct timespe= c *now) > { > + assert(qpair < VHOST_USER_MAX_VQS / 2); > tap4_handler(c, qpair, now); > tap6_handler(c, qpair, now); > } > @@ -1186,21 +1191,23 @@ void tap_add_packet(struct ctx *c, unsigned int q= pair, struct iov_tail *data, > proto_update_l2_buf(c->guest_mac); > } > =20 > + assert(qpair < VHOST_USER_MAX_VQS / 2); > + > switch (ntohs(eh->h_proto)) { > case ETH_P_ARP: > case ETH_P_IP: > - if (!pool_can_fit(pool_tap4, data)) { > + if (!pool_can_fit(pool_tap4[qpair], data)) { > tap4_handler(c, qpair, now); > - pool_flush(pool_tap4); > + pool_flush(pool_tap4[qpair]); > } > - packet_add(pool_tap4, data); > + packet_add(pool_tap4[qpair], data); > break; > case ETH_P_IPV6: > - if (!pool_can_fit(pool_tap6, data)) { > + if (!pool_can_fit(pool_tap6[qpair], data)) { > tap6_handler(c, qpair, now); > - pool_flush(pool_tap6); > + pool_flush(pool_tap6[qpair]); > } > - packet_add(pool_tap6, data); > + packet_add(pool_tap6[qpair], data); > break; > default: > break; > @@ -1238,7 +1245,7 @@ static void tap_passt_input(struct ctx *c, const st= ruct timespec *now) > ssize_t n; > char *p; > =20 > - tap_flush_pools(); > + tap_flush_pools(QPAIR_DEFAULT); > =20 > if (partial_len) { > /* We have a partial frame from an earlier pass. Move it to the > @@ -1321,7 +1328,7 @@ static void tap_pasta_input(struct ctx *c, const st= ruct timespec *now) > { > ssize_t n, len; > =20 > - tap_flush_pools(); > + tap_flush_pools(QPAIR_DEFAULT); > =20 > for (n =3D 0; > n <=3D (ssize_t)(sizeof(pkt_buf) - L2_MAX_LEN_PASTA); > @@ -1586,10 +1593,14 @@ static void tap_sock_tun_init(struct ctx *c) > */ > static void tap_sock_update_pool(void *base, size_t size) > { > - int i; > + unsigned int i; > =20 > - pool_tap4_storage =3D PACKET_INIT(pool_tap4, TAP_MSGS_IP4, base, size); > - pool_tap6_storage =3D PACKET_INIT(pool_tap6, TAP_MSGS_IP6, base, size); > + for (i =3D 0; i < VHOST_USER_MAX_VQS / 2; i++) { > + pool_tap4_storage[i] =3D PACKET_INIT(pool_tap4, TAP_MSGS_IP4, base, si= ze); > + pool_tap4[i] =3D (struct pool *)&pool_tap4_storage[i]; > + pool_tap6_storage[i] =3D PACKET_INIT(pool_tap6, TAP_MSGS_IP6, base, si= ze); > + pool_tap6[i] =3D (struct pool *)&pool_tap6_storage[i]; > + } > =20 > for (i =3D 0; i < TAP_SEQS; i++) { > tap4_l4[i].p =3D PACKET_INIT(pool_l4, UIO_MAXIOV, base, size); > diff --git a/tap.h b/tap.h > index 0f8e98fc5f84..7d77e071ba49 100644 > --- a/tap.h > +++ b/tap.h > @@ -121,7 +121,7 @@ void tap_handler_passt(struct ctx *c, uint32_t events, > int tap_sock_unix_open(char *sock_path); > void tap_sock_reset(struct ctx *c); > void tap_backend_init(struct ctx *c); > -void tap_flush_pools(void); > +void tap_flush_pools(unsigned int qpair); > void tap_handler(struct ctx *c, unsigned int qpair, const struct timespe= c *now); > void tap_add_packet(struct ctx *c, unsigned int qpair, struct iov_tail *= data, > const struct timespec *now); > diff --git a/vu_common.c b/vu_common.c > index 7239cb42f38b..0cd19b36737d 100644 > --- a/vu_common.c > +++ b/vu_common.c > @@ -177,7 +177,7 @@ static void vu_handle_tx(struct vu_dev *vdev, int ind= ex, > =20 > assert(QPAIR_IS_FROMGUEST(index)); > =20 > - tap_flush_pools(); > + tap_flush_pools(QPAIR_DEFAULT); > =20 > count =3D 0; > out_sg_count =3D 0; > --=20 > 2.54.0 >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --nhv7sLor9sXJF0ID Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmpwKhgACgkQzQJF27ox 2Gcxmg/+PGjz2N1Nnywm6HFfYEUDLAnzcphyP1/9EVUD34ED5nA8CnLSWqirXWU/ AB1jNp69NVCS0MgIUZua4YG5gJ94Cs9Vg0k6O3h75igRV/qAYP+PGLY+1B6PS+2f 5bfyPkrXNqPHfzcihj+ycBzxBOA6XfomMDOeT+PruOjdosV7tDUQBWvfVOON3ePj gcfOuEmHI2rz1EW0o6iU8iGA3+NwH1vGzY4GpaBN5MJtyOZ+GlDu66bkUT9TUpUD 0CQqIPAryOHvfV7dfhyxr4TzBWIf+ZMUvKHtpZ+YbYRPuPZ2lwbdDR6psEK5ffmL sbXxwMSE3TvvgYSpJ7c9WglpMDzYwPO5oL9EOTO+3pU6h3dGjk7g9k6vudwTZqK6 51qDayI3rCRPC7zdVjk9/pL9jAzefXOOyY5FDmI6AnRanTviaqUt0RY6NOu50WoO UzwBDWJoaa1vL32o7BKveuCEQ0ILcSiXpFClW05HuQgkC0V8imL1+oXFR6S/sM9s hQbHFGfTESlDrxCo05gIlOrmowQ86t2QiPY3Eq75b8svjj1jxv0nKxbPGQQ+a/Ou 08ZciSWCUQoRCAP+2AkbPpMgWrgOZ0Di4NSKyEJMmFu46/4ubXkua6Q+7RGnsTi7 AsPzk7WzLlSQ/zy8Lh8E2iEjnNPRsTpQMAW4Zre4NSvO9t2L6VE= =Fqi/ -----END PGP SIGNATURE----- --nhv7sLor9sXJF0ID--