From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202608 header.b=htQiUdlw; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id AF73F5A0265 for ; Mon, 03 Aug 2026 08:11:44 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202608; t=1785737502; bh=mUGGuwRM4hmZPYs3mAtTIe5kBaJpPR2YHz3wRyI/1PI=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=htQiUdlwNNEGtq9YJhfbUz3nF2pnVl9qZKl/jghryBHpGTU1KNkAJVpEXPSS/GPM7 bZQ+9zbXdY2itPFvzhY4OMXqTUqsSHuXCzrzvCsGUQIDQESHBRZDrdawl1Hht26Gca /P51jkDmN3fUe0hftaRslGSE+ro0mZff6hSCUkkZLIBiCCNOQ8gdbGg7OTzPM70GnV nsnzHmpBhplV+NZjnLC4Sm4v4xQqcJdzVgcX738qeksDawRaVsfnXi8HSozaiC0+z7 UfL2Sj7ssyYHsGF2RvCcpckGG8KflXHeFsIxDYqCpvKQnblgb4056BYmZRQM2UAM+p rcEUEbuWHk1Vg== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4hD5sp524hz4wCH; Mon, 03 Aug 2026 16:11:42 +1000 (AEST) Date: Mon, 3 Aug 2026 15:59:19 +1000 From: David Gibson To: Laurent Vivier Subject: Re: [PATCH v2 03/10] tcp: Make static buffers stack-local for thread safety Message-ID: References: <20260731162329.3552800-1-lvivier@redhat.com> <20260731162329.3552800-4-lvivier@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="7+a1Qw1sTyi+JL8h" Content-Disposition: inline In-Reply-To: <20260731162329.3552800-4-lvivier@redhat.com> Message-ID-Hash: EUJVOEU4VESYWYIIG52M7DGRGJBCFEHG X-Message-ID-Hash: EUJVOEU4VESYWYIIG52M7DGRGJBCFEHG X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --7+a1Qw1sTyi+JL8h Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jul 31, 2026 at 06:23:22PM +0200, Laurent Vivier wrote: > Static buffers shared across all call sites are not safe when multiple > worker threads handle TCP connections concurrently. >=20 > In tcp.c, move tcp_iov[] from file scope into tcp_data_from_tap() where > it is exclusively used. At UIO_MAXIOV (1024) entries of struct iovec > (16 bytes each), this adds 16 KiB to the stack frame. >=20 > In tcp_vu.c, move iov_vu[], elem[], and frame[] from file scope into > tcp_vu_data_from_sock() and pass them to tcp_vu_sock_recv() as > parameters. Also make iov_msg[] in tcp_vu_sock_recv() a local variable > instead of static, as it is only used within a single call. Combined, > these add roughly 80 KiB across the nested stack frames, which is > acceptable for per-thread stacks. >=20 > Signed-off-by: Laurent Vivier Reviewed-by: David Gibson > --- > tcp.c | 3 +-- > tcp_vu.c | 33 ++++++++++++++++++++------------- > 2 files changed, 21 insertions(+), 15 deletions(-) >=20 > diff --git a/tcp.c b/tcp.c > index e80a64e38f24..ee4ec599e055 100644 > --- a/tcp.c > +++ b/tcp.c > @@ -435,8 +435,6 @@ static socklen_t tcp_info_size; > /* Kernel reports delivery rate in TCP_INFO (kernel commit eb8329e0a04d)= */ > #define delivery_rate_cap tcp_info_cap(delivery_rate) > =20 > -/* sendmsg() to socket */ > -static struct iovec tcp_iov [UIO_MAXIOV]; > =20 > /* Pools for pre-opened sockets (in init) */ > int init_sock_pool4 [TCP_SOCK_POOL_SIZE]; > @@ -1945,6 +1943,7 @@ static int tcp_data_from_tap(const struct ctx *c, s= truct tcp_tap_conn *conn, > uint16_t max_ack_seq_wnd =3D conn->wnd_from_tap; > uint32_t max_ack_seq =3D conn->seq_ack_from_tap; > uint32_t seq_from_tap =3D conn->seq_from_tap; > + struct iovec tcp_iov[UIO_MAXIOV]; > struct msghdr mh =3D { .msg_iov =3D tcp_iov }; > size_t len; > ssize_t n; > diff --git a/tcp_vu.c b/tcp_vu.c > index c9a1e08747b5..b4baecc1f717 100644 > --- a/tcp_vu.c > +++ b/tcp_vu.c > @@ -35,9 +35,6 @@ > #include "vu_common.h" > #include > =20 > -static struct iovec iov_vu[VIRTQUEUE_MAX_SIZE]; > -static struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; > - > /** > * struct vu_frame - Descriptor for a TCP frame mapped to virtqueue elem= ents > * @idx_element: Index of first element in elem[] for this frame > @@ -46,13 +43,13 @@ static struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZ= E]; > * @num_iovec: Number of iovecs covering this frame's buffers > * @size: Total frame size including all headers > */ > -static struct vu_frame { > +struct vu_frame { > int idx_element; > int num_element; > int idx_iovec; > int num_iovec; > size_t size; > -} frame[VIRTQUEUE_MAX_SIZE]; > +}; > =20 > /** > * tcp_vu_hdrlen() - Sum size of all headers, from TCP to virtio-net > @@ -225,6 +222,9 @@ int tcp_vu_send_flag(const struct ctx *c, struct tcp_= tap_conn *conn, int flags, > * @v6: Set for IPv6 connections > * @already_sent: Number of bytes already sent > * @fillsize: Maximum bytes to fill in guest-side receiving window > + * @iov_vu: IO vector array for virtqueue buffers > + * @elem: Virtqueue element array > + * @frame: Frame descriptor array > * @elem_used: number of element (output) > * @frame_cnt: Pointer to store the number of frames (output) > * > @@ -234,9 +234,12 @@ int tcp_vu_send_flag(const struct ctx *c, struct tcp= _tap_conn *conn, int flags, > static ssize_t tcp_vu_sock_recv(const struct ctx *c, struct vu_virtq *vq, > const struct tcp_tap_conn *conn, bool v6, > uint32_t already_sent, size_t fillsize, > + struct iovec *iov_vu, > + struct vu_virtq_element *elem, > + struct vu_frame *frame, > int *elem_used, int *frame_cnt) > { > - static struct iovec iov_msg[VIRTQUEUE_MAX_SIZE + DISCARD_IOV_NUM]; > + struct iovec iov_msg[VIRTQUEUE_MAX_SIZE + DISCARD_IOV_NUM]; > const struct vu_dev *vdev =3D c->vdev; > struct msghdr mh_sock =3D { 0 }; > uint16_t mss =3D MSS_GET(conn); > @@ -253,16 +256,16 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c= , struct vu_virtq *vq, > iov_used =3D 0; > elem_cnt =3D 0; > *frame_cnt =3D 0; > - while (fillsize > 0 && elem_cnt < ARRAY_SIZE(elem) && > - iov_used < ARRAY_SIZE(iov_vu) && > - *frame_cnt < ARRAY_SIZE(frame)) { > + while (fillsize > 0 && elem_cnt < VIRTQUEUE_MAX_SIZE && > + iov_used < VIRTQUEUE_MAX_SIZE && > + *frame_cnt < VIRTQUEUE_MAX_SIZE) { > size_t frame_size, in_total; > int cnt; > =20 > cnt =3D vu_collect(vdev, vq, &elem[elem_cnt], > - ARRAY_SIZE(elem) - elem_cnt, > + VIRTQUEUE_MAX_SIZE - elem_cnt, > &iov_vu[iov_used], > - ARRAY_SIZE(iov_vu) - iov_used, &in_total, > + VIRTQUEUE_MAX_SIZE - iov_used, &in_total, > MIN(mss, fillsize) + hdrlen, > &frame_size); > if (cnt =3D=3D 0) > @@ -328,7 +331,8 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, = struct vu_virtq *vq, > if ((size_t)ret <=3D f->size - hdrlen) { > unsigned cnt; > =20 > - cnt =3D iov_skip_bytes(&iov_vu[f->idx_iovec], f->num_iovec, > + cnt =3D iov_skip_bytes(&iov_vu[f->idx_iovec], > + f->num_iovec, > MAX(hdrlen + ret, VNET_HLEN + ETH_ZLEN), > NULL); > if (cnt < (unsigned)f->num_iovec) > @@ -437,6 +441,9 @@ int tcp_vu_data_from_sock(const struct ctx *c, struct= tcp_tap_conn *conn, > uint32_t already_sent, const struct timespec *now, > unsigned int qpair) > { > + struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; > + struct iovec iov_vu[VIRTQUEUE_MAX_SIZE]; > + struct vu_frame frame[VIRTQUEUE_MAX_SIZE]; > uint32_t wnd_scaled =3D conn->wnd_from_tap << conn->ws_from_tap; > int toguest =3D QPAIR_TOGUEST_QUEUE(qpair); > struct vu_dev *vdev =3D c->vdev; > @@ -460,7 +467,7 @@ int tcp_vu_data_from_sock(const struct ctx *c, struct= tcp_tap_conn *conn, > * data from the socket > */ > len =3D tcp_vu_sock_recv(c, vq, conn, v6, already_sent, fillsize, > - &elem_cnt, &frame_cnt); > + iov_vu, elem, frame, &elem_cnt, &frame_cnt); > if (len < 0) { > if (len !=3D -EAGAIN && len !=3D -EWOULDBLOCK) { > tcp_rst(c, conn, now, qpair); > --=20 > 2.54.0 >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --7+a1Qw1sTyi+JL8h Content-Type: application/pgp-signature; name=signature.asc -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmpwLiwACgkQzQJF27ox 2GdjPA//bWNrAO5NDzW3Oj378aVwU7JZmj79isOYKTAkZRKrckO30/kBaTQc3Fs7 kqNXmZRGyUc75vsmSY0ZFER1G8O9SExCJE1Qy+IE5o1zRNQDjqy5EEyItXuJy1MJ s+M0Vi9LRj8OyDmxwKnz+sbA4wczLPVR5zsM0xMtAq/B/EIdFNobfhtpKOD1I6M4 Ygamyy4novGrK/6wNbWTmYsa2OZD9wHLWRL+W7krrS2sewHiauoBpPfHsvPM5tk5 M+HCJLC6BRrG6hkFdKBvegNKOzWtHZ3hn0hPvhqnTHC8Vj+z90nL5OkD+vTutIEB d50UtLi2mvG+KjKBbv7xK5W5So7jqqMKF4B3CJmvTpQajvaZmAXfP9nTM8zadZnN PtIboHo6pm5aSqICS3ToPa5pit3s0nR2gzAJvbmFheGJ5OAnp+KxP0J1fHKCqrwP /rZCn5ra6QxZbUeSl+3/ffPt/wqc2CguVk2PsuyYY0Yz3Edw+2amAQdDIcDluD0H PEZah153To8eMvRyXmCfPOf2aEt7IlvXGpoL9bW560+uNscP4atugyadBqGTMoym aXhyoX7Hfq37yZPckPB++xiyaXBLwtxuCQfboJTS8bPux0QkUgE0mkkSIfm2yf2P OKmtbbO4555DiWv6T7k+bpbq76/YEo+dNkq70HOnwXUOjl9iEHI= =n+Kf -----END PGP SIGNATURE----- --7+a1Qw1sTyi+JL8h--