From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=LhbGJlW8; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id C389A5A061C for ; Tue, 16 Jun 2026 19:11:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1781629864; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GMDvAxRTBY7AKNXk4H9V9pY5oK0PgP87exBMu6RhTI8=; b=LhbGJlW8IP0QnkwiiwIUfta0tQ7daJgflemvX1bTcapOh9Yw0kXiGdzfFm3DfRxpQg0iyL 5kQQRa0SkpHqYMUNzMWXfYXkwtNvj19HBEQV+suUB3gC2mb9YawcBMJvv5ruz4yewdifd8 pC1UCe8GwG6PzygjGovub4js4c5iBPk= Received: from mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (ec2-35-165-154-97.us-west-2.compute.amazonaws.com [35.165.154.97]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-561-3AA22YI8NpGei_wL7Vc98A-1; Tue, 16 Jun 2026 13:10:59 -0400 X-MC-Unique: 3AA22YI8NpGei_wL7Vc98A-1 X-Mimecast-MFC-AGG-ID: 3AA22YI8NpGei_wL7Vc98A_1781629858 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-06.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 36C6C1800EEC for ; Tue, 16 Jun 2026 17:10:58 +0000 (UTC) Received: from lenovo-t14s.redhat.corp (headnet05.pony-001.prod.iad2.dc.redhat.com [10.2.32.117]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id 64BF9195411C; Tue, 16 Jun 2026 17:10:57 +0000 (UTC) From: Laurent Vivier To: passt-dev@passt.top Subject: [PATCH 3/8] tcp: Make static buffers stack-local for thread safety Date: Tue, 16 Jun 2026 19:10:47 +0200 Message-ID: <20260616171052.3785909-4-lvivier@redhat.com> In-Reply-To: <20260616171052.3785909-1-lvivier@redhat.com> References: <20260616171052.3785909-1-lvivier@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Wv8gwXq_aNNXJS59hT7qWsmNShYCjozQDwvRtTtM1n4_1781629858 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit content-type: text/plain; charset="US-ASCII"; x-default=true Message-ID-Hash: FPEFGKLFDI725NEEMN6EAQNXJ34WGMET X-Message-ID-Hash: FPEFGKLFDI725NEEMN6EAQNXJ34WGMET X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Laurent Vivier X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: Static buffers shared across all call sites are not safe when multiple worker threads handle TCP connections concurrently. In tcp.c, move tcp_iov[] from file scope into tcp_data_from_tap() where it is exclusively used. At UIO_MAXIOV (1024) entries of struct iovec (16 bytes each), this adds 16 KiB to the stack frame. In tcp_vu.c, move iov_vu[], elem[], and frame[] from file scope into tcp_vu_data_from_sock() and pass them to tcp_vu_sock_recv() as parameters. Also make iov_msg[] in tcp_vu_sock_recv() a local variable instead of static, as it is only used within a single call. Combined, these add roughly 80 KiB across the nested stack frames, which is acceptable for per-thread stacks. Signed-off-by: Laurent Vivier --- tcp.c | 3 +-- tcp_vu.c | 33 ++++++++++++++++++++------------- 2 files changed, 21 insertions(+), 15 deletions(-) diff --git a/tcp.c b/tcp.c index 1549e14adaf4..f4fe866ba7c3 100644 --- a/tcp.c +++ b/tcp.c @@ -435,8 +435,6 @@ static socklen_t tcp_info_size; /* Kernel reports delivery rate in TCP_INFO (kernel commit eb8329e0a04d) */ #define delivery_rate_cap tcp_info_cap(delivery_rate) -/* sendmsg() to socket */ -static struct iovec tcp_iov [UIO_MAXIOV]; /* Pools for pre-opened sockets (in init) */ int init_sock_pool4 [TCP_SOCK_POOL_SIZE]; @@ -1900,6 +1898,7 @@ static int tcp_data_from_tap(const struct ctx *c, struct tcp_tap_conn *conn, uint16_t max_ack_seq_wnd = conn->wnd_from_tap; uint32_t max_ack_seq = conn->seq_ack_from_tap; uint32_t seq_from_tap = conn->seq_from_tap; + struct iovec tcp_iov[UIO_MAXIOV]; struct msghdr mh = { .msg_iov = tcp_iov }; size_t len; ssize_t n; diff --git a/tcp_vu.c b/tcp_vu.c index 4f76f599156f..9270ece43d17 100644 --- a/tcp_vu.c +++ b/tcp_vu.c @@ -35,9 +35,6 @@ #include "vu_common.h" #include -static struct iovec iov_vu[VIRTQUEUE_MAX_SIZE]; -static struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; - /** * struct vu_frame - Descriptor for a TCP frame mapped to virtqueue elements * @idx_element: Index of first element in elem[] for this frame @@ -46,13 +43,13 @@ static struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; * @num_iovec: Number of iovecs covering this frame's buffers * @size: Total frame size including all headers */ -static struct vu_frame { +struct vu_frame { int idx_element; int num_element; int idx_iovec; int num_iovec; size_t size; -} frame[VIRTQUEUE_MAX_SIZE]; +}; /** * tcp_vu_hdrlen() - Sum size of all headers, from TCP to virtio-net @@ -224,6 +221,9 @@ int tcp_vu_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags, * @v6: Set for IPv6 connections * @already_sent: Number of bytes already sent * @fillsize: Maximum bytes to fill in guest-side receiving window + * @iov_vu: IO vector array for virtqueue buffers + * @elem: Virtqueue element array + * @frame: Frame descriptor array * @elem_used: number of element (output) * @frame_cnt: Pointer to store the number of frames (output) * @@ -233,9 +233,12 @@ int tcp_vu_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags, static ssize_t tcp_vu_sock_recv(const struct ctx *c, struct vu_virtq *vq, const struct tcp_tap_conn *conn, bool v6, uint32_t already_sent, size_t fillsize, + struct iovec *iov_vu, + struct vu_virtq_element *elem, + struct vu_frame *frame, int *elem_used, int *frame_cnt) { - static struct iovec iov_msg[VIRTQUEUE_MAX_SIZE + DISCARD_IOV_NUM]; + struct iovec iov_msg[VIRTQUEUE_MAX_SIZE + DISCARD_IOV_NUM]; const struct vu_dev *vdev = c->vdev; struct msghdr mh_sock = { 0 }; uint16_t mss = MSS_GET(conn); @@ -252,16 +255,16 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, struct vu_virtq *vq, iov_used = 0; elem_cnt = 0; *frame_cnt = 0; - while (fillsize > 0 && elem_cnt < ARRAY_SIZE(elem) && - iov_used < ARRAY_SIZE(iov_vu) && - *frame_cnt < ARRAY_SIZE(frame)) { + while (fillsize > 0 && elem_cnt < VIRTQUEUE_MAX_SIZE && + iov_used < VIRTQUEUE_MAX_SIZE && + *frame_cnt < VIRTQUEUE_MAX_SIZE) { size_t frame_size, in_total; int cnt; cnt = vu_collect(vdev, vq, &elem[elem_cnt], - ARRAY_SIZE(elem) - elem_cnt, + VIRTQUEUE_MAX_SIZE - elem_cnt, &iov_vu[iov_used], - ARRAY_SIZE(iov_vu) - iov_used, &in_total, + VIRTQUEUE_MAX_SIZE - iov_used, &in_total, MIN(mss, fillsize) + hdrlen, &frame_size); if (cnt == 0) @@ -327,7 +330,8 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, struct vu_virtq *vq, if ((size_t)ret <= f->size - hdrlen) { unsigned cnt; - cnt = iov_skip_bytes(&iov_vu[f->idx_iovec], f->num_iovec, + cnt = iov_skip_bytes(&iov_vu[f->idx_iovec], + f->num_iovec, MAX(hdrlen + ret, VNET_HLEN + ETH_ZLEN), NULL); if (cnt < (unsigned)f->num_iovec) @@ -433,6 +437,9 @@ static void tcp_vu_prepare(const struct ctx *c, struct tcp_tap_conn *conn, int tcp_vu_data_from_sock(const struct ctx *c, struct tcp_tap_conn *conn, unsigned int qpair) { + struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; + struct iovec iov_vu[VIRTQUEUE_MAX_SIZE]; + struct vu_frame frame[VIRTQUEUE_MAX_SIZE]; uint32_t wnd_scaled = conn->wnd_from_tap << conn->ws_from_tap; int rx_queue = QPAIR_TOGUEST_QUEUE(qpair); struct vu_dev *vdev = c->vdev; @@ -477,7 +484,7 @@ int tcp_vu_data_from_sock(const struct ctx *c, struct tcp_tap_conn *conn, * data from the socket */ len = tcp_vu_sock_recv(c, vq, conn, v6, already_sent, fillsize, - &elem_cnt, &frame_cnt); + iov_vu, elem, frame, &elem_cnt, &frame_cnt); if (len < 0) { if (len != -EAGAIN && len != -EWOULDBLOCK) { tcp_rst(c, conn, qpair); -- 2.54.0