From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=GnVb2r6+; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id EBCCF5A0262 for ; Mon, 22 Jun 2026 09:44:50 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1782114289; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=Lxw+U9w6ZuvMFtFXk0YcFGXTyUNPBSFQI0Ws39iRluY=; b=GnVb2r6+cHm8MkS2/6FqrZfvA4d80UQqiaPkLtYSyCX4jTyM0QP3xuL6SIZEHRLxEJQpV8 5z/lZO6UoE5BfR0EXNWjxRX7GksYWGdCcsWc/dTmof7Z4UZt7vv7saBwlNpC6iOezHpKEz fW4ey1RkcLviqLD9PofpSJam2tfr+eM= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-353-oeSxzq0yO92iLidCj2QnNg-1; Mon, 22 Jun 2026 03:44:48 -0400 X-MC-Unique: oeSxzq0yO92iLidCj2QnNg-1 X-Mimecast-MFC-AGG-ID: oeSxzq0yO92iLidCj2QnNg_1782114287 Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-490ae461f8dso26935165e9.1 for ; Mon, 22 Jun 2026 00:44:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1782114287; x=1782719087; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Lxw+U9w6ZuvMFtFXk0YcFGXTyUNPBSFQI0Ws39iRluY=; b=Myzkrl8zwapmUNDOqxaFi2yAD93VxFhO5wEd5VQbk35sZv02AlLPeVJfPYkAOaLhjv KLufvA3ihxhi/rMB7+mazR5KxmOkH6gDK9obRZquqfxSgRt0o5IT/+/FlAMDFHY02FGe 9qA/QZS1ayoTqRB66E3pFa3lUM09ABsxaEQwn1OtCJwe17KKyNWI+TzmqBghKp7R/t2i 5YPhW+oeqVBtVmhM/9udOYDRhkfCOXFVSeIdUN8nrb+P4Vb6bvw+iEG/LW1oHKNG8RBj SgcyCOGScC/AkIMpGLpFdzOc4ZVNkWEUI03EYWm0Y2aFoiQwgo/0ADgAvS0Wn92ToYUL 6r1A== X-Gm-Message-State: AOJu0Yw3lPjWfWZM6hbMcv8H/rleUYzOPGgQDV/tHtd0qRjiXJap3aZb zflytdT0j0c/UBvPaiJRT6O44VKb93li+HAA+/VZAecPyzoFeZJ7c+oCf7E0jYidjO/sfFD9Kpy 6OJGnSa1sZ29vWLgxS/YyXJLU+w1R1P8Gx7Zk2eLMh7AkUzr94zYhJA== X-Gm-Gg: AfdE7cnfbpQOeNRmqJFAMHyKdQI/QSSReq/RQhtq3kZGIsqiAIQoCfG85LF0jIk2Jd4 viLeBrFQ4FeWI8jq7UF5q24jrz+FlKWbBSLChRNSOF3Y6rL5XqcqRFkd1t8H+4eTtnxdeC7IsCA lav2eUjrH5icWCWQt3K0MLEiipid64j8YkrvIbf3P0pflqk5FmRDMEQkA//sAV4sk8JLxSsOGMw foUVRo9O4y3hCYnXjiNynvRxGhcMqvrjtP6R99gvbIfSuSiY/ocEHtKqmBPvwnJ/Lkfpo3ZRfeW Au3dI6BNT/gKbF+Mh0Yw6LYKfRXbFhXutukIptjepnI7JJ3LGs8cHdnVBDaMrR2tOpCCuwV+utq w+//9/YE0UJ+idKHlz0rz4WzDxvwvaLIY7sZ8nPS25GWEyrO6/A== X-Received: by 2002:a05:600c:138a:b0:492:1e7f:d41e with SMTP id 5b1f17b1804b1-4923f14c484mr194856885e9.10.1782114286958; Mon, 22 Jun 2026 00:44:46 -0700 (PDT) X-Received: by 2002:a05:600c:138a:b0:492:1e7f:d41e with SMTP id 5b1f17b1804b1-4923f14c484mr194856555e9.10.1782114286447; Mon, 22 Jun 2026 00:44:46 -0700 (PDT) Received: from ?IPV6:2a01:e0a:e10:ef90:343a:68f:2e91:95c? ([2a01:e0a:e10:ef90:343a:68f:2e91:95c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4923ff8a9e3sm298617755e9.14.2026.06.22.00.44.45 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 22 Jun 2026 00:44:45 -0700 (PDT) Message-ID: Date: Mon, 22 Jun 2026 09:44:45 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v5 06/12] tcp: Pass queue pair explicitly through TCP send path To: David Gibson References: <20260616125130.1324274-1-lvivier@redhat.com> <20260616125130.1324274-7-lvivier@redhat.com> <499724cc-6262-402d-9e63-b38ec171a7b2@redhat.com> From: Laurent Vivier Autocrypt: addr=lvivier@redhat.com; keydata= xsFNBFYFJhkBEAC2me7w2+RizYOKZM+vZCx69GTewOwqzHrrHSG07MUAxJ6AY29/+HYf6EY2 WoeuLWDmXE7A3oJoIsRecD6BXHTb0OYS20lS608anr3B0xn5g0BX7es9Mw+hV/pL+63EOCVm SUVTEQwbGQN62guOKnJJJfphbbv82glIC/Ei4Ky8BwZkUuXd7d5NFJKC9/GDrbWdj75cDNQx UZ9XXbXEKY9MHX83Uy7JFoiFDMOVHn55HnncflUncO0zDzY7CxFeQFwYRbsCXOUL9yBtqLer Ky8/yjBskIlNrp0uQSt9LMoMsdSjYLYhvk1StsNPg74+s4u0Q6z45+l8RAsgLw5OLtTa+ePM JyS7OIGNYxAX6eZk1+91a6tnqfyPcMbduxyBaYXn94HUG162BeuyBkbNoIDkB7pCByed1A7q q9/FbuTDwgVGVLYthYSfTtN0Y60OgNkWCMtFwKxRaXt1WFA5ceqinN/XkgA+vf2Ch72zBkJL RBIhfOPFv5f2Hkkj0MvsUXpOWaOjatiu0fpPo6Hw14UEpywke1zN4NKubApQOlNKZZC4hu6/ 8pv2t4HRi7s0K88jQYBRPObjrN5+owtI51xMaYzvPitHQ2053LmgsOdN9EKOqZeHAYG2SmRW LOxYWKX14YkZI5j/TXfKlTpwSMvXho+efN4kgFvFmP6WT+tPnwARAQABzSNMYXVyZW50IFZp dmllciA8bHZpdmllckByZWRoYXQuY29tPsLBeAQTAQIAIgUCVgVQgAIbAwYLCQgHAwIGFQgC CQoLBBYCAwECHgECF4AACgkQ8ww4vT8vvjwpgg//fSGy0Rs/t8cPFuzoY1cex4limJQfReLr SJXCANg9NOWy/bFK5wunj+h/RCFxIFhZcyXveurkBwYikDPUrBoBRoOJY/BHK0iZo7/WQkur 6H5losVZtrotmKOGnP/lJYZ3H6OWvXzdz8LL5hb3TvGOP68K8Bn8UsIaZJoeiKhaNR0sOJyI YYbgFQPWMHfVwHD/U+/gqRhD7apVysxv5by/pKDln1I5v0cRRH6hd8M8oXgKhF2+rAOL7gvh jEHSSWKUlMjC7YwwjSZmUkL+TQyE18e2XBk85X8Da3FznrLiHZFHQ/NzETYxRjnOzD7/kOVy gKD/o7asyWQVU65mh/ECrtjfhtCBSYmIIVkopoLaVJ/kEbVJQegT2P6NgERC/31kmTF69vn8 uQyW11Hk8tyubicByL3/XVBrq4jZdJW3cePNJbTNaT0d/bjMg5zCWHbMErUib2Nellnbg6bc 2HLDe0NLVPuRZhHUHM9hO/JNnHfvgiRQDh6loNOUnm9Iw2YiVgZNnT4soUehMZ7au8PwSl4I KYE4ulJ8RRiydN7fES3IZWmOPlyskp1QMQBD/w16o+lEtY6HSFEzsK3o0vuBRBVp2WKnssVH qeeV01ZHw0bvWKjxVNOksP98eJfWLfV9l9e7s6TaAeySKRRubtJ+21PRuYAxKsaueBfUE7ZT 7zfOwU0EVgUmGQEQALxSQRbl/QOnmssVDxWhHM5TGxl7oLNJms2zmBpcmlrIsn8nNz0rRyxT 460k2niaTwowSRK8KWVDeAW6ZAaWiYjLlTunoKwvF8vP3JyWpBz0diTxL5o+xpvy/Q6YU3BN efdq8Vy3rFsxgW7mMSrI/CxJ667y8ot5DVugeS2NyHfmZlPGE0Nsy7hlebS4liisXOrN3jFz asKyUws3VXek4V65lHwB23BVzsnFMn/bw/rPliqXGcwl8CoJu8dSyrCcd1Ibs0/Inq9S9+t0 VmWiQWfQkz4rvEeTQkp/VfgZ6z98JRW7S6l6eophoWs0/ZyRfOm+QVSqRfFZdxdP2PlGeIFM C3fXJgygXJkFPyWkVElr76JTbtSHsGWbt6xUlYHKXWo+xf9WgtLeby3cfSkEchACrxDrQpj+ Jt/JFP+q997dybkyZ5IoHWuPkn7uZGBrKIHmBunTco1+cKSuRiSCYpBIXZMHCzPgVDjk4viP brV9NwRkmaOxVvye0vctJeWvJ6KA7NoAURplIGCqkCRwg0MmLrfoZnK/gRqVJ/f6adhU1oo6 z4p2/z3PemA0C0ANatgHgBb90cd16AUxpdEQmOCmdNnNJF/3Zt3inzF+NFzHoM5Vwq6rc1JP jfC3oqRLJzqAEHBDjQFlqNR3IFCIAo4SYQRBdAHBCzkM4rWyRhuVABEBAAHCwV8EGAECAAkF AlYFJhkCGwwACgkQ8ww4vT8vvjwg9w//VQrcnVg3TsjEybxDEUBm8dBmnKqcnTBFmxN5FFtI WlEuY8+YMiWRykd8Ln9RJ/98/ghABHz9TN8TRo2b6WimV64FmlVn17Ri6FgFU3xNt9TTEChq AcNg88eYryKsYpFwegGpwUlaUaaGh1m9OrTzcQy+klVfZWaVJ9Nw0keoGRGb8j4XjVpL8+2x OhXKrM1fzzb8JtAuSbuzZSQPDwQEI5CKKxp7zf76J21YeRrEW4WDznPyVcDTa+tz++q2S/Bp P4W98bXCBIuQgs2m+OflERv5c3Ojldp04/S4NEjXEYRWdiCxN7ca5iPml5gLtuvhJMSy36gl U6IW9kn30IWuSoBpTkgV7rLUEhh9Ms82VWW/h2TxL8enfx40PrfbDtWwqRID3WY8jLrjKfTd R3LW8BnUDNkG+c4FzvvGUs8AvuqxxyHbXAfDx9o/jXfPHVRmJVhSmd+hC3mcQ+4iX5bBPBPM oDqSoLt5w9GoQQ6gDVP2ZjTWqwSRMLzNr37rJjZ1pt0DCMMTbiYIUcrhX8eveCJtY7NGWNyx FCRkhxRuGcpwPmRVDwOl39MB3iTsRighiMnijkbLXiKoJ5CDVvX5yicNqYJPKh5MFXN1bvsB kmYiStMRbrD0HoY1kx5/VozBtc70OU0EB8Wrv9hZD+Ofp0T3KOr1RUHvCZoLURfFhSQ= In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: k9amzSUuDCHM1dmHMue4VCyrGcoFlndclraLmY7Wx_0_1782114287 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: OCHEO6LAES6QOUYNNKJKEUXCSWG7MC5C X-Message-ID-Hash: OCHEO6LAES6QOUYNNKJKEUXCSWG7MC5C X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 6/22/26 04:36, David Gibson wrote: > On Fri, Jun 19, 2026 at 07:07:56PM +0200, Laurent Vivier wrote: >> On 6/19/26 08:00, David Gibson wrote: >>> On Tue, Jun 16, 2026 at 02:51:24PM +0200, Laurent Vivier wrote: >>>> Thread a qpair parameter from the entry points (tcp_sock_handler, >>>> tcp_timer_handler, tcp_tap_handler, tcp_defer_handler) through every >>>> intermediate function down to the vhost-user send functions, so callers >>>> explicitly select the target RX virtqueue instead of hardcoding >>>> QPAIR_DEFAULT. >>>> >>>> Add a qpair parameter to tcp_send_flag(), tcp_data_from_sock(), >>>> tcp_rst_do() and its tcp_rst() macro, tcp_rewind_seq(), >>>> tcp_data_from_tap(), tcp_conn_from_sock_finish(), tcp_connect_finish(), >>>> tcp_tap_window_update(), tcp_conn_from_tap(), tcp_rst_no_conn(), >>>> tcp_keepalive(), and tcp_inactivity(). >>> >>> For the to-guest functions which take a connection parameter, this >>> seems odd to me. Can't they deduce the right queue from the >>> connection? >> >> The connection's qpair (conn->f.qpair) can change at any time when >> another thread processes a tap packet for the same flow and calls >> FLOW_MIGRATE(). > > Ahh. Hmm. Right. I guess I don't really know the concurrency model > you're going for. I had assumed that each flow was pinned to a > thread, and migrating it was an operation requiring a maybe complex > and heavyweight synchronization step. > >> If a socket event fires on qpair 0's epoll instance and we read >> conn->f.qpair during processing, another thread might change it >> concurrently via FLOW_MIGRATE(). The qpair from the epoll event is >> the stable, race-free reference for "which queue am I operating on." > > Ok. For call chains initiated on the tap side, that's clear enough - > it's the queue we got the initiating event on. For things initiaed on > the socket side it's less clear what "queue I am operating on" means. > I guess it means the queue associated with the epoll set tne > initiating event occurred on? Yes, qpairs are binded to an epoll fd. In the 3rd series (I didn't send it, 2nd is about to handle concurrency), each qpair is binded to an epollfd and each epollfd is handled by a thread. The socket-side qpair is the queue associated with the epoll instance that delivere the event. > >> So the explicit parameter is intentional: it carries the qpair from >> the epoll event through the call chain, independent of the mutable >> flow state. > > I guess I haven't looked at the multithread series yet, but it feels > like the picture has a gap. In general we freely access fields in the > connection. I had assumed that meant it was "owned" by the current > thread. If the owning thread can change midstream, don't we need some > other sort of synchronization accessing *anything* in the flow entry? > Yes, you're right, I missed that. I added lock to the flow table but only to access the table, not the individual entry. Perhaps we can do the migration at the end of the processing of the entry, perhaps in the post processing function? Thanks, Laurent