From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=dvm3+PGl; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 5F8175A0265 for ; Thu, 21 May 2026 10:30:49 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1779352248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=Fpxa7XLoCTMzqXLQodZ4qAUyvmN8RG/MvPuDEyEHtYk=; b=dvm3+PGllcpM9LrUbgX7ZtpOp5UcpW3EMByU2VhFKIt2EcclyOdTbXF2gh8Tf5b+uBrDe/ AbB4gbqNUtPvDNh4IwmC5SyXGWbKgNgCtlWwiK0oTAtI/2u6ESVUwYybxv+pClcJ5Ual+V RaS+HYEBjpAGhQblSgqksDa4CsgQuqM= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-284-LWqZOWzZPMSAu_uzVFTJbA-1; Thu, 21 May 2026 04:30:46 -0400 X-MC-Unique: LWqZOWzZPMSAu_uzVFTJbA-1 X-Mimecast-MFC-AGG-ID: LWqZOWzZPMSAu_uzVFTJbA_1779352246 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-44bf1ac8893so4554050f8f.0 for ; Thu, 21 May 2026 01:30:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779352245; x=1779957045; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Fpxa7XLoCTMzqXLQodZ4qAUyvmN8RG/MvPuDEyEHtYk=; b=VR+5Q9KX94jEfIPn8aCvC6PoFTBTNvPNIvMJHbw6s8XjAfsvByX7T+MHLICWGDc/1y dkvjj7PAC4fE+kAEnYAdmLU5QOSHgUG9KXdyPBr2s0U7OP31UAhX2BBI9O4ID9jkmVlA +zz0h2cv+Xv/kLfnLlLrXKQlSNHsqDKi1HLn1joU4xw+5Fx6YCeI2Z2JDSlrxHtKRINu ihZ5LSuNk2uWFkumr2iAaiWDiEnUuGmgG7A4dy+HY4F++rHJFT4YtO7b2E2aYFC/1xoe BGJcafS7cuEVl8hvEKQiCP/cgaurRTYR2yLPm1rJ8aopl4gKFQki73XQvd6NLxNoD/GR la+w== X-Gm-Message-State: AOJu0YxADoB2sbOq421bqvju6fXM04Y6i7Bu3Ojo6c4R1IcJ2QX01R45 SrEll0q1Na3Bcfdw7VKEV05xJgXXrJEnkq2ocLl+jBcorntNSc+DRUQXDACe4jA1B9LwBWIZyPX wyPawvbU2QQ+dpwa4y2rrFsKE5wj3HVL2mRQFCgCrYikWU9uvfxT7MT9LFbVQo7Y3 X-Gm-Gg: Acq92OH9xM8YSOQZfjAEiHd5Lo4wUdjq3ISCy85fZ/8NnchoXHJ7HhrtzyzXCgV1LhO g9CqtwyIhiH6KhudB1WYW7fbDNPnkAtay6UX5DoySPHwOcewaVUbUhpMQx34zQd9OK9Z96wBpHn GZ9G0xuoQD5Y7T7UyEPAS2bLeZ17/EpkbLGmMES9F49/2EBOOlKQo94WV0JXS/zRuEnXdoy4X0b AlYWclsOPdiWff16SXtoyQbHE9y8tvT2BYz8GQkqSBIlxsDu3hgabB+dbrc/DYFdqyU1BADnTbZ /RBtFiNcudMKsYO+5CFDTvi9KLIEzv+A2FHEzeBEZ2V6G0eOVo6uFObBO9nL4rYy4j11WkqSLTH AiooujJxeSSL1uhj04O2wuzaEeNsI8TkA6OG2DCc2HQdT97tBYttNymI= X-Received: by 2002:a05:6000:41f9:b0:45e:891c:648e with SMTP id ffacd0b85a97d-45ea38cd490mr2473773f8f.8.1779352245341; Thu, 21 May 2026 01:30:45 -0700 (PDT) X-Received: by 2002:a05:6000:41f9:b0:45e:891c:648e with SMTP id ffacd0b85a97d-45ea38cd490mr2473671f8f.8.1779352244655; Thu, 21 May 2026 01:30:44 -0700 (PDT) Received: from [192.168.100.100] (82-64-211-94.subs.proxad.net. [82.64.211.94]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-45eaa7dd9e6sm1053914f8f.16.2026.05.21.01.30.43 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 21 May 2026 01:30:44 -0700 (PDT) Message-ID: Date: Thu, 21 May 2026 10:30:43 +0200 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 00/10] vhost-user: Preparatory series for multiple iovec entries per virtqueue element To: Stefano Brivio References: <20260513115218.1662850-1-lvivier@redhat.com> <20260520173445.0658dfef@elisabeth> <20260520180708.275ec4de@elisabeth> <20260520181852.1f0119ff@elisabeth> <20260520225340.54490a21@elisabeth> From: Laurent Vivier Autocrypt: addr=lvivier@redhat.com; keydata= xsFNBFYFJhkBEAC2me7w2+RizYOKZM+vZCx69GTewOwqzHrrHSG07MUAxJ6AY29/+HYf6EY2 WoeuLWDmXE7A3oJoIsRecD6BXHTb0OYS20lS608anr3B0xn5g0BX7es9Mw+hV/pL+63EOCVm SUVTEQwbGQN62guOKnJJJfphbbv82glIC/Ei4Ky8BwZkUuXd7d5NFJKC9/GDrbWdj75cDNQx UZ9XXbXEKY9MHX83Uy7JFoiFDMOVHn55HnncflUncO0zDzY7CxFeQFwYRbsCXOUL9yBtqLer Ky8/yjBskIlNrp0uQSt9LMoMsdSjYLYhvk1StsNPg74+s4u0Q6z45+l8RAsgLw5OLtTa+ePM JyS7OIGNYxAX6eZk1+91a6tnqfyPcMbduxyBaYXn94HUG162BeuyBkbNoIDkB7pCByed1A7q q9/FbuTDwgVGVLYthYSfTtN0Y60OgNkWCMtFwKxRaXt1WFA5ceqinN/XkgA+vf2Ch72zBkJL RBIhfOPFv5f2Hkkj0MvsUXpOWaOjatiu0fpPo6Hw14UEpywke1zN4NKubApQOlNKZZC4hu6/ 8pv2t4HRi7s0K88jQYBRPObjrN5+owtI51xMaYzvPitHQ2053LmgsOdN9EKOqZeHAYG2SmRW LOxYWKX14YkZI5j/TXfKlTpwSMvXho+efN4kgFvFmP6WT+tPnwARAQABzSNMYXVyZW50IFZp dmllciA8bHZpdmllckByZWRoYXQuY29tPsLBeAQTAQIAIgUCVgVQgAIbAwYLCQgHAwIGFQgC CQoLBBYCAwECHgECF4AACgkQ8ww4vT8vvjwpgg//fSGy0Rs/t8cPFuzoY1cex4limJQfReLr SJXCANg9NOWy/bFK5wunj+h/RCFxIFhZcyXveurkBwYikDPUrBoBRoOJY/BHK0iZo7/WQkur 6H5losVZtrotmKOGnP/lJYZ3H6OWvXzdz8LL5hb3TvGOP68K8Bn8UsIaZJoeiKhaNR0sOJyI YYbgFQPWMHfVwHD/U+/gqRhD7apVysxv5by/pKDln1I5v0cRRH6hd8M8oXgKhF2+rAOL7gvh jEHSSWKUlMjC7YwwjSZmUkL+TQyE18e2XBk85X8Da3FznrLiHZFHQ/NzETYxRjnOzD7/kOVy gKD/o7asyWQVU65mh/ECrtjfhtCBSYmIIVkopoLaVJ/kEbVJQegT2P6NgERC/31kmTF69vn8 uQyW11Hk8tyubicByL3/XVBrq4jZdJW3cePNJbTNaT0d/bjMg5zCWHbMErUib2Nellnbg6bc 2HLDe0NLVPuRZhHUHM9hO/JNnHfvgiRQDh6loNOUnm9Iw2YiVgZNnT4soUehMZ7au8PwSl4I KYE4ulJ8RRiydN7fES3IZWmOPlyskp1QMQBD/w16o+lEtY6HSFEzsK3o0vuBRBVp2WKnssVH qeeV01ZHw0bvWKjxVNOksP98eJfWLfV9l9e7s6TaAeySKRRubtJ+21PRuYAxKsaueBfUE7ZT 7zfOwU0EVgUmGQEQALxSQRbl/QOnmssVDxWhHM5TGxl7oLNJms2zmBpcmlrIsn8nNz0rRyxT 460k2niaTwowSRK8KWVDeAW6ZAaWiYjLlTunoKwvF8vP3JyWpBz0diTxL5o+xpvy/Q6YU3BN efdq8Vy3rFsxgW7mMSrI/CxJ667y8ot5DVugeS2NyHfmZlPGE0Nsy7hlebS4liisXOrN3jFz asKyUws3VXek4V65lHwB23BVzsnFMn/bw/rPliqXGcwl8CoJu8dSyrCcd1Ibs0/Inq9S9+t0 VmWiQWfQkz4rvEeTQkp/VfgZ6z98JRW7S6l6eophoWs0/ZyRfOm+QVSqRfFZdxdP2PlGeIFM C3fXJgygXJkFPyWkVElr76JTbtSHsGWbt6xUlYHKXWo+xf9WgtLeby3cfSkEchACrxDrQpj+ Jt/JFP+q997dybkyZ5IoHWuPkn7uZGBrKIHmBunTco1+cKSuRiSCYpBIXZMHCzPgVDjk4viP brV9NwRkmaOxVvye0vctJeWvJ6KA7NoAURplIGCqkCRwg0MmLrfoZnK/gRqVJ/f6adhU1oo6 z4p2/z3PemA0C0ANatgHgBb90cd16AUxpdEQmOCmdNnNJF/3Zt3inzF+NFzHoM5Vwq6rc1JP jfC3oqRLJzqAEHBDjQFlqNR3IFCIAo4SYQRBdAHBCzkM4rWyRhuVABEBAAHCwV8EGAECAAkF AlYFJhkCGwwACgkQ8ww4vT8vvjwg9w//VQrcnVg3TsjEybxDEUBm8dBmnKqcnTBFmxN5FFtI WlEuY8+YMiWRykd8Ln9RJ/98/ghABHz9TN8TRo2b6WimV64FmlVn17Ri6FgFU3xNt9TTEChq AcNg88eYryKsYpFwegGpwUlaUaaGh1m9OrTzcQy+klVfZWaVJ9Nw0keoGRGb8j4XjVpL8+2x OhXKrM1fzzb8JtAuSbuzZSQPDwQEI5CKKxp7zf76J21YeRrEW4WDznPyVcDTa+tz++q2S/Bp P4W98bXCBIuQgs2m+OflERv5c3Ojldp04/S4NEjXEYRWdiCxN7ca5iPml5gLtuvhJMSy36gl U6IW9kn30IWuSoBpTkgV7rLUEhh9Ms82VWW/h2TxL8enfx40PrfbDtWwqRID3WY8jLrjKfTd R3LW8BnUDNkG+c4FzvvGUs8AvuqxxyHbXAfDx9o/jXfPHVRmJVhSmd+hC3mcQ+4iX5bBPBPM oDqSoLt5w9GoQQ6gDVP2ZjTWqwSRMLzNr37rJjZ1pt0DCMMTbiYIUcrhX8eveCJtY7NGWNyx FCRkhxRuGcpwPmRVDwOl39MB3iTsRighiMnijkbLXiKoJ5CDVvX5yicNqYJPKh5MFXN1bvsB kmYiStMRbrD0HoY1kx5/VozBtc70OU0EB8Wrv9hZD+Ofp0T3KOr1RUHvCZoLURfFhSQ= In-Reply-To: <20260520225340.54490a21@elisabeth> X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: -f0YcxU4mzvqUczuNunzfQ9jMGV2O4kG-MP6ZutKVJ0_1779352246 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: ZCLFOFJ26FHS4NNXG7IGBY2US743UPF7 X-Message-ID-Hash: ZCLFOFJ26FHS4NNXG7IGBY2US743UPF7 X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Jon Maloy , David GIbson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 5/20/26 22:53, Stefano Brivio wrote: > On Wed, 20 May 2026 18:18:52 +0200 > Stefano Brivio wrote: > >> On Wed, 20 May 2026 18:07:08 +0200 >> Stefano Brivio wrote: >> >>> On Wed, 20 May 2026 17:34:45 +0200 >>> Stefano Brivio wrote: >>> >>>> On Wed, 13 May 2026 13:52:08 +0200 >>>> Laurent Vivier wrote: >>>> >>>>> Currently, the vhost-user path assumes each virtqueue element contains >>>>> exactly one iovec entry covering the entire frame. This assumption >>>>> breaks as some virtio-net drivers (notably iPXE) provide descriptors where the >>>>> vnet header and the frame payload are in separate buffers, resulting in >>>>> two iovec entries per virtqueue element. >>>>> >>>>> This series refactors the vhost-user data path so that frame lengths, >>>>> header sizes, and padding are tracked and passed explicitly rather than >>>>> being derived from iovec sizes. This decoupling is a prerequisite for >>>>> correctly handling padding of multi-buffer frames. >>>> >>>> Sorry to bring (likely) bad news, but this series seems to introduce a >>>> regression: I got the migration/rampstream_in tests fail twice in a >>>> row, which I've never saw happening (I think I saw a single failure a >>>> long time ago when the machine had a high CPU load, but nothing else). >>>> >>>> I'm currently bisecting and the bisect seems to point towards the end >>>> of the series (probably 10/10), but I haven't finished yet. I'll keep >>>> you posted. I haven't spotted anything that might cause issues there. >>> >>> Yeah, that's the one :( >>> >>> $ git bisect bad >>> db798fc60f4c5869cb53168354e068fb4dabd91a is the first bad commit >>> commit db798fc60f4c5869cb53168354e068fb4dabd91a >>> Author: Laurent Vivier >>> Date: Wed May 13 13:52:18 2026 +0200 >>> >>> vhost-user: Centralise Ethernet frame padding in vu_collect() and vu_pad() I checked on my system with the commit previous to this series, bcc3d37a6e01 ("util: Fix changes to assert_with_msg()") and rampstream_in fails too (not everytime). > TCP/IPv4: sequence check, ramps, inbound ...failed. and rampstream_out hangs sometime too. I'm going to try with ealier commits. Thanks, Laurent >>> >>> The "TCP/IPv4: sequence check, ramps, inbound" test in rampstream_in >>> gets stuck, once the source is done with the migration, and passt on the >>> destination just printed: >>> >>> Accepted TCP_REPAIR helper, PID 13 >>> accepted connection from PID 16 >>> >>> I'll get captures and logs next. It seems to fail most of the times, >>> I had two failures in a row again. >> >> Log from passt --debug attached. Likely highlight: >> >> --- >> 13.2853: ================ Vhost user message ================ >> 13.2853: Request: VHOST_USER_SET_VRING_ADDR (9) >> 13.2853: Flags: 0x1 >> 13.2853: Size: 40 >> 13.2853: vhost_vring_addr: >> 13.2853: index: 0 >> 13.2853: flags: 0 >> 13.2853: desc_user_addr: 0x00007f0943f41000 >> 13.2853: used_user_addr: 0x00007f0943f42240 >> 13.2854: avail_user_addr: 0x00007f0943f42000 >> 13.2854: log_guest_addr: 0x000000001ff43240 >> 13.2854: Setting virtq addresses: >> 13.2854: vring_desc at 0x7f2e2e2ca000 >> 13.2854: vring_used at 0x7f2e2e2cb240 >> 13.2854: vring_avail at 0x7f2e2e2cb000 >> 13.2854: Last avail index != used index: 2163 != 1936 >> 13.2854: Got packet, but RX virtqueue not usable yet >> --- >> >> pcap file of that passt instance empty, it didn't have a chance to >> send/receive packets yet. > > ...but I bisected 10/10 itself, and realised that reverting the > iov_truncate() -> iov_skip_bytes() conversion in tcp_vu_sock_recv() > like this: > > --- > diff --git a/tcp_vu.c b/tcp_vu.c > index f6ac76e..ccc031e 100644 > --- a/tcp_vu.c > +++ b/tcp_vu.c > @@ -249,11 +249,7 @@ static ssize_t tcp_vu_sock_recv(const struct ctx > *c, struct vu_virtq *vq, if (!peek_offset_cap) > ret -= already_sent; > > - i = iov_skip_bytes(&iov_vu[DISCARD_IOV_NUM], iov_used, > - MAX(hdrlen + ret, VNET_HLEN + ETH_ZLEN), > - NULL); > - if ((size_t)i < iov_used) > - i++; > + i = iov_truncate(&iov_vu[DISCARD_IOV_NUM], iov_used, ret); > > /* adjust head count */ > while (*head_cnt > 0 && head[*head_cnt - 1] >= i) > --- > > hides / fixes the issue. > > I'm testing things on a kernel without SO_PEEK_OFF support for TCP, but > it doesn't seem to matter ('ret' at this point is the same before and > after your patch). > > I don't see what's wrong with your change though. It's not even about > replacing 'ret' with the padded version, because I can also reproduce > the issue with: > > i = iov_skip_bytes(&iov_vu[DISCARD_IOV_NUM], iov_used, ret, > NULL); > > For convenience, this is how I'm selecting the test without bothering > about variables in run(): > > --- > diff --git a/test/run b/test/run > index f858e55..25d7002 100755 > --- a/test/run > +++ b/test/run > @@ -71,6 +71,7 @@ run() { > perf_init > [ ${CI} -eq 1 ] && video_start ci > > +dont() { > exeter smoke/smoke.sh > exeter build/build.py > exeter build/static_checkers.sh > @@ -162,6 +163,10 @@ run() { > setup migrate > test migrate/iperf3_many_out6 > teardown migrate > +} > + VHOST_USER=1 > + VALGRIND=0 > + > setup migrate > test migrate/rampstream_in > teardown migrate > --- >