From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=hTmqe2y2; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 671A95A0262 for ; Wed, 20 May 2026 22:53:52 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1779310431; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iCwD7i0+RB4Vz7Lh9Q2jspiL4YKZ0biSyargOXo3zPI=; b=hTmqe2y2jENwkBfYO6ubKnrwIXSo+9ElOiKonsdGyN0B80mnS1E/kzP0zy4SdG+BKOKylv rvno15/1uN9fBWDKBPcareTrlg/MC2uATcSqRElugWrsyTnCtio0b8zYR989eZ9HLhZdpB B461+BFbfGix4RDn3xtyQfbD5uJNddI= Received: from mail-wr1-f69.google.com (mail-wr1-f69.google.com [209.85.221.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-685-_GuzjYvnOEK6ugezeDrHtg-1; Wed, 20 May 2026 16:53:46 -0400 X-MC-Unique: _GuzjYvnOEK6ugezeDrHtg-1 X-Mimecast-MFC-AGG-ID: _GuzjYvnOEK6ugezeDrHtg_1779310425 Received: by mail-wr1-f69.google.com with SMTP id ffacd0b85a97d-43d7a5b9678so3490659f8f.2 for ; Wed, 20 May 2026 13:53:46 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20251104; t=1779310425; x=1779915225; h=date:content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=iCwD7i0+RB4Vz7Lh9Q2jspiL4YKZ0biSyargOXo3zPI=; b=ejYGO3MSHeggvnHBimBUy8RhJnEckCExZNEfoQl3cNPXHfKvor2I3Bl7i1FkZuVwlo JHeXj+Y9R5C34W/OXmqK9QNFRv0jZseY4L47hRSLkVyzm5s+Jer8ouhRC/JC7NB1jzPm fmFgWDvfT9a437Q2igTkIkx8Mv3PJhr/Ev296+Xqf3p8enW+G2O4yyd1x1w+X6dtKmiJ 1K7NOhMn6Cf/AOFFxr6GbAcoTrpveBvoCnjwa7G6RQZ8BZgKk97DWzfyVxWb2SuO1BZ3 EtCqESU6i5Yv1hv5s5umJ7HkTrKbJKrepKajUPJa0edI3wfqh8k07GZf3t86+3PjPTkR UjuA== X-Gm-Message-State: AOJu0YyqBHSiAnnvK1yFCooInPGH3ZDsNaXoNoiwplApdrgRlgP/P1bT YoctLpQzOG4bPluWnOSoRXUcrDytFZVBNH/si2JRjbolwou9Ydu7iU76qoRNnGziiIAINXg2LxW i/N5bcfJOQgyvugWDMqLk4ToBEOeQLojTrZWszh+h6vJvKNOpoJ88Kw== X-Gm-Gg: Acq92OHVJOaBVFl6P/JW3zm3IbXaRIWgMLdb0vUzLHHzl9eQiUBs6AFr2zhtkjxV+GU rVCISQ7Rl6bHfUMTTLsg9J/ug6KKUZ2Uez4QLkjnMn56KxNfAr6qa8/JsDlihbig+4vWsRdc147 arxaR3BGJEooBHS6BRdrkIpkurz0xHdd3HdAlWDhtBJxuAAKOyN9seM5F30vtAHOQjEQSbPAqMi Tzm3NL1XYc5Xhbtyk72a57SxCY1yc7f+lDVcf72uCUqXon54lthvcsfgdF8vxmgTxVQsIGoKbn4 NkwoHX5XrqbtPcDjoVbnzcBooBoduMei9N/hXDZvebcyFp5FArevrcs/uAL0SgA8yCD48IbZgPv +4BhgyhefOrak4sJM8sgvotRVJgNUGdaXc0X7vPQmQ6vbrQYNNg== X-Received: by 2002:a05:6000:2681:b0:45e:93ec:2deb with SMTP id ffacd0b85a97d-45e93ec2e3bmr7899395f8f.16.1779310425350; Wed, 20 May 2026 13:53:45 -0700 (PDT) X-Received: by 2002:a05:6000:2681:b0:45e:93ec:2deb with SMTP id ffacd0b85a97d-45e93ec2e3bmr7899359f8f.16.1779310424862; Wed, 20 May 2026 13:53:44 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-45d9e767cb9sm54441947f8f.2.2026.05.20.13.53.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 20 May 2026 13:53:44 -0700 (PDT) From: Stefano Brivio To: Laurent Vivier Subject: Re: [PATCH v4 00/10] vhost-user: Preparatory series for multiple iovec entries per virtqueue element Message-ID: <20260520225340.54490a21@elisabeth> In-Reply-To: <20260520181852.1f0119ff@elisabeth> References: <20260513115218.1662850-1-lvivier@redhat.com> <20260520173445.0658dfef@elisabeth> <20260520180708.275ec4de@elisabeth> <20260520181852.1f0119ff@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 Date: Wed, 20 May 2026 22:53:43 +0200 (CEST) X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: aFg-JdGsnyIQKXmHNSgd2eLX94t9Q0qIzZsODhAlwMw_1779310425 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: PQL4G2BEJ5A3HCZTA5ILP2PZ5NZKJTT3 X-Message-ID-Hash: PQL4G2BEJ5A3HCZTA5ILP2PZ5NZKJTT3 X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Jon Maloy , David GIbson X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 20 May 2026 18:18:52 +0200 Stefano Brivio wrote: > On Wed, 20 May 2026 18:07:08 +0200 > Stefano Brivio wrote: > > > On Wed, 20 May 2026 17:34:45 +0200 > > Stefano Brivio wrote: > > > > > On Wed, 13 May 2026 13:52:08 +0200 > > > Laurent Vivier wrote: > > > > > > > Currently, the vhost-user path assumes each virtqueue element contains > > > > exactly one iovec entry covering the entire frame. This assumption > > > > breaks as some virtio-net drivers (notably iPXE) provide descriptors where the > > > > vnet header and the frame payload are in separate buffers, resulting in > > > > two iovec entries per virtqueue element. > > > > > > > > This series refactors the vhost-user data path so that frame lengths, > > > > header sizes, and padding are tracked and passed explicitly rather than > > > > being derived from iovec sizes. This decoupling is a prerequisite for > > > > correctly handling padding of multi-buffer frames. > > > > > > Sorry to bring (likely) bad news, but this series seems to introduce a > > > regression: I got the migration/rampstream_in tests fail twice in a > > > row, which I've never saw happening (I think I saw a single failure a > > > long time ago when the machine had a high CPU load, but nothing else). > > > > > > I'm currently bisecting and the bisect seems to point towards the end > > > of the series (probably 10/10), but I haven't finished yet. I'll keep > > > you posted. I haven't spotted anything that might cause issues there. > > > > Yeah, that's the one :( > > > > $ git bisect bad > > db798fc60f4c5869cb53168354e068fb4dabd91a is the first bad commit > > commit db798fc60f4c5869cb53168354e068fb4dabd91a > > Author: Laurent Vivier > > Date: Wed May 13 13:52:18 2026 +0200 > > > > vhost-user: Centralise Ethernet frame padding in vu_collect() and vu_pad() > > > > The "TCP/IPv4: sequence check, ramps, inbound" test in rampstream_in > > gets stuck, once the source is done with the migration, and passt on the > > destination just printed: > > > > Accepted TCP_REPAIR helper, PID 13 > > accepted connection from PID 16 > > > > I'll get captures and logs next. It seems to fail most of the times, > > I had two failures in a row again. > > Log from passt --debug attached. Likely highlight: > > --- > 13.2853: ================ Vhost user message ================ > 13.2853: Request: VHOST_USER_SET_VRING_ADDR (9) > 13.2853: Flags: 0x1 > 13.2853: Size: 40 > 13.2853: vhost_vring_addr: > 13.2853: index: 0 > 13.2853: flags: 0 > 13.2853: desc_user_addr: 0x00007f0943f41000 > 13.2853: used_user_addr: 0x00007f0943f42240 > 13.2854: avail_user_addr: 0x00007f0943f42000 > 13.2854: log_guest_addr: 0x000000001ff43240 > 13.2854: Setting virtq addresses: > 13.2854: vring_desc at 0x7f2e2e2ca000 > 13.2854: vring_used at 0x7f2e2e2cb240 > 13.2854: vring_avail at 0x7f2e2e2cb000 > 13.2854: Last avail index != used index: 2163 != 1936 > 13.2854: Got packet, but RX virtqueue not usable yet > --- > > pcap file of that passt instance empty, it didn't have a chance to > send/receive packets yet. ...but I bisected 10/10 itself, and realised that reverting the iov_truncate() -> iov_skip_bytes() conversion in tcp_vu_sock_recv() like this: --- diff --git a/tcp_vu.c b/tcp_vu.c index f6ac76e..ccc031e 100644 --- a/tcp_vu.c +++ b/tcp_vu.c @@ -249,11 +249,7 @@ static ssize_t tcp_vu_sock_recv(const struct ctx *c, struct vu_virtq *vq, if (!peek_offset_cap) ret -= already_sent; - i = iov_skip_bytes(&iov_vu[DISCARD_IOV_NUM], iov_used, - MAX(hdrlen + ret, VNET_HLEN + ETH_ZLEN), - NULL); - if ((size_t)i < iov_used) - i++; + i = iov_truncate(&iov_vu[DISCARD_IOV_NUM], iov_used, ret); /* adjust head count */ while (*head_cnt > 0 && head[*head_cnt - 1] >= i) --- hides / fixes the issue. I'm testing things on a kernel without SO_PEEK_OFF support for TCP, but it doesn't seem to matter ('ret' at this point is the same before and after your patch). I don't see what's wrong with your change though. It's not even about replacing 'ret' with the padded version, because I can also reproduce the issue with: i = iov_skip_bytes(&iov_vu[DISCARD_IOV_NUM], iov_used, ret, NULL); For convenience, this is how I'm selecting the test without bothering about variables in run(): --- diff --git a/test/run b/test/run index f858e55..25d7002 100755 --- a/test/run +++ b/test/run @@ -71,6 +71,7 @@ run() { perf_init [ ${CI} -eq 1 ] && video_start ci +dont() { exeter smoke/smoke.sh exeter build/build.py exeter build/static_checkers.sh @@ -162,6 +163,10 @@ run() { setup migrate test migrate/iperf3_many_out6 teardown migrate +} + VHOST_USER=1 + VALGRIND=0 + setup migrate test migrate/rampstream_in teardown migrate --- -- Stefano