From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=HtI53uy7; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 346045A0271 for ; Fri, 25 Jul 2025 16:18:42 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1753453121; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dWPVwyFgyaRUbD8i3IVC/ugIg2U5SmMwn/1ZR8XDqCw=; b=HtI53uy750kvR4l4qF/vZArIactFSXOGGWMWw3BWhODmZ2qZLkWMHVjuRxgfvsGjv59oFR hqZmf8KOhOF6km09tzBEBBTJ9mFn9pbw8olIbP48sCOzBqFenKPEo+xfStTKgu98TRsjvx 0fNy+QKqQT+R/YegI7U3b7KsLdrLhPA= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-494-DuJoWoREPW-2zZ8y9J2vjQ-1; Fri, 25 Jul 2025 10:18:39 -0400 X-MC-Unique: DuJoWoREPW-2zZ8y9J2vjQ-1 X-Mimecast-MFC-AGG-ID: DuJoWoREPW-2zZ8y9J2vjQ_1753453119 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-3b604541741so1569451f8f.3 for ; Fri, 25 Jul 2025 07:18:39 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1753453118; x=1754057918; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=ezNfWh7Cg+tSIUEVc8HCCcrml7rz5RDzvd5pix4eRzc=; b=h4mhaa41Z0DFDcyBzXZLkrkVMElLCPpqTSdvg+4sFnmF0zy1V6w3dMYmsy1EyVVuIv IDiUNZlnrRP4haj+uOcDW5yXgs1s1DKMT1SKrBWSwZxYdCfagJ+qpde95/FnQx7/YtAj 6e6rmeABjVxb6CYjJp9Aj/kygVBT8xNhP+cD6pRcRt10C0LrnS8MZ1UbybKWOwHMZhA8 na+SUoDw2fmll/03w8IeSDvfmyWDeb5XsJnpFa46JB828Hg8V0QWq4qFJrD2T7hyxjmp U8aVmdTaTtO2JeBY0diskiOsFpGsw202SCtU00grGSAgl8+JIMBdqpLo195VszNDoeDf +N/Q== X-Gm-Message-State: AOJu0YwG+RGnDBpsxU1v0cGq/mPAYKZJc3TgRbFwbGLM6MQpV13VFzGT /Ndc1VVNeuuQ6dhyH8ySPoAhTW0yRKp78B6AQfRLhiqsC0bF1u9M0d+YAP9z9m/KLjWp3CJdM3c sBjjWWfvuxy63N7VOFHOS9Jm+mPUgH4NTu5ilgKjWtMyIUzjUt3O4BH+VBf0Lj8LVEBpTAPb57U IWKh+dJuKskB/rGhEWCj2LBYXBRU+73YilSZsC X-Gm-Gg: ASbGncs8Zl8VYlsDARWW0Ynak07ifxKr2fBVIb2C50r5RH2pUKE8wlZ+oX82WVoWf7R Z1Nk2ieG27CagCCr5arNj6iA3R0qAI+y4fTWPckdxvY+SAQCnR67On3DWoPtNtpwBkgHdgWrnf2 DYdAk68Xy/dBhRKP+FmmJ2obMcF/sX6PclUO9GxyWCCT46BJXo7ml8b6uhzY50gfjePFqfJH0qL 2XPaSBmQs3Trl46lB4a1k4NA+pJo+8f7qfFdrn/7WSmv7VWaG1WopwUEKFehBVu6c9vHAESBOVb WZT4qtbWTyrJ4UWrQNHWmvtP5IYuRlSVT4rP7pbcbr60/+5yGHgwljSzR39ILEpC/5yZ X-Received: by 2002:a05:6000:2913:b0:3a4:e6e6:a026 with SMTP id ffacd0b85a97d-3b776664aaamr1938736f8f.28.1753453117644; Fri, 25 Jul 2025 07:18:37 -0700 (PDT) X-Google-Smtp-Source: AGHT+IH+sOhKXiGqPbBxjU+Yndjqc+cT7JfIyK9vazW5xFr8ga8NeV/fX23Dv6bT/sgzYyPPtiR5vw== X-Received: by 2002:a05:6000:2913:b0:3a4:e6e6:a026 with SMTP id ffacd0b85a97d-3b776664aaamr1938699f8f.28.1753453117066; Fri, 25 Jul 2025 07:18:37 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3b778ec2b63sm49022f8f.36.2025.07.25.07.18.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 25 Jul 2025 07:18:36 -0700 (PDT) Date: Fri, 25 Jul 2025 16:18:23 +0200 From: Stefano Brivio To: Laurent Vivier Subject: Re: [PATCH v7 00/31] Introduce discontiguous frames management Message-ID: <20250725161823.4329b411@elisabeth> In-Reply-To: References: <20250623110635.1478625-1-lvivier@redhat.com> <20250718204546.5ff05d49@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.49; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Y52Pfx11gykZOCuciGNGZFtBR41RhUl37TGrsC5uaxo_1753453119 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Message-ID-Hash: HD6UNSCQVMYTMFW7NCP6TTGDRJ3JAFAH X-Message-ID-Hash: HD6UNSCQVMYTMFW7NCP6TTGDRJ3JAFAH X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Fri, 25 Jul 2025 13:45:36 +0200 Laurent Vivier wrote: > On 24/07/2025 15:01, Laurent Vivier wrote: > > On 18/07/2025 20:45, Stefano Brivio wrote: =20 > >> On Mon, 23 Jun 2025 13:06:04 +0200 > >> Laurent Vivier wrote: > >> =20 > >>> This series introduces iov_tail to convey frame information > >>> between functions. > >>> > >>> This is only an API change, for the moment the memory pool > >>> is only able to store contiguous buffer, so, except for > >>> vhost-user in a special case, we only play with iovec array > >>> with only one entry. > >>> > >>> v7: > >>> =C2=A0=C2=A0=C2=A0=C2=A0 - Add a patch to fix comment style of 'Retur= n:' > >>> =C2=A0=C2=A0=C2=A0=C2=A0 - Fix ignore_arp()/accept_arp() > >>> =C2=A0=C2=A0=C2=A0=C2=A0 - Fix coverity error > >>> =C2=A0=C2=A0=C2=A0=C2=A0 - Fix several comments =20 > >> > >> I was about to apply this without 1/31 (I applied the v2 of it you sen= t > >> outside of this series instead, which is actually up to date) and with > >> the minor comment fix to 31/31... but the test perf/passt_vu_tcp fails > >> rather consistently now (and I triple checked without this series): > >> > >> - "TCP throughput over IPv6: guest to host" with MTU 1500 and 9000 > >> =C2=A0=C2=A0 bytes now reports between 0 and 0.6 Gbps. The guest kerne= l prints a > >> =C2=A0=C2=A0 series of two messages with ~1-10 =C2=B5s interval: > >> > >> [=C2=A0=C2=A0 21.159827] TCP: out of memory -- consider tuning tcp_mem > >> [=C2=A0=C2=A0 21.159831] TCP: out of memory -- consider tuning tcp_mem > >> > >> - "TCP throughput over IPv4: guest to host" never reports 0 Gbps, but > >> =C2=A0=C2=A0 the throughput figure for large MTU (65520 bytes) is very= low (5.4 > >> =C2=A0=C2=A0 Gbps in the last run). Here I'm getting four messages: > >> > >> [=C2=A0=C2=A0 40.807818] TCP: out of memory -- consider tuning tcp_mem > >> [=C2=A0=C2=A0 40.807829] TCP: out of memory -- consider tuning tcp_mem > >> [=C2=A0=C2=A0 40.807829] TCP: out of memory -- consider tuning tcp_mem > >> [=C2=A0=C2=A0 40.807830] TCP: out of memory -- consider tuning tcp_mem > >> > >> - on the reverse direction, "TCP throughput over IPv4: host to guest" > >> =C2=A0=C2=A0 (but not with IPv6), the iperf3 client gets SIGSEGV, but = not > >> =C2=A0=C2=A0 consistently, it happened once out of five times. > >> > >> To me it smells a bit like we're leaking virtqueue slots but I looked > >> again at the whole series and I couldn't find anything obvious... at > >> least not yet. > >> > >> UDP tests never fail and the throughput is the same as before. > >> =20 > >=20 > > I think the problem is the way we use the iovec array. > >=20 > > In tap4_handler() we have a packet_get() that provides a pointer to the= iovec array from=20 > > pool. Idx is 0, iovec idx is 0. > >=20 > > Then we have a pool_flush(), so first available idx is now 0 again. > >=20 > > And then we have packet_add() with the iovec idx (in "data") of the pre= vious packet_get()=20 > > that we try to add at the same index (as pool is empty again, and first= available idx is 0). > >=20 > > When I wrote this patch I guessed that when we release packet (pool_flu= sh()) we don't use=20 > > anymore the iovec array of the packet, it appears to be not true. =20 >=20 > Could you try the following patch (my iperf3 continue to crash = on my host=20 > system, not related to this problem), it's a little bit ugly (use of allo= ca()) but it's an=20 > easy fix. Weird, I don't get the "TCP: out of memory" messages anymore, but throughput still has some obvious issues: =3D=3D=3D perf/passt_vu_tcp > passt: throughput and latency Throughput in Gbps, latency in =C2=B5s, 6 threads at 3.6 GHz MTU: | 256B | 576B | 1280B | 1500B | 900= 0B | 65520B | |--------|--------|--------|--------|-----= ---|--------| TCP throughput over IPv6: guest to host | - | - | 0.6 | = 2.0 | 0 | 13.3 | TCP RR latency over IPv6: guest to host | - | - | - | = - | - | 38 | TCP CRR latency over IPv6: guest to host | - | - | - | = - | - | 95 | |--------|--------|--------|--------|-----= ---|--------| TCP throughput over IPv4: guest to host | 0 | 2.2 | 1.0 | = 1.7 | 9.9 | 16.7 | TCP RR latency over IPv4: guest to host | - | - | - | = - | - | 35 | TCP CRR latency over IPv4: guest to host | - | - | - | = - | - | 91 | |--------|--------|--------|--------|-----= ---|--------| ...we should never have < 0.1 Gbps, and with 64k MTU, we otherwise have 30 - 40 Gbps. Even more weird, iperf3 now reliably crashes for me too, on the other path (host to guest). It never did without this series. I wonder if the issue is actually "fixed" with this patch but the alloca() is so large as to actually mess with throughput. I can play with that if it helps, or bisect... let me know. I have to admit I didn't fully grasp the problem at hand, simply because I haven't read carefully what you wrote yet. --=20 Stefano