From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=a6v0wSf0; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTP id C73C25A004C for ; Wed, 25 Sep 2024 19:48:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1727286484; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rEX6K90GPkSo3hFk+gM6DnAJor96RWquhTQnTrMbcqM=; b=a6v0wSf0FIrFqIDQPRabSFfaxPpBsDRqYFSxHAZVdr4lkVXiM5AY8qzcgWIPMmWG6L7syy e9ATZaSzzvHvqe5Tgyg4Xa4oEYeC0UdNC4Hcjn6AfPENbuWqrTVxag2fq4zAiB/q17+F2N Z4foQt8pw7LF7Ji7LjGm8IsVgvKv2ng= Received: from mail-pf1-f198.google.com (mail-pf1-f198.google.com [209.85.210.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-639-HkZkMTgOMDaP8OZDHiEr4A-1; Wed, 25 Sep 2024 13:48:03 -0400 X-MC-Unique: HkZkMTgOMDaP8OZDHiEr4A-1 Received: by mail-pf1-f198.google.com with SMTP id d2e1a72fcca58-718d51f33a6so136840b3a.2 for ; Wed, 25 Sep 2024 10:48:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1727286482; x=1727891282; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=rEX6K90GPkSo3hFk+gM6DnAJor96RWquhTQnTrMbcqM=; b=Cc2jCBIxhHD92FUW33iqjMMFL6HkTVoldvaI6hl5AEBKQxlXmEal2YD+FeWBHCeg8j 78qY6nJjgcLNOCuAI906AxMyNOtMhA8e+ZIjGi4d5h1HzWE4nALdJcgehwQxuRr3ijQ7 h5JCyj+kXZvq1oxTbrhw+Jax7P3cVgpPYeehV80YOKO1wBEnKDLit5fKnTwxgXkRj8ug E+U9dGbqBsYx6Iv1FO2QUc9cMhhOXsu9EUa+5RLnzpXOu57257QMgFwXrclcU9av8Xjw QbOHRl80M58DTOkg87t70+TQxUKzu9oZP2OWx6UaLfMEPxW0UELbLciIxppndV8deOq+ MFBg== X-Gm-Message-State: AOJu0YyzGJopvy5uEFNCgOECeE9h6EpqyuzImFXNMO8Mvzpfuwdzn1ZU 1VxptRYrDBRiEECHAsJC120vHOMpmMRj4jogogqZeyaJfh3YUOoiyO7p1j9oLp+x8elideMvkuW VqKZEIv5a2j4xtdBYnH6EdrgDNdZoV4fzo3HSLA9ZYhm39z54m+GDbhOLnYz6oyPoBFNuUjRQoL BhrS7FWsKKXyjmB10phDtpoTcx+xbtTnIu X-Received: by 2002:a05:6a00:b81:b0:706:3329:5533 with SMTP id d2e1a72fcca58-71b0afa2c26mr4820898b3a.24.1727286482158; Wed, 25 Sep 2024 10:48:02 -0700 (PDT) X-Google-Smtp-Source: AGHT+IFvDKURH/wCG1eOAP9VcQU6V+vGZ2uLP/XcmSweB3QBCzp4sniNcXmJyFMIE/PiRuro/k2/CA== X-Received: by 2002:a05:6a00:b81:b0:706:3329:5533 with SMTP id d2e1a72fcca58-71b0afa2c26mr4820869b3a.24.1727286481645; Wed, 25 Sep 2024 10:48:01 -0700 (PDT) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71afc939068sm2999030b3a.133.2024.09.25.10.48.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Sep 2024 10:48:01 -0700 (PDT) Date: Wed, 25 Sep 2024 19:47:58 +0200 From: Stefano Brivio To: Jon Maloy Subject: Re: Problematic throughput numbers Message-ID: <20240925194758.242c7f5c@elisabeth> In-Reply-To: References: Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: MDDBJYNPSJQ3D4SOWTG4UTYLBQTQXCOM X-Message-ID-Hash: MDDBJYNPSJQ3D4SOWTG4UTYLBQTQXCOM X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, dgibson@redhat.com, lvivier@redhat.com X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Wed, 25 Sep 2024 12:36:43 -0400 Jon Maloy wrote: > I made many runs with iperf3 ns->host, and the results are puzzling me. This has nothing to do with the path affected by your patches, because you're connecting to a loopback address, and that's dealt with by spliced connections. See tcp_splice.c, and: https://passt.top/#pasta-pack-a-subtle-tap-abstraction "Handling of local traffic in pasta" in pasta(1) for more details. If you want to try out your patches, you could run the test suite, or use a non-loopback address from the container. > Over and over again, I see throughput practically collapse, with a two > orders of magnitude decrease. > Just to make sure this wasn't something introduced by me, I went back to > the master branch and disabled the SO_PEEK_OFF feature. > The result was the same. > The below log shows typical run, but it is highly variable. > Sometimes almost the whole series is in the 50-70 Gb/s range, and > sometimes almost all in the 100-300 Mb/s range. > When I added the kernel fix it didn't seem to make any difference. > > To me this is really worrying, and should be investigated. I can't reproduce this, and we didn't have user reports of anything of this sort, so far. To investigate this, I would suggest that you have a look with strace (as root) at what pasta is doing once the throughput decreases. Packet captures with -p / --pcap won't show anything, because we don't actually deal with packets on the spliced path. Using --trace and a log file might help, but you'll probably decrease throughput enough, that way, that the "good" condition won't be distinguishable from the "bad" one. -- Stefano