From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=none (p=none dis=none) header.from=gibson.dropbear.id.au Authentication-Results: passt.top; dkim=pass (2048-bit key; secure) header.d=gibson.dropbear.id.au header.i=@gibson.dropbear.id.au header.a=rsa-sha256 header.s=202408 header.b=YRygzDJI; dkim-atps=neutral Received: from mail.ozlabs.org (mail.ozlabs.org [IPv6:2404:9400:2221:ea00::3]) by passt.top (Postfix) with ESMTPS id E6AD15A004C for ; Thu, 26 Sep 2024 05:42:12 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202408; t=1727322125; bh=TpqMMX/uQdaV8c9WURYD/qzMTkzAO/iF29gSlIeHKs0=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=YRygzDJIEh+xyJmpQtlTU6HiXYcX3Art8FQ85hKMePAYKZBEixfFI2GyLVlvw0mhc QhHpu03sfhD5DBy3iRaRzBWWCTXbsgHH1pcoO/3V4DS0tIhkq86ZI5adSKAGWugok9 qDehNxkfITBdQ/acOOJ1yELlgTjNz6opwPwDq5ojfx6drBFb1QMHbTVYociBWTPoGx P5HKUdaF2oOEmI4hHDt/jIGbIwIY/0U1d7pnqWf0be5g0Mc8SnElSgIGhVtOOHUUvy 0tYRPS1Tyggiv74b4HTF7YVXC2I6L2pBHinOcwA5rqi/gDOxwqUfCI9Clf+6QzXu9P CSLseKbhBICdw== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4XDfY90PxQz4xQv; Thu, 26 Sep 2024 13:42:05 +1000 (AEST) Date: Thu, 26 Sep 2024 11:59:17 +1000 From: David Gibson To: Jon Maloy Subject: Re: Problematic throughput numbers Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="HGPJ0PKGvKYhrM+1" Content-Disposition: inline In-Reply-To: Message-ID-Hash: XYO4TVS5WSPRPL7KJKZ3BKWCFMHBUCEH X-Message-ID-Hash: XYO4TVS5WSPRPL7KJKZ3BKWCFMHBUCEH X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, sbrivio@redhat.com, dgibson@redhat.com, lvivier@redhat.com X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --HGPJ0PKGvKYhrM+1 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Sep 25, 2024 at 12:36:43PM -0400, Jon Maloy wrote: > I made many runs with iperf3 ns->host, and the results are puzzling me. > Over and over again, I see throughput practically collapse, with a two > orders of magnitude decrease. > Just to make sure this wasn't something introduced by me, I went back to = the > master branch and disabled the SO_PEEK_OFF feature. > The result was the same. > The below log shows typical run, but it is highly variable. > Sometimes almost the whole series is in the 50-70 Gb/s range, and sometim= es > almost all in the 100-300 Mb/s range. > When I added the kernel fix it didn't seem to make any difference. >=20 > To me this is really worrying, and should be investigated. Those numbers are certainly worrying. Like Stefano, though, I have encountered this myself. >=20 > ///jon >=20 >=20 >=20 >=20 > pasta NS->host (master branch, SO_PEEK_OFF disabled >=20 > ----------------------------------------------------------- > Server listening on 5201 (test #2) > ----------------------------------------------------------- > Accepted connection from 127.0.0.1, port 48354 > [=A0 5] local 127.0.0.1 port 5201 connected to 127.0.0.1 port 48360 > [ ID] Interval=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Transfer=A0=A0=A0=A0 Bitrate > [=A0 5]=A0=A0 0.00-1.00=A0=A0 sec=A0 7.61 GBytes=A0 65.3 Gbits/sec > [=A0 5]=A0=A0 1.00-2.00=A0=A0 sec=A0 7.59 GBytes=A0 65.2 Gbits/sec > [=A0 5]=A0=A0 2.00-3.00=A0=A0 sec=A0 7.55 GBytes=A0 64.8 Gbits/sec > [=A0 5]=A0=A0 3.00-4.00=A0=A0 sec=A0 7.69 GBytes=A0 66.1 Gbits/sec > [=A0 5]=A0=A0 4.00-5.00=A0=A0 sec=A0 7.55 GBytes=A0 64.8 Gbits/sec > [=A0 5]=A0=A0 5.00-6.00=A0=A0 sec=A0 7.60 GBytes=A0 65.3 Gbits/sec > [=A0 5]=A0=A0 6.00-7.00=A0=A0 sec=A0 7.52 GBytes=A0 64.6 Gbits/sec > [=A0 5]=A0=A0 7.00-8.00=A0=A0 sec=A0 1.02 GBytes=A0 8.73 Gbits/sec > [=A0 5]=A0=A0 8.00-9.00=A0=A0 sec=A0 3.50 MBytes=A0 29.4 Mbits/sec > [=A0 5]=A0=A0 9.00-10.00=A0 sec=A0 17.2 MBytes=A0=A0 145 Mbits/sec > [=A0 5]=A0 10.00-11.00=A0 sec=A0 64.0 MBytes=A0=A0 537 Mbits/sec > [=A0 5]=A0 11.00-12.00=A0 sec=A0 24.1 MBytes=A0=A0 202 Mbits/sec > [=A0 5]=A0 12.00-13.00=A0 sec=A0 25.4 MBytes=A0=A0 213 Mbits/sec > [=A0 5]=A0 13.00-14.00=A0 sec=A0 24.9 MBytes=A0=A0 209 Mbits/sec > [=A0 5]=A0 14.00-15.00=A0 sec=A0 21.4 MBytes=A0=A0 179 Mbits/sec > [=A0 5]=A0 15.00-16.00=A0 sec=A0 32.0 MBytes=A0=A0 268 Mbits/sec > [=A0 5]=A0 16.00-17.00=A0 sec=A0 10.9 MBytes=A0 91.2 Mbits/sec > [=A0 5]=A0 17.00-18.00=A0 sec=A0 27.6 MBytes=A0=A0 232 Mbits/sec > [=A0 5]=A0 18.00-19.00=A0 sec=A0 75.6 MBytes=A0=A0 634 Mbits/sec > [=A0 5]=A0 19.00-20.00=A0 sec=A0 21.1 MBytes=A0=A0 177 Mbits/sec > [=A0 5]=A0 20.00-21.00=A0 sec=A0=A0 109 MBytes=A0=A0 912 Mbits/sec > [=A0 5]=A0 21.00-22.00=A0 sec=A0 23.9 MBytes=A0=A0 200 Mbits/sec > [=A0 5]=A0 22.00-23.00=A0 sec=A0 62.6 MBytes=A0=A0 525 Mbits/sec > [=A0 5]=A0 23.00-24.00=A0 sec=A0 34.5 MBytes=A0=A0 289 Mbits/sec > [=A0 5]=A0 24.00-25.00=A0 sec=A0 54.4 MBytes=A0=A0 456 Mbits/sec > [=A0 5]=A0 25.00-26.00=A0 sec=A0 14.1 MBytes=A0=A0 118 Mbits/sec > [=A0 5]=A0 26.00-27.00=A0 sec=A0 38.8 MBytes=A0=A0 325 Mbits/sec > [=A0 5]=A0 27.00-28.00=A0 sec=A0 95.0 MBytes=A0=A0 797 Mbits/sec > [=A0 5]=A0 28.00-29.00=A0 sec=A0 23.2 MBytes=A0=A0 195 Mbits/sec > [=A0 5]=A0 29.00-30.00=A0 sec=A0 71.9 MBytes=A0=A0 603 Mbits/sec > [=A0 5]=A0 30.00-31.00=A0 sec=A0 28.8 MBytes=A0=A0 241 Mbits/sec > [=A0 5]=A0 31.00-32.00=A0 sec=A0 34.8 MBytes=A0=A0 292 Mbits/sec > [=A0 5]=A0 32.00-33.00=A0 sec=A0 19.4 MBytes=A0=A0 163 Mbits/sec > [=A0 5]=A0 33.00-34.00=A0 sec=A0 39.1 MBytes=A0=A0 328 Mbits/sec > [=A0 5]=A0 34.00-35.00=A0 sec=A0 31.4 MBytes=A0=A0 263 Mbits/sec > [=A0 5]=A0 35.00-36.00=A0 sec=A0 28.2 MBytes=A0=A0 237 Mbits/sec > [=A0 5]=A0 36.00-37.00=A0 sec=A0 48.5 MBytes=A0=A0 407 Mbits/sec > [=A0 5]=A0 37.00-38.00=A0 sec=A0 23.4 MBytes=A0=A0 196 Mbits/sec > [=A0 5]=A0 38.00-39.00=A0 sec=A0 71.4 MBytes=A0=A0 599 Mbits/sec > [=A0 5]=A0 39.00-40.00=A0 sec=A0 41.4 MBytes=A0=A0 347 Mbits/sec > [=A0 5]=A0 40.00-41.00=A0 sec=A0 15.2 MBytes=A0=A0 128 Mbits/sec > [=A0 5]=A0 41.00-42.00=A0 sec=A0 34.1 MBytes=A0=A0 286 Mbits/sec > [=A0 5]=A0 42.00-43.00=A0 sec=A0 31.1 MBytes=A0=A0 261 Mbits/sec > [=A0 5]=A0 43.00-44.00=A0 sec=A0 61.5 MBytes=A0=A0 516 Mbits/sec > [=A0 5]=A0 44.00-45.00=A0 sec=A0 38.2 MBytes=A0=A0 321 Mbits/sec > [=A0 5]=A0 45.00-46.00=A0 sec=A0 34.2 MBytes=A0=A0 287 Mbits/sec > [=A0 5]=A0 46.00-47.00=A0 sec=A0 33.8 MBytes=A0=A0 283 Mbits/sec > [=A0 5]=A0 47.00-48.00=A0 sec=A0 7.12 MBytes=A0 59.8 Mbits/sec > [=A0 5]=A0 48.00-49.00=A0 sec=A0 97.4 MBytes=A0=A0 817 Mbits/sec > [=A0 5]=A0 49.00-50.00=A0 sec=A0 13.6 MBytes=A0=A0 114 Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 Transfer=A0=A0=A0=A0 Bitrate > [=A0 5]=A0=A0 0.00-50.00=A0 sec=A0 55.7 GBytes=A0 9.57 Gbits/sec=A0=A0=A0= =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 > receiver > ----------------------------------------------------------- > Server listening on 5201 (test #3) > ----------------------------------------------------------- >=20 --=20 David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson --HGPJ0PKGvKYhrM+1 Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmb0v/UACgkQzQJF27ox 2GfhpQ//T+aNBJDYJ9Aji58P8bs36sYqENZPBD95U9OO1gifthENOK4S34STcM6M H73CHQGZK6M+m+zoM2Po+Afp63KIJ8f7Fv79gTMEmfG7VPje0and+vRamsmYcqI1 XwF8L9ONSljKEGJAc0c3GRJjmELCefu+mkJE36PrZh/q2NujY6B4hF2K2NUZ2ONu 5PwHGfF1Omzd9cV8rBkagBTDgQEzyL1+ZhHQG1ey2nHOBBjEFuQeQJR7KBpS94Bl F+wyBvs58vMFEuehW6/bMUwCd41A3K5GRcnSkcb5YiFcim2S4NG0OLB/Xvgjk/xG 7YiTa5e7w5whwcDCXLxSPhfJDs4jx0h7JQBnN6RdLLI8nmAOYtqOZisL4RD5+XCi r7TAOlW18eIDfG6SXysugjo01wteEGDshirafOlgoYKQ2lQ9FIY6GYSaccXGaYA3 dqEBpe4K5j+oGejdyVKwUCGOavxKQwQCVqYlsbbAVd8Q4IDM6QlgDwRn0awPILd6 pANiDiviamSvLnDKie8VTVs/edg8Lj2wON3fBP70kmRzLHW4PPJAceiMJFU+NBEK tWRz3JmQyLuqT4SW5iMkU2wXA3OQif053HpfkbTWg+iKTWjWCFYzlbgzRlNIm0nP I3DNGEprrjvaNhcQl/Xjp21uJyV4I7/2HvK/FZuUWeEj9EjhBPI= =GKDR -----END PGP SIGNATURE----- --HGPJ0PKGvKYhrM+1--