From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=Llhsi+yP; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 840DD5A0274 for ; Mon, 20 Jan 2025 18:28:53 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1737394132; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/K1ADfFFqnVqvlozMJRBlLGD/aPF7oeTDNog1894XsI=; b=Llhsi+yP11u4n1oQF/NHl8cvrdwiii78gz+D+O0Rovgqyzl697RSbBP2P7pSr5DVEqxyQY Lmh7BkmOlWquf+JC/H69rF3X74ovNoW6bMr26NDD+93qdWHQqIMFKj3Fn82nw74p7T/yj2 U6gkWdpzpAvxFkPTxeylSONBupsvFG8= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-178-cUiwmIK3NruMvt_NShCkpA-1; Mon, 20 Jan 2025 12:28:50 -0500 X-MC-Unique: cUiwmIK3NruMvt_NShCkpA-1 X-Mimecast-MFC-AGG-ID: cUiwmIK3NruMvt_NShCkpA Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-436379713baso23080545e9.2 for ; Mon, 20 Jan 2025 09:28:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737394128; x=1737998928; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=/K1ADfFFqnVqvlozMJRBlLGD/aPF7oeTDNog1894XsI=; b=hCy8AsTGR28lp2MIj+pp6VGyf90LKowIUZzZUdccCqBRHC7s/IJroXytT3af/6xGdy I73LiHVC16YQeGamvTXaHEYQakrIKy+6L6d69Sg3k+/IDpU3OkOjKZsqaxMK2foRbgv2 4RxNroKyz3WLgMNbrs8yhx1ZZAh8fnNL7E+6FlAaR5ajN2S2CiIVSDxH+tkVpSlONLCd quO8Nq4zymG1B/0qypG2Lh1UaeEyQBvAR8+Abq+/PHyBjg6CXLA4thlAMLX6W5vfhALM 0+wHUVC/RCKi6CUw1v3uvsCnqSCxL1sqm4HQPbFVAHSONxeyj1ky5vdWf+ewQkYmD1sd Farg== X-Gm-Message-State: AOJu0Yw0eFE21n4Pm0KNVXhGgmdFs+KCnKDdSzfeM61Vc0aoxMQmejfB sGaAQdV1xh1EEah8gxhgmxANUtRqXYaXB5L0fKNzy3CY1FJgbaYYgeG9UOhl6MYLh1P0CsF2Kja y4PfaQnrKISQ7JAgzxNdU5rEneq1RkHOS5iY5tYuPjI9f2KtuoZSBXuNyIg== X-Gm-Gg: ASbGncu85lR5z/qPW5YO5jmkQFjUQLNLW5XU3/NZ93xHX9fK0bfSUmpizFd9XLhTsn9 Q2IsBzBG/7Slm7t9kr3gfiMzPEpW83+YwSlTzidP4WhiUgA4pU4i2Kc7LKR/brMc542Z5D1VOjF 9uxGHF0xhgK5Bl3274IcbOcm/1uhgyCjDX6WcySky78Fv/Sg7y17M+zfMVHjUfPwTuaIrdjkXC+ A4YTwAFmAqJtAHTK1bOO6JO/n7QjP5iXDMGFcNpd2Ebs+lA7uZVl1+uRgU/LuETMjL6JML3gL1k yCbwIA== X-Received: by 2002:a05:600c:4f43:b0:434:a1d3:a326 with SMTP id 5b1f17b1804b1-438913bf07dmr126469145e9.6.1737394128281; Mon, 20 Jan 2025 09:28:48 -0800 (PST) X-Google-Smtp-Source: AGHT+IFytCNfl5nmQlu1XXmKrvYwh91fyRXLF7s62JET122niGGQcJVQaBEfke+URMrQ+tYuOv65eg== X-Received: by 2002:a05:600c:4f43:b0:434:a1d3:a326 with SMTP id 5b1f17b1804b1-438913bf07dmr126468925e9.6.1737394127862; Mon, 20 Jan 2025 09:28:47 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-437c16c4c12sm107350165e9.3.2025.01.20.09.28.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 20 Jan 2025 09:28:47 -0800 (PST) Date: Mon, 20 Jan 2025 18:28:45 +0100 From: Stefano Brivio To: David Gibson Subject: Re: [PATCH] tcp: Disable Nagle's algorithm (set TCP_NODELAY) on all sockets Message-ID: <20250120182845.734918ae@elisabeth> In-Reply-To: References: <20250117093405.1253554-1-sbrivio@redhat.com> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: pztlftC93HoOgrI-U_KvM7_IUQvSV_kqTSqwuFywvKg_1737394129 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: E25F2DEVEBETIAD5JP43Q3BULR7LF3CG X-Message-ID-Hash: E25F2DEVEBETIAD5JP43Q3BULR7LF3CG X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Mon, 20 Jan 2025 19:38:53 +1030 David Gibson wrote: > On Fri, Jan 17, 2025 at 10:34:05AM +0100, Stefano Brivio wrote: > > Following up on 725acd111ba3 ("tcp_splice: Set (again) TCP_NODELAY on > > both sides"), David argues that, in general, we don't know what kind > > of TCP traffic we're dealing with, on any side or path. > > > > TCP segments might have been delivered to our socket with a PSH flag, > > but we don't have a way to know about it. > > > > Similarly, the guest might send us segments with PSH or URG set, but > > we don't know if we should generally TCP_CORK sockets and uncork on > > those flags, because that would assume they're running a Linux kernel > > (and a particular version of it) matching the kernel that delivers > > outbound packets for us. > > > > Given that we can't make any assumption and everything might very well > > be interactive traffic, disable Nagle's algorithm on all non-spliced > > sockets as well. > > > > After all, John Nagle himself is nowadays recommending that delayed > > ACKs should never be enabled together with his algorithm, but we > > don't have a practical way to ensure that our environment is free from > > delayed ACKs (TCP_QUICKACK is not really usable for this purpose): > > > > https://news.ycombinator.com/item?id=34180239 > > > > Suggested-by: David Gibson > > Signed-off-by: Stefano Brivio > > --- > > tcp.c | 20 ++++++++++++++++++++ > > 1 file changed, 20 insertions(+) > > > > diff --git a/tcp.c b/tcp.c > > index 3b3193a..c570f42 100644 > > --- a/tcp.c > > +++ b/tcp.c > > @@ -756,6 +756,19 @@ static void tcp_sock_set_bufsize(const struct ctx *c, int s) > > trace("TCP: failed to set SO_SNDBUF to %i", v); > > } > > > > +/** > > + * tcp_sock_set_nodelay() - Set TCP_NODELAY option (disable Nagle's algorithm) > > + * @s: Socket, can be -1 to avoid check in the caller > > + */ > > +static void tcp_sock_set_nodelay(int s) > > +{ > > + if (s == -1) > > + return; > > + > > + if (setsockopt(s, SOL_TCP, TCP_NODELAY, &((int){ 1 }), sizeof(int))) > > + trace("TCP: failed to set TCP_NODELAY on socket %i", s); > > I think this deserves something a little stronger than trace(). It > might have a significant impact on latency, and if we get a failure > enabling a feature that's been in TCP longer than Linux has existed, > something pretty weird is going on. Hm, right, changed to debug(). For more than that, we'd need to have a print-once option. > > +} > > + > > /** > > * tcp_update_csum() - Calculate TCP checksum > > * @psum: Unfolded partial checksum of the IPv4 or IPv6 pseudo-header > > @@ -1285,6 +1298,7 @@ static int tcp_conn_new_sock(const struct ctx *c, sa_family_t af) > > return -errno; > > > > tcp_sock_set_bufsize(c, s); > > + tcp_sock_set_nodelay(s); > > > > return s; > > } > > @@ -2261,6 +2275,8 @@ static int tcp_sock_init_one(const struct ctx *c, const union inany_addr *addr, > > return s; > > > > tcp_sock_set_bufsize(c, s); > > + tcp_sock_set_nodelay(s); > > + > > This is a listening socket, not an accepted or connecting socket. > Does NODELAY even mean anything there? > > > return s; > > } > > > > @@ -2322,6 +2338,8 @@ static void tcp_ns_sock_init4(const struct ctx *c, in_port_t port) > > else > > s = -1; > > > > + tcp_sock_set_nodelay(s); > > Same here. > > > if (c->tcp.fwd_out.mode == FWD_AUTO) > > tcp_sock_ns[port][V4] = s; > > } > > @@ -2348,6 +2366,8 @@ static void tcp_ns_sock_init6(const struct ctx *c, in_port_t port) > > else > > s = -1; > > > > + tcp_sock_set_nodelay(s); > > And here. > > > if (c->tcp.fwd_out.mode == FWD_AUTO) > > tcp_sock_ns[port][V6] = s; > > } > > Do accept()ed sockets inherit NODELAY from the listening socket? Or > should we instead be setting NODELAY after the accept4() in > tcp_listen_handler()? In fact.. even if they do inhereit, would it be > simpler to just set it at accept() time? Oops. I just followed the example of tcp_sock_set_bufsize(), but it turns out that neither buffer sizes nor TCP_NODELAY is inherited from listening sockets! Sending a patch for that, and a v2 of this one. -- Stefano