From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: passt.top; dkim=pass (2048-bit key; unprotected) header.d=gmail.com header.i=@gmail.com header.a=rsa-sha256 header.s=20230601 header.b=ZqkTkpwk; dkim-atps=neutral Received: from mail-io1-xd2a.google.com (mail-io1-xd2a.google.com [IPv6:2607:f8b0:4864:20::d2a]) by passt.top (Postfix) with ESMTPS id 283605A061E for ; Tue, 28 Jan 2025 01:52:38 +0100 (CET) Received: by mail-io1-xd2a.google.com with SMTP id ca18e2360f4ac-844e39439abso130214739f.1 for ; Mon, 27 Jan 2025 16:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1738025557; x=1738630357; darn=passt.top; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=9tEeDmUjY4CieUAYofp5Zv8e/F4uQypMegDgyNLo0SA=; b=ZqkTkpwkVMQeT6qthLdMggx6bKY2UXrDxDYwXjv128tPLfg9TNXxt+tni9c+4N1e/b mWKVCvTCImI5cxf3UydHrKBoEnGFi+LfmApO1uETwVGTSAhAXKPxsCVtrPY9v5RvuI8B FhBbnGysJ4ci6ydIFJ9SP8tEfnCxmN8n+v+tfZo95JkQqZ7NiQsXs4JyY+Fkabbj66Rj oWiG/SEOwQ25YZBluZ3x7t/TfvC55DnGtjbnY/WOkLIjqoWIU6u/WwKxx/Bjg0NYiHfA aD108cKOlHA8u2bTZCkwfuOH8nXmySiINU/8TgUNhx1Jr5mivfAnpF+YRldC1wIA6qaR 7fGw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738025557; x=1738630357; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9tEeDmUjY4CieUAYofp5Zv8e/F4uQypMegDgyNLo0SA=; b=UIdu3K0p3OfOoNC79z6S6UXlxy2Cd/G1Yv0Qq+EfzJRhrypYg716NQ9+O78DLXnNFG WF15WW471T/gkaZ9X3zG69gqay4SHdbq/Xt9OfSInriaIp/8Jo7GFBt7GGTJxldojqKH fRlY0D2hLVJV2AEzNdL0OcoiQbtOrGdaarINbaZzO0kgjUEAVCbxfGdsn2EDvzGjjxKd 1pkqmd8mO2DT+iOqRpU9Ucwh7nLPJgxte0UolAD/E6PLFhCD4erQ8dH53zH8MFqHO2BI vh3lmf3u5cIdGvROzmUkZVOHgyZAYqvCMFgZeY5OIfk24FFT/fi7rSxG/823qafr8ccP x1eg== X-Forwarded-Encrypted: i=1; AJvYcCVJ06i4+eMNp16MwSvgzUDCbiMj0xUMHx7DmIwqxmFByejU+d/DNOZIJ6r6115omxSdiW820cYQuXA=@passt.top X-Gm-Message-State: AOJu0Yxzv7rj1MprZ1X3jOhuB/gltdn+5fRVnc/8xpAmXK9IskgpSBtk qd+6vUBcIgPdIuiJPb2yc6Uzt/0OZZzmg8rpkq7jKn6uxszIgGfV2efloMDbwXDvfRWjTf6jRor UN/zvtR22+bwkjoglljgJGXjNims= X-Gm-Gg: ASbGncv8guZj4mORIw9yqJQM56TmaIc1DpX2khUK5fpUT9b2KozsEEe5pV1LRASPFug nsiCl0DZgzicktsdcWMyO2eIiZ5F1vjseaxf8CJKqODH+Vj6RHPXTELS31AVs X-Google-Smtp-Source: AGHT+IGpJF52xzqROd7a+Z92UeGluIpLpNSHadhR7IvFddR48a66AR6n5IdpSajnwuWt+VVmH9euNAd1uh/RAHuXJoE= X-Received: by 2002:a05:6e02:3f90:b0:3cf:c85c:4f60 with SMTP id e9e14a558f8ab-3cfc85c503fmr121696475ab.11.1738025556763; Mon, 27 Jan 2025 16:52:36 -0800 (PST) MIME-Version: 1.0 References: <20250127231304.1465565-1-jmaloy@redhat.com> In-Reply-To: <20250127231304.1465565-1-jmaloy@redhat.com> From: Jason Xing Date: Tue, 28 Jan 2025 08:52:00 +0800 X-Gm-Features: AWEUYZkCN2V3SODRx6bgIpH56rlewnK2p1DuLiILfZgMJg4_Lu8CCfFAxlXr2dA Message-ID: Subject: Re: [net,v3] tcp: correct handling of extreme memory squeeze To: jmaloy@redhat.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-MailFrom: kerneljasonxing@gmail.com X-Mailman-Rule-Hits: nonmember-moderation X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation Message-ID-Hash: UCDG7ASO5KRPNJLLNFEXKXHDFTZSFWGS X-Message-ID-Hash: UCDG7ASO5KRPNJLLNFEXKXHDFTZSFWGS X-Mailman-Approved-At: Tue, 28 Jan 2025 07:52:28 +0100 CC: netdev@vger.kernel.org, davem@davemloft.net, kuba@kernel.org, passt-dev@passt.top, sbrivio@redhat.com, lvivier@redhat.com, dgibson@redhat.com, memnglong8.dong@gmail.com, ncardwell@google.com, eric.dumazet@gmail.com, edumazet@google.com X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Tue, Jan 28, 2025 at 7:13=E2=80=AFAM wrote: > > From: Jon Maloy > > Testing with iperf3 using the "pasta" protocol splicer has revealed > a bug in the way tcp handles window advertising in extreme memory > squeeze situations. > > Under memory pressure, a socket endpoint may temporarily advertise > a zero-sized window, but this is not stored as part of the socket data. > The reasoning behind this is that it is considered a temporary setting > which shouldn't influence any further calculations. > > However, if we happen to stall at an unfortunate value of the current > window size, the algorithm selecting a new value will consistently fail > to advertise a non-zero window once we have freed up enough memory. > This means that this side's notion of the current window size is > different from the one last advertised to the peer, causing the latter > to not send any data to resolve the sitution. > > The problem occurs on the iperf3 server side, and the socket in question > is a completely regular socket with the default settings for the > fedora40 kernel. We do not use SO_PEEK or SO_RCVBUF on the socket. > > The following excerpt of a logging session, with own comments added, > shows more in detail what is happening: > > // tcp_v4_rcv(->) > // tcp_rcv_established(->) > [5201<->39222]: =3D=3D=3D=3D Activating log @ net/ipv4/tcp_input.c/tc= p_data_queue()/5257 =3D=3D=3D=3D > [5201<->39222]: tcp_data_queue(->) > [5201<->39222]: DROPPING skb [265600160..265665640], reason: SKB_D= ROP_REASON_PROTO_MEM > [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469= 200, win_now 131184] > [copied_seq 259909392->260034360 (124968), unread = 5565800, qlen 85, ofoq 0] > [OFO queue: gap: 65480, len: 0] > [5201<->39222]: tcp_data_queue(<-) > [5201<->39222]: __tcp_transmit_skb(->) > [tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp-= >rcv_nxt 265600160] > [5201<->39222]: tcp_select_window(->) > [5201<->39222]: (inet_csk(sk)->icsk_ack.pending & ICSK_ACK_NOMEM)= ? --> TRUE > [tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp-= >rcv_nxt 265600160] > returning 0 > [5201<->39222]: tcp_select_window(<-) > [5201<->39222]: ADVERTISING WIN 0, ACK_SEQ: 265600160 > [5201<->39222]: [__tcp_transmit_skb(<-) > [5201<->39222]: tcp_rcv_established(<-) > [5201<->39222]: tcp_v4_rcv(<-) > > // Receive queue is at 85 buffers and we are out of memory. > // We drop the incoming buffer, although it is in sequence, and decide > // to send an advertisement with a window of zero. > // We don't update tp->rcv_wnd and tp->rcv_wup accordingly, which means > // we unconditionally shrink the window. > > [5201<->39222]: tcp_recvmsg_locked(->) > [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_= wnd: 262144, tp->rcv_nxt 265600160 > [5201<->39222]: [new_win =3D 0, win_now =3D 131184, 2 * win_now =3D 2= 62368] > [5201<->39222]: [new_win >=3D (2 * win_now) ? --> time_to_ack =3D 0] > [5201<->39222]: NOT calling tcp_send_ack() > [tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv= _nxt 265600160] > [5201<->39222]: __tcp_cleanup_rbuf(<-) > [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, = win_now 131184] > [copied_seq 260040464->260040464 (0), unread 5559696, q= len 85, ofoq 0] > returning 6104 bytes > [5201<->39222]: tcp_recvmsg_locked(<-) > > // After each read, the algorithm for calculating the new receive > // window in __tcp_cleanup_rbuf() finds it is too small to advertise > // or to update tp->rcv_wnd. > // Meanwhile, the peer thinks the window is zero, and will not send > // any more data to trigger an update from the interrupt mode side. > > [5201<->39222]: tcp_recvmsg_locked(->) > [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_= wnd: 262144, tp->rcv_nxt 265600160 > [5201<->39222]: [new_win =3D 262144, win_now =3D 131184, 2 * win_now = =3D 262368] > [5201<->39222]: [new_win >=3D (2 * win_now) ? --> time_to_ack =3D 0] > [5201<->39222]: NOT calling tcp_send_ack() > [tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv= _nxt 265600160] > [5201<->39222]: __tcp_cleanup_rbuf(<-) > [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, = win_now 131184] > [copied_seq 260099840->260171536 (71696), unread 542862= 4, qlen 83, ofoq 0] > returning 131072 bytes > [5201<->39222]: tcp_recvmsg_locked(<-) > > // The above pattern repeats again and again, since nothing changes > // between the reads. > > [...] > > [5201<->39222]: tcp_recvmsg_locked(->) > [5201<->39222]: __tcp_cleanup_rbuf(->) tp->rcv_wup: 265469200, tp->rcv_= wnd: 262144, tp->rcv_nxt 265600160 > [5201<->39222]: [new_win =3D 262144, win_now =3D 131184, 2 * win_now = =3D 262368] > [5201<->39222]: [new_win >=3D (2 * win_now) ? --> time_to_ack =3D 0] > [5201<->39222]: NOT calling tcp_send_ack() > [tp->rcv_wup: 265469200, tp->rcv_wnd: 262144, tp->rcv= _nxt 265600160] > [5201<->39222]: __tcp_cleanup_rbuf(<-) > [rcv_nxt 265600160, rcv_wnd 262144, snt_ack 265469200, = win_now 131184] > [copied_seq 265600160->265600160 (0), unread 0, qlen 0,= ofoq 0] > returning 54672 bytes > [5201<->39222]: tcp_recvmsg_locked(<-) > > // The receive queue is empty, but no new advertisement has been sent. > // The peer still thinks the receive window is zero, and sends nothing. > // We have ended up in a deadlock situation. > > Furthermore, we have observed that in these situations this side may > send out an updated 'th->ack_seq=C2=B4 which is not stored in tp->rcv_wup > as it should be. Backing ack_seq seems to be harmless, but is of > course still wrong from a protocol viewpoint. > > We fix this by updating the socket state correctly when a packet has > been dropped because of memory exhaustion and we have to advertize > a zero window. > > Further testing shows that the connection recovers neatly from the > squeeze situation, and traffic can continue indefinitely. > > Fixes: e2142825c120 ("net: tcp: send zero-window ACK when no memory") > Cc: Menglong Dong > Reviewed-by: Stefano Brivio > Signed-off-by: Jon Maloy I reckon that adding more description about why this case can be triggered (the reason behind this case is the other side using other kernels doesn't periodically send a window probe) is really necessary. Reviewed-by: Jason Xing Thanks, Jason