From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail.ozlabs.org (gandalf.ozlabs.org [150.107.74.76]) by passt.top (Postfix) with ESMTPS id 8B2675A02C3 for ; Mon, 13 May 2024 04:23:30 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gibson.dropbear.id.au; s=202312; t=1715567007; bh=dHX0FMgKWTNkHHpji0nQsP4icB7ni9jMrtbBoX5hmh4=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=OeSaQVWlk2gRRxU9EHbVbsPm/TymAuwbIejhwfx1eVnM8j2YWG6lEVhLTFnt/75In gg1ADoKiTY2OdEkcovEuplugiRHnxAfwNCwYlQTt5qqC4aTjBYfbE170pKOPkizV5K 3xzKlIHkxQEaKRTcPCRhBRENpWsLiSV3jW+2TPxZmnNv4iLojMiluKKziD15nwmF6Q Sy2bZe9jbl6mLo8KFhePdpjZfHwpfzJ47ailzXceq5F25xI1rs2usfSdHGUL81Zxay s9/LJEKxoyck1ivLfYoT1qZQCBosqQFRdRKzAOj7syUzWYshi/vKVPfyasrNqTFbsm tb7B7sXDQ8cDA== Received: by gandalf.ozlabs.org (Postfix, from userid 1007) id 4Vd3FC3gP8z4wcR; Mon, 13 May 2024 12:23:27 +1000 (AEST) Date: Mon, 13 May 2024 12:09:57 +1000 From: David Gibson To: Jon Maloy Subject: Re: [PATCH v3 1/3] tcp: move seq_to_tap update to when frame is queued Message-ID: References: <20240511152008.421750-1-jmaloy@redhat.com> <20240511152008.421750-2-jmaloy@redhat.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="Y7wei7SUKhihmAys" Content-Disposition: inline In-Reply-To: <20240511152008.421750-2-jmaloy@redhat.com> Message-ID-Hash: KTPA6EW4WNP3PCSQE3LYKSU5CDNW4SH4 X-Message-ID-Hash: KTPA6EW4WNP3PCSQE3LYKSU5CDNW4SH4 X-MailFrom: dgibson@gandalf.ozlabs.org X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, sbrivio@redhat.com, lvivier@redhat.com, dgibson@redhat.com X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: --Y7wei7SUKhihmAys Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, May 11, 2024 at 11:20:06AM -0400, Jon Maloy wrote: > commit a469fc393fa1 ("tcp, tap: Don't increase tap-side sequence counter = for dropped frames") > delayed update of conn->seq_to_tap until the moment the corresponding > frame has been successfully pushed out. This has the advantage that we > immediately can make a new attempt to transmit a frame after a failed > trasnmit, rather than waiting for the peer to later discover a gap and > trigger the fast retransmit mechanism to solve the problem. >=20 > This approach has turned out to cause a problem with spurious sequence > number updates during peer-initiated retransmits, and we have realized > it may not be the best way to solve the above issue. >=20 > We now restore the previous method, by updating the said field at the > moment a frame is added to the outqueue. To retain the advantage of > having a quick re-attempt based on local failure detection, we now scan > through the part of the outqueue that had do be dropped, and restore the > sequence counter for each affected connection to the most appropriate > value. >=20 > Signed-off-by: Jon Maloy >=20 > --- > v2: - Re-spun loop in tcp_revert_seq() and some other changes based on > feedback from Stefano Brivio. > - Added paranoid test to avoid that seq_to_tap becomes lower than > seq_ack_from_tap. > --- > tcp.c | 63 ++++++++++++++++++++++++++++++++++++++++++----------------- > 1 file changed, 45 insertions(+), 18 deletions(-) >=20 > diff --git a/tcp.c b/tcp.c > index 21d0af0..21cbfba 100644 > --- a/tcp.c > +++ b/tcp.c > @@ -411,13 +411,14 @@ static int tcp_sock_ns [NUM_PORTS][IP_VERSIONS]; > static union inany_addr low_rtt_dst[LOW_RTT_TABLE_SIZE]; > =20 > /** > - * tcp_buf_seq_update - Sequences to update with length of frames once s= ent > - * @seq: Pointer to sequence number sent to tap-side, to be updated > - * @len: TCP payload length > + * tcp_frame_ref - References needed by queued frames in case we need > + * to revert corresponding connection sequence numbers > + * @conn: Pointer to connection for this frame > + * @seq: Sequence number of the corresponding frame > */ > -struct tcp_buf_seq_update { > - uint32_t *seq; > - uint16_t len; > +struct tcp_frame_ref { > + struct tcp_tap_conn *conn; > + uint32_t seq; As noted in another mail, I think we could get the sequence number =66rom the actual frame buffer at revert time. That could be a follow up improvement, though. > }; > =20 > /* Static buffers */ > @@ -461,7 +462,7 @@ static struct tcp_payload_t tcp4_payload[TCP_FRAMES_M= EM]; > =20 > static_assert(MSS4 <=3D sizeof(tcp4_payload[0].data), "MSS4 is greater t= han 65516"); > =20 > -static struct tcp_buf_seq_update tcp4_seq_update[TCP_FRAMES_MEM]; > +static struct tcp_frame_ref tcp4_frame_ref[TCP_FRAMES_MEM]; > static unsigned int tcp4_payload_used; > =20 > static struct tap_hdr tcp4_flags_tap_hdr[TCP_FRAMES_MEM]; > @@ -483,7 +484,7 @@ static struct tcp_payload_t tcp6_payload[TCP_FRAMES_M= EM]; > =20 > static_assert(MSS6 <=3D sizeof(tcp6_payload[0].data), "MSS6 is greater t= han 65516"); > =20 > -static struct tcp_buf_seq_update tcp6_seq_update[TCP_FRAMES_MEM]; > +static struct tcp_frame_ref tcp6_frame_ref[TCP_FRAMES_MEM]; > static unsigned int tcp6_payload_used; > =20 > static struct tap_hdr tcp6_flags_tap_hdr[TCP_FRAMES_MEM]; > @@ -1261,25 +1262,50 @@ static void tcp_flags_flush(const struct ctx *c) > tcp4_flags_used =3D 0; > } > =20 > +/** > + * tcp_revert_seq() - Revert affected conn->seq_to_tap after failed tran= smission > + * @frames_ref: Array with connection and sequence number data > + * @first: Index of entry corresponding to first dropped frame > + * @last: Index of entry corresponding to last dropped frame > + */ > +static void tcp_revert_seq(struct tcp_frame_ref *frame_ref, int first, i= nt last) > +{ > + struct tcp_tap_conn *conn; > + int i; > + > + for (i =3D first; i <=3D last; i++) { > + conn =3D frame_ref[i].conn; 'conn' could be local to the loop body, and I think one of our static checkers is going to complain that it's not. > + > + if (SEQ_LE(conn->seq_to_tap, frame_ref[i].seq)) > + continue; > + > + conn->seq_to_tap =3D frame_ref[i].seq; > + > + if (SEQ_GE(conn->seq_to_tap, conn->seq_ack_from_tap)) > + continue; > + > + conn->seq_to_tap =3D conn->seq_ack_from_tap; > + } > +} > + > /** > * tcp_payload_flush() - Send out buffers for segments with data > * @c: Execution context > */ > static void tcp_payload_flush(const struct ctx *c) > { > - unsigned i; > size_t m; > =20 > m =3D tap_send_frames(c, &tcp6_l2_iov[0][0], TCP_NUM_IOVS, > tcp6_payload_used); > - for (i =3D 0; i < m; i++) > - *tcp6_seq_update[i].seq +=3D tcp6_seq_update[i].len; > + if (m !=3D tcp6_payload_used) > + tcp_revert_seq(tcp6_frame_ref, m, tcp6_payload_used - 1); > tcp6_payload_used =3D 0; > =20 > m =3D tap_send_frames(c, &tcp4_l2_iov[0][0], TCP_NUM_IOVS, > tcp4_payload_used); > - for (i =3D 0; i < m; i++) > - *tcp4_seq_update[i].seq +=3D tcp4_seq_update[i].len; > + if (m !=3D tcp4_payload_used) > + tcp_revert_seq(tcp4_frame_ref, m, tcp4_payload_used - 1); > tcp4_payload_used =3D 0; > } > =20 > @@ -2129,10 +2155,11 @@ static int tcp_sock_consume(const struct tcp_tap_= conn *conn, uint32_t ack_seq) > static void tcp_data_to_tap(const struct ctx *c, struct tcp_tap_conn *co= nn, > ssize_t dlen, int no_csum, uint32_t seq) > { > - uint32_t *seq_update =3D &conn->seq_to_tap; > struct iovec *iov; > size_t l4len; > =20 > + conn->seq_to_tap =3D seq + dlen; > + > if (CONN_V4(conn)) { > struct iovec *iov_prev =3D tcp4_l2_iov[tcp4_payload_used - 1]; > const uint16_t *check =3D NULL; > @@ -2142,8 +2169,8 @@ static void tcp_data_to_tap(const struct ctx *c, st= ruct tcp_tap_conn *conn, > check =3D &iph->check; > } > =20 > - tcp4_seq_update[tcp4_payload_used].seq =3D seq_update; > - tcp4_seq_update[tcp4_payload_used].len =3D dlen; > + tcp4_frame_ref[tcp4_payload_used].conn =3D conn; > + tcp4_frame_ref[tcp4_payload_used].seq =3D seq; > =20 > iov =3D tcp4_l2_iov[tcp4_payload_used++]; > l4len =3D tcp_l2_buf_fill_headers(c, conn, iov, dlen, check, seq); > @@ -2151,8 +2178,8 @@ static void tcp_data_to_tap(const struct ctx *c, st= ruct tcp_tap_conn *conn, > if (tcp4_payload_used > TCP_FRAMES_MEM - 1) > tcp_payload_flush(c); > } else if (CONN_V6(conn)) { > - tcp6_seq_update[tcp6_payload_used].seq =3D seq_update; > - tcp6_seq_update[tcp6_payload_used].len =3D dlen; > + tcp6_frame_ref[tcp6_payload_used].conn =3D conn; > + tcp6_frame_ref[tcp6_payload_used].seq =3D seq; > =20 > iov =3D tcp6_l2_iov[tcp6_payload_used++]; > l4len =3D tcp_l2_buf_fill_headers(c, conn, iov, dlen, NULL, seq); --=20 David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson --Y7wei7SUKhihmAys Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEEO+dNsU4E3yXUXRK2zQJF27ox2GcFAmZBdnQACgkQzQJF27ox 2GfVow//Rqs/MTH+2qc86zdHAmtCa8RZuheGvp+D8ia0n8K9v26gZsu6Jjv6X/60 wlrmKmSiRH/NawRpaZVRRWfyc4swgS6B+AgU9xQqEJnDA8pD3Sl+PJVAVjOv/Dfq qEEL22Gs3oGtk5W/2GmZK+BOXNrsmpekDKw+hERTv6LA6zziS2pGZif/tuEtOLnx rXWZofiMZEeeR/DZgvEP6HsPy1Boer0glSiXsFBBafGLFW4PFCrr+eMuNoOcWAuR ulGW8u38/jz/ZFaEzjeDS4mrwjO7cpqi69uxfhKuESat6sQx7s7EJQhAsjEFKbBj Jh5v11ewj2STu/cOy+Ho7wCmhncaXRqdRlCJPGTDDf7+CpWKEE+4kKfUW2lBxKku D3C889pTH1uTPgq/lCBbJ57hYNM6K+1xK1lpMY+0HYZr0kret3KBgDAmCmzpBuKd Hdr+CbsA6tzZx+eGYnPpPD4AyhvNEno4fD2FFoilz8RArreOUo2MNTIqj5Ec5iTZ d392YpoazWCPDOTpAs+4VQN9VXIGlPUkM8SeUgtbf2JzRBjFwXY1f55W3sN6yjDB ozRl2/qWL7petrk+wNXHD8Z78gZpRInxuMWSyaeduLCd3CTlTgKl9FJqHJ1PJhkY FoSZzjk/CqLt7Wrx62WkW+GE5dp9IB6NnybfSbLvsdcbY9esV+s= =nELR -----END PGP SIGNATURE----- --Y7wei7SUKhihmAys--