On Wed, Feb 14, 2024 at 10:34:50AM +1100, David Gibson wrote: > On Tue, Feb 13, 2024 at 04:49:01PM +0100, Eric Dumazet wrote: > > On Tue, Feb 13, 2024 at 4:28 PM Paolo Abeni wrote: > > > > > > On Tue, 2024-02-13 at 14:34 +0100, Eric Dumazet wrote: > > > > On Tue, Feb 13, 2024 at 2:02 PM Paolo Abeni wrote: > > > > > > > > > > On Tue, 2024-02-13 at 13:24 +0100, Eric Dumazet wrote: > > > > > > On Tue, Feb 13, 2024 at 11:49 AM Paolo Abeni wrote: > > > > > > > > > > > > > > @@ -2508,7 +2508,10 @@ static int tcp_recvmsg_locked(struct sock *sk, struct msghdr *msg, size_t len, > > > > > > > > WRITE_ONCE(*seq, *seq + used); > > > > > > > > copied += used; > > > > > > > > len -= used; > > > > > > > > - > > > > > > > > + if (flags & MSG_PEEK) > > > > > > > > + sk_peek_offset_fwd(sk, used); > > > > > > > > + else > > > > > > > > + sk_peek_offset_bwd(sk, used); > > > > > > > > > > > > Yet another cache miss in TCP fast path... > > > > > > > > > > > > We need to move sk_peek_off in a better location before we accept this patch. > > > > > > > > > > > > I always thought MSK_PEEK was very inefficient, I am surprised we > > > > > > allow arbitrary loops in recvmsg(). > > > > > > > > > > Let me double check I read the above correctly: are you concerned by > > > > > the 'skb_queue_walk(&sk->sk_receive_queue, skb) {' loop that could > > > > > touch a lot of skbs/cachelines before reaching the relevant skb? > > > > > > > > > > The end goal here is allowing an user-space application to read > > > > > incrementally/sequentially the received data while leaving them in > > > > > receive buffer. > > > > > > > > > > I don't see a better option than MSG_PEEK, am I missing something? > > > > > > > > > > > > This sk_peek_offset protocol, needing sk_peek_offset_bwd() in the non > > > > MSG_PEEK case is very strange IMO. > > > > > > > > Ideally, we should read/write over sk_peek_offset only when MSG_PEEK > > > > is used by the caller. > > > > > > > > That would only touch non fast paths. > > > > > > > > Since the API is mono-threaded anyway, the caller should not rely on > > > > the fact that normal recvmsg() call > > > > would 'consume' sk_peek_offset. > > > > > > Storing in sk_peek_seq the tcp next sequence number to be peeked should > > > avoid changes in the non MSG_PEEK cases. > > > > > > AFAICS that would need a new get_peek_off() sock_op and a bit somewhere > > > (in sk_flags?) to discriminate when sk_peek_seq is actually set. Would > > > that be acceptable? > > > > We could have a parallel SO_PEEK_OFFSET option, reusing the same socket field. > > > > The new semantic would be : Supported by TCP (so far), and tcp > > recvmsg() only reads/writes this field when MSG_PEEK is used. > > Applications would have to clear the values themselves. > > Those semantics would likely defeat the purpose of using SO_PEEK_OFF > for our use case, since we'd need an additional setsockopt() for every > non-PEEK recv() (which are all MSG_TRUNC in our case). Btw, Eric, If you're concerned about the extra access added to the "regular" TCP path, would you be happier with the original approach Jon proposed: that allowed a user to essentially supply an offset to an individial MSG_PEEK recvmsg() by inserting a dummy entry as msg_iov[0] with a NULL pointer and length to skip. It did the job for us, although I admit it's a little ugly, which I presume is why Paolo suggested we investigate SO_PEEK_OFF instead. I think the SO_PEEK_OFF approach is more elegant, but maybe the performance impact outweighs that. -- David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson