Re: [PATCH 0/4] Some improvements to the tap send path

[Laurent Vivier <lvivier@redhat.com>]

On 3/8/24 07:53, David Gibson wrote:
> This series has a handful of small improvements to the tap send path.
> See individual commit messages for the details.
> 
> I expect this will conflict with Laurent's upcoming work.  I hope the
> conflicts won't be too bad, and indeed will set us up for less
> duplication there in the end.

I'm working on patch that devides TCP buffers in several buffers pointed out by an IOV 
arrays and then provided to tap_send_frames(). I'm going to base my patch on this series.

The idea is:

A frame is made with 4 iovecs:

#define TCP_IOV_VNET    0
#define TCP_IOV_ETH     1
#define TCP_IOV_IP      2
#define TCP_IOV_PAYLOAD 3
#define TCP_IOV_NUM     4
typedef struct iovec tap_iovec_t[TCP_IOV_NUM];

The payload can be TCP + data or TCP + flags:

struct tcp_l2_flags_t {
         struct tcphdr th;
         char opts[OPT_MSS_LEN + OPT_WS_LEN + 1];
};

struct tcp_l2_payload_t {
         struct tcphdr th;       /*    20 bytes */
         uint8_t data[MSS];      /* 65516 bytes */
#ifdef __AVX2__
} __attribute__ ((packed, aligned(32)));
#else
} __attribute__ ((packed, aligned(__alignof__(unsigned int))));
#endif

static struct ethhdr            tcp4_eth_src;

static struct iphdr             tcp4_l2_ip[TCP_FRAMES_MEM];
static struct tcp_l2_payload_t  tcp4_l2_payload[TCP_FRAMES_MEM];

static struct tcp_buf_seq_update tcp4_l2_buf_seq_update[TCP_FRAMES_MEM];
static unsigned int tcp4_l2_buf_used;

static tap_iovec_t      tcp4_l2_iov             [TCP_FRAMES_MEM];

Initialization looks like:

	tcp4_eth_src.h_proto = htons_constant(ETH_P_IP);
	for (i = 0; i < TCP_FRAMES_MEM; i++) {
		...
                 /* headers */
                 tcp4_l2_ip[i] = iph;
                 tcp4_l2_payload[i].th = (struct tcphdr){
                                         .doff = sizeof(struct tcphdr) / 4,
                                         .ack = 1
                                 };
		...
		/* iovecs */
		iov = tcp4_l2_iov[i];
                 iov[TCP_IOV_ETH].iov_base = &tcp4_eth_src;
                 iov[TCP_IOV_ETH].iov_len = sizeof(struct ethhdr);
                 iov[TCP_IOV_IP].iov_base = &tcp4_l2_ip[i];
                 iov[TCP_IOV_IP].iov_len = sizeof(struct iphdr);
                 iov[TCP_IOV_PAYLOAD].iov_base = &tcp4_l2_payload[i];
		...
	}

Then to fill the payload header (data are received by tcp_data_from_sock()):

	iov = tcp4_l2_iov[tcp4_l2_buf_used++];
	iov[TCP_IOV_PAYLOAD].iov_len = tcp_l2_buf_fill_headers(c, conn,
                                                        iov, plen, check, seq);

And the frame is sent using:

	m = tap_send_iov(c, tcp4_l2_iov, tcp4_l2_buf_used);

For the moment (I'll rebase on your series), tap_send_iov_passt() looks like:

static size_t tap_send_iov_passt(const struct ctx *c, tap_iovec_t *tap_iov,
                                  size_t n)
{
         unsigned int i;

         for (i = 0; i < n; i++) {
                 uint32_t vnet_len;
                 int j;

                 vnet_len = 0;
                 for (j = TCP_IOV_ETH; j < TCP_IOV_NUM; j++)
                         vnet_len += tap_iov[i][j].iov_len;

                 tap_iov[i][TCP_IOV_VNET].iov_base = &vnet_len;
                 tap_iov[i][TCP_IOV_VNET].iov_len = sizeof(vnet_len);

                 if (!tap_send_frames_passt(c, tap_iov[i], TCP_IOV_NUM))
                         break;
         }

         return i;

}

Thanks,
Laurent
> 
> This is based on Laurent's patch fixing pcap_multiple() not to capture
> frames we failed to send.
> 
> David Gibson (4):
>    tap: Extend tap_send_frames() to allow multi-buffer frames
>    tap: Simplify some casts in the tap "slow path" functions
>    tap: Implement tap_send() "slow path" in terms of fast path
>    tap: Rename tap_iov_{base,len}
> 
>   arp.c |   4 +-
>   tap.c | 158 +++++++++++++++++++++++++++++++---------------------------
>   tap.h |  19 +++----
>   tcp.c |  20 ++++----
>   udp.c |  10 ++--
>   5 files changed, 111 insertions(+), 100 deletions(-)
>