From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=IBTQVaTW; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTP id BD0CC5A0276 for ; Fri, 06 Sep 2024 18:05:08 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1725638707; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=13aaMKgtXWhDcnYjumcd9z6OgTykpW7+ewviP3qdh28=; b=IBTQVaTWXLJeenDV91KhnlzywoNTor9zZqdfVZpf0x4tT3t9KFAVuxWtA3i9FX/FaSea5y XYXRWlbww9EeixkISL2qOiRDKPpvup5vveuzEn9yDxVm7uaTjoBEwb9sJNO99/06eYCfZx y/iOQYAH9RK8kFuKDpO9Z/aaBFwt+zU= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-543-1Laj4ywXNpuuKMnjIVCblw-1; Fri, 06 Sep 2024 12:05:06 -0400 X-MC-Unique: 1Laj4ywXNpuuKMnjIVCblw-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 2E7251953941 for ; Fri, 6 Sep 2024 16:05:05 +0000 (UTC) Received: from lenovo-t14s.redhat.com (unknown [10.39.193.22]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id D91771956086; Fri, 6 Sep 2024 16:05:03 +0000 (UTC) From: Laurent Vivier To: passt-dev@passt.top Subject: [PATCH v4 4/4] vhost-user: add vhost-user Date: Fri, 6 Sep 2024 18:04:49 +0200 Message-ID: <20240906160455.2088854-5-lvivier@redhat.com> In-Reply-To: <20240906160455.2088854-1-lvivier@redhat.com> References: <20240906160455.2088854-1-lvivier@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true Message-ID-Hash: JKKPYDJSVDDJ3W7MVXDQN3RTQNGURCVH X-Message-ID-Hash: JKKPYDJSVDDJ3W7MVXDQN3RTQNGURCVH X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: Laurent Vivier X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: add virtio and vhost-user functions to connect with QEMU. $ ./passt --vhost-user and # qemu-system-x86_64 ... -m 4G \ -object memory-backend-memfd,id=memfd0,share=on,size=4G \ -numa node,memdev=memfd0 \ -chardev socket,id=chr0,path=/tmp/passt_1.socket \ -netdev vhost-user,id=netdev0,chardev=chr0 \ -device virtio-net,mac=9a:2b:2c:2d:2e:2f,netdev=netdev0 \ ... Signed-off-by: Laurent Vivier --- Makefile | 6 +- checksum.c | 1 - conf.c | 23 +- epoll_type.h | 4 + isolation.c | 15 +- packet.c | 11 + packet.h | 8 +- passt.1 | 10 +- passt.c | 26 +- passt.h | 6 + pcap.c | 1 - tap.c | 111 +++++++-- tap.h | 5 +- tcp.c | 31 ++- tcp_buf.c | 8 +- tcp_internal.h | 3 +- tcp_vu.c | 656 +++++++++++++++++++++++++++++++++++++++++++++++++ tcp_vu.h | 12 + udp.c | 76 +++--- udp.h | 8 +- udp_internal.h | 34 +++ udp_vu.c | 386 +++++++++++++++++++++++++++++ udp_vu.h | 13 + vhost_user.c | 32 +-- virtio.c | 1 - vu_common.c | 35 +++ vu_common.h | 34 +++ 27 files changed, 1451 insertions(+), 105 deletions(-) create mode 100644 tcp_vu.c create mode 100644 tcp_vu.h create mode 100644 udp_internal.h create mode 100644 udp_vu.c create mode 100644 udp_vu.h create mode 100644 vu_common.c create mode 100644 vu_common.h diff --git a/Makefile b/Makefile index 01e95ac1b62c..e481a9430174 100644 --- a/Makefile +++ b/Makefile @@ -47,7 +47,8 @@ FLAGS += -DDUAL_STACK_SOCKETS=$(DUAL_STACK_SOCKETS) PASST_SRCS = arch.c arp.c checksum.c conf.c dhcp.c dhcpv6.c flow.c fwd.c \ icmp.c igmp.c inany.c iov.c ip.c isolation.c lineread.c log.c mld.c \ ndp.c netlink.c packet.c passt.c pasta.c pcap.c pif.c tap.c tcp.c \ - tcp_buf.c tcp_splice.c udp.c udp_flow.c util.c vhost_user.c virtio.c + tcp_buf.c tcp_splice.c tcp_vu.c udp.c udp_flow.c udp_vu.c util.c \ + vhost_user.c virtio.c vu_common.c QRAP_SRCS = qrap.c SRCS = $(PASST_SRCS) $(QRAP_SRCS) @@ -57,7 +58,8 @@ PASST_HEADERS = arch.h arp.h checksum.h conf.h dhcp.h dhcpv6.h flow.h fwd.h \ flow_table.h icmp.h icmp_flow.h inany.h iov.h ip.h isolation.h \ lineread.h log.h ndp.h netlink.h packet.h passt.h pasta.h pcap.h pif.h \ siphash.h tap.h tcp.h tcp_buf.h tcp_conn.h tcp_internal.h tcp_splice.h \ - udp.h udp_flow.h util.h vhost_user.h virtio.h + tcp_vu.h udp.h udp_flow.h udp_internal.h udp_vu.h util.h vhost_user.h \ + virtio.h vu_common.h HEADERS = $(PASST_HEADERS) seccomp.h C := \#include \nstruct tcp_info x = { .tcpi_snd_wnd = 0 }; diff --git a/checksum.c b/checksum.c index 006614fcbb28..aa5b7ae1cb66 100644 --- a/checksum.c +++ b/checksum.c @@ -501,7 +501,6 @@ uint16_t csum(const void *buf, size_t len, uint32_t init) * * Return: 16-bit folded, complemented checksum */ -/* cppcheck-suppress unusedFunction */ uint16_t csum_iov(const struct iovec *iov, size_t n, uint32_t init) { unsigned int i; diff --git a/conf.c b/conf.c index e29b6a9201e0..0633f82d231a 100644 --- a/conf.c +++ b/conf.c @@ -45,6 +45,7 @@ #include "lineread.h" #include "isolation.h" #include "log.h" +#include "vhost_user.h" /** * next_chunk - Return the next piece of a string delimited by a character @@ -759,9 +760,14 @@ static void usage(const char *name, FILE *f, int status) " default: same interface name as external one\n"); } else { fprintf(f, - " -s, --socket PATH UNIX domain socket path\n" + " -s, --socket, --socket-path PATH UNIX domain socket path\n" " default: probe free path starting from " UNIX_SOCK_PATH "\n", 1); + fprintf(f, + " --vhost-user Enable vhost-user mode\n" + " UNIX domain socket is provided by -s option\n" + " --print-capabilities print back-end capabilities in JSON format,\n" + " only meaningful for vhost-user mode\n"); } fprintf(f, @@ -1281,6 +1287,10 @@ void conf(struct ctx *c, int argc, char **argv) {"netns-only", no_argument, NULL, 20 }, {"map-host-loopback", required_argument, NULL, 21 }, {"map-guest-addr", required_argument, NULL, 22 }, + {"vhost-user", no_argument, NULL, 23 }, + /* vhost-user backend program convention */ + {"print-capabilities", no_argument, NULL, 24 }, + {"socket-path", required_argument, NULL, 's' }, { 0 }, }; const char *logname = (c->mode == MODE_PASTA) ? "pasta" : "passt"; @@ -1419,7 +1429,6 @@ void conf(struct ctx *c, int argc, char **argv) sizeof(c->ip6.ifname_out), "%s", optarg); if (ret <= 0 || ret >= (int)sizeof(c->ip6.ifname_out)) die("Invalid interface name: %s", optarg); - break; case 17: if (c->mode != MODE_PASTA) @@ -1458,6 +1467,16 @@ void conf(struct ctx *c, int argc, char **argv) conf_nat(optarg, &c->ip4.map_guest_addr, &c->ip6.map_guest_addr, NULL); break; + case 23: + if (c->mode == MODE_PASTA) { + err("--vhost-user is for passt mode only"); + usage(argv[0], stdout, EXIT_SUCCESS); + } + c->mode = MODE_VU; + break; + case 24: + vu_print_capabilities(); + break; case 'd': c->debug = 1; c->quiet = 0; diff --git a/epoll_type.h b/epoll_type.h index 0ad1efa0ccec..f3ef41584757 100644 --- a/epoll_type.h +++ b/epoll_type.h @@ -36,6 +36,10 @@ enum epoll_type { EPOLL_TYPE_TAP_PASST, /* socket listening for qemu socket connections */ EPOLL_TYPE_TAP_LISTEN, + /* vhost-user command socket */ + EPOLL_TYPE_VHOST_CMD, + /* vhost-user kick event socket */ + EPOLL_TYPE_VHOST_KICK, EPOLL_NUM_TYPES, }; diff --git a/isolation.c b/isolation.c index 45fba1e68b9d..c2a3c7b7911d 100644 --- a/isolation.c +++ b/isolation.c @@ -379,12 +379,19 @@ void isolate_postfork(const struct ctx *c) prctl(PR_SET_DUMPABLE, 0); - if (c->mode == MODE_PASTA) { - prog.len = (unsigned short)ARRAY_SIZE(filter_pasta); - prog.filter = filter_pasta; - } else { + switch (c->mode) { + case MODE_PASST: prog.len = (unsigned short)ARRAY_SIZE(filter_passt); prog.filter = filter_passt; + break; + case MODE_PASTA: + prog.len = (unsigned short)ARRAY_SIZE(filter_pasta); + prog.filter = filter_pasta; + break; + case MODE_VU: + prog.len = (unsigned short)ARRAY_SIZE(filter_vu); + prog.filter = filter_vu; + break; } if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) || diff --git a/packet.c b/packet.c index 37489961a37e..e5a78d079231 100644 --- a/packet.c +++ b/packet.c @@ -36,6 +36,17 @@ static int packet_check_range(const struct pool *p, size_t offset, size_t len, const char *start, const char *func, int line) { + if (p->buf_size == 0) { + int ret; + + ret = vu_packet_check_range((void *)p->buf, offset, len, start); + + if (ret == -1) + trace("cannot find region, %s:%i", func, line); + + return ret; + } + if (start < p->buf) { trace("packet start %p before buffer start %p, " "%s:%i", (void *)start, (void *)p->buf, func, line); diff --git a/packet.h b/packet.h index 8377dcf678bb..3f70e949c066 100644 --- a/packet.h +++ b/packet.h @@ -8,8 +8,10 @@ /** * struct pool - Generic pool of packets stored in a buffer - * @buf: Buffer storing packet descriptors - * @buf_size: Total size of buffer + * @buf: Buffer storing packet descriptors, + * a struct vu_dev_region array for passt vhost-user mode + * @buf_size: Total size of buffer, + * 0 for passt vhost-user mode * @size: Number of usable descriptors for the pool * @count: Number of used descriptors for the pool * @pkt: Descriptors: see macros below @@ -22,6 +24,8 @@ struct pool { struct iovec pkt[1]; }; +int vu_packet_check_range(void *buf, size_t offset, size_t len, + const char *start); void packet_add_do(struct pool *p, size_t len, const char *start, const char *func, int line); void *packet_get_do(const struct pool *p, const size_t idx, diff --git a/passt.1 b/passt.1 index 79d134dbe098..822714147be8 100644 --- a/passt.1 +++ b/passt.1 @@ -378,12 +378,20 @@ interface address are configured on a given host interface. .SS \fBpasst\fR-only options .TP -.BR \-s ", " \-\-socket " " \fIpath +.BR \-s ", " \-\-socket-path ", " \-\-socket " " \fIpath Path for UNIX domain socket used by \fBqemu\fR(1) or \fBqrap\fR(1) to connect to \fBpasst\fR. Default is to probe a free socket, not accepting connections, starting from \fI/tmp/passt_1.socket\fR to \fI/tmp/passt_64.socket\fR. +.TP +.BR \-\-vhost-user +Enable vhost-user. The vhost-user command socket is provided by \fB--socket\fR. + +.TP +.BR \-\-print-capabilities +Print back-end capabilities in JSON format, only meaningful for vhost-user mode. + .TP .BR \-F ", " \-\-fd " " \fIFD Pass a pre-opened, connected socket to \fBpasst\fR. Usually the socket is opened diff --git a/passt.c b/passt.c index ad6f0bc32df6..b64efeaf346c 100644 --- a/passt.c +++ b/passt.c @@ -74,6 +74,8 @@ char *epoll_type_str[] = { [EPOLL_TYPE_TAP_PASTA] = "/dev/net/tun device", [EPOLL_TYPE_TAP_PASST] = "connected qemu socket", [EPOLL_TYPE_TAP_LISTEN] = "listening qemu socket", + [EPOLL_TYPE_VHOST_CMD] = "vhost-user command socket", + [EPOLL_TYPE_VHOST_KICK] = "vhost-user kick socket", }; static_assert(ARRAY_SIZE(epoll_type_str) == EPOLL_NUM_TYPES, "epoll_type_str[] doesn't match enum epoll_type"); @@ -206,6 +208,7 @@ int main(int argc, char **argv) struct rlimit limit; struct timespec now; struct sigaction sa; + struct vu_dev vdev; clock_gettime(CLOCK_MONOTONIC, &log_start); @@ -262,6 +265,8 @@ int main(int argc, char **argv) pasta_netns_quit_init(&c); tap_sock_init(&c); + if (c.mode == MODE_VU) + vu_init(&c, &vdev); secret_init(&c); @@ -352,14 +357,31 @@ loop: tcp_timer_handler(&c, ref); break; case EPOLL_TYPE_UDP_LISTEN: - udp_listen_sock_handler(&c, ref, eventmask, &now); + if (c.mode == MODE_VU) { + udp_vu_listen_sock_handler(&c, ref, eventmask, + &now); + } else { + udp_buf_listen_sock_handler(&c, ref, eventmask, + &now); + } break; case EPOLL_TYPE_UDP_REPLY: - udp_reply_sock_handler(&c, ref, eventmask, &now); + if (c.mode == MODE_VU) + udp_vu_reply_sock_handler(&c, ref, eventmask, + &now); + else + udp_buf_reply_sock_handler(&c, ref, eventmask, + &now); break; case EPOLL_TYPE_PING: icmp_sock_handler(&c, ref); break; + case EPOLL_TYPE_VHOST_CMD: + vu_control_handler(&vdev, c.fd_tap, eventmask); + break; + case EPOLL_TYPE_VHOST_KICK: + vu_kick_cb(&vdev, ref, &now); + break; default: /* Can't happen */ ASSERT(0); diff --git a/passt.h b/passt.h index 031c9b669cc4..a98f043c7e64 100644 --- a/passt.h +++ b/passt.h @@ -25,6 +25,8 @@ union epoll_ref; #include "fwd.h" #include "tcp.h" #include "udp.h" +#include "udp_vu.h" +#include "vhost_user.h" /* Default address for our end on the tap interface. Bit 0 of byte 0 must be 0 * (unicast) and bit 1 of byte 1 must be 1 (locally administered). Otherwise @@ -94,6 +96,7 @@ struct fqdn { enum passt_modes { MODE_PASST, MODE_PASTA, + MODE_VU, }; /** @@ -227,6 +230,7 @@ struct ip6_ctx { * @no_ra: Disable router advertisements * @low_wmem: Low probed net.core.wmem_max * @low_rmem: Low probed net.core.rmem_max + * @vdev: vhost-user device */ struct ctx { enum passt_modes mode; @@ -287,6 +291,8 @@ struct ctx { int low_wmem; int low_rmem; + + struct vu_dev *vdev; }; void proto_update_l2_buf(const unsigned char *eth_d, diff --git a/pcap.c b/pcap.c index 46cc4b0d72b6..7e9c56090041 100644 --- a/pcap.c +++ b/pcap.c @@ -140,7 +140,6 @@ void pcap_multiple(const struct iovec *iov, size_t frame_parts, unsigned int n, * containing packet data to write, including L2 header * @iovcnt: Number of buffers (@iov entries) */ -/* cppcheck-suppress unusedFunction */ void pcap_iov(const struct iovec *iov, size_t iovcnt) { struct timespec now; diff --git a/tap.c b/tap.c index 852d83769c29..4ad5e9f4e148 100644 --- a/tap.c +++ b/tap.c @@ -58,6 +58,7 @@ #include "packet.h" #include "tap.h" #include "log.h" +#include "vhost_user.h" /* IPv4 (plus ARP) and IPv6 message batches from tap/guest to IP handlers */ static PACKET_POOL_NOINIT(pool_tap4, TAP_MSGS, pkt_buf); @@ -78,16 +79,22 @@ void tap_send_single(const struct ctx *c, const void *data, size_t l2len) struct iovec iov[2]; size_t iovcnt = 0; - if (c->mode == MODE_PASST) { + switch (c->mode) { + case MODE_PASST: iov[iovcnt] = IOV_OF_LVALUE(vnet_len); iovcnt++; - } - - iov[iovcnt].iov_base = (void *)data; - iov[iovcnt].iov_len = l2len; - iovcnt++; + /* fall through */ + case MODE_PASTA: + iov[iovcnt].iov_base = (void *)data; + iov[iovcnt].iov_len = l2len; + iovcnt++; - tap_send_frames(c, iov, iovcnt, 1); + tap_send_frames(c, iov, iovcnt, 1); + break; + case MODE_VU: + vu_send(c->vdev, data, l2len); + break; + } } /** @@ -406,10 +413,18 @@ size_t tap_send_frames(const struct ctx *c, const struct iovec *iov, if (!nframes) return 0; - if (c->mode == MODE_PASTA) + switch (c->mode) { + case MODE_PASTA: m = tap_send_frames_pasta(c, iov, bufs_per_frame, nframes); - else + break; + case MODE_PASST: m = tap_send_frames_passt(c, iov, bufs_per_frame, nframes); + break; + case MODE_VU: + /* fall through */ + default: + ASSERT(0); + } if (m < nframes) debug("tap: failed to send %zu frames of %zu", @@ -968,7 +983,7 @@ void tap_add_packet(struct ctx *c, ssize_t l2len, char *p) * tap_sock_reset() - Handle closing or failure of connect AF_UNIX socket * @c: Execution context */ -static void tap_sock_reset(struct ctx *c) +void tap_sock_reset(struct ctx *c) { info("Client connection closed%s", c->one_off ? ", exiting" : ""); @@ -979,6 +994,8 @@ static void tap_sock_reset(struct ctx *c) epoll_ctl(c->epollfd, EPOLL_CTL_DEL, c->fd_tap, NULL); close(c->fd_tap); c->fd_tap = -1; + if (c->mode == MODE_VU) + vu_cleanup(c->vdev); } /** @@ -1178,11 +1195,17 @@ static void tap_sock_unix_init(struct ctx *c) ev.data.u64 = ref.u64; epoll_ctl(c->epollfd, EPOLL_CTL_ADD, c->fd_tap_listen, &ev); - info("\nYou can now start qemu (>= 7.2, with commit 13c6be96618c):"); - info(" kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=%s", - c->sock_path); - info("or qrap, for earlier qemu versions:"); - info(" ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio"); + if (c->mode == MODE_VU) { + info("You can start qemu with:"); + info(" kvm ... -chardev socket,id=chr0,path=%s -netdev vhost-user,id=netdev0,chardev=chr0 -device virtio-net,netdev=netdev0 -object memory-backend-memfd,id=memfd0,share=on,size=$RAMSIZE -numa node,memdev=memfd0\n", + c->sock_path); + } else { + info("\nYou can now start qemu (>= 7.2, with commit 13c6be96618c):"); + info(" kvm ... -device virtio-net-pci,netdev=s -netdev stream,id=s,server=off,addr.type=unix,addr.path=%s", + c->sock_path); + info("or qrap, for earlier qemu versions:"); + info(" ./qrap 5 kvm ... -net socket,fd=5 -net nic,model=virtio"); + } } /** @@ -1192,8 +1215,8 @@ static void tap_sock_unix_init(struct ctx *c) */ void tap_listen_handler(struct ctx *c, uint32_t events) { - union epoll_ref ref = { .type = EPOLL_TYPE_TAP_PASST }; struct epoll_event ev = { 0 }; + union epoll_ref ref; int v = INT_MAX / 2; struct ucred ucred; socklen_t len; @@ -1233,6 +1256,10 @@ void tap_listen_handler(struct ctx *c, uint32_t events) trace("tap: failed to set SO_SNDBUF to %i", v); ref.fd = c->fd_tap; + if (c->mode == MODE_VU) + ref.type = EPOLL_TYPE_VHOST_CMD; + else + ref.type = EPOLL_TYPE_TAP_PASST; ev.events = EPOLLIN | EPOLLRDHUP; ev.data.u64 = ref.u64; epoll_ctl(c->epollfd, EPOLL_CTL_ADD, c->fd_tap, &ev); @@ -1294,21 +1321,52 @@ static void tap_sock_tun_init(struct ctx *c) epoll_ctl(c->epollfd, EPOLL_CTL_ADD, c->fd_tap, &ev); } +/** + * tap_sock_update_buf() - Set the buffer base and size for the pool of packets + * @base: Buffer base + * @size Buffer size + */ +void tap_sock_update_buf(void *base, size_t size) +{ + int i; + + pool_tap4_storage.buf = base; + pool_tap4_storage.buf_size = size; + pool_tap6_storage.buf = base; + pool_tap6_storage.buf_size = size; + + for (i = 0; i < TAP_SEQS; i++) { + tap4_l4[i].p.buf = base; + tap4_l4[i].p.buf_size = size; + tap6_l4[i].p.buf = base; + tap6_l4[i].p.buf_size = size; + } +} + /** * tap_sock_init() - Create and set up AF_UNIX socket or tuntap file descriptor * @c: Execution context */ void tap_sock_init(struct ctx *c) { - size_t sz = sizeof(pkt_buf); + size_t sz; + char *buf; int i; - pool_tap4_storage = PACKET_INIT(pool_tap4, TAP_MSGS, pkt_buf, sz); - pool_tap6_storage = PACKET_INIT(pool_tap6, TAP_MSGS, pkt_buf, sz); + if (c->mode == MODE_VU) { + buf = NULL; + sz = 0; + } else { + buf = pkt_buf; + sz = sizeof(pkt_buf); + } + + pool_tap4_storage = PACKET_INIT(pool_tap4, TAP_MSGS, buf, sz); + pool_tap6_storage = PACKET_INIT(pool_tap6, TAP_MSGS, buf, sz); for (i = 0; i < TAP_SEQS; i++) { - tap4_l4[i].p = PACKET_INIT(pool_l4, UIO_MAXIOV, pkt_buf, sz); - tap6_l4[i].p = PACKET_INIT(pool_l4, UIO_MAXIOV, pkt_buf, sz); + tap4_l4[i].p = PACKET_INIT(pool_l4, UIO_MAXIOV, buf, sz); + tap6_l4[i].p = PACKET_INIT(pool_l4, UIO_MAXIOV, buf, sz); } if (c->fd_tap != -1) { /* Passed as --fd */ @@ -1317,10 +1375,17 @@ void tap_sock_init(struct ctx *c) ASSERT(c->one_off); ref.fd = c->fd_tap; - if (c->mode == MODE_PASST) + switch (c->mode) { + case MODE_PASST: ref.type = EPOLL_TYPE_TAP_PASST; - else + break; + case MODE_PASTA: ref.type = EPOLL_TYPE_TAP_PASTA; + break; + case MODE_VU: + ref.type = EPOLL_TYPE_VHOST_CMD; + break; + } ev.events = EPOLLIN | EPOLLRDHUP; ev.data.u64 = ref.u64; diff --git a/tap.h b/tap.h index ec9e2acec460..c5447f7077eb 100644 --- a/tap.h +++ b/tap.h @@ -40,7 +40,8 @@ static inline struct iovec tap_hdr_iov(const struct ctx *c, */ static inline void tap_hdr_update(struct tap_hdr *thdr, size_t l2len) { - thdr->vnet_len = htonl(l2len); + if (thdr) + thdr->vnet_len = htonl(l2len); } void tap_udp4_send(const struct ctx *c, struct in_addr src, in_port_t sport, @@ -68,6 +69,8 @@ void tap_handler_pasta(struct ctx *c, uint32_t events, void tap_handler_passt(struct ctx *c, uint32_t events, const struct timespec *now); int tap_sock_unix_open(char *sock_path); +void tap_sock_reset(struct ctx *c); +void tap_sock_update_buf(void *base, size_t size); void tap_sock_init(struct ctx *c); void tap_flush_pools(void); void tap_handler(struct ctx *c, const struct timespec *now); diff --git a/tcp.c b/tcp.c index 77c62f053f15..2a4a5c8b46b7 100644 --- a/tcp.c +++ b/tcp.c @@ -304,6 +304,7 @@ #include "flow_table.h" #include "tcp_internal.h" #include "tcp_buf.h" +#include "tcp_vu.h" /* MSS rounding: see SET_MSS() */ #define MSS_DEFAULT 536 @@ -903,6 +904,7 @@ static void tcp_fill_header(struct tcphdr *th, * @dlen: TCP payload length * @check: Checksum, if already known * @seq: Sequence number for this segment + * @no_tcp_csum: Do not set TCP checksum * * Return: The IPv4 payload length, host order */ @@ -910,7 +912,7 @@ static size_t tcp_fill_headers4(const struct tcp_tap_conn *conn, struct tap_hdr *taph, struct iphdr *iph, struct tcphdr *th, size_t dlen, const uint16_t *check, - uint32_t seq) + uint32_t seq, bool no_tcp_csum) { const struct flowside *tapside = TAPFLOW(conn); const struct in_addr *src4 = inany_v4(&tapside->oaddr); @@ -929,7 +931,10 @@ static size_t tcp_fill_headers4(const struct tcp_tap_conn *conn, tcp_fill_header(th, conn, seq); - tcp_update_check_tcp4(iph, th); + if (no_tcp_csum) + th->check = 0; + else + tcp_update_check_tcp4(iph, th); tap_hdr_update(taph, l3len + sizeof(struct ethhdr)); @@ -945,13 +950,14 @@ static size_t tcp_fill_headers4(const struct tcp_tap_conn *conn, * @dlen: TCP payload length * @check: Checksum, if already known * @seq: Sequence number for this segment + * @no_tcp_csum: Do not set TCP checksum * * Return: The IPv6 payload length, host order */ static size_t tcp_fill_headers6(const struct tcp_tap_conn *conn, struct tap_hdr *taph, struct ipv6hdr *ip6h, struct tcphdr *th, - size_t dlen, uint32_t seq) + size_t dlen, uint32_t seq, bool no_tcp_csum) { const struct flowside *tapside = TAPFLOW(conn); size_t l4len = dlen + sizeof(*th); @@ -970,7 +976,10 @@ static size_t tcp_fill_headers6(const struct tcp_tap_conn *conn, tcp_fill_header(th, conn, seq); - tcp_update_check_tcp6(ip6h, th); + if (no_tcp_csum) + th->check = 0; + else + tcp_update_check_tcp6(ip6h, th); tap_hdr_update(taph, l4len + sizeof(*ip6h) + sizeof(struct ethhdr)); @@ -984,12 +993,14 @@ static size_t tcp_fill_headers6(const struct tcp_tap_conn *conn, * @dlen: TCP payload length * @check: Checksum, if already known * @seq: Sequence number for this segment + * @no_tcp_csum: Do not set TCP checksum * * Return: IP payload length, host order */ size_t tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn, struct iovec *iov, size_t dlen, - const uint16_t *check, uint32_t seq) + const uint16_t *check, uint32_t seq, + bool no_tcp_csum) { const struct flowside *tapside = TAPFLOW(conn); const struct in_addr *a4 = inany_v4(&tapside->oaddr); @@ -998,13 +1009,13 @@ size_t tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn, return tcp_fill_headers4(conn, iov[TCP_IOV_TAP].iov_base, iov[TCP_IOV_IP].iov_base, iov[TCP_IOV_PAYLOAD].iov_base, dlen, - check, seq); + check, seq, no_tcp_csum); } return tcp_fill_headers6(conn, iov[TCP_IOV_TAP].iov_base, iov[TCP_IOV_IP].iov_base, iov[TCP_IOV_PAYLOAD].iov_base, dlen, - seq); + seq, no_tcp_csum); } /** @@ -1237,6 +1248,9 @@ int tcp_prepare_flags(struct ctx *c, struct tcp_tap_conn *conn, */ int tcp_send_flag(struct ctx *c, struct tcp_tap_conn *conn, int flags) { + if (c->mode == MODE_VU) + return tcp_vu_send_flag(c, conn, flags); + return tcp_buf_send_flag(c, conn, flags); } @@ -1630,6 +1644,9 @@ static int tcp_sock_consume(const struct tcp_tap_conn *conn, uint32_t ack_seq) */ static int tcp_data_from_sock(struct ctx *c, struct tcp_tap_conn *conn) { + if (c->mode == MODE_VU) + return tcp_vu_data_from_sock(c, conn); + return tcp_buf_data_from_sock(c, conn); } diff --git a/tcp_buf.c b/tcp_buf.c index c31e9f31b438..7aa750596af8 100644 --- a/tcp_buf.c +++ b/tcp_buf.c @@ -321,7 +321,7 @@ int tcp_buf_send_flag(struct ctx *c, struct tcp_tap_conn *conn, int flags) return ret; } - l4len = tcp_l2_buf_fill_headers(conn, iov, optlen, NULL, seq); + l4len = tcp_l2_buf_fill_headers(conn, iov, optlen, NULL, seq, false); iov[TCP_IOV_PAYLOAD].iov_len = l4len; if (flags & DUP_ACK) { @@ -378,7 +378,8 @@ static void tcp_data_to_tap(struct ctx *c, struct tcp_tap_conn *conn, tcp4_frame_conns[tcp4_payload_used] = conn; iov = tcp4_l2_iov[tcp4_payload_used++]; - l4len = tcp_l2_buf_fill_headers(conn, iov, dlen, check, seq); + l4len = tcp_l2_buf_fill_headers(conn, iov, dlen, check, seq, + false); iov[TCP_IOV_PAYLOAD].iov_len = l4len; if (tcp4_payload_used > TCP_FRAMES_MEM - 1) tcp_payload_flush(c); @@ -386,7 +387,8 @@ static void tcp_data_to_tap(struct ctx *c, struct tcp_tap_conn *conn, tcp6_frame_conns[tcp6_payload_used] = conn; iov = tcp6_l2_iov[tcp6_payload_used++]; - l4len = tcp_l2_buf_fill_headers(conn, iov, dlen, NULL, seq); + l4len = tcp_l2_buf_fill_headers(conn, iov, dlen, NULL, seq, + false); iov[TCP_IOV_PAYLOAD].iov_len = l4len; if (tcp6_payload_used > TCP_FRAMES_MEM - 1) tcp_payload_flush(c); diff --git a/tcp_internal.h b/tcp_internal.h index aa8bb64f1f33..e7fe735bfcb4 100644 --- a/tcp_internal.h +++ b/tcp_internal.h @@ -91,7 +91,8 @@ void tcp_rst_do(struct ctx *c, struct tcp_tap_conn *conn); size_t tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn, struct iovec *iov, size_t dlen, - const uint16_t *check, uint32_t seq); + const uint16_t *check, uint32_t seq, + bool no_tcp_csum); int tcp_update_seqack_wnd(const struct ctx *c, struct tcp_tap_conn *conn, int force_seq, struct tcp_info *tinfo); int tcp_prepare_flags(struct ctx *c, struct tcp_tap_conn *conn, int flags, diff --git a/tcp_vu.c b/tcp_vu.c new file mode 100644 index 000000000000..206f95980eaf --- /dev/null +++ b/tcp_vu.c @@ -0,0 +1,656 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* tcp_vu.c - TCP L2 vhost-user management functions + * + * Copyright Red Hat + * Author: Laurent Vivier + */ + +#include +#include +#include + +#include + +#include + +#include +#include + +#include "util.h" +#include "ip.h" +#include "passt.h" +#include "siphash.h" +#include "inany.h" +#include "vhost_user.h" +#include "tcp.h" +#include "pcap.h" +#include "flow.h" +#include "tcp_conn.h" +#include "flow_table.h" +#include "tcp_vu.h" +#include "tcp_internal.h" +#include "checksum.h" +#include "vu_common.h" + +/** + * struct tcp_payload_t - TCP header and data to send segments with payload + * @th: TCP header + * @data: TCP data + */ +struct tcp_payload_t { + struct tcphdr th; + uint8_t data[IP_MAX_MTU - sizeof(struct tcphdr)]; +}; + +/** + * struct tcp_flags_t - TCP header and data to send zero-length + * segments (flags) + * @th: TCP header + * @opts TCP options + */ +struct tcp_flags_t { + struct tcphdr th; + char opts[OPT_MSS_LEN + OPT_WS_LEN + 1]; +}; + +/* vhost-user */ +static const struct virtio_net_hdr vu_header = { + .flags = VIRTIO_NET_HDR_F_DATA_VALID, + .gso_type = VIRTIO_NET_HDR_GSO_NONE, +}; + +static struct iovec iov_vu[VIRTQUEUE_MAX_SIZE]; +static struct vu_virtq_element elem[VIRTQUEUE_MAX_SIZE]; + +/** + * tcp_vu_l2_hdrlen() - return the size of the header in level 2 frame (TCP) + * @vdev: vhost-user device + * @v6: Set for IPv6 packet + * + * Return: Return the size of the header + */ +static size_t tcp_vu_l2_hdrlen(const struct vu_dev *vdev, bool v6) +{ + size_t l2_hdrlen; + + l2_hdrlen = vdev->hdrlen + sizeof(struct ethhdr) + + sizeof(struct tcphdr); + + if (v6) + l2_hdrlen += sizeof(struct ipv6hdr); + else + l2_hdrlen += sizeof(struct iphdr); + + return l2_hdrlen; +} + +/** + * tcp_vu_pcap() - Capture a single frame to pcap file (TCP) + * @c: Execution context + * @tapside: Address information for one side of the flow + * @iov: Pointer to the array of IO vectors + * @iov_used: Length of the array + * @l4len: IPv4 Payload length + */ +static void tcp_vu_pcap(const struct ctx *c, const struct flowside *tapside, + struct iovec *iov, int iov_used, size_t l4len) +{ + const struct in_addr *src = inany_v4(&tapside->oaddr); + const struct in_addr *dst = inany_v4(&tapside->eaddr); + const struct vu_dev *vdev = c->vdev; + char *base = iov[0].iov_base; + size_t size = iov[0].iov_len; + struct tcp_payload_t *bp; + uint32_t sum; + + if (!*c->pcap) + return; + + if (src && dst) { + bp = vu_payloadv4(vdev, base); + sum = proto_ipv4_header_psum(l4len, IPPROTO_TCP, + *src, *dst); + } else { + bp = vu_payloadv6(vdev, base); + sum = proto_ipv6_header_psum(l4len, IPPROTO_TCP, + &tapside->oaddr.a6, + &tapside->eaddr.a6); + } + iov[0].iov_base = &bp->th; + iov[0].iov_len = size - ((char *)iov[0].iov_base - base); + bp->th.check = 0; + bp->th.check = csum_iov(iov, iov_used, sum); + + /* set iov for pcap logging */ + iov[0].iov_base = base + vdev->hdrlen; + iov[0].iov_len = size - vdev->hdrlen; + + pcap_iov(iov, iov_used); + + /* restore iov[0] */ + iov[0].iov_base = base; + iov[0].iov_len = size; +} + +/** + * tcp_vu_send_flag() - Send segment with flags to vhost-user (no payload) + * @c: Execution context + * @conn: Connection pointer + * @flags: TCP flags: if not set, send segment only if ACK is due + * + * Return: negative error code on connection reset, 0 otherwise + */ +int tcp_vu_send_flag(struct ctx *c, struct tcp_tap_conn *conn, int flags) +{ + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + const struct flowside *tapside = TAPFLOW(conn); + struct virtio_net_hdr_mrg_rxbuf *vh; + struct iovec l2_iov[TCP_NUM_IOVS]; + size_t l2len, l4len, optlen; + struct iovec in_sg; + struct ethhdr *eh; + int nb_ack; + int ret; + + elem[0].out_num = 0; + elem[0].out_sg = NULL; + elem[0].in_num = 1; + elem[0].in_sg = &in_sg; + ret = vu_queue_pop(vdev, vq, &elem[0]); + if (ret < 0) + return 0; + + if (elem[0].in_num < 1) { + debug("virtio-net receive queue contains no in buffers"); + vu_queue_rewind(vq, 1); + return 0; + } + + vh = elem[0].in_sg[0].iov_base; + + vh->hdr = vu_header; + if (vdev->hdrlen == sizeof(struct virtio_net_hdr_mrg_rxbuf)) + vh->num_buffers = htole16(1); + + l2_iov[TCP_IOV_TAP].iov_base = NULL; + l2_iov[TCP_IOV_TAP].iov_len = 0; + l2_iov[TCP_IOV_ETH].iov_base = (char *)elem[0].in_sg[0].iov_base + vdev->hdrlen; + l2_iov[TCP_IOV_ETH].iov_len = sizeof(struct ethhdr); + + eh = l2_iov[TCP_IOV_ETH].iov_base; + + memcpy(eh->h_dest, c->guest_mac, sizeof(eh->h_dest)); + memcpy(eh->h_source, c->our_tap_mac, sizeof(eh->h_source)); + + if (CONN_V4(conn)) { + struct tcp_flags_t *payload; + struct iphdr *iph; + uint32_t seq; + + l2_iov[TCP_IOV_IP].iov_base = (char *)l2_iov[TCP_IOV_ETH].iov_base + + l2_iov[TCP_IOV_ETH].iov_len; + l2_iov[TCP_IOV_IP].iov_len = sizeof(struct iphdr); + l2_iov[TCP_IOV_PAYLOAD].iov_base = (char *)l2_iov[TCP_IOV_IP].iov_base + + l2_iov[TCP_IOV_IP].iov_len; + + eh->h_proto = htons(ETH_P_IP); + + iph = l2_iov[TCP_IOV_IP].iov_base; + *iph = (struct iphdr)L2_BUF_IP4_INIT(IPPROTO_TCP); + + payload = l2_iov[TCP_IOV_PAYLOAD].iov_base; + payload->th = (struct tcphdr){ + .doff = offsetof(struct tcp_flags_t, opts) / 4, + .ack = 1 + }; + + seq = conn->seq_to_tap; + ret = tcp_prepare_flags(c, conn, flags, &payload->th, payload->opts, &optlen); + if (ret <= 0) { + vu_queue_rewind(vq, 1); + return ret; + } + + l4len = tcp_l2_buf_fill_headers(conn, l2_iov, optlen, NULL, seq, + true); + /* keep the following assignment for clarity */ + /* cppcheck-suppress unreadVariable */ + l2_iov[TCP_IOV_PAYLOAD].iov_len = l4len; + + l2len = l4len + sizeof(*iph) + sizeof(struct ethhdr); + } else { + struct tcp_flags_t *payload; + struct ipv6hdr *ip6h; + uint32_t seq; + + l2_iov[TCP_IOV_IP].iov_base = (char *)l2_iov[TCP_IOV_ETH].iov_base + + l2_iov[TCP_IOV_ETH].iov_len; + l2_iov[TCP_IOV_IP].iov_len = sizeof(struct ipv6hdr); + l2_iov[TCP_IOV_PAYLOAD].iov_base = (char *)l2_iov[TCP_IOV_IP].iov_base + + l2_iov[TCP_IOV_IP].iov_len; + + eh->h_proto = htons(ETH_P_IPV6); + + ip6h = l2_iov[TCP_IOV_IP].iov_base; + *ip6h = (struct ipv6hdr)L2_BUF_IP6_INIT(IPPROTO_TCP); + + payload = l2_iov[TCP_IOV_PAYLOAD].iov_base; + payload->th = (struct tcphdr){ + .doff = offsetof(struct tcp_flags_t, opts) / 4, + .ack = 1 + }; + + seq = conn->seq_to_tap; + ret = tcp_prepare_flags(c, conn, flags, &payload->th, payload->opts, &optlen); + if (ret <= 0) { + vu_queue_rewind(vq, 1); + return ret; + } + + l4len = tcp_l2_buf_fill_headers(conn, l2_iov, optlen, NULL, seq, + true); + /* keep the following assignment for clarity */ + /* cppcheck-suppress unreadVariable */ + l2_iov[TCP_IOV_PAYLOAD].iov_len = l4len; + + l2len = l4len + sizeof(*ip6h) + sizeof(struct ethhdr); + } + l2len += vdev->hdrlen; + ASSERT(l2len <= elem[0].in_sg[0].iov_len); + + elem[0].in_sg[0].iov_len = l2len; + tcp_vu_pcap(c, tapside, &elem[0].in_sg[0], 1, l4len); + + vu_queue_fill(vq, &elem[0], l2len, 0); + nb_ack = 1; + + if (flags & DUP_ACK) { + struct iovec in_sg_dup; + + elem[1].out_num = 0; + elem[1].out_sg = NULL; + elem[1].in_num = 1; + elem[1].in_sg = &in_sg_dup; + ret = vu_queue_pop(vdev, vq, &elem[1]); + if (ret == 0) { + if (elem[1].in_num < 1 || elem[1].in_sg[0].iov_len < l2len) { + vu_queue_rewind(vq, 1); + } else { + memcpy(elem[1].in_sg[0].iov_base, vh, l2len); + nb_ack++; + + tcp_vu_pcap(c, tapside, &elem[1].in_sg[0], 1, + l4len); + + vu_queue_fill(vq, &elem[1], l2len, 1); + } + } + } + + vu_queue_flush(vq, nb_ack); + vu_queue_notify(vdev, vq); + + return 0; +} + +/** tcp_vu_sock_recv() - Receive datastream from socket into vhost-user buffers + * @c: Execution context + * @conn: Connection pointer + * @v4: Set for IPv4 connections + * @fillsize: Number of bytes we can receive + * @datalen: Size of received data (output) + * + * Return: Number of iov entries used to store the data + */ +static ssize_t tcp_vu_sock_recv(struct ctx *c, + struct tcp_tap_conn *conn, bool v4, + size_t fillsize, ssize_t *data_len) +{ + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + static struct iovec in_sg[VIRTQUEUE_MAX_SIZE]; + struct msghdr mh_sock = { 0 }; + uint16_t mss = MSS_GET(conn); + static int in_sg_count; + int s = conn->sock; + size_t l2_hdrlen; + int segment_size; + int iov_cnt; + ssize_t ret; + + l2_hdrlen = tcp_vu_l2_hdrlen(vdev, !v4); + + iov_cnt = 0; + in_sg_count = 0; + segment_size = 0; + *data_len = 0; + while (fillsize > 0 && iov_cnt < VIRTQUEUE_MAX_SIZE - 1 && + in_sg_count < ARRAY_SIZE(in_sg)) { + + elem[iov_cnt].out_num = 0; + elem[iov_cnt].out_sg = NULL; + elem[iov_cnt].in_num = ARRAY_SIZE(in_sg) - in_sg_count; + elem[iov_cnt].in_sg = &in_sg[in_sg_count]; + ret = vu_queue_pop(vdev, vq, &elem[iov_cnt]); + if (ret < 0) + break; + + if (elem[iov_cnt].in_num < 1) { + warn("virtio-net receive queue contains no in buffers"); + break; + } + + in_sg_count += elem[iov_cnt].in_num; + + ASSERT(elem[iov_cnt].in_num == 1); + ASSERT(elem[iov_cnt].in_sg[0].iov_len >= l2_hdrlen); + + if (segment_size == 0) { + iov_vu[iov_cnt + 1].iov_base = + (char *)elem[iov_cnt].in_sg[0].iov_base + l2_hdrlen; + iov_vu[iov_cnt + 1].iov_len = + elem[iov_cnt].in_sg[0].iov_len - l2_hdrlen; + } else { + iov_vu[iov_cnt + 1].iov_base = elem[iov_cnt].in_sg[0].iov_base; + iov_vu[iov_cnt + 1].iov_len = elem[iov_cnt].in_sg[0].iov_len; + } + + if (iov_vu[iov_cnt + 1].iov_len > fillsize) + iov_vu[iov_cnt + 1].iov_len = fillsize; + + segment_size += iov_vu[iov_cnt + 1].iov_len; + if (vdev->hdrlen != sizeof(struct virtio_net_hdr_mrg_rxbuf)) { + segment_size = 0; + } else if (segment_size >= mss) { + iov_vu[iov_cnt + 1].iov_len -= segment_size - mss; + segment_size = 0; + } + fillsize -= iov_vu[iov_cnt + 1].iov_len; + + iov_cnt++; + } + if (iov_cnt == 0) + return 0; + + mh_sock.msg_iov = iov_vu; + mh_sock.msg_iovlen = iov_cnt + 1; + + do + ret = recvmsg(s, &mh_sock, MSG_PEEK); + while (ret < 0 && errno == EINTR); + + if (ret < 0) { + vu_queue_rewind(vq, iov_cnt); + if (errno != EAGAIN && errno != EWOULDBLOCK) { + ret = -errno; + tcp_rst(c, conn); + } + return ret; + } + if (!ret) { + vu_queue_rewind(vq, iov_cnt); + + if ((conn->events & (SOCK_FIN_RCVD | TAP_FIN_SENT)) == SOCK_FIN_RCVD) { + int retf = tcp_vu_send_flag(c, conn, FIN | ACK); + if (retf) { + tcp_rst(c, conn); + return retf; + } + + conn_event(c, conn, TAP_FIN_SENT); + } + return 0; + } + + *data_len = ret; + return iov_cnt; +} + +/** + * tcp_vu_prepare() - Prepare the packet header + * @c: Execution context + * @conn: Connection pointer + * @first: Pointer to the array of IO vectors + * @data_len: Packet data length + * @check: Checksum, if already known + * + * Return: Level-4 length + */ +static size_t tcp_vu_prepare(const struct ctx *c, + struct tcp_tap_conn *conn, struct iovec *first, + size_t data_len, const uint16_t **check) +{ + const struct flowside *toside = TAPFLOW(conn); + const struct vu_dev *vdev = c->vdev; + struct iovec l2_iov[TCP_NUM_IOVS]; + char *base = first->iov_base; + struct ethhdr *eh; + size_t l4len; + + /* we guess the first iovec provided by the guest can embed + * all the headers needed by L2 frame + */ + + l2_iov[TCP_IOV_TAP].iov_base = NULL; + l2_iov[TCP_IOV_TAP].iov_len = 0; + l2_iov[TCP_IOV_ETH].iov_base = base + vdev->hdrlen; + l2_iov[TCP_IOV_ETH].iov_len = sizeof(struct ethhdr); + + eh = l2_iov[TCP_IOV_ETH].iov_base; + + memcpy(eh->h_dest, c->guest_mac, sizeof(eh->h_dest)); + memcpy(eh->h_source, c->our_tap_mac, sizeof(eh->h_source)); + + /* initialize header */ + if (inany_v4(&toside->eaddr) && inany_v4(&toside->oaddr)) { + struct tcp_payload_t *payload; + struct iphdr *iph; + + ASSERT(first[0].iov_len >= vdev->hdrlen + + sizeof(struct ethhdr) + sizeof(struct iphdr) + + sizeof(struct tcphdr)); + + l2_iov[TCP_IOV_IP].iov_base = (char *)l2_iov[TCP_IOV_ETH].iov_base + + l2_iov[TCP_IOV_ETH].iov_len; + l2_iov[TCP_IOV_IP].iov_len = sizeof(struct iphdr); + l2_iov[TCP_IOV_PAYLOAD].iov_base = (char *)l2_iov[TCP_IOV_IP].iov_base + + l2_iov[TCP_IOV_IP].iov_len; + + + eh->h_proto = htons(ETH_P_IP); + + iph = l2_iov[TCP_IOV_IP].iov_base; + *iph = (struct iphdr)L2_BUF_IP4_INIT(IPPROTO_TCP); + payload = l2_iov[TCP_IOV_PAYLOAD].iov_base; + payload->th = (struct tcphdr){ + .doff = offsetof(struct tcp_payload_t, data) / 4, + .ack = 1 + }; + + l4len = tcp_l2_buf_fill_headers(conn, l2_iov, data_len, *check, + conn->seq_to_tap, true); + /* keep the following assignment for clarity */ + /* cppcheck-suppress unreadVariable */ + l2_iov[TCP_IOV_PAYLOAD].iov_len = l4len; + + *check = &iph->check; + } else { + struct tcp_payload_t *payload; + struct ipv6hdr *ip6h; + + ASSERT(first[0].iov_len >= vdev->hdrlen + + sizeof(struct ethhdr) + sizeof(struct ipv6hdr) + + sizeof(struct tcphdr)); + + l2_iov[TCP_IOV_IP].iov_base = (char *)l2_iov[TCP_IOV_ETH].iov_base + + l2_iov[TCP_IOV_ETH].iov_len; + l2_iov[TCP_IOV_IP].iov_len = sizeof(struct ipv6hdr); + l2_iov[TCP_IOV_PAYLOAD].iov_base = (char *)l2_iov[TCP_IOV_IP].iov_base + + l2_iov[TCP_IOV_IP].iov_len; + + + eh->h_proto = htons(ETH_P_IPV6); + + ip6h = l2_iov[TCP_IOV_IP].iov_base; + *ip6h = (struct ipv6hdr)L2_BUF_IP6_INIT(IPPROTO_TCP); + + payload = l2_iov[TCP_IOV_PAYLOAD].iov_base; + payload->th = (struct tcphdr){ + .doff = offsetof(struct tcp_payload_t, data) / 4, + .ack = 1 + }; +; + l4len = tcp_l2_buf_fill_headers(conn, l2_iov, data_len, NULL, + conn->seq_to_tap, true); + /* keep the following assignment for clarity */ + /* cppcheck-suppress unreadVariable */ + l2_iov[TCP_IOV_PAYLOAD].iov_len = l4len; + } + + return l4len; +} + +/** + * tcp_vu_data_from_sock() - Handle new data from socket, queue to vhost-user, + * in window + * @c: Execution context + * @conn: Connection pointer + * + * Return: Negative on connection reset, 0 otherwise + */ +int tcp_vu_data_from_sock(struct ctx *c, struct tcp_tap_conn *conn) +{ + uint32_t wnd_scaled = conn->wnd_from_tap << conn->ws_from_tap; + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + const struct flowside *tapside = TAPFLOW(conn); + uint16_t mss = MSS_GET(conn); + size_t l2_hdrlen, fillsize; + int i, iov_cnt, iov_used; + int v4 = CONN_V4(conn); + uint32_t already_sent = 0; + const uint16_t *check; + struct iovec *first; + int segment_size; + int num_buffers; + ssize_t len; + + if (!vu_queue_enabled(vq) || !vu_queue_started(vq)) { + flow_err(conn, + "Got packet, but RX virtqueue not usable yet"); + return 0; + } + + already_sent = conn->seq_to_tap - conn->seq_ack_from_tap; + + if (SEQ_LT(already_sent, 0)) { + /* RFC 761, section 2.1. */ + flow_trace(conn, "ACK sequence gap: ACK for %u, sent: %u", + conn->seq_ack_from_tap, conn->seq_to_tap); + conn->seq_to_tap = conn->seq_ack_from_tap; + already_sent = 0; + } + + if (!wnd_scaled || already_sent >= wnd_scaled) { + conn_flag(c, conn, STALLED); + conn_flag(c, conn, ACK_FROM_TAP_DUE); + return 0; + } + + /* Set up buffer descriptors we'll fill completely and partially. */ + + fillsize = wnd_scaled; + + if (peek_offset_cap) + already_sent = 0; + + iov_vu[0].iov_base = tcp_buf_discard; + iov_vu[0].iov_len = already_sent; + fillsize -= already_sent; + + /* collect the buffers from vhost-user and fill them with the + * data from the socket + */ + iov_cnt = tcp_vu_sock_recv(c, conn, v4, fillsize, &len); + if (iov_cnt <= 0) + return iov_cnt; + + len -= already_sent; + if (len <= 0) { + conn_flag(c, conn, STALLED); + vu_queue_rewind(vq, iov_cnt); + return 0; + } + + conn_flag(c, conn, ~STALLED); + + /* Likely, some new data was acked too. */ + tcp_update_seqack_wnd(c, conn, 0, NULL); + + /* initialize headers */ + l2_hdrlen = tcp_vu_l2_hdrlen(vdev, !v4); + iov_used = 0; + num_buffers = 0; + check = NULL; + segment_size = 0; + + /* iov_vu is an array of buffers and the buffer size can be + * smaller than the segment size we want to use but with + * num_buffer we can merge several virtio iov buffers in one packet + * we need only to set the packet headers in the first iov and + * num_buffer to the number of iov entries + */ + for (i = 0; i < iov_cnt && len; i++) { + + if (segment_size == 0) + first = &iov_vu[i + 1]; + + if (iov_vu[i + 1].iov_len > (size_t)len) + iov_vu[i + 1].iov_len = len; + + len -= iov_vu[i + 1].iov_len; + iov_used++; + + segment_size += iov_vu[i + 1].iov_len; + num_buffers++; + + if (segment_size >= mss || len == 0 || + i + 1 == iov_cnt || vdev->hdrlen != sizeof(struct virtio_net_hdr_mrg_rxbuf)) { + struct virtio_net_hdr_mrg_rxbuf *vh; + size_t l4len; + + if (i + 1 == iov_cnt) + check = NULL; + + /* restore first iovec base: point to vnet header */ + first->iov_base = (char *)first->iov_base - l2_hdrlen; + first->iov_len = first->iov_len + l2_hdrlen; + + vh = first->iov_base; + + vh->hdr = vu_header; + if (vdev->hdrlen == sizeof(struct virtio_net_hdr_mrg_rxbuf)) + vh->num_buffers = htole16(num_buffers); + + l4len = tcp_vu_prepare(c, conn, first, segment_size, &check); + + tcp_vu_pcap(c, tapside, first, num_buffers, l4len); + + conn->seq_to_tap += segment_size; + + segment_size = 0; + num_buffers = 0; + } + } + + /* release unused buffers */ + vu_queue_rewind(vq, iov_cnt - iov_used); + + /* send packets */ + vu_send_frame(vdev, vq, elem, &iov_vu[1], iov_used); + + conn_flag(c, conn, ACK_FROM_TAP_DUE); + + return 0; +} diff --git a/tcp_vu.h b/tcp_vu.h new file mode 100644 index 000000000000..b433c3e0d06f --- /dev/null +++ b/tcp_vu.h @@ -0,0 +1,12 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright Red Hat + * Author: Laurent Vivier + */ + +#ifndef TCP_VU_H +#define TCP_VU_H + +int tcp_vu_send_flag(struct ctx *c, struct tcp_tap_conn *conn, int flags); +int tcp_vu_data_from_sock(struct ctx *c, struct tcp_tap_conn *conn); + +#endif /*TCP_VU_H */ diff --git a/udp.c b/udp.c index 8a93aad6272b..b7df5c117596 100644 --- a/udp.c +++ b/udp.c @@ -109,8 +109,7 @@ #include "pcap.h" #include "log.h" #include "flow_table.h" - -#define UDP_MAX_FRAMES 32 /* max # of frames to receive at once */ +#include "udp_internal.h" /* "Spliced" sockets indexed by bound port (host order) */ static int udp_splice_ns [IP_VERSIONS][NUM_PORTS]; @@ -118,20 +117,8 @@ static int udp_splice_init[IP_VERSIONS][NUM_PORTS]; /* Static buffers */ -/** - * struct udp_payload_t - UDP header and data for inbound messages - * @uh: UDP header - * @data: UDP data - */ -static struct udp_payload_t { - struct udphdr uh; - char data[USHRT_MAX - sizeof(struct udphdr)]; -#ifdef __AVX2__ -} __attribute__ ((packed, aligned(32))) -#else -} __attribute__ ((packed, aligned(__alignof__(unsigned int)))) -#endif -udp_payload[UDP_MAX_FRAMES]; +/* UDP header and data for inbound messages */ +static struct udp_payload_t udp_payload[UDP_MAX_FRAMES]; /* Ethernet header for IPv4 frames */ static struct ethhdr udp4_eth_hdr; @@ -311,6 +298,7 @@ static void udp_splice_send(const struct ctx *c, size_t start, size_t n, /** * udp_update_hdr4() - Update headers for one IPv4 datagram + * @c: Execution context * @ip4h: Pre-filled IPv4 header (except for tot_len and saddr) * @bp: Pointer to udp_payload_t to update * @toside: Flowside for destination side @@ -318,8 +306,9 @@ static void udp_splice_send(const struct ctx *c, size_t start, size_t n, * * Return: size of IPv4 payload (UDP header + data) */ -static size_t udp_update_hdr4(struct iphdr *ip4h, struct udp_payload_t *bp, - const struct flowside *toside, size_t dlen) +size_t udp_update_hdr4(const struct ctx *c, + struct iphdr *ip4h, struct udp_payload_t *bp, + const struct flowside *toside, size_t dlen) { const struct in_addr *src = inany_v4(&toside->oaddr); const struct in_addr *dst = inany_v4(&toside->eaddr); @@ -336,13 +325,17 @@ static size_t udp_update_hdr4(struct iphdr *ip4h, struct udp_payload_t *bp, bp->uh.source = htons(toside->oport); bp->uh.dest = htons(toside->eport); bp->uh.len = htons(l4len); - csum_udp4(&bp->uh, *src, *dst, bp->data, dlen); + if (c->mode != MODE_VU) + csum_udp4(&bp->uh, *src, *dst, bp->data, dlen); + else + bp->uh.check = 0; return l4len; } /** * udp_update_hdr6() - Update headers for one IPv6 datagram + * @c: Execution context * @ip6h: Pre-filled IPv6 header (except for payload_len and addresses) * @bp: Pointer to udp_payload_t to update * @toside: Flowside for destination side @@ -350,8 +343,9 @@ static size_t udp_update_hdr4(struct iphdr *ip4h, struct udp_payload_t *bp, * * Return: size of IPv6 payload (UDP header + data) */ -static size_t udp_update_hdr6(struct ipv6hdr *ip6h, struct udp_payload_t *bp, - const struct flowside *toside, size_t dlen) +size_t udp_update_hdr6(const struct ctx *c, + struct ipv6hdr *ip6h, struct udp_payload_t *bp, + const struct flowside *toside, size_t dlen) { uint16_t l4len = dlen + sizeof(bp->uh); @@ -365,19 +359,29 @@ static size_t udp_update_hdr6(struct ipv6hdr *ip6h, struct udp_payload_t *bp, bp->uh.source = htons(toside->oport); bp->uh.dest = htons(toside->eport); bp->uh.len = ip6h->payload_len; - csum_udp6(&bp->uh, &toside->oaddr.a6, &toside->eaddr.a6, bp->data, dlen); + if (c->mode != MODE_VU) { + csum_udp6(&bp->uh, &toside->oaddr.a6, &toside->eaddr.a6, + bp->data, dlen); + } else { + /* O is an invalid checksum for UDP IPv6 and dropped by + * the kernel stack, even if the checksum is disabled by virtio + * flags. We need to put any non-zero value here. + */ + bp->uh.check = 0xffff; + } return l4len; } /** * udp_tap_prepare() - Convert one datagram into a tap frame + * @c: Execution context * @mmh: Receiving mmsghdr array * @idx: Index of the datagram to prepare * @toside: Flowside for destination side */ -static void udp_tap_prepare(const struct mmsghdr *mmh, unsigned idx, - const struct flowside *toside) +static void udp_tap_prepare(const struct ctx *c, const struct mmsghdr *mmh, + unsigned idx, const struct flowside *toside) { struct iovec (*tap_iov)[UDP_NUM_IOVS] = &udp_l2_iov[idx]; struct udp_payload_t *bp = &udp_payload[idx]; @@ -385,13 +389,15 @@ static void udp_tap_prepare(const struct mmsghdr *mmh, unsigned idx, size_t l4len; if (!inany_v4(&toside->eaddr) || !inany_v4(&toside->oaddr)) { - l4len = udp_update_hdr6(&bm->ip6h, bp, toside, mmh[idx].msg_len); + l4len = udp_update_hdr6(c, &bm->ip6h, bp, toside, + mmh[idx].msg_len); tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip6h) + sizeof(udp6_eth_hdr)); (*tap_iov)[UDP_IOV_ETH] = IOV_OF_LVALUE(udp6_eth_hdr); (*tap_iov)[UDP_IOV_IP] = IOV_OF_LVALUE(bm->ip6h); } else { - l4len = udp_update_hdr4(&bm->ip4h, bp, toside, mmh[idx].msg_len); + l4len = udp_update_hdr4(c, &bm->ip4h, bp, toside, + mmh[idx].msg_len); tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip4h) + sizeof(udp4_eth_hdr)); (*tap_iov)[UDP_IOV_ETH] = IOV_OF_LVALUE(udp4_eth_hdr); @@ -408,7 +414,7 @@ static void udp_tap_prepare(const struct mmsghdr *mmh, unsigned idx, * * #syscalls recvmsg */ -static bool udp_sock_recverr(int s) +bool udp_sock_recverr(int s) { const struct sock_extended_err *ee; const struct cmsghdr *hdr; @@ -495,7 +501,7 @@ static int udp_sock_recv(const struct ctx *c, int s, uint32_t events, } /** - * udp_listen_sock_handler() - Handle new data from socket + * udp_buf_listen_sock_handler() - Handle new data from socket * @c: Execution context * @ref: epoll reference * @events: epoll events bitmap @@ -503,8 +509,8 @@ static int udp_sock_recv(const struct ctx *c, int s, uint32_t events, * * #syscalls recvmmsg */ -void udp_listen_sock_handler(const struct ctx *c, union epoll_ref ref, - uint32_t events, const struct timespec *now) +void udp_buf_listen_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now) { struct mmsghdr *mmh_recv = ref.udp.v6 ? udp6_mh_recv : udp4_mh_recv; int n, i; @@ -527,7 +533,7 @@ void udp_listen_sock_handler(const struct ctx *c, union epoll_ref ref, if (pif_is_socket(batchpif)) { udp_splice_prepare(mmh_recv, i); } else if (batchpif == PIF_TAP) { - udp_tap_prepare(mmh_recv, i, + udp_tap_prepare(c, mmh_recv, i, flowside_at_sidx(batchsidx)); } @@ -561,7 +567,7 @@ void udp_listen_sock_handler(const struct ctx *c, union epoll_ref ref, } /** - * udp_reply_sock_handler() - Handle new data from flow specific socket + * udp_buf_reply_sock_handler() - Handle new data from flow specific socket * @c: Execution context * @ref: epoll reference * @events: epoll events bitmap @@ -569,8 +575,8 @@ void udp_listen_sock_handler(const struct ctx *c, union epoll_ref ref, * * #syscalls recvmmsg */ -void udp_reply_sock_handler(const struct ctx *c, union epoll_ref ref, - uint32_t events, const struct timespec *now) +void udp_buf_reply_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now) { const struct flowside *fromside = flowside_at_sidx(ref.flowside); flow_sidx_t tosidx = flow_sidx_opposite(ref.flowside); @@ -594,7 +600,7 @@ void udp_reply_sock_handler(const struct ctx *c, union epoll_ref ref, if (pif_is_socket(topif)) udp_splice_prepare(mmh_recv, i); else if (topif == PIF_TAP) - udp_tap_prepare(mmh_recv, i, toside); + udp_tap_prepare(c, mmh_recv, i, toside); } if (pif_is_socket(topif)) { diff --git a/udp.h b/udp.h index fb42e1c50d70..77b29260e8d1 100644 --- a/udp.h +++ b/udp.h @@ -9,10 +9,10 @@ #define UDP_TIMER_INTERVAL 1000 /* ms */ void udp_portmap_clear(void); -void udp_listen_sock_handler(const struct ctx *c, union epoll_ref ref, - uint32_t events, const struct timespec *now); -void udp_reply_sock_handler(const struct ctx *c, union epoll_ref ref, - uint32_t events, const struct timespec *now); +void udp_buf_listen_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now); +void udp_buf_reply_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now); int udp_tap_handler(const struct ctx *c, uint8_t pif, sa_family_t af, const void *saddr, const void *daddr, const struct pool *p, int idx, const struct timespec *now); diff --git a/udp_internal.h b/udp_internal.h new file mode 100644 index 000000000000..7dd45753698f --- /dev/null +++ b/udp_internal.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later + * Copyright (c) 2021 Red Hat GmbH + * Author: Stefano Brivio + */ + +#ifndef UDP_INTERNAL_H +#define UDP_INTERNAL_H + +#include "tap.h" /* needed by udp_meta_t */ + +#define UDP_MAX_FRAMES 32 /* max # of frames to receive at once */ + +/** + * struct udp_payload_t - UDP header and data for inbound messages + * @uh: UDP header + * @data: UDP data + */ +struct udp_payload_t { + struct udphdr uh; + char data[USHRT_MAX - sizeof(struct udphdr)]; +#ifdef __AVX2__ +} __attribute__ ((packed, aligned(32))); +#else +} __attribute__ ((packed, aligned(__alignof__(unsigned int)))); +#endif + +size_t udp_update_hdr4(const struct ctx *c, + struct iphdr *ip4h, struct udp_payload_t *bp, + const struct flowside *toside, size_t dlen); +size_t udp_update_hdr6(const struct ctx *c, + struct ipv6hdr *ip6h, struct udp_payload_t *bp, + const struct flowside *toside, size_t dlen); +bool udp_sock_recverr(int s); +#endif /* UDP_INTERNAL_H */ diff --git a/udp_vu.c b/udp_vu.c new file mode 100644 index 000000000000..39a7a30b209c --- /dev/null +++ b/udp_vu.c @@ -0,0 +1,386 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* udp_vu.c - UDP L2 vhost-user management functions + * + * Copyright Red Hat + * Author: Laurent Vivier + */ + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include "checksum.h" +#include "util.h" +#include "ip.h" +#include "siphash.h" +#include "inany.h" +#include "passt.h" +#include "pcap.h" +#include "log.h" +#include "vhost_user.h" +#include "udp_internal.h" +#include "flow.h" +#include "flow_table.h" +#include "udp_flow.h" +#include "udp_vu.h" +#include "vu_common.h" + +/* vhost-user */ +static const struct virtio_net_hdr vu_header = { + .flags = VIRTIO_NET_HDR_F_DATA_VALID, + .gso_type = VIRTIO_NET_HDR_GSO_NONE, +}; + +static struct iovec iov_vu [VIRTQUEUE_MAX_SIZE]; +static struct vu_virtq_element elem [VIRTQUEUE_MAX_SIZE]; +static struct iovec in_sg[VIRTQUEUE_MAX_SIZE]; +static int in_sg_count; + +/** + * udp_vu_l2_hdrlen() - return the size of the header in level 2 frame (UDP) + * @vdev: vhost-user device + * @v6: Set for IPv6 packet + * + * Return: Return the size of the header + */ +static size_t udp_vu_l2_hdrlen(const struct vu_dev *vdev, bool v6) +{ + size_t l2_hdrlen; + + l2_hdrlen = vdev->hdrlen + sizeof(struct ethhdr) + + sizeof(struct udphdr); + + if (v6) + l2_hdrlen += sizeof(struct ipv6hdr); + else + l2_hdrlen += sizeof(struct iphdr); + + return l2_hdrlen; +} + +/** + * udp_vu_sock_recv() - Receive datagrams from socket into vhost-user buffers + * @c: Execution context + * @s_in: Source socket address, filled in by recvmsg() + * @s: Socket to receive from + * @events: epoll events bitmap + * @v6: Set for IPv6 connections + * @datalen: Size of received data (output) + * + * Return: Number of iov entries used to store the datagram + */ +static int udp_vu_sock_recv(const struct ctx *c, union sockaddr_inany *s_in, + int s, uint32_t events, bool v6, ssize_t *data_len) +{ + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + int virtqueue_max, iov_cnt, idx, iov_used; + size_t fillsize, size, off, l2_hdrlen; + struct virtio_net_hdr_mrg_rxbuf *vh; + struct msghdr msg = { 0 }; + char *base; + + ASSERT(!c->no_udp); + + /* Clear any errors first */ + if (events & EPOLLERR) { + while (udp_sock_recverr(s)) + ; + } + + if (!(events & EPOLLIN)) + return 0; + + /* compute L2 header length */ + + if (vu_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) + virtqueue_max = VIRTQUEUE_MAX_SIZE; + else + virtqueue_max = 1; + + l2_hdrlen = udp_vu_l2_hdrlen(vdev, v6); + + msg.msg_name = s_in; + msg.msg_namelen = sizeof(union sockaddr_inany); + + fillsize = USHRT_MAX; + iov_cnt = 0; + in_sg_count = 0; + while (fillsize && iov_cnt < virtqueue_max && + in_sg_count < ARRAY_SIZE(in_sg)) { + int ret; + + elem[iov_cnt].out_num = 0; + elem[iov_cnt].out_sg = NULL; + elem[iov_cnt].in_num = ARRAY_SIZE(in_sg) - in_sg_count; + elem[iov_cnt].in_sg = &in_sg[in_sg_count]; + ret = vu_queue_pop(vdev, vq, &elem[iov_cnt]); + if (ret < 0) + break; + in_sg_count += elem[iov_cnt].in_num; + + if (elem[iov_cnt].in_num < 1) { + err("virtio-net receive queue contains no in buffers"); + vu_queue_rewind(vq, iov_cnt); + return 0; + } + ASSERT(elem[iov_cnt].in_num == 1); + ASSERT(elem[iov_cnt].in_sg[0].iov_len >= l2_hdrlen); + + if (iov_cnt == 0) { + base = elem[iov_cnt].in_sg[0].iov_base; + size = elem[iov_cnt].in_sg[0].iov_len; + + /* keep space for the headers */ + iov_vu[0].iov_base = base + l2_hdrlen; + iov_vu[0].iov_len = size - l2_hdrlen; + } else { + iov_vu[iov_cnt].iov_base = elem[iov_cnt].in_sg[0].iov_base; + iov_vu[iov_cnt].iov_len = elem[iov_cnt].in_sg[0].iov_len; + } + + if (iov_vu[iov_cnt].iov_len > fillsize) + iov_vu[iov_cnt].iov_len = fillsize; + + fillsize -= iov_vu[iov_cnt].iov_len; + + iov_cnt++; + } + if (iov_cnt == 0) + return 0; + + msg.msg_iov = iov_vu; + msg.msg_iovlen = iov_cnt; + + *data_len = recvmsg(s, &msg, 0); + if (*data_len < 0) { + vu_queue_rewind(vq, iov_cnt); + return 0; + } + + /* restore original values */ + iov_vu[0].iov_base = base; + iov_vu[0].iov_len = size; + + /* count the numbers of buffer filled by recvmsg() */ + idx = iov_skip_bytes(iov_vu, iov_cnt, l2_hdrlen + *data_len, + &off); + /* adjust last iov length */ + if (idx < iov_cnt) + iov_vu[idx].iov_len = off; + iov_used = idx + !!off; + + /* release unused buffers */ + vu_queue_rewind(vq, iov_cnt - iov_used); + + vh = (struct virtio_net_hdr_mrg_rxbuf *)base; + vh->hdr = vu_header; + if (vdev->hdrlen == sizeof(struct virtio_net_hdr_mrg_rxbuf)) + vh->num_buffers = htole16(iov_used); + + return iov_used; +} + +/** + * udp_vu_prepare() - Prepare the packet header + * @c: Execution context + * @toside: Address information for one side of the flow + * @datalen: Packet data length + * + * Return:i Level-4 length + */ +static size_t udp_vu_prepare(const struct ctx *c, + const struct flowside *toside, ssize_t data_len) +{ + const struct vu_dev *vdev = c->vdev; + struct ethhdr *eh; + size_t l4len; + + /* ethernet header */ + eh = vu_eth(vdev, iov_vu[0].iov_base); + + memcpy(eh->h_dest, c->guest_mac, sizeof(eh->h_dest)); + memcpy(eh->h_source, c->our_tap_mac, sizeof(eh->h_source)); + + /* initialize header */ + if (inany_v4(&toside->eaddr) && inany_v4(&toside->oaddr)) { + struct iphdr *iph = vu_ip(vdev, iov_vu[0].iov_base); + struct udp_payload_t *bp = vu_payloadv4(vdev, + iov_vu[0].iov_base); + + eh->h_proto = htons(ETH_P_IP); + + *iph = (struct iphdr)L2_BUF_IP4_INIT(IPPROTO_UDP); + + l4len = udp_update_hdr4(c, iph, bp, toside, data_len); + } else { + struct ipv6hdr *ip6h = vu_ip(vdev, iov_vu[0].iov_base); + struct udp_payload_t *bp = vu_payloadv6(vdev, + iov_vu[0].iov_base); + + eh->h_proto = htons(ETH_P_IPV6); + + *ip6h = (struct ipv6hdr)L2_BUF_IP6_INIT(IPPROTO_UDP); + + l4len = udp_update_hdr6(c, ip6h, bp, toside, data_len); + } + + return l4len; +} + +/** + * udp_vu_pcap() - Capture a single frame to pcap file (UDP) + * @c: Execution context + * @toside: ddress information for one side of the flow + * @l4len: IPv4 Payload length + * @iov_used: Length of the array + */ +static void udp_vu_pcap(const struct ctx *c, const struct flowside *toside, + size_t l4len, int iov_used) +{ + const struct in_addr *src4 = inany_v4(&toside->oaddr); + const struct in_addr *dst4 = inany_v4(&toside->eaddr); + const struct vu_dev *vdev = c->vdev; + char *base = iov_vu[0].iov_base; + size_t size = iov_vu[0].iov_len; + struct udp_payload_t *bp; + uint32_t sum; + + if (!*c->pcap) + return; + + if (src4 && dst4) { + bp = vu_payloadv4(vdev, base); + sum = proto_ipv4_header_psum(l4len, IPPROTO_UDP, *src4, *dst4); + } else { + bp = vu_payloadv6(vdev, base); + sum = proto_ipv6_header_psum(l4len, IPPROTO_UDP, + &toside->oaddr.a6, + &toside->eaddr.a6); + bp->uh.check = 0; /* by default, set to 0xffff */ + } + + iov_vu[0].iov_base = &bp->uh; + iov_vu[0].iov_len = size - ((char *)iov_vu[0].iov_base - base); + + bp->uh.check = csum_iov(iov_vu, iov_used, sum); + + /* set iov for pcap logging */ + iov_vu[0].iov_base = base + vdev->hdrlen; + iov_vu[0].iov_len = size - vdev->hdrlen; + pcap_iov(iov_vu, iov_used); + + /* restore iov_vu[0] */ + iov_vu[0].iov_base = base; + iov_vu[0].iov_len = size; +} + +/** + * udp_vu_listen_sock_handler() - Handle new data from socket + * @c: Execution context + * @ref: epoll reference + * @events: epoll events bitmap + * @now: Current timestamp + */ +void udp_vu_listen_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now) +{ + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + bool v6 = ref.udp.v6; + int i; + + for (i = 0; i < UDP_MAX_FRAMES; i++) { + union sockaddr_inany s_in; + flow_sidx_t batchsidx; + uint8_t batchpif; + ssize_t data_len; + int iov_used; + + iov_used = udp_vu_sock_recv(c, &s_in, ref.fd, + events, v6, &data_len); + if (iov_used <= 0) + return; + + batchsidx = udp_flow_from_sock(c, ref, &s_in, now); + batchpif = pif_at_sidx(batchsidx); + + if (batchpif == PIF_TAP) { + size_t l4len; + + l4len = udp_vu_prepare(c, flowside_at_sidx(batchsidx), + data_len); + udp_vu_pcap(c, flowside_at_sidx(batchsidx), l4len, + iov_used); + vu_send_frame(vdev, vq, elem, iov_vu, iov_used); + } else if (flow_sidx_valid(batchsidx)) { + flow_sidx_t fromsidx = flow_sidx_opposite(batchsidx); + struct udp_flow *uflow = udp_at_sidx(batchsidx); + + flow_err(uflow, + "No support for forwarding UDP from %s to %s", + pif_name(pif_at_sidx(fromsidx)), + pif_name(batchpif)); + } else { + debug("Discarding 1 datagram without flow"); + } + } +} + +/** + * udp_vu_reply_sock_handler() - Handle new data from flow specific socket + * @c: Execution context + * @ref: epoll reference + * @events: epoll events bitmap + * @now: Current timestamp + */ +void udp_vu_reply_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now) +{ + flow_sidx_t tosidx = flow_sidx_opposite(ref.flowside); + const struct flowside *toside = flowside_at_sidx(tosidx); + struct vu_dev *vdev = c->vdev; + struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; + struct udp_flow *uflow = udp_at_sidx(ref.flowside); + uint8_t topif = pif_at_sidx(tosidx); + bool v6 = ref.udp.v6; + int i; + + ASSERT(!c->no_udp); + ASSERT(uflow); + + for (i = 0; i < UDP_MAX_FRAMES; i++) { + union sockaddr_inany s_in; + ssize_t data_len; + int iov_used; + + iov_used = udp_vu_sock_recv(c, &s_in, ref.fd, + events, v6, &data_len); + if (iov_used <= 0) + return; + flow_trace(uflow, "Received 1 datagram on reply socket"); + uflow->ts = now->tv_sec; + + if (topif == PIF_TAP) { + size_t l4len; + + l4len = udp_vu_prepare(c, toside, data_len); + udp_vu_pcap(c, toside, l4len, iov_used); + vu_send_frame(vdev, vq, elem, iov_vu, iov_used); + } else { + uint8_t frompif = pif_at_sidx(ref.flowside); + + flow_err(uflow, + "No support for forwarding UDP from %s to %s", + pif_name(frompif), pif_name(topif)); + } + } +} diff --git a/udp_vu.h b/udp_vu.h new file mode 100644 index 000000000000..ba7018d3bf01 --- /dev/null +++ b/udp_vu.h @@ -0,0 +1,13 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright Red Hat + * Author: Laurent Vivier + */ + +#ifndef UDP_VU_H +#define UDP_VU_H + +void udp_vu_listen_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now); +void udp_vu_reply_sock_handler(const struct ctx *c, union epoll_ref ref, + uint32_t events, const struct timespec *now); +#endif /* UDP_VU_H */ diff --git a/vhost_user.c b/vhost_user.c index 6008a8adc967..7df503e3d8b1 100644 --- a/vhost_user.c +++ b/vhost_user.c @@ -52,7 +52,6 @@ * this is part of the vhost-user backend * convention. */ -/* cppcheck-suppress unusedFunction */ void vu_print_capabilities(void) { info("{"); @@ -162,9 +161,7 @@ static void vmsg_close_fds(const struct vhost_user_msg *vmsg) */ static void vu_remove_watch(const struct vu_dev *vdev, int fd) { - /* Placeholder to add passt related code */ - (void)vdev; - (void)fd; + epoll_ctl(vdev->context->epollfd, EPOLL_CTL_DEL, fd, NULL); } /** @@ -428,7 +425,6 @@ static bool map_ring(struct vu_dev *vdev, struct vu_virtq *vq) * * Return: 0 if the zone is in a mapped memory region, -1 otherwise */ -/* cppcheck-suppress unusedFunction */ int vu_packet_check_range(void *buf, size_t offset, size_t len, const char *start) { @@ -518,6 +514,14 @@ static bool vu_set_mem_table_exec(struct vu_dev *vdev, } } + /* As vu_packet_check_range() has no access to the number of + * memory regions, mark the end of the array with mmap_addr = 0 + */ + ASSERT(vdev->nregions < VHOST_USER_MAX_RAM_SLOTS - 1); + vdev->regions[vdev->nregions].mmap_addr = 0; + + tap_sock_update_buf(vdev->regions, 0); + return false; } @@ -646,9 +650,12 @@ static bool vu_get_vring_base_exec(struct vu_dev *vdev, */ static void vu_set_watch(const struct vu_dev *vdev, int fd) { - /* Placeholder to add passt related code */ - (void)vdev; - (void)fd; + union epoll_ref ref = { .type = EPOLL_TYPE_VHOST_KICK, .fd = fd }; + struct epoll_event ev = { 0 }; + + ev.data.u64 = ref.u64; + ev.events = EPOLLIN; + epoll_ctl(vdev->context->epollfd, EPOLL_CTL_ADD, fd, &ev); } /** @@ -688,7 +695,6 @@ static int vu_wait_queue(const struct vu_virtq *vq) * * Return: number of bytes sent, -1 if there is an error */ -/* cppcheck-suppress unusedFunction */ int vu_send(struct vu_dev *vdev, const void *buf, size_t size) { struct vu_virtq *vq = &vdev->vq[VHOST_USER_RX_QUEUE]; @@ -876,7 +882,6 @@ static void vu_handle_tx(struct vu_dev *vdev, int index, * @ref: epoll reference information * @now: Current timestamp */ -/* cppcheck-suppress unusedFunction */ void vu_kick_cb(struct vu_dev *vdev, union epoll_ref ref, const struct timespec *now) { @@ -1122,11 +1127,11 @@ static bool vu_set_vring_enable_exec(struct vu_dev *vdev, * @c: execution context * @vdev: vhost-user device */ -/* cppcheck-suppress unusedFunction */ void vu_init(struct ctx *c, struct vu_dev *vdev) { int i; + c->vdev = vdev; vdev->context = c; vdev->hdrlen = 0; for (i = 0; i < VHOST_USER_MAX_QUEUES; i++) { @@ -1143,7 +1148,6 @@ void vu_init(struct ctx *c, struct vu_dev *vdev) * vu_cleanup() - Reset vhost-user device * @vdev: vhost-user device */ -/* cppcheck-suppress unusedFunction */ void vu_cleanup(struct vu_dev *vdev) { unsigned int i; @@ -1191,8 +1195,7 @@ void vu_cleanup(struct vu_dev *vdev) */ static void vu_sock_reset(struct vu_dev *vdev) { - /* Placeholder to add passt related code */ - (void)vdev; + tap_sock_reset(vdev->context); } static bool (*vu_handle[VHOST_USER_MAX])(struct vu_dev *vdev, @@ -1220,7 +1223,6 @@ static bool (*vu_handle[VHOST_USER_MAX])(struct vu_dev *vdev, * @fd: vhost-user message socket * @events: epoll events */ -/* cppcheck-suppress unusedFunction */ void vu_control_handler(struct vu_dev *vdev, int fd, uint32_t events) { struct vhost_user_msg msg = { 0 }; diff --git a/virtio.c b/virtio.c index 237395396606..31e56def2c23 100644 --- a/virtio.c +++ b/virtio.c @@ -562,7 +562,6 @@ void vu_queue_unpop(struct vu_virtq *vq) * @vq: Virtqueue * @num: Number of element to unpop */ -/* cppcheck-suppress unusedFunction */ bool vu_queue_rewind(struct vu_virtq *vq, unsigned int num) { if (num > vq->inuse) diff --git a/vu_common.c b/vu_common.c new file mode 100644 index 000000000000..5b469da9731f --- /dev/null +++ b/vu_common.c @@ -0,0 +1,35 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* Copyright Red Hat + * Author: Laurent Vivier + * + * common_vu.c - vhost-user common UDP and TCP functions + */ + +#include +#include + +#include "util.h" +#include "passt.h" +#include "vhost_user.h" +#include "vu_common.h" + +/** + * vu_send_frame() - Send one frame to the vhost-user interface + * @vdev: vhost-user device + * @vq: vhost-user virtqueue + * @elem: virtqueue element array to send back to the virqueue + * @iov_vu: iovec array containing the data to send + * @iov_used: Length of the array + */ +void vu_send_frame(const struct vu_dev *vdev, struct vu_virtq *vq, + struct vu_virtq_element *elem, const struct iovec *iov_vu, + int iov_used) +{ + int i; + + for (i = 0; i < iov_used; i++) + vu_queue_fill(vq, &elem[i], iov_vu[i].iov_len, i); + + vu_queue_flush(vq, iov_used); + vu_queue_notify(vdev, vq); +} diff --git a/vu_common.h b/vu_common.h new file mode 100644 index 000000000000..d2ea46bd379b --- /dev/null +++ b/vu_common.h @@ -0,0 +1,34 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later + * Copyright Red Hat + * Author: Laurent Vivier + * + * vhost-user common UDP and TCP functions + */ + +#ifndef VU_COMMON_H +#define VU_COMMON_H + +static inline void *vu_eth(const struct vu_dev *vdev, void *base) +{ + return ((char *)base + vdev->hdrlen); +} + +static inline void *vu_ip(const struct vu_dev *vdev, void *base) +{ + return (struct ethhdr *)vu_eth(vdev, base) + 1; +} + +static inline void *vu_payloadv4(const struct vu_dev *vdev, void *base) +{ + return (struct iphdr *)vu_ip(vdev, base) + 1; +} + +static inline void *vu_payloadv6(const struct vu_dev *vdev, void *base) +{ + return (struct ipv6hdr *)vu_ip(vdev, base) + 1; +} + +void vu_send_frame(const struct vu_dev *vdev, struct vu_virtq *vq, + struct vu_virtq_element *elem, const struct iovec *iov_vu, + int iov_used); +#endif /* VU_COMMON_H */ -- 2.46.0