[PATCH v13 00/10] Use true MAC address of LAN local remote hosts

[PATCH v13 00/10] Use true MAC address of LAN local remote hosts

[Jon Maloy]

Bug #120 asks us to use the true MAC addresses of LAN local
remote hosts, since some programs need this information.
These commits introduces this for ARP, NDP, UDP, TCP and
ICMP.

---
v3: Updated according to feedback from Stefano and David:
    - Made the ARP/NDP lookup call filter out the requested address
      by itself, qualified by the index if the template interface
    - Moved the flow specific MAC address from struct flowside to
      struct flow_common.

v4: - Updated according to feedback from David and Stefan
    - Added a cache table for ARP/NDP table contents

v5: - Updated according to feedback from David and Stefan
    - Added cache table entries to FIFO/LRU queue
    - New criteria for when to consult ARP/NDP

v6: - Simplified and merged mac cache table commits
    - Other changes after feedback from David.
v7: - Fixes in patch #2 based on feedback from David
      and Stefano.
v8: - Redesigned netlink and cache table part to be based
      on a subscription model.
v8: - Small fix to patch #2 so that we cover the case
      when a MAC addess for a host has changed.
    - Added a commit where we send a gratuitous ARP/
      unsolicitated NA to the guest when a new host is
      added to the neighbour cache table.
v10: - Some fixes after feedback from David Gibson
     - Reordered: Moved patch #9 to position #3.
     - Added synchronization step between ARP/NDP
       table contents and the neigbour table at
       initialization. This reduces the number of
       "false" ARP/NDP replies drastically, but not
       completly. 
     - (Next step could be to scan over the flow table
       and update affeced entries when we receive
       a MAC address update.)
v11: - Corrected the gratuitous ARP implementation to use
       the "ARP Announcement" model instead of the
       "Gratuitous ARP reply" model.
v12: - Updated based on feedback from David and Stefano
     - Added special handling of default GW and loopback
       addresses.
v13: - Updated based on discussion with David and Stefano
     - Conceptually moved to only considering guest-side
       visible addresss. A lot of things became simpler
       and clearer through this change. Thank you, David.
     - Introduced a 'permanent' flag in the special
       entries representing addessed mapping to own host
       and conditionally the guest gw. This flag indicates
       those entries cannot be altered by possible remote hosts
       shadowed by these addresses. Suggested by Stefano.
     - Reordered patch ##4 and 5, since #5 cannot work
       correctly for NDP unsolicited NA until #4 is in place.
     - Added a new commit #2 to get later access to the flag
       no_map_gw. It was wrong to call fwd_neigh_table_init()
       from inside conf(), it has to be done in main() after
       random_init() and tap_backend_init().

Jon Maloy (10):
  netlink: add subscription on changes in NDP/ARP table
  passt: add no_map_gw flag to struct ctx
  fwd: Add cache table for ARP/NDP contents
  arp/ndp: respond with true MAC address of LAN local remote hosts
  arp/ndp: send ARP announcement / unsolicited NA when neigbour entry
    added
  flow: add MAC address of LAN local remote hosts to flow
  udp: forward external source MAC address through tap interface
  tcp: forward external source MAC address through tap interface
  tap: change signature of function tap_push_l2h()
  icmp: let icmp use mac address from flowside structure

 arp.c          |  50 ++++++++++-
 arp.h          |   2 +
 conf.c         |  10 +--
 epoll_type.h   |   2 +
 flow.c         |   2 +
 flow.h         |   2 +
 fwd.c          | 232 +++++++++++++++++++++++++++++++++++++++++++++++++
 fwd.h          |   7 ++
 icmp.c         |   8 +-
 inany.c        |   1 +
 ndp.c          |  16 +++-
 ndp.h          |   1 +
 netlink.c      | 218 +++++++++++++++++++++++++++++++++++++++++++++-
 netlink.h      |   4 +
 passt.c        |  17 ++--
 passt.h        |   4 +-
 pasta.c        |   2 +-
 tap.c          |  24 ++---
 tap.h          |   7 +-
 tcp.c          |  20 ++++-
 tcp.h          |   2 +-
 tcp_buf.c      |  37 ++++----
 tcp_internal.h |   4 +-
 tcp_vu.c       |   5 +-
 udp.c          |  57 +++++++-----
 udp.h          |   2 +-
 util.h         |   2 +
 27 files changed, 650 insertions(+), 88 deletions(-)

-- 
2.50.1

[PATCH v13 01/10] netlink: add subscription on changes in NDP/ARP table

[Jon Maloy]

The solution to bug https://bugs.passt.top/show_bug.cgi?id=120
requires the ability to translate from an IP address to its
corresponding MAC address in cases where those are present in
the ARP or NDP tables.

To keep track of the contents of these tables we add a netlink
based neighbour subscription feature.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>

---
v3: - Added an attribute contianing NDA_DST to sent message, so
      that we let the kernel do the filtering of the IP address
      and return only one entry.
    - Added interface index to the call signature. Since the only
      interface we know is the template interface, this limits
      the number of hosts that will be seen as 'network segment
      local' from a PASST viewpoint.
v4: - Made loop independent of attribute order.
    - Ignoring L2 addresses which are not of size ETH_ALEN.
v5: - Changed return value of new function, so caller can know if
      a MAC address really was found.
v6: - Removed warning printout which had ended up in the wrong
      commit.
v8: - Changed to neighbour event subscription model
    - netlink: arp/ndp table subscription
v10:- Updated according to David's latest comments on v8
    - Added functionaly where we initially read current
      state of ARP/NDP tables
v12:- Updates based on feedback from David and Stefano
v13:- Updates based on feedback from David and Stefano
---
 epoll_type.h |   2 +
 netlink.c    | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 netlink.h    |   4 +
 passt.c      |   7 ++
 4 files changed, 224 insertions(+), 4 deletions(-)

diff --git a/epoll_type.h b/epoll_type.h
index 12ac59b..a90ffb6 100644
--- a/epoll_type.h
+++ b/epoll_type.h
@@ -44,6 +44,8 @@ enum epoll_type {
 	EPOLL_TYPE_REPAIR_LISTEN,
 	/* TCP_REPAIR helper socket */
 	EPOLL_TYPE_REPAIR,
+	/* Netlink neighbour subscription socket */
+	EPOLL_TYPE_NL_NEIGH,
 
 	EPOLL_NUM_TYPES,
 };
diff --git a/netlink.c b/netlink.c
index 9fe70f2..99dcb72 100644
--- a/netlink.c
+++ b/netlink.c
@@ -26,6 +26,7 @@
 #include <arpa/inet.h>
 #include <netinet/in.h>
 #include <netinet/if_ether.h>
+#include <net/if_arp.h>
 
 #include <linux/netlink.h>
 #include <linux/rtnetlink.h>
@@ -40,6 +41,10 @@
 #define RTNH_NEXT_AND_DEC(rtnh, attrlen)				\
 	((attrlen) -= RTNH_ALIGN((rtnh)->rtnh_len), RTNH_NEXT(rtnh))
 
+/* Convenience macro borrowed from kernel */
+#define NUD_VALID				\
+	(NUD_PERMANENT | NUD_NOARP | NUD_REACHABLE | NUD_PROBE | NUD_STALE)
+
 /* Netlink expects a buffer of at least 8kiB or the system page size,
  * whichever is larger.  32kiB is recommended for more efficient.
  * Since the largest page size on any remotely common Linux setup is
@@ -50,9 +55,10 @@
 #define NLBUFSIZ 65536
 
 /* Socket in init, in target namespace, sequence (just needs to be monotonic) */
-int nl_sock	= -1;
-int nl_sock_ns	= -1;
-static int nl_seq = 1;
+int nl_sock		 = -1;
+int nl_sock_ns		 = -1;
+static int nl_sock_neigh = -1;
+static int nl_seq	 = 1;
 
 /**
  * nl_sock_init_do() - Set up netlink sockets in init or target namespace
@@ -325,7 +331,6 @@ unsigned int nl_get_ext_if(int s, sa_family_t af)
 
 	if (status < 0)
 		warn("netlink: RTM_GETROUTE failed: %s", strerror_(-status));
-
 	if (defifi) {
 		if (ndef > 1) {
 			info("Multiple default %s routes, picked first",
@@ -1103,3 +1108,205 @@ int nl_link_set_flags(int s, unsigned int ifi,
 
 	return nl_do(s, &req, RTM_NEWLINK, 0, sizeof(req));
 }
+
+/**
+ * nl_neigh_msg_read() - Interpret a neigbour state message from netlink
+ * @c:		Execution context
+ * @nh:	Message to be read
+ */
+static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
+{
+	struct ndmsg *ndm = NLMSG_DATA(nh);
+	struct rtattr *rta = (struct rtattr *)(ndm + 1);
+	size_t na = NLMSG_PAYLOAD(nh, sizeof(*ndm));
+	char ip_str[INET6_ADDRSTRLEN];
+	char mac_str[ETH_ADDRSTRLEN];
+	const uint8_t *lladdr = NULL;
+	const void *dst = NULL;
+	size_t lladdr_len = 0;
+	uint8_t mac[ETH_ALEN];
+	union inany_addr addr;
+	size_t dstlen = 0;
+
+	if (nh->nlmsg_type == NLMSG_DONE)
+		return;
+
+	if (nh->nlmsg_type == NLMSG_ERROR) {
+		warn("nlmsg_type error at msg read");
+		return;
+	}
+
+	if (nh->nlmsg_type != RTM_NEWNEIGH &&
+	    nh->nlmsg_type != RTM_DELNEIGH)
+		return;
+
+	for (; RTA_OK(rta, na); rta = RTA_NEXT(rta, na)) {
+		switch (rta->rta_type) {
+		case NDA_DST:
+			dst = RTA_DATA(rta);
+			dstlen = RTA_PAYLOAD(rta);
+			break;
+		case NDA_LLADDR:
+			lladdr = RTA_DATA(rta);
+			lladdr_len = RTA_PAYLOAD(rta);
+			break;
+		default:
+			break;
+		}
+	}
+
+	if (!dst)
+		return;
+
+
+	if (ndm->ndm_family == AF_INET &&
+	     ndm->ndm_ifindex != c->ifi4)
+		return;
+
+	if (ndm->ndm_family == AF_INET6 &&
+	     ndm->ndm_ifindex != c->ifi6)
+		return;
+
+	if (ndm->ndm_family != AF_INET &&
+	    ndm->ndm_family != AF_INET6)
+		return;
+
+	if (ndm->ndm_family == AF_INET &&
+	    dstlen != sizeof(struct in_addr)) {
+		warn("netlink: wrong address length in AF_INET notification");
+		return;
+	}
+	if (ndm->ndm_family == AF_INET6 &&
+	    dstlen != sizeof(struct in6_addr)) {
+		warn("netlink: wrong address length in AF_INET6 notification");
+		return;
+	}
+	inany_from_af(&addr, ndm->ndm_family, dst);
+	inany_ntop(dst, ip_str, sizeof(ip_str));
+
+	if (nh->nlmsg_type == RTM_DELNEIGH) {
+		trace("neigh table delete: %s", ip_str);
+		return;
+	}
+	if (!(ndm->ndm_state & NUD_VALID)) {
+		trace("neigh table: invalid state for %s", ip_str);
+		return;
+	}
+	if (nh->nlmsg_type != RTM_NEWNEIGH || !lladdr) {
+		warn("netlink: notification with illegal msg for %s", ip_str);
+		return;
+	}
+	if (lladdr_len != ETH_ALEN || ndm->ndm_type != ARPHRD_ETHER)
+		return;
+
+	memcpy(mac, lladdr, ETH_ALEN);
+	eth_ntop(mac, mac_str, sizeof(mac_str));
+	trace("neigh table update: %s / %s", ip_str, mac_str);
+}
+
+/**
+ * nl_neigh_sync() - Read current contents ARP/NDP tables
+ * @c:		Execution context
+ * @proto:	Protocol, AF_INET or AF_INET6
+ * @ifi:	Interface index
+ */
+static void nl_neigh_sync(const struct ctx *c, int proto, int ifi)
+{
+	struct {
+		struct nlmsghdr nlh;
+		struct ndmsg    ndm;
+	} req = {
+		.nlh = {0},
+		.ndm.ndm_family  = proto,
+		.ndm.ndm_ifindex = ifi,
+		.ndm.ndm_state   = 0,
+		.ndm.ndm_flags   = 0,
+		.ndm.ndm_type    = 0
+	};
+	struct nlmsghdr *nh;
+	char buf[NLBUFSIZ];
+	ssize_t status;
+	uint32_t seq;
+
+	seq = nl_send(nl_sock_neigh, &req, RTM_GETNEIGH, NLM_F_DUMP, sizeof(req));
+	nl_foreach_oftype(nh, status, nl_sock_neigh, buf, seq, RTM_NEWNEIGH)
+		nl_neigh_msg_read(c, nh);
+	if (status < 0)
+		warn("netlink: RTM_GETNEIGH failed: %s", strerror_(-status));
+}
+
+/**
+ * nl_neigh_notify_handler() - Non-blocking drain of pending neighbour updates
+ * @c:		Execution context
+ */
+void nl_neigh_notify_handler(const struct ctx *c)
+{
+	char buf[NLBUFSIZ];
+
+	for (;;) {
+		ssize_t n = recv(nl_sock_neigh, buf, sizeof(buf), MSG_DONTWAIT);
+		struct nlmsghdr *nh = (struct nlmsghdr *)buf;
+
+		if (n < 0) {
+			struct nlmsgerr *errmsg;
+
+			if (nh->nlmsg_type != NLMSG_ERROR)
+				return;
+
+			errmsg = (struct nlmsgerr *)NLMSG_DATA(nh);
+			warn("netlink: recv() failed: error %i", errmsg->error);
+			return;
+		}
+		for (; NLMSG_OK(nh, n); nh = NLMSG_NEXT(nh, n))
+			nl_neigh_msg_read(c, nh);
+	}
+}
+
+/**
+ * nl_neigh_notify_init() - Open netlink sock and subscribe to neighbour events
+ * @c:		Execution context
+ *
+ * Return: 0 on success, -1 on failure
+ */
+int nl_neigh_notify_init(const struct ctx *c)
+{
+	union epoll_ref ref = {
+		.type = EPOLL_TYPE_NL_NEIGH
+	};
+	struct epoll_event ev = {
+		.events = EPOLLIN
+	};
+	struct sockaddr_nl addr = {
+		.nl_family = AF_NETLINK,
+		.nl_groups = RTMGRP_NEIGH,
+	};
+
+	if (nl_sock_neigh >= 0) {
+		warn("RTMGRP_NEIGH already exists");
+		return 0;
+	}
+	nl_sock_neigh = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE);
+	if (nl_sock_neigh < 0) {
+		warn("Failed create RTMGRP_NEIGH socket");
+		return -1;
+	}
+	if (bind(nl_sock_neigh, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
+		close(nl_sock_neigh);
+		nl_sock_neigh = -1;
+		warn("Failed bind RTMGRP_NEIGH socket");
+		return -1;
+	}
+
+	ev.data.u64 = ref.u64;
+	if (epoll_ctl(c->epollfd, EPOLL_CTL_ADD, nl_sock_neigh, &ev) == -1) {
+		close(nl_sock_neigh);
+		nl_sock_neigh = -1;
+		warn("Failed do epoll_ctrl() on RTMGRP_NEIGH socket");
+		return -1;
+	}
+
+	nl_neigh_sync(c, AF_INET, c->ifi4);
+	nl_neigh_sync(c, AF_INET6, c->ifi6);
+
+	return 0;
+}
diff --git a/netlink.h b/netlink.h
index b51e99c..8f1e9b9 100644
--- a/netlink.h
+++ b/netlink.h
@@ -17,6 +17,8 @@ int nl_route_dup(int s_src, unsigned int ifi_src,
 		 int s_dst, unsigned int ifi_dst, sa_family_t af);
 int nl_addr_get(int s, unsigned int ifi, sa_family_t af,
 		void *addr, int *prefix_len, void *addr_l);
+bool nl_neigh_mac_get(int s, const union inany_addr *addr, int ifi,
+		      unsigned char *mac);
 int nl_addr_set(int s, unsigned int ifi, sa_family_t af,
 		const void *addr, int prefix_len);
 int nl_addr_get_ll(int s, unsigned int ifi, struct in6_addr *addr);
@@ -28,5 +30,7 @@ int nl_link_set_mac(int s, unsigned int ifi, const void *mac);
 int nl_link_set_mtu(int s, unsigned int ifi, int mtu);
 int nl_link_set_flags(int s, unsigned int ifi,
 		      unsigned int set, unsigned int change);
+int nl_neigh_notify_init(const struct ctx *c);
+void nl_neigh_notify_handler(const struct ctx *c);
 
 #endif /* NETLINK_H */
diff --git a/passt.c b/passt.c
index 31fbb75..e21d6ba 100644
--- a/passt.c
+++ b/passt.c
@@ -53,6 +53,7 @@
 #include "vu_common.h"
 #include "migrate.h"
 #include "repair.h"
+#include "netlink.h"
 
 #define EPOLL_EVENTS		8
 
@@ -79,6 +80,7 @@ char *epoll_type_str[] = {
 	[EPOLL_TYPE_VHOST_KICK]		= "vhost-user kick socket",
 	[EPOLL_TYPE_REPAIR_LISTEN]	= "TCP_REPAIR helper listening socket",
 	[EPOLL_TYPE_REPAIR]		= "TCP_REPAIR helper socket",
+	[EPOLL_TYPE_NL_NEIGH]		= "netlink neighbour subscription socket",
 };
 static_assert(ARRAY_SIZE(epoll_type_str) == EPOLL_NUM_TYPES,
 	      "epoll_type_str[] doesn't match enum epoll_type");
@@ -322,6 +324,8 @@ int main(int argc, char **argv)
 
 	pcap_init(&c);
 
+	nl_neigh_notify_init(&c);
+
 	if (!c.foreground) {
 		if ((devnull_fd = open("/dev/null", O_RDWR | O_CLOEXEC)) < 0)
 			die_perror("Failed to open /dev/null");
@@ -414,6 +418,9 @@ loop:
 		case EPOLL_TYPE_REPAIR:
 			repair_handler(&c, eventmask);
 			break;
+		case EPOLL_TYPE_NL_NEIGH:
+			nl_neigh_notify_handler(&c);
+			break;
 		default:
 			/* Can't happen */
 			ASSERT(0);
-- 
2.50.1

[PATCH v13 02/10] passt: add no_map_gw flag to struct ctx

[Jon Maloy]

Later in this series we will introduce a new initialization function
which needs access to the 'no_map_gw' flag. This flag is currently
a local variable inside the conf() call, where it is too early to
call the new function. The new function needs both ctx->hash_secret
and a valid ctx->fd_tap to be set, neither of which are initialized
at that point.

We therefore add the 'no_map_flag' to struct ctx, so it can be carried
along and used by later functions.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>
---
 conf.c  | 10 +++++-----
 passt.h |  1 +
 2 files changed, 6 insertions(+), 5 deletions(-)

diff --git a/conf.c b/conf.c
index 66b9e63..6143b54 100644
--- a/conf.c
+++ b/conf.c
@@ -1410,7 +1410,7 @@ fail:
  */
 void conf(struct ctx *c, int argc, char **argv)
 {
-	int netns_only = 0, no_map_gw = 0;
+	int netns_only = 0;
 	const struct option options[] = {
 		{"debug",	no_argument,		NULL,		'd' },
 		{"quiet",	no_argument,		NULL,		'q' },
@@ -1442,7 +1442,7 @@ void conf(struct ctx *c, int argc, char **argv)
 		{"no-ra",	no_argument,		&c->no_ra,	1 },
 		{"no-splice",	no_argument,		&c->no_splice,	1 },
 		{"freebind",	no_argument,		&c->freebind,	1 },
-		{"no-map-gw",	no_argument,		&no_map_gw,	1 },
+		{"no-map-gw",	no_argument,		&c->no_map_gw,	1 },
 		{"ipv4-only",	no_argument,		NULL,		'4' },
 		{"ipv6-only",	no_argument,		NULL,		'6' },
 		{"one-off",	no_argument,		NULL,		'1' },
@@ -1656,7 +1656,7 @@ void conf(struct ctx *c, int argc, char **argv)
 			break;
 		case 21:
 			conf_nat(optarg, &c->ip4.map_host_loopback,
-				 &c->ip6.map_host_loopback, &no_map_gw);
+				 &c->ip6.map_host_loopback, &c->no_map_gw);
 			break;
 		case 22:
 			conf_nat(optarg, &c->ip4.map_guest_addr,
@@ -2005,11 +2005,11 @@ void conf(struct ctx *c, int argc, char **argv)
 		}
 	}
 
-	if (c->ifi4 && !no_map_gw &&
+	if (c->ifi4 && !c->no_map_gw &&
 	    IN4_IS_ADDR_UNSPECIFIED(&c->ip4.map_host_loopback))
 		c->ip4.map_host_loopback = c->ip4.guest_gw;
 
-	if (c->ifi6 && !no_map_gw &&
+	if (c->ifi6 && !c->no_map_gw &&
 	    IN6_IS_ADDR_UNSPECIFIED(&c->ip6.map_host_loopback))
 		c->ip6.map_host_loopback = c->ip6.guest_gw;
 
diff --git a/passt.h b/passt.h
index 0075eb4..830bc36 100644
--- a/passt.h
+++ b/passt.h
@@ -310,6 +310,7 @@ struct ctx {
 	int no_ndp;
 	int no_ra;
 	int no_splice;
+	int no_map_gw;
 	int host_lo_to_ns_lo;
 	int freebind;
 
-- 
2.50.1

[PATCH v13 03/10] fwd: Add cache table for ARP/NDP contents

[Jon Maloy]

We add a cache table to keep track of the contents of the kernel ARP
and NDP tables. The table is fed from the just introduced netlink based
neigbour subscription function. The new table eliminates the need for
explicit netlink calls to find a host's MAC address.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>

---
v5: - Moved to earlier in series to reduce rebase conflicts
v6: - Sqashed the hash list commit and the FIFO/LRU queue commit
    - Removed hash lookup. We now only use linear lookup in a
      linked list
    - Eliminated dynamic memory allocation.
    - Ensured there is only one call to clock_gettime()
    - Using MAC_ZERO instead of the previously dedicated definitions
v7: - NOW using MAC_ZERO where needed
    - I am still using linear back-off for empty cache entries. Even
      an incoming, flow-creating packet from a local host gives no
      guarantee that its MAC address is in the ARP table, so we must
      allow for a few new attempts at first possible occasions. Only
      after several failed lookups can we conclude that we probably
      never will succeed. Hence the back-off.
    - Fixed a bug that David inadvertently made me aware of: I only
      intended to set the initial expiry value to MAC_CACHE_RENEWAL
      when an ARP/NDP table lookup was successful.
    - Improved struct and function description comments.
v8: - Total re-design of table, adapting to the new, subscription
      based way of updating it.
v9: - Catering for MAC address change for an existing host.
v10: - Changes according to feedback from David Gibson
v12: - Changes according to feedback from David and Stefano
     - Added dummy entries for loopback and default GW addresses
v13: - Changes according to feedback and discussions with David
       and Stefano
---
 fwd.c     | 222 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
 fwd.h     |   7 ++
 netlink.c |   3 +
 passt.c   |   1 +
 4 files changed, 233 insertions(+)

diff --git a/fwd.c b/fwd.c
index 250cf56..d062ebc 100644
--- a/fwd.c
+++ b/fwd.c
@@ -26,6 +26,7 @@
 #include "passt.h"
 #include "lineread.h"
 #include "flow_table.h"
+#include "netlink.h"
 
 /* Empheral port range: values from RFC 6335 */
 static in_port_t fwd_ephemeral_min = (1 << 15) + (1 << 14);
@@ -33,6 +34,227 @@ static in_port_t fwd_ephemeral_max = NUM_PORTS - 1;
 
 #define PORT_RANGE_SYSCTL	"/proc/sys/net/ipv4/ip_local_port_range"
 
+#define NEIGH_TABLE_SLOTS    1024
+#define NEIGH_TABLE_SIZE     (NEIGH_TABLE_SLOTS / 2)
+static_assert((NEIGH_TABLE_SLOTS & (NEIGH_TABLE_SLOTS - 1)) == 0,
+	      "NEIGH_TABLE_SLOTS must be a power of two");
+
+/**
+ * struct neigh_table_entry - Entry in the ARP/NDP table
+ * @next:	Next entry in slot or free list
+ * @addr:	IP address of represented host
+ * @mac:	MAC address of represented host
+ * @permanent:	Entry cannot be altered or freed by notification
+ */
+struct neigh_table_entry {
+	struct neigh_table_entry *next;
+	union inany_addr addr;
+	uint8_t mac[ETH_ALEN];
+	bool permanent;
+};
+
+/**
+ * struct neigh_table - Cache of ARP/NDP table contents
+ * @entries:	Entries to be plugged into the hash slots when allocated
+ * @slots:	Hash table slots
+ * @free:	Linked list of unused entries
+ */
+struct neigh_table {
+	struct neigh_table_entry entries[NEIGH_TABLE_SIZE];
+	struct neigh_table_entry *slots[NEIGH_TABLE_SLOTS];
+	struct neigh_table_entry *free;
+};
+
+static struct neigh_table neigh_table;
+
+/**
+ * neigh_table_slot() - Hash key to a number within the table range
+ * @c:		Execution context
+ * @key:	The key to be used for the hash
+ *
+ * Return: the resulting hash value
+ */
+static size_t neigh_table_slot(const struct ctx *c,
+			       const union inany_addr *key)
+{
+	struct siphash_state st = SIPHASH_INIT(c->hash_secret);
+	uint32_t i;
+
+	inany_siphash_feed(&st, key);
+	i = siphash_final(&st, sizeof(*key), 0);
+
+	return ((size_t)i) & (NEIGH_TABLE_SIZE - 1);
+}
+
+/**
+ * fwd_neigh_table_find() - Find a MAC table entry
+ * @c:		Execution context
+ * @addr:	Neighbour address to be used as key for the lookup
+ *
+ * Return: the matching entry, if found. Otherwise NULL
+ */
+static struct neigh_table_entry *fwd_neigh_table_find(const struct ctx *c,
+						      const union inany_addr *addr)
+{
+	size_t slot = neigh_table_slot(c, addr);
+	struct neigh_table_entry *e = neigh_table.slots[slot];
+
+	while (e && !inany_equals(&e->addr, addr))
+		e = e->next;
+
+	return e;
+}
+
+/**
+ * fwd_neigh_table_update() - Allocate or update neighbour table entry
+ * @c:		Execution context
+ * @addr:	IP address used to determine insertion slot and store in entry
+ * @mac:	The MAC address associated with the neighbour address
+ * @permanent:	Created entry cannot be altered or freed
+ */
+void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
+			    const uint8_t *mac, bool permanent)
+{
+	struct neigh_table *t = &neigh_table;
+	struct neigh_table_entry *e;
+	union inany_addr daddr;
+	ssize_t slot;
+
+	/* We only add guest-side visible addresses */
+	if (!nat_inbound(c, addr, &daddr))
+		return;
+
+	/* MAC address might change sometimes */
+	e = fwd_neigh_table_find(c, &daddr);
+	if (e) {
+		if (!e->permanent)
+			memcpy(e->mac, mac, ETH_ALEN);
+		return;
+	}
+
+	e = t->free;
+	if (!e) {
+		debug("Failed to allocate neighbour table entry");
+		return;
+	}
+	t->free = e->next;
+	slot = neigh_table_slot(c, &daddr);
+	e->next = t->slots[slot];
+	t->slots[slot] = e;
+
+	memcpy(&e->addr, &daddr, sizeof(daddr));
+	memcpy(e->mac, mac, ETH_ALEN);
+	e->permanent = permanent;
+}
+
+/**
+ * fwd_neigh_table_free() - Remove an entry from a slot and add it to free list
+ * @c:		Execution context
+ * @addr:	IP address used to find the slot for the entry
+ */
+void fwd_neigh_table_free(const struct ctx *c, const union inany_addr *addr)
+{
+	ssize_t slot = neigh_table_slot(c, addr);
+	struct neigh_table *t = &neigh_table;
+	struct neigh_table_entry *e, **prev;
+
+	prev = &t->slots[slot];
+	e = t->slots[slot];
+	while (e && !inany_equals(&e->addr, addr)) {
+		prev = &e->next;
+		e = e->next;
+	}
+
+	if (!e || e->permanent)
+		return;
+
+	*prev = e->next;
+	e->next = t->free;
+	t->free = e;
+	memset(&e->addr, 0, sizeof(*addr));
+	memset(e->mac, 0, ETH_ALEN);
+}
+
+/**
+ * fwd_neigh_mac_get() - Look up MAC address in the ARP/NDP table
+ * @c:		Execution context
+ * @addr:	Neighbour IP address used as lookup key
+ * @mac:	Buffer for returned MAC address
+ */
+void fwd_neigh_mac_get(const struct ctx *c, const union inany_addr *addr,
+		       uint8_t *mac)
+{
+	const struct neigh_table_entry *e = fwd_neigh_table_find(c, addr);
+
+	if (e)
+		memcpy(mac, e->mac, ETH_ALEN);
+	else
+		memcpy(mac, c->our_tap_mac, ETH_ALEN);
+}
+
+/**
+ * fwd_neigh_table_init() - Initialize the neighbour table
+ * @c:		Execution context
+ */
+void fwd_neigh_table_init(const struct ctx *c)
+{
+	union inany_addr mhl = inany_from_v4(c->ip4.map_host_loopback);
+	union inany_addr mga = inany_from_v4(c->ip4.map_guest_addr);
+	union inany_addr ggw = inany_from_v4(c->ip4.guest_gw);
+	struct neigh_table *t = &neigh_table;
+	struct neigh_table_entry *e;
+	int i;
+
+	memset(t, 0, sizeof(*t));
+
+	for (i = 0; i < NEIGH_TABLE_SIZE; i++) {
+		e = &t->entries[i];
+		e->next = t->free;
+		t->free = e;
+	}
+
+	/* Blocker entries to stop events from hosts using these addresses */
+	if (!inany_is_unspecified4(&mhl))
+		fwd_neigh_table_update(c, &mhl, c->our_tap_mac, true);
+
+	if (!inany_is_unspecified4(&ggw) && !c->no_map_gw)
+		fwd_neigh_table_update(c, &ggw, c->our_tap_mac, true);
+
+	if (!inany_is_unspecified4(&mga) && !inany_equals(&mhl, &mga)) {
+		uint8_t mac[ETH_ALEN];
+		int rc;
+
+		rc = nl_link_get_mac(nl_sock, c->ifi4, mac);
+		if (rc < 0) {
+			debug("Couldn't get ip4 MAC addr: %s", strerror_(-rc));
+			memcpy(mac, c->our_tap_mac, ETH_ALEN);
+		}
+		fwd_neigh_table_update(c, &mga, mac, true);
+	}
+
+	mhl = *(union inany_addr *)&c->ip6.map_host_loopback;
+	mga = *(union inany_addr *)&c->ip4.map_guest_addr;
+	ggw = *(union inany_addr *)&c->ip4.guest_gw;
+
+	if (!inany_is_unspecified6(&mhl))
+		fwd_neigh_table_update(c, &mhl, c->our_tap_mac, true);
+
+	if (!inany_is_unspecified6(&ggw) && !c->no_map_gw)
+		fwd_neigh_table_update(c, &ggw, c->our_tap_mac, true);
+
+	if (!inany_is_unspecified6(&mga) && !inany_equals(&mhl, &mga)) {
+		uint8_t mac[ETH_ALEN];
+		int rc;
+
+		rc = nl_link_get_mac(nl_sock, c->ifi6, mac);
+		if (rc < 0) {
+			debug("Couldn't get ip6 MAC addr: %s", strerror_(-rc));
+			memcpy(mac, c->our_tap_mac, ETH_ALEN);
+		}
+		fwd_neigh_table_update(c, &mga, mac, true);
+	}
+}
+
 /** fwd_probe_ephemeral() - Determine what ports this host considers ephemeral
  *
  * Work out what ports the host thinks are emphemeral and record it for later
diff --git a/fwd.h b/fwd.h
index 65c7c96..352f3b5 100644
--- a/fwd.h
+++ b/fwd.h
@@ -56,5 +56,12 @@ uint8_t fwd_nat_from_splice(const struct ctx *c, uint8_t proto,
 			    const struct flowside *ini, struct flowside *tgt);
 uint8_t fwd_nat_from_host(const struct ctx *c, uint8_t proto,
 			  const struct flowside *ini, struct flowside *tgt);
+void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
+			    const uint8_t *mac, bool permanent);
+void fwd_neigh_table_free(const struct ctx *c,
+			  const union inany_addr *addr);
+void fwd_neigh_mac_get(const struct ctx *c, const union inany_addr *addr,
+		       uint8_t *mac);
+void fwd_neigh_table_init(const struct ctx *c);
 
 #endif /* FWD_H */
diff --git a/netlink.c b/netlink.c
index 99dcb72..61796fb 100644
--- a/netlink.c
+++ b/netlink.c
@@ -1186,10 +1186,12 @@ static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
 
 	if (nh->nlmsg_type == RTM_DELNEIGH) {
 		trace("neigh table delete: %s", ip_str);
+		fwd_neigh_table_free(c, &addr);
 		return;
 	}
 	if (!(ndm->ndm_state & NUD_VALID)) {
 		trace("neigh table: invalid state for %s", ip_str);
+		fwd_neigh_table_free(c, &addr);
 		return;
 	}
 	if (nh->nlmsg_type != RTM_NEWNEIGH || !lladdr) {
@@ -1202,6 +1204,7 @@ static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
 	memcpy(mac, lladdr, ETH_ALEN);
 	eth_ntop(mac, mac_str, sizeof(mac_str));
 	trace("neigh table update: %s / %s", ip_str, mac_str);
+	fwd_neigh_table_update(c, &addr, mac, false);
 }
 
 /**
diff --git a/passt.c b/passt.c
index e21d6ba..98fc430 100644
--- a/passt.c
+++ b/passt.c
@@ -324,6 +324,7 @@ int main(int argc, char **argv)
 
 	pcap_init(&c);
 
+	fwd_neigh_table_init(&c);
 	nl_neigh_notify_init(&c);
 
 	if (!c.foreground) {
-- 
2.50.1

[PATCH v13 04/10] arp/ndp: respond with true MAC address of LAN local remote hosts

[Jon Maloy]

When we receive an ARP request or NDP neigbour solicitation over
the tap interface for a host on the local network segment attached
to the template interface, we respond with that host's real MAC
address, if available.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>

---
v3: - Added helper function to find out if a remote ip address is subject
      to NAT. This filters out local host addresses which should be
      presented with the passt/pasta local MAC address 9a:55:9a:55:9a:55 even
      though it is on the local segment.
    - Adapted to the change in nl_mac_get() function, so that we now consider
      only the template interface when checking the ARP/NDP table.
v4: - Moved NAT check into the function nat_outbound() to obtain more
      precise criteria for when NAT is used. We may in theory
      have NAT even if original and translated addresses are equal, and
      we want to catch this case.
    - I chose to keep the wrapper funtion inany_nat(), but moved it to
      fwd.h/fwd.c and renamed it to fwd_inany_nat().
v5: - Simplified criteria for when we do ARP/NDP lookup. Now, we
      just try with the potentially translated address after an
      attempted NAT check.
    - Using the new ARP/NDP cache table instead of using netlink
      directly.
v6: - Fixes after feedback from David:
    - Renamed nat_outbound() to fwd_nat_outbound()
    - Eliminated unnecessary temporary variable in arp.c::arp()
v12: - Some minor changes after feedback from Stefano
v13: - Removed call to nat_outbound() before MAC resolution, as
       we are now handling guest-side visible addresses only.
---
 arp.c   | 8 ++++++--
 inany.c | 1 +
 ndp.c   | 4 ++++
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/arp.c b/arp.c
index ad088b1..b4a686f 100644
--- a/arp.c
+++ b/arp.c
@@ -69,6 +69,7 @@ static bool ignore_arp(const struct ctx *c,
  */
 int arp(const struct ctx *c, struct iov_tail *data)
 {
+	union inany_addr tgt;
 	struct {
 		struct ethhdr eh;
 		struct arphdr ah;
@@ -102,8 +103,11 @@ int arp(const struct ctx *c, struct iov_tail *data)
 	resp.ah.ar_hln = ah->ar_hln;
 	resp.ah.ar_pln = ah->ar_pln;
 
-	/* ARP message */
-	memcpy(resp.am.sha,		c->our_tap_mac,	sizeof(resp.am.sha));
+	/* MAC address to return in ARP message */
+	inany_from_af(&tgt, AF_INET, am->tip);
+	fwd_neigh_mac_get(c, &tgt, resp.am.sha);
+
+	/* Rest of ARP message */
 	memcpy(resp.am.sip,		am->tip,	sizeof(resp.am.sip));
 	memcpy(resp.am.tha,		am->sha,	sizeof(resp.am.tha));
 	memcpy(resp.am.tip,		am->sip,	sizeof(resp.am.tip));
diff --git a/inany.c b/inany.c
index 65a39f9..7680439 100644
--- a/inany.c
+++ b/inany.c
@@ -16,6 +16,7 @@
 #include "ip.h"
 #include "siphash.h"
 #include "inany.h"
+#include "fwd.h"
 
 const union inany_addr inany_loopback4 = INANY_INIT4(IN4ADDR_LOOPBACK_INIT);
 const union inany_addr inany_any4 = INANY_INIT4(IN4ADDR_ANY_INIT);
diff --git a/ndp.c b/ndp.c
index 588b48f..7cc6e48 100644
--- a/ndp.c
+++ b/ndp.c
@@ -196,6 +196,7 @@ static void ndp_send(const struct ctx *c, const struct in6_addr *dst,
 static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
 	    const struct in6_addr *addr)
 {
+	union inany_addr tgt;
 	struct ndp_na na = {
 		.ih = {
 			.icmp6_type		= NA,
@@ -215,6 +216,9 @@ static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
 
 	memcpy(na.target_l2_addr.mac, c->our_tap_mac, ETH_ALEN);
 
+	inany_from_af(&tgt, AF_INET6, addr);
+	fwd_neigh_mac_get(c, &tgt, na.target_l2_addr.mac);
+
 	ndp_send(c, dst, &na, sizeof(na));
 }
 
-- 
2.50.1

[PATCH v13 05/10] arp/ndp: send ARP announcement / unsolicited NA when neigbour entry added

[Jon Maloy]

ARP announcements and unsolicited NAs should be handled with caution
because of the risk of malignant users emitting them to disturb
network communication.

There is however one case we where we know it is legitimate
and safe for us to send out such messages: The one time we switch
from using ctx->own_tap_mac to a MAC address received via the
recently added neigbour subscription function. Later changes to
the MAC address of a host in an existing entry cannot be fully
trusted, so we abstain from doing it in such cases.

When sending this type of messages, we notice that the guest accepts
the update, but shortly later asks for a confirmation in the form of
a regular ARP/NS request. This is responded to with the new value,
and we have exactly the effect we wanted.

This commit adds this functionality.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>

---
v10: -Made small changes based of feedback from David G.
v11: -Moved from 'Gratuitous ARP reply' model to 'ARP Announcement'
      model.
v12: -Excluding loopback and default GW addresses from the ARP/NA
      announcement to be sent to the guest
v13: -Filtering out all announcements of our_tap_mac instead of
      explicitly comparing a set of IP addresses.
     -Changed annc.am.tha in arp_announce() to MAC_ZERO, which is
      'unknown' according to RFC826.
     -Renamed ndp_send_unsolicited_na() to ndp_unsolicited_na()
---
 arp.c | 42 ++++++++++++++++++++++++++++++++++++++++++
 arp.h |  2 ++
 fwd.c | 10 ++++++++++
 ndp.c | 10 ++++++++++
 ndp.h |  1 +
 5 files changed, 65 insertions(+)

diff --git a/arp.c b/arp.c
index b4a686f..18fe1c7 100644
--- a/arp.c
+++ b/arp.c
@@ -150,3 +150,45 @@ void arp_send_init_req(const struct ctx *c)
 	debug("Sending initial ARP request for guest MAC address");
 	tap_send_single(c, &req, sizeof(req));
 }
+
+/**
+ * arp_announce() - Send an ARP announcement for an IPv4 host
+ * @c:		Execution context
+ * @ip:	IPv4 address we announce as owned by @mac
+ * @mac:	MAC address to advertise for @ip
+ */
+void arp_announce(const struct ctx *c, struct in_addr *ip,
+		  const unsigned char *mac)
+{
+	char ip_str[INET_ADDRSTRLEN];
+	char mac_str[ETH_ADDRSTRLEN];
+	struct {
+		struct ethhdr eh;
+		struct arphdr ah;
+		struct arpmsg am;
+	} __attribute__((__packed__)) annc;
+
+	/* Ethernet header */
+	annc.eh.h_proto = htons(ETH_P_ARP);
+	memcpy(annc.eh.h_dest, MAC_BROADCAST, sizeof(annc.eh.h_dest));
+	memcpy(annc.eh.h_source, mac, sizeof(annc.eh.h_source));
+
+	/* ARP header */
+	annc.ah.ar_op = htons(ARPOP_REQUEST);
+	annc.ah.ar_hrd = htons(ARPHRD_ETHER);
+	annc.ah.ar_pro = htons(ETH_P_IP);
+	annc.ah.ar_hln = ETH_ALEN;
+	annc.ah.ar_pln = 4;
+
+	/* ARP message */
+	memcpy(annc.am.sha, mac, sizeof(annc.am.sha));
+	memcpy(annc.am.sip, ip, sizeof(annc.am.sip));
+	memcpy(annc.am.tha, MAC_ZERO, sizeof(annc.am.tha));
+	memcpy(annc.am.tip, ip, sizeof(annc.am.tip));
+
+	inet_ntop(AF_INET, ip, ip_str, sizeof(ip_str));
+	eth_ntop(mac, mac_str, sizeof(mac_str));
+	debug("Announcing ARP for %s / %s", ip_str, mac_str);
+
+	tap_send_single(c, &annc, sizeof(annc));
+}
diff --git a/arp.h b/arp.h
index d5ad0e1..4862e90 100644
--- a/arp.h
+++ b/arp.h
@@ -22,5 +22,7 @@ struct arpmsg {
 
 int arp(const struct ctx *c, struct iov_tail *data);
 void arp_send_init_req(const struct ctx *c);
+void arp_announce(const struct ctx *c, struct in_addr *ip,
+		  const unsigned char *mac);
 
 #endif /* ARP_H */
diff --git a/fwd.c b/fwd.c
index d062ebc..3a61901 100644
--- a/fwd.c
+++ b/fwd.c
@@ -27,6 +27,8 @@
 #include "lineread.h"
 #include "flow_table.h"
 #include "netlink.h"
+#include "arp.h"
+#include "ndp.h"
 
 /* Empheral port range: values from RFC 6335 */
 static in_port_t fwd_ephemeral_min = (1 << 15) + (1 << 14);
@@ -145,6 +147,14 @@ void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
 	memcpy(&e->addr, &daddr, sizeof(daddr));
 	memcpy(e->mac, mac, ETH_ALEN);
 	e->permanent = permanent;
+
+	if (!memcmp(mac, c->our_tap_mac, ETH_ALEN))
+		return;
+
+	if (inany_v4(&daddr))
+		arp_announce(c, inany_v4(&daddr), e->mac);
+	else
+		ndp_unsolicited_na(c, &daddr.a6);
 }
 
 /**
diff --git a/ndp.c b/ndp.c
index 7cc6e48..1e41933 100644
--- a/ndp.c
+++ b/ndp.c
@@ -222,6 +222,16 @@ static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
 	ndp_send(c, dst, &na, sizeof(na));
 }
 
+/**
+ * ndp_unsolicited_na() - Send unsolicited NA
+ * @c:         Execution context
+ * @addr:      IPv6 address to advertise
+ */
+void ndp_unsolicited_na(const struct ctx *c, const struct in6_addr *addr)
+{
+	ndp_na(c, &in6addr_ll_all_nodes, addr);
+}
+
 /**
  * ndp_ra() - Send an NDP Router Advertisement (RA) message
  * @c:		Execution context
diff --git a/ndp.h b/ndp.h
index 781ea86..56b756d 100644
--- a/ndp.h
+++ b/ndp.h
@@ -12,5 +12,6 @@ int ndp(const struct ctx *c, const struct in6_addr *saddr,
 	struct iov_tail *data);
 void ndp_timer(const struct ctx *c, const struct timespec *now);
 void ndp_send_init_req(const struct ctx *c);
+void ndp_unsolicited_na(const struct ctx *c, const struct in6_addr *addr);
 
 #endif /* NDP_H */
-- 
2.50.1

[PATCH v13 06/10] flow: add MAC address of LAN local remote hosts to flow

[Jon Maloy]

When communicating with remote hosts on the local network, some guest
applications want to see the real MAC address of that host instead
of PASST/PASTA's own tap address. The flow_common structure is a
convenient location for storing that address, so we do that in this
commit.

Note that we don´t add actual usage of this address here, that will
be done in later commits.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>

---
v3: - Moved the remote host macaddress from struct flowside to
      struct flow_common. I chose to call it 'omac' as suggested
      by David, although in my understanding the correct name would be
      'emac'. (In general I find the address naming scheme confusing.)
    - Adapted to new signature of function nl_mac_get(), now passing
      it the index of the template interface.
v4: - Renamed flow_commeon->omac to flow_common->tap_omac to make is
      role in the code clearer
v5: - Modified the criteria for ARP/NDP table lookup like in the
      previous commits.
    - Removed the PIF_TAP lookup case, as David suggested, and did
      instead give the flow->tap_omac field a value marking it as
      non-initialized.
    - Calling the cache table instead of netlink for ARP/NDP lookup.
    - Unconditionally using the potentially translated IP address
      in the lookup, instead of only if NAT really was applied.
v6: - Using MAC_ZERO instead of own definitions
v12:- Using MAC_UNDEF (==ff:ff:ff:ff:ff:ff) instead of MAC_ZERO,
      which is a legal MAC address.
v13: - Removed call to nat_outbound() before MAC resolution, as
       we are now handling guest-side visible addresses only.
     - Using tgt->oaddr instead of ini->eaddr as lookup key
       for fwd_neigh_mac_get(), for the same reason as above.
---
 flow.c | 2 ++
 flow.h | 2 ++
 util.h | 2 ++
 3 files changed, 6 insertions(+)

diff --git a/flow.c b/flow.c
index feefda3..a57d7b9 100644
--- a/flow.c
+++ b/flow.c
@@ -449,6 +449,7 @@ struct flowside *flow_target(const struct ctx *c, union flow *flow,
 
 	switch (f->pif[INISIDE]) {
 	case PIF_TAP:
+		memcpy(f->tap_omac, MAC_UNDEF, ETH_ALEN);
 		tgtpif = fwd_nat_from_tap(c, proto, ini, tgt);
 		break;
 
@@ -458,6 +459,7 @@ struct flowside *flow_target(const struct ctx *c, union flow *flow,
 
 	case PIF_HOST:
 		tgtpif = fwd_nat_from_host(c, proto, ini, tgt);
+		fwd_neigh_mac_get(c, &tgt->oaddr, f->tap_omac);
 		break;
 
 	default:
diff --git a/flow.h b/flow.h
index cac618a..f342895 100644
--- a/flow.h
+++ b/flow.h
@@ -177,6 +177,7 @@ int flowside_connect(const struct ctx *c, int s,
  * @type:	Type of packet flow
  * @pif[]:	Interface for each side of the flow
  * @side[]:	Information for each side of the flow
+ * @tap_omac: MAC address of remote endpoint as seen from the guest
  */
 struct flow_common {
 #ifdef __GNUC__
@@ -192,6 +193,7 @@ struct flow_common {
 #endif
 	uint8_t		pif[SIDES];
 	struct flowside	side[SIDES];
+	uint8_t		tap_omac[6];
 };
 
 #define FLOW_INDEX_BITS		17	/* 128k - 1 */
diff --git a/util.h b/util.h
index 22eaac5..6fc8f5d 100644
--- a/util.h
+++ b/util.h
@@ -101,6 +101,8 @@ void abort_with_msg(const char *fmt, ...)
 	((uint8_t [ETH_ALEN]){ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff })
 #define MAC_ZERO		((uint8_t [ETH_ALEN]){ 0 })
 #define MAC_IS_ZERO(addr)	(!memcmp((addr), MAC_ZERO, ETH_ALEN))
+#define MAC_UNDEF		MAC_BROADCAST
+#define MAC_IS_UNDEF(addr)	(!memcmp((addr), MAC_UNDEF, ETH_ALEN))
 
 #ifndef __bswap_constant_16
 #define __bswap_constant_16(x)						\
-- 
2.50.1

[PATCH v13 07/10] udp: forward external source MAC address through tap interface

[Jon Maloy]

We forward the incoming MAC address through the tap interface when
receiving incoming packets from network local hosts.

This is a part of the solution to bug
https://bugs.passt.top/show_bug.cgi?id=120

Signed-off-by: Jon Maloy <jmaloy@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

---
v3: - Adapted to the move of external MAC address from struct flowside
      to struct flow_common
v4: - Changed signature of udp_tap_prepare() to take a MAC address
      instead of a flow.
    - Eliminated initialization of MAC source address in all frames,
      since those now are set per send occasion anyway.
v5: - Added lookup in ARP/NDP table on incoming messages in
      case flow->tap_omac wasn't initialized at flow creation,
      i.e., the flow was initiated from the guest.
v6: - Using MAC_IS_ZERO
v12: - Using MAC_IS_UNDEF() instead of MAC_IS_ZERO()
v13: - No changes
---
 passt.c |  2 +-
 udp.c   | 45 +++++++++++++++++++++++++--------------------
 udp.h   |  2 +-
 3 files changed, 27 insertions(+), 22 deletions(-)

diff --git a/passt.c b/passt.c
index 98fc430..52d70db 100644
--- a/passt.c
+++ b/passt.c
@@ -164,7 +164,7 @@ static void timer_init(struct ctx *c, const struct timespec *now)
 void proto_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s)
 {
 	tcp_update_l2_buf(eth_d, eth_s);
-	udp_update_l2_buf(eth_d, eth_s);
+	udp_update_l2_buf(eth_d);
 }
 
 /**
diff --git a/udp.c b/udp.c
index 86585b7..3acc401 100644
--- a/udp.c
+++ b/udp.c
@@ -133,11 +133,8 @@ static int udp_splice_init[IP_VERSIONS][NUM_PORTS];
 /* UDP header and data for inbound messages */
 static struct udp_payload_t udp_payload[UDP_MAX_FRAMES];
 
-/* Ethernet header for IPv4 frames */
-static struct ethhdr udp4_eth_hdr;
-
-/* Ethernet header for IPv6 frames */
-static struct ethhdr udp6_eth_hdr;
+/* Ethernet headers for IPv4 and IPv6 frames */
+static struct ethhdr udp_eth_hdr[UDP_MAX_FRAMES];
 
 /**
  * struct udp_meta_t - Pre-cooked headers for UDP packets
@@ -210,12 +207,13 @@ void udp_portmap_clear(void)
 /**
  * udp_update_l2_buf() - Update L2 buffers with Ethernet and IPv4 addresses
  * @eth_d:	Ethernet destination address, NULL if unchanged
- * @eth_s:	Ethernet source address, NULL if unchanged
  */
-void udp_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s)
+void udp_update_l2_buf(const unsigned char *eth_d)
 {
-	eth_update_mac(&udp4_eth_hdr, eth_d, eth_s);
-	eth_update_mac(&udp6_eth_hdr, eth_d, eth_s);
+	int i;
+
+	for (i = 0; i < UDP_MAX_FRAMES; i++)
+		eth_update_mac(&udp_eth_hdr[i], eth_d, NULL);
 }
 
 /**
@@ -238,6 +236,7 @@ static void udp_iov_init_one(const struct ctx *c, size_t i)
 
 	*siov = IOV_OF_LVALUE(payload->data);
 
+	tiov[UDP_IOV_ETH] = IOV_OF_LVALUE(udp_eth_hdr[i]);
 	tiov[UDP_IOV_TAP] = tap_hdr_iov(c, &meta->taph);
 	tiov[UDP_IOV_PAYLOAD].iov_base = payload;
 
@@ -253,9 +252,6 @@ static void udp_iov_init(const struct ctx *c)
 {
 	size_t i;
 
-	udp4_eth_hdr.h_proto = htons_constant(ETH_P_IP);
-	udp6_eth_hdr.h_proto = htons_constant(ETH_P_IPV6);
-
 	for (i = 0; i < UDP_MAX_FRAMES; i++)
 		udp_iov_init_one(c, i);
 }
@@ -352,31 +348,34 @@ size_t udp_update_hdr6(struct ipv6hdr *ip6h, struct udp_payload_t *bp,
  * udp_tap_prepare() - Convert one datagram into a tap frame
  * @mmh:	Receiving mmsghdr array
  * @idx:	Index of the datagram to prepare
+ * @tap_omac:	MAC address of remote endpoint as seen from the guest
  * @toside:	Flowside for destination side
  * @no_udp_csum: Do not set UDP checksum
  */
 static void udp_tap_prepare(const struct mmsghdr *mmh,
-			    unsigned idx, const struct flowside *toside,
+			    unsigned int idx,
+			    const uint8_t *tap_omac,
+			    const struct flowside *toside,
 			    bool no_udp_csum)
 {
 	struct iovec (*tap_iov)[UDP_NUM_IOVS] = &udp_l2_iov[idx];
+	struct ethhdr *eh = (*tap_iov)[UDP_IOV_ETH].iov_base;
 	struct udp_payload_t *bp = &udp_payload[idx];
 	struct udp_meta_t *bm = &udp_meta[idx];
 	size_t l4len;
 
+	eth_update_mac(eh, NULL, tap_omac);
 	if (!inany_v4(&toside->eaddr) || !inany_v4(&toside->oaddr)) {
 		l4len = udp_update_hdr6(&bm->ip6h, bp, toside,
 					mmh[idx].msg_len, no_udp_csum);
-		tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip6h) +
-			       sizeof(udp6_eth_hdr));
-		(*tap_iov)[UDP_IOV_ETH] = IOV_OF_LVALUE(udp6_eth_hdr);
+		tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip6h) + ETH_HLEN);
+		eh->h_proto = htons_constant(ETH_P_IPV6);
 		(*tap_iov)[UDP_IOV_IP] = IOV_OF_LVALUE(bm->ip6h);
 	} else {
 		l4len = udp_update_hdr4(&bm->ip4h, bp, toside,
 					mmh[idx].msg_len, no_udp_csum);
-		tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip4h) +
-			       sizeof(udp4_eth_hdr));
-		(*tap_iov)[UDP_IOV_ETH] = IOV_OF_LVALUE(udp4_eth_hdr);
+		tap_hdr_update(&bm->taph, l4len + sizeof(bm->ip4h) + ETH_HLEN);
+		eh->h_proto = htons_constant(ETH_P_IP);
 		(*tap_iov)[UDP_IOV_IP] = IOV_OF_LVALUE(bm->ip4h);
 	}
 	(*tap_iov)[UDP_IOV_PAYLOAD].iov_len = l4len;
@@ -801,13 +800,19 @@ static void udp_buf_sock_to_tap(const struct ctx *c, int s, int n,
 				flow_sidx_t tosidx)
 {
 	const struct flowside *toside = flowside_at_sidx(tosidx);
+	struct udp_flow *uflow = udp_at_sidx(tosidx);
+	uint8_t *omac = uflow->f.tap_omac;
 	int i;
 
 	if ((n = udp_sock_recv(c, s, udp_mh_recv, n)) <= 0)
 		return;
 
+	/* Find if neighbour table has a recorded MAC address */
+	if (MAC_IS_UNDEF(omac))
+		fwd_neigh_mac_get(c, &toside->oaddr, omac);
+
 	for (i = 0; i < n; i++)
-		udp_tap_prepare(udp_mh_recv, i, toside, false);
+		udp_tap_prepare(udp_mh_recv, i, omac, toside, false);
 
 	tap_send_frames(c, &udp_l2_iov[0][0], UDP_NUM_IOVS, n);
 }
diff --git a/udp.h b/udp.h
index 8f8531a..dd6e5ad 100644
--- a/udp.h
+++ b/udp.h
@@ -21,7 +21,7 @@ int udp_sock_init(const struct ctx *c, int ns, const union inany_addr *addr,
 		  const char *ifname, in_port_t port);
 int udp_init(struct ctx *c);
 void udp_timer(struct ctx *c, const struct timespec *now);
-void udp_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s);
+void udp_update_l2_buf(const unsigned char *eth_d);
 
 /**
  * union udp_listen_epoll_ref - epoll reference for "listening" UDP sockets
-- 
2.50.1

[PATCH v13 08/10] tcp: forward external source MAC address through tap interface

[Jon Maloy]

We forward the incoming mac address through the tap interface when
receiving incoming packets from network local hosts.

This is a part of the solution to bug
https://bugs.passt.top/show_bug.cgi?id=120

Signed-off-by: Jon Maloy <jmaloy@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

---
v3: - Adapted to the signature change in nl_mac_get() function, so that
      we now consider only the template interface when checking the
      ARP/NDP table.
v4: - Adapted to previous name changes in this series
v5: - Added lookup in ARP/NDP cache and/or table on incoming messages
      in  case flow->tap_omac wasn't initialized at flow creation,
      i.e., the flow was initiated from the guest.
v6: - Using MAC_IS_ZERO() instead of mac_undefined()
    - I ignored David's suggestion to move the test for
      undefined flow->tap_omac to tcp_{buf,vu}_data_from_sock(), at
      least for now. It looks like a sub-optimization to first have to
      look up the flow instance in these calls, and then do the
      test for MAC_IS_ZERO(), even though we might in theory end up
      with fewer such calls (how often?).
v12: - Using MAC_IS_UNDEF() instead of MAC_IS_ZERO()
     - Minor update after feedback from Stefano
v13: - Unchanged
---
 passt.c        |  7 +++----
 passt.h        |  3 +--
 pasta.c        |  2 +-
 tap.c          |  2 +-
 tcp.c          | 14 ++++++++++++--
 tcp.h          |  2 +-
 tcp_buf.c      | 37 +++++++++++++++++--------------------
 tcp_internal.h |  4 ++--
 tcp_vu.c       |  5 ++---
 9 files changed, 40 insertions(+), 36 deletions(-)

diff --git a/passt.c b/passt.c
index 52d70db..a9770b3 100644
--- a/passt.c
+++ b/passt.c
@@ -159,11 +159,10 @@ static void timer_init(struct ctx *c, const struct timespec *now)
 /**
  * proto_update_l2_buf() - Update scatter-gather L2 buffers in protocol handlers
  * @eth_d:	Ethernet destination address, NULL if unchanged
- * @eth_s:	Ethernet source address, NULL if unchanged
  */
-void proto_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s)
+void proto_update_l2_buf(const unsigned char *eth_d)
 {
-	tcp_update_l2_buf(eth_d, eth_s);
+	tcp_update_l2_buf(eth_d);
 	udp_update_l2_buf(eth_d);
 }
 
@@ -314,7 +313,7 @@ int main(int argc, char **argv)
 	if ((!c.no_udp && udp_init(&c)) || (!c.no_tcp && tcp_init(&c)))
 		_exit(EXIT_FAILURE);
 
-	proto_update_l2_buf(c.guest_mac, c.our_tap_mac);
+	proto_update_l2_buf(c.guest_mac);
 
 	if (c.ifi4 && !c.no_dhcp)
 		dhcp_init();
diff --git a/passt.h b/passt.h
index 830bc36..79041c0 100644
--- a/passt.h
+++ b/passt.h
@@ -327,7 +327,6 @@ struct ctx {
 	bool migrate_exit;
 };
 
-void proto_update_l2_buf(const unsigned char *eth_d,
-			 const unsigned char *eth_s);
+void proto_update_l2_buf(const unsigned char *eth_d);
 
 #endif /* PASST_H */
diff --git a/pasta.c b/pasta.c
index 687406b..a42cfd8 100644
--- a/pasta.c
+++ b/pasta.c
@@ -411,7 +411,7 @@ void pasta_ns_conf(struct ctx *c)
 		}
 	}
 
-	proto_update_l2_buf(c->guest_mac, NULL);
+	proto_update_l2_buf(c->guest_mac);
 }
 
 /**
diff --git a/tap.c b/tap.c
index 95d309b..6c4809a 100644
--- a/tap.c
+++ b/tap.c
@@ -1104,7 +1104,7 @@ void tap_add_packet(struct ctx *c, struct iov_tail *data,
 		memcpy(c->guest_mac, eh->h_source, ETH_ALEN);
 		debug("New guest MAC address observed: %s",
 		      eth_ntop(c->guest_mac, bufmac, sizeof(bufmac)));
-		proto_update_l2_buf(c->guest_mac, NULL);
+		proto_update_l2_buf(c->guest_mac);
 	}
 
 	switch (ntohs(eh->h_proto)) {
diff --git a/tcp.c b/tcp.c
index 0f9e9b3..eb8c682 100644
--- a/tcp.c
+++ b/tcp.c
@@ -919,8 +919,10 @@ static void tcp_fill_header(struct tcphdr *th,
 
 /**
  * tcp_fill_headers() - Fill 802.3, IP, TCP headers
+ * @c:			Execution context
  * @conn:		Connection pointer
  * @taph:		tap backend specific header
+ * @eh:		Pointer to Ethernet header
  * @ip4h:		Pointer to IPv4 header, or NULL
  * @ip6h:		Pointer to IPv6 header, or NULL
  * @th:			Pointer to TCP header
@@ -929,14 +931,15 @@ static void tcp_fill_header(struct tcphdr *th,
  * @seq:		Sequence number for this segment
  * @no_tcp_csum:	Do not set TCP checksum
  */
-void tcp_fill_headers(const struct tcp_tap_conn *conn,
-		      struct tap_hdr *taph,
+void tcp_fill_headers(const struct ctx *c, struct tcp_tap_conn *conn,
+		      struct tap_hdr *taph, struct ethhdr *eh,
 		      struct iphdr *ip4h, struct ipv6hdr *ip6h,
 		      struct tcphdr *th, struct iov_tail *payload,
 		      const uint16_t *ip4_check, uint32_t seq, bool no_tcp_csum)
 {
 	const struct flowside *tapside = TAPFLOW(conn);
 	size_t l4len = iov_tail_size(payload) + sizeof(*th);
+	uint8_t *omac = conn->f.tap_omac;
 	size_t l3len = l4len;
 	uint32_t psum = 0;
 
@@ -962,6 +965,7 @@ void tcp_fill_headers(const struct tcp_tap_conn *conn,
 			psum = proto_ipv4_header_psum(l4len, IPPROTO_TCP,
 						      *src4, *dst4);
 		}
+		eh->h_proto = htons_constant(ETH_P_IP);
 	}
 
 	if (ip6h) {
@@ -982,8 +986,14 @@ void tcp_fill_headers(const struct tcp_tap_conn *conn,
 						      &ip6h->saddr,
 						      &ip6h->daddr);
 		}
+		eh->h_proto = htons_constant(ETH_P_IPV6);
 	}
 
+	/* Find if neighbour table has a recorded MAC address */
+	if (MAC_IS_UNDEF(omac))
+		fwd_neigh_mac_get(c, &tapside->oaddr, omac);
+	eth_update_mac(eh, NULL, omac);
+
 	tcp_fill_header(th, conn, seq);
 
 	if (no_tcp_csum)
diff --git a/tcp.h b/tcp.h
index 234a803..c1b8385 100644
--- a/tcp.h
+++ b/tcp.h
@@ -24,7 +24,7 @@ int tcp_init(struct ctx *c);
 void tcp_timer(struct ctx *c, const struct timespec *now);
 void tcp_defer_handler(struct ctx *c);
 
-void tcp_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s);
+void tcp_update_l2_buf(const unsigned char *eth_d);
 
 extern bool peek_offset_cap;
 
diff --git a/tcp_buf.c b/tcp_buf.c
index cc106bc..2058225 100644
--- a/tcp_buf.c
+++ b/tcp_buf.c
@@ -40,8 +40,7 @@
 /* Static buffers */
 
 /* Ethernet header for IPv4 and IPv6 frames */
-static struct ethhdr		tcp4_eth_src;
-static struct ethhdr		tcp6_eth_src;
+static struct ethhdr		tcp_eth_hdr[TCP_FRAMES_MEM];
 
 static struct tap_hdr		tcp_payload_tap_hdr[TCP_FRAMES_MEM];
 
@@ -67,12 +66,13 @@ static struct iovec	tcp_l2_iov[TCP_FRAMES_MEM][TCP_NUM_IOVS];
 /**
  * tcp_update_l2_buf() - Update Ethernet header buffers with addresses
  * @eth_d:	Ethernet destination address, NULL if unchanged
- * @eth_s:	Ethernet source address, NULL if unchanged
  */
-void tcp_update_l2_buf(const unsigned char *eth_d, const unsigned char *eth_s)
+void tcp_update_l2_buf(const unsigned char *eth_d)
 {
-	eth_update_mac(&tcp4_eth_src, eth_d, eth_s);
-	eth_update_mac(&tcp6_eth_src, eth_d, eth_s);
+	int i;
+
+	for (i = 0; i < TCP_FRAMES_MEM; i++)
+		eth_update_mac(&tcp_eth_hdr[i], eth_d, NULL);
 }
 
 /**
@@ -85,9 +85,6 @@ void tcp_sock_iov_init(const struct ctx *c)
 	struct iphdr iph = L2_BUF_IP4_INIT(IPPROTO_TCP);
 	int i;
 
-	tcp6_eth_src.h_proto = htons_constant(ETH_P_IPV6);
-	tcp4_eth_src.h_proto = htons_constant(ETH_P_IP);
-
 	for (i = 0; i < ARRAY_SIZE(tcp_payload); i++) {
 		tcp6_payload_ip[i] = ip6;
 		tcp4_payload_ip[i] = iph;
@@ -149,13 +146,15 @@ void tcp_payload_flush(const struct ctx *c)
 
 /**
  * tcp_l2_buf_fill_headers() - Fill 802.3, IP, TCP headers in pre-cooked buffers
+ * @c:		Execution context
  * @conn:	Connection pointer
  * @iov:	Pointer to an array of iovec of TCP pre-cooked buffers
  * @check:	Checksum, if already known
  * @seq:	Sequence number for this segment
  * @no_tcp_csum: Do not set TCP checksum
  */
-static void tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn,
+static void tcp_l2_buf_fill_headers(const struct ctx *c,
+				    struct tcp_tap_conn *conn,
 				    struct iovec *iov, const uint16_t *check,
 				    uint32_t seq, bool no_tcp_csum)
 {
@@ -164,6 +163,7 @@ static void tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn,
 	struct tap_hdr *taph = iov[TCP_IOV_TAP].iov_base;
 	const struct flowside *tapside = TAPFLOW(conn);
 	const struct in_addr *a4 = inany_v4(&tapside->oaddr);
+	struct ethhdr *eh = iov[TCP_IOV_ETH].iov_base;
 	struct ipv6hdr *ip6h = NULL;
 	struct iphdr *ip4h = NULL;
 
@@ -172,7 +172,7 @@ static void tcp_l2_buf_fill_headers(const struct tcp_tap_conn *conn,
 	else
 		ip6h = iov[TCP_IOV_IP].iov_base;
 
-	tcp_fill_headers(conn, taph, ip4h, ip6h, th, &tail,
+	tcp_fill_headers(c, conn, taph, eh, ip4h, ip6h, th, &tail,
 			 check, seq, no_tcp_csum);
 }
 
@@ -194,14 +194,12 @@ int tcp_buf_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags)
 	int ret;
 
 	iov = tcp_l2_iov[tcp_payload_used];
-	if (CONN_V4(conn)) {
+	if (CONN_V4(conn))
 		iov[TCP_IOV_IP] = IOV_OF_LVALUE(tcp4_payload_ip[tcp_payload_used]);
-		iov[TCP_IOV_ETH].iov_base = &tcp4_eth_src;
-	} else {
+	else
 		iov[TCP_IOV_IP] = IOV_OF_LVALUE(tcp6_payload_ip[tcp_payload_used]);
-		iov[TCP_IOV_ETH].iov_base = &tcp6_eth_src;
-	}
 
+	iov[TCP_IOV_ETH] = IOV_OF_LVALUE(tcp_eth_hdr[tcp_payload_used]);
 	payload = iov[TCP_IOV_PAYLOAD].iov_base;
 	seq = conn->seq_to_tap;
 	ret = tcp_prepare_flags(c, conn, flags, &payload->th,
@@ -212,7 +210,7 @@ int tcp_buf_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags)
 	tcp_frame_conns[tcp_payload_used++] = conn;
 	l4len = optlen + sizeof(struct tcphdr);
 	iov[TCP_IOV_PAYLOAD].iov_len = l4len;
-	tcp_l2_buf_fill_headers(conn, iov, NULL, seq, false);
+	tcp_l2_buf_fill_headers(c, conn, iov, NULL, seq, false);
 
 	if (flags & DUP_ACK) {
 		struct iovec *dup_iov = tcp_l2_iov[tcp_payload_used];
@@ -260,11 +258,10 @@ static void tcp_data_to_tap(const struct ctx *c, struct tcp_tap_conn *conn,
 			check = &iph->check;
 		}
 		iov[TCP_IOV_IP] = IOV_OF_LVALUE(tcp4_payload_ip[tcp_payload_used]);
-		iov[TCP_IOV_ETH].iov_base = &tcp4_eth_src;
 	} else if (CONN_V6(conn)) {
 		iov[TCP_IOV_IP] = IOV_OF_LVALUE(tcp6_payload_ip[tcp_payload_used]);
-		iov[TCP_IOV_ETH].iov_base = &tcp6_eth_src;
 	}
+	iov[TCP_IOV_ETH].iov_base = &tcp_eth_hdr[tcp_payload_used];
 	payload = iov[TCP_IOV_PAYLOAD].iov_base;
 	payload->th.th_off = sizeof(struct tcphdr) / 4;
 	payload->th.th_x2 = 0;
@@ -272,7 +269,7 @@ static void tcp_data_to_tap(const struct ctx *c, struct tcp_tap_conn *conn,
 	payload->th.ack = 1;
 	payload->th.psh = push;
 	iov[TCP_IOV_PAYLOAD].iov_len = dlen + sizeof(struct tcphdr);
-	tcp_l2_buf_fill_headers(conn, iov, check, seq, false);
+	tcp_l2_buf_fill_headers(c, conn, iov, check, seq, false);
 	if (++tcp_payload_used > TCP_FRAMES_MEM - 1)
 		tcp_payload_flush(c);
 }
diff --git a/tcp_internal.h b/tcp_internal.h
index 5cb6cba..f55025c 100644
--- a/tcp_internal.h
+++ b/tcp_internal.h
@@ -174,8 +174,8 @@ void tcp_rst_do(const struct ctx *c, struct tcp_tap_conn *conn);
 
 struct tcp_info_linux;
 
-void tcp_fill_headers(const struct tcp_tap_conn *conn,
-		      struct tap_hdr *taph,
+void tcp_fill_headers(const struct ctx *c, struct tcp_tap_conn *conn,
+		      struct tap_hdr *taph, struct ethhdr *eh,
 		      struct iphdr *ip4h, struct ipv6hdr *ip6h,
 		      struct tcphdr *th, struct iov_tail *payload,
 		      const uint16_t *ip4_check, uint32_t seq, bool no_tcp_csum);
diff --git a/tcp_vu.c b/tcp_vu.c
index 3ec3538..1c81ce3 100644
--- a/tcp_vu.c
+++ b/tcp_vu.c
@@ -135,7 +135,7 @@ int tcp_vu_send_flag(const struct ctx *c, struct tcp_tap_conn *conn, int flags)
 	flags_elem[0].in_sg[0].iov_len = hdrlen + optlen;
 	payload = IOV_TAIL(flags_elem[0].in_sg, 1, hdrlen);
 
-	tcp_fill_headers(conn, NULL, ip4h, ip6h, th, &payload,
+	tcp_fill_headers(c, conn, NULL, eh, ip4h, ip6h, th, &payload,
 			 NULL, seq, !*c->pcap);
 
 	if (*c->pcap) {
@@ -308,7 +308,6 @@ static void tcp_vu_prepare(const struct ctx *c, struct tcp_tap_conn *conn,
 	eh = vu_eth(base);
 
 	memcpy(eh->h_dest, c->guest_mac, sizeof(eh->h_dest));
-	memcpy(eh->h_source, c->our_tap_mac, sizeof(eh->h_source));
 
 	/* initialize header */
 
@@ -332,7 +331,7 @@ static void tcp_vu_prepare(const struct ctx *c, struct tcp_tap_conn *conn,
 	th->ack = 1;
 	th->psh = push;
 
-	tcp_fill_headers(conn, NULL, ip4h, ip6h, th, &payload,
+	tcp_fill_headers(c, conn, NULL, eh, ip4h, ip6h, th, &payload,
 			 *check, conn->seq_to_tap, no_tcp_csum);
 	if (ip4h)
 		*check = &ip4h->check;
-- 
2.50.1

[PATCH v13 09/10] tap: change signature of function tap_push_l2h()

[Jon Maloy]

In the next commit it must be possible for the callers of function
tap_push_l2h() to specify which source MAC address should be
added to the ethernet header sent over the tap interface. As a
preparation, we now add a new argument to that function, still
without any logical changes.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

---
v3: -Improved comment for src_mac argument, as suggested by Stefano.
v4: -Re-added comment in tap.c and remove redundant argument check
     in tap_push_l2()
v5: -Ensured that MAC argument to tap_push_l2h() never is NULL.
v6: -Clarification of comment about ARP lookup
v12: - Minor updates after feedback from Stefano
v13: - No changes
---
 tap.c | 16 +++++++++-------
 tap.h |  3 ++-
 tcp.c |  6 ++++--
 3 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/tap.c b/tap.c
index 6c4809a..37a7092 100644
--- a/tap.c
+++ b/tap.c
@@ -171,17 +171,19 @@ const struct in6_addr *tap_ip6_daddr(const struct ctx *c,
  * tap_push_l2h() - Build an L2 header for an inbound packet
  * @c:		Execution context
  * @buf:	Buffer address at which to generate header
+ * @src_mac:	MAC address to be used as source for message
  * @proto:	Ethernet protocol number for L3
  *
  * Return: pointer at which to write the packet's payload
  */
-void *tap_push_l2h(const struct ctx *c, void *buf, uint16_t proto)
+void *tap_push_l2h(const struct ctx *c, void *buf,
+		   const void *src_mac, uint16_t proto)
 {
 	struct ethhdr *eh = (struct ethhdr *)buf;
 
-	/* TODO: ARP table lookup */
+	/* TODO: ARP lookup on tap side */
 	memcpy(eh->h_dest, c->guest_mac, ETH_ALEN);
-	memcpy(eh->h_source, c->our_tap_mac, ETH_ALEN);
+	memcpy(eh->h_source, src_mac, ETH_ALEN);
 	eh->h_proto = ntohs(proto);
 	return eh + 1;
 }
@@ -261,7 +263,7 @@ void tap_udp4_send(const struct ctx *c, struct in_addr src, in_port_t sport,
 {
 	size_t l4len = dlen + sizeof(struct udphdr);
 	char buf[USHRT_MAX];
-	struct iphdr *ip4h = tap_push_l2h(c, buf, ETH_P_IP);
+	struct iphdr *ip4h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IP);
 	struct udphdr *uh = tap_push_ip4h(ip4h, src, dst, l4len, IPPROTO_UDP);
 	char *data = tap_push_uh4(uh, src, sport, dst, dport, in, dlen);
 
@@ -281,7 +283,7 @@ void tap_icmp4_send(const struct ctx *c, struct in_addr src, struct in_addr dst,
 		    const void *in, size_t l4len)
 {
 	char buf[USHRT_MAX];
-	struct iphdr *ip4h = tap_push_l2h(c, buf, ETH_P_IP);
+	struct iphdr *ip4h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IP);
 	struct icmphdr *icmp4h = tap_push_ip4h(ip4h, src, dst,
 					       l4len, IPPROTO_ICMP);
 
@@ -367,7 +369,7 @@ void tap_udp6_send(const struct ctx *c,
 {
 	size_t l4len = dlen + sizeof(struct udphdr);
 	char buf[USHRT_MAX];
-	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, ETH_P_IPV6);
+	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IPV6);
 	struct udphdr *uh = tap_push_ip6h(ip6h, src, dst,
 					  l4len, IPPROTO_UDP, flow);
 	char *data = tap_push_uh6(uh, src, sport, dst, dport, in, dlen);
@@ -389,7 +391,7 @@ void tap_icmp6_send(const struct ctx *c,
 		    const void *in, size_t l4len)
 {
 	char buf[USHRT_MAX];
-	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, ETH_P_IPV6);
+	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IPV6);
 	struct icmp6hdr *icmp6h = tap_push_ip6h(ip6h, src, dst, l4len,
 						IPPROTO_ICMPV6, 0);
 
diff --git a/tap.h b/tap.h
index 21db4d2..02f7761 100644
--- a/tap.h
+++ b/tap.h
@@ -70,7 +70,8 @@ static inline void tap_hdr_update(struct tap_hdr *thdr, size_t l2len)
 }
 
 unsigned long tap_l2_max_len(const struct ctx *c);
-void *tap_push_l2h(const struct ctx *c, void *buf, uint16_t proto);
+void *tap_push_l2h(const struct ctx *c, void *buf,
+		   const void *src_mac, uint16_t proto);
 void *tap_push_ip4h(struct iphdr *ip4h, struct in_addr src,
 		     struct in_addr dst, size_t l4len, uint8_t proto);
 void *tap_push_uh4(struct udphdr *uh, struct in_addr src, in_port_t sport,
diff --git a/tcp.c b/tcp.c
index eb8c682..27c38b5 100644
--- a/tcp.c
+++ b/tcp.c
@@ -2004,7 +2004,8 @@ static void tcp_rst_no_conn(const struct ctx *c, int af,
 		return;
 
 	if (af == AF_INET) {
-		struct iphdr *ip4h = tap_push_l2h(c, buf, ETH_P_IP);
+		struct iphdr *ip4h = tap_push_l2h(c, buf, c->our_tap_mac,
+						  ETH_P_IP);
 		const struct in_addr *rst_src = daddr;
 		const struct in_addr *rst_dst = saddr;
 
@@ -2014,7 +2015,8 @@ static void tcp_rst_no_conn(const struct ctx *c, int af,
 					      *rst_src, *rst_dst);
 
 	} else {
-		struct ipv6hdr *ip6h = tap_push_l2h(c, buf, ETH_P_IPV6);
+		struct ipv6hdr *ip6h = tap_push_l2h(c, buf, c->our_tap_mac,
+						    ETH_P_IPV6);
 		const struct in6_addr *rst_src = daddr;
 		const struct in6_addr *rst_dst = saddr;
 
-- 
2.50.1

[PATCH v13 10/10] icmp: let icmp use mac address from flowside structure

[Jon Maloy]

Even ICMP needs to be updated to use the external MAC address instead
of just the own tap address when applicable. We do that here.

Signed-off-by: Jon Maloy <jmaloy@redhat.com>
Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

---
v3: - Adapted to the move of external MAC address from struct flowside
      to struct flow_common
v4: - Adapted to name changes in previous commits in this series
v5: - Added conditional lookup in ARP/NDP if the flow's tap_omac is
      undefined
v6: - Looking up MAC of ICMP generating node in udp_send_tap_icmp4/6()
      when available, instead trusting the contents of flow->tap_omac.
v12: - Using MAC_IS_UNDEF() instead of MAC_IS_ZERO()
     - Comment update after feedback from Stefano
v13: - No changes
---
 icmp.c |  8 ++++++--
 ndp.c  |  2 +-
 tap.c  | 10 ++++++----
 tap.h  |  4 ++--
 udp.c  | 12 ++++++++++--
 5 files changed, 25 insertions(+), 11 deletions(-)

diff --git a/icmp.c b/icmp.c
index 6dffafb..93b394a 100644
--- a/icmp.c
+++ b/icmp.c
@@ -125,17 +125,21 @@ void icmp_sock_handler(const struct ctx *c, union epoll_ref ref)
 	flow_dbg(pingf, "echo reply to tap, ID: %"PRIu16", seq: %"PRIu16,
 		 ini->eport, seq);
 
+	/* Find if neighbour table has a recorded MAC address */
+	if (MAC_IS_UNDEF(pingf->f.tap_omac))
+		fwd_neigh_mac_get(c, &ini->oaddr, pingf->f.tap_omac);
+
 	if (pingf->f.type == FLOW_PING4) {
 		const struct in_addr *saddr = inany_v4(&ini->oaddr);
 		const struct in_addr *daddr = inany_v4(&ini->eaddr);
 
 		ASSERT(saddr && daddr); /* Must have IPv4 addresses */
-		tap_icmp4_send(c, *saddr, *daddr, buf, n);
+		tap_icmp4_send(c, *saddr, *daddr, buf, pingf->f.tap_omac, n);
 	} else if (pingf->f.type == FLOW_PING6) {
 		const struct in6_addr *saddr = &ini->oaddr.a6;
 		const struct in6_addr *daddr = &ini->eaddr.a6;
 
-		tap_icmp6_send(c, saddr, daddr, buf, n);
+		tap_icmp6_send(c, saddr, daddr, buf, pingf->f.tap_omac, n);
 	}
 	return;
 
diff --git a/ndp.c b/ndp.c
index 1e41933..22a185f 100644
--- a/ndp.c
+++ b/ndp.c
@@ -184,7 +184,7 @@ static void ndp_send(const struct ctx *c, const struct in6_addr *dst,
 {
 	const struct in6_addr *src = &c->ip6.our_tap_ll;
 
-	tap_icmp6_send(c, src, dst, buf, l4len);
+	tap_icmp6_send(c, src, dst, buf, c->our_tap_mac, l4len);
 }
 
 /**
diff --git a/tap.c b/tap.c
index 37a7092..640a860 100644
--- a/tap.c
+++ b/tap.c
@@ -277,13 +277,14 @@ void tap_udp4_send(const struct ctx *c, struct in_addr src, in_port_t sport,
  * @src:	IPv4 source address
  * @dst:	IPv4 destination address
  * @in:		ICMP packet, including ICMP header
+ * @src_mac:	MAC address to be used as source for message
  * @l4len:	ICMP packet length, including ICMP header
  */
 void tap_icmp4_send(const struct ctx *c, struct in_addr src, struct in_addr dst,
-		    const void *in, size_t l4len)
+		    const void *in, const void *src_mac, size_t l4len)
 {
 	char buf[USHRT_MAX];
-	struct iphdr *ip4h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IP);
+	struct iphdr *ip4h = tap_push_l2h(c, buf, src_mac, ETH_P_IP);
 	struct icmphdr *icmp4h = tap_push_ip4h(ip4h, src, dst,
 					       l4len, IPPROTO_ICMP);
 
@@ -384,14 +385,15 @@ void tap_udp6_send(const struct ctx *c,
  * @src:	IPv6 source address
  * @dst:	IPv6 destination address
  * @in:		ICMP packet, including ICMP header
+ * @src_mac:	MAC address to be used as source for message
  * @l4len:	ICMP packet length, including ICMP header
  */
 void tap_icmp6_send(const struct ctx *c,
 		    const struct in6_addr *src, const struct in6_addr *dst,
-		    const void *in, size_t l4len)
+		    const void *in, const void *src_mac, size_t l4len)
 {
 	char buf[USHRT_MAX];
-	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, c->our_tap_mac, ETH_P_IPV6);
+	struct ipv6hdr *ip6h = tap_push_l2h(c, buf, src_mac, ETH_P_IPV6);
 	struct icmp6hdr *icmp6h = tap_push_ip6h(ip6h, src, dst, l4len,
 						IPPROTO_ICMPV6, 0);
 
diff --git a/tap.h b/tap.h
index 02f7761..1864173 100644
--- a/tap.h
+++ b/tap.h
@@ -91,7 +91,7 @@ void tap_udp4_send(const struct ctx *c, struct in_addr src, in_port_t sport,
 		   struct in_addr dst, in_port_t dport,
 		   const void *in, size_t dlen);
 void tap_icmp4_send(const struct ctx *c, struct in_addr src, struct in_addr dst,
-		    const void *in, size_t l4len);
+		    const void *in, const void *src_mac, size_t l4len);
 const struct in6_addr *tap_ip6_daddr(const struct ctx *c,
 				     const struct in6_addr *src);
 void *tap_push_ip6h(struct ipv6hdr *ip6h,
@@ -103,7 +103,7 @@ void tap_udp6_send(const struct ctx *c,
 		   uint32_t flow, void *in, size_t dlen);
 void tap_icmp6_send(const struct ctx *c,
 		    const struct in6_addr *src, const struct in6_addr *dst,
-		    const void *in, size_t l4len);
+		    const void *in, const void *src_mac, size_t l4len);
 void tap_send_single(const struct ctx *c, const void *data, size_t l2len);
 size_t tap_send_frames(const struct ctx *c, const struct iovec *iov,
 		       size_t bufs_per_frame, size_t nframes);
diff --git a/udp.c b/udp.c
index 3acc401..3981e9b 100644
--- a/udp.c
+++ b/udp.c
@@ -400,6 +400,8 @@ static void udp_send_tap_icmp4(const struct ctx *c,
 	struct in_addr eaddr = toside->eaddr.v4mapped.a4;
 	in_port_t eport = toside->eport;
 	in_port_t oport = toside->oport;
+	union inany_addr saddr_any;
+	uint8_t tap_omac[ETH_ALEN];
 	struct {
 		struct icmphdr icmp4h;
 		struct iphdr ip4h;
@@ -421,7 +423,10 @@ static void udp_send_tap_icmp4(const struct ctx *c,
 	tap_push_uh4(&msg.uh, eaddr, eport, oaddr, oport, in, dlen);
 	memcpy(&msg.data, in, dlen);
 
-	tap_icmp4_send(c, saddr, eaddr, &msg, msglen);
+	/* Try to obtain the MAC address of the generating node */
+	saddr_any = inany_from_v4(saddr);
+	fwd_neigh_mac_get(c, &saddr_any, tap_omac);
+	tap_icmp4_send(c, saddr, eaddr, &msg, tap_omac, msglen);
 }
 
 
@@ -445,6 +450,7 @@ static void udp_send_tap_icmp6(const struct ctx *c,
 	const struct in6_addr *eaddr = &toside->eaddr.a6;
 	in_port_t eport = toside->eport;
 	in_port_t oport = toside->oport;
+	uint8_t tap_omac[ETH_ALEN];
 	struct {
 		struct icmp6_hdr icmp6h;
 		struct ipv6hdr ip6h;
@@ -466,7 +472,9 @@ static void udp_send_tap_icmp6(const struct ctx *c,
 	tap_push_uh6(&msg.uh, eaddr, eport, oaddr, oport, in, dlen);
 	memcpy(&msg.data, in, dlen);
 
-	tap_icmp6_send(c, saddr, eaddr, &msg, msglen);
+	/* Try to obtain the MAC address of the generating node */
+	fwd_neigh_mac_get(c, (union inany_addr *) saddr, tap_omac);
+	tap_icmp6_send(c, saddr, eaddr, &msg, tap_omac, msglen);
 }
 
 /**
-- 
2.50.1

Re: [PATCH v13 01/10] netlink: add subscription on changes in NDP/ARP table

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 12565 bytes --]

On Sun, Oct 12, 2025 at 03:33:28PM -0400, Jon Maloy wrote:
> The solution to bug https://bugs.passt.top/show_bug.cgi?id=120
> requires the ability to translate from an IP address to its
> corresponding MAC address in cases where those are present in
> the ARP or NDP tables.
> 
> To keep track of the contents of these tables we add a netlink
> based neighbour subscription feature.
> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>

One important comment, several nits.

> ---
> v3: - Added an attribute contianing NDA_DST to sent message, so
>       that we let the kernel do the filtering of the IP address
>       and return only one entry.
>     - Added interface index to the call signature. Since the only
>       interface we know is the template interface, this limits
>       the number of hosts that will be seen as 'network segment
>       local' from a PASST viewpoint.
> v4: - Made loop independent of attribute order.
>     - Ignoring L2 addresses which are not of size ETH_ALEN.
> v5: - Changed return value of new function, so caller can know if
>       a MAC address really was found.
> v6: - Removed warning printout which had ended up in the wrong
>       commit.
> v8: - Changed to neighbour event subscription model
>     - netlink: arp/ndp table subscription
> v10:- Updated according to David's latest comments on v8
>     - Added functionaly where we initially read current
>       state of ARP/NDP tables
> v12:- Updates based on feedback from David and Stefano
> v13:- Updates based on feedback from David and Stefano
> ---
>  epoll_type.h |   2 +
>  netlink.c    | 215 ++++++++++++++++++++++++++++++++++++++++++++++++++-
>  netlink.h    |   4 +
>  passt.c      |   7 ++
>  4 files changed, 224 insertions(+), 4 deletions(-)
> 
> diff --git a/epoll_type.h b/epoll_type.h
> index 12ac59b..a90ffb6 100644
> --- a/epoll_type.h
> +++ b/epoll_type.h
> @@ -44,6 +44,8 @@ enum epoll_type {
>  	EPOLL_TYPE_REPAIR_LISTEN,
>  	/* TCP_REPAIR helper socket */
>  	EPOLL_TYPE_REPAIR,
> +	/* Netlink neighbour subscription socket */
> +	EPOLL_TYPE_NL_NEIGH,
>  
>  	EPOLL_NUM_TYPES,
>  };
> diff --git a/netlink.c b/netlink.c
> index 9fe70f2..99dcb72 100644
> --- a/netlink.c
> +++ b/netlink.c
> @@ -26,6 +26,7 @@
>  #include <arpa/inet.h>
>  #include <netinet/in.h>
>  #include <netinet/if_ether.h>
> +#include <net/if_arp.h>
>  
>  #include <linux/netlink.h>
>  #include <linux/rtnetlink.h>
> @@ -40,6 +41,10 @@
>  #define RTNH_NEXT_AND_DEC(rtnh, attrlen)				\
>  	((attrlen) -= RTNH_ALIGN((rtnh)->rtnh_len), RTNH_NEXT(rtnh))
>  
> +/* Convenience macro borrowed from kernel */
> +#define NUD_VALID				\
> +	(NUD_PERMANENT | NUD_NOARP | NUD_REACHABLE | NUD_PROBE | NUD_STALE)
> +
>  /* Netlink expects a buffer of at least 8kiB or the system page size,
>   * whichever is larger.  32kiB is recommended for more efficient.
>   * Since the largest page size on any remotely common Linux setup is
> @@ -50,9 +55,10 @@
>  #define NLBUFSIZ 65536
>  
>  /* Socket in init, in target namespace, sequence (just needs to be monotonic) */
> -int nl_sock	= -1;
> -int nl_sock_ns	= -1;
> -static int nl_seq = 1;
> +int nl_sock		 = -1;
> +int nl_sock_ns		 = -1;
> +static int nl_sock_neigh = -1;
> +static int nl_seq	 = 1;
>  
>  /**
>   * nl_sock_init_do() - Set up netlink sockets in init or target namespace
> @@ -325,7 +331,6 @@ unsigned int nl_get_ext_if(int s, sa_family_t af)
>  
>  	if (status < 0)
>  		warn("netlink: RTM_GETROUTE failed: %s", strerror_(-status));
> -

Nit: unrelated change.

>  	if (defifi) {
>  		if (ndef > 1) {
>  			info("Multiple default %s routes, picked first",
> @@ -1103,3 +1108,205 @@ int nl_link_set_flags(int s, unsigned int ifi,
>  
>  	return nl_do(s, &req, RTM_NEWLINK, 0, sizeof(req));
>  }
> +
> +/**
> + * nl_neigh_msg_read() - Interpret a neigbour state message from netlink
> + * @c:		Execution context
> + * @nh:	Message to be read
> + */
> +static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
> +{
> +	struct ndmsg *ndm = NLMSG_DATA(nh);
> +	struct rtattr *rta = (struct rtattr *)(ndm + 1);
> +	size_t na = NLMSG_PAYLOAD(nh, sizeof(*ndm));
> +	char ip_str[INET6_ADDRSTRLEN];
> +	char mac_str[ETH_ADDRSTRLEN];
> +	const uint8_t *lladdr = NULL;
> +	const void *dst = NULL;
> +	size_t lladdr_len = 0;
> +	uint8_t mac[ETH_ALEN];
> +	union inany_addr addr;
> +	size_t dstlen = 0;
> +
> +	if (nh->nlmsg_type == NLMSG_DONE)
> +		return;
> +
> +	if (nh->nlmsg_type == NLMSG_ERROR) {
> +		warn("nlmsg_type error at msg read");
> +		return;
> +	}
> +
> +	if (nh->nlmsg_type != RTM_NEWNEIGH &&
> +	    nh->nlmsg_type != RTM_DELNEIGH)
> +		return;
> +
> +	for (; RTA_OK(rta, na); rta = RTA_NEXT(rta, na)) {
> +		switch (rta->rta_type) {
> +		case NDA_DST:
> +			dst = RTA_DATA(rta);
> +			dstlen = RTA_PAYLOAD(rta);
> +			break;
> +		case NDA_LLADDR:
> +			lladdr = RTA_DATA(rta);
> +			lladdr_len = RTA_PAYLOAD(rta);
> +			break;
> +		default:
> +			break;
> +		}
> +	}
> +
> +	if (!dst)
> +		return;
> +
> +

Nit: extra newline.

> +	if (ndm->ndm_family == AF_INET &&
> +	     ndm->ndm_ifindex != c->ifi4)
> +		return;
> +
> +	if (ndm->ndm_family == AF_INET6 &&
> +	     ndm->ndm_ifindex != c->ifi6)
> +		return;
> +
> +	if (ndm->ndm_family != AF_INET &&
> +	    ndm->ndm_family != AF_INET6)
> +		return;
> +
> +	if (ndm->ndm_family == AF_INET &&
> +	    dstlen != sizeof(struct in_addr)) {
> +		warn("netlink: wrong address length in AF_INET notification");
> +		return;
> +	}
> +	if (ndm->ndm_family == AF_INET6 &&
> +	    dstlen != sizeof(struct in6_addr)) {
> +		warn("netlink: wrong address length in AF_INET6 notification");
> +		return;
> +	}
> +	inany_from_af(&addr, ndm->ndm_family, dst);
> +	inany_ntop(dst, ip_str, sizeof(ip_str));
> +
> +	if (nh->nlmsg_type == RTM_DELNEIGH) {
> +		trace("neigh table delete: %s", ip_str);
> +		return;
> +	}
> +	if (!(ndm->ndm_state & NUD_VALID)) {
> +		trace("neigh table: invalid state for %s", ip_str);
> +		return;
> +	}
> +	if (nh->nlmsg_type != RTM_NEWNEIGH || !lladdr) {
> +		warn("netlink: notification with illegal msg for %s", ip_str);
> +		return;
> +	}
> +	if (lladdr_len != ETH_ALEN || ndm->ndm_type != ARPHRD_ETHER)
> +		return;
> +
> +	memcpy(mac, lladdr, ETH_ALEN);
> +	eth_ntop(mac, mac_str, sizeof(mac_str));
> +	trace("neigh table update: %s / %s", ip_str, mac_str);
> +}
> +
> +/**
> + * nl_neigh_sync() - Read current contents ARP/NDP tables
> + * @c:		Execution context
> + * @proto:	Protocol, AF_INET or AF_INET6
> + * @ifi:	Interface index
> + */
> +static void nl_neigh_sync(const struct ctx *c, int proto, int ifi)
> +{
> +	struct {
> +		struct nlmsghdr nlh;
> +		struct ndmsg    ndm;
> +	} req = {
> +		.nlh = {0},
> +		.ndm.ndm_family  = proto,
> +		.ndm.ndm_ifindex = ifi,
> +		.ndm.ndm_state   = 0,
> +		.ndm.ndm_flags   = 0,
> +		.ndm.ndm_type    = 0

Nit: unspecified initializer fields are assumed to be 0 (as long as
there's at least one initializer field).

> +	};
> +	struct nlmsghdr *nh;
> +	char buf[NLBUFSIZ];
> +	ssize_t status;
> +	uint32_t seq;
> +
> +	seq = nl_send(nl_sock_neigh, &req, RTM_GETNEIGH, NLM_F_DUMP, sizeof(req));
> +	nl_foreach_oftype(nh, status, nl_sock_neigh, buf, seq, RTM_NEWNEIGH)
> +		nl_neigh_msg_read(c, nh);
> +	if (status < 0)
> +		warn("netlink: RTM_GETNEIGH failed: %s", strerror_(-status));
> +}
> +
> +/**
> + * nl_neigh_notify_handler() - Non-blocking drain of pending neighbour updates
> + * @c:		Execution context
> + */
> +void nl_neigh_notify_handler(const struct ctx *c)
> +{
> +	char buf[NLBUFSIZ];
> +
> +	for (;;) {
> +		ssize_t n = recv(nl_sock_neigh, buf, sizeof(buf), MSG_DONTWAIT);
> +		struct nlmsghdr *nh = (struct nlmsghdr *)buf;
> +
> +		if (n < 0) {
> +			struct nlmsgerr *errmsg;
> +
> +			if (nh->nlmsg_type != NLMSG_ERROR)
> +				return;

This isn't right.  It looks like you're conflating errors reported by
recv() with errors reported as netlink messages.  If n < 0, it means
recv() has failed, and nh->nlmsg_type is uninitialized.  recv()
succeeding, but the buffer containing an NLMSG_ERROR is a different
case.  There are two different semantic levels, each of which reports
errors in its own way, each needs to be handled separately.

> +			errmsg = (struct nlmsgerr *)NLMSG_DATA(nh);
> +			warn("netlink: recv() failed: error %i", errmsg->error);
> +			return;
> +		}
> +		for (; NLMSG_OK(nh, n); nh = NLMSG_NEXT(nh, n))
> +			nl_neigh_msg_read(c, nh);
> +	}
> +}
> +
> +/**
> + * nl_neigh_notify_init() - Open netlink sock and subscribe to neighbour events
> + * @c:		Execution context
> + *
> + * Return: 0 on success, -1 on failure
> + */
> +int nl_neigh_notify_init(const struct ctx *c)
> +{
> +	union epoll_ref ref = {
> +		.type = EPOLL_TYPE_NL_NEIGH
> +	};
> +	struct epoll_event ev = {
> +		.events = EPOLLIN
> +	};
> +	struct sockaddr_nl addr = {
> +		.nl_family = AF_NETLINK,
> +		.nl_groups = RTMGRP_NEIGH,
> +	};
> +
> +	if (nl_sock_neigh >= 0) {
> +		warn("RTMGRP_NEIGH already exists");
> +		return 0;
> +	}
> +	nl_sock_neigh = socket(AF_NETLINK, SOCK_RAW | SOCK_CLOEXEC, NETLINK_ROUTE);
> +	if (nl_sock_neigh < 0) {
> +		warn("Failed create RTMGRP_NEIGH socket");
> +		return -1;
> +	}
> +	if (bind(nl_sock_neigh, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> +		close(nl_sock_neigh);
> +		nl_sock_neigh = -1;
> +		warn("Failed bind RTMGRP_NEIGH socket");
> +		return -1;
> +	}
> +
> +	ev.data.u64 = ref.u64;
> +	if (epoll_ctl(c->epollfd, EPOLL_CTL_ADD, nl_sock_neigh, &ev) == -1) {
> +		close(nl_sock_neigh);
> +		nl_sock_neigh = -1;
> +		warn("Failed do epoll_ctrl() on RTMGRP_NEIGH socket");
> +		return -1;
> +	}

We can probably simplify this a bit with the epoll_add() helper in one
of Laurent's draft patches.  I guess it just depends which lands first.

> +
> +	nl_neigh_sync(c, AF_INET, c->ifi4);
> +	nl_neigh_sync(c, AF_INET6, c->ifi6);
> +
> +	return 0;
> +}
> diff --git a/netlink.h b/netlink.h
> index b51e99c..8f1e9b9 100644
> --- a/netlink.h
> +++ b/netlink.h
> @@ -17,6 +17,8 @@ int nl_route_dup(int s_src, unsigned int ifi_src,
>  		 int s_dst, unsigned int ifi_dst, sa_family_t af);
>  int nl_addr_get(int s, unsigned int ifi, sa_family_t af,
>  		void *addr, int *prefix_len, void *addr_l);
> +bool nl_neigh_mac_get(int s, const union inany_addr *addr, int ifi,
> +		      unsigned char *mac);
>  int nl_addr_set(int s, unsigned int ifi, sa_family_t af,
>  		const void *addr, int prefix_len);
>  int nl_addr_get_ll(int s, unsigned int ifi, struct in6_addr *addr);
> @@ -28,5 +30,7 @@ int nl_link_set_mac(int s, unsigned int ifi, const void *mac);
>  int nl_link_set_mtu(int s, unsigned int ifi, int mtu);
>  int nl_link_set_flags(int s, unsigned int ifi,
>  		      unsigned int set, unsigned int change);
> +int nl_neigh_notify_init(const struct ctx *c);
> +void nl_neigh_notify_handler(const struct ctx *c);
>  
>  #endif /* NETLINK_H */
> diff --git a/passt.c b/passt.c
> index 31fbb75..e21d6ba 100644
> --- a/passt.c
> +++ b/passt.c
> @@ -53,6 +53,7 @@
>  #include "vu_common.h"
>  #include "migrate.h"
>  #include "repair.h"
> +#include "netlink.h"
>  
>  #define EPOLL_EVENTS		8
>  
> @@ -79,6 +80,7 @@ char *epoll_type_str[] = {
>  	[EPOLL_TYPE_VHOST_KICK]		= "vhost-user kick socket",
>  	[EPOLL_TYPE_REPAIR_LISTEN]	= "TCP_REPAIR helper listening socket",
>  	[EPOLL_TYPE_REPAIR]		= "TCP_REPAIR helper socket",
> +	[EPOLL_TYPE_NL_NEIGH]		= "netlink neighbour subscription socket",
>  };
>  static_assert(ARRAY_SIZE(epoll_type_str) == EPOLL_NUM_TYPES,
>  	      "epoll_type_str[] doesn't match enum epoll_type");
> @@ -322,6 +324,8 @@ int main(int argc, char **argv)
>  
>  	pcap_init(&c);
>  
> +	nl_neigh_notify_init(&c);
> +
>  	if (!c.foreground) {
>  		if ((devnull_fd = open("/dev/null", O_RDWR | O_CLOEXEC)) < 0)
>  			die_perror("Failed to open /dev/null");
> @@ -414,6 +418,9 @@ loop:
>  		case EPOLL_TYPE_REPAIR:
>  			repair_handler(&c, eventmask);
>  			break;
> +		case EPOLL_TYPE_NL_NEIGH:
> +			nl_neigh_notify_handler(&c);
> +			break;
>  		default:
>  			/* Can't happen */
>  			ASSERT(0);
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v13 02/10] passt: add no_map_gw flag to struct ctx

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 3051 bytes --]

On Sun, Oct 12, 2025 at 03:33:29PM -0400, Jon Maloy wrote:
> Later in this series we will introduce a new initialization function
> which needs access to the 'no_map_gw' flag. This flag is currently
> a local variable inside the conf() call, where it is too early to
> call the new function. The new function needs both ctx->hash_secret
> and a valid ctx->fd_tap to be set, neither of which are initialized
> at that point.
> 
> We therefore add the 'no_map_flag' to struct ctx, so it can be carried
> along and used by later functions.

Nit 'no_map_gw', not 'no_map_flag'.

> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

I'll have to revisit this with the flexible forwarding stuff, but this
won't make it any harder.

> ---
>  conf.c  | 10 +++++-----
>  passt.h |  1 +
>  2 files changed, 6 insertions(+), 5 deletions(-)
> 
> diff --git a/conf.c b/conf.c
> index 66b9e63..6143b54 100644
> --- a/conf.c
> +++ b/conf.c
> @@ -1410,7 +1410,7 @@ fail:
>   */
>  void conf(struct ctx *c, int argc, char **argv)
>  {
> -	int netns_only = 0, no_map_gw = 0;
> +	int netns_only = 0;
>  	const struct option options[] = {
>  		{"debug",	no_argument,		NULL,		'd' },
>  		{"quiet",	no_argument,		NULL,		'q' },
> @@ -1442,7 +1442,7 @@ void conf(struct ctx *c, int argc, char **argv)
>  		{"no-ra",	no_argument,		&c->no_ra,	1 },
>  		{"no-splice",	no_argument,		&c->no_splice,	1 },
>  		{"freebind",	no_argument,		&c->freebind,	1 },
> -		{"no-map-gw",	no_argument,		&no_map_gw,	1 },
> +		{"no-map-gw",	no_argument,		&c->no_map_gw,	1 },
>  		{"ipv4-only",	no_argument,		NULL,		'4' },
>  		{"ipv6-only",	no_argument,		NULL,		'6' },
>  		{"one-off",	no_argument,		NULL,		'1' },
> @@ -1656,7 +1656,7 @@ void conf(struct ctx *c, int argc, char **argv)
>  			break;
>  		case 21:
>  			conf_nat(optarg, &c->ip4.map_host_loopback,
> -				 &c->ip6.map_host_loopback, &no_map_gw);
> +				 &c->ip6.map_host_loopback, &c->no_map_gw);
>  			break;
>  		case 22:
>  			conf_nat(optarg, &c->ip4.map_guest_addr,
> @@ -2005,11 +2005,11 @@ void conf(struct ctx *c, int argc, char **argv)
>  		}
>  	}
>  
> -	if (c->ifi4 && !no_map_gw &&
> +	if (c->ifi4 && !c->no_map_gw &&
>  	    IN4_IS_ADDR_UNSPECIFIED(&c->ip4.map_host_loopback))
>  		c->ip4.map_host_loopback = c->ip4.guest_gw;
>  
> -	if (c->ifi6 && !no_map_gw &&
> +	if (c->ifi6 && !c->no_map_gw &&
>  	    IN6_IS_ADDR_UNSPECIFIED(&c->ip6.map_host_loopback))
>  		c->ip6.map_host_loopback = c->ip6.guest_gw;
>  
> diff --git a/passt.h b/passt.h
> index 0075eb4..830bc36 100644
> --- a/passt.h
> +++ b/passt.h
> @@ -310,6 +310,7 @@ struct ctx {
>  	int no_ndp;
>  	int no_ra;
>  	int no_splice;
> +	int no_map_gw;
>  	int host_lo_to_ns_lo;
>  	int freebind;
>  
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v13 03/10] fwd: Add cache table for ARP/NDP contents

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 12752 bytes --]

On Sun, Oct 12, 2025 at 03:33:30PM -0400, Jon Maloy wrote:
> We add a cache table to keep track of the contents of the kernel ARP
> and NDP tables. The table is fed from the just introduced netlink based
> neigbour subscription function. The new table eliminates the need for
> explicit netlink calls to find a host's MAC address.

Last sentence only really makes sense in the context of earlier
versions of this series, which won't be in the final commit log.

> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>
> 
> ---
> v5: - Moved to earlier in series to reduce rebase conflicts
> v6: - Sqashed the hash list commit and the FIFO/LRU queue commit
>     - Removed hash lookup. We now only use linear lookup in a
>       linked list
>     - Eliminated dynamic memory allocation.
>     - Ensured there is only one call to clock_gettime()
>     - Using MAC_ZERO instead of the previously dedicated definitions
> v7: - NOW using MAC_ZERO where needed
>     - I am still using linear back-off for empty cache entries. Even
>       an incoming, flow-creating packet from a local host gives no
>       guarantee that its MAC address is in the ARP table, so we must
>       allow for a few new attempts at first possible occasions. Only
>       after several failed lookups can we conclude that we probably
>       never will succeed. Hence the back-off.
>     - Fixed a bug that David inadvertently made me aware of: I only
>       intended to set the initial expiry value to MAC_CACHE_RENEWAL
>       when an ARP/NDP table lookup was successful.
>     - Improved struct and function description comments.
> v8: - Total re-design of table, adapting to the new, subscription
>       based way of updating it.
> v9: - Catering for MAC address change for an existing host.
> v10: - Changes according to feedback from David Gibson
> v12: - Changes according to feedback from David and Stefano
>      - Added dummy entries for loopback and default GW addresses
> v13: - Changes according to feedback and discussions with David
>        and Stefano
> ---
>  fwd.c     | 222 ++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  fwd.h     |   7 ++
>  netlink.c |   3 +
>  passt.c   |   1 +
>  4 files changed, 233 insertions(+)
> 
> diff --git a/fwd.c b/fwd.c
> index 250cf56..d062ebc 100644
> --- a/fwd.c
> +++ b/fwd.c
> @@ -26,6 +26,7 @@
>  #include "passt.h"
>  #include "lineread.h"
>  #include "flow_table.h"
> +#include "netlink.h"
>  
>  /* Empheral port range: values from RFC 6335 */
>  static in_port_t fwd_ephemeral_min = (1 << 15) + (1 << 14);
> @@ -33,6 +34,227 @@ static in_port_t fwd_ephemeral_max = NUM_PORTS - 1;
>  
>  #define PORT_RANGE_SYSCTL	"/proc/sys/net/ipv4/ip_local_port_range"
>  
> +#define NEIGH_TABLE_SLOTS    1024
> +#define NEIGH_TABLE_SIZE     (NEIGH_TABLE_SLOTS / 2)
> +static_assert((NEIGH_TABLE_SLOTS & (NEIGH_TABLE_SLOTS - 1)) == 0,
> +	      "NEIGH_TABLE_SLOTS must be a power of two");
> +
> +/**
> + * struct neigh_table_entry - Entry in the ARP/NDP table
> + * @next:	Next entry in slot or free list
> + * @addr:	IP address of represented host
> + * @mac:	MAC address of represented host
> + * @permanent:	Entry cannot be altered or freed by notification
> + */
> +struct neigh_table_entry {
> +	struct neigh_table_entry *next;
> +	union inany_addr addr;
> +	uint8_t mac[ETH_ALEN];
> +	bool permanent;
> +};
> +
> +/**
> + * struct neigh_table - Cache of ARP/NDP table contents
> + * @entries:	Entries to be plugged into the hash slots when allocated
> + * @slots:	Hash table slots
> + * @free:	Linked list of unused entries
> + */
> +struct neigh_table {
> +	struct neigh_table_entry entries[NEIGH_TABLE_SIZE];
> +	struct neigh_table_entry *slots[NEIGH_TABLE_SLOTS];
> +	struct neigh_table_entry *free;
> +};
> +
> +static struct neigh_table neigh_table;
> +
> +/**
> + * neigh_table_slot() - Hash key to a number within the table range
> + * @c:		Execution context
> + * @key:	The key to be used for the hash
> + *
> + * Return: the resulting hash value
> + */
> +static size_t neigh_table_slot(const struct ctx *c,
> +			       const union inany_addr *key)
> +{
> +	struct siphash_state st = SIPHASH_INIT(c->hash_secret);
> +	uint32_t i;
> +
> +	inany_siphash_feed(&st, key);
> +	i = siphash_final(&st, sizeof(*key), 0);
> +
> +	return ((size_t)i) & (NEIGH_TABLE_SIZE - 1);
> +}
> +
> +/**
> + * fwd_neigh_table_find() - Find a MAC table entry
> + * @c:		Execution context
> + * @addr:	Neighbour address to be used as key for the lookup
> + *
> + * Return: the matching entry, if found. Otherwise NULL
> + */
> +static struct neigh_table_entry *fwd_neigh_table_find(const struct ctx *c,
> +						      const union inany_addr *addr)
> +{
> +	size_t slot = neigh_table_slot(c, addr);
> +	struct neigh_table_entry *e = neigh_table.slots[slot];
> +
> +	while (e && !inany_equals(&e->addr, addr))
> +		e = e->next;
> +
> +	return e;
> +}
> +
> +/**
> + * fwd_neigh_table_update() - Allocate or update neighbour table entry
> + * @c:		Execution context
> + * @addr:	IP address used to determine insertion slot and store in entry
> + * @mac:	The MAC address associated with the neighbour address
> + * @permanent:	Created entry cannot be altered or freed
> + */
> +void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
> +			    const uint8_t *mac, bool permanent)
> +{
> +	struct neigh_table *t = &neigh_table;
> +	struct neigh_table_entry *e;
> +	union inany_addr daddr;
> +	ssize_t slot;
> +
> +	/* We only add guest-side visible addresses */
> +	if (!nat_inbound(c, addr, &daddr))
> +		return;

To me, it makes more sense to have the nat_inbound() in the caller.
That way the hash table is just a hash table, with no specific
knowledge of what the contents mean.  The caller is the thing which
knows it has a host side address, and wants to update a corresponding
entry in a table indexed by guest side address.  It also means that
the fwd_neigh_table_*() functions will consistently take guest side
addresses.

> +	/* MAC address might change sometimes */
> +	e = fwd_neigh_table_find(c, &daddr);
> +	if (e) {
> +		if (!e->permanent)
> +			memcpy(e->mac, mac, ETH_ALEN);
> +		return;
> +	}
> +
> +	e = t->free;
> +	if (!e) {
> +		debug("Failed to allocate neighbour table entry");
> +		return;
> +	}
> +	t->free = e->next;
> +	slot = neigh_table_slot(c, &daddr);
> +	e->next = t->slots[slot];
> +	t->slots[slot] = e;
> +
> +	memcpy(&e->addr, &daddr, sizeof(daddr));
> +	memcpy(e->mac, mac, ETH_ALEN);
> +	e->permanent = permanent;
> +}
> +
> +/**
> + * fwd_neigh_table_free() - Remove an entry from a slot and add it to free list
> + * @c:		Execution context
> + * @addr:	IP address used to find the slot for the entry
> + */
> +void fwd_neigh_table_free(const struct ctx *c, const union inany_addr *addr)
> +{
> +	ssize_t slot = neigh_table_slot(c, addr);
> +	struct neigh_table *t = &neigh_table;
> +	struct neigh_table_entry *e, **prev;
> +
> +	prev = &t->slots[slot];
> +	e = t->slots[slot];
> +	while (e && !inany_equals(&e->addr, addr)) {
> +		prev = &e->next;
> +		e = e->next;
> +	}
> +
> +	if (!e || e->permanent)
> +		return;
> +
> +	*prev = e->next;
> +	e->next = t->free;
> +	t->free = e;
> +	memset(&e->addr, 0, sizeof(*addr));
> +	memset(e->mac, 0, ETH_ALEN);
> +}
> +
> +/**
> + * fwd_neigh_mac_get() - Look up MAC address in the ARP/NDP table
> + * @c:		Execution context
> + * @addr:	Neighbour IP address used as lookup key
> + * @mac:	Buffer for returned MAC address
> + */
> +void fwd_neigh_mac_get(const struct ctx *c, const union inany_addr *addr,
> +		       uint8_t *mac)
> +{
> +	const struct neigh_table_entry *e = fwd_neigh_table_find(c, addr);
> +
> +	if (e)
> +		memcpy(mac, e->mac, ETH_ALEN);
> +	else
> +		memcpy(mac, c->our_tap_mac, ETH_ALEN);
> +}
> +
> +/**
> + * fwd_neigh_table_init() - Initialize the neighbour table
> + * @c:		Execution context
> + */
> +void fwd_neigh_table_init(const struct ctx *c)
> +{
> +	union inany_addr mhl = inany_from_v4(c->ip4.map_host_loopback);
> +	union inany_addr mga = inany_from_v4(c->ip4.map_guest_addr);
> +	union inany_addr ggw = inany_from_v4(c->ip4.guest_gw);
> +	struct neigh_table *t = &neigh_table;
> +	struct neigh_table_entry *e;
> +	int i;
> +
> +	memset(t, 0, sizeof(*t));
> +
> +	for (i = 0; i < NEIGH_TABLE_SIZE; i++) {
> +		e = &t->entries[i];
> +		e->next = t->free;
> +		t->free = e;
> +	}
> +
> +	/* Blocker entries to stop events from hosts using these addresses */
> +	if (!inany_is_unspecified4(&mhl))
> +		fwd_neigh_table_update(c, &mhl, c->our_tap_mac, true);

Here mhl is already a guest side address - and may not have a
corresponding address host side - so we definitely don't want the
nat_inbound() inside fwd_neigh_table_update().


> +	if (!inany_is_unspecified4(&ggw) && !c->no_map_gw)
> +		fwd_neigh_table_update(c, &ggw, c->our_tap_mac, true);

Same here.

> +	if (!inany_is_unspecified4(&mga) && !inany_equals(&mhl, &mga)) {
> +		uint8_t mac[ETH_ALEN];
> +		int rc;
> +
> +		rc = nl_link_get_mac(nl_sock, c->ifi4, mac);
> +		if (rc < 0) {
> +			debug("Couldn't get ip4 MAC addr: %s", strerror_(-rc));
> +			memcpy(mac, c->our_tap_mac, ETH_ALEN);
> +		}
> +		fwd_neigh_table_update(c, &mga, mac, true);

And here.

> +	}
> +
> +	mhl = *(union inany_addr *)&c->ip6.map_host_loopback;
> +	mga = *(union inany_addr *)&c->ip4.map_guest_addr;
> +	ggw = *(union inany_addr *)&c->ip4.guest_gw;
> +
> +	if (!inany_is_unspecified6(&mhl))
> +		fwd_neigh_table_update(c, &mhl, c->our_tap_mac, true);

..and so forth.

> +	if (!inany_is_unspecified6(&ggw) && !c->no_map_gw)
> +		fwd_neigh_table_update(c, &ggw, c->our_tap_mac, true);
> +
> +	if (!inany_is_unspecified6(&mga) && !inany_equals(&mhl, &mga)) {
> +		uint8_t mac[ETH_ALEN];
> +		int rc;
> +
> +		rc = nl_link_get_mac(nl_sock, c->ifi6, mac);
> +		if (rc < 0) {
> +			debug("Couldn't get ip6 MAC addr: %s", strerror_(-rc));
> +			memcpy(mac, c->our_tap_mac, ETH_ALEN);
> +		}
> +		fwd_neigh_table_update(c, &mga, mac, true);
> +	}
> +}
> +
>  /** fwd_probe_ephemeral() - Determine what ports this host considers ephemeral
>   *
>   * Work out what ports the host thinks are emphemeral and record it for later
> diff --git a/fwd.h b/fwd.h
> index 65c7c96..352f3b5 100644
> --- a/fwd.h
> +++ b/fwd.h
> @@ -56,5 +56,12 @@ uint8_t fwd_nat_from_splice(const struct ctx *c, uint8_t proto,
>  			    const struct flowside *ini, struct flowside *tgt);
>  uint8_t fwd_nat_from_host(const struct ctx *c, uint8_t proto,
>  			  const struct flowside *ini, struct flowside *tgt);
> +void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
> +			    const uint8_t *mac, bool permanent);
> +void fwd_neigh_table_free(const struct ctx *c,
> +			  const union inany_addr *addr);
> +void fwd_neigh_mac_get(const struct ctx *c, const union inany_addr *addr,
> +		       uint8_t *mac);
> +void fwd_neigh_table_init(const struct ctx *c);
>  
>  #endif /* FWD_H */
> diff --git a/netlink.c b/netlink.c
> index 99dcb72..61796fb 100644
> --- a/netlink.c
> +++ b/netlink.c
> @@ -1186,10 +1186,12 @@ static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
>  
>  	if (nh->nlmsg_type == RTM_DELNEIGH) {
>  		trace("neigh table delete: %s", ip_str);
> +		fwd_neigh_table_free(c, &addr);

You'll need a nat_inbound() on the delete side as well.  Again, I
think it makes more sense here than in fwd_neigh_table_free(), but at
the moment it's in neither.

>  		return;
>  	}
>  	if (!(ndm->ndm_state & NUD_VALID)) {
>  		trace("neigh table: invalid state for %s", ip_str);
> +		fwd_neigh_table_free(c, &addr);
>  		return;
>  	}
>  	if (nh->nlmsg_type != RTM_NEWNEIGH || !lladdr) {
> @@ -1202,6 +1204,7 @@ static void nl_neigh_msg_read(const struct ctx *c, struct nlmsghdr *nh)
>  	memcpy(mac, lladdr, ETH_ALEN);
>  	eth_ntop(mac, mac_str, sizeof(mac_str));
>  	trace("neigh table update: %s / %s", ip_str, mac_str);
> +	fwd_neigh_table_update(c, &addr, mac, false);
>  }
>  
>  /**
> diff --git a/passt.c b/passt.c
> index e21d6ba..98fc430 100644
> --- a/passt.c
> +++ b/passt.c
> @@ -324,6 +324,7 @@ int main(int argc, char **argv)
>  
>  	pcap_init(&c);
>  
> +	fwd_neigh_table_init(&c);
>  	nl_neigh_notify_init(&c);
>  
>  	if (!c.foreground) {
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v13 04/10] arp/ndp: respond with true MAC address of LAN local remote hosts

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4151 bytes --]

On Sun, Oct 12, 2025 at 03:33:31PM -0400, Jon Maloy wrote:
> When we receive an ARP request or NDP neigbour solicitation over
> the tap interface for a host on the local network segment attached
> to the template interface, we respond with that host's real MAC
> address, if available.
> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>
> 
> ---
> v3: - Added helper function to find out if a remote ip address is subject
>       to NAT. This filters out local host addresses which should be
>       presented with the passt/pasta local MAC address 9a:55:9a:55:9a:55 even
>       though it is on the local segment.
>     - Adapted to the change in nl_mac_get() function, so that we now consider
>       only the template interface when checking the ARP/NDP table.
> v4: - Moved NAT check into the function nat_outbound() to obtain more
>       precise criteria for when NAT is used. We may in theory
>       have NAT even if original and translated addresses are equal, and
>       we want to catch this case.
>     - I chose to keep the wrapper funtion inany_nat(), but moved it to
>       fwd.h/fwd.c and renamed it to fwd_inany_nat().
> v5: - Simplified criteria for when we do ARP/NDP lookup. Now, we
>       just try with the potentially translated address after an
>       attempted NAT check.
>     - Using the new ARP/NDP cache table instead of using netlink
>       directly.
> v6: - Fixes after feedback from David:
>     - Renamed nat_outbound() to fwd_nat_outbound()
>     - Eliminated unnecessary temporary variable in arp.c::arp()
> v12: - Some minor changes after feedback from Stefano
> v13: - Removed call to nat_outbound() before MAC resolution, as
>        we are now handling guest-side visible addresses only.
> ---
>  arp.c   | 8 ++++++--
>  inany.c | 1 +
>  ndp.c   | 4 ++++
>  3 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/arp.c b/arp.c
> index ad088b1..b4a686f 100644
> --- a/arp.c
> +++ b/arp.c
> @@ -69,6 +69,7 @@ static bool ignore_arp(const struct ctx *c,
>   */
>  int arp(const struct ctx *c, struct iov_tail *data)
>  {
> +	union inany_addr tgt;
>  	struct {
>  		struct ethhdr eh;
>  		struct arphdr ah;
> @@ -102,8 +103,11 @@ int arp(const struct ctx *c, struct iov_tail *data)
>  	resp.ah.ar_hln = ah->ar_hln;
>  	resp.ah.ar_pln = ah->ar_pln;
>  
> -	/* ARP message */
> -	memcpy(resp.am.sha,		c->our_tap_mac,	sizeof(resp.am.sha));
> +	/* MAC address to return in ARP message */
> +	inany_from_af(&tgt, AF_INET, am->tip);
> +	fwd_neigh_mac_get(c, &tgt, resp.am.sha);
> +
> +	/* Rest of ARP message */
>  	memcpy(resp.am.sip,		am->tip,	sizeof(resp.am.sip));
>  	memcpy(resp.am.tha,		am->sha,	sizeof(resp.am.tha));
>  	memcpy(resp.am.tip,		am->sip,	sizeof(resp.am.tip));
> diff --git a/inany.c b/inany.c
> index 65a39f9..7680439 100644
> --- a/inany.c
> +++ b/inany.c
> @@ -16,6 +16,7 @@
>  #include "ip.h"
>  #include "siphash.h"
>  #include "inany.h"
> +#include "fwd.h"
>  
>  const union inany_addr inany_loopback4 = INANY_INIT4(IN4ADDR_LOOPBACK_INIT);
>  const union inany_addr inany_any4 = INANY_INIT4(IN4ADDR_ANY_INIT);
> diff --git a/ndp.c b/ndp.c
> index 588b48f..7cc6e48 100644
> --- a/ndp.c
> +++ b/ndp.c
> @@ -196,6 +196,7 @@ static void ndp_send(const struct ctx *c, const struct in6_addr *dst,
>  static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
>  	    const struct in6_addr *addr)
>  {
> +	union inany_addr tgt;
>  	struct ndp_na na = {
>  		.ih = {
>  			.icmp6_type		= NA,
> @@ -215,6 +216,9 @@ static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
>  
>  	memcpy(na.target_l2_addr.mac, c->our_tap_mac, ETH_ALEN);

The line above is obsoleted by the additions below isn't it?


> +	inany_from_af(&tgt, AF_INET6, addr);
> +	fwd_neigh_mac_get(c, &tgt, na.target_l2_addr.mac);
> +
>  	ndp_send(c, dst, &na, sizeof(na));
>  }
>  
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v13 05/10] arp/ndp: send ARP announcement / unsolicited NA when neigbour entry added

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5996 bytes --]

On Sun, Oct 12, 2025 at 03:33:32PM -0400, Jon Maloy wrote:
> ARP announcements and unsolicited NAs should be handled with caution
> because of the risk of malignant users emitting them to disturb
> network communication.
> 
> There is however one case we where we know it is legitimate
> and safe for us to send out such messages: The one time we switch
> from using ctx->own_tap_mac to a MAC address received via the
> recently added neigbour subscription function. Later changes to
> the MAC address of a host in an existing entry cannot be fully
> trusted, so we abstain from doing it in such cases.
> 
> When sending this type of messages, we notice that the guest accepts
> the update, but shortly later asks for a confirmation in the form of
> a regular ARP/NS request. This is responded to with the new value,
> and we have exactly the effect we wanted.
> 
> This commit adds this functionality.
> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>
> 
> ---
> v10: -Made small changes based of feedback from David G.
> v11: -Moved from 'Gratuitous ARP reply' model to 'ARP Announcement'
>       model.
> v12: -Excluding loopback and default GW addresses from the ARP/NA
>       announcement to be sent to the guest
> v13: -Filtering out all announcements of our_tap_mac instead of
>       explicitly comparing a set of IP addresses.
>      -Changed annc.am.tha in arp_announce() to MAC_ZERO, which is
>       'unknown' according to RFC826.
>      -Renamed ndp_send_unsolicited_na() to ndp_unsolicited_na()
> ---
>  arp.c | 42 ++++++++++++++++++++++++++++++++++++++++++
>  arp.h |  2 ++
>  fwd.c | 10 ++++++++++
>  ndp.c | 10 ++++++++++
>  ndp.h |  1 +
>  5 files changed, 65 insertions(+)
> 
> diff --git a/arp.c b/arp.c
> index b4a686f..18fe1c7 100644
> --- a/arp.c
> +++ b/arp.c
> @@ -150,3 +150,45 @@ void arp_send_init_req(const struct ctx *c)
>  	debug("Sending initial ARP request for guest MAC address");
>  	tap_send_single(c, &req, sizeof(req));
>  }
> +
> +/**
> + * arp_announce() - Send an ARP announcement for an IPv4 host
> + * @c:		Execution context
> + * @ip:	IPv4 address we announce as owned by @mac
> + * @mac:	MAC address to advertise for @ip
> + */
> +void arp_announce(const struct ctx *c, struct in_addr *ip,
> +		  const unsigned char *mac)
> +{
> +	char ip_str[INET_ADDRSTRLEN];
> +	char mac_str[ETH_ADDRSTRLEN];
> +	struct {
> +		struct ethhdr eh;
> +		struct arphdr ah;
> +		struct arpmsg am;
> +	} __attribute__((__packed__)) annc;
> +
> +	/* Ethernet header */
> +	annc.eh.h_proto = htons(ETH_P_ARP);
> +	memcpy(annc.eh.h_dest, MAC_BROADCAST, sizeof(annc.eh.h_dest));
> +	memcpy(annc.eh.h_source, mac, sizeof(annc.eh.h_source));
> +
> +	/* ARP header */
> +	annc.ah.ar_op = htons(ARPOP_REQUEST);
> +	annc.ah.ar_hrd = htons(ARPHRD_ETHER);
> +	annc.ah.ar_pro = htons(ETH_P_IP);
> +	annc.ah.ar_hln = ETH_ALEN;
> +	annc.ah.ar_pln = 4;
> +
> +	/* ARP message */
> +	memcpy(annc.am.sha, mac, sizeof(annc.am.sha));
> +	memcpy(annc.am.sip, ip, sizeof(annc.am.sip));
> +	memcpy(annc.am.tha, MAC_ZERO, sizeof(annc.am.tha));
> +	memcpy(annc.am.tip, ip, sizeof(annc.am.tip));

It seems weird to have sip == tip, but sha != tha.  Is this the way to
do this suggested in the RFCs?  If so, might be worth quoting chapter
and verse in a comment.

> +	inet_ntop(AF_INET, ip, ip_str, sizeof(ip_str));
> +	eth_ntop(mac, mac_str, sizeof(mac_str));
> +	debug("Announcing ARP for %s / %s", ip_str, mac_str);
> +
> +	tap_send_single(c, &annc, sizeof(annc));
> +}
> diff --git a/arp.h b/arp.h
> index d5ad0e1..4862e90 100644
> --- a/arp.h
> +++ b/arp.h
> @@ -22,5 +22,7 @@ struct arpmsg {
>  
>  int arp(const struct ctx *c, struct iov_tail *data);
>  void arp_send_init_req(const struct ctx *c);
> +void arp_announce(const struct ctx *c, struct in_addr *ip,
> +		  const unsigned char *mac);
>  
>  #endif /* ARP_H */
> diff --git a/fwd.c b/fwd.c
> index d062ebc..3a61901 100644
> --- a/fwd.c
> +++ b/fwd.c
> @@ -27,6 +27,8 @@
>  #include "lineread.h"
>  #include "flow_table.h"
>  #include "netlink.h"
> +#include "arp.h"
> +#include "ndp.h"
>  
>  /* Empheral port range: values from RFC 6335 */
>  static in_port_t fwd_ephemeral_min = (1 << 15) + (1 << 14);
> @@ -145,6 +147,14 @@ void fwd_neigh_table_update(const struct ctx *c, const union inany_addr *addr,
>  	memcpy(&e->addr, &daddr, sizeof(daddr));
>  	memcpy(e->mac, mac, ETH_ALEN);
>  	e->permanent = permanent;
> +
> +	if (!memcmp(mac, c->our_tap_mac, ETH_ALEN))
> +		return;
> +
> +	if (inany_v4(&daddr))
> +		arp_announce(c, inany_v4(&daddr), e->mac);
> +	else
> +		ndp_unsolicited_na(c, &daddr.a6);
>  }
>  
>  /**
> diff --git a/ndp.c b/ndp.c
> index 7cc6e48..1e41933 100644
> --- a/ndp.c
> +++ b/ndp.c
> @@ -222,6 +222,16 @@ static void ndp_na(const struct ctx *c, const struct in6_addr *dst,
>  	ndp_send(c, dst, &na, sizeof(na));
>  }
>  
> +/**
> + * ndp_unsolicited_na() - Send unsolicited NA
> + * @c:         Execution context
> + * @addr:      IPv6 address to advertise
> + */
> +void ndp_unsolicited_na(const struct ctx *c, const struct in6_addr *addr)
> +{
> +	ndp_na(c, &in6addr_ll_all_nodes, addr);
> +}
> +
>  /**
>   * ndp_ra() - Send an NDP Router Advertisement (RA) message
>   * @c:		Execution context
> diff --git a/ndp.h b/ndp.h
> index 781ea86..56b756d 100644
> --- a/ndp.h
> +++ b/ndp.h
> @@ -12,5 +12,6 @@ int ndp(const struct ctx *c, const struct in6_addr *saddr,
>  	struct iov_tail *data);
>  void ndp_timer(const struct ctx *c, const struct timespec *now);
>  void ndp_send_init_req(const struct ctx *c);
> +void ndp_unsolicited_na(const struct ctx *c, const struct in6_addr *addr);
>  
>  #endif /* NDP_H */
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v13 06/10] flow: add MAC address of LAN local remote hosts to flow

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 4082 bytes --]

On Sun, Oct 12, 2025 at 03:33:33PM -0400, Jon Maloy wrote:
> When communicating with remote hosts on the local network, some guest
> applications want to see the real MAC address of that host instead
> of PASST/PASTA's own tap address. The flow_common structure is a
> convenient location for storing that address, so we do that in this
> commit.
> 
> Note that we don´t add actual usage of this address here, that will
> be done in later commits.
> 
> Signed-off-by: Jon Maloy <jmaloy@redhat.com>

Reviewed-by: David Gibson <david@gibson.dropbear.id.au>

> 
> ---
> v3: - Moved the remote host macaddress from struct flowside to
>       struct flow_common. I chose to call it 'omac' as suggested
>       by David, although in my understanding the correct name would be
>       'emac'. (In general I find the address naming scheme confusing.)
>     - Adapted to new signature of function nl_mac_get(), now passing
>       it the index of the template interface.
> v4: - Renamed flow_commeon->omac to flow_common->tap_omac to make is
>       role in the code clearer
> v5: - Modified the criteria for ARP/NDP table lookup like in the
>       previous commits.
>     - Removed the PIF_TAP lookup case, as David suggested, and did
>       instead give the flow->tap_omac field a value marking it as
>       non-initialized.
>     - Calling the cache table instead of netlink for ARP/NDP lookup.
>     - Unconditionally using the potentially translated IP address
>       in the lookup, instead of only if NAT really was applied.
> v6: - Using MAC_ZERO instead of own definitions
> v12:- Using MAC_UNDEF (==ff:ff:ff:ff:ff:ff) instead of MAC_ZERO,
>       which is a legal MAC address.
> v13: - Removed call to nat_outbound() before MAC resolution, as
>        we are now handling guest-side visible addresses only.
>      - Using tgt->oaddr instead of ini->eaddr as lookup key
>        for fwd_neigh_mac_get(), for the same reason as above.
> ---
>  flow.c | 2 ++
>  flow.h | 2 ++
>  util.h | 2 ++
>  3 files changed, 6 insertions(+)
> 
> diff --git a/flow.c b/flow.c
> index feefda3..a57d7b9 100644
> --- a/flow.c
> +++ b/flow.c
> @@ -449,6 +449,7 @@ struct flowside *flow_target(const struct ctx *c, union flow *flow,
>  
>  	switch (f->pif[INISIDE]) {
>  	case PIF_TAP:
> +		memcpy(f->tap_omac, MAC_UNDEF, ETH_ALEN);
>  		tgtpif = fwd_nat_from_tap(c, proto, ini, tgt);
>  		break;
>  
> @@ -458,6 +459,7 @@ struct flowside *flow_target(const struct ctx *c, union flow *flow,
>  
>  	case PIF_HOST:
>  		tgtpif = fwd_nat_from_host(c, proto, ini, tgt);
> +		fwd_neigh_mac_get(c, &tgt->oaddr, f->tap_omac);
>  		break;
>  
>  	default:
> diff --git a/flow.h b/flow.h
> index cac618a..f342895 100644
> --- a/flow.h
> +++ b/flow.h
> @@ -177,6 +177,7 @@ int flowside_connect(const struct ctx *c, int s,
>   * @type:	Type of packet flow
>   * @pif[]:	Interface for each side of the flow
>   * @side[]:	Information for each side of the flow
> + * @tap_omac: MAC address of remote endpoint as seen from the guest
>   */
>  struct flow_common {
>  #ifdef __GNUC__
> @@ -192,6 +193,7 @@ struct flow_common {
>  #endif
>  	uint8_t		pif[SIDES];
>  	struct flowside	side[SIDES];
> +	uint8_t		tap_omac[6];
>  };
>  
>  #define FLOW_INDEX_BITS		17	/* 128k - 1 */
> diff --git a/util.h b/util.h
> index 22eaac5..6fc8f5d 100644
> --- a/util.h
> +++ b/util.h
> @@ -101,6 +101,8 @@ void abort_with_msg(const char *fmt, ...)
>  	((uint8_t [ETH_ALEN]){ 0xff, 0xff, 0xff, 0xff, 0xff, 0xff })
>  #define MAC_ZERO		((uint8_t [ETH_ALEN]){ 0 })
>  #define MAC_IS_ZERO(addr)	(!memcmp((addr), MAC_ZERO, ETH_ALEN))
> +#define MAC_UNDEF		MAC_BROADCAST
> +#define MAC_IS_UNDEF(addr)	(!memcmp((addr), MAC_UNDEF, ETH_ALEN))
>  
>  #ifndef __bswap_constant_16
>  #define __bswap_constant_16(x)						\
> -- 
> 2.50.1
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]