Re: [PATCH v3 4/4] fwd: Direct inbound spliced forwards to the guest's external address

[Stefano Brivio <sbrivio@redhat.com>]

On Wed, 16 Oct 2024 19:39:40 +1100
David Gibson <david@gibson.dropbear.id.au> wrote:

> On Wed, Oct 16, 2024 at 04:46:52PM +1100, David Gibson wrote:
> > On Wed, Oct 16, 2024 at 02:15:19PM +1100, David Gibson wrote:  
> > > On Thu, Oct 10, 2024 at 04:57:32PM +1100, David Gibson wrote:  
> > > > On Wed, Oct 09, 2024 at 10:44:33PM +0200, Stefano Brivio wrote:  
> > > > > On Wed, 9 Oct 2024 15:07:21 +0200
> > > > > Stefano Brivio <sbrivio@redhat.com> wrote:  
> > > [snip]  
> > > > > > > @@ -447,20 +447,35 @@ uint8_t fwd_nat_from_host(const struct ctx *c, uint8_t proto,
> > > > > > >  	    (proto == IPPROTO_TCP || proto == IPPROTO_UDP)) {
> > > > > > >  		/* spliceable */
> > > > > > >  
> > > > > > > -		/* Preserve the specific loopback adddress used, but let the
> > > > > > > -		 * kernel pick a source port on the target side
> > > > > > > +		/* The traffic will go over the guest's 'lo' interface, but by
> > > > > > > +		 * default use its external address, so we don't inadvertently
> > > > > > > +		 * expose services that listen only on the guest's loopback
> > > > > > > +		 * address.  That can be overridden by --host-lo-to-ns-lo which
> > > > > > > +		 * will instead forward to the loopback address in the guest.
> > > > > > > +		 *
> > > > > > > +		 * In either case, let the kernel pick the source address to
> > > > > > > +		 * match.
> > > > > > >  		 */
> > > > > > > -		tgt->oaddr = ini->eaddr;
> > > > > > > +		if (inany_v4(&ini->eaddr)) {
> > > > > > > +			if (c->host_lo_to_ns_lo)
> > > > > > > +				tgt->eaddr = inany_loopback4;
> > > > > > > +			else
> > > > > > > +				tgt->eaddr = inany_from_v4(c->ip4.addr_seen);
> > > > > > > +			tgt->oaddr = inany_any4;
> > > > > > > +		} else {
> > > > > > > +			if (c->host_lo_to_ns_lo)
> > > > > > > +				tgt->eaddr = inany_loopback6;
> > > > > > > +			else
> > > > > > > +				tgt->eaddr.a6 = c->ip6.addr_seen;    
> > > > > > 
> > > > > > Either this...
> > > > > >   
> > > > > > > +			tgt->oaddr = inany_any6;    
> > > > > > 
> > > > > > or this (and not something before this patch, up to 3/4) make the
> > > > > > "TCP/IPv6: host to ns (spliced): big transfer" test in pasta/tcp hang,
> > > > > > sometimes (about one in three/four runs), that's what I mistakenly
> > > > > > reported as coming from Laurent's series at:  
> > > > 
> > > > Huh, interesting.  Just got back from my leave and ran that group of
> > > > tests in a loop this afternoon, but didn't manage to reproduce.  I
> > > > have administrivia that will probably fill the rest of this week, but
> > > > I'll look into this as soon as I can.  
> > > 
> > > I reproduced the problem on passt.top, and I have a partial idea
> > > what's going on.  As you say it's seeming like the address (addr_seen
> > > == addr in this case) isn't properly ready.  This is over splice, but
> > > on the tap interface, I see the container sending NS messages for its
> > > own address - seems like it's doing DAD.  But more importantly, we're
> > > answering those NS messages with NA messages, because we answer all
> > > NS.  i.e. we're making the DAD fail.  What I'm not sure of is how this
> > > ever worked at all.  --config-net makes sense, since we disable DAD,
> > > but our test suite has always been using NDP+DHCP instead of
> > > --config-net.
> > > 
> > > So, AFACT, we'll always fail guest DAD attempts, both IPv6, which
> > > happens most of the time and for IPv4 via ARP, which is used much more
> > > rarely.  I think we need to be more selective in what NS or ARP
> > > lookups we resopnd to.  The question is what approach to take:  
> > 
> > Hmm... no.. there's more to this.
> > 
> > Usually DAD requests have :: as the source address, and we *do*
> > exclude those from getting replies.  In this case though, we're
> > getting NS requests for the assigned address from what looks like the
> > SLAAC address.  So, I do think it would be wise to explicitly exclude
> > these: we shouldn't be giving NA responses for an address that ought
> > to belong to the guest, even if it doesn't look like a DAD.
> > 
> > But, I'm not sure what's triggering this.  Is for some reason the DHCP
> > address not "taking", so the container is trying to locate it on the
> > network instead?  Or _is_ this DAD, but under some circumstances
> > rather than using :: as the source address it uses another configured
> > address.  
> 
> Ok.. I've understood a bit more.  While timing is a factor here, it
> looks like the main reason I wasn't seeing it on my machine is what
> I'd consider a bug in the Debian version of the dhclient-script:
> when adding an IPv6 address, it returns without waiting for DAD to
> complete (i.e. for the address to be non-tentative).

Oops. On one hand, I would feel inclined to propose a fix for the
Debian and Ubuntu packages. On the other hand, I wonder if it's
universally considered a bug: the DHCPv6 client did its job at that
point, and it's debatable whether dhclient should wait for the address
to be usable before forking to background.

That is, arguably, the job of dhclient's is to request and configure an
address. It's not a network configuration daemon. There might be many
other reasons why that address is unusable, and yet dhclient is not
responsible for them.

By the way, I guess it's just an issue for test scripts like this one.

> There's also an additional bug, which doesn't cause this problem, I
> think, but caused some problems when I was investigating.  DHCPv6
> needs the link-local SLAAC address already configured and
> non-tentative.  The Fedora dhclient-script waits for that too at the
> PREINIT6 stage, but the Debian one doesn't, meaning if you attempt
> dhclient -6 immediately after starting the namespace it will fail to
> bind the UDP address it needs.

Right, and that's fine for us because we have a 2-second delay after
SLAAC. This looks to me a bit more like a real bug, but again, there
might be many other reasons why dhclient can't use a link-local
address. One might argue that it's the responsibility of the
user/caller to invoke dhclient when appropriate.

In that sense, you might be wondering why there's a 2-second delay
after SLAAC, but no delay after invoking dhclient -6: the reason is
that I was convinced that one wouldn't need DAD once a DHCPv6 client
configures an address. The server is already checking that, I thought.

Well, no. RFC 8415 18.2.10.1:

  https://datatracker.ietf.org/doc/html/rfc8415#section-18.2.10.1

says:

   If the client can operate with the addresses and/or prefixes obtained
   from the server:

   [...]

   -  The client MUST perform duplicate address detection as per
      Section 5.4 of [RFC4862], which does list some exceptions, on each
      of the received addresses in any IAs on which it has not performed
      duplicate address detection during processing of any of the
      previous Reply messages from the server.  The client performs the
      duplicate address detection before using the received addresses
      for any traffic.  If any of the addresses are found to be in use
      on the link, the client sends a Decline message to the server for
      those addresses as described in Section 18.2.8.

> I still think it's a good idea not to give NA messages for the guest
> assigned address, but we'll need a different workaround for this
> issue.

I read the rest of your reasoning about it, but the nice thing of the
current behaviour (and that's why I added that single check on the
source address in ndp()) is that the guest can really use whatever
address it wants, regardless of what we tried to configure, and we'll
resolve any other address.

If we receive a neighbour solicitation for the guest assigned address,
and the source address is not unspecified, it means that the guest is
_not_ using the assigned address, and it's actually trying to reach it.

> I guess we'll have to manually wait for DAD to complete in the
> DHCP tests, which will be kind of mucky :/

Alternatively, we could use the same trick as added by commit
f4e9f26480ef ("pasta: Disable neighbour solicitations on device up
to prevent DAD"): disable neighbour solicitations, run dhclient -6,
set 'nodad' on the address, and re-enable neighbour solicitations.

This works for me:

--
diff --git a/test/pasta/dhcp b/test/pasta/dhcp
index 41556b8..76aa723 100644
--- a/test/pasta/dhcp
+++ b/test/pasta/dhcp
@@ -34,9 +34,12 @@ nsout	MTU ip -j link show | jq -rM '.[] | select(.ifname == "__IFNAME__").mtu'
 check	[ __MTU__ = 65520 ]
 
 test	DHCPv6: address
+ns	ip link set dev __IFNAME__ arp off
 ns	/sbin/dhclient -6 --no-pid __IFNAME__
 hout	HOST_IFNAME6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").dev] | .[0]'
 nsout	ADDR6 ip -j -6 addr show|jq -rM '[.[] | select(.ifname == "__IFNAME__").addr_info[] | select(.prefixlen == 128).local] | .[0]'
+ns	ip addr change __ADDR6__ dev __IFNAME__ nodad
+ns	ip link set dev __IFNAME__ arp on
 hout	HOST_ADDR6 ip -j -6 addr show|jq -rM '[.[] | select(.ifname == "__HOST_IFNAME6__").addr_info[] | select(.scope == "global" and .deprecated != true).local] | .[0]'
 check	[ __ADDR6__ = __HOST_ADDR6__ ]
 
--

Adding 2-second delays as we have them for NDP doesn't look that bad:

$ grep --exclude-dir=demo -rn "dhclient -6"
pasta/dhcp:37:ns	/sbin/dhclient -6 --no-pid __IFNAME__
passt_in_ns/dhcp:54:guest	/sbin/dhclient -6 __IFNAME__
passt/dhcp:51:guest	/sbin/dhclient -6 __IFNAME__
perf/passt_tcp:117:guest	dhclient -6 -x
perf/passt_tcp:118:guest	dhclient -6 __IFNAME__
two_guests/basic:40:guest1	/sbin/dhclient -6 __IFNAME1__
two_guests/basic:41:guest2	/sbin/dhclient -6 __IFNAME2__

given that we don't need it on dhclient -x, tests would take about
12 seconds longer.

Or we could switch to the arp off / nodad / arp on approach for
everything, including SLAAC:

$ grep --exclude-dir=demo --exclude-dir=mbuto --exclude-dir=distro --exclude-dir=memory -rn "sleep[ $(printf '\t')].*2"
pasta/ndp:21:sleep	2
passt_in_ns/icmp:29:ns	ip addr add 2001:db8::1 dev __IFNAME_NS__ && sleep 2 # DAD
passt/ndp:19:guest	ip link set dev __IFNAME__ up && sleep 2
two_guests/basic:39:sleep	2

and save slightly less than 8 seconds. If you ask me, I would have a
slight preference for the nodad approach.

-- 
@@ -34,9 +34,12 @@ nsout	MTU ip -j link show | jq -rM '.[] | select(.ifname == "__IFNAME__").mtu'
 check	[ __MTU__ = 65520 ]
 
 test	DHCPv6: address
+ns	ip link set dev __IFNAME__ arp off
 ns	/sbin/dhclient -6 --no-pid __IFNAME__
 hout	HOST_IFNAME6 ip -j -6 route show|jq -rM '[.[] | select(.dst == "default").dev] | .[0]'
 nsout	ADDR6 ip -j -6 addr show|jq -rM '[.[] | select(.ifname == "__IFNAME__").addr_info[] | select(.prefixlen == 128).local] | .[0]'
+ns	ip addr change __ADDR6__ dev __IFNAME__ nodad
+ns	ip link set dev __IFNAME__ arp on
 hout	HOST_ADDR6 ip -j -6 addr show|jq -rM '[.[] | select(.ifname == "__HOST_IFNAME6__").addr_info[] | select(.scope == "global" and .deprecated != true).local] | .[0]'
 check	[ __ADDR6__ = __HOST_ADDR6__ ]
 
--

Adding 2-second delays as we have them for NDP doesn't look that bad:

$ grep --exclude-dir=demo -rn "dhclient -6"
pasta/dhcp:37:ns	/sbin/dhclient -6 --no-pid __IFNAME__
passt_in_ns/dhcp:54:guest	/sbin/dhclient -6 __IFNAME__
passt/dhcp:51:guest	/sbin/dhclient -6 __IFNAME__
perf/passt_tcp:117:guest	dhclient -6 -x
perf/passt_tcp:118:guest	dhclient -6 __IFNAME__
two_guests/basic:40:guest1	/sbin/dhclient -6 __IFNAME1__
two_guests/basic:41:guest2	/sbin/dhclient -6 __IFNAME2__

given that we don't need it on dhclient -x, tests would take about
12 seconds longer.

Or we could switch to the arp off / nodad / arp on approach for
everything, including SLAAC:

$ grep --exclude-dir=demo --exclude-dir=mbuto --exclude-dir=distro --exclude-dir=memory -rn "sleep[ $(printf '\t')].*2"
pasta/ndp:21:sleep	2
passt_in_ns/icmp:29:ns	ip addr add 2001:db8::1 dev __IFNAME_NS__ && sleep 2 # DAD
passt/ndp:19:guest	ip link set dev __IFNAME__ up && sleep 2
two_guests/basic:39:sleep	2

and save slightly less than 8 seconds. If you ask me, I would have a
slight preference for the nodad approach.

-- 
Stefano