On Wed, Oct 16, 2024 at 05:26:48PM +0200, Stefano Brivio wrote: > On Wed, 16 Oct 2024 19:39:40 +1100 > David Gibson wrote: > > > On Wed, Oct 16, 2024 at 04:46:52PM +1100, David Gibson wrote: > > > On Wed, Oct 16, 2024 at 02:15:19PM +1100, David Gibson wrote: > > > > On Thu, Oct 10, 2024 at 04:57:32PM +1100, David Gibson wrote: > > > > > On Wed, Oct 09, 2024 at 10:44:33PM +0200, Stefano Brivio wrote: > > > > > > On Wed, 9 Oct 2024 15:07:21 +0200 > > > > > > Stefano Brivio wrote: > > > > [snip] > > > > > > > > @@ -447,20 +447,35 @@ uint8_t fwd_nat_from_host(const struct ctx *c, uint8_t proto, > > > > > > > > (proto == IPPROTO_TCP || proto == IPPROTO_UDP)) { > > > > > > > > /* spliceable */ > > > > > > > > > > > > > > > > - /* Preserve the specific loopback adddress used, but let the > > > > > > > > - * kernel pick a source port on the target side > > > > > > > > + /* The traffic will go over the guest's 'lo' interface, but by > > > > > > > > + * default use its external address, so we don't inadvertently > > > > > > > > + * expose services that listen only on the guest's loopback > > > > > > > > + * address. That can be overridden by --host-lo-to-ns-lo which > > > > > > > > + * will instead forward to the loopback address in the guest. > > > > > > > > + * > > > > > > > > + * In either case, let the kernel pick the source address to > > > > > > > > + * match. > > > > > > > > */ > > > > > > > > - tgt->oaddr = ini->eaddr; > > > > > > > > + if (inany_v4(&ini->eaddr)) { > > > > > > > > + if (c->host_lo_to_ns_lo) > > > > > > > > + tgt->eaddr = inany_loopback4; > > > > > > > > + else > > > > > > > > + tgt->eaddr = inany_from_v4(c->ip4.addr_seen); > > > > > > > > + tgt->oaddr = inany_any4; > > > > > > > > + } else { > > > > > > > > + if (c->host_lo_to_ns_lo) > > > > > > > > + tgt->eaddr = inany_loopback6; > > > > > > > > + else > > > > > > > > + tgt->eaddr.a6 = c->ip6.addr_seen; > > > > > > > > > > > > > > Either this... > > > > > > > > > > > > > > > + tgt->oaddr = inany_any6; > > > > > > > > > > > > > > or this (and not something before this patch, up to 3/4) make the > > > > > > > "TCP/IPv6: host to ns (spliced): big transfer" test in pasta/tcp hang, > > > > > > > sometimes (about one in three/four runs), that's what I mistakenly > > > > > > > reported as coming from Laurent's series at: > > > > > > > > > > Huh, interesting. Just got back from my leave and ran that group of > > > > > tests in a loop this afternoon, but didn't manage to reproduce. I > > > > > have administrivia that will probably fill the rest of this week, but > > > > > I'll look into this as soon as I can. > > > > > > > > I reproduced the problem on passt.top, and I have a partial idea > > > > what's going on. As you say it's seeming like the address (addr_seen > > > > == addr in this case) isn't properly ready. This is over splice, but > > > > on the tap interface, I see the container sending NS messages for its > > > > own address - seems like it's doing DAD. But more importantly, we're > > > > answering those NS messages with NA messages, because we answer all > > > > NS. i.e. we're making the DAD fail. What I'm not sure of is how this > > > > ever worked at all. --config-net makes sense, since we disable DAD, > > > > but our test suite has always been using NDP+DHCP instead of > > > > --config-net. > > > > > > > > So, AFACT, we'll always fail guest DAD attempts, both IPv6, which > > > > happens most of the time and for IPv4 via ARP, which is used much more > > > > rarely. I think we need to be more selective in what NS or ARP > > > > lookups we resopnd to. The question is what approach to take: > > > > > > Hmm... no.. there's more to this. > > > > > > Usually DAD requests have :: as the source address, and we *do* > > > exclude those from getting replies. In this case though, we're > > > getting NS requests for the assigned address from what looks like the > > > SLAAC address. So, I do think it would be wise to explicitly exclude > > > these: we shouldn't be giving NA responses for an address that ought > > > to belong to the guest, even if it doesn't look like a DAD. > > > > > > But, I'm not sure what's triggering this. Is for some reason the DHCP > > > address not "taking", so the container is trying to locate it on the > > > network instead? Or _is_ this DAD, but under some circumstances > > > rather than using :: as the source address it uses another configured > > > address. > > > > Ok.. I've understood a bit more. While timing is a factor here, it > > looks like the main reason I wasn't seeing it on my machine is what > > I'd consider a bug in the Debian version of the dhclient-script: > > when adding an IPv6 address, it returns without waiting for DAD to > > complete (i.e. for the address to be non-tentative). > > Oops. On one hand, I would feel inclined to propose a fix for the > Debian and Ubuntu packages. On the other hand, I wonder if it's > universally considered a bug: the DHCPv6 client did its job at that > point, and it's debatable whether dhclient should wait for the address > to be usable before forking to background. > > That is, arguably, the job of dhclient's is to request and configure an > address. It's not a network configuration daemon. There might be many > other reasons why that address is unusable, and yet dhclient is not > responsible for them. Hrm... I guess. Counterpoints.. - Most other failures to get a usable address will result in a visible error - dhclient has a --dad-wait-time option which seems to imply that the script should wait for DAD - The upstream script version waits for DAD In any case I filed a report for it https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085231 > By the way, I guess it's just an issue for test scripts like this one. Why do you guess that? > > There's also an additional bug, which doesn't cause this problem, I > > think, but caused some problems when I was investigating. DHCPv6 > > needs the link-local SLAAC address already configured and > > non-tentative. The Fedora dhclient-script waits for that too at the > > PREINIT6 stage, but the Debian one doesn't, meaning if you attempt > > dhclient -6 immediately after starting the namespace it will fail to > > bind the UDP address it needs. > > Right, and that's fine for us because we have a 2-second delay after > SLAAC. This looks to me a bit more like a real bug, but again, there > might be many other reasons why dhclient can't use a link-local > address. One might argue that it's the responsibility of the > user/caller to invoke dhclient when appropriate. Here I think it's a much clearer argument that it's a real bug. We play fast and loose with it for mbuto, but dhclient can typically be called on an interface that isn't even up: the PREINIT/PREINIT6 script actions are specifically for this, they'll bring the interface up ready for the client to do its thing. I'd say the script is failing to do its job for PREINIT6 if there isn't a usable link-local address at the end. I filed a report: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1085229 > In that sense, you might be wondering why there's a 2-second delay > after SLAAC, but no delay after invoking dhclient -6: the reason is > that I was convinced that one wouldn't need DAD once a DHCPv6 client > configures an address. The server is already checking that, I thought. > > Well, no. RFC 8415 18.2.10.1: > > https://datatracker.ietf.org/doc/html/rfc8415#section-18.2.10.1 > > says: > > If the client can operate with the addresses and/or prefixes obtained > from the server: > > [...] > > - The client MUST perform duplicate address detection as per > Section 5.4 of [RFC4862], which does list some exceptions, on each > of the received addresses in any IAs on which it has not performed > duplicate address detection during processing of any of the > previous Reply messages from the server. The client performs the > duplicate address detection before using the received addresses > for any traffic. If any of the addresses are found to be in use > on the link, the client sends a Decline message to the server for > those addresses as described in Section 18.2.8. Indeed. > > I still think it's a good idea not to give NA messages for the guest > > assigned address, but we'll need a different workaround for this > > issue. > > I read the rest of your reasoning about it, but the nice thing of the > current behaviour (and that's why I added that single check on the > source address in ndp()) is that the guest can really use whatever > address it wants, regardless of what we tried to configure, and we'll > resolve any other address. Hrm. I suppose. Fwiw we already make the equivalent exclusion for ARP > If we receive a neighbour solicitation for the guest assigned address, > and the source address is not unspecified, it means that the guest is > _not_ using the assigned address, and it's actually trying to reach it. > > > I guess we'll have to manually wait for DAD to complete in the > > DHCP tests, which will be kind of mucky :/ > > Alternatively, we could use the same trick as added by commit > f4e9f26480ef ("pasta: Disable neighbour solicitations on device up > to prevent DAD"): disable neighbour solicitations, run dhclient -6, > set 'nodad' on the address, and re-enable neighbour solicitations. > > This works for me: Ok. More complex, but faster, I guess. I'll try implementing this. -- David Gibson (he or they) | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you, not the other way | around. http://www.ozlabs.org/~dgibson