Re: [PATCH v3] treewide: By default, don't quit source after migration, keep sockets open

[David Gibson <david@gibson.dropbear.id.au>]

[-- Attachment #1: Type: text/plain, Size: 5372 bytes --]

On Fri, Jul 25, 2025 at 10:21:12AM +0200, Stefano Brivio wrote:
> On Fri, 25 Jul 2025 16:50:23 +1000
> David Gibson <david@gibson.dropbear.id.au> wrote:
> > On Fri, Jul 25, 2025 at 07:10:58AM +0200, Stefano Brivio wrote:
> > > On Fri, 25 Jul 2025 14:04:17 +1000
> > > David Gibson <david@gibson.dropbear.id.au> wrote:
[snip]
> > > > > v3:
> > > > >   - Nir reported occasional failures (connections being reset)
> > > > >     with both v1 and v2, because, in KubeVirt's usage, we quit as
> > > > >     QEMU exits. Disable --one-off after migration as source, and
> > > > >     document this exception    
> > > > 
> > > > This seems like an awful, awful hack.  
> > > 
> > > Well, of course, it is, and long term it should be fixed in
> > > either KubeVirt or libvirt (even though I'm not sure how, see below)
> > > instead.  
> > 
> > But this hack means that even when it's fixed we'll still have this
> > wildly counterintuitive behaviour that every future user will have to
> > work around.
> 
> No, why? We can change that as well. We changed semantics of options
> in the past and I don't see an issue doing that as long as we
> coordinate things to a reasonable extent (like we do with Podman and
> rootlesskit and with distributions and LSMs...).

Ah, ok.  That kind of changes everything.  It thought this indicated
commiting to these semantics indefinitely.  I think the co-ordination
you're suggesting may be messier than you think, but if we're
explicitly willing to (technically) break compatibility to remove this
again, then, sure, ok.  It's still gross, but fine.

[snip]
> > > We found out that the guest is generally suspended during that while,
> > > but sometimes it might even have already exited. The pod remains,
> > > though, as long as it's needed. That's the only certainty we have.  
> > 
> > Keeping the pod around is fine.  What needs to change is that the
> > guest's IP(s) needs to be removed from the source host before qemu
> > (and therefore passt) is terminated.  The pod must have at least one
> > other IP, or it would be impossible to perform the migration in the
> > first place.
> 
> Maybe, yes. I'm not sure if it's doable.

KubeVirt and/or libvirt really need to figure out how to make it
doable, because having two simultaneously active things with the same
IP on the same network is bound to cause trouble.  At some point it's
likely to cause trouble that we can't hack our way around in passt.

> > This essentially matches the situation for bridged networking: with
> > the source guest suspended the source host will no longer respond to
> > the guest IP
> > 
> > > So, do we want to drop --one-off from the libvirt integration, and have
> > > libvirt manage passt's lifecycle entirely (note that all users outside
> > > KubeVirt don't use migration, so we would make the general case vastly
> > > more complicated for the sake of correctness for a single usage...)?  
> > 
> > Hmm.. if I understand correctly the network swizzling is handled by
> > KubeVirt, not libvirt.
> 
> That's OVN-Kubernetes in KubeVirt's case.

To clarify, I'm not talking so much about what actually makes the
network arrangements, but what component initiates the changeover.

> 
> > I'm hoping that means there's a suitable point
> > at which it can remove the IP without having to alter libvirt.
> 
> I hope so too, eventually. Or we could make sure that QEMU is alive as
> long as needed, this is probably easier to ensure from virt-launcher.

Right.  There's kind of a basic mismatch here - libvirt aims to manage
the whole migration, end-to-end, so you just tell it to migrate and it
does.  But in the k8s context, something else needs to do network
jiggery-pokery co-ordinated with that, so it kind of breaks the model.

passt adds a further complication in that the host owns the IP, to
share it with the guest.  That means suspending the guest is no longer
sufficient to prevent that source responding on that IP.

So, maybe that means libvirt logically ought to be handling the IP
assignment/de-assignment when migrating with passt.  But... no doubt
that would conflict with OVN-Kubernetes wanting to own the network
configuration.  Yet again, libvirt's wish to manage everything (in the
manner of a Xen machine circa 2010) does not play well with the kind
of circumstances people try to use it.

> I haven't looked at the details yet, but in passt it's one line and we
> can drop it later as needed, in KubeVirt it's probably much more
> complicated than that.
> 
> > > Well, we can try to do that. Except that libvirt doesn't know either
> > > for how long this traffic will reach the source pod (that's a KubeVirt
> > > concept). So it should implement the same hack: let it outlive QEMU on
> > > migration... as long as we have that issue in KubeVirt.
> > > 
> > > But I asked KubeVirt people, and it turns out that it's extremely
> > > complicated to fix this in KubeVirt. So, actually, I don't see another
> > > way to fix this in the short term. And without KubeVirt using this we
> > > could also drop the whole feature...
> 

-- 
David Gibson (he or they)	| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you, not the other way
				| around.
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]