From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=UNQLv18V; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by passt.top (Postfix) with ESMTPS id 3E1BD5A0622 for ; Thu, 27 Feb 2025 05:32:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1740630746; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kKLTJSz5fBH7jB7I4wsZ2vHWkf9deRZ+C+gnu86J2zA=; b=UNQLv18V3RQl2wJ7oQyb0tBnLhFFjOW1NLdjbAvMvyAvM4aPoivi8fG+KsjUBQXykY1cgE p0KzMUaGUQmmMLHpFYER5g3XtnQGfBHCDVINBOfvEcHoDUPldqBlLiPK8S/+cJvvSyLiI5 gUkZuzLbivvrGGwp5zO1BDaI1BW9zZo= Received: from mail-wr1-f71.google.com (mail-wr1-f71.google.com [209.85.221.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-205-ruoL86ciN_e8PTSIgMnMyg-1; Wed, 26 Feb 2025 23:32:24 -0500 X-MC-Unique: ruoL86ciN_e8PTSIgMnMyg-1 X-Mimecast-MFC-AGG-ID: ruoL86ciN_e8PTSIgMnMyg_1740630743 Received: by mail-wr1-f71.google.com with SMTP id ffacd0b85a97d-38f55ccb04bso262343f8f.3 for ; Wed, 26 Feb 2025 20:32:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740630743; x=1741235543; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kKLTJSz5fBH7jB7I4wsZ2vHWkf9deRZ+C+gnu86J2zA=; b=ojMrxbguwMNYgcdG0MpCMeCTp7lD6lMYxTEikIQFNWb9vKhu18g5gOpas7B8wI8+Jk 0KyXccwFc7Kz6hqk+gSI34mOl7D//QgAqGkMyyTNsHNicaNkIfzHRLMxNfnn5D9DQb45 Ue7SzDjJACDBdsaAq7RB6cRsvlF29yJBDUl9gXDEmlGhGskPQxa0CmoW6X21LSUgRkU6 RotWhMbBHMyAJ4xhf/B6XaWjpBOzktK6EWlgNJ/QuJuqtE0j0XqPyDQ3sdvcuZ7pcIdu QgWQ1MOKjGYlV03HaGZ0sAq24ULqZwNEYsEeRn8hcAVrHjxnxhy2PtN3gvPRNIM3pEsU GTwA== X-Gm-Message-State: AOJu0YzjBprNHEsxQrpQUNw3clIZbwkJgThxhn/iGEFM5z0QRZiYwBpP jU06VfgxjzyRXVN3YHO3gAZnA1GaKjwZqOxWlUZQcY+UOxnhtONz5yYxgjnO7cN7ntyUbZLna+K 5dxeQtT0xaOe305R7uEk6Q1pFNlAZooPqXmucry9HqW0eBS4UGw== X-Gm-Gg: ASbGncttIkO951w2fm5N5WUKBfz92WPCC4SqN4/q8Vcf5atvm+duXcbfa/6WGZBo6nA kQkXeIXYeVtv3gk5Vi+fsPuFW4U9V3UtuV3DyNcp1Tb/2+6oOjl5a4hfabmnUB27tiKMBkShLjB pD83ajT/1QqGCy+7hT1BOKATnFdz4Qu92KqCneNc5se6GzJnTrLVCAWgOCenJVNHAqokrcMa9zY yIGs9XkthcrRhVXBDrmqDNzp5LfIx1wFhi0w6k+rjEOrZHV7TujwP8QAdVGXaUZUKCHw1QBh4St IN4e0rQIaLPc9ckWE48Za0JVYpA0IVd+h69L6fX7DK8C X-Received: by 2002:a5d:59ad:0:b0:390:deff:f233 with SMTP id ffacd0b85a97d-390defff408mr2721925f8f.24.1740630743378; Wed, 26 Feb 2025 20:32:23 -0800 (PST) X-Google-Smtp-Source: AGHT+IHXlTLG8EBjqXAPOgkHtIt9YWAxpdpLWLT6Tj3StPJ8dTfwd56TkoGpVvEfRuoPmbOJ8bRw7A== X-Received: by 2002:a5d:59ad:0:b0:390:deff:f233 with SMTP id ffacd0b85a97d-390defff408mr2721915f8f.24.1740630742975; Wed, 26 Feb 2025 20:32:22 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-390e4844c2bsm738253f8f.77.2025.02.26.20.32.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 26 Feb 2025 20:32:22 -0800 (PST) Date: Thu, 27 Feb 2025 05:32:19 +0100 From: Stefano Brivio To: David Gibson Subject: Re: [PATCH v2 0/2] More graceful handling of migration without passt-repair Message-ID: <20250227053219.01e76539@elisabeth> In-Reply-To: References: <20250225055132.3677190-1-david@gibson.dropbear.id.au> <20250225184316.407247f4@elisabeth> <20250226090948.3d1fff91@elisabeth> <20250226122412.33009f77@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: XxrOb3z4yXwIHsD-vmeE8hmK8uj0ehZS8LhqrF8Qq5s_1740630743 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: NZGKO32ZDFDJVT5L52IIP4K3LG2C72AO X-Message-ID-Hash: NZGKO32ZDFDJVT5L52IIP4K3LG2C72AO X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 27 Feb 2025 12:43:41 +1100 David Gibson wrote: > On Wed, Feb 26, 2025 at 12:24:12PM +0100, Stefano Brivio wrote: > > On Wed, 26 Feb 2025 19:51:11 +1100 > > David Gibson wrote: > > > > > On Wed, Feb 26, 2025 at 09:09:48AM +0100, Stefano Brivio wrote: > > > > On Wed, 26 Feb 2025 11:27:32 +1100 > > > > David Gibson wrote: > > > > > > > > > On Tue, Feb 25, 2025 at 06:43:16PM +0100, Stefano Brivio wrote: > > > > > > On Tue, 25 Feb 2025 16:51:30 +1100 > > > > > > David Gibson wrote: > > > > > > > > > > > > > From Red Hat internal testing we've had some reports that if > > > > > > > attempting to migrate without passt-repair, the failure mode is uglier > > > > > > > than we'd like. The migration fails, which is somewhat expected, but > > > > > > > we don't correctly roll things back on the source, so it breaks > > > > > > > network there as well. > > > > > > > > > > > > > > Handle this more gracefully allowing the migration to proceed in this > > > > > > > case, but allow TCP connections to break > > > > > > > > > > > > > > I've now tested this reasonably: > > > > > > > * I get a clean migration if there are now active flows > > > > > > > * Migration completes, although connections are broken if > > > > > > > passt-repair isn't connected > > > > > > > * Basic test suite (minus perf) > > > > > > > > > > > > > > I didn't manage to test with libvirt yet, but I'm pretty convinced the > > > > > > > behaviour should be better than it was. > > > > > > > > > > > > I did, and it is. The series looks good to me and I would apply it as > > > > > > it is, but I'm waiting a bit longer in case you want to try out some > > > > > > variations based on my tests as well. Here's what I did. > > > > > > > > > > [snip] > > > > > > > > > > Thanks for the detailed instructions. More complex than I might have > > > > > liked, but oh well. > > > > > > > > > > > $ virsh migrate --verbose --p2p --live --unsafe alpine --tunneled qemu+ssh://88.198.0.161:10951/session > > > > > > Migration: [97.59 %]error: End of file while reading data: : Input/output error > > > > > > > > > > > > ...despite --verbose the error doesn't tell much (perhaps I need > > > > > > LIBVIRT_DEBUG=1 instead?), but passt terminates at this point. With > > > > > > this series (I just used 'make install' from the local build), migration > > > > > > succeeds instead: > > > > > > > > > > > > $ virsh migrate --verbose --p2p --live --unsafe alpine --tunneled qemu+ssh://88.198.0.161:10951/session > > > > > > Migration: [100.00 %] > > > > > > > > > > > > Now, on the target, I still have to figure out how to tell libvirt > > > > > > to start QEMU and prepare for the migration (equivalent of '-incoming' > > > > > > as we use in our tests), instead of just starting a new instance like > > > > > > it does. Otherwise, I have no chance to start passt-repair there. > > > > > > Perhaps it has something to do with persistent mode described here: > > > > > > > > > > Ah. So I'm pretty sure virsh migrate will automatically start qemu > > > > > with --incoming on the target. > > > > > > > > ("-incoming"), yes, see src/qemu/qemu_migration.c, > > > > qemuMigrationDstPrepare(). > > > > > > > > > IIUC the problem here is more about > > > > > timing: we want it to start it early, so that we have a chance to > > > > > start passt-repair and let it connect before the migration actually > > > > > happens. > > > > > > > > For the timing itself, we could actually wait for passt-repair to be > > > > there, with a timeout (say, 100ms). > > > > > > I guess. That still requires some way for KubeVirt (or whatever) to > > > know at least roughly when it needs to launch passt-repair, and I'm > > > not sure if that's something we can currently get from libvirt. > > > > KubeVirt sets up the target pod, and that's when it should be done (if > > we have an inotify mechanism or similar). I can't point to an exact code > > path yet but there's something like that. > > Right, but that approach does require inotify and starting > passt-repair before passt, which might be fine, but I have the concern > noted below. To avoid that we'd need notification after passt & qemu > are started on the target, but before the migration is actually > initiated which I don't think libvirt provides. > > > > > We could also modify passt-repair to set up an inotify watcher if the > > > > socket isn't there yet. > > > > > > Maybe, yes. This kind of breaks our "passt starts first, passt-repair > > > connects to it" model though, and I wonder if we need to revisit the > > > security implications of that. > > > > I don't think it actually breaks that model for security purposes, > > because the guest doesn't have anyway a chance to cause a connection to > > passt-repair. The guest is still suspended (or missing) at that point. > > I wasn't thinking of threat models coming from the guest, but an > attack from an unrelated process impersonating passt in order to > access passt-repair's superpowers. Then an inotify watch shouldn't substantially change things. The attacker could create the socket earlier and obtain the same outcome. > [...] > > > We could even think of deferring switching repair mode off until the > > right address is there, by the way. That would make a difference to > > us. > > Do you mean by blocking? Or by returning to normal operation with the > flow flagged somehow to be "woken up" by a netlink monitor? The latter. I don't think we should block connectivity (with new addresses) meanwhile. -- Stefano