From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=JxAenQ8K; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id C71465A0272 for ; Wed, 05 Feb 2025 06:47:18 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738734437; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SlvwxSCTzp02ysVm6eHRxSiu9Z+5k0MMyRR01l7vnIY=; b=JxAenQ8K7oSSEkLECTZtqrDNviWDOhB7tEjex/lcehQNGahEj5XXqJcSlGU6wdGrgpyNJO UV59LbwM0Z4mmhsStfFGuCD4TLu5W5ZCMPdle4INDw+eF7kAioxZtLU9PgT9g0U7+nW7vR TsepruDbjuVlAcTFgESfmn4hTEJazxU= Received: from mail-wm1-f71.google.com (mail-wm1-f71.google.com [209.85.128.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-20-Y-8Dw9OxOIivtaaGvd52Lg-1; Wed, 05 Feb 2025 00:47:15 -0500 X-MC-Unique: Y-8Dw9OxOIivtaaGvd52Lg-1 X-Mimecast-MFC-AGG-ID: Y-8Dw9OxOIivtaaGvd52Lg Received: by mail-wm1-f71.google.com with SMTP id 5b1f17b1804b1-4362b9c1641so32792815e9.3 for ; Tue, 04 Feb 2025 21:47:15 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738734434; x=1739339234; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=SlvwxSCTzp02ysVm6eHRxSiu9Z+5k0MMyRR01l7vnIY=; b=Dt/HcRsyZdHNfb3hMYP/HArHTj+U7pb/Xs2ZzXJnyvsUUhlfzJakavxnlg5oHz7HHl d9ungUjedmt54jjddzpNo7g2Tsdn7dtn+3fONk46XVXTtzK5cxuNMKOGRlP05/ixIW2O +cV3NisB6WY0NB2m0xtxfFSmKUpvLB5z9LohPRnvcC4s4jUzM4++EBoztrAlyRz6Kzbq wDA7XoML187hT+gh7lUFnNPBKWAEr0TIPnSav26iuUff3NECp7iKX/wfTTN9YwQWnPe0 olUPHjzWDrgrxvNwfuZyio9+gwQ7SWtysJpogGMC/Bt4cuw0daC0OwQ6u2pKUrxB7ZlE Ba3g== X-Forwarded-Encrypted: i=1; AJvYcCUTneiPZX9eqOoOOGU6q4BI/z28sMCIAT6PERR9TKXZRD61acJcCDvpKHibQ8iixWWcM6Q3bs5Lol4=@passt.top X-Gm-Message-State: AOJu0YwPXwP6ZzpQyiBHi/kYaD6LX2JBxWZajFIPDLcwUyuBXJITbi3D SKTREPDs6zaZEL4zI/ECf1giR4y94JiUJovFKM8JRphhZfKl57gyNK5b65jgTrY+KlFd3PBXLaJ yFGWOgjFEc6RcQa8RBcoKm17ii/hangwVisqMP8IT1VOIJV23dQ== X-Gm-Gg: ASbGncuR2h+ZBe0XWaD9WpH3/z8XTpFuqI7m9lA7jLpbiJoMChS8uy2N5pqXUx4Fag1 wBQcPefqsdL4mHtIjy7FSLDMZZL44FD3onbdAiu17oS/rrLCym7kt4Gg8yPhot/UhNliu8Fj1dp pyRHsYtFfN23JQZXRnAjONPv8UGRBgdvZKqhx5rC9hhoQeHLjMtP8jebnwnZ7l7UhyEXFFggPv4 KS9DXGV3fUen8UEa1cJXkLsRzDN4sAMr4tMJZxnzZrPypyL70vUZdjZW+A7fZibLIWTR2bYe96f vyxsKkXuc2Wop19u X-Received: by 2002:a05:600c:294:b0:434:9d62:aa23 with SMTP id 5b1f17b1804b1-4390e050da9mr5184675e9.20.1738734434239; Tue, 04 Feb 2025 21:47:14 -0800 (PST) X-Google-Smtp-Source: AGHT+IGjeCUenzyrV1lhk/diZYyWhqedWZM9Cn60Ta2oWfJXXN3g2b/3HH3S1nzgdwhhMrgB35lv/A== X-Received: by 2002:a05:600c:294:b0:434:9d62:aa23 with SMTP id 5b1f17b1804b1-4390e050da9mr5184535e9.20.1738734433830; Tue, 04 Feb 2025 21:47:13 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [2a10:fc81:a806:d6a9::1]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4390d94d7c7sm9666285e9.14.2025.02.04.21.47.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 04 Feb 2025 21:47:12 -0800 (PST) Date: Wed, 5 Feb 2025 06:47:08 +0100 From: Stefano Brivio To: German Maglione , Hanna Czenczek Subject: Re: [PATCH v5 5/6] vhost_user: Make source quit after reporting migration state Message-ID: <20250205064708.1af3ef28@elisabeth> In-Reply-To: References: <20250205003904.2797491-1-sbrivio@redhat.com> <20250205003904.2797491-6-sbrivio@redhat.com> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: ebp_V4WktXPw7SjpS1nazHQ5YkDIGR5WeLGyjXoyYE0_1738734434 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: C4VPY3QOHSHX5XJ2SS2EEIN34RELLQUK X-Message-ID-Hash: C4VPY3QOHSHX5XJ2SS2EEIN34RELLQUK X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: David Gibson , passt-dev@passt.top, Laurent Vivier X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: To: German whom I feel okay to To: now that https://github.com/kubevirt/kubevirt/pull/13756 is merged, and Hanna who knows one thing or two about vhost-user based migration. This is Stefano from your neighbouring mailing list, passt-dev. David is wondering: On Wed, 5 Feb 2025 13:09:42 +1100 David Gibson wrote: > On Wed, Feb 05, 2025 at 01:39:03AM +0100, Stefano Brivio wrote: > > On migration, the source process asks passt-helper to set TCP sockets > > in repair mode, dumps the information we need to migrate connections, > > and closes them. > > > > At this point, we can't pass them back to passt-helper using > > SCM_RIGHTS, because they are closed, from that perspective, and > > sendmsg() will give us EBADF. But if we don't clear repair mode, the > > port they are bound to will not be available for binding in the > > target. > > > > Terminate once we're done with the migration and we reported the > > state. This is equivalent to clearing repair mode on the sockets we > > just closed. > > As we've discussed, quitting still makes sense, but the description > above is not really accurate. Perhaps, > > === > Once we've passed the migration's "point of no return", there's no way > to resume the guest on the source side, because we no longer own the > connections. There's not really anything we can do except exit. > === > > Except.. thinking about it, I'm not sure that's technically true. > After migration, the source qemu enters a kind of limbo state. I > suppose for the case of to-disk migration (savevm) the guest can > actually be resumed. Which for us is not really compatible with > completing at least a local migration properly. Not really sure what > to do about that. > > I think it's also technically possible to use monitor commands to boot > up essentially an entirely new guest instance in the original qemu, > in which case for us it would make sense to basically reset ourselves > (flush the low table). > > Hrm.. we really need to know the sequence of events in a bit more > detail to get this right (not that this stops improving the guts of > the logic in the meantime). > > I'm asking around to see if I can find who did the migration stuff or > virtiofsd, so we can compare notes. ...what we should be doing in the source passt at different stages of moving our TCP connections (or failing to move them) over to the target, which might be inspired by what you're doing with your... filesystem things in virtiofsd. We're taking for granted that as long as we have a chance to detect failure (e.g. we can't dump sequence numbers from a TCP socket in the source), we should use that to abort the whole thing. Once we're past that point, we have several options. And actually, that's regardless of failure: because we also have the question of what to do if we see that nothing went wrong. We can exit in the source, for example (this is what patch implements): wait for VHOST_USER_CHECK_DEVICE_STATE, report that, and quit. Or we can just clear up all our connections and resume (start from a blank state). Or do that, only if we didn't smell failure. Would you have some pointers, general guidelines, ideas? I know that the topic is a bit broad, but I'm hopeful that you have a lot of clear answers for us. :) Thanks. > > Signed-off-by: Stefano Brivio > > --- > > vhost_user.c | 11 +++++++++++ > > 1 file changed, 11 insertions(+) > > > > diff --git a/vhost_user.c b/vhost_user.c > > index b107d0f..70773d6 100644 > > --- a/vhost_user.c > > +++ b/vhost_user.c > > @@ -997,6 +997,8 @@ static bool vu_send_rarp_exec(struct vu_dev *vdev, > > return false; > > } > > > > +static bool quit_on_device_state = false; > > + > > /** > > * vu_set_device_state_fd_exec() - Set the device state migration channel > > * @vdev: vhost-user device > > @@ -1024,6 +1026,9 @@ static bool vu_set_device_state_fd_exec(struct vu_dev *vdev, > > migrate_request(vdev->context, msg->fds[0], > > direction == VHOST_USER_TRANSFER_STATE_DIRECTION_LOAD); > > > > + if (direction == VHOST_USER_TRANSFER_STATE_DIRECTION_SAVE) > > + quit_on_device_state = true; > > + > > /* We don't provide a new fd for the data transfer */ > > vmsg_set_reply_u64(msg, VHOST_USER_VRING_NOFD_MASK); > > > > @@ -1201,4 +1206,10 @@ void vu_control_handler(struct vu_dev *vdev, int fd, uint32_t events) > > > > if (reply_requested) > > vu_send_reply(fd, &msg); > > + > > + if (quit_on_device_state && > > + msg.hdr.request == VHOST_USER_CHECK_DEVICE_STATE) { > > + info("Migration complete, exiting"); > > + exit(EXIT_SUCCESS); > > + } > > } -- Stefano