From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=I1X9vMQY; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 1F4275A061E for ; Fri, 31 Jan 2025 06:46:32 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1738302390; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kbNcjMLHxARiZCg3i9Jq6NlAn7qq/dgVD9TYZH4wnxU=; b=I1X9vMQY9Wh25PlC0Cy5aqG8aJCyJ/+nc9Ow3R1Zd6mLVsUjhz3QT9P5xSDPVW40Sg6PQp JbpSlO+O6woJVr2QPRBUZY2QP5RdpL68PBJnAyNhTL9Vny5gUyGmRknTSVQMdi7iuTYoU4 3dO6fe4pnEoPN5Dsij/YlqiUC6MsotY= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-168-H3PDL3wFMke3pxJp6wCV7g-1; Fri, 31 Jan 2025 00:46:29 -0500 X-MC-Unique: H3PDL3wFMke3pxJp6wCV7g-1 X-Mimecast-MFC-AGG-ID: H3PDL3wFMke3pxJp6wCV7g Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-43631d8d9c7so7327355e9.1 for ; Thu, 30 Jan 2025 21:46:28 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738302386; x=1738907186; h=content-transfer-encoding:mime-version:organization:references :in-reply-to:message-id:subject:cc:to:from:date:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=kbNcjMLHxARiZCg3i9Jq6NlAn7qq/dgVD9TYZH4wnxU=; b=GbucKEd4KJkrMxrz6gT9DXE0OcqgriSgsL8VmNq4BzfxVATLIrgROaNWhuG2eSPLwo 3Wnl3Ar3T/vfkvSgIi9bc6l9TvtVuZMc8o9Xwce6djCVpwuzV3qFudyf9SftMj68CyRN UsB4PmdX3v3tqhiH6CojnJKofCHLa/DEd+UibkgvYZtTlJhSr0ftiGTddCXnWuikzW+y oV1HYrBbwVkSCZPilycMWDctLywr3Kf3i18MotDp43Bz5ZoRa7WFrcT50gxP1UgvIvGu RCF4MdgxZi4TAQdzdpwpjA8ifKdCvwWMGHWTOkX6HvtMGYsL6Iy6M0exKEX2Z4qiOHKc qzLg== X-Gm-Message-State: AOJu0Yzfk5Vkb1ooLWXysg7N9XxFGDPEP9G/NyFwGNZaKujtHpK72lkK QHZGICAgQRwRFY0XBnU4TyS0gBxrnpdpmiTs/YS7dZvLHLejwcHrGPgYfhR68z4gcriHtGKEv+V T1BlWCEs7VmTZDN5wgIuEJMBpN+hwLCcHUf/k8DtEGUKlJSfcUsWahjP2ZA== X-Gm-Gg: ASbGncsTez0ZmdOfDKfSuD/CzrgZNn66nq0xgxgvFbtKiaeuYKEyc48C5K2kJI1xPWU 1kXuT/zbMoUYot+xInMPHneWg07tLtzIjtsmnmbNXehF721pLsqJhkmkgSzzuZJ9dCkM9XhZaKa U6tq5j1Vl5k8tCSPtMVxoe3mZ4Y1gTeLms4GSTWQIc0mytD66+BB9VHG/G0jO5kZMVYxkqKU/BW VA++b2GUY18b5i0naIxGclSazpNxvApwQeQzkHy9ir3h0xOY0Tw5bXHCWklpxeZoUx2cQw57XJ6 HdigLLMmYWpzUYU8daHYIca3QANGBzAEow== X-Received: by 2002:a7b:ce17:0:b0:434:e65e:457b with SMTP id 5b1f17b1804b1-438ef87af53mr3982735e9.3.1738302386479; Thu, 30 Jan 2025 21:46:26 -0800 (PST) X-Google-Smtp-Source: AGHT+IFn9su6Q0ooR1OrKA/Hzka4wFfNJDT1/Y8EWvmPcC8jAUncMsbK60a1gbIw/fI7EaY0RdvpGw== X-Received: by 2002:a7b:ce17:0:b0:434:e65e:457b with SMTP id 5b1f17b1804b1-438ef87af53mr3982615e9.3.1738302386079; Thu, 30 Jan 2025 21:46:26 -0800 (PST) Received: from maya.myfinge.rs (ifcgrfdd.trafficplex.cloud. [176.103.220.4]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c5c1cf853sm3680809f8f.84.2025.01.30.21.46.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 Jan 2025 21:46:24 -0800 (PST) Date: Fri, 31 Jan 2025 06:46:21 +0100 From: Stefano Brivio To: David Gibson Subject: Re: [PATCH 6/7] Introduce facilities for guest migration on top of vhost-user infrastructure Message-ID: <20250131064621.6cddad7a@elisabeth> In-Reply-To: References: <20250127231532.672363-1-sbrivio@redhat.com> <20250127231532.672363-7-sbrivio@redhat.com> <20250128075001.3557d398@elisabeth> <20250129083350.220a7ab0@elisabeth> <20250130055522.39acb265@elisabeth> <20250130093236.117c3fd0@elisabeth> Organization: Red Hat X-Mailer: Claws Mail 4.2.0 (GTK 3.24.41; x86_64-pc-linux-gnu) MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: EO0eBglvTlnjAQ7f61R5bSUHi19xEs-R3PGj0xCdojk_1738302388 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Message-ID-Hash: LUQCUT55MMVU5BHLOVL7GSJ7LZFZRYD7 X-Message-ID-Hash: LUQCUT55MMVU5BHLOVL7GSJ7LZFZRYD7 X-MailFrom: sbrivio@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top, Laurent Vivier X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On Thu, 30 Jan 2025 19:54:17 +1100 David Gibson wrote: > On Thu, Jan 30, 2025 at 09:32:36AM +0100, Stefano Brivio wrote: > > On Thu, 30 Jan 2025 18:38:22 +1100 > > David Gibson wrote: > > > > > Right, but in the present draft you pay that cost whether or not > > > you're actually using the flows. Unfortunately a busy server with > > > heaps of active connections is exactly the case that's likely to be > > > most sensitve to additional downtime, but there's not really any > > > getting around that. A machine with a lot of state will need either > > > high downtime or high migration bandwidth. > > > > It's... sixteen megabytes. A KubeVirt node is only allowed to perform up > > to _four_ migrations in parallel, and that's our main use case at the > > moment. "High downtime" is kind of relative. > > Certainly. But I believe it's typical to aim for downtimes in the > ~100ms range. > > > > But, I'm really hoping we can move relatively quickly to a model where > > > a guest with only a handful of connections _doesn't_ have to pay that > > > 128k flow cost - and can consequently migrate ok even with quite > > > constrained migration bandwidth. In that scenario the size of the > > > header could become significant. > > > > I think the biggest cost of the full flow table transfer is rather code > > that's a bit quicker to write (I just managed to properly set sequences > > on the target, connections don't quite "flow" yet) but relatively high > > maintenance (as you mentioned, we need to be careful about every single > > field) and easy to break. > > Right. And with this draft we can't even change the size of the flow > table without breaking migration. That seems like a thing we might > well want to change. Why? The size of the flow table hasn't changed since it was added. I don't see a reason to improve this if we don't want to transfer the flow table anyway. > > I would like to quickly complete the whole flow first, because I think > > we can inform design and implementation decisions much better at that > > point, and we can be sure it's feasible, > > That's fair. > > > but I'm not particularly keen > > to merge this patch like it is, if we can switch it relatively swiftly > > to an implementation where we model a smaller fixed-endian structure > > with just the stuff we need. > > So, there are kind of two parts to this: > > 1) Only transferring active flow entries, and not transferring the > hash table > > I think this is pretty easy. It could be done with or without > preserving flow indicies. Preserving makes for debug log continuity > between the ends, but not preserving lets us change the size of the > flow table without breaking migration. I would just add prints on migration showing how old flow indices map to new ones. > 2) Only transferring the necessary pieces of each entry, and using a > fixed representation of each piece > > This is harder. Not *super* hard, I think, but definitely trickier > than (1) > > > And again, to be a bit more sure of which stuff we need in it, the full > > flow is useful to have implemented. > > > > Actually the biggest complications I see in switching to that approach, > > from the current point, are that we need to, I guess: > > > > 1. model arrays (not really complicated by itself) > > So here, I actually think this is simpler if we don't attempt to have > a declarative approach to defining the protocol, but just write > functions to implement it. > > > 2. have a temporary structure where we store flows instead of using the > > flow table directly (meaning that the "data model" needs to logically > > decouple source and destination of the copy) > > Right.. I'd really prefer to "stream" in the entries one by one, > rather than having a big staging area. That's even harder to do > declaratively, but I think the other advantages are worth it. > > > 3. batch stuff to some extent. We'll call socket() and connect() once > > for each socket anyway, obviously, but sending one message to the > > TCP_REPAIR helper for each socket looks like a rather substantial > > and avoidable overhead > > I don't think this actually has a lot of bearing on the protocol. I'd > envisage migrate_target() decodes all the information into the > target's flow table, then migrate_target_post() steps through all the > flows re-establishing the connections. Since we've already parsed the > protocol at that point, we can make multiple passes: one to gather > batches and set TCP_REPAIR, another through each entry to set the > values, and a final one to clear TCP_REPAIR in batches. Ah, right, I didn't think of using the target flow table directly. That has the advantage that the current code I'm writing to reactivate flows from the flow table can be recycled as it is. > > > > > It's both easier to do > > > > > and a bigger win in most cases. That would dramatically reduce the > > > > > size sent here. > > > > > > > > Yep, feel free. > > > > > > It's on my queue for the next few days. > > > > To me this part actually looks like the biggest priority after/while > > getting the whole thing to work, because we can start right with a 'v1' > > which looks more sustainable. > > > > And I would just get stuff working on x86_64 in that case, without even > > implementing conversions and endianness switches etc. > > Right. Given the number of options here, I think it would be safest > to go in expecting to go through a few throwaway protocol versions > before reaching one we're happy enough to support long term. > > To ease that process, I'm wondering if we should, add a default-off > command line option to enable migration. For now, enabling it would > print some sort of "migration is experimental!" warning. Once we have > a stream format we're ok with, we can flip it to on-by-default, but we > don't maintain receive compatibility for the experimental versions > leading up to that. It looks like unnecessary code churn to me. It doesn't need to be merged if it's work in progress. You can also push stuff to a temporary branch if needed. It can also be merged and not documented for a while, as long as it doesn't break existing functionality. -- Stefano