From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: passt.top; dmarc=pass (p=none dis=none) header.from=redhat.com Authentication-Results: passt.top; dkim=pass (1024-bit key; unprotected) header.d=redhat.com header.i=@redhat.com header.a=rsa-sha256 header.s=mimecast20190719 header.b=S548kmIF; dkim-atps=neutral Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by passt.top (Postfix) with ESMTPS id 721955A0272 for ; Fri, 20 Dec 2024 08:56:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1734681392; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=SL+nm+Q7U3i5e9y8SkAg66gv8rDT577l33RgYWsIfL0=; b=S548kmIFlzvGQ0/kgcjixh0ZNu+hE/WpYaH2W2M4OjKI0bGPaRWLfATd62b4z1OZjmNBaA lvlS62MCZijRcP0BZXGrJHpEGhI3ek7luovpqxwKOdLvBC4hpQVwcBrQVCoHUZqX5OB4NC yuNCsyh0KdyhzVPUCjSM+iSd5DIe+9E= Received: from mail-wm1-f69.google.com (mail-wm1-f69.google.com [209.85.128.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-68-IwEBHNI-MHGc6FOt2wtPHA-1; Fri, 20 Dec 2024 02:56:30 -0500 X-MC-Unique: IwEBHNI-MHGc6FOt2wtPHA-1 X-Mimecast-MFC-AGG-ID: IwEBHNI-MHGc6FOt2wtPHA Received: by mail-wm1-f69.google.com with SMTP id 5b1f17b1804b1-436379713baso8780425e9.2 for ; Thu, 19 Dec 2024 23:56:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1734681389; x=1735286189; h=content-transfer-encoding:in-reply-to:autocrypt:from :content-language:references:cc:to:subject:user-agent:mime-version :date:message-id:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=SL+nm+Q7U3i5e9y8SkAg66gv8rDT577l33RgYWsIfL0=; b=HHJDBAvKu+f6PSZLxzS9a5axuowsMhllpdIupsMVBNrNFjS+c3atqAvga7rVhuOU7V RhW6kV1Rcpzysm15Ae9BlkqFlu4QI72wjI7kNwN8sIijcvYl9Gz07TYDSuLtHVRyWdaA 7ASvCuEvcvYG5OAuV4wgG6h3DOpzshtT/jG1n59sxiH9/rxtbcXZovvfYrCFZdDNyADX NbsPaQX+p/hLxaTv24gvJ/Qm0MbCHqaDCWAkvq10XLhaExtC/AEQO/QgO26bUN+SUWrT oYobtlAkSakBdCcJ1u2wX7xiEKNda3fIONyVOoNgYVgSt6EMwLLNHLMVcdBA5srm728B ixTA== X-Gm-Message-State: AOJu0YykhqLTy8quknLvXGKomib2GkwZdLbtJmd154C8akOVn4dfuml5 rG/OwofASorqlpqd9sIZXJXztMVuyP6YfUuJSck2CtNX4KprhjtVDY9YPCHy2ASxSQEPbmlD2rc YlczPK45mH6EsBAsU4rw94BYpdoPY+R8iH96OWQRRS2IEFm4Qj/2iAI9f6A== X-Gm-Gg: ASbGnctm8lXrw9xseRZjyW2fV+hoSx4R/DvzkLwLvlJbPR2ptXqIku2Vhyih28lvE5A NVOzRdzjUrt3y1j+FPydlUxj1VlpgaWbt+wqtkWF4U1v1N0SSuNKcMByPnzD0ULp7LPYGnftSSD cvxX5hYofs7P3IgGU/nvaYid6dFYdcri8e7IZAWUVF6IfvSdqsvC7Jxoj0DDo5fS1vOdlGkDHaQ OjYaYAvRYdtACPQKc7Mw1fef1++DlCn5lKhC2nYt2ETLQUJ9HDvL6yhu7ZsGMP2ESOY9YfljtCg x2Fw+Gyy7/YzWcVF73c= X-Received: by 2002:a05:600c:1c87:b0:434:a10f:c3 with SMTP id 5b1f17b1804b1-4366864320emr15611265e9.9.1734681388807; Thu, 19 Dec 2024 23:56:28 -0800 (PST) X-Google-Smtp-Source: AGHT+IEPuoJHU7Mt8SO3WywbKxemPKLk6SPCNJD0XTjLrREXkGy4q97ySYKdgUqdD5tu0odeLfOkzA== X-Received: by 2002:a05:600c:1c87:b0:434:a10f:c3 with SMTP id 5b1f17b1804b1-4366864320emr15611105e9.9.1734681388463; Thu, 19 Dec 2024 23:56:28 -0800 (PST) Received: from ?IPV6:2a01:e0a:e10:ef90:343a:68f:2e91:95c? ([2a01:e0a:e10:ef90:343a:68f:2e91:95c]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-436611ea387sm38035655e9.6.2024.12.19.23.56.27 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Dec 2024 23:56:28 -0800 (PST) Message-ID: <61d7f502-6f4d-40bc-b3a5-feed4229585d@redhat.com> Date: Fri, 20 Dec 2024 08:56:27 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 8/9] vhost-user: add VHOST_USER_SET_DEVICE_STATE_FD command To: Stefano Brivio References: <20241219111400.2352110-1-lvivier@redhat.com> <20241219111400.2352110-9-lvivier@redhat.com> <20241219204759.65bc5353@elisabeth> From: Laurent Vivier Autocrypt: addr=lvivier@redhat.com; keydata= xsFNBFYFJhkBEAC2me7w2+RizYOKZM+vZCx69GTewOwqzHrrHSG07MUAxJ6AY29/+HYf6EY2 WoeuLWDmXE7A3oJoIsRecD6BXHTb0OYS20lS608anr3B0xn5g0BX7es9Mw+hV/pL+63EOCVm SUVTEQwbGQN62guOKnJJJfphbbv82glIC/Ei4Ky8BwZkUuXd7d5NFJKC9/GDrbWdj75cDNQx UZ9XXbXEKY9MHX83Uy7JFoiFDMOVHn55HnncflUncO0zDzY7CxFeQFwYRbsCXOUL9yBtqLer Ky8/yjBskIlNrp0uQSt9LMoMsdSjYLYhvk1StsNPg74+s4u0Q6z45+l8RAsgLw5OLtTa+ePM JyS7OIGNYxAX6eZk1+91a6tnqfyPcMbduxyBaYXn94HUG162BeuyBkbNoIDkB7pCByed1A7q q9/FbuTDwgVGVLYthYSfTtN0Y60OgNkWCMtFwKxRaXt1WFA5ceqinN/XkgA+vf2Ch72zBkJL RBIhfOPFv5f2Hkkj0MvsUXpOWaOjatiu0fpPo6Hw14UEpywke1zN4NKubApQOlNKZZC4hu6/ 8pv2t4HRi7s0K88jQYBRPObjrN5+owtI51xMaYzvPitHQ2053LmgsOdN9EKOqZeHAYG2SmRW LOxYWKX14YkZI5j/TXfKlTpwSMvXho+efN4kgFvFmP6WT+tPnwARAQABzSNMYXVyZW50IFZp dmllciA8bHZpdmllckByZWRoYXQuY29tPsLBeAQTAQIAIgUCVgVQgAIbAwYLCQgHAwIGFQgC CQoLBBYCAwECHgECF4AACgkQ8ww4vT8vvjwpgg//fSGy0Rs/t8cPFuzoY1cex4limJQfReLr SJXCANg9NOWy/bFK5wunj+h/RCFxIFhZcyXveurkBwYikDPUrBoBRoOJY/BHK0iZo7/WQkur 6H5losVZtrotmKOGnP/lJYZ3H6OWvXzdz8LL5hb3TvGOP68K8Bn8UsIaZJoeiKhaNR0sOJyI YYbgFQPWMHfVwHD/U+/gqRhD7apVysxv5by/pKDln1I5v0cRRH6hd8M8oXgKhF2+rAOL7gvh jEHSSWKUlMjC7YwwjSZmUkL+TQyE18e2XBk85X8Da3FznrLiHZFHQ/NzETYxRjnOzD7/kOVy gKD/o7asyWQVU65mh/ECrtjfhtCBSYmIIVkopoLaVJ/kEbVJQegT2P6NgERC/31kmTF69vn8 uQyW11Hk8tyubicByL3/XVBrq4jZdJW3cePNJbTNaT0d/bjMg5zCWHbMErUib2Nellnbg6bc 2HLDe0NLVPuRZhHUHM9hO/JNnHfvgiRQDh6loNOUnm9Iw2YiVgZNnT4soUehMZ7au8PwSl4I KYE4ulJ8RRiydN7fES3IZWmOPlyskp1QMQBD/w16o+lEtY6HSFEzsK3o0vuBRBVp2WKnssVH qeeV01ZHw0bvWKjxVNOksP98eJfWLfV9l9e7s6TaAeySKRRubtJ+21PRuYAxKsaueBfUE7ZT 7zfOwU0EVgUmGQEQALxSQRbl/QOnmssVDxWhHM5TGxl7oLNJms2zmBpcmlrIsn8nNz0rRyxT 460k2niaTwowSRK8KWVDeAW6ZAaWiYjLlTunoKwvF8vP3JyWpBz0diTxL5o+xpvy/Q6YU3BN efdq8Vy3rFsxgW7mMSrI/CxJ667y8ot5DVugeS2NyHfmZlPGE0Nsy7hlebS4liisXOrN3jFz asKyUws3VXek4V65lHwB23BVzsnFMn/bw/rPliqXGcwl8CoJu8dSyrCcd1Ibs0/Inq9S9+t0 VmWiQWfQkz4rvEeTQkp/VfgZ6z98JRW7S6l6eophoWs0/ZyRfOm+QVSqRfFZdxdP2PlGeIFM C3fXJgygXJkFPyWkVElr76JTbtSHsGWbt6xUlYHKXWo+xf9WgtLeby3cfSkEchACrxDrQpj+ Jt/JFP+q997dybkyZ5IoHWuPkn7uZGBrKIHmBunTco1+cKSuRiSCYpBIXZMHCzPgVDjk4viP brV9NwRkmaOxVvye0vctJeWvJ6KA7NoAURplIGCqkCRwg0MmLrfoZnK/gRqVJ/f6adhU1oo6 z4p2/z3PemA0C0ANatgHgBb90cd16AUxpdEQmOCmdNnNJF/3Zt3inzF+NFzHoM5Vwq6rc1JP jfC3oqRLJzqAEHBDjQFlqNR3IFCIAo4SYQRBdAHBCzkM4rWyRhuVABEBAAHCwV8EGAECAAkF AlYFJhkCGwwACgkQ8ww4vT8vvjwg9w//VQrcnVg3TsjEybxDEUBm8dBmnKqcnTBFmxN5FFtI WlEuY8+YMiWRykd8Ln9RJ/98/ghABHz9TN8TRo2b6WimV64FmlVn17Ri6FgFU3xNt9TTEChq AcNg88eYryKsYpFwegGpwUlaUaaGh1m9OrTzcQy+klVfZWaVJ9Nw0keoGRGb8j4XjVpL8+2x OhXKrM1fzzb8JtAuSbuzZSQPDwQEI5CKKxp7zf76J21YeRrEW4WDznPyVcDTa+tz++q2S/Bp P4W98bXCBIuQgs2m+OflERv5c3Ojldp04/S4NEjXEYRWdiCxN7ca5iPml5gLtuvhJMSy36gl U6IW9kn30IWuSoBpTkgV7rLUEhh9Ms82VWW/h2TxL8enfx40PrfbDtWwqRID3WY8jLrjKfTd R3LW8BnUDNkG+c4FzvvGUs8AvuqxxyHbXAfDx9o/jXfPHVRmJVhSmd+hC3mcQ+4iX5bBPBPM oDqSoLt5w9GoQQ6gDVP2ZjTWqwSRMLzNr37rJjZ1pt0DCMMTbiYIUcrhX8eveCJtY7NGWNyx FCRkhxRuGcpwPmRVDwOl39MB3iTsRighiMnijkbLXiKoJ5CDVvX5yicNqYJPKh5MFXN1bvsB kmYiStMRbrD0HoY1kx5/VozBtc70OU0EB8Wrv9hZD+Ofp0T3KOr1RUHvCZoLURfFhSQ= In-Reply-To: <20241219204759.65bc5353@elisabeth> X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: Ez3pzOpxZ9gkQMGODsX-BX5PWGX9BfFPhSrw9CXeldQ_1734681389 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Message-ID-Hash: KQBFDTXI6SU5FAS3VMZKQXAYSG2LWKP4 X-Message-ID-Hash: KQBFDTXI6SU5FAS3VMZKQXAYSG2LWKP4 X-MailFrom: lvivier@redhat.com X-Mailman-Rule-Misses: dmarc-mitigation; no-senders; approved; emergency; loop; banned-address; member-moderation; nonmember-moderation; administrivia; implicit-dest; max-recipients; max-size; news-moderation; no-subject; digests; suspicious-header CC: passt-dev@passt.top X-Mailman-Version: 3.3.8 Precedence: list List-Id: Development discussion and patches for passt Archived-At: Archived-At: List-Archive: List-Archive: List-Help: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: On 19/12/2024 20:47, Stefano Brivio wrote: > On Thu, 19 Dec 2024 12:13:59 +0100 > Laurent Vivier wrote: > >> +/** >> + * vu_migrate() -- Send/receive passt insternal state to/from QEMU > > Magic! > >> + * @vdev: vhost-user device >> + * @events: epoll events >> + */ >> +void vu_migrate(struct vu_dev *vdev, uint32_t events) >> +{ >> + int ret; >> + >> + /* TODO: collect/set passt internal state >> + * and use vdev->device_state_fd to send/receive it >> + */ >> + debug("vu_migrate fd %d events %x", vdev->device_state_fd, events); >> + if (events & EPOLLOUT) { > > I haven't really reviewed the series yet, but I have a preliminary > question: how does the hypervisor tell us that we're writing too much? The hypervisor is not aware of anything. It's only a bridge between the two passt instances. But normally the destination side can report an error, this aborts the migration on both sides. In QEMU, the code is: while (true) { ssize_t read_ret; /* read the data from the backend */ read_ret = RETRY_ON_EINTR(read(read_fd, transfer_buf, chunk_size)); if (read_ret < 0) { ret = -errno; error_setg_errno(errp, -ret, "Failed to receive state"); goto fail; } assert(read_ret <= chunk_size); qemu_put_be32(f, read_ret); if (read_ret == 0) { /* EOF */ break; } /* send to destination QEMU */ qemu_put_buffer(f, transfer_buf, read_ret); } /* * Back-end will not really care, but be clean and close our end of the pipe * before inquiring the back-end about whether transfer was successful */ close(read_fd); On the other side: while (true) { size_t this_chunk_size = qemu_get_be32(f); ssize_t write_ret; const uint8_t *transfer_pointer; if (this_chunk_size == 0) { /* End of state */ break; } if (transfer_buf_size < this_chunk_size) { transfer_buf = g_realloc(transfer_buf, this_chunk_size); transfer_buf_size = this_chunk_size; } if (qemu_get_buffer(f, transfer_buf, this_chunk_size) < this_chunk_size) { error_setg(errp, "Failed to read state"); ret = -EINVAL; goto fail; } transfer_pointer = transfer_buf; while (this_chunk_size > 0) { write_ret = RETRY_ON_EINTR( write(write_fd, transfer_pointer, this_chunk_size) ); if (write_ret < 0) { ret = -errno; error_setg_errno(errp, -ret, "Failed to send state"); goto fail; } else if (write_ret == 0) { error_setg(errp, "Failed to send state: Connection is closed"); ret = -ECONNRESET; goto fail; } assert(write_ret <= this_chunk_size); this_chunk_size -= write_ret; transfer_pointer += write_ret; } } /* * Close our end, thus ending transfer, before inquiring the back-end about * whether transfer was successful */ close(write_fd); Moreover, I think it's important to know, the source side is stopped at the end of migration but it is in a state it can be restarted. > I guess we'll do a short write and we'll need to go back to EPOLLOUT? > There's no minimum chunk size we can write, correct? Correct. It works even if there is no write() in this code. The hypervisor reads until we close the file descriptor (it's a pipe in fact). > >> + debug("Saving backend state"); >> + >> + /* send some stuff */ >> + ret = write(vdev->device_state_fd, "PASST", 6); >> + /* value to be returned by VHOST_USER_CHECK_DEVICE_STATE */ >> + vdev->device_state_result = ret == -1 ? -1 : 0; >> + /* Closing the file descriptor signals the end of transfer */ >> + epoll_ctl(vdev->context->epollfd, EPOLL_CTL_DEL, >> + vdev->device_state_fd, NULL); >> + close(vdev->device_state_fd); >> + vdev->device_state_fd = -1; >> + } else if (events & EPOLLIN) { > > ...and similarly here, I guess we'll get a short read? We must read until we get a close(). > >> + char buf[6]; >> + >> + debug("Loading backend state"); >> + /* read some stuff */ >> + ret = read(vdev->device_state_fd, buf, sizeof(buf)); >> + /* value to be returned by VHOST_USER_CHECK_DEVICE_STATE */ >> + if (ret != sizeof(buf)) { >> + vdev->device_state_result = -1; >> + } else { >> + ret = strncmp(buf, "PASST", sizeof(buf)); >> + vdev->device_state_result = ret == 0 ? 0 : -1; >> + } >> + } else if (events & EPOLLHUP) { >> + debug("Closing migration channel"); >> + >> + /* The end of file signals the end of the transfer. */ >> + epoll_ctl(vdev->context->epollfd, EPOLL_CTL_DEL, >> + vdev->device_state_fd, NULL); >> + close(vdev->device_state_fd); >> + vdev->device_state_fd = -1; >> + } >> +} > Thanks, Laurent