[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[RFC PATCH 0/9] Support for Virtio-fs daemon crash reconnection
From: |
Jiachen Zhang |
Subject: |
[RFC PATCH 0/9] Support for Virtio-fs daemon crash reconnection |
Date: |
Wed, 16 Dec 2020 00:21:10 +0800 |
Hi, all
We implement virtio-fs crash reconnection in this patchset. The crash
reconnection of virtiofsd here is completely transparent to guest, no
remount in guest is needed, even the inflight requests can be handled
normally after reconnection. We are looking forward to any comments.
Thanks,
Jiachen
OVERVIEW:
To support virtio-fs crash reconnection, we need to support the recovery
of 1) inflight FUSE request, and 2) virtiofsd internal status information.
Fortunately, QEMU's vhost-user reconnection framework already supports
inflight I/O tracking by using VHOST_USER_GET_INFLIGHT_FD and
VHOST_USER_SET_INFLIGHT_FD (see 5ad204bf2 and 5f9ff1eff for details).
As the FUSE requests are transferred by virtqueue I/O requests, by using
the vhost-user inflight I/O tracking, we can recover the inflight FUSE
requests.
To support virtiofsd internal status recovery, we introduce 4 new
vhost-user message types. As shown in the following diagram, two of them
are used to persist shared lo_maps and opened fds to QEMU, the other two
message types are used to restore the status when reconnecting.
VHOST_USER_SLAVE_SHM
VHOST_USER_SLAVE_FD
+--------------+ Persist +--------------------+
| <---------------------+ |
| QEMU | | Virtio-fs Daemon |
| +---------------------> |
+--------------+ Restore +--------------------+
VHOST_USER_SET_SHM
VHOST_USER_SET_FD
Although the 4 newly added message types are to support virtiofsd
reconnection in this patchset, it might be potential in other situation.
So we keep in mind to make them more general when add them to vhost
related source files. VHOST_USER_SLAVE_SHM and VHOST_USER_SET_SHM can be
used for memory sharing between a vhost-user daemon and QEMU,
VHOST_USER_SLAVE_FD and VHOST_USER_SET_FD would be useful if we want to
shared opened fds between QEMU process and vhost-user daemon process.
USAGE and NOTES:
- The commits are rebased to a recent QEMU master commit b4d939133dca0fa2b.
- ",reconnect=1" should be added to the "-chardev socket" of vhost-user-fs-pci
in the QEMU command line, for example:
qemu-system-x86_64 ... \
-chardev socket,id=char0,path=/tmp/vhostqemu,reconnect=1 \
-device vhost-user-fs-pci,queue-size=1024,chardev=char0,tag=myfs \
...
- We add new options for virtiofsd to enable or disable crash reconnection.
And some options are not supported by crash reconnection. So add following
options to virtiofsd to enable reconnection:
virtiofsd ... -o reconnect -o no_mount_ns -o no_flock -o no_posix_lock
-o no_xattr ...
- The reasons why virtiofsd-side locking, extended attributes, and mount
namespace are not supported is explained in the commit message of the 6th
patch (virtiofsd: Add two new options for crash reconnection).
- The 9th patch is a work-around that will not affect the overall correctness.
We remove the qsort related codes because we found that when resubmit_num is
larger than 64, seccomp will kill the virtiofsd process.
- Support for dax version virtiofsd is very possible and requires almost no
additional change to this patchset.
Jiachen Zhang (9):
vhost-user-fs: Add support for reconnection of vhost-user-fs backend
vhost: Add vhost-user message types for sending shared memory and file
fds
vhost-user-fs: Support virtiofsd crash reconnection
libvhost-user: Add vhost-user message types for sending shared memory
and file fds
virtiofsd: Convert the struct lo_map array to a more flatten layout
virtiofsd: Add two new options for crash reconnection
virtiofsd: Persist/restore lo_map and opened fds to/from QEMU
virtiofsd: Ensure crash consistency after reconnection
virtiofsd: (work around) Comment qsort in inflight I/O tracking
contrib/libvhost-user/libvhost-user.c | 106 +++-
contrib/libvhost-user/libvhost-user.h | 70 +++
docs/interop/vhost-user.rst | 41 ++
hw/virtio/vhost-user-fs.c | 334 ++++++++++-
hw/virtio/vhost-user.c | 123 ++++
hw/virtio/vhost.c | 42 ++
include/hw/virtio/vhost-backend.h | 6 +
include/hw/virtio/vhost-user-fs.h | 16 +-
include/hw/virtio/vhost.h | 42 ++
tools/virtiofsd/fuse_lowlevel.c | 24 +-
tools/virtiofsd/fuse_virtio.c | 44 ++
tools/virtiofsd/fuse_virtio.h | 1 +
tools/virtiofsd/helper.c | 9 +
tools/virtiofsd/passthrough_helpers.h | 2 +-
tools/virtiofsd/passthrough_ll.c | 830 ++++++++++++++++++--------
tools/virtiofsd/passthrough_seccomp.c | 1 +
16 files changed, 1413 insertions(+), 278 deletions(-)
--
2.20.1
- [RFC PATCH 0/9] Support for Virtio-fs daemon crash reconnection,
Jiachen Zhang <=
- [RFC PATCH 1/9] vhost-user-fs: Add support for reconnection of vhost-user-fs backend, Jiachen Zhang, 2020/12/15
- [RFC PATCH 2/9] vhost: Add vhost-user message types for sending shared memory and file fds, Jiachen Zhang, 2020/12/15
- [RFC PATCH 3/9] vhost-user-fs: Support virtiofsd crash reconnection, Jiachen Zhang, 2020/12/15
- [RFC PATCH 4/9] libvhost-user: Add vhost-user message types for sending shared memory and file fds, Jiachen Zhang, 2020/12/15
- [RFC PATCH 5/9] virtiofsd: Convert the struct lo_map array to a more flatten layout, Jiachen Zhang, 2020/12/15
- [RFC PATCH 6/9] virtiofsd: Add two new options for crash reconnection, Jiachen Zhang, 2020/12/15
- [RFC PATCH 7/9] virtiofsd: Persist/restore lo_map and opened fds to/from QEMU, Jiachen Zhang, 2020/12/15
- [RFC PATCH 8/9] virtiofsd: Ensure crash consistency after reconnection, Jiachen Zhang, 2020/12/15
- [RFC PATCH 9/9] virtiofsd: (work around) Comment qsort in inflight I/O tracking, Jiachen Zhang, 2020/12/15
- Re: [RFC PATCH 0/9] Support for Virtio-fs daemon crash reconnection, no-reply, 2020/12/15