[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Attaching 9P VirtFS to PowerNV
From: |
Ryan Burns |
Subject: |
Re: Attaching 9P VirtFS to PowerNV |
Date: |
Thu, 4 Mar 2021 10:44:22 -0800 |
User-agent: |
Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.7.0 |
On 3/4/21 2:02 AM, Greg Kurz wrote:
> On Wed, 3 Mar 2021 11:19:37 -0800
> Ryan Burns <ryanxcv@gmail.com> wrote:
>
>> Hi Murilo, thanks for your response.
>>
>
> Hi Ryan,
>
> It is probably best to Cc QEMU 9p maintainers and qemu-devel@nongnu.org
> so that people who can provide help have a chance to see your question.
>
> Luckily, I happen to spend most of my time on PPC :)
Ah good to know, appreciate the CCs.
>
>> On 3/3/21 8:37 AM, Murilo Opsfelder Araújo wrote:
>>> Hi, Ryan.
>>>
>>> On Wednesday, March 3, 2021 4:59:09 AM -03 Ryan Burns wrote:
>>>> Hi all,
>>>>
>>>> I'm trying to port NixOS to run on QEMU's PowerNV platform. The system
>>>> is able to load the kernel, initrd, and a virtio-blk-pci device without
>
> Very cool ! Cc'ing PowerNV maintainer Cédric Le Goater. He'll probably be
> interested in your initiative.
Great, thanks!
I've already found his PowerNV wiki page on github very helpful :)
>
>>>> issues, but fails trying to mount the nix store from the host via -virtfs.
>>>>
>>>> I noticed that unlike on other platforms, I needed to specify the PCIe
>>>> bus for the virtio-blk-pci device to appear as /dev/vda. Do I need to
>>>> do something similar for the 9P virtfs?
>
> Probably. The -virtfs option is sugar for -fsdev + -device virtio-9p-pci.
>
> You can check the QEMU manual page or the wiki to see how to convert
> your command line:
>
> https://wiki.qemu.org/Documentation/9psetup#Starting_the_Guest_directly
>
> See below for further explanation.
>
>>>>
>>>>
>>>> Here is the relevant snippet from my boot log:
>>>>
>>>>
>>>> /dev/vda: recovering journal
>>>> /dev/vda: clean, 231/32768 files, 10160/131072 blocks
>>>> mounting /dev/vda on /...
>>>> mounting store on /nix/.ro-store...
>>>> [ 5.972587][ T170] 9pnet_virtio: no channels available for device store
>>>> [ 5.972587][ T170] 9pnet_virtio: no channels available for device store
>
> This message means that 9pnet_virtio doesn't know about the mount
> tags that you have passed to -virtfs, i.e. very likely because the
> virtio-9p-pci device isn't seen by the guest.
>
>>>> mount: mounting store on /mnt-root/nix/.ro-store failed: No such file or
>>>> directory
>>>
>>> The message suggests that /mnt-root/nix/.ro-store does not exist at the
>>> moment
>>> mount is called.
>>>
>>> Are you 100% sure that the previous mkdir was successful? Is "/" writable
>>> at
>>> the moment mkdir was called? Shouldn't the script check for the return
>>> code
>>> of
>>> mkdir?
>>>
>>> https://github.com/NixOS/nixpkgs/blob/
>>> 545bafa2d49d5184714d94ea4c568316d9ce7b5c/nixos/modules/system/boot/stage-1-
>>> init.sh#L390
>>> mkdir -p "/mnt-root$mountPoint"
>>
>> The previous mkdir was successful - "/" is always a fresh qcow2 image.
>> I added some debugging commands to the stage 1 script to be sure -
>> the mkdir command exits with status 0 and the mountpoint does exist.
>>
>>>
>>> Besides that, I'd also check if filesystem type is missing from mount
>>> arguments,
>>> as per:
>>>
>>> https://wiki.qemu.org/Documentation/9psetup#Mounting_the_shared_path
>>
>> The stage 1 script does this by generating an fstab on-the-fly with the
>> correct options set. Here is what it looks like at mount time:
>>
>> /dev/vda /mnt-root/ ext4 defaults,
>> store /mnt-root/nix/.ro-store 9p trans=virtio,version=9p2000.L,cache=loose,
>>
>> I think this is all correct, right? I might add that these VM scripts are
>> pretty well-tested on x86 and ARM, which is why I thought I'd ask on this
>> mailing list - I assume the most likely explanation is that I'm simply
>> using a faulty QEMU command-line or kernel config.
>>
>
> The command line to start a PowerNV machine with multiple PCIe
> devices requires to specify the bus for all of them.
>
> This is what I do to boot a fedora guest with a virtio-scsi,
> a virtio-9p and a virtio-net device:
>
> qemu-system-ppc64 \
> -machine powernv9 \
> -m 2G \
> -kernel ${kernel} -initrd ${initrd} -bios ${skiboot_fw} \
> -serial mon:stdio \
> -nographic -nodefaults \
> -drive file=${fedora_image},if=none,id=drive0 \
> -device virtio-scsi-pci,bus=pcie.0 \
> -device scsi-hd,drive=drive0,bootindex=1 \
> -netdev tap,helper=/usr/libexec/qemu-bridge-helper,br=virbr0,id=netdev0
> \
> -device virtio-net-pci,netdev=netdev0,mac=${mac},bus=pcie.1 \
> -fsdev local,id=fsdev0,path=${path},security_model=none \
> -device virtio-9p-pci,fsdev=fsdev0,mount_tag=host,bus=pcie.2
You are spot on, this solved my issue perfectly!
Look at this beautiful login prompt!
Welcome to NixOS 21.03 (Okapi)!
[ OK ] Created slice system-getty.slice.
<snip>
[ OK ] Reached target Login Prompts.
<<< Welcome to NixOS 21.03pre-git (ppc64le) - hvc0 >>>
nixos login:
>
> Cheers,
>
> --
> Greg
>
>>>
>>> Cheers!
>>>
>>>> And here is what my generated qemu command line invocation looks like:
>>>>
>>>>
>>>> exec
>>>> /nix/store/r2jn54cpvmp2620f6ky8yy2dfd3lglwz-qemu-5.2.0/bin/qemu-system-ppc64
>>>> -machine powernv \
>>>> -name nixos \
>>>> -m 2048 \
>>>> -smp 1 \
>>>> -device virtio-rng-pci \
>>>> -net nic,netdev=user.0,model=virtio -netdev
>>>> user,id=user.0${QEMU_NET_OPTS:+,$QEMU_NET_OPTS} \
>>>> -virtfs local,path=/nix/store,security_model=none,mount_tag=store \
>>>> -virtfs local,path=$TMPDIR/xchg,security_model=none,mount_tag=xchg \
>>>> -virtfs
>>>> local,path=${SHARED_DIR:-$TMPDIR/xchg},security_model=none,mount_tag=shared
>>>> \
>>>> -drive
>>>> cache=writeback,file=$NIX_DISK_IMAGE,id=drive1,if=none,index=1,werror=report
>>>> -device virtio-blk-pci,bus=pcie.0,drive=drive1 \
>>>> -kernel
>>>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>>>> kernel -initrd
>>>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>>>> initrd -append "$(cat
>>>> /nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pre-git/
>>>> kernel-params)
>>>> init=/nix/store/4mw3zzh1k4xhlysfmcgvpgx9zjz032c7-nixos-system-nixos-21.03pr
>>>> e-git/init
>>>> regInfo=/nix/store/dnrqa7qar8jakalrz85qn3wslq5k7fmm-closure-info-powerpc64l
>>>> e-unknown-linux-gnu/registration console=ttyAMA0,115200n8
>>>> $QEMU_KERNEL_PARAMS" -nographic \
>>>
>>> --
>>> Murilo
>>>
>>
>