On 10/11/19 9:22 AM, Guoheyi wrote:
Hi folks,
We observed Linux on VM occasionally (at very low rate) got soft lockup
when a remote cdrom is attached. The guest hangs up at below call trace:
That's certainly a new one to me :)
[Tue Oct8 23:02:53 2019]ata_scsi_queuecmd+0xe0/0x2a0 [libata]
[Tue Oct8 23:02:53 2019]scsi_dispatch_cmd+0xec/0x288
[Tue Oct8 23:02:53 2019]scsi_queue_rq+0x5f4/0x6b8
[Tue Oct8 23:02:53 2019]blk_mq_dispatch_rq_list+0xb0/0x690
[Tue Oct8 23:02:53 2019]blk_mq_do_dispatch_sched+0x8c/0x130
[Tue Oct8 23:02:53 2019]blk_mq_sched_dispatch_requests+0x128/0x1f0
[Tue Oct8 23:02:53 2019]__blk_mq_run_hw_queue+0x9c/0x128
[Tue Oct8 23:02:53 2019]__blk_mq_delay_run_hw_queue+0x198/0x1d8
[Tue Oct8 23:02:53 2019]blk_mq_run_hw_queue+0x68/0x180
[Tue Oct8 23:02:53 2019]blk_mq_sched_insert_request+0xbc/0x210
[Tue Oct8 23:02:53 2019]blk_execute_rq_nowait+0x118/0x168
[Tue Oct8 23:02:53 2019]blk_execute_rq+0x74/0xd8
[Tue Oct8 23:02:53 2019]__scsi_execute+0xd8/0x1e0
[Tue Oct8 23:02:53 2019]sr_check_events+0xd4/0x2c8 [sr_mod]
[Tue Oct8 23:02:53 2019]cdrom_check_events+0x34/0x50 [cdrom]
[Tue Oct8 23:02:53 2019]sr_block_check_events+0xdc/0x108 [sr_mod]
[Tue Oct8 23:02:53 2019]disk_check_events+0x60/0x198
[Tue Oct8 23:02:53 2019]disk_events_workfn+0x24/0x30
[Tue Oct8 23:02:53 2019]process_one_work+0x1b4/0x3f8
[Tue Oct8 23:02:53 2019]worker_thread+0x54/0x470
[Tue Oct8 23:02:53 2019]kthread+0x134/0x138
[Tue Oct8 23:02:53 2019]ret_from_fork+0x10/0x18
We are running the whole stack on ARM64 platforms, using rcdrom on host
to connect a remote cdrom, which is appeared as "/dev/sr0" on the host.
Our Linux kernel version is 4.19.36 and qemu version is 2.8.1, which is
fairly old but I checked the mainline and found the work flow does not
change much. And KVM is enabled.
We provide the remote cdrom to guest as a block device, attached under
ICH SATA bus.
The work flow should be like this (please correct me if I was wrong):
1. There is a kworker thread in guest kernel which will check cdrom
status periodically.
2. The call of "ata_scsi_queuecmd" in guest will write AHCI port
register "PORT_CMD_ISSUE", so this VCPU thread is trapped out to qemu.
3. qemu will grab the BQL and then dispatch the access to
ahci_port_write().
4. For this is a "get event status notification" command, qemu finally
goes to cmd_get_event_status_notification() and then cdrom_is_inserted().
via
cmd_get_event_status_notification (SCSI 0x4A)
event_status_media
blk_is_inserted
5. In cdrom_is_inserted(), an ioctl to cdrom fd is issued.
Using the bdrv_host_cdrom BlockDriver, for the .bdrv_is_inserted callback.
However, in the last step, we found the ioctl() may have large latency,
for it is a virtual device of remote cdrom, when the remote server is
busy and of poor performance. We have observed more than 8 seconds
latency in half an hour test, and the latency might reach more than 20
seconds when guest soft lockup occurred.
I'm not sure what can be done here. the host_cdrom driver has a few
methods to query state (cdrom_is_inserted, cdrom_eject,
cdrom_lock_medium) and in general code is going to rely on
bdrv_is_inserted returning a truthful answer.
(I'm not sure we have callbacks established to tell when the backing
media we are ourselves relying on has gone away. Maybe it could be
added, but it's not there now. We could maybe cache the answer if we had
something reliable.)
You could always try using the host_device driver instead of the
host_cdrom one, which will just treat it as a "normal" block device
instead of a CDROM one, and doesn't use any cdrom specific ioctls. It
might avoid the costly call.
My question is, is there any way to get around of this issue? Does it
make sense for qemu to setup an IO thread to issue this ioctl() and let
the VCPU thread return to guest as soon as possible? Or it is kernel's
responsibility to break up the long time ioctl() and return to user space?
Yeah, I think you could probably try to make this change -- the code is
unfortunately very callback-hell-ish with poor abstraction boundaries,
but obviously the data transfer commands already defer to bottom halves.
(Ideally, I think every ATAPI command would just immediately defer to a
bottom half, but I don't know what effect that would have on callers of
ide_atapi_cmd. I'd have to audit it, and it's quite squiggly.)
"Patches welcome" as they say, but it's quite messy down there.