[Qemu-commits] [qemu/qemu] d81e1e: blockdev-backup: don't check aio_context too early

[Peter Maydell]

  Branch: refs/heads/master
  Home:   https://github.com/qemu/qemu
  Commit: d81e1efbea7d19c2f142d300df56538c73800590
      
https://github.com/qemu/qemu/commit/d81e1efbea7d19c2f142d300df56538c73800590
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M blockdev.c

  Log Message:
  -----------
  blockdev-backup: don't check aio_context too early

in blockdev_backup_prepare, we check to make sure that the target is
associated with a compatible aio context. However, do_blockdev_backup is
called later and has some logic to move the target to a compatible
aio_context. The transaction version will fail certain commands
needlessly early as a result.

Allow blockdev_backup_prepare to simply call do_blockdev_backup, which
will ultimately decide if the contexts are compatible or not.

Note: the transaction version has always disallowed this operation since
its initial commit bd8baecd (2014), whereas the version of
qmp_blockdev_backup at the time, from commit c29c1dd312f, tried to
enforce the aio_context switch instead. It's not clear, and I can't see
from the mailing list archives at the time, why the two functions take a
different approach. It wasn't until later in efd7556708b (2016) that the
standalone version tried to determine if it could set the context or
not.

Reported-by: aihua liang <address@hidden>
Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=1683498
Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 8b6f5f8b9f3bec5cbeebefab34bae0102a2581b3
      
https://github.com/qemu/qemu/commit/8b6f5f8b9f3bec5cbeebefab34bae0102a2581b3
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests.py: do not use infinite waits

Cap waits to 60 seconds so that iotests can fail gracefully if something
goes wrong.

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: f6f4b3f045ea18e3fa93a50cd0462236c428d62e
      
https://github.com/qemu/qemu/commit/f6f4b3f045ea18e3fa93a50cd0462236c428d62e
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M python/qemu/__init__.py

  Log Message:
  -----------
  QEMUMachine: add events_wait method

Instead of event_wait which looks for a single event, add an events_wait
which can look for any number of events simultaneously. However, it
will still only return one at a time, whichever happens first.

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: d6a79af0e641806d6bd6a42a4920e294b5db179c
      
https://github.com/qemu/qemu/commit/d6a79af0e641806d6bd6a42a4920e294b5db179c
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  iotests.py: rewrite run_job to be pickier

Don't pull events out of the queue that don't belong to us;
be choosier so that we can use this method to drive jobs that
were launched by transactions that may have more jobs.

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: ba7704f2228f16ed61b9903801e28e17666c7e38
      
https://github.com/qemu/qemu/commit/ba7704f2228f16ed61b9903801e28e17666c7e38
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    A tests/qemu-iotests/256
    A tests/qemu-iotests/256.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: add iotest 256 for testing blockdev-backup across iothread contexts

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Reviewed-by: Max Reitz <address@hidden>
[mreitz: Moved from 250 to 256]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 9e8dfad045211fbb8f4184285747907517550f23
      
https://github.com/qemu/qemu/commit/9e8dfad045211fbb8f4184285747907517550f23
  Author: John Snow <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M python/qemu/__init__.py

  Log Message:
  -----------
  event_match: always match on None value

Before, event_match didn't always recurse if the event value was not a
dictionary, and would instead check for equality immediately.

By delaying equality checking to post-recursion, we can allow leaf
values like "5" to match "None" and take advantage of the generic
None-returns-True clause.

This makes the matching a little more obviously consistent at the
expense of being able to check for explicit None values, which is
probably not that important given what this function is used for.

Signed-off-by: John Snow <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: a3bd71b5773a3664692601e6e181f108e1e4aa41
      
https://github.com/qemu/qemu/commit/a3bd71b5773a3664692601e6e181f108e1e4aa41
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/175
    M tests/qemu-iotests/175.out

  Log Message:
  -----------
  iotests: Filter 175's allocation information

It is possible for an empty file to take up blocks on a filesystem, for
example:

$ qemu-img create -f raw test.img 1G
Formatting 'test.img', fmt=raw size=1073741824
$ mkfs.ext4 -I 128 -q test.img
$ mkdir test-mount
$ sudo mount -o loop test.img test-mount
$ sudo touch test-mount/test-file
$ stat -c 'blocks=%b' test-mount/test-file
blocks=8

These extra blocks (one cluster) are apparently used for metadata,
because they are always there, on top of blocks used for data:

$ sudo dd if=/dev/zero of=test-mount/test-file bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.00135339 s, 775 MB/s
$ stat -c 'blocks=%b' test-mount/test-file
blocks=2056

Make iotest 175 take this into account.

Reported-by: Thomas Huth <address@hidden>
Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Nir Soffer <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: d9efe9384ecb68dcb971b439099e9bead653f96b
      
https://github.com/qemu/qemu/commit/d9efe9384ecb68dcb971b439099e9bead653f96b
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/219

  Log Message:
  -----------
  iotests: Fix intermittent failure in 219

In 219, we wait for the job to make progress before we emit its status.
This makes the output reliable.  We do not wait for any more progress if
the job's current-progress already matches its total-progress.

Unfortunately, there is a bug: Right after the job has been started,
it's possible that total-progress is still 0.  In that case, we may skip
the first progress-making step and keep ending up 64 kB short.

To fix that bug, we can simply wait for total-progress to reach 4 MB
(the image size) after starting the job.

Reported-by: Karen Mezick <address@hidden>
Buglink: https://bugzilla.redhat.com/show_bug.cgi?id=1686651
Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
[mreitz: Adjusted commit message as per John's proposal]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 6653d1318d623681fc7e849e9ae5069ad144864a
      
https://github.com/qemu/qemu/commit/6653d1318d623681fc7e849e9ae5069ad144864a
  Author: Andrey Shinkevich <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M hw/block/fdc.c

  Log Message:
  -----------
  hw/block/fdc: floppy command FIFO memory initialization

The uninitialized memory allocated for the command FIFO of the
floppy controller during the VM hardware initialization incurs
many unwanted reports by Valgrind when VM state is being saved.
That verbosity hardens a search for the real memory issues when
the iotests run. Particularly, the patch eliminates 20 unnecessary
reports of the Valgrind tool in the iotest #169.

Signed-off-by: Andrey Shinkevich <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 549fb88045253cb246f900a9db447800313865c4
      
https://github.com/qemu/qemu/commit/549fb88045253cb246f900a9db447800313865c4
  Author: Vladimir Sementsov-Ogievskiy <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/254

  Log Message:
  -----------
  iotests: restrict 254 to support only qcow2

Test fails at least for qcow, because of different cluster sizes in
base and top (and therefore different granularities of bitmaps we are
trying to merge).

The test aim is to check block-dirty-bitmap-merge between different
nodes functionality, no needs to check all formats. So, let's just drop
support for anything except qcow2.

Reported-by: Max Reitz <address@hidden>
Signed-off-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: f22356d95524347ff255ae58a96f0f7052018d4e
      
https://github.com/qemu/qemu/commit/f22356d95524347ff255ae58a96f0f7052018d4e
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Fix options leakage in img_rebase()

img_rebase() can leak a QDict in two occasions.  Fix it.

Coverity: CID 1401416
Fixes: d16699b64671466b42079c45b89127aeea1ca565
Fixes: 330c72957196e0ae382abcaa97ebf4eb9bc8574f
Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 681b86ac506bf994a92f17e5a8dedb278aab6ede
      
https://github.com/qemu/qemu/commit/681b86ac506bf994a92f17e5a8dedb278aab6ede
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M qapi/block-core.json

  Log Message:
  -----------
  qapi/block-core: Overlays are not snapshots

A snapshot is something that reflects the state of something at a
certain point in time.  It does not change.

The file our snapshot commands create (or the node they install) is not
a snapshot, as it does change over time.  It is an overlay.  We cannot
do anything about the parameter names, but we can at least adjust the
descriptions to reflect that fact.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: a2bb6f8c920066fba7bd3fdddf214ea40c0fafab
      
https://github.com/qemu/qemu/commit/a2bb6f8c920066fba7bd3fdddf214ea40c0fafab
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M blockdev.c
    M tests/qemu-iotests/085.out

  Log Message:
  -----------
  blockdev: Overlays are not snapshots

There are error messages which refer to an overlay node as the snapshot.
That is wrong, those are two different things.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Message-id: address@hidden
Reviewed-by: John Snow <address@hidden>
Reviewed-by: Alberto Garcia <address@hidden>
Signed-off-by: Max Reitz <address@hidden>


  Commit: 3d96cb91d7812c27142e0c722482bfd35661e50c
      
https://github.com/qemu/qemu/commit/3d96cb91d7812c27142e0c722482bfd35661e50c
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M qemu-img.c

  Log Message:
  -----------
  qemu-img: Move quiet into ImgConvertState

Move img_convert()'s quiet flag into the ImgConvertState so it is
accessible by nested functions.  -q dictates that it suppresses anything
but errors, so if those functions want to emit warnings, they need to
query this flag first.  (There currently are no such warnings, but there
will be as of the next patch.)

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 8eaac025fb43b16a38671e6b80e7059cd8af3403
      
https://github.com/qemu/qemu/commit/8eaac025fb43b16a38671e6b80e7059cd8af3403
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M qemu-img-cmds.hx
    M qemu-img.c
    M qemu-img.texi

  Log Message:
  -----------
  qemu-img: Add salvaging mode to convert

This adds a salvaging mode (--salvage) to qemu-img convert which ignores
read errors and treats the respective areas as containing only zeroes.
This can be used for instance to at least partially recover the data
from terminally corrupted qcow2 images.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 16789db3de64147c7271d30a9d379b2a8aa2b81e
      
https://github.com/qemu/qemu/commit/16789db3de64147c7271d30a9d379b2a8aa2b81e
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M block/blkdebug.c
    M qapi/block-core.json

  Log Message:
  -----------
  blkdebug: Add @iotype error option

This new error option allows users of blkdebug to inject errors only on
certain kinds of I/O operations.  Users usually want to make a very
specific operation fail, not just any; but right now they simply hope
that the event that triggers the error injection is followed up with
that very operation.  That may not be true, however, because the block
layer is changing (including blkdebug, which may increase the number of
types of I/O operations on which to inject errors).

The new option's default has been chosen to keep backwards
compatibility.

Note that similar to the internal representation, we could choose to
expose this option as a list of I/O types.  But there is no practical
use for this, because as described above, users usually know exactly
which kind of operation they want to make fail, so there is no need to
specify multiple I/O types at once.  In addition, exposing this option
as a list would require non-trivial changes to qemu_opts_absorb_qdict().

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: f8cec157cb7f04cb247786f8776ebfcebd9b61be
      
https://github.com/qemu/qemu/commit/f8cec157cb7f04cb247786f8776ebfcebd9b61be
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M block/blkdebug.c
    M qapi/block-core.json

  Log Message:
  -----------
  blkdebug: Add "none" event

Together with @iotypes and @sector, this can be used to trap e.g. the
first read or write access to a certain sector without having to know
what happens internally in the block layer, i.e. which "real" events
happen right before such an access.

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 1adb0b5e0f7c5c4f707b1ffa98e2b15ef51ebbc5
      
https://github.com/qemu/qemu/commit/1adb0b5e0f7c5c4f707b1ffa98e2b15ef51ebbc5
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M block/blkdebug.c
    M qapi/block-core.json

  Log Message:
  -----------
  blkdebug: Inject errors on .bdrv_co_block_status()

Signed-off-by: Max Reitz <address@hidden>
Reviewed-by: Eric Blake <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: 0b1eb0ce7d734affdc181ca583d12947d17e47bf
      
https://github.com/qemu/qemu/commit/0b1eb0ce7d734affdc181ca583d12947d17e47bf
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    A tests/qemu-iotests/251
    A tests/qemu-iotests/251.out
    M tests/qemu-iotests/group

  Log Message:
  -----------
  iotests: Test qemu-img convert --salvage

This test converts a simple image to another, but blkdebug injects
block_status and read faults at some offsets.  The resulting image
should be the same as the input image, except that sectors that could
not be read have to be 0.

Signed-off-by: Max Reitz <address@hidden>
Message-id: address@hidden
Tested-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
[mreitz: Dropped superfluous printf from _filter_offsets, as suggested
         by Vladimir; disable test for VDI and IMGOPTSSYNTAX]
Signed-off-by: Max Reitz <address@hidden>


  Commit: 21c1ce592a144188dfe59b9e156a97da412a59a2
      
https://github.com/qemu/qemu/commit/21c1ce592a144188dfe59b9e156a97da412a59a2
  Author: Max Reitz <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M tests/qemu-iotests/082
    M tests/qemu-iotests/082.out

  Log Message:
  -----------
  iotests: Test qemu-img convert -C --salvage

We do not support this combination (yet), so this should yield an error
message.

Signed-off-by: Max Reitz <address@hidden>
Tested-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Reviewed-by: Vladimir Sementsov-Ogievskiy <address@hidden>
Message-id: address@hidden
Signed-off-by: Max Reitz <address@hidden>


  Commit: f3d0bec9f80e4ed7796fffa834ba0a53f2094f7f
      
https://github.com/qemu/qemu/commit/f3d0bec9f80e4ed7796fffa834ba0a53f2094f7f
  Author: Peter Maydell <address@hidden>
  Date:   2019-06-14 (Fri, 14 Jun 2019)

  Changed paths:
    M block/blkdebug.c
    M blockdev.c
    M hw/block/fdc.c
    M python/qemu/__init__.py
    M qapi/block-core.json
    M qemu-img-cmds.hx
    M qemu-img.c
    M qemu-img.texi
    M tests/qemu-iotests/082
    M tests/qemu-iotests/082.out
    M tests/qemu-iotests/085.out
    M tests/qemu-iotests/175
    M tests/qemu-iotests/175.out
    M tests/qemu-iotests/219
    A tests/qemu-iotests/251
    A tests/qemu-iotests/251.out
    M tests/qemu-iotests/254
    A tests/qemu-iotests/256
    A tests/qemu-iotests/256.out
    M tests/qemu-iotests/group
    M tests/qemu-iotests/iotests.py

  Log Message:
  -----------
  Merge remote-tracking branch 'remotes/maxreitz/tags/pull-block-2019-06-14' 
into staging

Block patches:
- Allow blockdev-backup from nodes that are not in qemu's main AIO
  context to newly added nodes
- Add salvaging mode to qemu-img convert
- Minor fixes to tests, documentation, and for less Valgrind annoyance

# gpg: Signature made Fri 14 Jun 2019 14:38:11 BST
# gpg:                using RSA key 91BEB60A30DB3E8857D11829F407DB0061D5CF40
# gpg:                issuer "address@hidden"
# gpg: Good signature from "Max Reitz <address@hidden>" [full]
# Primary key fingerprint: 91BE B60A 30DB 3E88 57D1  1829 F407 DB00 61D5 CF40

* remotes/maxreitz/tags/pull-block-2019-06-14:
  iotests: Test qemu-img convert -C --salvage
  iotests: Test qemu-img convert --salvage
  blkdebug: Inject errors on .bdrv_co_block_status()
  blkdebug: Add "none" event
  blkdebug: Add @iotype error option
  qemu-img: Add salvaging mode to convert
  qemu-img: Move quiet into ImgConvertState
  blockdev: Overlays are not snapshots
  qapi/block-core: Overlays are not snapshots
  qemu-img: Fix options leakage in img_rebase()
  iotests: restrict 254 to support only qcow2
  hw/block/fdc: floppy command FIFO memory initialization
  iotests: Fix intermittent failure in 219
  iotests: Filter 175's allocation information
  event_match: always match on None value
  iotests: add iotest 256 for testing blockdev-backup across iothread contexts
  iotests.py: rewrite run_job to be pickier
  QEMUMachine: add events_wait method
  iotests.py: do not use infinite waits
  blockdev-backup: don't check aio_context too early

Signed-off-by: Peter Maydell <address@hidden>


Compare: https://github.com/qemu/qemu/compare/5ec2eca83dc4...f3d0bec9f80e