Re: [Qemu-devel] Exploring Sphinx, autodoc, apidoc, and coverage tools for python/qemu

[John Snow]

On 7/26/19 5:16 PM, Eduardo Habkost wrote:
> CCing Cleber and Gabriel.  Comments at the "conclusions" section
> below:
> 
> On Wed, Jul 24, 2019 at 05:06:41PM -0400, John Snow wrote:
>> Has anyone on this list experimented with these tools?
>>
>> I was hoping to use them to document things like the python/machine.py
>> and python/qmp.py modules to help demonstrate some of our internal
>> tooling API (for test writers, GSoC/Outreachy interns, folks who want to
>> script QEMU at a level between writing a CLI driver and using libvirt.)
>>
>> What follows below is my process trying to enable this and some of the
>> problems I'm still stuck with, summarized below at the end of this more
>> exploratory text.
>>
>>
>> Enabling autodoc:
>>
>> First, it appears as if enabling the "sphinx-autodoc" tool is not
>> sufficient for actually generating anything at all when you invoke the
>> sphinx-generated "make html" target. It just enables understanding
>> certain directives.
>>
>> So apparently you need to generate module "stubs" using sphinx-autodoc.
>> Sphinx uses the sphinx-autodoc extension to understand how to consume
>> the directives in these stubs.
>>
>> That strikes me as odd, because these stubs might need to be changed
>> frequently as code comes and goes; it seems strange that it isn't
>> integrated at the top level. (Do I have the wrong idea on how these
>> tools should be used?)
>>
>> So you need to run:
>>> sphinx-apidoc --separate --module-first -o docs/ python/qemu/
>>
>> which generates stubs to docs:
>>
>> Creating file docs/qemu.machine.rst.
>> Creating file docs/qemu.qmp.rst.
>> Creating file docs/qemu.qtest.rst.
>> Creating file docs/qemu.rst.
>> Creating file docs/modules.rst.
>>
>> And then you can edit e.g. the top-level index.rst TOC in docs/index.rst
>> to look like this:
>>
>> ```
>> .. toctree::
>>    :maxdepth: 2
>>    :caption: Contents:
>>
>>    interop/index
>>    devel/index
>>    specs/index
>>    modules
>> ```
>>
>> And then finally generating the build; manually removing the -W option
>> from the Makefile: there are a lot of warnings in here.
>>
>>> sphinx-build -n -b html -D version=4.0.92 -D release="4.0.92
>> (v4.1.0-rc2-34-g160802eb07-dirty)" -d .doctrees/
>> /home/bos/jhuston/src/qemu/docs/ docs/
>>
>> Great! that will generate output to docs/index.html which indeed shows
>> APIdoc comments generated from our Python files. Good.
>>
>> However, where this gets a little strange is if you look at the
>> generated stubs. For instance, qemu.machine.rst looks like this:
>>
>> ```
>> .. automodule:: qemu.machine
>>     :members:
>>     :undoc-members:
>>     :show-inheritance:
>> ```
>>
>> :undoc-members: says that we want to "document" any members that don't
>> have a matching apidoc comment by generating a stub.
>>
>> Oops, but the presence of that stub will cause the sphinx coverage tool
>> to happily report 100% coverage.
>>
>> Further oops, pylint doesn't understand apidoc comments and can't be
>> used as the linter in this case, either.
>>
>> You can edit the stubs to remove these directives, but these stubs are
>> generated -- and it doesn't appear like there's a command line option to
>> change this behavior. ...Hmm.
>>
>> And either way, the coverage tool only generates a report and not
>> something with an error code that I could use to gate the build. Same
>> goes for the general build: if I remove the :undoc-members: parameter,
>> there's nothing in the autodoc module that appears to throw warnings
>> when it encounters undocumented parameters or members.
>>
>> That seems disappointing, because it's hard to keep docstrings up to
>> date unless they are checked conclusively at build time.
>>
>>
>> Conclusions:
>>
>> - the autodoc documentation page doesn't seem to document examples of
>> how you're expected to write meaningful docstrings for the tool to extract.
> 
> I had the same impression when I read sphinx/autodoc
> documentation.
> 
>>
>> - autodoc fools the coverage tool into reporting 100% coverage.
>>
>> - autodoc can be configured to omit non-documented members to allow the
>> coverage tool to work, but the configuration is auto-generated and
>> defaults to always generating documentation for these entities.
>>
>> - coverage tool doesn't appear like it can be used for gating the build
>> natively for missing python docs; it only generates a report.
>>
>> - Even if we script to block on a non-empty report, the coverage tool
>> only works at the function/class level and does not understand the
>> concept of missing parameter or return value tags.
>>
>> - It would seem that it would be the Autodoc module's job to be
>> responsible for understanding incomplete documentation, but doesn't
>> appear to. The :param name: syntax is just a ReST "field list" and isn't
>> parsed semantically by autodoc, sadly.
> 
> I wonder if there are other Python documentation coverage tools
> outside Sphinx.  Googling for [python docstring coverage]
> resulted in a few different projects, but I don't know which ones
> would understand Sphinx-compatible rST docstrings.
> 
>>
>>
>> It looks to me, at a glance, that there's nothing in Sphinx that knows
>> how to look for and warn about undocumented parameters, exception types,
>> return values, etc. Hopefully I've missed something and it is possible.
> 
> Personally, my first priority would be generating reasonable
> documentation from docstrings, preferably integrated with the
> rest of the QEMU docs.  Gating based on docstring coverage checks
> would be just nice to have.
> 

Right, and for THIS, we can simply do a pylint check. we can probably
target some pylint subset when we drop python2 support entirely and
start keeping some minimum code quality metric.

--js