Re: sphinx-build is really slow, any way to improve that?

[Daniel P . Berrangé]

On Tue, Sep 06, 2022 at 02:41:13PM +0100, Peter Maydell wrote:
> On Tue, 6 Sept 2022 at 08:55, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > On Mon, Sep 05, 2022 at 10:21:55PM +0100, Peter Maydell wrote:
> > > On Mon, 5 Sept 2022 at 20:51, Claudio Fontana <cfontana@suse.de> wrote:
> > > > when I build qemu, there is a lot of time spent at the end of the build 
> > > > where one cpu goes 100% on sphinx-build.
> > > >
> > > > Is there some way to parallelize that? It seems it is the current 
> > > > bottleneck for rebuilds for me..
> > >
> > > It's a big fat python program, so I suspect not, but
> > > maybe I'm wrong.
> >
> > It annoys me too and I've had a look at what it is doing in the past and
> > failed to find an obvious way to improve it. I fear this could be an
> > inherant limitation of the way we use sphinx to build the docs as a
> > complete manual, as compared to say treating each docs source file as
> > a distinct standalone web page.
> 
> IIRC sphinx really really wants to process the whole document tree
> in one go. You can see this in the way that for example the
> HTML build process creates HTML files for the top-level rst
> files that are supposed to be only for the manpage -- it will
> suck in and process everything, not just the files reachable
> via whatever top level file you point it at.

Yeah, thats why I think we're limited by what sphinx upstream can do
for us. They need to be able to parallelize stuff in their loading
and generation code.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|