Hi Andrew (and hi Valentin, nice to see you on the list again!),
I'm with you in the animating spirit of your observations about spacing (especially horizontal spacing). Proportional spacing (which I helped Han-Wen scope when he implemented it more than a decade ago) is very powerful indeed. But I've engraved every single one of my scores (lots of tuplets and rhythmic processes of all sorts going on, though I don't claim a complexist mantle) in Lily for 13 years now, and reaching *full mastery* of horizontal spacing in Lily still eludes me ... though I hasten to add that my ability to control spacing of all types in Lily continues to improve year over year, and the resulting layout of the scores continues to improve quite significantly as a result. And I would never attempt what I do now in my own music in other engraving package.
Sometime soon I'd like to find the time to summarize some of what I've learned about Lily's spacing (and also what still eludes me, which is rather a bit), but it's a daunting task. In the meantime I want to point to a bug that I've needed to report to the -bug list for sometime now (will open formally on -bug soon) and which might help explain some of what you've experienced with spacing sections:
While I can't vouch for the complete set of behaviors that you get when you use \newSpacingSection, using the command *does largely seem to work* for me as a way of laying out rhythmically complex music. The basic idea (which I'm sure you know, and have probably tried in your engraving of complexist scores) is: call \newSpacingSection at the beginning of a bar (otherwise Lily will ignore the command); and at that same moment, reset propotionalNotationDuration to best match the contents (rhythmic density, primarily) of the measure at hand. It was a years-long evolution getting there, but I now do this *to every measure* in all my scores, and it seems to be the best first-order strategy for optimizing spacing. (Why "first-order"? Because there are still always a million-and-one second- and third-order steps involving turning off the horizontal widths of items, which is pretty much exactly what you were pointing to earlier in this thread.)
The outstanding bug, though, is this: Lily systematically overstretches the horizontal width of system-final measures that carry a \newSpacingSection at their start! The factor of over-stretching (by eye-balling) is very close to 35/24. Meaning that the last measure of any system that you spacing-control in the way I've described here (with \newSpacingSection and proportionalNotationDuration) will be almost exactly 35/24 *longer* than what you've specified. It's a truly unique behavior (bug) in Lily; nothing else in the system seems to work quite this way.
The workaround is, of course, that scale your proportionalNotationDuration by the reciprocal of the overstretching, ie, 24/35.
Later on a much fuller discussion (and *documentation*) of the entire collection of spacing concerns would be truly useful. Hope this small breadcrumb might be useful for you in the nearterm.
Trevor.