Re: Presentation: "Publisher-grade LilyPond" in Ottawa

[Boris Shingarov]

Hi Graham and all Lilyponders,

I was quite astonished to hear that my slides were understood to mean anything pessimistic or negative.  If they give people this impression, then it is a defect of the slides which I will fix.

But right now, let me address some of the apparent misunderstandings.

Are you seriously going to submit a paper to CMJ saying "a volunteer
open-source project has limited resources for fixing bugs" ???
  

This is not what we are saying in our presentation and/or paper.
What we are saying is: "We attempted a publication of a major critical edition through a major publishing house, using software from a volunteer open-source project with limited resources.  We first hoped that this software would be immediately (or almost immediately) suitable for critical-edition work.  We found issues blocking this work.  These issues are orthogonal to the main direction of the LilyPond project.  We fixed them, making the book possible.  Future work includes making these solutions useful for the wide LilyPond audience, not just for the immediate needs of this particular book".

Oh, and I hope that this "paper to appear" includes an excellent
reason why you didn't use lilypond-book and LaTeX, which unlike
lilypond itself is *designed* to mix music and text.
  

Two reasons, really.

Reason one, we did investigate that route.  Lilypond-book and LaTeX do not achieve the kind of music/text integration that the editor demanded.  Another issue was integrating with their existing database of variant readings.  And for me, there was the obstacle that it was not what the end user was asked for.

Reason two, the cry for help was heard all over the mailing list starting many, many months ago.  Anyone with a LaTeX-based solution yet?  Anyone even *suggesting* a LaTeX-based solution yet?  After a year and non-trivial (thousands of Euros) bounties offered?

I agree that the presentation has a disproportionately large first (introductory) part, which makes it seem like a "presentation about LilyPond", while it is really the second part which is the essence.  There is a background why I chose to extend this first part to be somewhat detailed.  Before the presentation at OCLUG, there were two "micro-presentations" I did in a more informal setting.  They ended in somewhat curious Q&A discussions.

The first question -- asked by a guy who is well-known to everyone in the digital typography and content engineering field through dozens of papers and monographs -- he couldn't believe that today, in 2010, when everything programmable into a computer had already been programmed, today there are still books that can not be printed using already written software.  He demanded examples of problems (in music typesetting) which still waited for their programming solutions.  So 45 minutes were devoted to these examples.

At another micro-presentation, there was another question.  "I thought music typography was completely solved by Unicode?  That is, all you need to publish any score, is LaTeX and a font with all the music symbols in it?"  So, another 15 minutes to explain why LaTeX+Feta != music engraving system.

So this is the kind of questions typically asked by those in the audience who care at all.  This is why I felt an extended introduction to LilyPond was needed -- although the real subject of the presentation is mostly orthogonal to LilyPond per se.

his slides would discourage any potential
developers from joining

If there is a person who gets this impression from the slides, then the slides are definitely defective and need fixing ASAP.  I am not sure if the presentation had the same defect or it's just the slides (there certainly was a lot more explaining of the reasons and the background), but they are most definitely not supposed to convey any negative ideas.

"The community showed no interest in addressing these issues", well I am certainly not saying it's the community's *fault* -- the point is that there are other dimensions, which are of concern to [at least one] top publisher, in connection with [at least one] type of book, and which are orthogonal to the dimension of the direction of work of the LilyPond project.  After all, none of these issues originated from me; they originated from the musicologist doing the chant research, and the editor of the book -- before I even started looking at LilyPond -- and were rejected as not even being valid LilyPond problems.  The typical answers were,

"I think that you'll be better served by others"
http://www.mail-archive.com/address@hidden/msg52031.html

"Your post is absolutely unnecessary"
http://www.mail-archive.com/address@hidden/msg52334.html

"I recommend magic"
http://www.mail-archive.com/address@hidden/msg52026.html

Does this represent a problem enough to "discourage" people?  I think it does not, for two reasons.  First, people are smart to take such things with a grain of salt.  If you look through the lkml archives through the last decade, you will see a lot darker atmosphere in that community, yet there are plenty of active kernel developers.  Second, and I insist to repeat it over and over, the main LilyPond project focus, and what we are doing in our "Publisher-Grade" project, are different, orthogonal dimensions.

"Patches were hard or impossible to contribute back" -- this is through no fault of the community.  In fact, working with Joe Neeman -- both to initially produce patches, and to make them conform to the LilyPond standards so they can be pushed into the main codebase, -- has been a highest pleasure, and his help and guidance are priceless.  His time and patience are highly appreciated.  The help from other developers, has also been great (Neil, Carl, my sincere thanks!!)  But this is not what I am talking about; all this work together with Joe and others, it's all about "the LilyPond dimension", while the problems blocking the publisher, are in those other dimensions, and LilyPond being a volunteer project, no one has incentives to do that not-very-core, not-very-exciting work.

So it's a question of the proverbial "scratching one's own itch".

Simplest example: a patch fixes a bug (a Blocker for our real-life project).  The fix is used in production for some time, and seems to be working fine.  Code review on Rietveld, patch looks good to the reviewers.  The only problem delaying its push, is the absence of a test case.  Ok, I spend some time adding a test case, but bump into problems.  Joe is so kind to offer his help in debugging; but to get through the debugging of the problem would seem to require half a day, maybe a day.  Do I understand the importance of having a test case?  Yes!  Do I want a test case?  Yes!!!  But it's about customer value. "A defect is anything that leads to customer dissatisfaction".  Well, the end-user does not care about having a test case.  The time is very tight, and we have other pressing priorities (like other Blocker bugs).  He would appreciate the inclusion of the work in the main LilyPond codebase, but not to the extent of stopping work on the other Blocker bug in order to write the test case so that the fix gets pushed to the main repo.  If it's an hour, yes.  But five hours...let it sit in the vendor tree.

There are also patches that require a lot more work to become universally useful to all LilyPond users, not just to our book here.  When a patch takes 3 days from start to deployment in production, and half a day to polish to be pushed into main repo, that's fine, I do that half-day.  When a patch takes the same 3 days, and would need another 5 days to be able to conform to LilyPond's standards, I might, but the customer would not appreciate it, so *having* the patch would in effect constitute a defect.

What's the way out of this?
Well I *hope* that by solving these "other-dimension" problems, we will be able to make LilyPond a platform for serious projects which will support its development.  That way, we will be able to afford real professional development; afford full-blown solutions, to meet all standards, and be useful to the community at large; get everything from my vendor tree into the main tree, and have no need for a vendor tree any more.  This is what I meant when I wrote that last "Future Work" slide.

On 06/12/2010 11:28 AM, Reinhold Kainhofer wrote:

Flowing text with music is simply out of the scope of lilypond. That would 
mean that a full text layout system is implemented, too. Otherwise things like 
hyphenation or widows/orphans cannot be properly handled. Ideally, of course, 
we would have someone / some company working on that, but the resources are 
simply limited.
  

Dear Dr. Kainhofer, your posts on the list have been very reassuring to me, that my work on LilyPond is going in the right direction.  What you observe about your work on your critical editions, agrees very well with my understanding of publisher-grade work.

Widows/orphans are already implemented (and even pushed! back in January I think).  Although the implementation is very limited (for example, the length of the line is not taken into account), but the Editor is pleased with the resulting look of the book, and then again, "a defect is anything that leads to customer dissatisfaction".

Yes, the text layout system is extremely primitive.  Even worse than hyphenation, the line-breaker is a simply greedy word-wrapper, and worst of all, the stretchability of the embedded music does not mingle well with that of text.  But again, the result pleases the Editor, and the first volume of the book will be accepted with this slightly imperfect stretching.  If we get support to continue our work to improve LilyPond, there will be further progress in the text layout system.

And, as you note, fragment line breaking is extremely complex...

The key here is to be very conscious of what limitations we are willing to accept.
If we make our mind that we only have the resources to within the limitations of the greedy line breaking, fragment line breaking suddenly becomes very easy.  We have it working in production since February.

I first tried lilypond, too, 
but soon switched to latex. I don't have to mix music with text, but only 
insert some figures

That's the thing -- we HAVE to mix music with text, and because if the specifics of the plainchant we are dealing with, fragments HAVE to flow seamlessly with the text.

I don't think there is zero interest in having the problems fixed, rather the 
contrary. However, there is zero time to address them ourselves.

What I am trying to do, is create some sort of professional LilyPond ecosystem, where people would be allowed to spend serious amount of time on LilyPond work, but where problems would actually get fixed.  If a publishing project is willing to spend many thousand dollars to fix a certain problem, and the only kind of response they get in a year, is "I recommend magic" and "Your post is absolutely unnecessary", that kind of interest-in-having-the-problems-fixed does not really matter to the publishing project.

And, btw, I'm very interested in footnotes, as I'd need them with my critical 
editions, too...

At this point, we temporarily put them on the back burner.
When the question about footnotes was asked on the list, no one raised their hand saying "I'll do it".  Plus, footnotes would be probably less good in the book we are preparing than endnotes, because of the huge amount of variants.  If we place the critical apparatus in footnotes, it will occupy almost the whole page, making the book difficult to use for practical singing.

BTW, have you fixes for the vertical alignment been pushed yet?

You mean, the vertical space estimation?  There were several bugs screwing height-estimation.  Some of those fixes are pushed, and some are still sitting on Rietveld.  Which ones specifically were you interested in?  Or do you mean the text / embedded score alignment?  The fix to that, requires as a prerequisite the "multi-line embedded score" fix which is sitting on Rietveld.

Boris