Re: [Cfengine-develop] my own branch of the tree

[Luke A. Kanies]

On Wed, 14 Jan 2004 address@hidden wrote:

>
> Luke,
>
> I agree with much of what you say, but you seem to mix rational sense
> with spurious complaints. I would also like iteration. I just don't know
> how to do it. Do you? I do know that none of this has got anything to do
> with the parser. As I explained at LISA, you cannot write the kind of
> parser you want until the language is completely known. As long as it
> continues to develop, it makes no sense to go on about this. It also has
> nothing to do with CVS. I get patches all the time, I just don't get
> development. Also "everyone I speak to" cuts no weight with me. I talk
> to people too.

No, I don't have a great solution for iteration right now, but I think
that partly is the fault of the parser.  If I were to be able to just
willy-nilly modify the syntax, I would probably do something that looked a
heckuva lot like ruby, rather than perl:

control:
        list = ( 1 2 3 4 )

copy:
        ${list}.each { |item|
                /path/to/files/${item} dest=/new/path/${item}
        }

links:
        ${list}.each { |item|
                /usr/local/bin -> /usr/local/software/${item}/bin/*
        }

I'm not saying that this is _the_ solution, but once you get used to it,
it continues to make sense.

Another option is to implicitly support iteration so that the syntax does
not change at all:

control:
        list = ( 1 2 3 4 )

copy:
        /path/to/files/${list} dest=/new/path/${list}

links:
        /new/place/${list} -> /old/place/${list}

I know links already kind of supports this, but it's using a standard
string variable, not an actual list variable.  One of the real problems
with this type of iteration is that you can't then do nested iterations,
and you can't really do anything resembling hashes (associative arrays)
either, and I think they're also important.

However, I definitely think the parser is related to making cfengine
better.  There are a number of aspects of the parser that limit
abstraction in cfengine.  For instance, I have multiple cfengine servers,
and I want to collect their IP addresses into a variable and use that
variable for access control:

control:
        servers = ( 170.150.135.232 170.150.135.235 )
        AllowConnectionsFrom = ( ${servers} )
        TrustKeysFrom = ( ${servers} )

But I can't do that because of the way the parser is written.  I could
point out the details, but essentially, it looks for a value and is
willing to split that value based on space, _or_ it looks for a variable
and interprets the value of the variable as one value.  What's worse, this
value is then not sanity checked.

Until recently it was possible to easily bypass sanity checks on class
names by using ExecResult with AddClasses and AddInstallable.  This is
because the parser does variable interpolation at the same level as it
does value acceptance; it instead needs to do all of the interpolation
prior to value acceptance, which would automatically make sanity checking
happen to variable values.  Bash is a great example of how to do this
right; in fact, there's a diagram in the Bash book by O'Reilly which walks
through the way that bash parses a command.  It does every item (variable
interpolation, tokenizing, line splitting, alias lookup, etc.) in a
specific order, and everything is tightly defined.  As a result, you get
very little confusion about why something happens the way it does.

> You have done good things with cfengine and show great insight im many
> things. If I don't fully trust you it is because you seem too
> emotionally attached to your own vision. I would love to harness your
> talents. Can you enter into a discussion and accept a viewpoint that you
> do not agree with? Every message from you is intense to the point of
> flaming. I embrace the rational and shy away from heated blows.

Hmmm.  Yes, I believe that I can accept a viewpoint that I don't agree
with; at least, I have many times in the past.  I am not terribly happy at
accepting the idea that what I want to do is somehow "wrong" or "not the
cfengine way", however.

I am emotionally attached to my vision, I agree with that, but the reason
I'm emotionally attached is because every time I try to do anything with
cfengine I have to go through a painful process of discovery, figuring out
what it does differently from what I expect, reading the source to figure
out why it does it that way and if I can work around it, and etc.

I'm not exaggerating when I say that 4/5 times that I make a significant
expansion to my cfengine configuration (meaning that I begin managing a
different aspect of my systems) I have to read through a lot of source
code.  Discovering the above problem with the server list was definitely
not fun, nor was discovering what characters AddClasses and AddInstallable
split on (yep, they're different; nope, they're not documented; yep, I
should have submitted a patch to the docs).

The amount of pain that I go through while trying to use cfengine for
abstraction is what makes me emotionally invested in it.  I really do not
understand how anyone else is able to avoid running into these problems,
but I have spoken to others who do run into them.

I get frustrated, Mark, because it often seems like you believe that if
it's not a problem for you it is not a problem for anyone else, and that
is definitely not the case.  You and I obviously do things very
differently, and our relative frustrations with cfengine are a reflection
of that.

> I am very interested to hear about what you intend to do about iteration.
> If you suggest something that I find convincing, I will happily work out
> a solution that works for both of us. If you think that I will just open the
> code for you to do whatever takes your fancy then, no that would be 
> irresponsible.

As I said above, I don't necessarily have a solution for iteration.  I
think that if it were up to me, I would use the ruby-style example above.
But if it were up to me, I would probably rewrite most of the parser,
which I think will be necessary to support such a thing.

We've already discussed having write access to the repository; I don't
necessarily need that, but it takes a good bit of effort to build an
environment of ongoing development around a project, and there is no such
environment around cfengine.  As far as I can tell, it's currently a one
man project and you are quite busy, which means it doesn't advance very
quickly.

I am of the opinion that cfengine's code base needs significant
refactoring, which makes sense given its age and history.  You are
probably never going to have the time to do that refactoring, which means
you need to find a way to encourage others to help you with it.  I don't
know exactly how one encourages this, but I do have some ideas, and at
least providing read access to CVS is one of them.

I expect you would get many more patches if you had read access to CVS,
and I also expect that you get many more people testing the latest version
of the code, which is pretty important for vetting patches.

> I enjoyed speaking to you in private. So let's try to elevate the discussion
> from politics to solutions.

I definitely enjoyed speaking with you at LISA, and I am hoping to work
something out so we can both be happy.  I do think there's an element of
politics to resolve, however, as usual.

At this point you are the final arbiter of what "the cfengine way" is.  I
apparently often produce recommendations that you don't believe fit into
that category but that I do.  The question of how to resolve this
conflicts is a question of politics, unless it can somehow be reduced to a
purely technical discussion.

If you are able to provide technical justifications for what you don't
like about my recommendations then I agree that politics can be left out,
but if the justifications are more "because I don't think they fit", then
it becomes a political problem.  I'm fine not dwelling on this issue, but
I think it's going to rear its ugly head a number of times, because it has
already come up with most of my recommendations.

On a technical level, though, I would like to take cfengine's parsing
towards the way that bash does its parsing.  No, not the syntax, not the
function, just the parsing.  Bash not only has a very good parsing
mechanism, its actual parsing code is some of the cleanest I've seen, and
I think cfengine would do very well to learn from it.

-- 
"Anyone who considers arithmatical methods of producing random digits
is, of course, in a state of sin."    -- John Von Neumann