Re: [Pika-dev] xl as config file, programmatic manipulation pitfall

[Jeremy Shaw]

At Thu, 30 Sep 2004 15:07:14 -0700 (PDT),
Thomas Lord wrote:
> 
> 
>     > From: Jeremy Shaw <address@hidden>
> 
>     > It is my suggestion that the xl comments be designed so that programs
>     > can be programmatically transformed without screwing up the comments.
> 
> Absolutely.
> 
> Comments should be first-class syntactic elements and should be
> explicitly "attached" to something else in the parse to which the
> comment refers.

Agreed.
 
> Additionally, comments have an internal and mostly-opaque-to-tools
> concept of formatting.  For example, if my comment contains an
> "ascii art" table of keywords and values, then all source-level
> transformations that preserve that comment should also preserve 
> the ascii art, no?

Definately.
 
> In some sense, these issues are largely orthogonal to the parts of xl
> i've been concentrating on: semantics and operational model.

That is what I am (still) trying to figure out. In most languages,
code and comments are strictly different. The code is the only thing
the compiler looks at, and the comments are useless pieces of text
that get ignored by the compiler. Even if my comment says,

/* !!!! DO NOT CHANGE THE && TO AN || IN THE NEXT LINE !!!!*/

it has no real power. I want to know what would happen if we had the
compiler look not only at code, but also at comments.

The closest example I have seen is adding optional type signatures in
haskell. In haskell, I might write a function like:

myfunc a = [a]

(Note: this is a function that takes a value of any type and turns it
into a one element list).

For which the compiler will infer the type:

myfunc :: forall t. t -> [t]

(Note: t is a type variable. The above roughly translates to 'the
function myfunc takes a value of any conceivable type and returns a
list of values of that same type).

Even though haskell does not (usually) require type signatures, many
people still put them in. So I might instead write:

myfunc :: t -> [t]
myfunc a = [a]

I have not changed the functionality or meaning of the code at all.  I
have simply documented the type signature that I want this function to
have.

However, this 'documentation' is enforced by the compiler. If you come
along and try to change the code to this:

myfunc :: t -> [t]
myfunc a = 'r':[a]

(Note: (:) is like cons from lisp)

The compiler will generate the compile time error:

    Cannot unify the type-signature variable `t' with the type `Char'
        Expected type: Char
        Inferred type: t
    In the list element: a
    In the second argument of `(:)', namely `[a]'

You now have two choices:

 (1) Update the documentation (by updating the type signature, or
     perhaps removing the type signature altogether).

 (2) Realize that the documentation is correct, and that your code is
     wrong.

On the other hand, the type signature can also be used to change the
meaning of the function. Going back to my original function
defination, I can specify a more specific type signature:

myfunc :: Int -> [Int]
myfunc a = [a]

So, in that instance, the type signature is more than just a
'comment'.

What I find interesting about the above examples is that the
documentation is optional, but has enforcement power. If you modify
code, you have to update or remove the documentation. I think removing
the documentation is possible a better option than leaving incorrect
documentation. Of course, the tight binding between code and
documentation in the above examples, severely limits what you can say,
because the compiler has to be able to understand it.

I guess, unit tests could also be considered a form of active
comments. I wonder what would happen if you allowed unit tests to be
added directly inline with the code. Imagine if instead of only being
able to test whole functions, you could even unit test certain blocks
of code inside a function... I think that is the sort of stuff people
who write software for airplanes and telephone switches like to do...

> Another issue is whether or not to add explicit *internal* structure
> to comments.   For example, if part of a comment is a "doc string"
> that an interactive system might want to translate into a help
> message: can that doc string part of the comment be explicitly
> specified at the xl syntax level rather than at the level of some
> post-processor that parses the strings an xl parser has identified as
> a comment?

I think there is a benefit to be gained. Many languages now have some
sort of javadoc type documentation, but the compiler typically does
not know anything about the autodoc language, so it can not provide
any automation or checking. And, often times, the third party tool
that turns the comments into html/ps/pdf only parses the comments and
does not understand the actual source.

For something like a config file, people *will* write GUIs for editing
it. So it seems proper to provide the ability to send useful
information to a GUI. Some tooltip text for each field would be a
start, but even better would be to provide something so that the GUI
could do smart layout and input validation. It seems reasonable that
the layout hints and input validation could be expressed as bits of xl
code that are sent to the GUI.

>   What about that hypothetical ascii-art table: should it
> be generically parsable in such a way that xl code can do an ASSOC
> lookup in the table?

That is the type of thing I am interesting in finding out. In that
particular example, you prevent the comments and the code from getting
out a sync, which seems like a good thing. Is it pragmatic? I don't
know...

> Yet another issue is code->code transforms.  Structured comments
> complicate those.  Do you have any thoughts about how to write, for
> example, a pattern-matching/template-based-rewrite system that handles
> structured comments gracefully? 

> (abstracting away the "1") but would the same pattern also match
> a hypothetical construct like:
> 
> 
>       for (  /* start at 1 not 0 because ... */->(x = 1);
> 
> where my little pseudo-syntax is intended to indicate a comment
> attached to a particular subexpression?

I think this is where things get tricky -- perhaps too tricky. If I
change the x = 1, to x = 2, then your comment is wrong. If a person is
editing the code by hand, hopefully they will notice and update
it. But, let's say that the value was updated by some program -- the
program is definately not going to update the comment.

That sort of problem occurs all the time if you try to do something
like write a GUI that modifies samba config files. The config file
contains all sorts of useful comments. If you let a program modify the
file, over time you start to get comment rot. You end up with lots of
carefully preserved comments that have completely lost their original
useful context.

I am not sure what to do in the above case? Maybe the program that
updates the x = 1 to x = 2, also gives the user the opportunity to
update the comment? Or maybe comments have some sort of rating to
indicate their likelyhood of validity.

You might write:

>       for (  /* start at 1 not 0 because ... */->(x = 1);

but if some program edits that line it changes it to:

>       for (  /? start at 1 not 0 because ... ?/->(x = 2);

To indicate that the comment may be out of date ?

That's all I have for now...

Jeremy Shaw.
--

This message contains information which may be confidential and privileged. 
Unless you are the 
addressee (or authorized to receive for the addressee), you may not use, copy 
or disclose to anyone 
the message or any information contained in the message. If you have received 
the message in error, 
please advise the sender and delete the message.  Thank you.