bug#9088: Java, JARS primary?

[Stefano Lattarini]

On 05/17/2013 05:50 AM, Michael Zucchi wrote:
> On 16/05/13 19:39, Stefano Lattarini wrote:
>> On 05/16/2013 05:57 AM, Michael Zucchi wrote:
>>> On 15/05/13 22:39, Stefano Lattarini wrote:
>>>> On 05/15/2013 01:52 PM, Michael Zucchi wrote:
>>>>> On 14/05/13 03:47, Stefano Lattarini wrote:
>>>>>
>>> Well if that's a requirement, then it just has to be added right?  It's
>>> not impossible, it's only software ...
>>>
>> I failed to parse this.  Can you rephrase?
> 
> It means the problem is an economic one, not a technical one.
> 
> It's a mostly facetious saying coined by a fellow work-mate which
> basically means 'we can do whatever you want (but it will cost you)'.
> 
> Or broken down logically, the technical problem:
> 
> a) We are solving a problem.
> b) We are using a general purpose computer to solve it.
> c) Any problem can be solved by a general purpose computer.
> d) Therefore, we can solve any problem.
> 
> (where of course, problem is a computer related problem, which this
> clearly is)
> 
> Here the cost is time that could be spent elsewhere, and maybe some
> lost/grey hairs.
> 
>>>> Doing too much pre-processing in automake.in (that is, at Automake time)
>>>> would prevent us from being able to grasp GNU make specific stuff.  So
>>>> let's try to keep such pre-processing at a minimum.
>>>
>>> I'm only talking about iterating a list of strings and printing them in
>>> various ways, i'm pretty sure it does that kind of thing already.
>>>
>> Yes, but the price is reduce flexibility at Automake time.  Which might
>> be irrelevant or very important, depending on the expected usage patterns
>> of a feature.
> 
> I'm lost here.  You seem to be talking about something I haven't even
> written yet.
>
My point was that, if you need to parse and analyze the content of a
make variable (let's call it FOO) at automake runtime rather than at
make runtime, the contents of that variable will have to be rather
"static"; that is, you lose at least two abilities:

  1. Even if you just assume your users will use GNU make, you won't
     be able to employ GNU make specific functions in the definition
     of FOO, because automake doesn't grasp them anyway.

  2. Your users will likely not be able to override that variable
     at make runtime (which might be nice for debugging or testing
     purpose).

Now, depending on the situation, the intended uses for a feature, etc.,
the loss of those two abilities can either be irrelevant, merely annoying,
or tragic.

> [SNIP]

>>> My initial version listed every file, and used the gnu-make specific
>>> extensions in order to make it not "totally suck".   But then I ran into
>>> problems with how to handle resources and rather than work on solving
>>> those in a way which could fit into an automake.am fragment, I thought
>>> i'd try the approach I had desired from the start.
>>>
>> Given my ignorance of Java, I'm fully ready to change my mind on
>> this; but then I need examples that are much more clear, complete,
>> self-explanatory and *working*.  I think you should start perusing
>> the java tests in the 't/' subdirectory of the Automake source tree,
>> and start writing similar tests that show how you envision your
>> feature to work in different scenarios.  At this stage, using GNU
>> make specific constructs is OK; I promise I won't bother you about
>> them anymore.
> 
> Well the makefile works as much as it was designed to.
>
What I mean is that I'd like to have a test script in 't/', say
't/java-new.sh', that demonstrate how your features work, and that
I can run with a simple "make check TESTS=t/java-new".  That kind
of automation is going great lengths toward making me more willing
to try out your code, think about possible problems, and clarify my
doubts (hopefully).

> I don't really understand how the unit tests work.
>
You can try starting from an existing test, and modify it as fit.  If
you have doubts, take a look at the 't/README' file.  If you get stuck,
just ask here for help.

> It's a bit frustrating trying to make you enough of a java build expert
> in a handful of emails when i'm not one myself.  Perhaps it's just a lot
> simpler than you expect - it has far fewer complexities than
> cross-platform c libraries.
>
I sincerely hope so ;-)  I think most of the issues here are just due to
newbie misunderstanding on my part.  Just be patient.

>>> This particular target was all very simple until i added the LIBADD
>>> dependencies which meant i could no longer just use $^ as i had.
>>>
>>> I couldn't think of a way of having $^ just list .java files whilst
>>> still depending on the LIBADD stuff for running the compiler but if that
>>> was possible the problem would vanish.
>>>
>> Consider that $^ is not portable to all non-GNU makes, IIRC (but I'm not
>> 100% sure of this, I'll have to double check), so that we'd likely still
>> need to rewrite such idioms when we make the transition from your
>> prototype from the final implementation.  Not a big deal right now, but
>> something worth keeping in mind.
> 
> Really?
>
Eh, I'm not really sure.  I know for a fact that '$<' is not portable when
used outside an inference rule (and that limitation is compliant with POSIX).
As for $^, I'd have to double-check on real-world make implementations, but
I hold no great hopes, given that it's not even named in POSIX:

  <http://pubs.opengroup.org/onlinepubs/009695399/utilities/make.html>

> Bummer.  How does one get the list of requirements then, or is
> make pretty much hard-coded for C with a single source file using $<?
>
No, you can't even use '$<' in non-inference rules.  You just have to
explicitly spell out your deps in the recipe.

> Is $? portable?
>
Yes (see link above to POSIX spec).

> This is what java.am uses.  Not that it's any use here.
>
Indeed.

> Obviously it can just use the variables instead, but it then needs to
> handle srcdir manually.
>
There are several Automake-generated rules that do so already (see e.g.,
lib/am/yacc.am); so that wouldn't be something new (sadly).

>>> Actually the end result I need explicitly for jar is this:
>>>
>>>  -C $(srcdir)/src a/a.txt \
>>>  -C $(srcdir)/src b/b.txt \
>>>  -C $(srcdir)/res c/c.txt
>>>
>> But how can the code know, in general, to only strip the first component
>> of the resources file?  That is, why:
>>
>>     -C res c/c.txt
>>
>> and not
>>
>>     -C src/c c.txt
>>
>> ?
>>
>> And it gets worse; if you have a resource in a/b/c/d/foo.txt, which one
>> of these possibilities should be used:
>>
>>    -C . a/b/c/foo.txt
>>    -C a b/c/foo.txt
>>    -C a/b c/foo.txt
>>    -C a/b/c foo.txt
>>
>> ?
> 
> Oh he finally gets it!!!
>
(BTW, this comes across as a little disparaging.  Which is strange, given
your polite tone so far.  So I'll assume you weren't being sarcastic here,
and it's just a "language fail" on my part).

> Yes, this is the problem that needs solving.  Which is why EXTRA_DIST's
> mechanism isn't any use, etc, and so on.
> 
> I'm sorry i took so long to enunciate it but i thought 'relative root'
> was fairly obvious.  Have you never used zip or tar?
>
Of course, but I was missing some context here.  Now my understanding is
roughly this:

  - Placement of *.class files in a precise point of a directory layout
    is extremely important in the Java world.

  - There is an API that allow simple access from Java objects to data
    files ("Resources", in Java lingo).  For that to work properly, the
    accessed data file must be in the same directory of the *.class file
    containing the class of the object that is accessing the data.

  - All of the above remains valid when the directory structure is
    "embedded" in a Jar file.

Please correct me if I'm still harboring some misunderstanding.

> Your name suggests there may simply be a language barrier here.
> 
The language barrier being referred more to the Java language rather
than the English language, though ;-)

>>> b.2)
>>> foo_bar_RESOURCESDIRS = src res
>>>
>> (or foo_bar_RESOURCE_PREFIXES even)
>>
>>> foo_bar_RESOURCES = src/a/a.txt src/b/b.txt root/c/c.txt
>>>
>>> (use regex/sed to strip either root: it can be safely assumed they are
>>> independent trees so this should be easy)
>>>
>> This sounds the best approach so far.  I'd love to see a preliminary
>> implementation of that ;-)
> 
> Ok.  This the one that seems the most simple to implement too.
> 
> When I have something I will return.
>
OK, thanks.

>> I fear HACKING is quite incomplete in this regard; the best reference
>> for what can and can't be done portably with POSIX tools in the
>> Autconf manual, section 11 "Portable Shell Programming":
> 
> Ok thanks.
> 
>>>  Even if
>>> every file was listed by hand these capabilities are still going to be
>>> required.
>>>
>>> I notice that automake tends to just use `` instead:
>>>
>> What do you intend with `` exactly?  Are you referring to command
>> substitutions?  If yes, how are they related to command line limits
>> issues.
> 
> What i was talking about was putting the command line arguments in a
> file, vs putting the command line arguments in a variable.
>
> automake tends to use variables.
>
The question here is more subtle than it seems. I lack the skills to
clearly explain it in a short time, but maybe the comments below can
help to shed some light on the issue.  If not, I'll try to give a more
complete explanation.  Just let me know.

> e.g. as a file (a toy example, i know it's wrong)
> 
> some.jar: $(SRCS)
>       rm -f srcs
>       for n in $(SRCS) ; do \
>
But if the content of $(SRCS) is too long, the command line of the shell
invoked by make would already been exceeded here, before any command
line length for a java process invoked by the shell can be exceeded.  In
fact, the shell wouldn't even get to run, because the 'exec' system call
invoked by make to launch the '/bin/sh' process would fail with an
"Argument list too long" error.

>               echo $n | sed 's|^|$(srcdir)|' >> srcs ; \
>       done
>       javac @srcs
>       rm -f srcs
> 
> (the "@srcs" reads the arguments from the file)
> 
> vs
> (more or less how java.am does it)
> 
> some.jar: $(SRCS)
>       list1='$(SRCS); list2="" ; for n in $$list1 ; do \
>               entry=`echo $n | sed 's|^|$(srcdir)|'` ; \
>               list2="$$list2 $$n" ; \
>       done ; \
>       javac $$list2
> 
> And as to how they're related to command line limits?  `everything
> inside is a command line is it not`?
>
Not necessarily.  That is a command substitution, so, AFAIK, the shell
will simply fork to execute it, without any need to call 'exec'.  So no
command line length can be exceeded.  Example on my Debian machine:

  $ long_line=$(python -c 'print ": "*10000000') # Create long dummy cmdline
  $ echo ${#long_line} # Verify it's very long indeed
  20000000
  $ bash -c "$long_line" # Too long to be run by the shell.
  bash: /bin/bash: Argument list too long
  $ var=`$long_line` # But not too long to be used in a command substitution
  $ echo ${var+set}
  set

> Every rule action is a command line?
>
Basically, yes.  For example, if we have:

  foo:
    echo 1
    echo 2.1; \
    echo 2.2
    echo 3

then to build 'foo' make will run three command lines:

  echo 1

  echo 2.1; echo 2.2

  echo 3

>>> aren't command line
>>> limits a portability issue anymore, or is it just not likely a problem
>>> with c source?
>>>
>> Mostly, it's *extremely* difficult to exceed command line length limits
>> with a list of sources, unless you are going to build a program or
>> library (or Jar file, in our case) that has thousands of sources ...
> 
> Ok i'll just ignore it at this point.
>
Good choice IMO.

Regards,
  Stefano