Re: [Quilt-dev] test suite improvements

[John Vandenberg]

Andreas,

The series of patches you were looking through were very old, created
to get the MSYS port operational.  I have been redoing them all again,
from scratch.

The patches that I proposed last week for 0.43 are here:

http://zeroj.hda0.net/quilt-0.43-patches/

The first part of that series is mostly redundant since your commits,
and the rest is stale.  I have refreshed all of the useful components,
except for the 'reference.diff' patch.  It takes a different approach
to the building of quilt.1 and README, improving the build times
considerably.  Is it worth me refreshing that patch for 0.43?

Here is an updated series against trunk:

http://zeroj.hda0.net/quilt-0.43-patches-v2/

On 1/14/06, Andreas Gruenbacher <address@hidden> wrote:
> John,
>
> I quickly checked some of your patches. Here are some comments.
>
> On Friday 13 January 2006 12:29, John Vandenberg wrote:
> > Hi again,
> >
> > In order to complete the portability series that I have been working
> > on, there are a few improvements that I would like to propose for the
> > test harness/suite.

> > 2.1 regex support for output lines that commence with >~
> > 2.2 failed commands to use the separators !~, =~, != and == instead of |
> > and ?
>
> http://zeroj.hda0.net/quilt/patches/test-regex.diff looks good. Please make
> sure that test/run will parse both '>~foo' and '>~ foo' at the beginning of
> lines. Putting a test case in test/*.test that fails on purpose is not a good
> idea.

I forgot to mention, that patch also changes the output of failed cases from

<expected>  <sep>  <actual>

to

<actual>  <comparitor>  <expected>

> I have changed things so that the test suite runs against the source tree
> instead of the installed version, so
> http://zeroj.hda0.net/quilt/patches/test-this-quilt.diff should be obsolete.

That patch has been obsolete for quite some time.  Hopefully my
patches have improved somewhat since then.

There is minor issue with your changes: symlinks are not created in
compat/ , causing make check to fail when configure
--with-<name>=<prog> is used.  the attached patch resolves that; is it
ok for me to commit it?  (btw, thanks Jean for the bash tip)

> > 3. local-quilt-x.diff introduces a 'make check'.  This becomes very
> > important as more of the ./configure information becomes necessary to
> > accurately test the quilt built in the local workpit.  Is it ok to
> > remove test/Makefile?  The only alternative that I can see is to
> > replace it with test/Makefile.in.
>
> I didn't find this patch.

Sorry for the confusion caused by having so many sets of patches lying
around. I will re-organise how I host/submit these a bit better.

> > 4. regularly running all of the tests on msys is really slow.  I would
> > like to record which tests have run to completion, and pick up the
> > testing on the last test case to fail.  I wrote a rough patch to do
> > this last year; any suggestions on how this should be done in order to
> > be committed?
> >
> > http://zeroj.hda0.net/quilt/patches/test-make-depend.diff
>
> Can you remove *.ok at the end of a full, successful test suite run? Then the
> test suite can be re-run without a clean in between, and ``make status'' also
> shouldn't be needed. Could you please try to set TESTS in test/Makefile to
> the tests that should be tried (using filter-out) instead of what you are
> doing right now? This allows ``make TESTS=...''. Something like
> ``$(filter-out $(patsubst %.ok,%.test,$(wildcard *.ok)),$(TESTS))'' should
> do.

Thanks for the suggestions; I'll look into this.

> > 5. Also, when one command in a test case fails, the rest are sure to
> > follow.  I would like to change the test harness so that it only
> > continues past the first error if an option (--ignore-error) is
> > enabled.  I think this behaviour should be disabled by default.
>
> Agreed. I'm fine with changing the default behavior.
>
> Other stuff:
>
> http://zeroj.hda0.net/quilt/patches/test-ignore-on-failure.diff
> Could you please use -k as in ``make -k'' instead of -q in test/run?
> I don't see what the ``$?'' syntax in *.test is good for.

Ignore this.  The intent was to allow particular lines in the test
case to fail without the entire test being considered a failure.  I
havent needed that recently.

> http://zeroj.hda0.net/quilt/patches/patch-2.5.diff has changes like:
> -       > 1 out of 1 hunk FAILED -- rejects in file Oberon.txt
> +       >~ 1 out of 1 hunk FAILED -- (saving )?rejects (in file Oberon.txt|to
> \/tmp\/.*)
>
> Instead, please fix cleanup_patch_output() as shown in the attached
> fix-cleanup_patch_output.diff.

thanks for the pointer.

> http://zeroj.hda0.net/quilt/patches/test-msys.diff:
> Please use $(filter-out some.test,$TESTS) instead of patsubst. I don't think
> it's great to hardcode "MINGW32" in anything but configure.in, but in this
> case it might be better than the alternatives.

I'll write two new autoconf macros to check whether the platform
supports POSIX file permissions, and if patch -e works.

> http://zeroj.hda0.net/quilt/patches/msys-prefix-colon.diff
> I hope we'll find better fixes for those things.

This patch was what led me to start the 'run quilt locally' patches. 
This shouldnt be necessary anymore.

> http://zeroj.hda0.net/quilt/patches/bash-2.04.diff
> Which tests need bash extensions and don't run with a /bin/sh?

I'll redo the testing on Solaris and fix any tests so they are
portable to /bin/sh

> http://zeroj.hda0.net/quilt/patches/quilt-exec.diff
> What's that?

Please ignore.

Thanks for taking the time to work through these.

--
John

local-compat.diff
Description: Text document