Re: Failure of expensive test "datamash-valgrind.sh" for git HEAD

[Shawn Wagner]

It's not /crashing/ in the test, it's exiting with an error. I suspect the idea of this particular test is to look for buffer overflows with really long input strings, and it should be using valgrind's --error-exitcode option and checking for that instead of any failure status.

(Fixed all the small memory leaks I found with -fsanitize=address in the normal test scripts, btw)

On Tue, May 31, 2022 at 2:23 PM Tim Rice <trice@posteo.net> wrote:

Hey Erik,

>the "expensive" test "datamash-valgrind.sh" fails on my Ubuntu 18.04
>GNU/Linux x86-64 laptop with current git HEAD:
>
>```
>$ make check-expensive TESTS=tests/datamash-valgrind.sh
>[...]
>custom-format failed
>FAIL: tests/datamash-valgrind.sh
>[...]
>```

Yeah, I was looking at this too. I want to make sure all expensive tests pass before opening 1.8 up for testing. This is not the only expensive test that fails for me, btw. The one for doing i/o on a full filesystem is also a bit wonky.

I thought the --format test problem could be because of something weird on my local machine. The test was added at the same time as the --format flag. I figured, surely it must have been working when it was added?

And if datamash is intended to fail for this test, then what is the point of including inputs that don't cause it to crash? It would be clearer to go straight to the input which is too big.

Because it doesn't do that, I think datamash is intended to succeed. The conclusion is that the test inputs need to be pruned down.

~ Tim