I'm using 20140122 because I noticed the addition of the --line-buffer option. Thanks for that by the way. Then i saw --compress and I know this will be very useful, but I'm running into issues using it.
I tend to use parallel to process streaming data and I'm doing something that looks like this:
yes whatisthisnow | head -n100000000 | parallel --pipe -N100000 -L1 -j512 cat 2> cattest.log > cattest.out
if i add --compress:
* my log file has a bunch of lines like: "lzop: <stdin>: not a lzop file"
* i do get streaming output (but is it per job like the default?)
* not all of the lzop processes exit and parallel hangs indefinitely
most of these issues seem to be intermittent but it seems like the bigger the input or higher my -N number (or both) the more frequently things go wrong.
if i add --line-buffer to that I don't get any output at all ever.
i do hope there is an easy way to resolve these issues and enable me to use lzop for temp files while still getting lines as soon as they are available.
thanks,
-Derek