bug#15578: Parameter -d or --direct to open files with flag O_DIRECT?

[Kyle Sallee]

On Thu, Oct 10, 2013 at 4:21 PM, Pádraig Brady <address@hidden> wrote:

>
> One possibility worth mentioning, would be to add a files0-from=F option
> to dd,
> like du,sort,wc already have.
>

Wonderful compromise.  I agree.
I appreciate how it already continues an established and understood
convention
in parameters for coreutils utilities rather than creating something new.

Oddly enough I might have never used the -files0-from= parameter for du
sort wc
However, that might sometimes save on an invocation of cat within a pipe.
But I am not often working with null terminated lists of files.
Merely creating that list might sometimes require an invocation of "tr"
to translate the linefeed to null.
However, I am aware that "find" can generate lists of null terminated
output.



> Now those have it because they need to operate on the complete input set,
> for accumulation or sorting, and thus can't resort to separated runs
> with xargs or whatever.


That makes sense.



> dd might use it as it has a very different command
> syntax to the standard tools. So that would allow a general method
> to efficiently read many files.
>

Since dd might be the only utility from coreutils that allows
specification of additional open flags
then dd would have been my second choice for modification after cat.



> Another related thing to consider is the above would allow a single
> process to handle everything, but it might be better to split the
> load into a process per CPU.
>

The type of processing that dd does to transliterate character sets
and do conversion can be processor intensive
when the amount of data being processed is immense.
Therefore, the benefit of concurrent threaded processing of blocks
while outputting the processed blocks linearly
could allow dd to expedite processing in proportion
to the amount of processors allowed.
However, I anticipate problems could occur when processing around block
edges.
Yet individual processing of files concurrently avoids that potential
problem.

Perhaps concurrent processing of files with dd
would most often be accomplished with xargs?
That is because the output might not be desired as a single data block.
Therefore, if dd allowed specification of multiple input files
then it should probably also allow the specification
of a list of an equal amount of output files?

The cost for invoking dd through xargs
to gain concurrent processing of individual files
is low in comparison for the boon gained
from using two or more processors concurrently.
Therefore, the added complexity might not be worth
the additional speed when nearly the same benefit
can be gained by invoking dd by xargs -P

Aside from the rejected modification to cat,
and the proposed modification to dd,
and writing a small C helper program;
I wonder what possibilities exist
that have not yet been contemplated?

Perhaps a BASH plugin can be created
that reads a list of file names on stdin
and a list of open flags as parameters
and then opens those files and writes their output to stdout?
Chet might accept a patch for a such a plugin.
Because BASH does not install plugins by default
and lacks an inter-POSIX suggested installation location;
the availability and use of the plugins go unnoticed
by almost everyone who lacks specialization in shell scripting.
Or more succinctly stated few people might possess
an opinion concerning BASH plugins.
And if the plugin is rejected
then I can still regress to the idea of writing
a standalone C program for the task.

Although my original suggestion might have seemed otherwise,
I am not in favor of rapid mutation of coreutils
as opposed to carefully considered evolution.
I appreciate exploring different possibilities for implementing
functionality
and carefully weighing the boons and the banes to arrive at a conclusion
that becomes best for the long term goals of coreutils.

I am 100% satisfied with coreutils.
I wanted an eloquent solution for a spliter of an idea.
I arrogantly and hastily expected that coreutils
should immediately accommodate.
Please forgive my lack of objectivity, insistence, and persistence.

Please pardon the delayed response.
I wanted to avoid a rapid rate of communication
becoming a burden or a cause for irritation.
And I hoped that a day's worth of other activities
would grant opportunity for a fresh perspective
upon resuming consideration.



> thanks,
> Pádraig.
>

Thanks for the continued attention.
I am satisfied with the outcome.
>From the conversation I gained a better understanding
of how to accomplish the task.
The outcome is appreciated.

I previously expected that "cat" --direct would suffice.
Yet now I also want to mitigate the necessity of invoking "cat" using
"xargs."
I should have realized that before sending the first email.

coreutils should be expected to do what coreutils does,
but not expected to be the most expeditious implementation.
I wanted unreasonable performance.

Please forgive that I created a bother
and selfishly utilized other Pádraig Brady's time
instead of thinking for myself.
That aside I should not omit saying;
thanks for maintaining and sharing coreutils.
Shell scripting would be insufferable without coreutils.  :)