Re: [Bug-wget] [Bug-Wget] Issues with Metalink support

[Darshit Shah]

I completely agree with Angel, I don't think Wget should bother about
fragmentation. It would be nice to reduce as much fragmentation as
possible, but actually going out of the way to do this doesn't seem very
right to me.
As Random Coder states, users who care about fragmentation should either
manually copy the file, run a defragmentation tool, or more importantly use
a starter File System that handles this for you. We *may* state this in the
man/info pages to ensure that everyone knows what exactly is going on.

Wget could, in theory, use fallocate() for linux, posix_fallocate() for
other posix-compliant systems and SetFileInformationByHandle (is this
available on older versions of Windows?) for Windows systems. It isn't
going out of the way by a large extent but ensures Wget plays well on each
system. However, this is going to lead to way too many code paths and ifdef
statements, and personally speaking, I'd rather we use only
posix_fallocate() everywhere and the Windows SysCalls for Windows.



On Sun, Apr 6, 2014 at 9:41 PM, Ángel González <address@hidden> wrote:

>  On 06/04/14 01:09, L Walsh wrote:
>
> Sorry for the long delay answering this but I thought
> I would mention a specific reason that such is done
> on windows (that may apply to linux in various degrees
> depending on filesystem type used and file-system activity).
>
> To answer the question, there is a reason, but
> its importance would be specific to each user's use case.
>
> It is consistent with how some files from the internet are
> downloaded, copied or extracted on windows.
>
> I.e. IE will download things to a tmp dir (usually
> under the user's home dir on windows), then
> move it into place when it is done.  This prevents partly
> transfered files from appearing in the destination.
>
> Downloading this way can, also, *allow* for allocating
> sufficient contiguous space at the destination in 1
> allocation, and then copying the file
> into place -- this allows for less fragmentation at the
> final destination.  This is more true with larger
> files and slower downloads that might stretch over several
> or more minutes.  Other activity on the disk
> is likely and if writes occur, they might happen in the
> middle of where the downloaded file _could_ have had
> contiguous space.
>
> So putting a file that is likely to be fragmented as it
> is downloaded due to other processes running, into
> a 'tmp' location, can allow for knowing the full size
> and allocating the full amount for the file so it can
> be contiguous on disk.
>
> If %TEMP% is in the same drive as the final folder, you still
> have fragmentation.
>
>
>  It can't allocate the full amount for the file at
> the destination until it has the whole thing locally, since
> if the download is interrupted, the destination would contain
> a file that looks to be the right size, but would have
> an incomplete download in it.
>
> It's possible -with some FS- with the Linux-specific fallocate() syscall,
> but that's hardly portable :)
> From Vista onwards, 
> SetFileInformationByHandle(*FILE_ALLOCATION_INFO*<http://msdn.microsoft.com/en-us/library/windows/desktop/aa364214%28v=vs.85%29.aspx>
> )
> seems able to also do that.
> I would make it fail gracefully for the EOLed versions, but seems perfectly
> fine to use.
>
>
> Anyway -- the behavior of copying it to a tmp is a useful
> feature to have -- IF you have the space.  It would be
> a "nice" (not required) feature if there was an option on
> how to do this (i.e. store file directly on download, or
> use a tmpdir and then move (or copy) the file into the
> final location.
>
> Always going direct is safest if user is tight on diskspace,
> but has the deficit of often causing more disk fragmentation.
>
> Not if you do something like calling posix_fallocate(2)
> (but it does change the file size)
>
>
> (FWIW, I don't really care one way or the other, but wanted
> to tell you why it might be useful)...
>
> Cheers!
> Linda
>
>
> If you don't want to download with the final filename, I vote for
> downloading at the same folder with another extension and
> renaming.
>
> I don't think wget should care about fragmentation, though.
>
> Looking a bit the available options, and trying to get the best from
> both sides, I think we should download with the file in place, trying
> to preallocate the blocks (fallocate, SetFileInformationByHandle)
> when possible, but not worrying too much if it can't.
>
> Cheers
>
>


-- 
Thanking You,
Darshit Shah