bug-wget
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Bug-wget] [PATCH] Keep fetched URLs in POSIX extended attributes


From: Sean Burford
Subject: Re: [Bug-wget] [PATCH] Keep fetched URLs in POSIX extended attributes
Date: Wed, 27 Jul 2016 09:39:53 +1000

Hi,

Sorry, my previous replies to various concerns were all direct rather than
to the list.  In summary, I promised to fix the style issues and I also
agreed that DEBUG on failure to set xattr was a good approach.  Also, I
confirmed that I work for Google.

Attached is a patch for the style issues and DEBUG.



On Fri, Jul 22, 2016 at 10:39 PM, Ander Juaristi <address@hidden> wrote:

> Also, shouldn't we print something when xattrs are not supported? In
> Linux fsetxattr() returns -1 when that is the case.
>
> <pedantic_comment>
> Extended attributes are not supported by all filesystems, eg:
>
> dd if=/dev/zero bs=64M count=1 of=./minix.img
> mkfs.minix ./minix.img
> mount -o loop -t minix minix.img /tmp/minix-test
>
> And then:
> wget --xattr -O /tmp/minix-test/archive.html http://archive.org
> getfattr -d /tmp/minix-test/archive.html
> <empty>
>
> Yeah, Minix is not used in the real world, but NTFS is (sometimes in the
> form of portable USB drives) and I think it also does not support
> extended attributes. Didn't try it - more hassle to set up in 5 minutes.
>
> If we trust Wikipedia [1], other filesystems from the MS world (such as
> FAT) do not support xattrs either.
> </pedantic_comment>
>
> [1] https://en.wikipedia.org/wiki/Extended_file_attributes#Linux
>
> On 21/07/16 06:33, Sean Burford wrote:
> > Hi,
> >
> > I find it useful to keep track of where files are downloaded from.  POSIX
> > extended attributes provide a lightweight portable method of keeping this
> > information across Linux, OS/X, FreeBSD and many other platforms.
> >
> > This compliments wget's existing WARC support, which serves a related but
> > different use case closer to tcpdump or tar for web pages.  Extended
> > attributes can provide a quick answer to "where did I get this file from
> > again?"
> >
> > This patch changes:
> > *   autoconf detects whether extended attributes are available and
> enables
> > the code if they are.
> > *   The new flags --xattr and --no-xattr control whether xattr is
> enabled.
> > *   The new command "xattr = (on|off)" can be used in ~/.wgetrc or
> > /etc/wgetrc
> > *   The original and redirected URLs are recorded as shown below.
> > *   This works for both single fetches and recursive mode.
> >
> > Here is an example, where http://archive.org redirects to
> > https://archive.org:
> > $ wget --xattr http://archive.org
> > ...
> > $ getfattr -d index.html
> > user.xdg.origin.url="https://archive.org/";
> > user.xdg.referrer.url="http://archive.org/";
> >
> > These attributes were chosen based on those stored by Google Chrome (
> > https://bugs.chromium.org/p/chromium/issues/detail?id=45903) and curl (
> > https://github.com/curl/curl/blob/master/src/tool_xattr.c)
> >
>
>


-- 
Sean Burford <address@hidden>

Attachment: 0001-xattr-Style-fixes-and-DEBUG-on-setxattr-failure.patch
Description: Text Data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]