Re: [Bug-wget] supporting rsync?

[Micah Cowan]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Karl Berry wrote:
>     Probably not any time soon.
> 
> Is that code for "forget it, it's never going to happen"?  Or is there
> actually some hope that you (or someone -- not me, sadly) would work on
> it within, say, a year?  Just trying to get a read ...

Okay, to put it more precisely: I'm not really motivated to build
support for this into Wget directly. Eventually, at some point, it'd be
very nice for Wget to support protocol "plugins" of some sort, that
handle arbitrary protocols. At that point, I'd be happy to see (but
perhaps not personally implement) support for rsync added to Wget.

The problem is, we're still a pretty long way off from that point. There
are more serious things that need attention with Wget, and we're a very
small part-time team. And when we do "get to it", it'll require a pretty
serious code reorganization (which work, however, really needs to get
done anyway).

>     But I'm curious: why not just use rsync? 
> 
> Because it would be useful to have just one program which could be used
> to "download" stuff over whatever protocol.  What's so special about
> http[s] and ftp that they should be the only ones supported :)?  There's
> also bittorrent, for that matter ... a la aria2.sf.net.
> 
> You could say the same about ftp.  There are tons of ftp programs out
> there.  But it's still very useful to be able to give ftp to wget.

Yes, but there's a specific value wget adds over typical ftp clients
that it doesn't over rsync: ftp clients don't typically provide
fine-tuned recursion and selective fetching capabilities, and they
frequently don't handle temporary network problems as well as Wget does.

Rsync on the other hand, provides every Wget capability I can think of,
and then some.

Rsync also differs from http and ftp in the respect that those two are
more clearly "web" protocols, which is really what Wget was intended
for. When Wget is trawling through an HTTP site, and encounters an FTP
URL, it'll happily start grabbing those, too. They're both sort of "part
of the web", where it's less clear that rsync is. I'm not sure I'd wish
Wget to automatically fetch something over rsync if it encountered an
rsync URL in an HTML document—a contrived example, to be sure, but
perhaps it illustrates the difference in user expectations for those
protocols.

>     Seems to me that rsync does it better than we could hope to :)
> 
> I was imagining that the implementation in wget could translate
> into exec-ing rsync, more or less.  Certainly reimplementing all of
> rsync would be crazy.  Maybe they provide a library?

Well, naturally we'd either use a library or the rsync binary. But a lot
of work would still go into what amounts to re-fitting rsync's
conventions to our own. We'd need to translate what our -A and -R
options mean to rsync's own pattern system (which, incidentally, is
somewhat more powerful than Wget's). In order to properly support
everything that the "rsync" client offers, we'd most likely wind up
offering an option that allows you to supply parameters directly to the
wrapped client, at which point, what are we offering on top of rsync?
Why not simply invoke rsync directly, rather than go through a
cumbersome wrapper?

- --
Micah J. Cowan
Programmer, musician, typesetting enthusiast, gamer.
Maintainer of GNU Wget and GNU Teseq
http://micah.cowan.name/
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkpKnikACgkQ7M8hyUobTrH5RACfcddStA4jGbQo32biHtmL2qc7
igcAn3biBZMSXPQwm0AOczCRtaabmfsu
=z6sA
-----END PGP SIGNATURE-----