Re: [Bug-wget] wget2.0 / niwt / refactoring

[Tim Ruehsen]

Am Monday 13 August 2012 schrieb Micah Cowan:
> On 08/13/2012 03:06 AM, Daniel Stenberg wrote:
> > On Mon, 13 Aug 2012, Tim Ruehsen wrote:
> >> But we should not forget about a monolithic, backward-compatibel (to
> >> wget 1.x) wget 2.0. We all agree, it is time to redesign wget's code
> >> architecture to have a clean codebase for new features to implement,
> >> to increase readability/hackability, to increase modularity and to
> >> increase code reusability (wget-derived libraries). Not to forget
> >> current C language / library conventions and features like C99 and
> >> Posix compatibility. On older systems wget 1.x would stay the
> >> preferred version.
> > 
> > You should know I talk of my own project here, but I would like to
> > remind people that when considering an overhaul of a (portable) HTTP/FTP
> > project, there might be reasons to think about letting the transfers be
> > handled by libcurl. It already supports more protocols, more SSL
> > flavours and more features than the wget transfer engine and it exists
> > as a completely free and open, proven and well documented library with a
> > stable API today. Most of mget's "Not Yet Implemented" features are
> > already supported by libcurl.
> > 
> > This said, I realize it is not kosher for a GNU project since libcurl is
> > not GNU and I believe it will be a primary factor that will prevent this
> > from happening. I still believe it would be smart from a technical
> > perspecitive.
> 
> I don't know about it not being kosher. There's certainly no rule
> against using Free Software that comes from outside GNU.
> 
> That said, when a GNU alternative is available, RMS often prefers it be
> used instead. The promoted (but not required) policy to consider Guile
> first for extension/configuration language is an example. And he had
> previously said that once GNUTLS support in wget were complete, that
> Wget should consider dropping OpenSSL support. There may well be more
> than simply "prefer GNU" in that one, considering that OpenSSL requires
> a license exemption for the GPL (which slowed us down considerably when
> we moved to GPL 3.0, as they wanted to take time to use the right
> language for the exemption with the new license, but didn't really get
> around to it for a while - volunteer legal organizations tend to be
> overworked I think :) ).
> 
> > Also, this is not a new idea and it has been turned down before:
> >   http://daniel.haxx.se/blog/2007/10/27/wget-going-libcurl/ and
> >   http://www.mail-archive.com/address@hidden/msg01129.html
> 
> Well, "turned down" may not be accurate. "Didn't get to it" may be more
> likely. Certainly, it was considered to be a "Wget 2.x" feature as
> opposed to "Wget 1.x", considering the major overhaul it represents, but
> I agree that it'd be a smart way to instantly expand wget's protocol
> support, expand the testing base for a major piece of wget's
> functionality (to the union of curl's and wget's current user bases),
> and push off maintenance of much of wget's to be someone else's
> maintenance problem ;)
> 
> > So this is basically a set of patches on top of the stock Wget? If it
> > is, have those patches been posted here? If it isn't, how exactly does
> > it relate to wget other than being a similar tool?
> 
> It shares no code with current Wget, AFAICT. What I think he's
> proposing, is to turn his new Mget into the foundation for a new Wget
> 2.0, pulling features one-by-one from current Wget 1.x, but redesigning
> them to be cleaner, until you have a Wget 2.x, functionally equivalent
> to Wget 1.x, but completely rewritten.

90% correct. I already rewrote the basic parts for Mget, so a big bunch of 
work is done.
And than, we don't have to rewrite everything else BUT we should review 
everything.
Let's take the WARC support as an example: why should we rewrite it ? But when 
integrating that code into Wget2.0, we are forced to review, maybe clean it up 
(or maybe not if it is OK), read the specs, note what's missing and what's 
done.
It is a bit like cleaning up the roof before you move - you have to grab 
everything and decide if it is worth keeping or not. Well, maybe a not so good 
comparison, but i hope you get what I mean.


> Personally, I'm inclined to favor an approach that builds Wget 2.x from
> a Wget 1.x base, and in particular its bevy of test cases (if they're up
> to date? I don't know if they are), to rewrite things bit by bit to be
> cleaner, taking advantage of the tests to ensure feature stability. I
> have yet to see a true, complete rewrite-from-scratch of a codebase that
> didn't result in loss of features, so I suspect a rewritten Wget 2.x
> would either fall short of its backward-compatibility goal, or it would
> never reach completion.
> 
> A piece-wise improved Wget 1.x has the advantage of never being
> functionally incomplete, while at the same time being able to take
> immediate advantage of any design improvements. IMO, a 30% cleaned-up
> Wget is more useful than an 80% completed Wget-rewrite.

Your are right, but there is a compromise.
Taking Wget1.x code and carefully adjust it to fit into the redesigned Wget2.x 
has the chance to get most of the Wget1.x features plus a code review (and 
maybe redesign). The old code would be 'recycled' (much of Wget1.x code is 
definitely worth keeping) and transformed into modern C99 / Posix style code.

I do not want to reinvent the wheel and I am not willing to throw away well 
tested code.

Without someone shouting HERE, there will be no Wget2.x within the next 2 
years... For the moment I can't see any (free and willing) developer resources 
on this list. Does someone prove me wrong ?

Tim