Re: [Bug-wget] Google Summer of Code 2016

[Kushagra Singh]

Hi,

Sorry for the late reply, I have exams going on at my institute, wasn't
able to have a look because of that.

I looked at the feature request on savannah [1], it extends the regex
feature. However this feature (--accept-regex) is not available in Wget2,
so I guess this is for Wget?

Right now, the url is checked against the accept regex in download_child()
(src/recur.c line 667). According to what I understand by reading the
discussion thread on savannah, we will call an external program, which is
expected to already exist and let it decided whether the uri gets accepted
or not. We will pass the uri and other variables by setting environment
variables using setnv() and unsetenv(). Has a similar feature (which calls
an external program) been implemented before? It would be great if it has,
I can have a look and get started. And how exactly will we be calling the
external program? Using system calls?

As for GSoC, I would prefer to work on a project on Wget, since I have a
fair idea how things work there. Is there anything on the wish list for
wget? If not, implementing FTP(S) in Wget2 seems like a fun project, would
love to work on that.


[1] https://savannah.gnu.org/bugs/?45803

Thanks,
Kushagra

On Thu, Mar 3, 2016 at 3:51 PM, Tim Ruehsen <address@hidden> wrote:

> Just more ideas for you, Kushagra:
>
> There are many command line options from Wget still missing in Wget2, you
> should have a look at
> https://github.com/rockdaboot/wget2/wiki anyways - feel free to work on
> the
> wiki yourself (e.g. fork the wiki pages:
> https://help.github.com/articles/adding-and-editing-wiki-pages-locally/
> or let
> me know and I'll give you write access).
>
> You can search the Wget bug tracker
> (https://savannah.gnu.org/bugs/?group=wget) for wishlist items.
> My favorite is https://savannah.gnu.org/bugs/?45803.
> Special popen(2|3) functions/code is already in libwget/ directory.
> E.g., that would allow Wget2 to be used as part of a recursive website
> malware
> checker.
>
> The authorization code in the test suite is not complete/not implemented -
> I
> once tested authorization (MD5, MD5-sess) 'by hand' with my local Apache.
> But
> a automated test is badly needed.
>
> We thought of a statistic module (very basic code exists) for spider mode
> to
> output diagnostics very detailed. Missing pages, response times, server
> load
> (e.g. using the RTT/ping time), etc.
>
> Tim
>
> On Wednesday 02 March 2016 10:51:02 Kushagra Singh wrote:
> > Hi,
> >
> > Thanks for the quick reply. I went through the repository and the issues,
> > and found a couple of things I would like to work on.
> >
> > I have a couple of questions about Wget2. Is it a complete rewrite of the
> > Wget project, available at git://git.savannah.gnu.org/wget.git, or are
> we
> > using existing code and extending functionality? I guess it is the second
> > one because I saw `libwget` in the repo. However if such is the case,
> then
> > how do we change existing functions in wget? For example, implementing
> [2]
> > would require making changes to the file cookies.c, which is present in
> > /src in the wget repo, but not in /src in the wget2 repo.
> >
> > I was looking at #43 [1], and have already submitted a patch for
> > consideration for the first suggestion [2]. The second suggestion
> mentioned
> > [3] is one of the things I'd like to work on, however this is not
> something
> > which will take three months :)
> >
> > Another project I am interested in, is implementing FTPS. I saw this
> listed
> > under one of the ideas of GSoC 2015, but I'm not sure whether it was
> > implemented, as I didn't see it under 'Development Status' in the wget2
> > readme on Github.
> >
> > Also, in #67 [4], we are talking about adhering to some specific parts of
> > RFC 7230. I'm not sure which all parts would be right, as the discussion
> > thread mentions that it won't be good to stick to each point of the RFC.
> > WDYT?
> >
> >
> > [1] https://github.com/rockdaboot/wget2/issues/43
> > [2] https://tools.ietf.org/html/draft-west-leave-secure-cookies-alone-04
> > [3] https://tools.ietf.org/html/draft-west-cookie-prefixes-05
> > [4] https://github.com/rockdaboot/wget2/issues/67
> >
> > On Tue, Mar 1, 2016 at 9:57 PM, Giuseppe Scrivano <address@hidden>
> wrote:
> > > Kushagra Singh <address@hidden> writes:
> > > > Hi,
> > > >
> > > > Will we be taking part in GSoC this year? I would really like to
> work on
> > >
> > > a
> > >
> > > > project related to Wget this summer. Any specific ideas that are of
> > > > importance to the community presently?
> > >
> > > yes, we will be take part in GSoC.  I think we would like to see more
> > > work happening on wget2, at the moment there is a list of issues on
> > >
> > > github that can be useful to you to pick some ideas to work on:
> > >   https://github.com/rockdaboot/wget2/issues
> > >
> > > Could you take a look at it?  Do you see anything interesting that you
> > > would like to work on?
> > >
> > > Regards,
> > > Giuseppe
>