Re: [Bug-wget] Anyone want to add libcares support to wget?

[Ben Greear]

On 03/17/2016 01:58 AM, Tim Ruehsen wrote:
> Tim
>
> On Wednesday 16 March 2016 07:35:44 Ben Greear wrote:
> > On 03/15/2016 06:40 PM, Ángel González wrote:
> > > On 15/03/16 17:03, Ben Greear wrote:
> > > > I want to use wget on multi-homed machines, and I use routing rules to
> > > > route pkts out specific interfaces based on the source IP.
> > > >
> > > > So, I need to be able to bind to a local IP address, including for the
> > > > resolver logic, which is why I want to use libcares.
> > > >
> > > > To test, I think it would be sufficient to put two IPs on a port
> > > > (1.1.1.1, 1.1.1.2), and then use cmd-line args to choose one or the
> > > > other.  Sniff and watch for proper source address.
> > > >
> > > > For DNS servers, have 1.1.1.3 and 1.1.1.4 be specified.  Sniff and watch
> > > > to see that the resolver destination IP is as specified.  This should be
> > > > completely independent of whatever is in /etc/resolv.conf, etc.
> > > >
> > > > I (and others) have patched libcares and curl in the past to handle all
> > > > of this, so I know libcares can do it.  I think the main work is to
> > > > handle new cmd-line arguments, pass appropriate info to libcares, and
> > > > make wget use libcares for it's resolve logic.
> > > >
> > > > Thanks,
> > > > Ben
> > >
> > > I think it's ok to add a --bind parameter to wget to choose the interface
> > > for download, but I don't think it should use a different resolver just
> > > for this.
> > >
> > > You _could_ change the libc nameservers from the application by changing
> > > _res.nsaddr_list after res_init(). It may be a bit fragile, but should
> > > work fine. And even better than changing each application, I would
> > > recommend you to create a simple LD_PRELOAD library that overrides
> > > res_init() & friends, and allows you to specify the target dns server
> > > with an environment variable. It's a pity there's no RESOLV_CONF var
> > > similar to RESOLV_HOST_CONF, for using a separate /etc/resolv.conf.
> > wget can already bind to a local IP.  It might be nice to add support for
> > SO_BINDTODEVICE, but not sure it is required for what I need.
>
> @Ángel The LD_PRELOAD idea is good. I really like it !
>
> > The rest of your suggestions are total hacks!
> >
> > Anyway, I'm quite sure of what I want, and it should not be horribly
> > difficult to accomplish.  I'll work on it if no one else is interested.
>
> Thanks for your offer.
> I expect a configure option to switch such a feature on (default=off) to avoid
> a new dependency (c-ares) for a feature that just one out of a million users
> need. How would you design an automated test ?

I will test it on my own systems...I'm not sure about automated testing, maybe
it will be more clear after I have the changes done.

> Did you consider using a container (e.g. docker) for such a task ? Easy to set
> up and you'll have your feature not only for wget. IMO, that is much more
> flexible. (It was Giuseppe's idea during a private talk).

Containers will not work for me.  I need to scale to thousands of
instances on modest hardware.  I'm certain the libcares and binding
approach will work because we do similar things with curl and other
programs already.

Thanks,
Ben


--
Ben Greear <address@hidden>
Candela Technologies Inc  http://www.candelatech.com