Re: LYNX-DEV Since Lynx won't, what will?

[Jim Dennis]

> Jim Dennis describes his script for doing a mirror type function with
> Lynx and -crawl, etc., in response to my message last week wanting a
> program to do something similar.

        Please feel free to let me know if  you use my script and find
        any bugs in it.

> Jim, I agree that it just doesn't make sense to add this kind of
> functionality to Lynx when there are already tools out there that do
> such things, and do them quite well. (After all, we don't want Lynx
> to have everything, including the kitchen sink, like emacs has done!)

        Umm!  That's not what  I  said.  I'm  the one who   originally
        posted the request TO ADD the -mirror function.

        My script was  a feasibility study and  prototype  to show how
        this might be done.  I'm hoping to tickle some coder's fingers
        into integrating my  design into C  code.  Maybe I'll bite the
        bullet and work on it myself (though my C coding skills aren't
        yet up to the task -- so it will take  me ten times as long as
        it would for a "real" programmer).

        On the one hand I agree about "freaping creaturism" and on the
        other I balance  the benefit  against  the cost.  One the  one
        hand the -crawl and -traversal switches are "features" without
        specific benefits attributed   to them.  (I  use -traversal to
        detect    errors,  and obviously  I  use    it as  part of  my
        "lynxmirror" script; I have used -crawl  to create or update a
        set of "infobot"  pages   for procmail to avoid   ugly  little
        "dynamic  fetch" recipes   --  but  I   don't know   what  the
        authors/contributors of these functions had in mind).

        My conclusion (from this feasibility test)  is that the amount
        of extra   code would be  minimal  and the added functionality
        would   be considerable.  In  part   I'm thinking in terms  of
        "marketing" Lynx to a core set of webmasters and sysadmins.

        My hope is that a number of  webmasters and sysadmins will use
        Lynx   as  their prefered   "web  automation"  tool  and  will
        consequently put furhter pressure  on their designers  to have
        Lynx friendly sites.  The benefit I  receive from this is that
        I   can work with  my preferred  tools for browsing  as a side
        effect. (I like  keyboards    and text mode  and    don't link
        electronic rodents and "squinting").
        
> Go to one of the GNU sites and get a copy of wget. It is an excellent
> program, has lots of options to tailor your usage to exactly what
> you want to do, and works very well for mirroring a site or a portion
> thereof. You CAN tell it to fetch just particular types of files, or
> not to fetch certain types. You can tell it to traverse to a limited
> depth or recurse all the way. You can tell it not to follow links to
> parent directories, to ignore certain directory or file name patterns,
> and much more.

> Scott

        I'm sure that wget is an excellent tool.  I'll probably *have*
        to get a copy  of  it for  other reasons.  However  it doesn't
        serve some  of my other purposes  and I  still think a -mirror
        switch is a "good thing"  to add to  Lynx.   If no one  agrees
        with me then either I'll find the time to climb the C hill and
        add  it  myself (and maybe  nobody will  include that patch in
        there code) or it will never get done.

        If I do  implement a -mirror  set of patches  I'll try for the
        following behavior: -mirror [path] [URL] will work essentially
        like  -traversal and my  script (except that  it wouldn't save
        the .dat files unless  you also include the -traversal switch.
        If you left off the URL I'd expect a set of URL's from <stdin>
        which I would assume was  a previously created (and presumably
        filtered or edited)  traverse.dat (the mirror  would then only
        fetch  those times listed and not  do a further traversal.  If
        you   didn't include a path  it  would default  to the corrent
        directory.  If it  found filename  collisions it would  rename
        the old files with emacs style "versioning"

        Finer  granularity of control  would not be directly available
        with my  proposed  set  of  additions.  However it   could  be
        written  so that  the API is  available  -- so that one  could
        create a UI for it (beyond the implicit UI of the command line
        switches).  

        One of the other enhancement requests I've seen at browser.org
        is the one  calling  for   some  sort  of embedded   scripting
        language.  I followed that up with  the recommendation that we
        TCL for this task (since  TCL is  specifically designed to  be
        embedded).  I've also recommended that  we look at ctk (curses
        tk) since   it's  designed  to  work with  TCL   and  we could
        seriously benefit  from the layout manager   for use in tables
        (among other things).

        An embedded TCL  would be ideal  for controlling the mirroring
        features.  In  addition would be neat  to see Safe-TCL quietly
        succeed in  delivering  on  some the  promises  that  Java has
        hyped.

        (That's what I like about GNU  software -- as close to working
        anarchy as I've ever seen).

--
Jim Dennis  (800) 938-4078              address@hidden
Proprietor, Starshine Technical Services:  http://www.starshine.org
        PGP  1024/2ABF03B1 Jim Dennis <address@hidden>
        Key fingerprint =  2524E3FEF0922A84  A27BDEDB38EBB95A 
;
; To UNSUBSCRIBE:  Send a mail message to address@hidden
;                  with "unsubscribe lynx-dev" (without the
;                  quotation marks) on a line by itself.
;