Re: lynx-dev [PATCH] remove extension to EXTERNAL command, extend CERN rules support for mailto: URLs

[Klaus Weide]

Executive summary:
I. The approach of using lynxexec/lynxprog is not appropriate for
   general use (unless lynxexec/lynxprog were heavily revised).
II. Some concrete problems with the implementation are discussed.
III. Using lynxcgi instead of lynxexec/lynxprog should be examined.
IV. I suggest to go back to the previous, EXTERNAL-based approach
   and possibly extend that.


On Sun, 25 Jun 2000, Vlad Harchev wrote:

>  In the recent messages on the thread "[PATCH]", I stated that I changed my
> mind (thanks to Mike Castle) on the usability of extension to the EXTERNAL
> command. As everybody remembers, that extension allowed users to set some
> actions defined by EXTERNAL setting as "default" handlers for URLs with that
> pattern - i.e. the action would be invoked on the matching URL even if user
> activated the link using right-arrow or enter key, not just '.'. The original
> patch I posted had some flaws - it didn't support 'g', 'G', 'E' and
> 'x' commands.

Or maybe those were advantages.  They allowed a way to acces the builtin
normal ACTIVATE-action, in spite of EXTERNAL handlers.

> It turned out that CERN rules already provided functionality of
> redirecting URLs to lynxexec: URL that allows invoking any programs (well, the
> set of programs can be limited). Such support for redirecting URLs via CERN
> rules is global and it covers all lynx commands. The only disavantage in
> using CERN rules was inability to redirect mailto: URLs. 
> 
>  I fixed that problem with redirecting mailto: URLs and it seems that now it's
> better to remove support for extended functionality of EXTERNAL setting, since
> it duplicates functionality of CERN rules and it not finished (doesn't 
> support 
> 'g', 'G', 'E' and 'x' commands).

I disagree, see below.

>  Here is a piece of lynx.cfg that tells lynx to handle all ftp:// links
> using 'links' browser:
> ------------------
> #this is needed to allow exec'ing programs
> LOCAL_EXECUTION_LINKS_ON_BUT_NOT_REMOTE:TRUE
> 
> RULE:PermitRedirection ftp:*
> #it's necessary to allow redirection
> 
> RULE:Redirect     ftp:*     "lynxprog:links ftp:*"
> #the main rule
> ------------------

You didn't give any example how this would work for "mailto:";, which seems
to be the main focus here.  So let me make one up, by analogy, and then
comment on it:

| ------------------
| #this is needed to allow exec'ing programs

You need to have a lynx compiled with EXEC_LINKS in the first place.
And, quoting from userdefs.h:
 * I strongly recommend that you define neither one
 * of these since execution links/scripts can represent
 * very serious security risk to your system and its
 * users.  If you do define these I suggest that
 * you only allow users to execute files/scripts
 * that reside on your local machine.
 *
 * YOU HAVE BEEN WARNED!

| LOCAL_EXECUTION_LINKS_ON_BUT_NOT_REMOTE:TRUE

No, this won't be enough in general.  You will need
LOCAL_EXECUTION_LINKS_ALWAYS_ON:TRUE if you want to allow
handling "mailto:"; from anywhere (not just in local files), or you
will have to mess with TRUSTED_EXEC or ALWAYS_TRUSTED_EXEC rules.

*This will make lynx wide-open for executing nearly everything!*

| RULE:PermitRedirection mailto:*
| #it's necessary to allow redirection

No it isn't, the way that your change is implemented.  You don't check
'HTPermitRedir' at all in getfile()!.

| RULE:Redirect     mailto:*     "lynxprog:pine -blahblah mailto:*";
| #the main rule

I don't know what the exact command would be, so I put  pine -blahblah
here.  Maybe it's  pico -blahblah  instead.

Anyway, think what happens if the mailto-UTL contains some special shell
characters.  For example, "mailto:address@hidden;subject=Re:%20something";.
Or is that '&' instead of '&'? - anyway, you have to be prepared for both,
or anything else whether allowed by mailto URL RFCs or not.

So you'll at least want to put some "'" quotes around the string matched by
the *, in order to protect from shell characters.  But then (1) this
can be subverted if the mailto-URL itself contained "'" characters, and (2)
exec_ok() will reject the resulting lynxprog - except when the most wide-
open setting, LOCAL_EXECUTION_LINKS_ALWAYS_ON:TRUE, is used.

| ------------------

In summary, this is highly dangerous and cannot be recommended for general
use.  It may be acceptable for you and me to play around with it, after we
have carefully read all the warnings in lynx.cfg and cernrules and elsewhere
about execution links.  It's definitely not something to recommend to folks
for whom '.' vs. Enter is a big issue!

Execution links (lynxexec and lynxprog) were just not meant to be used this
way.  They were meant to be used in a carefully controlled environment,
where someone is making damn sure that no dangerous lynxexec/lynxprog link
can be executed.  Using this for links that may come from anywhere is not
appropriate.  You'd have to carefully revise the security model for this.

II. Now, some comments on the implementation.  I think it's still a good
idea to make "mailto" redirectable or, more generally, subject to cernrules
handling; if only for advanced users who are aware of the dangers.
But it should be done differently from Vlad's patch.

>  Here is a patch that removes the effect of the patch extending functionality 
> of EXTERNAL command, and allows redirecting of mailto: URLs. Documentation is
> touched too.
> 
>  Notes: This patch is effectively 3 parts - one reverses modifications
> made by previous patch, second touches /samples/cernrules.txt, third modifies
> LYGetline.c to allow redirecting mailto: URLs using CERN rules.

> --- lynx2.8.4dev4-0/samples/cernrules.txt     Thu Oct 21 19:56:48 1999
> +++ lynx2.8.4dev4/samples/cernrules.txt       Sun Jun 25 19:28:40 2000
> @@ -536,7 +536,7 @@
>  # Rules processing does not apply to all URL schemes.  Some are
>  # handled differently from the generic access code, therefore rules
>  # for such URLs will never be "seen".  This limitation applies at
> -# least to lynxexec:, lynxprog:, mailto:, LYNXHIST:, LYNXMESSAGES:,
> +# least to lynxexec:, lynxprog:, LYNXHIST:, LYNXMESSAGES:,
>  # LYNXCFG:, and LYNXCOMPILEOPTS: URLs.  You shouldn't be tempted
>  # to try to redirect most of these schemes anyway, but this also
>  # makes it impossible to disable them with "Fail" rules.

No.  You patch doesn't do enough to warrant taking "mailto:"; out of that
list.  AFAICS, you cannot make a mailto URL "Fail" or "Pass".
You also allow redirection to anything, without regard to the
restrictions that apply in the generic case (starting with

                            if (!HTPermitRedir &&
                               (url_type == ...
).

That's the result of trying to handle redirection from mailto as a
special case in getfile(), rather than using the generic mechanism
that is already in HTLoadDocument()/HTLoad().  You are reinventing the
wheel (or rather: you copy the design from somewhere else, but forget
some important aspects, for example that it has to be round...).

> diff -ru lynx2.8.4dev4-0/src/LYGetFile.c lynx2.8.4dev4/src/LYGetFile.c
> --- lynx2.8.4dev4-0/src/LYGetFile.c   Tue Jun 27 09:41:30 2000
> +++ lynx2.8.4dev4/src/LYGetFile.c     Tue Jun 27 11:43:00 2000
> @@ -2,6 +2,9 @@
>  #include <HTTP.h>
>  #include <HTAnchor.h>            /* Anchor class */
>  #include <HTAccess.h>
> +#ifndef NO_RULES
> +#include <HTRules.h>
> +#endif
>  #include <HTParse.h>
>  #include <LYCurses.h>
>  #include <GridText.h>
> @@ -536,6 +539,19 @@
>                       HTParentAnchor *tmpanchor;
>                       CONST char *title;
>                       char *tmptitle = NULL;
> +                     
> +#ifndef NO_RULES
> +                     char* newurl = HTTranslate(doc->address);
> +                     if (!newurl) {
> +                         if (redirecting_url && strcmp(redirecting_url 
> ,doc->address)) {
> +                             StrAllocCopy(doc->address,redirecting_url);
> +                             goto Try_Redirected_URL;
> +                         };
> +                         FREE(redirecting_url);
> +                     } else {
> +                         FREE(newurl);//no need in it - it's just a copy
> +                     };
> +#endif                       
>  
>                       title = "";
>                       if ((tmpanchor = HTAnchor_parent(

Now, to improve this (in order to make mailto less of a special case
that needs special documentation etc.), you could either copy more
code from the generic copde path into the MAILTO_URL_TYPE branch of
getfile() - code for handling the different possible returns from
HTTranslate, for handling HTPermitRedir, etc.  Or you could just
let mailto be handled by the generic code, and get equal treatment
automatically.  I hope you agree that the second alternative is
preferred.

To do that, the simplest and most straightforward way is to let
(ACTIVATION of) mailto URLs *always* be handled via HTLoad(), instead
of just sometimes.  So just take the 'if (url_type == MAILTO_URL_TYPE)'
out of getfile(), don't let getfile() call reply_by_mail() directly.
Instead, write a small 'adapter' that has the same calling conventions
as other HTProtocol handlers (HTLoad*/LYLoad* functions), does the
special processing (mostly about setting title/subject) now removed
form getfile(), and calls reply_by_mail().  See code appended to this
message.

III. Lynxcgi scripts are generally "safer" than execution links (lynxexec/
lynxprog) if reasonably configured.  Talking about UNIX, there is no
system() call and therefore no shell command parsing involved (execpt
in the script itself, if it happens to be written that way).  Therefore
no issues with unsafe shell characters that are hard to avoid.  Parsing
of arguments is under control of the script.

Disadvantages:
 1. Not available on all systems/ports.
 2. CGI scripts don't have access to the terminal directly, they are
    not supposed to interact with the user directly (only via stdout/
    stdin file descriptors 1,2[,0 if POST]).

Re 1, well it should work on all UNIX(-ish) systems.

Re 2: If an interactive process is really needed - as in the case of
composing mail - this can be worked around.  (I am talking about UNIX
here - don't know how this plays out elsewhere.) The child process
running the script does not have standard file descriptors open to
the terminal, but it *can* reopen the terminal for input/output.
It is still associated with the same controlling tty as its lynx parent
process.  For example
      ----------
      #! /bin/sh
      echo "Content-type: text/plain"
      echo ""
      ARGS="$QUERY_STRING"  # will have everything after '?' in the URL
      # Add more sophisticated argument parsing here, possibly using
      # $PATH_INFO / $PATH_TRANSLATED etc. - depending on how the RULEs
      # that convert mailto -> lynxcgi:something look.
      #
      /somewhere/bin/the-real-command "$ARGS" </dev/tty >/dev/tty 2>&1
      echo "the-real-command returned exit code $?."
      exit
      ----------

(A disadvantage with this is that the-real-command does things to the
terminal that lynx doesn't know about.  At least a REFRESH (^L) will
be needed to clean up the screen.)

IV. As for the problem the original "[PATCH]" tried to solve - I still
don't know completely what it was.  If it is only that some users want
to be able to do with Enter/Return what they already could do with '.' -
well it *did* solve that "problem" (apart from the question of interaction
with other key commands, already discussed), and did it better than the
second attempt.  Better because:
- It doesn't require to open up lynx to execution of all sorts of
  execution links.
  It only requires opening up execution of one command specifically
  configured in lynx.cfg.
- It doesn't have big issues with dangerous shell characters.  (Again,
  talking about UNIX).  That is because EXTERN handling contains logic
  to escape the URL parameter if necessary.  The EXTERNAL command will 
  be passed a single-quoted string, quoted in a way that even single-
  and double-quote characters in the URL are passed along properly.
  (Of course it's still up to the invoked command to deal with the
  string safely.  If it doesn't do that, that's not lynx's fault.)
- It still may allow access to the normal lynx handling, circumventing
  the EXTERNAL command.

So I recommend to go back to the first-attempt patch and try to improve
that.  I don't personally see the need for this "Enter does wild things"
kind of behavior, but if someone wants it, I would prefer to see an
implementation that is based on you first attempt, rather than dangerous
lynxexec stuff.

  Klaus


; To UNSUBSCRIBE: Send "unsubscribe lynx-dev" to address@hidden