[Savannah-cvs] [Mirmon] (edit) now have dl:/root/bin/mirrors-*.sh consistently, more doc on process

[karlberry]

??changed:
-We run mirmon for mirrors of ftp.gnu.org and of dl.sv.nongnu.org.
We run mirmon for mirrors of ftp.gnu.org and of dl.sv.nongnu.org and, as a 
favor, of ctan.org.

??changed:
-Mirror request page for both gnu and nongnu is 
http://www.gnu.org/server/mirror.html.
-
-dl.sv also runs the multiplexor for CTAN, although we do not run mirmon.  
/root/mirmon/ctan, /root/bin/ctan-mirror.sh, etc.
-
-The list of mirrors for gnu mirmon is automatically created from 
``http://www.gnu.org/prep/ftp.html`` by ``/root/bin/gnu-mirmon.pl``, which runs 
nightly from ``dl:/etc/cron.d/mirmon``.
-
-``/usr/local/bin/gnu-mirmon2geoip.pl`` uses state file to build 
gnu-download.txt, which is used by ``Apache2::Geo::Mirror->auto_redirect`` to 
do the redirecting.
-
-Process for adding new nongnu mirrors at end of DownloadArea page.
ctan urls:

- ctan mirror list - http://ctan.org/mirrors
- ctan mirmon conf - maintained on ctan
- ctan mirmon web page - http://dante.ctan.org/mirmon/
- ctan multiplexor example - http://mirror.ctan.org/systems/knuth

For people wishing to run gnu or nongnu mirrors, 
http://www.gnu.org/server/mirror.html has information.  The process for adding 
new nongnu mirrors is at the end of the DownloadArea page.

All the gory details of how it works follow.

The Apache stanzas for the three redirectors are in 
dl:/etc/apache2/sites-enabled/download.  Here are the primary !ServerName's and 
the actual url's that redirect::

  download.savannah.gnu.org    # nongnu: http://dl.sv.gnu.org/releases-redirect
  ftpmirror.gnu.org            #    gnu: http://ftpmirror.gnu.org
  ctan-mirror.savannah.gnu.org #   ctan: http://mirror.ctan.org

All three redirectors have this line with the basic geography/ip data::

  PerlSetVar GeoIPDBFile "/usr/local/share/GeoIP/GeoIP.dat"

This file is updated monthly from dl:/etc/cron.d/maxmind cron.

In addition, each redirector has a data file listing its available
(http) hosts::

  PerlSetVar GeoIPMirror "/usr/local/share/GeoIP/download.txt"     # nongnu
  PerlSetVar GeoIPMirror "/usr/local/share/GeoIP/gnu-download.txt" # gnu
  PerlSetVar GeoIPMirror "/usr/local/share/GeoIP/ctan-mirror.txt"  # ctan

These files are updated hourly from dl:/etc/cron.d/mirmon cron, which runs 
dl:/root/bin/mirrors-{ctan,gnu,nongnu}.sh to do so.  Put another way, those 
three scripts output the !GeoIPMirror files above.  In addition, each reads two 
files.  First, the mirmon state::

  nongnu: /var/lib/mirmon/state
     gnu: /var/lib/mirmon/state-gnu
    ctan: /root/mirmon/ctan/mirmon.state

Second, the list of all mirrors (with country) for that redirector::

  nongnu: /srv/download/00_MIRRORS.txt
     gnu: /usr/local/share/GeoIP/gnu-mirror-list.txt
    ctan: /root/mirmon/ctan/mirror.txt

Each of these files is handled differently:

- For nongnu (/srv/download/00_MIRRORS.txt), we edit the file directly
  by hand whenever a mirror is added or removed.
  (see end of http://savannah.gnu.org/maintenance/DownloadArea.)

- For gnu (/usr/local/share/GeoIP/gnu-mirror-list.txt),
  /root/bin/gnu-mirmon.pl runs nightly from mirmon cron to update our list
  from the master at http://www.gnu.org/prep/ftp.html (which is itself
  generated from the plain text http://www.gnu.org/prep/FTP, but that's
  done by the webmasters, not savannah).

- For ctan (/root/mirmon/ctan/mirror.txt), the
  /root/bin/ctan-mirror.sh script gets the list from the dante data.

In all cases, we filter out mirrors that are too old (based on the
mirmon data), so they don't get redirected to.  The common script 
/root/bin/mirmon-to-geoip does that.

Finally, the actual Perl handler that runs to do the redirection is this::

  PerlResponseHandler Apache2::Geo::Mirror->auto_redirect

Which auto_redirect function is defined here (not /usr/lib/perl5/Geo/Mirror.pm, 
that's an older version)::

  /usr/local/share/perl/5.10.0/Apache2/Geo/Mirror.pm


--
forwarded from http://savannah.gnu.org/maintenance/address@hidden/maintenance