Re: Git mirrors

[John Wiegley]

>>>>> Óscar Fuentes <address@hidden> writes:

> AFAIK, the only one keeping the savannah git mirror up to date is Andreas
> Schwab and he is doing that manually. I'm not sure what's the actual process
> he is using, but if it is fast-import/fast-export the worst thing that could
> happen is a failed push because the other part pushed first. In that case a
> pull should fix things. But he has the authoritative word on this.

Sure enough, I'm stepping on whoever else is running an automated mirroring
process.  Just about every hour I'm re-deleting branches that they are
re-pushing:

,----
| ======================================================================
| Wed Oct 12 18:00:01 CDT 2011
| ======================================================================
| receiving incremental file list
| 
| sent 357 bytes  received 20797 bytes  475.37 bytes/sec
| total size is 376714147  speedup is 17808.18
| To savannah:/srv/git/emacs.git
|  - [deleted]         old-branches/cedet-branch
|  - [deleted]         old-branches/emacs-unicode
|  - [deleted]         old-branches/gerd_defvaralias
|  - [deleted]         old-branches/gnus-5_10-branch
|  - [deleted]         old-branches/multi-tty
|  - [deleted]         old-branches/pending
|  - [deleted]         old-branches/rmail-mbox-branch
|  - [deleted]         other-branches/gerd_0001
|  - [deleted]         other-branches/gerd_int
|  - [deleted]         other-branches/miles-orphaned-changes
|  - [deleted]         other-branches/thomas-posix1996
|  * [new branch]      tmp -> tmp
`----

Who do I talk to so that they disable their mirror, or at least refine it?
Below is the script I'm now using.  It relies upon a Python script, also
attached.

John

------------------------------------------------------------------------
#!/bin/bash

# Many thanks to Andreas Schwab <address@hidden> for this script.  It
# is currently maintained by John Wiegley <address@hidden>.

test last-sync-start -nt last-sync-ready || touch -r last-sync-start last-sync
touch last-sync-start

echo ======================================================================
date
echo ======================================================================

rsync -av --del --delete-excluded \
    --exclude=obsolete_packs \
    --exclude=lock/ \
    bzr.sv.gnu.org::bzr/emacs/ emacs.bzr/

dc emacs.bzr
find * -path '*/.bzr/branch/last-revision' | while read r; do
  test $r -ot ../last-sync && continue
  b=${r%/.bzr/*}
  b2=$b
  test $b = trunk && b2=master
  test $b = master && b2=bzr/master
  echo $b
  bzr fast-export --plain --marks=../bzr-marks -b $b2 $b | (
    cd ../emacs.git
    git fast-import --quiet --export-marks=../git-marks 
--import-marks=../git-marks
  )
done

dc ../emacs.git
git push --mirror savannah:/srv/git/emacs.git

cd ..
bin/correlate | gzip -c > commit-map.txt.gz

touch last-sync-ready

exit 0

### emacs-mirror.sh ends here

------------------------------------------------------------------------

#!/usr/bin/env python

# This is bin/correlate

import re
import os

def load_marks(path, inserter):
    marks = open(path)
    for line in marks.readlines():
        match = re.match(':([0-9]+) (.+)', line)
        assert match
        ident, info = match.groups()
        inserter(table, ident, info)

    marks.close()

def insert_at(table, key, value, pos):
    if key not in table:
        table[key] = [None, None]
    table[key][pos] = value

table = {}

load_marks('bzr-marks',
           lambda table, ident, info: insert_at(table, ident, info, 0))
load_marks('git-marks',
           lambda table, ident, info: insert_at(table, ident, info, 1))

for key in table:
    print table[key][0], table[key][1]