Re: [Savannah-hackers-public] Check that mirrors are not corrupted?

[Ian Kelling]

Thérèse Godefroy <godef.th@free.fr> writes:

> Hello,
>
> In the last 2 months, we had 2 cases of mirrors that were reachable and
> up to date according to Mirmon, but were actually corrupted. Of course,
> these mirrors are not listed anymore.
>
> - RT #1664168 2020-12-15
>
>> recently I discovered incidentally that the
>> http://savannah-nongnu-org.ip-connect.vn.ua mirror is not always serving
>> the correct files. It seems some files just contain '0' (zeros) after
>> downloading them.
>> 
>> Two examples discovered, due to invalid md5sums, are:
>> 
>> http://savannah-nongnu-org.ip-connect.vn.ua/freetype/freetype-2.4.11.tar.bz2
>> http://savannah-nongnu-org.ip-connect.vn.ua/freetype/freetype-2.4.11.tar.gz
>
>
> - RT #1684954 2021-02-14
>
>> I try to download http://ftpmirror.gnu.org/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz 
>> from Athens / Greece
>> Most of the times I get redirected to 
>> http://ftp.ntua.gr/pub/gnu/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz which is not found 
>> (404). The correct URL for the NTUA mirror is 
>> http://ftp.ntua.gr/pub/gnu/gcc/releases/gcc-7.3.0/gcc-7.3.0.tar.xz (note the 
>> “releases/” subfolder).
>> Occasionally I get redirected to ftp.cc.uoc.gr which works.
>
> This case is even more bothersome than the first one, because the issue
> is with gcc, and the filetree has been tampered with.
>
> Can you think of a way to automatically check that a mirror is correct?
> Thanks!
>
> All the best,
> Thérèse

The only way is to download everything on the mirrors and check their
sha256sum against whats on the ftp. It would be good to develop that
program and run it on fencepost, since the ftp is shared there on
/srv/data/ftp-mirror/ftp. I'd put a 1+ second delay between each file
download, this is fine if it takes a month or more to go through all the
mirrors, because any malicous mirror could easily have the modified file
only served to specific ip addresses, or just ip addresses from a
specific region of the world.

- Ian