[GNUnet-developers] MySQL/mine ideas.

[Jan Marco Alkema]

Hi all,

I am very happy with the current discussion about the use of (MySQL)
database for gnunet ---). I mine opinion a good database design is crucial
for the use and acceptation of gnunet.

>(I hear grinding of teeth somewhere).

Mine only "drive" is to make a product with high value/quality. I find it
mine obligation to give feedback if I see improvements in the product.

Today I have reviewed mine ideas I wrote in September 2002. See appendix for
mine ideas.

If someone has better ideas please let me known ----),

Greetings Jan Marco

Purpose:

The purpose of www.foonsearch.com is the ability to perform the next five
operations on intellectual property:

1.      Searching;
2.      Identifying;
3.      Categorization;
4.      (Automatically) distribute;
5.      (Legally) Obtaining.

Business Model

Foonsearch is based on a social economic model. Each node is forming an
opinion on all the other nodes it is in contact with. Depending on that
opinion, the node will decide which requests it will perform.

As long as a node is not busy, it will serve all requests. If it gets busy,
it will prefer requests from nodes that will pay for the requested services.
At run time the server and client can make contracts about the execution of
the requests.

The node gives credit for nodes that execute transactions in a proper way
and takes credit for nodes who frustrate the Foonsearch network.


I think of the next main buttons should be on the website:

. Home
. Register
. Info
. Catalog
. Search
. Download
. Transaction
. Games

A short explanation of the buttons:

. Register
- Give a GlobalUniqueIDentifier (GUID) {GUID foonsearch server/client =
HASH160(public key) + MD5(public key) + CRC32(public key)} to an end user of
the Foonsearch network;
- I'll collect routines out of http://www.oscommerce.com/ to make the
register function in PHP;
- I will use the concepts of "Caller-Id". On mine second telephone line I
will install a speech/voice modem and with a speech program I can generate a
Pincode (password in numbers). I can see the telephone number on my Linux
server and search the entry in the telephone book (See megafoon.da.ru/ and
http://www.mycgiserver.com/~foonsearch). People get 15 minutes to enter the
Pincode and then I'm "sure" that the person is who he tells he is. The
Secret Public key will be sent to the end user.

. Info
- Documentation (manuals);
- FAQ;
- News;
- Business Model;
- Contact info;
- Forum;
- Project information;
- Encrypted mail;
- ICQ, IRC;

. Catalog
You must can make "footprints" (footprint = "HASH160(File) + MD5(file) +
CRC32(file)" + file size) of a Cdrom, Audio CD, DVD, files and distributions
(zip/tarball archive). After categorization they can be found on the
Internet very easily. If you known what you want you can ask the right
query. The catalog program have to decompress with different compression
techniques (zip, tar, gz, etc) because in a archive the files exist. Files
and tracks (of Audio CD's) are the smallest "atoms" of the Foonsearch
network.

. Search
Here can a Cdrom, Audio CD, DVD or file(s) be found. For instance you want
"windows2000 professional" . Out of the root (MySQL) database you known what
files the Windows2000 distribution exist. First will be looked which files
you have locally. Only the files you haven't will be retrieved from the
other nodes of the foonsearch network. An other example. If the last version
of MySQL has two new files. The receiving node (Node A) will perform a query
to the neighbor foonsearch servers. The neighbor server who have the two
files will give a message to node A. Node A ask one of the replying nodes to
perform the extraction, compressing of the extraction and sending the result
to Node A).

. Download
Here a download can be performed. There will be different GUI routines for
foonsearch: Command line, Windows API, Java client, PHP on apache web server
and a GTK interface. I prefer JAVA and the PHP implementation.

Example: In the PHP environment the end user can click on a distribution,
track, file, etc. If these information has no copy rights it is easy to
download. If the intellectual property has copy right the person can set it
open for the world. That person faces the law. The permissions can be set in
the MySQL database. If the person has prohibited to copy the intellectual
property out of the Foonsearch server. The record on the screen of the end
user will have an URL with host "localhost". The end user will automatically
go to the installed and running foonsearch HTTP-gateway to retrieve it from
the neighbor foonsearch servers. To make a first java client I will use the
java client of the Dutch telephone book (decompile the class files).

. Transaction
You can give 'sell' and 'buy' transactions. Also you can bid on a
transaction. For example: I would give you 2 euro if you haven't enough
network capacity to give me the last CD of the Dutch telephone book before
noon tomorrow.
I will work with Peter Kaas to port Perl routines to C++ of a Dutch bank
(Girotel/postbank). The execution of transactions I will perform in the
running process (fshcore). An amount of money is only meant for coordination
of the scarce resources (network band capacity, cpu tics, memory, disk
space). N.B. I haven't commercial intentions in making of the transaction
module.

. Games
I'll will use games (blackjack, IQ-tests, etc) to check the working of the
transaction module. You must can give an option that you like to play cards.
If someone (person B) would actually like to play cards he performs a query
on the foonsearch network. The message will be give to the end users. If
there are enough replies person B can play card. If a person wins a play a
transaction may be executed. This relies on the "assigned (implicit)
contract".


Foonsearch core programs:
- cdrecord;
- grubclient;
- fshcore;
- fshdownload;
- fshpeerinfo;
- fshstatus;
- fshcatalog;
- fshcataloghost;
- freedb;
- girotel-gto
- database synchronization.

The next (MySQL) databases (Distribution, Keyword, Release, Filetrack,
Volume, Host, Mount, Access, Transfer, Crontab, Transaction and Log) I
suggest:

Distribution:
In this table the distributions will be inserted. For example Redhat, "The
river" of Bruce Springsteen, Fatal attraction DvD, etc.

Record format: Distribution number, "HASH160(name, publishing year,
publisher, copy right, etc) + MD5(name, publishing year, publisher, copy
right, etc) + CRC(name, publishing year, publisher, copy right, etc)", name,
publishing year, publisher, copy right, etc.

Keyword:
In mine opinion the most keywords will be assigned to distributions, but if
you wish you could assign it to a specific release or specific FileTrack.

Record format: Type (D=Distribution, R=Release,F=FileTrack),
(Distribution|Release|FileTrack) Number, Keyword

Release:
In this table the different FileTrack are mentioned of which a specific
distribution exist of.

Record format: Release number, Distribution number, Operating System,
FileTrack number, Type (data, audio, life, studio).

FileTrack:
In this table all independent footprints of files and footprints of (audio)
tracks are been given. The table will automatically be filled by the catalog
program.

Record format: FileTrack number, path, MD5,  FileTrack size, Read/Write
rights.
Foonsearch server can retrieve the FileTracks depending on the set rights in
this table. Un registered Foonsearch (server without GUID) servers would
have the lowest priority.

Volume:
In this table are the different volumes been given of a specific
distribution. For example Redhat 7.3 has nine volumes. Eight of the volumes
are CD and one has a DvD format. A lot of Volume information can be
retrieved out of Cd-grabbers source code like cdrecord.

Record format: Volume number, Distribution number, Version number, FileTrack
number,  Number of sectors, Number of tracks, etc.

Host :
It should be possible to have more black boxs (foonsearch servers) running
on one server. The only restriction is an unique port number on the goal
server.

Format: Host number , "HASH160(public foonsearch key) + MD5(public
foonsearch key) + CRC32(public foonsearch key)", DNS host name, Ip address,
port number.

Mount:
In this table will be the actual mount will be given of a specific Volume.
Note: Not all mounts must be online. A foonsearch server can ask another
foonsearch server to put it in the Cdom/DvD player manually. More likely the
Cdrom/Dvd will be copied to hard disk. After the Cdrom/Dvd is transferred to
requesting foonsearch server (transfer of files and transfer of MD5 + file
size) the Cdrom/Dvd will automatically be removed of the local hard disk.

Format: Mount number, Volume number, URL (related to Host table)

Access:
The read right in FileTrack works like the "access deny thing". In this
table a foonsearch server can grant another foonsearch access to prohibited
FileTracks like the "host allow thing". After a FileTrack is transferred the
footprints are exchanged. We are always sure that the transfer has
succeeded. The number of copies will be reduced by one. If you have made a
transaction. For instance: Some GUID will give you five euro for "MCAFEE
Virusscan NA-519-0003-UK-1". Here you will give that GUID access to copy it
once. If the transfer of less than 34.5 Mb (transfer files should first be
zipped) is successful (see transfer table) the transaction can be executed
in the transaction table.
Note: You should also set the rights here for upload to a foonsearch server.
You must prohibited that someone floods yours hard disk.

Record format: Access number, Distribution number, Release number, FileTrack
number, IP-number, GUID, new read rights, Max number of copies.

Transfer:
In this table FileTrack transfers will accurate be monitored which
FileTracks are sent and received and to which GUID/Ip-address. You can see
that one specific foonsearch server uses very much resources of your
foonsearch server. In mine opinion it should be very dynamical and flexible
based on "rules". You must easy give to stop the current FileTrack transfer
and let it resume later in time. The transport of the FileTrack information
that has performed must not be harmed. The end user can give you a bid to
give him more network band capacity if he can't wait till later.

In mine option there are two techniques for sending large files:
1)      The Winrar technique: Zip the file and split the file in equal fixed
blocks. Each block has a MD5 (or CRC check). Better suggestion is to split
the files in runtime of 10 minutes transfer time with MD5 and file size.
2)      Zip the file and send it over and after the exchange of the footprints
(MD5 and file size) the files can be equal. If they are not the same: Binary
selection technique with MD5 (and file size) can be processed till the
blocks are the same.

Record format: Transfernumber, FileTracknumber, IP-adress, GUID, status.

Crontab:
The crontab table has two purposes:
1)      Stopping and starting of a job. For instance the foonsearch engine 
delays
a request (for sending a FileTrack). In this table an entry will be inserted
to resume the transfer later.
2)      Configure, starting and stopping a Foonsearch application.

Pseudo code for CronTab:
If (current_release < last_release)
{
  If (current_patch <  last_patch)
    {
1.      Retrieve files which are changed in last release;
2.      Execute the <./configure + options> out of the Configstartstop table;
3.      Execute the <Make> option out of the Configstartstop table;
4.      Execute the <Make install> out of the Configstartstop table;
5.      Kill the running processes (info stands in Configstartstop table);
6.      Throw away the not used files. (files which were in the old release but
are not in the last release).
7.      Start up the program (running demons).
   }
}

Record format: Distribution number, release number, Operating system, Type
(start, stop, configure, remove), script

Transaction:
This table can be filled like the record structure of a banking system.

Record format: Transactionnumber, GUID_from, GUID_to, amount of money,
Conditions, Info.

Log:
In this table all errors, warnings will be logged. Normal the running
process will generate the log. You could choose a specific foonsearch server
to monitor many other foonsearch servers. This could be in a "private"
company network.

Record format: Host Number, message type (error, warning, debug),
Information.

Mine stages are:

1)      "Java GUI" with MySQL SQL database via ODBC.
2)      "program" fshcore (gnunetd + vsftpd) with ODBC connection in Linux and
Windows environment. Integrate ftp with gnunet concepts. Make gnunet also
available as a Linux service.
3)      "program" fshstatus, fshinsert and fshdownload in a MySQL environment;
4)      Make the Montana Linux server slave for the rootserver
(www.foonsearch.com). The Montana server can see in the MySQL database of
www.foonsearch.com that there is a new release of Foonsearch. Automatically
he will download the last release and will reboot the Foonsearch system.
5)      "program" the cdrecord routines.
6)      "program" the rootserver routines.
7)      "program" the website www.foonsearch.com.

The "next step" must be "Not how many files you have shared, but how many
database you have shared."

Greetings Jan Marco

Appendix A: A catalog example:

./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Operating System
Installation" -s 1 -d 1
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Operating System
Installation" -s 2 -d 2
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Operating System
Installation" -s 3 -d 3
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Source Code" -s 1 -d 4
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Source Code" -s 2 -d 5
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Applications StarOffice
5.2" -d 6
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Documentation" -d 7
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Productivity
Applications" -d 8
./catalogcd -o "linux" -n "Redhat" -v "7.3" -t "Personal DVD" -d 9

Appendix B: Relevant Links:
http://www.kazaa.com
http://edonkey2000.com
http://www.overnet.com/
http://www.grub.org
http://sourceforge.net/projects/girotel-gto/
http://www.oscommerce.com/
http://megafoon.da.ru/
http://www.mycgiserver.com/~foonsearch
http://sourceforge.net/projects/freedb/
http://megafoon.da.ru/
http://members.lycos.nl/xim/mps.php