[GNUnet-SVN] [gnunet] branch master updated (c756d928b -> 8b48d9288)

[gnunet]

This is an automated email from the git hooks/post-receive script.

rexxnor pushed a change to branch master
in repository gnunet.

    from c756d928b Build peerstore before ats
     new b889d108b added documentation for the ascension tool
     new 2798cec3a updated ascension documentation; added developer section
     new d91f5dcc3 fixed typos, improved ascension documentation
     new 8b48d9288 fixed up documentation for ascension

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 doc/handbook/chapters/developer.texi    | 118 +++++++++++++++++++++++++++++++-
 doc/handbook/chapters/installation.texi |  31 ++++++++-
 doc/handbook/chapters/user.texi         |  41 ++++++++++-
 3 files changed, 186 insertions(+), 4 deletions(-)

diff --git a/doc/handbook/chapters/developer.texi 
b/doc/handbook/chapters/developer.texi
index d9c92247b..2da262b34 100644
--- a/doc/handbook/chapters/developer.texi
+++ b/doc/handbook/chapters/developer.texi
@@ -7743,6 +7743,7 @@ record types.
 * The GNS Client-Service Protocol::
 * Hijacking the DNS-Traffic using gnunet-service-dns::
 * Serving DNS lookups via GNS on W32::
+* Importing DNS Zones into GNS::
 @end menu
 
 @node libgnunetgns
@@ -8073,6 +8074,119 @@ applications that use alternative means of resolving 
names (such as
 sending queries to a DNS server directly by themselves).
 This includes some of well known utilities, like "ping" and "nslookup".
 
address@hidden Importing DNS Zones into GNS
address@hidden Importing DNS Zones into GNS
+
address@hidden %**end of header
+
+This section discusses the challenges and problems faced when writing the
+ascension tool. It also takes a look at possible improvements in the future.
+
address@hidden
+* Conversions between DNS and GNS::
+* DNS Zone Size::
+* Performance::
address@hidden menu
+
address@hidden Conversions between DNS and GNS
address@hidden Conversions between DNS and GNS
+
+The differences between the two name systems lies in the details
+and is not always transparent. For instance an SRV record is converted to a
+gnunet only BOX record.
+
+This is done by converting to a BOX record from an existing SRV record:
+
address@hidden
+# SRV
+# _service._proto.name. TTL class SRV priority weight port target
+_sip._tcp.example.com. 14000 IN        SRV     0 0 5060 www.example.com.
+# BOX
+# TTL BOX flags port protocol recordtype priority weight port target
+14000 BOX n 5060 6 33 0 0 5060 www.example.com
address@hidden example
+
+Other records that have such a transformation is the MX record type, as well as
+the SOA record type.
+
+Transformation of a SOA record into GNS works as described in the following
+example. Very important to note are the rname and mname keys.
address@hidden
+# BIND syntax for a clean SOA record
+@   IN SOA master.example.com. hostmaster.example.com. (
+    2017030300 ; serial
+    3600       ; refresh
+    1800       ; retry
+    604800     ; expire
+    600 )      ; ttl
+# Recordline for adding the record
+gnunet-namestore -z example.com -a -n @ -t SOA -V rname=master.example.com \
+    mname=hostmaster.example.com 2017030300,3600,1800,604800,600 -e 7200s
address@hidden example
+
+The transformation of MX records is done in a simple way.
address@hidden
+# mail.example.com. 3600 IN MX 10 mail.example.com.
+gnunet-namestore -z example.com -n mail -R 3600 MX n 10,mail
address@hidden example
+
+Finally, one of the biggest struggling points were the NS records that are 
found
+in top level domain zones. The intended behaviour for those is to add GNS2DNS
+records for those so that gnunet-gns can resolve records for those domains on
+its own. This requires migration of the DNS GLUE records as well, provided that
+they are within the same zone.
+
+A solution was found by creating a hierarchical zone structure in GNS and 
linking
+the zones using PKEY records to one another. This allows the resolution of the
+nameservers to work within GNS while not taking control over unwanted zones.
+
address@hidden DNS Zone Size
address@hidden DNS Zone Size
+
+Another very big problem exists with very large zones. When migrating a small
+zone the delay between adding of records and their expiry is negligible. 
However
+when working with a TLD zone that has more that 1 million records this delay
+becomes a problem.
+
+Records will start to expire well before the zone has finished migrating. This
+causes unwanted anomalies when trying to resolve records.
+
+A good solution has not been found yet. One of the idea that floated around was
+that the records should be added with the s (shadow) flag to keep the records
+resolvable even if they expired. However this would introduce the problem of 
how
+to detect if a record has been removed from the zone and would require deletion
+of said record(s).
+
+Another problem that still persists is how to refresh records. Expired records
+are still displayed when calling gnunet-namestore but do not resolve with
+gnunet-gns. When doing incremental zone transfers this becomes especially
+apparent.
+
address@hidden Performance
address@hidden Performance
+The performance when migrating a zone using the ascension tool is limited by a
+handful of factors. First of all ascension is written in python3 and calls the
+CLI tools of gnunet. Furthermore all the records that are added to the same
+label are signed using the zones private key. This signing operation is very
+resource heavy and was optimized during development by adding the '-R'
+(Recordline) option to gnunet-namestore. This allows to add multiple records
+at once using the CLI.
+
+The result of this was a much faster migration of TLD zones, as most records
+with the same label have two name servers.
+
+Another improvement that could be made is with the addition of multiple threads
+when opening the gnunet CLI tools. This could be implemented by simply creating
+more workers in the program but performance improvements were not tested.
+
+During the entire development of the ascension tool sqlite was used as a
+database backend. Other backends need to be tested in the future.
+
+In conclusion there are many bottlenecks still around in the program, namely 
the
+signing process and the single threaded implementation. In the future a 
solution
+that uses the c api would be cleaner and better.
+
+
 @cindex GNS Namecache
 @node GNS Namecache
 @section GNS Namecache
@@ -8768,14 +8882,14 @@ regular expressions.
 
 Using the REST subsystem, you can expose REST-based APIs or services.
 The REST service is designed as a pluggable architecture.
-To create a new REST endpoint, simply add a library in the form 
+To create a new REST endpoint, simply add a library in the form
 ``plugin_rest_*''.
 The REST service will automatically load all REST plugins on startup.
 
 @strong{Configuration}
 
 The REST service can be configured in various ways.
-The reference config file can be found in 
+The reference config file can be found in
 @file{src/rest/rest.conf}:
 @example
 [rest]
diff --git a/doc/handbook/chapters/installation.texi 
b/doc/handbook/chapters/installation.texi
index c05f776f2..bdff20802 100644
--- a/doc/handbook/chapters/installation.texi
+++ b/doc/handbook/chapters/installation.texi
@@ -35,7 +35,7 @@ The mandatory libraries and applications are
 @item libextractor
 @item libidn
 @item libmicrohttpd @geq{}0.9.52
address@hidden libnss 
address@hidden libnss
 @item libunistring
 @item gettext
 @item glibc
@@ -1427,6 +1427,7 @@ and @code{[transport-https_client]} section of the 
configuration:
 * GNS Proxy Setup::
 * Setup of the GNS CA::
 * Testing the GNS setup::
+* Migrating existing DNS zones into GNS::
 @end menu
 
 
@@ -1693,6 +1694,34 @@ configured proxy) should give you a valid SSL 
certificate for
 @c FIXME: Image does not exist, create it or save it from Drupal?
 @c @image{images/gnunethpgns.png,5in,, picture of homepage.gnu in Webbrowser}
 
address@hidden Migrating existing DNS zones into GNS
address@hidden Migrating existing DNS zones into GNS
+
+To migrate an existing zone into GNS use the ascension tool.
+
+Ascension transfers entire zones into GNS by doing incremental zone transfers
+and then adding the records to GNS.
+
+You can find the source code here: @code{https://gnunet.org/git/ascension.git/}
+
+The software can be installed into a python virtual environment like this:
address@hidden
+$ python3 -m venv .venv
+$ source .venv/bin/activate
+$ python3 setup.py install
address@hidden example
+
+Or installed globally like this (not recommended):
address@hidden
+$ sudo python3 setup.py install
address@hidden example
+
+The advantage of using a virtual environment is, that all the dependencies can
+be installed separately in different versions without touching your existing
+python installation and its dependencies.
+
+Using the tool is discussed in the user section of the documentation.
+
 
 @node Configuring the GNUnet VPN
 @subsection Configuring the GNUnet VPN
diff --git a/doc/handbook/chapters/user.texi b/doc/handbook/chapters/user.texi
index ea41bbb6c..0703adafc 100644
--- a/doc/handbook/chapters/user.texi
+++ b/doc/handbook/chapters/user.texi
@@ -1453,6 +1453,7 @@ the user. This results in non-unique name-value mappings 
as
 * Using Public Keys as Top Level Domains::
 * Resource Records in GNS::
 * Synchronizing with legacy DNS::
+* Migrating an existing DNS zone into GNS::
 @end menu
 
 
@@ -1891,6 +1892,43 @@ DNS zone and that do not have a local gns service in 
use, it
 is thus advisable to disable the namecache by setting the
 option ``DISABLE'' to ``YES'' in section ``[namecache]''.
 
address@hidden Migrating an existing DNS zone into GNS
address@hidden Migrating an existing DNS zone into GNS
+
+After installing the tool according to the README file you have the following
+options:
address@hidden
+Ascension
+
+Usage:
+    ascension.py <domain> [-d]
+    ascension.py <domain> -p <port> [-d]
+    ascension.py <domain> -ns <transferns> [-d]
+    ascension.py <domain> -ns <transferns> -p <port> [-d]
+    ascension.py -h | --help
+    ascension.py -v | --version
+
+Options:
+    <port>          Port for zone transfer
+    <domain>        Domain to migrate
+    <transferns>    DNS Server that does the zone transfer
+    -d --debug      Enable debugging
+    -h --help       Show this screen.
+    -v --version    Show version.
address@hidden example
+
+To migrate the Syrian top level domain - one of the few top level domains that
+still supports zone transfers - use the following command:
+
address@hidden
+$ ascension sy. -ns ns1.tld.sy.
address@hidden example
+
+The program will continue to run as a daemon and update once the refresh time
+specified in the zones SOA record has elapsed.
+
+At this point you might want to write for example a systemd unit file to start
+and enable the service, so that your zone is migrated automatically.
 
 @node re@:claim Identity Provider
 @section re@:claim Identity Provider
@@ -1962,7 +2000,7 @@ Further, the "ticket" can be re-used later to retrieve 
up-to-date attributes in
 
 To list all given authorizations (tickets) you can execute:
 @example
-$ gnunet-reclaim -e "friend" -T (TODO there is only a C and REST API for this 
at this time) 
+$ gnunet-reclaim -e "friend" -T (TODO there is only a C and REST API for this 
at this time)
 @end example
 
 
@@ -2292,3 +2330,4 @@ protocol (--tcp or --udp) and you will again receive an 
IP address
 that will terminate at the respective peer's service.
 
 
+

-- 
To stop receiving notification emails like this one, please contact
address@hidden