[circle] Circle Design

[rodrigob]

Hi,
In this post I will try to present my personal vision of what should be the
Circle new internal design. I have discussed the topic several times with
ThomasV (over the circle chat obviously).

I profit the Gui/Core separation topic to post the design, but it try to attact
various other topics too.

Today the Circle is one of the most avanced P2P applications. Unfortunatelly his
internal design make the code hard to manage and extend, truncating most attempt
of extension, plugins, etc... At this point cleaning and documenting the code
internals is an important thing.

The Circle has been develloped since various years and have a very interesant
and smart design, but obviously, being one of the "first of his class" some
future issues where not detected. Actually Python applications are much common,
and there is much more experience in the field of software design. Python itself
has grow, and now new concepts are aviable.

Personnally I think that Twisted Matrix is the better tool to devellop python
network applications. It is stable, well documented, pure pythonic, have proved
to "work when needed", and have a very smart design. Using Twisted enforce
following smart design, which is more good than bad. Also Twisted is almost a
"standard" in the area of python network applications, thus his nomenclature and
concepts are more and more common. Using Twisted will ease future extensions or
coupling to other protocols.

Actually I agree that "twistifying" the Circle is not a central aspect, but I
believe that it is "a good thing to do" in order to standarize, clean and prune
the Circle implementation. I have already studyied this topic, and "twistifying"
the Circle is not an easy task but it is possible. I will let this topic for a
future discussion, just I have to mention that job will be centered in
converting the 80 'task' functions to Twisted Flows.

The design descripted here does not enforce a "twistification", but have been
created having it in mind.

The core ideas are very, very simple. Maybe the description can be somewhat
tedious, but getting it coded, in practice is a conceptally simple thing.

The idea is: the Circle can be separated in three layers. The Distributed Hash
Table maintenance, as set of Services implemented on top of the DHT, and the 
Gui.

The whole design follow the "Components" (components, interfaces, adaptators)
software design style, and play a little with the class definitions. The spirit
is similar to the Twisted design and to the Zope3 design
(http://twistedmatrix.com/documents/howto/components
http://dev.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/FrontPage ).

The whole dynamic is somewhat like this:

There is an application that exist and instanciate a DHT_node (that meet the
IDHT_node interface). Then it instanciate various Services objects (that meet
the IService interface), and register them to the DHT_node object. At this point
(omitting some minors details) the application will be running and
providing/interacting through the different services.

If the application is a gui application then more things happens. The
application will check (through a method defined in the interface) what services
are registered in the DHT_node, in order to be sure that the minimal services
assuptions are covered.

Now comes the funny part that touch the actual topic Core/Gui separation, so I
will describe it in more detail.
The DHT_node and Services implementation only take care of provide the strictly
core functionalities, and thus are the core code. For each service (and, maybe,
for the node too) a IGui interface Adaptator is defined. This adaptators will
provide the functions to match the IGui interface, this include every specific
functions of the Services that every Gui should use.

Thus for each services a class for the IService will exist, and a class for the
Gui adaptator. Notice that the Gui adaptator provide code that is Independet of
the underlying User Interface engine (GTK, QT, Tk, text, Windows, remote access,
etc...), the gui adaptator define the methods of the services that are specific
of the gui, and that any user interface could make use of.
Using the Gui adaptators there is a very nice way to separate core code from gui
code and to define the specifics methods of the service (and, maybe, of the
dht_node) that will be aviable for the graphical user interface. 

Thus the graphical application will retrieve the services objects and just doing
"the_service = IGui(the_service_instace)" they will retreive the IGui interface
(an object instance) to that service.

Using this structure make the application more flexible and better defined. The
IDHT_node and IService interfaces documents what have to provide a new DHT
implementation (Kademlia, CAN, Viceroy), the IService define what/how a new
service over the DHT have to provide (some base methods, a long running "task",
and rpc methods that extend the protocol). The DHT implementation explain the
low level protocol topics, the Service implementation explain the service
specific behaviour, the Gui Adapators implementation explain the aviable methods
and how they map to the service internal methods.

To ease to catch the idea, I attach a simple picture (see circle.png).

That is the conceptual aspect.
In practice the job would be:
- define the IDHT_node, IService and IGui interfaces
- Move and adapt the code to match the Node interface
- Separate the related portions of code into the Services class (that match the
IService interface) and the BaseGui classes (that matche the IGui interface)
- Move and adapt the Gtk code run as an interface to the new structure
- Adapt the daemon code to use the new structure

This refactoring will probably imply some bugs insertions, but it is a
reasonable cost to pay.

Having this architecture the plugins extensions will be much more easy to
devellop (most of them registering a new service and extending the platform
specific gui). The devellopment of new service will be much easier because the
interface will be defined and documented. The DHT_node abstraction will ease the
change of the underlying protocol, because the related code will be compacted in
one class, and his change will not affect the services code.

This design also allow to implement various of the desire of flexibilty
enhancement. For example, as shown to ThomasV the "Gui in a daemon"  is very
easy to do and do not require sockets usage. The use of adaptators will make a
very minimal use of resource when no gui is used.
The abstraction also allow to devellop remote accesable interfaces, just by
replacing the platform dependent gui by a rpc object.  

Ok, long post, thinks this ideas, I will try to create the first piece of code
to give a more tangible idea. I have already sent to ThomasV a first glance of
the Node/Services separation. I'm not confident enough with the code to do the
refactoring by myself but I have time and motivation to help with at the high
and low level.

RodrigoB.

circle.png
Description: PNG image