Re: [Demexp-dev] Current status of development

[David MENTRE]

Hello Lionel,

2006/2/21, Lionel Eyraud <address@hidden>:
> Solution 2/ does indeed simplify things. But I am mainly concerned
> about scalability issues : storing more and more information on the
> server will inevitably lead to problems if you ever have a high number
> of users. Maybe it's not your main concern yet ;)

Yes, scalability issues are not yet a concern. ;)

Right now, I would like current code to be able to handle 1.000 to
10.000 users. It should be enough for the first experimental phase of
the project.


> But I think I would like an architecture where the main subject of
> demexp (voting) is separated from the cosmetic aspects and the history
> of each user.

I agree, that's why I started initially on solution 3/, separated web
proxy that stores client votes and seen questions.

> We could imagine several "history" servers, that are
> either proxies for the user and vote in their place to the central
> demexp server, or serve as a complementary service while the user still
> votes directly to the central server. I think I like best this last
> solution.

Yes but:
 - it complicates the architecture. From an engineering point of view,
maintaining two servers (two set of binaries, with configuration
files, socket ports, etc.) with there own state is much more
complicated that just one server (Murphy is not your friend ;-) ;

 - it complicates the programming (for the same reasons as above). A
distributed system is much more complicated to design and debug than a
centralized one;

  - one must consider a user migrating between the web interface and
the GTK client. A centralized server could manage this
configuration[1].

So, right now, I'm still in favor of option 2/ (everything on the server).

> In the very long term, I think you might need to have a
> hierarchical structure of demexp servers, that collect votes from a
> number of users and reports them to a central server. I know you do not
> think about that yet, but  I think it is a good thing to keep it in
> mind.

Other have suggested P2P demexp servers. I would like to keep sci-fi
suggestions after the 1.0 release. ;)


> Additionally, after having looked at the code of the server,

Nice!

> I have the
> feeling that it keeps all information of all questions, votes,
> participants, ... in memory while it is up, saves this information in
> an XML file each time it is changed, and never reads it back...
> (correct me if I'm wrong :)).

You're right.

> Still with scalability in mind, I am not
> sure that this approach will support a reaonably large number of
> participants/questions. I think only data about current or recently
> accessed objects should be kept in memory; and I also think that the
> data should be stored in a way that allows to read only a part of it.
> Personnally, I would recommend using a database system, since that's
> what they're designed for -- efficiently storing large data.

I completely agree. However, I'm using the current design only because
we are still at the design phase. For example, I don't know much about
the data structures needed for the delegation system.

In short, once the delegation system is designed and implemented, we
could look at storing all data in a database and keeping XML files
only for backups.


> Of course, this is my own opinion, and since you know this project much
> better that I do, feel free to disagree :)

I hope you understand my position but, in turn, feel free to disagree.
As Felix posts have shown, I don't think at all the options. My
position could seem a bit boring (and doing code that handles all
error cases *is* boring :-) but I think this makes the difference
between code that indeed works and unmanageable software.

Best wishes,
d.

[1] Of course, it should also be possibe with a proxy, but the design
is not entirely trivial.