Re: [Tsp-devel] dtest: a distributed test framework

[Frederik Deweerdt]

On Wed, Apr 25, 2007 at 01:56:00AM +0200, Eric Noulard wrote:
> 2007/4/24, Frederik Deweerdt <address@hidden>:
> >I'm not strongly attached to XMLRPC, it just seemed handy to me.
> >> I like the idea of being able to build autonomous binary which
> >> could be deployed without python being installed I wonder
> >> if it could be optional.
> >You mean in case python is already installed on the target?
> 
> No I mean if the python
> scripting does ONLY takes place on the machine you initially
> "make test".
> That is imagine your launch AND test_engine scripts are pure python
> and you do scp/sftp/ssh through a Python SSH implementation
> 
> (twisted conch :  http://twistedmatrix.com/projects/conch/
> or
> paramiko: http://www.lag.net/paramiko/
> are options)
This would have the advantage of getting rid from the horrendous shell
scripts that glue everything together at the moment.
> 
> then you have one (or several) "local" python scripts which
> "only" handles ssh connection whose stdin/stdout/stderr are
> directly controlled by the local python scripts.
> Thoses scripts may barrier/ok locally.
I agree that the test results could come back through stdout, specially
if we use TAP, which only defines an output format.
However, I don't see how you'll get barriers from ssh: we want the
barriers to be inside the running code that is:
client                  server
wait_for_barrier(1)
                        do some setup
                        wait for connections
                        wait_for_barrier(1)
========== Do the actual testing ============

That's also why XMLRPC is interesting, is that you can report the test
results and use the barriers from whatever language you like.

> The remote target stdout results comes from SSH link.
> The remote target file results should be scp from target to
> dtest local host. (this is a weakness of my scheme)
OK, but we don't really need them if we've got stdout?
> 
> With this scheme you may even drive distributed linux test
> from a Windows box or the converse or mixed
> distributed  systems too :=)
That would be great, indeed. The binary thing prevented that. However,
cxFreeze can produce executables for windows, so I think that this was
already doable to a certain extent (ie using cygwin/cmake for win).

While writing, I was thinking that maybe defining a "typical" TSP test
could be helpful. Do we agree that the following is typical? In fact,
what we want dtest to be would depend on what we want to do with it :)

client                          server
TSP_consumer_init && OK(1)      TSP_provider_init && OK(2)
barrier(1)                      
                                TSP_provider_run && OK(3)
                                barrier(1)
TSP_consumer_connect_url && OK(4)
TSP_consumer_[...] && OK(X)
barrier(2)
                                TSP_datapool_push_next_item && OK(X+1)
                                barrier(2)
                                barrier(end)
Get item, check value && OK (X+2)
barrier(end)

Regards,
Frederik