[Gnumed-devel] pipelining experiment

[Syan Tan]

 

Some holiday fun while Karsten's away:-

 

 the pyPgSQL module uses cursors so it takes I think 3 -4 round trips (create cursor. execute cursor, fetch result, close cursor) for every sql statement , so this slows down  the access remotely salaam.

I used the script from test-area/sjtan/schemascan.py  to retrieve a Kirk's record ( which is Id= 12),

and this took some time too.    schemascan2.py reorganizes the single threaded single connection retrieval into

a multi threaded worker pool each with it's own connection, and uses a Queue.Queue to submit statements to available workers. 

 I was able to use about 30 threads; It looks faster, but using unix "time python schemascan2.py" the user time is still about a minute.

( Each statement will still take 3 round trips because of the use of sql cursor, although now there is some pipelining, due to concurrently running connections ).

How long does it take running as a batch file to psql  ?

If the XMLRPCServer serving yaml, text, xml , serialized python object or whatever, did materialize, what would be the gain ?

( using a web server, one could retrieve a constructed record in 10 seconds , I think).