Re: [Demexp-dev] How to get through the network an unknown amount of data?

[David MENTRE]

Hello John,

skaller <address@hidden> writes:

>> One obvious answer would be to use an API similar to Unix API to list
>> files in a directory:
>> 
>>  - a function start_listing() to start the listing;
>> 
>>  - a function continue_listing() to have the let's say next 10
>>    items of the listing.
>> 
>> However, I'm wondering if such a design would play nicely with OCaml
>> data structures like Hashtbl? Any idea how to stop and restart an
>> iter-like operation on an OCaml Hashtbl.t?
>> 
>> Any idea of a better design?
>
> You may want to look at ExtLib's Enum concept.

Thanks for the pointer. I'll look at it.

> The standard way to fetch things from a database
> is that your 'select' request returns a cursor object;
> that is, an iterator. So you have operations:
>
> let it = get_selection <description> in
> try while true do
>   let data = get_data it in
>   <do something with data>
>   advance it
> done with End_of_data ->
> close it

Yes, it looks more OCamlish. :)

Regarding the network side, I've asked the advice of some
colleagues. They advise me to use a simple index-based iterator (so
similar to your proposal, but where the iterator is explicit).

Needed network API (as seen from client side) :

 - num_elements = get_number_of_elements(): returns the maximum number
   of elements;

 - get_elements_range(index_base, size): returns /size/ elements,
   starting at /index_base/.

the client do a:

 i := 0
 while !i < get_number_of_elements() do
     el_list = get_elements_range(i, 10)
     <process el_list>
     i := i + 10
 done


The advantages of this approach:

 - all the context is in the client, the server does not need to
   maintain state per client;

 - the degenerated case where the client can only fetch one element at a
   time is included (size = 1);

 - this approach is compatible with indexes as seen in real SQL
   databases.

So I'll use this API, except if somebody has a better proposal (with a
rationale ;).

It remains to code all of that. :)


Yours,
d.
-- 
 David Mentré <address@hidden>