Re: [Gnumed-devel] Results tracking

[Karsten Hilbert]

> at work , the secretary showed me a stack of paper to be scanned
> in about 15 cm high each day. 50 x 5 x 15 = 25 metres a year.
> I can see why one might want to a separate service for blobs.
How we (my parents, that is) do it:

Scan the documents at a local workstation into a queue
directory on the server. This can happen in parallel with as
many scanner workplaces you can afford. Works as fast as the
server can accept files via Samba. The EMR isn't affected
beyond searching for a patient and calling the scan
application from within it - just as any regular EMR task
would require.

On (possibly yet other) machine(s) add metadata to on-disk
scans (the patient metadata is added during the scan process
already). Write to another staging directory on the server.

At night when no one is working with the EMR and only the
server is running cron kicks off an import script that scans
through the staging directory of the previous step and imports
everything it finds there. This is where the database of the
EMR comes into play.

All this is done with GnuMed.

> Our single server chokes in throughput sometimes : not sure
> if it's our lack of scsi drives, lack of cpu performance, lack of
> gigabyte switch and network interface cards on all computers,
> or using network file system locking to implement concurrency
> for medical record access, or just the an excessively sized
> not highly scalable database system used for our emr
Most likely it is "reckless" programming where separation of
concerns isn't properly obeyed. Eg. do not bother to build up
load on the EMR database unless *really* needed. Eg. do
everything possible in staging without touching the EMR
database (may use another database, though). Do the actual
import with some sensible load balancing, eg. a) do it async.
and only import so many blobs in a given amount of time (eg.
leave significant gaps for other processes to get at the
data). This is how auto_vacuum works on PostgreSQL, too.

All this pretty much applies to importing lab data as well: Do
not mix fetching and importing. If fetching hangs/fails then
importing won't be affected/hang, too, if separation is
properly used.

Karsten
-- 
GPG key ID E4071346 @ wwwkeys.pgp.net
E167 67FD A291 2BEA 73BD  4537 78B9 A9F9 E407 1346