Filesystem as database (was Re: [Gnumed-devel] address@hidden: Announcing the OpenHRE.org Portal])

[Tim Churches]

On Fri, 2004-09-03 at 18:49, Karsten Hilbert wrote:
> > BTW ,  just as idle conjecture, what would the performance of a record 
> > system that is directories "a  b c d ..." and patient
> > surname, firstnames, date of birth  as file names, and records as text 
> > files , perform like, say with 10,000 patients?
> > i.e. using unix commands find to find the patient record; text editor to 
> > change it; scripts to run diff on ~ and original name files, and
> > archiving the diff; grep to read diffs and apply commands such as 
> > "script"; cron to create document subdirectories and ln -s to link
> > to imported documents in  a import document directory.
> A person I talked to here in Germany uses that scheme for
> document archival and claims performance is great.
> 
> However, this has no ACID properties that I can see. Neither
> referential integrity.

Here's another exa,ple: http://www.vnunet.com/comment/1157781

Using the filesystem as a database is not a silly idea, although any
indexing/access system beyond the primary key (the filename) needs to be
maintained by the application. But recent discusion re phonetic lookups
etc demonstrates that back-end SQL databases don't completely solve the
indexing/access problems anyway. Regarding referential integrity, yes,
you would need to adopt a hierarchical or network storage model, and
consistency is then the application's problem - but it still is, in
part, with any system using a relational database too. Atomic updates
are probably satisfied with a journaling filesystem. I suspect that
speed would be better than an SQL database for many operations. Indeed,
many of the XML database servers which are appearing are just layers on
top of a filesystem - either an operating system filesystem or an
equivalent such as a BSD database.

One final observation: Gnumed already uses a filesystem database to
store and access all its Python code. If a Python module contained in a
file is incompletely retreived or the lines of which it consusts are
retrieved in the wrong order, then your application fails. But that
never happens. Of course, code is not updated as often as database
records, but otherwise the requirements are similar. Then consider how
CVS works...
-- 

Tim C

PGP/GnuPG Key 1024D/EAF993D0 available from keyservers everywhere
or at http://members.optushome.com.au/tchur/pubkey.asc
Key fingerprint = 8C22 BF76 33BA B3B5 1D5B  EB37 7891 46A9 EAF9 93D0