[VM] using VM to manage large mail archives

[markd]

Hi Uday,

I am changing the subject of this mail thread to be more
appropriate.  Hopefully, it's of interest to others and not just
overloading more mailboxes.

Uday Reddy <address@hidden> writes:
> I am not sure that mbox is that much of a limit, once you figure out how to
> archive old mail.  

It's not the format so much as the need to read an entire mbox
into memory.

> I keep about 3 months worth of email in my "INBOX" and
> archive older mail into 3 month quantities of mbox's.  Virtual folders then
> help me combine these mbox's into whatever combinations I want.  Virtual
> folders are definitely your friend here.

I am way beyond being able to sort through mail to decide what
to keep.  I just automatically archive everything on delivery
and keep per-month folders. Things I need to deal with stay in
the INBOX and get periodically flushed when it gets too big.

It is much less work to just archive everything and dig things
out as needed.  It's frequently been a surprise what mail was
needed from the archive.  Not necessarily stuff I would have
saved if I manually archiving.

The volume has gotten absurd, especially when I have to work on
grants with people who insist on e-mailing around Word documents
as a work flow.  My archive for December is approaching a
gigabyte :-(  

This is way beyond anything that will work with reading mbox
files into memory.  Even the current INBOX gets overload and
ends up spending way too much time garbage collecting.

What I would really like is the ability to do start a search on
the archive within vm, get back a list of hits in something
similar to a summary buffer and then look through them with vm
as if normal email.

Current thinking is that running a private imap server on top of
my mail archive might be an approach.
 
> Then we have the IMAP folders and external (headers-only) message feature of
> VM.  These already provide a powerful "abstraction layer" but only for IMAP
> users.  Eventually, this kind of thing will also become possible to other
> forms of external storage: maildir etc.

YES!  This is exactly the example it had in mind.  Forgive me
for speculating, since I don't know a lot of the internals of
VM.  What I am seeing VM starting to evolve into an user
interface to various mail back ends rather than an internal
format and the special-case external ones.  Better integrating
powerful back-end search engines could come from this.

Cheers,
Mark