[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [Sks-devel] SKS RAM usage gone haywire
From: |
Phil Pennock |
Subject: |
Re: [Sks-devel] SKS RAM usage gone haywire |
Date: |
Sun, 15 Feb 2009 15:00:35 -0800 |
On 2009-02-15 at 10:05 -0500, Yaron Minsky wrote:
> Ari is right: there's nothing inherent about the algorithm that should
> require an ever-growing use of memory. OCaml itself is very careful about
> reclaiming unreferenced memory, but that of course does not preclude a
> memory leak in the code.
>
> So far, I have no real clue as to what is going wrong. I could imagine that
> the caching at some level is overly aggressive. There are a number of
> configuration variables that control how much caching there is. Some of
> these are explicit caching numbers that are used by the actual DB, and some
> if it is caching that the prefix-tree datastructure does on its own. For
> instance, there is a bound (defaulted to 1000) on the number of in-memory
> nodes of the prefix-tree.
>
> The idea that some weird query or a server in an unusual state is exercising
> some bug that blows up the memory utilization seems possible as well.
>
> Has anyone confirmed if it's the db or recon process that is blowing up in
> memory? That would help figure out what's going on. For instance, it's
> pretty unlikely that a query from a web-crawler would cause the recon
> process to explode in size.
It's recon, and the problem has stopped since I took keys.nayr.net out
of my config and that was the most recent change before things went
ballistic.
Ryan, sorry to name you and point figures publicly while still
investigating, but since at least one other person has seen the same
failure, warning trumps politeness. :(
-Phil
- [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/13
- Re: [Sks-devel] SKS RAM usage gone haywire, Daniel Kahn Gillmor, 2009/02/14
- Re: [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/14
- Re: [Sks-devel] SKS RAM usage gone haywire, Daniel Kahn Gillmor, 2009/02/14
- Re: [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/14
- Re: [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/14
- Re: [Sks-devel] SKS RAM usage gone haywire, Ari Trachtenberg, 2009/02/15
- Re: [Sks-devel] SKS RAM usage gone haywire, Yaron Minsky, 2009/02/15
- Re: [Sks-devel] SKS RAM usage gone haywire,
Phil Pennock <=
- Re: [Sks-devel] SKS RAM usage gone haywire, Ryan, 2009/02/15
- Re: [Sks-devel] SKS RAM usage gone haywire, Daniel Kahn Gillmor, 2009/02/15
- Re: [Sks-devel] SKS RAM usage gone haywire, John Marshall, 2009/02/15
- Re: [Sks-devel] SKS RAM usage gone haywire, Ryan Hunt, 2009/02/16
- Re: [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/16
- Re: [Sks-devel] SKS RAM usage gone haywire, Ryan Hunt, 2009/02/16
- Re: [Sks-devel] SKS RAM usage gone haywire, Phil Pennock, 2009/02/16
- Re: [Sks-devel] SKS RAM usage gone haywire, Ari Trachtenberg, 2009/02/17
- Re: [Sks-devel] SKS RAM usage gone haywire, Azamat S. Kalimoulline, 2009/02/17
- Re: [Sks-devel] SKS RAM usage gone haywire, Azamat S. Kalimoulline, 2009/02/17