Re: Memory Leak

[Sukanta Basu]

I have run valgrind with both the serial and the parallel versions of
my code. I am not able to interpret the results (I have never used
valgrind before).

For serial code, I used: valgrind --leak-check=yes
--log-file=ValgrindSerial.out octave < MLp_Les &
The output is available at:
ftp://152.1.40.22/array1/MISC/ValgrindSerial.out (size: 1.3 MB)

For parallel code, I used: valgrind --leak-check=yes
--log-file=ValgrindParallel.out mpirun -np 2 octave -q --eval MLp_Les
&
The output is available at:
ftp://152.1.40.22/array1/MISC/ValgrindParallel.out (size: 37.9 KB)

I will be grateful if someone could interpret the results.

Thanks,
Sukanta


On Mon, Jan 21, 2013 at 10:14 AM, Sukanta Basu <address@hidden> wrote:
>
> Hi All,
>
> I would also like to answer James' question: "Is there someway to stop and 
> save the current state at a certain iteration, then restart the simulation at 
> that point?"
>
> I have a restart option in the code and I am using it frequently these days. 
> When a simulation crashes due to the lack of memory, I have to restart it. At 
> the beginning, the memory consumption is almost the same (statistically) as a 
> new run. Then, it grows again until it reaches the limit.
>
> Sukanta
>
>
> On Mon, Jan 21, 2013 at 10:04 AM, Sukanta Basu <address@hidden> wrote:
>>
>> Dear All,
>>
>> Thanks for your response and suggestions. I would like to give you a bit 
>> more background about this code and the errors I am getting.
>>
>> 1. Originally, I wrote this code in Matlab ~8 years ago. I have used this 
>> code extensively (for publications) and never detected any memory leak in 
>> Matlab. I am not an "expert" in octave/matlab, but I follow all the "basics" 
>> (e.g., pre-allocation of matrices). In other words, none of the matrices are 
>> growing in size through iteration.
>>
>> 2. About 6 years ago, I used MatlabMPI to parallelize the code. Again, no 
>> problem with memory. The code was not very efficient because of latency of 
>> MatlabMPI.
>>
>> 3. I created a serial version of the code in Octave ~2 months ago. At the 
>> beginning, I did not notice any memory issues. After posting the memory 
>> issue, I started looking into it more carefully. For example, I started a 
>> simulation last night:
>> 2:15 pm (Jan 20): 1.47 GB (swap: 0.192 GB)
>> 8:15 pm (Jan 20): 1.53 GB (swap: 0.192 GB)
>> 9:50 am (Jan 21): 2.21 GB (swap: 0.192 GB)
>> So, the serial code does have "small" amount of memory leak.
>>
>> 4. I created a parallel version of the code in Octave with openmpi_ext 
>> package ~1 month ago. The parallelization is much faster than MatlabMPI (as 
>> expected). However, the memory leak using this code is severe. The same job 
>> (described in item #3) will cross 24 GB within the same time-period.
>>
>> From these experiments, I think the leak is coming from openmpi_ext package. 
>> I would like to note that I have experience working with large-scale MPI 
>> codes; so, it is unlikely due to any faulty implementation of MPI in my code.
>>
>> I do not have any experience with valgrind or similar codes. However, I will 
>> be happy to email anyone my code. Unfortunately, it is a rather bulky code 
>> with several subroutines. It will be difficult for me to trim down this code 
>> to reproduce the same error.
>>
>> If anyone could look into the openmpi_ext package that would be great! I am 
>> using v1.1.0 (the latest).
>>
>> Best regards,
>> Sukanta
>>
>>
>> On Mon, Jan 21, 2013 at 8:42 AM, Jordi Gutiérrez Hermoso <address@hidden> 
>> wrote:
>>>
>>> On 19 January 2013 22:46, Sukanta Basu <address@hidden> wrote:
>>> > I am using Octave 3.6.2 on Ubuntu 12.10. I have written a code for
>>> > solving certain fluid dynamics problem (~5000 lines) using Octave
>>> > and the openmpi_ext toolbox. I am noticing a strange problem with
>>> > Octave. The memory usage slowly increases with each iteration.
>>>
>>> If there is indeed a memory leak, it can be in several possible
>>> locations.
>>>
>>> It can be in the openmpi_ext package (due to trademark issues with The
>>> Mathworks, we don't generally refer to Octave packages as
>>> "toolboxes"), which would be a bug in this package.
>>>
>>> It could be in in Open MPI. In that case, we should forward the bug to
>>> the OpenM PI maintainers.
>>>
>>> It can be intrinsic in your m-file code, if you're inadvertently
>>> growing an array without bound.
>>>
>>> It can be in Octave itself, which would be a bug in Octave.
>>>
>>> At any rate, this shouldn't happen, and you have given no usable
>>> diagnostic information to find the memory leak. If you are unfamiliar
>>> with hunting memory leaks (using valgrind and such), then try to post
>>> your code which exhibits the memory leak so that other people can hunt
>>> it down. Optimally, you should trim down your code to a working sample
>>> that exhibits the problem.
>>>
>>> - Jordi G. H.
>>
>>
>>
>>
>> --
>> Sukanta Basu
>> Associate Professor
>> North Carolina State University
>> http://www4.ncsu.edu/~sbasu5/
>
>
>
>
> --
> Sukanta Basu
> Associate Professor
> North Carolina State University
> http://www4.ncsu.edu/~sbasu5/




--
Sukanta Basu
Associate Professor
North Carolina State University
http://www4.ncsu.edu/~sbasu5/