Re: Diff crash when comparing 2 big files with error message "diff.exe: memory exhausted"

[Bob Proulx]

Kissinger Chen wrote:
> Today I copy these big pure text file to another computer. This computer
> is a 32-bit computer with windows server 2003 enterprise edition(sp1) 
> And has 8G memory. 

Even though the system has 8G of memory as a 32-bit program an
individual program cannot address more than 4G of it.  Usually there
is a limitation below that value.  Typical GNU/Linux based programs
can access up to 3G of memory on a 32-bit system.  I do not know what
limitations your MS system imposes as I do not use such systems.

> Then I compare these files with diff.exe(2.8.7 version). But I get
> the same result: For some big files, the diff work well and I can
> get the differences, for this case, 1059M memory have been consumed
> by diff.exe and 6614M memory are available.(I check the memory usage
> by widows task manager)

Running out of memory at 1059M seems suspiciously like your program is
limited to 1G of memory.  This is probably a limitation is the way the
program was built on your MS operating system.  You are probably not
able to use the rest of the memory even though it is available.

> For some other big files, the diff will fail with message:"diff.exe:
> memory exhausted". for this case, 1007M memory have been consumed by
> diff.exe and 6667M memory are free.

This appears to be a limition to 1G in the way the program was built
for your MS operating system.  Since I do not use MS I cannot help you
further with this problem.

> You can see now I have enough memory, and I remember that a process can
> consume 2000M memory 
> in 32-bit computer.

Typically on 32-bit GNU/Linux systems programs can access up to 3G of
memory.

> So maybe I encounter the very extreme cases. diff.exe will report error
> even there is enough
> memory.

But diff hasn't been able to allocate that memory from the system.
Your diff.exe program has failed to allocate more memory than 1G in
both of the cases you showed.  It doesn't matter if the memory is in
the system if the system does not give it to the program to use.

I do not use MS operating systems and so cannot suggest what you would
need to do to produce a program that could make use of the memory
there.  You might ask your question on one of the MS mailing lists
such as the Cygwin mailing lists.  They might have an answer for you
specific to your operating system.  Here we are using the GNU
operating system and it does not have this restriction.

GNU diff needs enough memory to read both files into memory before
the diff.  Because both of the files you have are 500M that exceeds
the 1G that your system is allowing you to use for diff.

> Has this issue been reported? And if convenient, would you please
> tell me how can I handle this issue? thanks.

I searched the web and found several different reports of people
having issues with diff running out of memory.  Some suggested using
the rdiff program from the rsync suite.  That seems to be more
efficient but has a completely different format and so if you need a
normal diff format isn't suitable.  I didn't see any other
suggestions.

The brute force solution would be to move to a 64-bit operating system
with more memory.  Otherwise you will simply need to reduce the size
of the files so as to reduce the amount of required memory.

Bob