RE: Diff crash when comparing 2 big files with error message "diff.exe: memory exhausted"

[Kissinger Chen]

HI, Bob,

Thanks for your answer. 

Today I copy these big pure text file to another computer. This computer
is a 32-bit computer with windows server 2003 enterprise edition(sp1) 
And has 8G memory. 

Then I compare these files with diff.exe(2.8.7 version). But I get the
same 
result: For some big files, the diff work well and I can get the
differences, for this case, 1059M memory have been consumed by diff.exe
and 6614M memory are available.(I check the memory usage by widows task
manager)
 
For some other big files, the diff will fail with message:"diff.exe:
memory 
exhausted". for this case, 1007M memory have been consumed by diff.exe
and 6667M memory are free.

You can see now I have enough memory, and I remember that a process can
consume 2000M memory 
in 32-bit computer.

So maybe I encounter the very extreme cases. diff.exe will report error
even there is enough
memory. Has this issue been reported? And if convenient, would you
please tell me how can I 
handle this issue? thanks.

Thanks again.

Kissinger




-----Original Message-----
From: Bob Proulx [mailto:address@hidden 
Sent: Tuesday, October 14, 2008 02:49
To: Kissinger Chen
Cc: address@hidden
Subject: Re: Diff crash when comparing 2 big files with error message
"diff.exe: memory exhausted"

Kissinger Chen wrote:
> This is Kissinger, a Gnuwin32 user. Today I use diff to compare some 
> big files and try to retrieve the differences between files.
>  
> For some big files, the diff work well and I can get the differences.

Sounds good.

> But for some other big files, the diff will crash with such error 
> message:"diff.exe: memory exhausted". These files are more then 500 
> MB, pure text files.
> And I use the diff of 2.8.7, my OS is windows XP.

GNU diff is a program and requires a certain amount of memory to analyze
files.  Bigger files with complex differences may consume more memory
than smaller files.  When diff encounters an error it will report it.
Running out of memory is an error and is being reported.
Running out of memory in the case of processing large files is not a
program crash.  The program did not crash.  The program detected that it
was out of resources and reported this error condition.

> It seems there is an issue in the Gnuwin32 diff tool. Has this issue 
> been reported? I can not search this issue from help.

The error message you have received is stating "memory exhausted"
meaning that there is not enough virtual process memory for the program
to run to successful completion.  This is a physical limitation of your
platform.  You have run out of memory.

How much memory do you have on your system?

How much memory is diff currently consuming before running out?

> If convenient, can you tell me how to resolve this issue? Is there any

> another diff-like tool? Thanks.

Here are some suggestions:

1. If your 32-bit system has less than 4G of memory then adding more
   memory to your system may allow this program to complete.  32-bit
   systems have a maximum addressable memory of 4G so that is the
   upper limit possible.

2. Run this task on a different system that has more memory than your
   current machine.

3. Run this task on a 64-bit system with more than 4G of memory.  A
   64-bit operating system has much more capability to address very
   large amounts of physical memory.

Note that in extreme cases diff may consume very large amounts of memory
and still may exhaust all available memory.  You may need to change the
problem statement in order to reduce the system resources needed to
accomplish what you want to accomplish.  You may need to reorganize what
you are trying to do in order to make the problem tractable within
current system memory limitations.

Bob


This email was sent to you by Thomson Reuters, the global news and information 
company.
Any views expressed in this message are those of the individual sender, except 
where the sender specifically states them to be the views of Thomson Reuters.