RE: gsort problem (TAKE 2)

[Anthony Tiemens]

Hi Paul,

you were spot on, there was a huge record (>3000bytes) in the file.  Appears 
the file was copied from another server and was corrupted even though they had 
the same filesize.  After removing the huge record, it was sorted ok.  Also 
recopied original file, did a chsum first to makesure it was ok, sort ran ok.

thanks for your help,

Anthony

-----Original Message-----
From: Paul Eggert [mailto:address@hidden
Sent: Friday, 9 June 2006 3:49 PM
To: Anthony Tiemens
Cc: address@hidden
Subject: Re: gsort problem (TAKE 2)


"Anthony Tiemens" <address@hidden> writes:

> debug output of binaary created with "-g" compile option.
>
> /CML/medw/tmp/orasupptest_gsort> gdb ./gsort-HPUX-5.94-6.7.06
> HP gdb 3.1 for PA-RISC 1.1 or 2.0 (narrow), HP-UX 11.00.
> Copyright 1986 - 2001 Free Software Foundation, Inc.
> Hewlett-Packard Wildebeest 3.1 (based on GDB) is covered by the
> GNU General Public License. Type "show copying" to see the conditions to
> change it and/or distribute copies. Type "show warranty" for warranty/support.
> ..
> (gdb) b xalloc_die
> Breakpoint 1 at 0x125b8: file xalloc-die.c, line 38.
> (gdb) r -T . -k 5n,5n -t ";" bigfile.txt -o bigfilesorted.txt
> Starting program: /CML/medw/tmp/orasupptest_gsort/./gsort-HPUX-5.94-6.7.06 -T 
> . -k 5n,5n -t ";" bigfile.txt -o bigfilesorted.txt
> ls
>
> Breakpoint 1, xalloc_die () at xalloc-die.c:38
> 38      xalloc-die.c: No such file or directory.
> (gdb) ls
> Undefined command: "ls".  Try "help".
> (gdb) where
> #0  xalloc_die () at xalloc-die.c:38
> #1  0x19244 in xnrealloc_inline (p=0x0, n=5728, s=1) at xmalloc.c:75
> #2  0x19298 in xrealloc (p=0x77e7ec90, n=5728) at xmalloc.c:91

Thanks.  It appears that realloc(0x77e7ec90, 5728) failed (i.e.,
returned NULL).  So realloc thought that memory was exhausted even
though 'sort' was asking for only 5728 bytes, which is a small number
these days.  That is a bit surprising.

Are you using the standard memory allocation routines
malloc/realloc/etc., or some unusual library?  If you are using the
standard library, can you try rebuilding it with (say) mpatrol
<http://directory.fsf.org/all/mpatrol.html> and see whether that
isolates the problem?  My understanding is that you need not recompile
'sort' to use mpatrol.  (mpatrol is not as good as valgrind, but it is
better than nothing.)

> (gdb) print merge_buffer_size
> $1 = 185835729

Wow.  That is a huge number.  It indicates that there is at least one
line of text in your file that is 185 MB long.  Or, that there is a
bug in 'sort' or in your compiler or debugger or whatnot.

Can you check all the lines in your file again?  Something like this
should do it:

gawk '{if (maxlen < length) maxlen = length} END { print maxlen}' bigfile.txt

This email and any attachments may contain privileged and confidential 
information and are intended for the named addressee only. If you have received 
this e-mail in error, please notify the sender and delete this e-mail 
immediately. Any confidentiality, privilege or copyright is not waived or lost 
because this e-mail has been sent to you in error. It is your responsibility to 
check this e-mail and any attachments for viruses.  No warranty is made that 
this material is free from computer virus or any other defect or error.  Any 
loss/damage incurred by using this material is not the sender's responsibility. 
 The sender's entire liability will be limited to resupplying the material.