Re: [Bug-gnupod] Bug in Pearlscript for 0.98.3

[H. Langos]

Hi Achim,

On Thu, Jul 24, 2008 at 11:19:17PM +0200, Achim Flammenkamp wrote:
> Hi
> 
> If have downloaded a certain mp3 file form the Internet.
> "Terry Jacks' Season in the Sun"
> It has illegal charcters in the mp3-id-tags for artist, album, title & 
> comment.
> With mp3info I can set correct "looking" ones.
> But when adding this mp3-file with gnupod_addsong to my mounted ipod-shuffle,
> then the next add of a correct mp3-song gives an error.
> Also I can not make a new ITuneDB getting also an error (without adding a 
> further mp3 file).
> 
> The reason seems that in /mnt/ipod/iPod_Control/.gnupod/GNUtunesDB.xml
> the Terry Jacks-song added an xml-enty with illegal characters.
> Do you want to have this misterious mp3-file which caused the script to fail?

I've got a whole bunch of mp3files that cause gnupod to fail. But if
there are no copyright problems you can send it to my email address.
One more for my test pool. Would be nice to see where exaclty on those
tags gnupod breaks.

gnupod currently has some serious limitations regarding the handling 
of illegal unicode sequences. Once they are in there, gnupod is unable 
to read its own xml file. 

I worked with Adrian on this problem but he has very little time these 
days, so don't hold your breath waiting for the next release to fix it. 


> Errormessage:
> 
> not well-formed (invalid token) at line 90, column 140, byte 31402 at 
> /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux/XML/Parser.pm line 187
> You see I use Perl-5.8.8 on an x86-64 Linux-(2.6.23 Kernel) running 
> Gentoo-Distri.
> 
> 
> I wonder why I can not remove these illegal values with mp3info ?
> They seem to be invisible for mp3info :-/

"mp3info" will only give you the old and limited version 1 id3 tags. 

Those id3v1 tags have a static size, only ascii (or iso-8859-1) text,
and they are located at the end of mp3 files. 

"id3v2.x" tags on the other hand are located at the start of a file, 
their size is not limited and they can contain not only text in different
utf encodings but also binary data like album art in jpg or png format.

If you want to see the v2 tags you'll have to install the "id3v2" tool 
and run "id3v2 -l filename". See http://www.id3.org/Home for details on
the whole id3 mess.


>From what you've posted I'd say the BOMs (Byte order markers) in front 
of the utf8 strings could be the culprit. I will not comment on the whole 
UTF8 BOM issue. You can read here about it:
http://blogs.msdn.com/michkap/archive/2005/01/20/357028.aspx 

id3v2 tags already contain one byte of information at the start of each 
string to signal what kind of text encoding is used. Still most 
UTF16-encoded tags include a BOM. But with UTF16 it is quite common to 
see BOMs as there are two ways of ordering your bytes. With utf8 there 
is even less reason for a BOM but gnupod should be able to handle them
anyway.

More to the point, gnupod should instruct libmp3 (MP3::Info) to return 
the tags in a parsed and uniformly encoded way instead of mucking around
with raw tags. 

Only if libmp3 fails to strip those BOMs, then gnupod should get rid of
them.

cheers
-henrik