Re: setlocale() [Was: Re: NSNumberFormater test fails]

[Eric Wasylishen]

Btw, I attached a test program test.m which shows how the current AppKit locale 
(and libc locale) affects various ways of printing decimal points:

Mac OS 7.2, locale set to French Canadian in SystemPreferences:
new-host-2:~ ericw$ echo $LANG
fr_CA.UTF-8
new-host-2:~ ericw$ gcc test.m -framework Foundation -framework AppKit
new-host-2:~ ericw$ ./a.out 
2012-01-29 17:52:58.642 a.out[1215:707] Launched. current locale: fr_CA
2012-01-29 17:52:58.643 a.out[1215:707] NSLog Decimal test: 1.23
printf decimal test: 1.23
2012-01-29 17:52:58.644 a.out[1215:707] Calling setlocale(LC_ALL, "")...
2012-01-29 17:52:58.645 a.out[1215:707] NSLog bonjour! 1.23
printf bonjour! 1,23
2012-01-29 17:52:58.645 a.out[1215:707] -[NSString stringWithFormat:]: 1.23
^C

GNUstep trunk:
address@hidden:~$ export LC_ALL=fr_CA.UTF-8
address@hidden:~$ gcc `gnustep-config --objc-flags` test.m `gnustep-config 
--gui-libs` -o test
address@hidden:~$ ./test
2012-02-10 16:16:32.203 test[14990] Launched. current locale: fr_CA
2012-02-10 16:16:32.210 test[14990] NSLog Decimal test: 1.23
printf decimal test: 1,23
2012-02-10 16:16:32.211 test[14990] Calling setlocale(LC_ALL, "")...
2012-02-10 16:16:32.211 test[14990] NSLog bonjour! 1.23
printf bonjour! 1,23
2012-02-10 16:16:32.211 test[14990] -[NSString stringWithFormat:]: 1.23

The only difference is the first "printf decimal test:" on GNUstep uses the 
comma decimal separator, because of the setlocale called by +[NSObject 
initialize]. On Mac OS, the libc locale is still "C".

One interesting thing is neither NSLog nor -[NSString stringWithFormat:], on 
Cocoa or GNUstep, use the locale's decimal point (regardless of the setting of 
the AppKit locale, or the libc locale.)


On 2012-02-08, at 12:29 PM, Fred Kiefer wrote:

> I think you are right as far as ICU is concerned, when we use ICU we should 
> use the function uloc_setDefault() to select the locale we want. But 
> currently the code we have is not ICU only, it works with a mixture of ICU 
> and glibc and it should work without ICU. We could try to make sure that all 
> our calls that need locale information of any sort, go through a wrapper that 
> uses the corresponding ICU function when that is available. If that is 
> achieved we could only use uloc_setDefault(), when ICU gets used and 
> everything should work. (And fall back to the old setlocale() call in the 
> other case.
> But is this achievable? Who is willing to check which of our used glibc 
> functions use any locale information? And to rewrite all of these? Just think 
> of the work in NSLog and the removal of all printf calls and the like.

Actually, I don't think there would be much work to do. From what I've seen, 
gnustep-base doesn't use the libc locale system much (if at all). For example, 
GSFormat.m uses NSLocale to get the decimal separator character (it does have 
fallback code to print a decimal using printf, but from what I can see that 
will never get used.)  

The only place I've found the libc locale used is in GSLocale.m, in the 
implementation of GSDomainFromDefaultLocale().

I attached a "first try" at a patch which does the following:

- deprecates GSSetLocale and GSSetLocaleC - they now do nothing.
- removes the call to GSSetLocaleC in +[NSObject initialize].
- adds a function to GSLocale.m, GSDefaultLanguageLocale(), which returns the 
locale for LC_MESSAGES. refactors two parts of NSUserDefaults that were calling 
GSSetLocale to get LC_MESSAGES to use GSDefaultLanguageLocale() instead.
- rewrite parts of GSLocale.m which need to use the libc locale. Now they call 
setlocale(LC_ALL, ""), and afterwards restore the C locale to what it was 
previously.

Of course if we were to apply this I would want to do a more careful scan of 
base for uses of the libc locale.

> I am not that sure whether the selection of the locale is really up to the 
> application code.
> When an application gets started it has a right to expect that its supporting 
> libraries are set up to a sensible default. This may not be true for tools, 
> but should be the case for applications that display a user interface.

I agree with your basic point… but I would expect the GNUstep locale system to 
be set up, not the libc locale.

Eric

locale3.diff
Description: Binary data

test.m
Description: Binary data

> On 08.02.2012 19:25, Eric Wasylishen wrote:
>> Hi,
>> I just had a look in to this problem. While it sounds like there is 
>> certainly a bug in libicu - it should not break if the libc locale is 
>> changed - I am very skeptical that setting the libc locale as we do in 
>> +[NSObject initialize] (or anywhere else... IIRC it's also done in 
>> NSUserDefaults) is a good idea.
>> 
>> Just to recap, +[NSObject initialize] does setlocale(LC_ALL, ""); which 
>> reads the current locale from the LANG environment variable (and others)[1] 
>> and sets all of the libc locale settings to that locale - so after 
>> +[NSObject initiazlize], printf("%g", 1.23) will output "1,23" if your 
>> system locale is French, for example.
>> 
>> My main problem with this is, I don't think any shared library really has 
>> the right to change this setting… if an application/tool wanted to switch 
>> from the default C locale to the current system locale, that should be the 
>> application's decision, since it has global consequences for everything 
>> running in that process (changing the semantics of printf!). But there would 
>> hardly be a point to doing that because GNUstep provides more powerful 
>> formatting anyway (NSNumberFormatter, etc.)
>> 
>>>  The "official" way of setting the locale in ICU is using uloc_setDefault()
>> 
>> According to the ICU docs, the notion of locale in ICU is totally 
>> independent of libc's. For number formatting, the ICU default locale only 
>> has an effect if you pass NULL for the locale when calling unum_open.
>> 
>> So setting the libc locale should have no effect on ICU's default locale 
>> (not true because of the bug mentioned below), and vice-versa - setting the 
>> ICU locale has no effect on the system locale.
>> 
>> Eric
>> 
>> [1] actually more complicated, at least for glibc: 
>> http://www.gnu.org/software/libc/manual/html_mono/libc.html#Locale-Categories
>> 
>> On 2012-01-23, at 6:43 AM, Stefan Bidi wrote:
>> 
>>> On Mon, Jan 23, 2012 at 3:01 AM, Fred Kiefer<address@hidden>  wrote:
>>> That bug description is not accurate. When running the NSNumberformatter 
>>> test program we only call setlocale() twice, once with "" and once with 
>>> NULL as the locale. That would be supported behaviour according to the bug 
>>> description, but clearly it is not.
>>> 
>>> I'll attach that modified version to the bug report.
>>> 
>>> I changed your test program to call setlocale() and now it also reports NaN 
>>> (See attachment). This really makes me wonder whether it is such a great 
>>> idea to use an internationalisation library that only supports English :-(
>>> 
>>> In all fairness, it has been classified as bug in the library.  The 
>>> "official" way of setting the locale in ICU is using uloc_setDefault().  To 
>>> get the locale it's uloc_getDefault().  But I see your point and am a 
>>> little surprised that this is even an issue.  I'm even more surprised that 
>>> it keeps getting pushed off to a later release.  It seems to have 
>>> originally been scheduled for 4.6, then 4.8 and now 5.0.
>>> 
>>> BTW: Is there a reason why the macro STRING_FROM_NUMBER calls the 
>>> conversion twice, even when it was successful on the first attempt? I don't 
>>> like the use of macros that much it really makes it hard to tell what is 
>>> going on and code in macros never gets as much review as normal code.
>>> 
>>> No, it's wrong.  I saw that when mucking around in there, too, but didn't a 
>>> fix commit (I'll do so when I get home, today).  Don't ask me how I managed 
>>> to screw that up... I don't know either.
>>> 
>>> On 21.01.2012 23:00, Stefan Bidi wrote:
>>> After running a few more tests and still not understanding what is going on
>>> I went to good and found this bug report:
>>> http://bugs.icu-project.org/trac/ticket/8214
>>> 
>>> Seems that ICU does not like it when we use setlocale().
>>> 
>>> On Sat, Jan 21, 2012 at 1:53 PM, Stefan Bidi<address@hidden>   wrote:
>>> 
>>> I am completely baffled by this bug.  I've been trying to debug this for
>>> the last 3 hrs and have gotten absolutely no where.  I added a unum_open
>>> and unum_formatDouble call in -init and I still get NaN when
>>> LANG=de_DE.UTF-8.  The test program continues to work without a hitch,
>>> though.  Something about how we handle the NSNumberFormatterInternal
>>> structure is screwing up UNumberFormat (I also added a unum_open and
>>> unum_formatDouble call in basic10_4.m and it worked fine).
>>> 
>>> 
>>> On Sat, Jan 21, 2012 at 11:00 AM, Stefan Bidi<address@hidden>wrote:
>>> 
>>> On Sat, Jan 21, 2012 at 10:41 AM, Fred Kiefer<address@hidden>   wrote:
>>> 
>>> Your test code works fine here and results in 1.234 as expected. My
>>> $LANG is de_DE.UTF-8. And with $LANG set to C the test run fine.
>>> Strange enough currentLocale ends up being en_US_POSIX
>>> 
>>> 
>>> -currentLocale looks for a Locale default, if it exists that's what it
>>> uses.  Do you have that set?
>>> 
>>> This still wouldn't explain why UNumberFormat is returning NaN.  Both you
>>> and Philippe have a valid locale.  On the plus side, if I set
>>> LANG=de_DE.UTF-8 I can reproduce this.  I'll go try to figure out what's
>>> going on.
> 
> 
> _______________________________________________
> Gnustep-dev mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnustep-dev