NLS on MSVC strikes back!

I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
support is broke.
Could someone please confirm this.

My test:

Gevik=#
Gevik=# show lc_messages ;
lc_messages
----------------------------
English_United States.1252
(1 row)

Gevik=#
Gevik=# set lc_messages TO 'de_DE.utf8';
SET
Gevik=# show lc_messages ;
lc_messages
-------------
de_DE.utf8
(1 row)

Gevik=# select to_char(now(), 'TMDay, DD TMMonth YYYY');
to_char
---------------------------
Tuesday, 12 February 2008
(1 row)

Gevik=#
Gevik=# set lc_messages TO 'es_ES.utf8';
SET
Gevik=# show lc_messages ;
lc_messages
-------------
es_ES.utf8
(1 row)

Gevik=# select to_char(now(), 'TMDay, DD TMMonth YYYY');
to_char
---------------------------
Tuesday, 12 February 2008
(1 row)

Gevik Babakhani wrote:
> I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
> support is broke.
> Could someone please confirm this.
>
> My test:
>
>
> Gevik=#
> Gevik=# set lc_messages TO 'de_DE.utf8';
> SET
> Gevik=# show lc_messages ;
> lc_messages
> -------------
> de_DE.utf8
> (1 row)
>
> Gevik=# select to_char(now(), 'TMDay, DD TMMonth YYYY');
> to_char
> ---------------------------
> Tuesday, 12 February 2008
> (1 row)
>
>
>

Should that not be lc_time you are setting? lc_messages is for, uh,
messages.

cheers

andrew

> Should that not be lc_time you are setting? lc_messages is
> for, uh, messages.

No. The same thing works on 8.2.6

Gevik Babakhani wrote:
> > Should that not be lc_time you are setting? lc_messages is
> > for, uh, messages.
>
> No. The same thing works on 8.2.6

Well, that should be considered a bug, not a feature. Perhaps it was
fixed in 8.3.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

> Well, that should be considered a bug, not a feature.
> Perhaps it was fixed in 8.3.
>

This is 8.3 I am testing with.

Gevik Babakhani wrote:
> > Well, that should be considered a bug, not a feature.
> > Perhaps it was fixed in 8.3.
>
> This is 8.3 I am testing with.

Right, I know that. But I didn't see you trying lc_time as Andrew
suggested.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Gevik Babakhani wrote:
>> Well, that should be considered a bug, not a feature.
>> Perhaps it was fixed in 8.3.
>>
>>
>
> This is 8.3 I am testing with.
>
>
>

You are missing Alvaro's point. He is saying that the behaviour you
relied on in 8.2.6 was a bug, and possibly you can no longer rely on the
buggy behaviour in 8.3.

cheers

andrew

> Right, I know that. But I didn't see you trying lc_time as
> Andrew suggested.
>

Did that too, but no luck :(

Gevik Babakhani wrote:
> I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
> support is broke.
> Could someone please confirm this.

What I can confirm is that lc_messages is supposed to work for to_char,
because it shows the localized output for me as I change lc_messages;
and it doesn't if I change lc_time.

In any case,

> Gevik=# show lc_messages ;
> lc_messages
> ----------------------------
> English_United States.1252
> (1 row)
>
> Gevik=#
> Gevik=# set lc_messages TO 'de_DE.utf8';
> SET

I wonder if this is really valid. Shouldn't you be using a locale name
like 'German_Germany.65001' or some such? I thought Windows did not
recognize the de_DE.utf8 form of names. Perhaps it failed to raise an
error here? It does for me:

alvherre=# set lc_time to 'fr_CA.utf8';
SET
alvherre=# set lc_time to 'de_DE.utf8';
ERROR: valor no válido para el parámetro «lc_time»: «de_DE.utf8»

Suffice to say I don't have the de_DE locale installed.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> Gevik Babakhani wrote:
>
>> I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
>> support is broke.
>> Could someone please confirm this.
>>
>
> What I can confirm is that lc_messages is supposed to work for to_char,
> because it shows the localized output for me as I change lc_messages;
> and it doesn't if I change lc_time.
>
>

FSVO "supposed to".

The CVS history is somewhat murky.

On Nov 24 2006 this was committed dor src/backend/utils/adt/formatting.c:

r 1.114: Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.

A few hours later it was supposedly reverted: r 1.115: Revert (too late
in beta): Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.

Apparently this was never returned to (it should probably have been put
on the patches hold queue).

I can't see offhand what else might have caused the behaviour change.

cheers

andrew

Hi

--Document.--
Table 9-22.
TM prefix translation mode (print localized day and month names based on lc_messages)
--
http://winpg.jp/~saito/pg_work/NLS_TO_CHAR_JP.png
But, Although Japanese is out of condition....

Does the fundamental specification change?

Regards,
Hiroshi Saito

----- Original Message -----
From: "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>

> Gevik Babakhani wrote:
>> I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
>> support is broke.
>> Could someone please confirm this.
>
> What I can confirm is that lc_messages is supposed to work for to_char,
> because it shows the localized output for me as I change lc_messages;
> and it doesn't if I change lc_time.
>
> In any case,
>
>> Gevik=# show lc_messages ;
>> lc_messages
>> ----------------------------
>> English_United States.1252
>> (1 row)
>>
>> Gevik=#
>> Gevik=# set lc_messages TO 'de_DE.utf8';
>> SET
>
> I wonder if this is really valid. Shouldn't you be using a locale name
> like 'German_Germany.65001' or some such? I thought Windows did not
> recognize the de_DE.utf8 form of names. Perhaps it failed to raise an
> error here? It does for me:
>
> alvherre=# set lc_time to 'fr_CA.utf8';
> SET
> alvherre=# set lc_time to 'de_DE.utf8';
> ERROR: valor no válido para el parámetro «lc_time»: «de_DE.utf8»
>
> Suffice to say I don't have the de_DE locale installed.
>
> --
> Alvaro Herrera http://www.CommandPrompt.com/
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly

Andrew Dunstan wrote:
>
>
> Alvaro Herrera wrote:
> > Gevik Babakhani wrote:
> >
> >> I have compiled 8.3 (CVS HEAD) with NLS on a XP box. But it seems that NLS
> >> support is broke.
> >> Could someone please confirm this.
> >>
> >
> > What I can confirm is that lc_messages is supposed to work for to_char,
> > because it shows the localized output for me as I change lc_messages;
> > and it doesn't if I change lc_time.
> >
> >
>
> FSVO "supposed to".
>
> The CVS history is somewhat murky.
>
> On Nov 24 2006 this was committed dor src/backend/utils/adt/formatting.c:
>
> r 1.114: Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.
>
> A few hours later it was supposedly reverted: r 1.115: Revert (too late
> in beta): Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.
>
> Apparently this was never returned to (it should probably have been put
> on the patches hold queue).

Added to TODO list:

o Use LC_TIME for localized weekday/month names, rather than
LC_MESSAGES

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On Nov 24 2006 this was committed dor src/backend/utils/adt/formatting.c:

> r 1.114: Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.

> A few hours later it was supposedly reverted: r 1.115: Revert (too late
> in beta): Fix to_char() locale handling to honor LC_TIME, not LC_MESSAGES.

> Apparently this was never returned to (it should probably have been put
> on the patches hold queue).

to_char's month/day name localization is implemented with gettext() not
strftime(), which is why it depends on LC_MESSAGES not LC_TIME. I seem
to recall that we didn't like the side-effects of the patch you are
mentioning, and so it ended up being rejected outright.

regards, tom lane

Hiroshi Saito wrote:
> Hi
>
> --Document.--
> Table 9-22.
> TM prefix translation mode (print localized day and month names based on lc_messages)
> --
> http://winpg.jp/~saito/pg_work/NLS_TO_CHAR_JP.png
> But, Although Japanese is out of condition....
>
> Does the fundamental specification change?

Humm, I cannot read Japanese so as far as I can make, this is working fine?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

> to_char's month/day name localization is implemented with
> gettext() not strftime(), which is why it depends on
> LC_MESSAGES not LC_TIME. I seem to recall that we didn't
> like the side-effects of the patch you are mentioning, and so
> it ended up being rejected outright.

Correct. I have been looking for the cause of this problem and
these are my finding:

1. The platforms discussed and tested here are MS Windows XP and 2003 using
Microsoft VC++ 2005 and NOT MINGW.

2. As Tom described above, to_char is implemented with gettext which depends
on LC_MESSAGES. I guess this is okay.

3. Changing LC_MESSAGES is done in pg_locale.c:94:pg_perm_setlocale by
setting LC_MESSAGES environment variable. This works for MINGW (tested by
installing PG 8.2.6), but it does not work for MSVC++ due different locale
handling of gettext. Please see gettext sources:1087:localenames.c

4. Locale names are different in MS Windows. I created a C app to test
gettext on Windows.
setting LC_MESSAGES to Spanisg_Spain and German_Germany works but es_ES and
de_DE do not :(

SET LC_MESSAGES to '....' has no effect because:

A. gettext compiled/linked in MSVC looks for the locale of the current
thread and NOT the LC_MESSAGES,LANGIAGE,LANG... environment variables. See
gettext's sources....

B. Given current thread's locale Spanish_Spain, gettext fails to find a
directory called "Spanish_Spain" in share/locale. As result English names
are returned.

A possible solutions:

- Keep to_char/LC_MESSAGES handling and create a fix for B.
Because we do not want to change/maintain our own version of Gettext this
would also involve creating a different directory/name structure for
Windows. For example share\locale\de would be share\locale\German_Germany.

Any thoughts,

Regards,
Gevik Babakhani
------------------------------------------------
PostgreSQL NL http://www.postgresql.nl
TrueSoftware BV http://www.truesoftware.nl
------------------------------------------------

Gevik Babakhani wrote:

> 4. Locale names are different in MS Windows. I created a C app to test
> gettext on Windows.
> setting LC_MESSAGES to Spanisg_Spain and German_Germany works but es_ES and
> de_DE do not :(
>
> SET LC_MESSAGES to '....' has no effect because:
>
> A. gettext compiled/linked in MSVC looks for the locale of the current
> thread and NOT the LC_MESSAGES,LANGIAGE,LANG... environment variables. See
> gettext's sources....
>
> B. Given current thread's locale Spanish_Spain, gettext fails to find a
> directory called "Spanish_Spain" in share/locale. As result English names
> are returned.

What this really means is that locale support is completely broken in
the MSVC build, i.e. you cannot get localized strings at all (not just
to_char()). Is this correct? If so, this is a serious problem.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Feb 12, 2008 4:44 PM, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
> What this really means is that locale support is completely broken in
> the MSVC build, i.e. you cannot get localized strings at all (not just
> to_char()). Is this correct? If so, this is a serious problem.

Not judging by Hiroshi's screenshot. I don't speak Japanese either,
but it certainly didn't say 'tuesday' after he adjusted LC_MESSAGES.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com
The Oracle-compatible database company

Dave Page wrote:
> On Feb 12, 2008 4:44 PM, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
> > What this really means is that locale support is completely broken in
> > the MSVC build, i.e. you cannot get localized strings at all (not just
> > to_char()). Is this correct? If so, this is a serious problem.
>
> Not judging by Hiroshi's screenshot. I don't speak Japanese either,
> but it certainly didn't say 'tuesday' after he adjusted LC_MESSAGES.

But it has the string "mingw" in the path, so I suspect it's not MSVC ...
Hiroshi-san, can you confirm?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Gevik Babakhani wrote:
>
>> 4. Locale names are different in MS Windows. I created a C app to test
>> gettext on Windows.
>> setting LC_MESSAGES to Spanisg_Spain and German_Germany works but es_ES and
>> de_DE do not :(
>>
>> SET LC_MESSAGES to '....' has no effect because:
>>
>> A. gettext compiled/linked in MSVC looks for the locale of the current
>> thread and NOT the LC_MESSAGES,LANGIAGE,LANG... environment variables. See
>> gettext's sources....
>>
>> B. Given current thread's locale Spanish_Spain, gettext fails to find a
>> directory called "Spanish_Spain" in share/locale. As result English names
>> are returned.
>
> What this really means is that locale support is completely broken in
> the MSVC build, i.e. you cannot get localized strings at all (not just
> to_char()). Is this correct? If so, this is a serious problem.

I've certainly managed to get output in Swedish more than once...
Annoying as hell because it's the default if you build with NLS and has
your English language Windows configured with Swedish time/date format :)

//Magnus

> What this really means is that locale support is completely
> broken in the MSVC build, i.e. you cannot get localized
> strings at all (not just to_char()). Is this correct? If
> so, this is a serious problem.
>

The way one could confirm this is by:

1. rename share/locale/de to share/locale/German_Germany
2. start a dosbox
3. before starting PG, set an env variable SET LANGUAGE=German_Germany in
that dosbox, which will force all child threads to have that locale.
3. start PG form that dosbox and SET LC_MESSAGES to "es_ES" or
"Spanish_Spain" or "es_ES.utf8" or anything else...; select
to_char(now(),'TMDay TMMonth YYYY');
4. The result will be in German and not Spanish as one expects.

If handling locale in to_char will remain using gettext then I can start
working on a patch to fix this.

Regards,
Gevik Babakhani
------------------------------------------------
PostgreSQL NL http://www.postgresql.nl
TrueSoftware BV http://www.truesoftware.nl
------------------------------------------------

Hi Alvaro-san.

Yes, However, It is not in good condition at the reason a message catalog still is not enough.
Then, I have tried these adjustments with the problem of a locale. Still, it is not much time.
But, My condition is not good so that I'm may be influenza. :-(

Regards,
Hiroshi Saito

----- Original Message -----
From: "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>

> Hiroshi Saito wrote:
>> Hi
>>
>> --Document.--
>> Table 9-22.
>> TM prefix translation mode (print localized day and month names based on lc_messages)
>> --
>> http://winpg.jp/~saito/pg_work/NLS_TO_CHAR_JP.png
>> But, Although Japanese is out of condition....
>>
>> Does the fundamental specification change?
>
> Humm, I cannot read Japanese so as far as I can make, this is working fine?
>
> --
> Alvaro Herrera http://www.CommandPrompt.com/
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support

From: "Alvaro Herrera" <alvherre(at)commandprompt(dot)com>

> Dave Page wrote:
>> On Feb 12, 2008 4:44 PM, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
>> > What this really means is that locale support is completely broken in
>> > the MSVC build, i.e. you cannot get localized strings at all (not just
>> > to_char()). Is this correct? If so, this is a serious problem.
>>
>> Not judging by Hiroshi's screenshot. I don't speak Japanese either,
>> but it certainly didn't say 'tuesday' after he adjusted LC_MESSAGES.
>
> But it has the string "mingw" in the path, so I suspect it's not MSVC ...
> Hiroshi-san, can you confirm?

Yeah, However, I go to a hospital from now.... sorry.
I will check it, after returning.

Regards,
Hiroshi Saito