Re: Encoding and i18n

Reading the commit message about the TZ encoding issue I'm curious why this
isn't a more widespread problem. How does gettext now what encoding we want
messages in? How do we prevent things like to_char(now(),'month') from
producing strings in an encoding different from the database's encoding?

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Gregory Stark wrote:
>
> Reading the commit message about the TZ encoding issue I'm curious why this
> isn't a more widespread problem. How does gettext now what encoding we want
> messages in? How do we prevent things like to_char(now(),'month') from
> producing strings in an encoding different from the database's encoding?

The PO files include encoding information, so it's easy for the server
to recode them from that to the server (or client) encoding, as
appropriate.

Of course, then it is up to the translator to get it right ... but I
think when he doesn't, people notice fairly quickly.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

"Alvaro Herrera" <alvherre(at)commandprompt(dot)com> writes:

> Gregory Stark wrote:
>>
>> Reading the commit message about the TZ encoding issue I'm curious why this
>> isn't a more widespread problem. How does gettext now what encoding we want
>> messages in? How do we prevent things like to_char(now(),'month') from
>> producing strings in an encoding different from the database's encoding?
>
> The PO files include encoding information, so it's easy for the server
> to recode them from that to the server (or client) encoding, as
> appropriate.

So does the _() macro automatically recode it to the current server encoding?

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> Reading the commit message about the TZ encoding issue I'm curious why this
> isn't a more widespread problem. How does gettext now what encoding we want
> messages in? How do we prevent things like to_char(now(),'month') from
> producing strings in an encoding different from the database's encoding?

The short answer is it's all a house of cards, and if you troll
the archives you will find plenty of bug reports traceable to
misconfiguration in this area. The recent attempt to enforce
that nl_langinfo(CODESET) matches the database encoding is a first
step towards making this more bulletproof, but we're finding out
that even that is harder than it looks.

regards, tom lane

Gregory Stark wrote:
> "Alvaro Herrera" <alvherre(at)commandprompt(dot)com> writes:
>
> > Gregory Stark wrote:
> >>
> >> Reading the commit message about the TZ encoding issue I'm curious why this
> >> isn't a more widespread problem. How does gettext now what encoding we want
> >> messages in? How do we prevent things like to_char(now(),'month') from
> >> producing strings in an encoding different from the database's encoding?
> >
> > The PO files include encoding information, so it's easy for the server
> > to recode them from that to the server (or client) encoding, as
> > appropriate.
>
> So does the _() macro automatically recode it to the current server encoding?

Well, I'm not sure if it's _(), elog() or what, but it does get recoded.
If I have a different client_encoding and get a NOTICE, then both the
server and client get a message in the corresponding encoding.

In fact this is the reason for the most common "PANIC: stack overflow"
in elog.c error stack. When a message needs to be recoded but the
recoding procedure errors out, it wants to report that and this one also
fails, you get infinite recursion and nothing can get reported.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> So does the _() macro automatically recode it to the current server encoding?

From the gettext manual:

---

gettext not only looks up a translation in a message catalog. It also
converts the translation on the fly to the desired output character
set. This is useful if the user is working in a different character set
than the translator who created the message catalog, because it avoids
distributing variants of message catalogs which differ only in the
character set.

The output character set is, by default, the value of nl_langinfo
(CODESET), which depends on the LC_CTYPE part of the current locale. But
programs which store strings in a locale independent way (e.g. UTF-8)
can request that gettext and related functions return the translations
in that encoding, by use of the bind_textdomain_codeset function.

---

We don't currently call bind_textdomain_codeset, in part because of the
lack of portability of names for codesets.

regards, tom lane

"Alvaro Herrera" <alvherre(at)commandprompt(dot)com> writes:

> Gregory Stark wrote:
>
>> So does the _() macro automatically recode it to the current server encoding?
>
> Well, I'm not sure if it's _(), elog() or what, but it does get recoded.
> If I have a different client_encoding and get a NOTICE, then both the
> server and client get a message in the corresponding encoding.

Actually I was thinking about things like formatting.c which take localized
strings and return them as data which can end up in the database. If they're
in the wrong encoding then they'll be invalidly encoded strings in the
database.

> In fact this is the reason for the most common "PANIC: stack overflow"
> in elog.c error stack. When a message needs to be recoded but the
> recoding procedure errors out, it wants to report that and this one also
> fails, you get infinite recursion and nothing can get reported.

Ouch

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Gregory Stark wrote:
> "Alvaro Herrera" <alvherre(at)commandprompt(dot)com> writes:
>
> > Gregory Stark wrote:
> >
> >> So does the _() macro automatically recode it to the current server encoding?
> >
> > Well, I'm not sure if it's _(), elog() or what, but it does get recoded.
> > If I have a different client_encoding and get a NOTICE, then both the
> > server and client get a message in the corresponding encoding.
>
> Actually I was thinking about things like formatting.c which take localized
> strings and return them as data which can end up in the database. If they're
> in the wrong encoding then they'll be invalidly encoded strings in the
> database.

Oh, I didn't think of that. Let me see if I can get an invalid string
into the database that way.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
>> Actually I was thinking about things like formatting.c which take localized
>> strings and return them as data which can end up in the database. If they're
>> in the wrong encoding then they'll be invalidly encoded strings in the
>> database.
>>
>
> Oh, I didn't think of that. Let me see if I can get an invalid string
> into the database that way.
>
>

I was quite certain when we closed most of these holes recently that we
hadn't caught them all, so this wouldn't surprise me in the least.

cheers

andrew

Andrew Dunstan wrote:
>
>
> Alvaro Herrera wrote:
>>> Actually I was thinking about things like formatting.c which take
>>> localized
>>> strings and return them as data which can end up in the database. If
>>> they're
>>> in the wrong encoding then they'll be invalidly encoded strings in the
>>> database.
>>
>> Oh, I didn't think of that. Let me see if I can get an invalid string
>> into the database that way.
>
> I was quite certain when we closed most of these holes recently that we
> hadn't caught them all, so this wouldn't surprise me in the least.

It seems to work correctly:

alvherre=# drop table week;
DROP TABLE
alvherre=# create table week (a text);
CREATE TABLE
alvherre=# \encoding utf8
alvherre=# insert into week select to_char(now()-'3 days'::interval, 'tmday');
INSERT 0 1
alvherre=# \encoding latin1
alvherre=# insert into week select to_char(now()-'3 days'::interval, 'tmday');
INSERT 0 1
alvherre=# select * from week;
a
-----------
miércoles
miércoles
(2 lignes)

I tried on both a UTF8 and Latin1 terminal and it works OK in all cases.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> I tried on both a UTF8 and Latin1 terminal and it works OK in all cases.

The cases that would be interesting involve to_char's locale-specific
format codes (eg Dy) along with LC_TIME settings that are deliberately
incompatible with the database encoding. client_encoding is not relevant.

It's not real clear to me whether, on a Unix machine, there is even
supposed to be any difference between setting LC_TIME=es_ES.iso88591 and
setting it to es_ES.utf8. Since nl_langinfo(CODESET) is supposedly
determined only by LC_CTYPE, you could argue that strftime's results
should be in that encoding regardless, and that the codeset component of
other LC_ variables should be ignored. Some experimentation suggests
that at least in glibc it doesn't work that way, and that there is in
fact no principled way for you to find out what encoding strftime is
giving you :-(.

$ LANG=es_ES.utf8 date
sb oct 6 14:11:30 EDT 2007
$ LANG=es_ES.iso88591 date
sb oct 6 14:11:42 EDT 2007
$ LANG=en_US.iso88591 LC_TIME=es_ES.utf8 date
sb oct 6 14:12:10 EDT 2007
$ LC_CTYPE=en_US.iso88591 LC_TIME=es_ES.utf8 date
sb oct 6 14:12:34 EDT 2007

Perhaps a workable fix for this would be to try to mangle the LC_ settings
we pass to setlocale() so that they all have the same codeset component
(if any). It looks like the convention of ".foo" being a codeset name
is fairly well standardized, even if the spelling of the codeset name is
not ...

regards, tom lane

"Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you
> could argue that strftime's results should be in that encoding regardless,

It seems to me we aren't actually using strftime any more in any case. We seem
to be using things like _("Monday") instead. Except in my tests I don't get
any French dates even when the server is started in French mode. I think we
just don't have localizations for those strings yet.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
>> Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you
>> could argue that strftime's results should be in that encoding regardless,

> It seems to me we aren't actually using strftime any more in any case.

Sorry, I was using strftime as a generic standin for "everything that
LC_TIME affects". Trace the usage of backend/utils/adt/pg_locale.c
to see what's really at stake there.

The practical issues would likely be things like type money using a
currency symbol that's given in the wrong encoding.

And of course you did get the point that we already know a bogus
LC_MESSAGES setting leads directly to error-stack-overflow PANIC.

regards, tom lane

Gregory Stark wrote:

> It seems to me we aren't actually using strftime any more in any case. We seem
> to be using things like _("Monday") instead. Except in my tests I don't get
> any French dates even when the server is started in French mode. I think we
> just don't have localizations for those strings yet.
>
This was already discussed [1]. I proposed a patch (that was rejected)
because it calls setlocale() in every template pattern in to_char()
IIRC. I coded a patch to implement the setlocale() caching mechanism but
didn't send it. :( I'll take a look and this.

[1] http://archives.postgresql.org/pgsql-hackers/2006-11/msg00523.php

--
Euler Taveira de Oliveira
http://www.timbira.com/

Am Sonntag, 7. Oktober 2007 schrieb Gregory Stark:
> "Tom Lane" <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:
> > Since nl_langinfo(CODESET) is supposedly determined only by LC_CTYPE, you
> > could argue that strftime's results should be in that encoding
> > regardless,
>
> It seems to me we aren't actually using strftime any more in any case. We
> seem to be using things like _("Monday") instead.

I seem to recall that we don't use strftime *yet*, exactly because of this
sort of issue. This was discussed before the 8.2 release.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Am Samstag, 6. Oktober 2007 schrieb Tom Lane:
> It's not real clear to me whether, on a Unix machine, there is even
> supposed to be any difference between setting LC_TIME=es_ES.iso88591 and
> setting it to es_ES.utf8.  Since nl_langinfo(CODESET) is supposedly
> determined only by LC_CTYPE, you could argue that strftime's results
> should be in that encoding regardless, and that the codeset component of
> other LC_ variables should be ignored.  Some experimentation suggests
> that at least in glibc it doesn't work that way, and that there is in
> fact no principled way for you to find out what encoding strftime is
> giving you :-(.

It might be useful to research whether that behavior is following the spec
(POSIX or whatever).

--
Peter Eisentraut
http://developer.postgresql.org/~petere/