Re: Locale + encoding combinations

I'm working on some code for pgInstaller that will check the locale and
encoding selected by the user are a valid combination.

The changes recently added to initdb (which highlighted the UTF-8 issue
on Windows that Tom posted about) appear to only allow the default
encoding for the locale to be selected. For example, for me that would be:

"English_United Kingdom.1252"

However, setlocale() will also accept other valid combinations on
Windows, which initdb will not, for example:

"English_United Kingdom.28591" (Latin1)

Is there any reason not to accept other combinations that setlocale() is
happy with?

Regards, Dave

Dave Page wrote:
> Is there any reason not to accept other combinations that setlocale()
> is happy with?

setlocale() sets the locale. How does it "accept" a "combination"?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Dave Page wrote:
>> Is there any reason not to accept other combinations that setlocale()
>> is happy with?
>
> setlocale() sets the locale. How does it "accept" a "combination"?
>

setlocale(LC_CTYPE, "English_United Kingdom.65001")

will return null (and not change anything) because it doesn't like the
combination of the locale and that encoding (UTF-8).

setlocale(LC_CTYPE, "English_United Kingdom.1252")

will return "English_United Kingdom.1252" and set the locale accordingly
because WIN1252 is a valid encoding for that locale. Similarly, LATIN1
and numerous other encodings are accepted in combination with that locale.

Should initdb allow any combination that setlocale() accepts, or should
it *only* accept the default encoding for the specified locale?

/D

Dave Page wrote:
> setlocale(LC_CTYPE, "English_United Kingdom.65001")
>
> will return null (and not change anything) because it doesn't like
> the combination of the locale and that encoding (UTF-8).

The reason that that call fails is probably that the operating system
does not provide such a locale. But that's not what we are interested
in. We are interested in compatibility between *existing* operating
system locales and *PostgreSQL* encoding names.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Dave Page wrote:
>> setlocale(LC_CTYPE, "English_United Kingdom.65001")
>>
>> will return null (and not change anything) because it doesn't like
>> the combination of the locale and that encoding (UTF-8).
>
> The reason that that call fails is probably that the operating system
> does not provide such a locale.

It doesn't - UTF-8/65001 is a pseudo codepage on Windows with no NLS
file defining collation rules etc. as we already discussed.

> But that's not what we are interested
> in. We are interested in compatibility between *existing* operating
> system locales and *PostgreSQL* encoding names.

Yes.

Let me put my question another way.

Latin1 is a perfectly valid encoding for my locale English_United
Kingdom. It is accepted by setlocale for LC_ALL.

Why does initdb reject it? Why does it insist the encoding is not valid
for the locale?

/D

Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
> Latin1 is a perfectly valid encoding for my locale English_United
> Kingdom. It is accepted by setlocale for LC_ALL.
>
> Why does initdb reject it? Why does it insist the encoding is not valid
> for the locale?

Because initdb works with a finite list of known matches, and your particular
combination might not be in that list -- yet.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
>> Latin1 is a perfectly valid encoding for my locale English_United
>> Kingdom. It is accepted by setlocale for LC_ALL.
>>
>> Why does initdb reject it? Why does it insist the encoding is not valid
>> for the locale?
>
> Because initdb works with a finite list of known matches, and your particular
> combination might not be in that list -- yet.

So is it just a case of us generating a list of matches that may be
Windows specific, or is there more to it than that?

/D

Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
> So is it just a case of us generating a list of matches that may be
> Windows specific, or is there more to it than that?

You want to peruse src/port/chklocale.c. There is already explicit Windows
support in there, so maybe you just need to add on your particular cases.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
>> So is it just a case of us generating a list of matches that may be
>> Windows specific, or is there more to it than that?
>
> You want to peruse src/port/chklocale.c. There is already explicit Windows
> support in there, so maybe you just need to add on your particular cases.

Yup, found that - thanks. I'll look at updating that list.

/D

Dave Page wrote:
> Peter Eisentraut wrote:
>> Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
>>> So is it just a case of us generating a list of matches that may be
>>> Windows specific, or is there more to it than that?
>> You want to peruse src/port/chklocale.c. There is already explicit Windows
>> support in there, so maybe you just need to add on your particular cases.
>
> Yup, found that - thanks. I'll look at updating that list.

OK so I added the appropriate entries (and posted the patch to
-patches), but my original question remains: why can I only select the
*default* encoding for the chosen locale, but not other ones that are
also be valid according to setlocale? Is this a bug, or is there some
technical reason?

/D

Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
> my original question remains: why can I only select the
> *default* encoding for the chosen locale, but not other ones that are
> also be valid according to setlocale? Is this a bug, or is there some
> technical reason?

One locale works only with one encoding. There are no "default" or perhaps
alternative encodings for one locale; there is only one. The whole point of
the exercise is to determine what the spelling of that one encoding is in
PostgreSQL.

Perhaps you are confused about the naming. These are all entirely separate
locales:

en_GB.iso88591
en_GB.iso885915
en_GB.utf8

Someone was friendly enough to include the name of the encoding used by the
locale into its name, but that doesn't mean that en_GB has three alternative
encodings or something.

At least that's the model we have on POSIX platforms.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Dave Page <dpage(at)postgresql(dot)org> writes:
> However, setlocale() will also accept other valid combinations on
> Windows, which initdb will not, for example:
> "English_United Kingdom.28591" (Latin1)
> Is there any reason not to accept other combinations that setlocale() is
> happy with?

Are you certain that that acceptance actually represents support?
Have you checked that it rejects combinations involving real code
pages (ie, NOT 65001) that don't really work with the locale?

regards, tom lane

Peter Eisentraut wrote:
> Am Mittwoch, 10. Oktober 2007 schrieb Dave Page:
>> my original question remains: why can I only select the
>> *default* encoding for the chosen locale, but not other ones that are
>> also be valid according to setlocale? Is this a bug, or is there some
>> technical reason?
>
> One locale works only with one encoding. There are no "default" or perhaps
> alternative encodings for one locale; there is only one. The whole point of
> the exercise is to determine what the spelling of that one encoding is in
> PostgreSQL.
>
> Perhaps you are confused about the naming. These are all entirely separate
> locales:
>
> en_GB.iso88591
> en_GB.iso885915
> en_GB.utf8
>
> Someone was friendly enough to include the name of the encoding used by the
> locale into its name, but that doesn't mean that en_GB has three alternative
> encodings or something.
>
> At least that's the model we have on POSIX platforms.

OK, sorting out my terminology deficiencies has helped - thanks. The
problem seems to be:

initdb --locale "English_United Kingdom.28591"

works, but

initdb -E LATIN1 --locale "English_United Kingdom"

does not. That's good (albeit inconsistent), I know how to fix
pgInstaller now. What isn't so good is:

============
C:\pg>bin\initdb --locale "English_United Kingdom.99999" -D data
initdb: invalid locale name "English_United Kingdom.99999"
initdb: invalid locale name "English_United Kingdom.99999"
initdb: invalid locale name "English_United Kingdom.99999"
initdb: invalid locale name "English_United Kingdom.99999"
initdb: invalid locale name "English_United Kingdom.99999"
initdb: invalid locale name "English_United Kingdom.99999"
The files belonging to this database system will be owned by user "Dave".
This user must also own the server process.

The database cluster will be initialized with locale English_United
Kingdom.1252
.
The default database encoding has accordingly been set to WIN1252.
===========

Shouldn't that have failed?

Regards, Dave

Tom Lane wrote:
> Dave Page <dpage(at)postgresql(dot)org> writes:
>> However, setlocale() will also accept other valid combinations on
>> Windows, which initdb will not, for example:
>> "English_United Kingdom.28591" (Latin1)
>> Is there any reason not to accept other combinations that setlocale() is
>> happy with?
>
> Are you certain that that acceptance actually represents support?
> Have you checked that it rejects combinations involving real code
> pages (ie, NOT 65001) that don't really work with the locale?

It fails with ones that Microsoft have decided don't belong in my
language group and therefore aren't installed. It accepts all the others
I've tried, but then from the sample I've looked, they all have
0-9a-zA-Z in them so I guess they're all capable of handling English.

Regards, Dave.

Dave Page <dpage(at)postgresql(dot)org> writes:
> Tom Lane wrote:
>> Are you certain that that acceptance actually represents support?
>> Have you checked that it rejects combinations involving real code
>> pages (ie, NOT 65001) that don't really work with the locale?

> It fails with ones that Microsoft have decided don't belong in my
> language group and therefore aren't installed. It accepts all the others
> I've tried, but then from the sample I've looked, they all have
> 0-9a-zA-Z in them so I guess they're all capable of handling English.

That doesn't exactly fill me with confidence. Maybe you need to make
some tests involving a non-English base locale?

regards, tom lane

Tom Lane wrote:
> Dave Page <dpage(at)postgresql(dot)org> writes:
>> Tom Lane wrote:
>>> Are you certain that that acceptance actually represents support?
>>> Have you checked that it rejects combinations involving real code
>>> pages (ie, NOT 65001) that don't really work with the locale?
>
>> It fails with ones that Microsoft have decided don't belong in my
>> language group and therefore aren't installed. It accepts all the others
>> I've tried, but then from the sample I've looked, they all have
>> 0-9a-zA-Z in them so I guess they're all capable of handling English.
>
> That doesn't exactly fill me with confidence. Maybe you need to make
> some tests involving a non-English base locale?

Hmm, I'm guessing these probably shouldn't work:

Dave(at)SNAKE:~$ setlc "Japanese_Japan.28605"
Japanese_Japan.28605
Dave(at)SNAKE:~$ setlc "Japanese_Japan.28595"
Japanese_Japan.28595
Dave(at)SNAKE:~$ setlc "Russian_Russia.1252"
Russian_Russia.1252
Dave(at)SNAKE:~$ setlc "Russian_Russia.28591"
Russian_Russia.28591

1252 == WIN1252
28591 == LATIN1
28605 == LATIN9
28595 == ISO8859-5 (Cyrillic)
28597 == ISO8859-7 (Greek)

In fact, it looks like it'll allow me to use anything thats installed,
regardless of whether they're liekly to be compatible. So much for
trusting setlocale() :-(

/D

Dave Page <dpage(at)postgresql(dot)org> writes:
> OK so I added the appropriate entries (and posted the patch to
> -patches), but my original question remains: why can I only select the
> *default* encoding for the chosen locale, but not other ones that are
> also be valid according to setlocale? Is this a bug, or is there some
> technical reason?

Well, the chklocale code is designed around the assumption that there
*is* only one encoding for which a locale setting will work, with
C/POSIX being a special case.

I think we are talking a bit at cross-purposes here, because the Windows
equivalent to this notion seems to be "English_United Kingdom.1252"
whereas you seem to be defining locale as just "English_United Kingdom".
Does it not work the way you want if you make the installer pass locale
strings of the first form to initdb?

regards, tom lane

Tom Lane wrote:
> Dave Page <dpage(at)postgresql(dot)org> writes:
>> OK so I added the appropriate entries (and posted the patch to
>> -patches), but my original question remains: why can I only select the
>> *default* encoding for the chosen locale, but not other ones that are
>> also be valid according to setlocale? Is this a bug, or is there some
>> technical reason?
>
> Well, the chklocale code is designed around the assumption that there
> *is* only one encoding for which a locale setting will work, with
> C/POSIX being a special case.
>
> I think we are talking a bit at cross-purposes here, because the Windows
> equivalent to this notion seems to be "English_United Kingdom.1252"
> whereas you seem to be defining locale as just "English_United Kingdom".
> Does it not work the way you want if you make the installer pass locale
> strings of the first form to initdb?

Yes, it seems it does (see my previous email to Peter):
http://archives.postgresql.org/pgsql-hackers/2007-10/msg00447.php

So I guess that's how I'll fix the installer. There is another issue
though as I mentioned in the post above - that it complains about an
invalid encoding specifier on the encoding name, then ignores it and
uses the default which seems wrong to me.

/D

Dave Page <dpage(at)postgresql(dot)org> writes:
> In fact, it looks like it'll allow me to use anything thats installed,
> regardless of whether they're liekly to be compatible. So much for
> trusting setlocale() :-(

Yech :-(. Count on Microsloth to get this wrong. Anyone have any ideas
on how to tell if a locale setting *really* works on Windows?

regards, tom lane

Dave Page <dpage(at)postgresql(dot)org> writes:
> ... There is another issue
> though as I mentioned in the post above - that it complains about an
> invalid encoding specifier on the encoding name, then ignores it and
> uses the default which seems wrong to me.

Yeah, if you look at chklocale() in initdb.c this is clearly how it
works, but there's a comment
/* should we exit here? */
so whoever wrote it wasn't all that convinced it was the right behavior.

Given that 8.3 is raising the stakes for having a correct locale
specification at initdb time, it seems right to me to error out if a
bogus locale switch is given, rather than whining and then substituting
the environment default. Any objections?

That still leaves us with the problem of how to tell whether a locale
spec is bad on Windows. Judging by your example, Windows checks whether
the code page is present but not whether it is sane for the base locale.
What happens when there's a mismatch --- eg, what encoding do system
messages come out in?

regards, tom lane

Tom Lane wrote
> That still leaves us with the problem of how to tell whether a locale
> spec is bad on Windows. Judging by your example, Windows checks whether
> the code page is present but not whether it is sane for the base locale.
> What happens when there's a mismatch --- eg, what encoding do system
> messages come out in?

I'm not sure how to test that specifically, but it seems that accented
characters simply fall back to their undecorated equivalents if the
encoding is not appropriate, eg:

Dave(at)SNAKE:~$ ./setlc French_France.1252
Locale: French_France.1252
The date is: sam. 01 of août 2007
Dave(at)SNAKE:~$ ./setlc French_France.28597
Locale: French_France.28597
The date is: sam. 01 of aout 2007

(the encodings used there are WIN1252 and ISO8859-7 (Greek)).

I'm happy to test further is you can suggest how I can figure out the
encoding actually output.

Regards, Dave.

On 10/12/07, Dave Page <dpage(at)postgresql(dot)org> wrote:
> Tom Lane wrote
> > That still leaves us with the problem of how to tell whether a locale
> > spec is bad on Windows. Judging by your example, Windows checks whether
> > the code page is present but not whether it is sane for the base locale.
> > What happens when there's a mismatch --- eg, what encoding do system
> > messages come out in?
>
> I'm not sure how to test that specifically, but it seems that accented
> characters simply fall back to their undecorated equivalents if the
> encoding is not appropriate, eg:
>
> Dave(at)SNAKE:~$ ./setlc French_France.1252
> Locale: French_France.1252
> The date is: sam. 01 of août 2007
> Dave(at)SNAKE:~$ ./setlc French_France.28597
> Locale: French_France.28597
> The date is: sam. 01 of aout 2007
>
> (the encodings used there are WIN1252 and ISO8859-7 (Greek)).
>
> I'm happy to test further is you can suggest how I can figure out the
> encoding actually output.

The encoding output is the one you specified. Keep in mind,
underneath Windows is mostly working with Unicode, so all characters
exist and the locale rules specify their behavior there. The encoding
is just the byte stream it needs to force them all into after doing
whatever it does to them. As you've seen, it uses some sort of
best-fit mapping I don't know the details of. (It will drop accent
marks and choose characters with similar shape where possible, by
default.)

I think it's a bit more complex for input/transform cases where you
operate on the byte stream directly without intermediate conversion to
Unicode, which is why UTF-8 doesn't work as a codepage, but again I
don't have the details nearby. I can try to do more digging if
needed.

Trevor Talbot wrote:
> The encoding output is the one you specified.

OK.

> Keep in mind,
> underneath Windows is mostly working with Unicode, so all characters
> exist and the locale rules specify their behavior there. The encoding
> is just the byte stream it needs to force them all into after doing
> whatever it does to them. As you've seen, it uses some sort of
> best-fit mapping I don't know the details of. (It will drop accent
> marks and choose characters with similar shape where possible, by
> default.)

Right, that makes sense. The codepages used by setlocale etc. are just
translation tables to/from the internal unicode representation.

> I think it's a bit more complex for input/transform cases where you
> operate on the byte stream directly without intermediate conversion to
> Unicode, which is why UTF-8 doesn't work as a codepage, but again I
> don't have the details nearby. I can try to do more digging if
> needed.

It does (sort of) work as a codepage, it just doesn't have the NLS file
to define how things like UPPER() and LOWER() should work.

Regards, Dave

On Fri, Oct 12, 2007 at 06:03:52AM -0700, Trevor Talbot wrote:
> On 10/12/07, Dave Page <dpage(at)postgresql(dot)org> wrote:
> > Tom Lane wrote
> > > That still leaves us with the problem of how to tell whether a locale
> > > spec is bad on Windows. Judging by your example, Windows checks whether
> > > the code page is present but not whether it is sane for the base locale.
> > > What happens when there's a mismatch --- eg, what encoding do system
> > > messages come out in?
> >
> > I'm not sure how to test that specifically, but it seems that accented
> > characters simply fall back to their undecorated equivalents if the
> > encoding is not appropriate, eg:
> >
> > Dave(at)SNAKE:~$ ./setlc French_France.1252
> > Locale: French_France.1252
> > The date is: sam. 01 of août 2007
> > Dave(at)SNAKE:~$ ./setlc French_France.28597
> > Locale: French_France.28597
> > The date is: sam. 01 of aout 2007
> >
> > (the encodings used there are WIN1252 and ISO8859-7 (Greek)).
> >
> > I'm happy to test further is you can suggest how I can figure out the
> > encoding actually output.
>
> The encoding output is the one you specified. Keep in mind,
> underneath Windows is mostly working with Unicode, so all characters
> exist and the locale rules specify their behavior there. The encoding
> is just the byte stream it needs to force them all into after doing
> whatever it does to them. As you've seen, it uses some sort of
> best-fit mapping I don't know the details of. (It will drop accent
> marks and choose characters with similar shape where possible, by
> default.)
>
> I think it's a bit more complex for input/transform cases where you
> operate on the byte stream directly without intermediate conversion to
> Unicode, which is why UTF-8 doesn't work as a codepage, but again I
> don't have the details nearby. I can try to do more digging if
> needed.

Just so the non-windows-savvy people get it.. When Windows documentation or
users refer to Unicode, they mean UTF-16.

//Magnus