Re: [COMMITTERS] pgsql: Re-allow UTF8 encodings on win32.

Log Message:
-----------
Re-allow UTF8 encodings on win32. Since UTF8 is converted to
UTF16 before being used, all (valid) locales will work for this.

Modified Files:
--------------
pgsql/src/backend/commands:
dbcommands.c (r1.201 -> r1.202)
(http://developer.postgresql.org/cvsweb.cgi/pgsql/src/backend/commands/dbcommands.c?r1=1.201&r2=1.202)
pgsql/src/bin/initdb:
initdb.c (r1.146 -> r1.147)
(http://developer.postgresql.org/cvsweb.cgi/pgsql/src/bin/initdb/initdb.c?r1=1.146&r2=1.147)

mha(at)postgresql(dot)org (Magnus Hagander) writes:
> Re-allow UTF8 encodings on win32. Since UTF8 is converted to
> UTF16 before being used, all (valid) locales will work for this.

So where do we stand on the Windows locale/encoding business --- are
we happy with the behavior now, or does it still need work?

regards, tom lane

Tom Lane wrote:
> mha(at)postgresql(dot)org (Magnus Hagander) writes:
>> Re-allow UTF8 encodings on win32. Since UTF8 is converted to
>> UTF16 before being used, all (valid) locales will work for this.
>
> So where do we stand on the Windows locale/encoding business --- are
> we happy with the behavior now, or does it still need work?

I think we're good. But I'd like to hear some verification from somebody
else. Specifically, I'd like to hear a signoff from someone who can
actually do "real tests" on a locale that's not US and not Swedish.
Also, I'd like to hear from the Japanese people (Hiroshi? Can you do
this?) that we didn't break it for them. I don't think we did, but I
want to be sure :)

Hiroshi, and whomever else can help to test, this is only testing the
backend, not the installer. The installer may need a few minor tweaks
still once the backend is considered fixed. And what needs to be tested
is CVS HEAD as of today.

//Magnus

Hi.

Um, It seems that it only passed the strict check of chklocale.c. Probably, It may
enable mistaken selection...However, I will clarify a problem by the test.

Regards,
Hiroshi Saito

From: "Magnus Hagander" <magnus(at)hagander(dot)net>

> Tom Lane wrote:
>> mha(at)postgresql(dot)org (Magnus Hagander) writes:
>>> Re-allow UTF8 encodings on win32. Since UTF8 is converted to
>>> UTF16 before being used, all (valid) locales will work for this.
>>
>> So where do we stand on the Windows locale/encoding business --- are
>> we happy with the behavior now, or does it still need work?
>
> I think we're good. But I'd like to hear some verification from somebody
> else. Specifically, I'd like to hear a signoff from someone who can
> actually do "real tests" on a locale that's not US and not Swedish.
> Also, I'd like to hear from the Japanese people (Hiroshi? Can you do
> this?) that we didn't break it for them. I don't think we did, but I
> want to be sure :)
>
> Hiroshi, and whomever else can help to test, this is only testing the
> backend, not the installer. The installer may need a few minor tweaks
> still once the backend is considered fixed. And what needs to be tested
> is CVS HEAD as of today.
>
> //Magnus

2007/10/16, Magnus Hagander <magnus(at)hagander(dot)net>:
> Tom Lane wrote:
> > mha(at)postgresql(dot)org (Magnus Hagander) writes:
> >> Re-allow UTF8 encodings on win32. Since UTF8 is converted to
> >> UTF16 before being used, all (valid) locales will work for this.
> >
> > So where do we stand on the Windows locale/encoding business --- are
> > we happy with the behavior now, or does it still need work?
>
> I think we're good. But I'd like to hear some verification from somebody
> else. Specifically, I'd like to hear a signoff from someone who can
> actually do "real tests" on a locale that's not US and not Swedish.
> Also, I'd like to hear from the Japanese people (Hiroshi? Can you do
> this?) that we didn't break it for them. I don't think we did, but I
> want to be sure :)
>
> Hiroshi, and whomever else can help to test, this is only testing the
> backend, not the installer. The installer may need a few minor tweaks
> still once the backend is considered fixed. And what needs to be tested
> is CVS HEAD as of today.
>
> //Magnus
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
>

I can test it with czech locale. Can I download binaries anywhere?

Pavel

Hi.

> I can test it with czech locale. Can I download binaries anywhere?
http://winpg.jp/~saito/pg83/postgresql-8.3beta-cvs.tgz
It is a thing after regression test.(MinGW+gcc)

Regards,
Hiroshi Saito

Hi.

> Um, It seems that it only passed the strict check of chklocale.c. Probably, It may
> enable mistaken selection...However, I will clarify a problem by the test.

First, it is one problem....
http://winpg.jp/~saito/pg83/pg83b1-err.txt

And a test continues....

Hi.

Second, it is big problem....
http://winpg.jp/~saito/pg83/pg83b1-err2.txt
It is text serch config error.
However, It passes initdb.(locale=Japanese_Japan.932 ... This is ShiftJIS locale)

And a test continues....

Regards,
Hiroshi Saito

Hiroshi Saito wrote:
> Hi.
>
>> Um, It seems that it only passed the strict check of chklocale.c.
>> Probably, It may enable mistaken selection...However, I will clarify a
>> problem by the test.
>
> First, it is one problem....
> http://winpg.jp/~saito/pg83/pg83b1-err.txt
>
> And a test continues....

But SJIS isn't supposed to work, no?

//Magnus

Hiroshi Saito wrote:
> Hi.
>
> Second, it is big problem....
> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
> It is text serch config error.
> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
> ShiftJIS locale)
>
> And a test continues....

What text search config would you expect?

//Magnus

Hi.

> Hiroshi Saito wrote:
>> Hi.
>>
>> Second, it is big problem....
>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>> It is text serch config error.
>> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
>> ShiftJIS locale)
>>
>> And a test continues....
>
> What text search config would you expect?

This problem here is that locale of initdb passes Japanese_Japan.932.

Regards,
Hiroshi Saito

Hiroshi Saito wrote:
> Hi.
>
> Second, it is big problem....
> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
> It is text serch config error.
> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
> ShiftJIS locale)
>
> And a test continues....

The changes that were made were only to re-enable UTF-8.

SJIS wasn't ever supported as a server encoding
(http://www.postgresql.org/docs/8.2/interactive/multibyte.html). The
fact that initdb continues if you use Japanese_Japan.932 is an
inconsistency I reported previously but has yet to be fixed.

/D

From: "Dave Page" <dpage(at)postgresql(dot)org>

> Hiroshi Saito wrote:
>> Hi.
>>
>> Second, it is big problem....
>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>> It is text serch config error.
>> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
>> ShiftJIS locale)
>>
>> And a test continues....
>
> The changes that were made were only to re-enable UTF-8.

Yes, Please see,
http://winpg.jp/~saito/pg83/pg83b1-err2.txt
Is that initdb is successful a problem as for this?

>
> SJIS wasn't ever supported as a server encoding
> (http://www.postgresql.org/docs/8.2/interactive/multibyte.html). The
> fact that initdb continues if you use Japanese_Japan.932 is an
> inconsistency I reported previously but has yet to be fixed.

Yes, However, Encoding and locale are not equivalent.

Regards,
Hiroshi Saito

Hiroshi Saito wrote:
> From: "Dave Page" <dpage(at)postgresql(dot)org>
>
>> Hiroshi Saito wrote:
>>> Hi.
>>>
>>> Second, it is big problem....
>>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>>> It is text serch config error.
>>> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
>>> ShiftJIS locale)
>>>
>>> And a test continues....
>>
>> The changes that were made were only to re-enable UTF-8.
>
> Yes, Please see,
> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
> Is that initdb is successful a problem as for this?

Oh, sorry - misread that. I chatted with Magnus about that. It is
correct, but misleading. pg_control will say Japanese_Japan.932 as well
iirc, even though it is really Japanese_Japan.65001.

Regards, Dave

Dave Page wrote:
> Hiroshi Saito wrote:
>> Hi.
>>
>> Second, it is big problem....
>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>> It is text serch config error.
>> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
>> ShiftJIS locale)
>>
>> And a test continues....
>
> The changes that were made were only to re-enable UTF-8.
>
> SJIS wasn't ever supported as a server encoding
> (http://www.postgresql.org/docs/8.2/interactive/multibyte.html). The
> fact that initdb continues if you use Japanese_Japan.932 is an
> inconsistency I reported previously but has yet to be fixed.

That is a good point, if unrelated to this very discussion. Do we want
to change that thing to an exit instead of complain-and-continue? I
think yes?

//Magnus

Dave Page wrote:
> Hiroshi Saito wrote:
>> From: "Dave Page" <dpage(at)postgresql(dot)org>
>>
>>> Hiroshi Saito wrote:
>>>> Hi.
>>>>
>>>> Second, it is big problem....
>>>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>>>> It is text serch config error.
>>>> However, It passes initdb.(locale=Japanese_Japan.932 ... This is
>>>> ShiftJIS locale)
>>>>
>>>> And a test continues....
>>> The changes that were made were only to re-enable UTF-8.
>> Yes, Please see,
>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>> Is that initdb is successful a problem as for this?
>
> Oh, sorry - misread that. I chatted with Magnus about that. It is
> correct, but misleading. pg_control will say Japanese_Japan.932 as well
> iirc, even though it is really Japanese_Japan.65001.

Not so. The locale is Japanese_Japan, really. That's the only part
that's relevant for UTF16 encodings, which is what we use to do UTF8. We
specifically *don't* try to use Japanese_Japan.65001.

//Magnus

Hi.

From: "Dave Page" <dpage(at)postgresql(dot)org>
>> Yes, Please see,
>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>> Is that initdb is successful a problem as for this?
>
> Oh, sorry - misread that. I chatted with Magnus about that. It is
> correct, but misleading. pg_control will say Japanese_Japan.932 as well
> iirc, even though it is really Japanese_Japan.65001.

But, Please see.
http://winpg.jp/~saito/pg83/pg83b1-err3.txt
Japanese_Japan.65001 is error...
Japanese_Japan is true.

Regards,
Hiroshi Saito

Hiroshi Saito wrote:
> Hi.
>
> From: "Dave Page" <dpage(at)postgresql(dot)org>
>>> Yes, Please see,
>>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>>> Is that initdb is successful a problem as for this?
>>
>> Oh, sorry - misread that. I chatted with Magnus about that. It is
>> correct, but misleading. pg_control will say Japanese_Japan.932 as well
>> iirc, even though it is really Japanese_Japan.65001.
>
> But, Please see.
> http://winpg.jp/~saito/pg83/pg83b1-err3.txt
> Japanese_Japan.65001 is error...
> Japanese_Japan is true.

Yes, that is expected. If you explicitly ask for the .65001 locale it
will try the one that doesn't have the proper NLS files, and that
shouldn't work. If you just put in Japanese_Japan, it will use the UTF16
locale.

//Magnus

Magnus Hagander wrote:
> Not so. The locale is Japanese_Japan, really. That's the only part
> that's relevant for UTF16 encodings, which is what we use to do UTF8. We
> specifically *don't* try to use Japanese_Japan.65001.

Thats not what I mean. From a *usability* perspective, Hiroshi should
see Japanese_Japan.65001 because he's selected UTF-8 in Japanese_Japan.
He shouldn't see Japanese_Japan.932 because that definitely isn't what
he selected.

/D

> But, Please see.
> http://winpg.jp/~saito/pg83/pg83b1-err3.txt
> Japanese_Japan.65001 is error...
> Japanese_Japan is true.

However, The test of this state is continued.
But but but, Sorry, I face to a bed...

Regards,
Hiroshi Saito

Dave Page wrote:
> Magnus Hagander wrote:
>> Not so. The locale is Japanese_Japan, really. That's the only part
>> that's relevant for UTF16 encodings, which is what we use to do UTF8. We
>> specifically *don't* try to use Japanese_Japan.65001.
>
> Thats not what I mean. From a *usability* perspective, Hiroshi should
> see Japanese_Japan.65001 because he's selected UTF-8 in Japanese_Japan.
> He shouldn't see Japanese_Japan.932 because that definitely isn't what
> he selected.

I'l grant you that from a usbility perspective, he should see
Japanese_Japan. Not the .65001 part, though.

//Magnus

Hiroshi Saito wrote:
> Hi.
>
> From: "Dave Page" <dpage(at)postgresql(dot)org>
>>> Yes, Please see,
>>> http://winpg.jp/~saito/pg83/pg83b1-err2.txt
>>> Is that initdb is successful a problem as for this?
>>
>> Oh, sorry - misread that. I chatted with Magnus about that. It is
>> correct, but misleading. pg_control will say Japanese_Japan.932 as well
>> iirc, even though it is really Japanese_Japan.65001.
>
> But, Please see.
> http://winpg.jp/~saito/pg83/pg83b1-err3.txt
> Japanese_Japan.65001 is error...
> Japanese_Japan is true.

Yes, we're faking utf-8 support using utf-16. Specifying it as you have
there bypasses the workaround and tries to use the 65001 codepage which
then fails because LC_CTYPE cannot be set to .65001 in any locale.

/D

Magnus Hagander wrote:
> Dave Page wrote:
>> Magnus Hagander wrote:
>>> Not so. The locale is Japanese_Japan, really. That's the only part
>>> that's relevant for UTF16 encodings, which is what we use to do UTF8. We
>>> specifically *don't* try to use Japanese_Japan.65001.
>> Thats not what I mean. From a *usability* perspective, Hiroshi should
>> see Japanese_Japan.65001 because he's selected UTF-8 in Japanese_Japan.
>> He shouldn't see Japanese_Japan.932 because that definitely isn't what
>> he selected.
>
> I'l grant you that from a usbility perspective, he should see
> Japanese_Japan. Not the .65001 part, though.

Well, that depends on whether we care that we're actually faking the
utf-8 support and/or we want to keep the message consistent with what
you'd see in other locales.

/D

2007/10/16, Hiroshi Saito <z-saito(at)guitar(dot)ocn(dot)ne(dot)jp>:
> Hi.
>
> > I can test it with czech locale. Can I download binaries anywhere?
> http://winpg.jp/~saito/pg83/postgresql-8.3beta-cvs.tgz
> It is a thing after regression test.(MinGW+gcc)
>

I have problem, there isn't libintl-2.dll

Pavel

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Dave Page wrote:
>> SJIS wasn't ever supported as a server encoding
>> (http://www.postgresql.org/docs/8.2/interactive/multibyte.html). The
>> fact that initdb continues if you use Japanese_Japan.932 is an
>> inconsistency I reported previously but has yet to be fixed.

> That is a good point, if unrelated to this very discussion. Do we want
> to change that thing to an exit instead of complain-and-continue? I
> think yes?

Yeah, I thought we'd agreed to that a few days ago.

regards, tom lane

Hi.

From: "Pavel Stehule" <pavel(dot)stehule(at)gmail(dot)com>
>> > I can test it with czech locale. Can I download binaries anywhere?
>> http://winpg.jp/~saito/pg83/postgresql-8.3beta-cvs.tgz
>> It is a thing after regression test.(MinGW+gcc)
>>
>
> I have problem, there isn't libintl-2.dll

Ooops, sorry, it is full-build.
Please, this is minimum composition
http://winpg.jp/~saito/pg83/postgresql-8.3beta-cvs-minbin.tgz
Thanks.

Regards,
Hiroshi Saito

Hi.

From: "Magnus Hagander" <magnus(at)hagander(dot)net>

>> But, Please see.
>> http://winpg.jp/~saito/pg83/pg83b1-err3.txt
>> Japanese_Japan.65001 is error...
>> Japanese_Japan is true.
>
> Yes, that is expected. If you explicitly ask for the .65001 locale it
> will try the one that doesn't have the proper NLS files, and that
> shouldn't work. If you just put in Japanese_Japan, it will use the UTF16
> locale.

Umm, As for result ...
initdb -E UTF8 --locale=Japanese_Japan -D../data
http://winpg.jp/~saito/pg83/pg83b1-err4.txt
It seems that it is only complemented.

Regards,
Hiroshi Saito

Hiroshi Saito wrote:
> Hi.
>
> From: "Magnus Hagander" <magnus(at)hagander(dot)net>
>
>>> But, Please see.
>>> http://winpg.jp/~saito/pg83/pg83b1-err3.txt
>>> Japanese_Japan.65001 is error...
>>> Japanese_Japan is true.
>>
>> Yes, that is expected. If you explicitly ask for the .65001 locale it
>> will try the one that doesn't have the proper NLS files, and that
>> shouldn't work. If you just put in Japanese_Japan, it will use the UTF16
>> locale.
>
> Umm, As for result ... initdb -E UTF8 --locale=Japanese_Japan -D../data
> http://winpg.jp/~saito/pg83/pg83b1-err4.txt
> It seems that it is only complemented.

Yes, that is expected, though not entirely to my tastes. The cluster
should still actually be in utf-8 however.

/D

I did some test, but without success,

Pavel

I have win2003 Server .. with czech locales support.

I:\PGSQL\BIN>initdb -D ../data -L i:\pgsql\share

The files belonging to this database system will be owned by user "postgres".

This user must also own the server process.

The database cluster will be initialized with locale Czech_Czech Republic.1250.

could not determine encoding for locale "Czech_Czech Republic.1250": codeset is

"CP1250"

INITDB: could not find suitable encoding for locale Czech_Czech Republic.1250

Rerun INITDB with the -E option.

Try "INITDB --help" for more information.

I:\PGSQL\BIN>

I:\PGSQL\BIN>initdb -E UTF-8 -D ../data -L i:\pgsql\share

The files belonging to this database system will be owned by user "postgres".

This user must also own the server process.

The database cluster will be initialized with locale Czech_Czech Republic.1250.

could not determine encoding for locale "Czech_Czech Republic.1250": codeset is

"CP1250"

INITDB: could not find suitable text search configuration for locale Czech_Czech

Republic.1250

The default text search configuration will be set to "simple".

fixing permissions on existing directory ../data ... ok

creating subdirectories ... ok

selecting default max_connections ... 10

selecting default shared_buffers/max_fsm_pages ... 400kB/20000

creating configuration files ... ok

creating template1 database in ../data/base/1 ... FATAL: could not select a sui

table default timezone

DETAIL: It appears that your GMT time zone uses leap seconds. PostgreSQL does n

ot support leap seconds.

child process exited with exit code 1

INITDB: removing contents of data directory "../data"

I:\PGSQL\BIN>initdb -E win1250 --locale="Czech_Czech Republic.1250" -D ../data -

L i:\pgsql\share

The files belonging to this database system will be owned by user "postgres".

This user must also own the server process.

The database cluster will be initialized with locale Czech_Czech Republic.1250.

could not determine encoding for locale "Czech_Czech Republic.1250": codeset is

"CP1250"

INITDB: could not find suitable text search configuration for locale Czech_Czech

Republic.1250

The default text search configuration will be set to "simple".

fixing permissions on existing directory ../data ... ok

creating subdirectories ... ok

selecting default max_connections ... 10

selecting default shared_buffers/max_fsm_pages ... 400kB/20000

creating configuration files ... ok

creating template1 database in ../data/base/1 ... FATAL: could not select a sui

table default timezone

DETAIL: It appears that your GMT time zone uses leap seconds. PostgreSQL does n

ot support leap seconds.

child process exited with exit code 1

INITDB: removing contents of data directory "../data"

"Pavel Stehule" <pavel(dot)stehule(at)gmail(dot)com> writes:
> could not determine encoding for locale "Czech_Czech Republic.1250": codeset is

> "CP1250"

Hm, we seem to have missed an entry for PG_WIN1250. Fixed.

regards, tom lane