A rough roadmap for internationalization fixes

OK, I've been spreading rumours about fixing the internationalization
problems, so let me make it a bit more clear. Here are the problems that
need to be fixed:

- Only one locale per process possible.

- Only one gettext-language per process possible.

- lc_collate and lc_ctype need to be held fixed in the entire cluster.

- Gettext relies on iconv character set conversion, which relies on
lc_ctype, which leads to a complete screw-up in the server because of
the previous item.

- Locale fixed per cluster, but encoding fixed per database, unware
of each other, don't get along.

- No support for upper/lower with multibyte encoding.

- Implementation of Unicode horribly incomplete.

These are all dependent on each other and sort of flow into each other.

Here is a proposed ordering of steps toward improving the situation:

1. Take out the character set conversion routines from the backend and
make them a library of their own. This could possibly be modelled after
iconv, but not necessarily. Or we might conclude that we can just use
iconv in the first place.

2. Reimplement gettext to use 1. and allow switching of language and
encoding at run-time.

3. Implement Unicode collation algorithm and character classification
routines that are aware of 1. Use that in place of system locale
routines.

4. Allow choice of locale per database. (This should be fairly easy after
3.)

5. Allow choice of locale per column and implement collation coercion
according to SQL standard.

This could easily take a long time, but I feel that even if we have to
stop after 2., 3., or 4. at feature freeze, we'd be a lot farther.

Comments? Anything else that needs fixing?

--
Peter Eisentraut peter_e(at)gmx(dot)net

> OK, I've been spreading rumours about fixing the internationalization
> problems, so let me make it a bit more clear. Here are the problems that
> need to be fixed:
>
> - Only one locale per process possible.
>
> - Only one gettext-language per process possible.
>
> - lc_collate and lc_ctype need to be held fixed in the entire cluster.
>
> - Gettext relies on iconv character set conversion, which relies on
> lc_ctype, which leads to a complete screw-up in the server because of
> the previous item.
>
> - Locale fixed per cluster, but encoding fixed per database, unware
> of each other, don't get along.
>
> - No support for upper/lower with multibyte encoding.
>
> - Implementation of Unicode horribly incomplete.
>
> These are all dependent on each other and sort of flow into each other.
>
> Here is a proposed ordering of steps toward improving the situation:
>
> 1. Take out the character set conversion routines from the backend and
> make them a library of their own. This could possibly be modelled after
> iconv, but not necessarily. Or we might conclude that we can just use
> iconv in the first place.

How do you handle user-defined conversions?

> 2. Reimplement gettext to use 1. and allow switching of language and
> encoding at run-time.
>
> 3. Implement Unicode collation algorithm and character classification
> routines that are aware of 1. Use that in place of system locale
> routines.

I don't see a relationship between Unicode and the one you are going
to replace the system locale routines. If you are going to the
direction for an "Unicode central" implementation, I will object.

> 4. Allow choice of locale per database. (This should be fairly easy after
> 3.)
>
> 5. Allow choice of locale per column and implement collation coercion
> according to SQL standard.
>
> This could easily take a long time, but I feel that even if we have to
> stop after 2., 3., or 4. at feature freeze, we'd be a lot farther.
>
> Comments? Anything else that needs fixing?
>
> --
> Peter Eisentraut peter_e(at)gmx(dot)net
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: Have you searched our list archives?
>
> http://archives.postgresql.org
>

On Mon, 24 Nov 2003, Peter Eisentraut wrote:

> 1. Take out the character set conversion routines from the backend and
> make them a library of their own. This could possibly be modelled after
> iconv, but not necessarily. Or we might conclude that we can just use
> iconv in the first place.
>
> 2. Reimplement gettext to use 1. and allow switching of language and
> encoding at run-time.

Force all translations to be in unicode and convert to other client
encodings if needed. There is no need to support translations stored using
different encodings.

> 3. Implement Unicode collation algorithm and character classification
> routines that are aware of 1. Use that in place of system locale
> routines.

Couldn't we use some library that already have this, like glib (or
something else). If it's not up to what we need, than fix that library
instead.

--
/Dennis

Tatsuo Ishii writes:

> > 3. Implement Unicode collation algorithm and character classification
> > routines that are aware of 1. Use that in place of system locale
> > routines.
>
> I don't see a relationship between Unicode and the one you are going
> to replace the system locale routines. If you are going to the
> direction for an "Unicode central" implementation, I will object.

The Unicode collation algorithm works for any character set, not only for
Unicode. It just happens to be published by the Unicode consortium. So
basically this is just a concrete alternative to making up our own out of
thin air. Also, the Unicode collation algorithm gives us the flexibility
to define customizations of collations that users frequently want, such as
ignoring or not ignoring punctuation.

Actually, what will more likely happen is that we'll define a collation as
a collection of one or more support functions, the equivalents of
strxfrm() and possibly a few more. Then it will be up to those functions
to define the collation order. The server will provide utility functions
that will facilitate implementing a collation order that follows the
Unicode collation algorithm, but you could just as well implement one
using memcmp() or whatever you like.

--
Peter Eisentraut peter_e(at)gmx(dot)net

Dennis Bjorklund writes:

> Force all translations to be in unicode and convert to other client
> encodings if needed. There is no need to support translations stored using
> different encodings.

Tell that to the Japanese.

> Couldn't we use some library that already have this, like glib (or
> something else). If it's not up to what we need, than fix that library
> instead.

I wasn't aware that glib had this. I'll look.

--
Peter Eisentraut peter_e(at)gmx(dot)net

On Tue, 25 Nov 2003, Peter Eisentraut wrote:

> > Force all translations to be in unicode and convert to other client
> > encodings if needed. There is no need to support translations stored using
> > different encodings.
>
> Tell that to the Japanese.

I've always thought unicode was enough to even represent Japanese. Then
the client encoding can be something else that we can convert to. In any
way, the encoding of the message catalog has to be known to the system so
it can be converted to the correct encoding for the client.

> > Couldn't we use some library that already have this, like glib (or
> > something else). If it's not up to what we need, than fix that library
> > instead.
>
> I wasn't aware that glib had this. I'll look.

And I don't really know what demands pg has, but glib has a lot of support
functions for utf-8. At least we should take a look at it.

--
/Dennis

> On Tue, 25 Nov 2003, Peter Eisentraut wrote:
>
> > > Force all translations to be in unicode and convert to other client
> > > encodings if needed. There is no need to support translations stored using
> > > different encodings.
> >
> > Tell that to the Japanese.
>
> I've always thought unicode was enough to even represent Japanese. Then
> the client encoding can be something else that we can convert to. In any
> way, the encoding of the message catalog has to be known to the system so
> it can be converted to the correct encoding for the client.

I'm tired of telling that Unicode is not that perfect. Another gottcha
with Unicode is the UTF-8 encoding (currently we use) consumes 3
bytes for each Kanji character, while other encodings consume only 2
bytes. IMO 3/2 storage ratio could not be neglected for database use.
--
Tatsuo Ishii

On Tue, 25 Nov 2003, Tatsuo Ishii wrote:

> I'm tired of telling that Unicode is not that perfect. Another gottcha
> with Unicode is the UTF-8 encoding (currently we use) consumes 3
> bytes for each Kanji character, while other encodings consume only 2
> bytes. IMO 3/2 storage ratio could not be neglected for database use.

I'm aware of how utf-8 works and I was talking about the message
cataloges. It does not affect what you store in the database in any way.

--
/Dennis

On Tue, 25 Nov 2003, Tatsuo Ishii wrote:

> I'm tired of telling that Unicode is not that perfect. Another gottcha
> with Unicode is the UTF-8 encoding (currently we use) consumes 3
> bytes for each Kanji character, while other encodings consume only 2
> bytes. IMO 3/2 storage ratio could not be neglected for database use.

The rest of the world seems to select unicode as the way to handle
different languages in the UI of programs. For example gnome supports
nothing but unicode. How is that handled in your country? I know that you
are tired of people who don't understand how difficult it is for you, but
I really would like to know. Is gnome not used over there because of this?

About storing data in the database, I would expect it to work with any
encoding, just like I would expect pg to be able to store images in any
format.

I'll try to not mention unicode near you in the feature :-)

--
/Dennis

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Actually, what will more likely happen is that we'll define a collation as
> a collection of one or more support functions, the equivalents of
> strxfrm() and possibly a few more. Then it will be up to those functions
> to define the collation order. The server will provide utility functions
> that will facilitate implementing a collation order that follows the
> Unicode collation algorithm, but you could just as well implement one
> using memcmp() or whatever you like.

That sounds like a good plan to me. Personally I'd want a
memcmp()-based collation implementation available, so that people who
don't care about sorting anything beyond 7-bit ASCII don't need to pay
a lot of overhead.

We have seen over and over that strcoll() is depressingly slow in some
locales (at least on some platforms). Do you have any feeling for the
real-world performance of the Unicode algorithm?

regards, tom lane

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Dennis Bjorklund writes:
>> Couldn't we use some library that already have this, like glib (or
>> something else). If it's not up to what we need, than fix that library
>> instead.

> I wasn't aware that glib had this. I'll look.

Of course the trouble with relying on glibc is that we'd have no solution
for platforms that don't use glibc.

It might be okay to rely on glibc for a first-cut implementation,
realizing that we couldn't do everything at once anyway.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>
> > I wasn't aware that glib had this. I'll look.
>
> Of course the trouble with relying on glibc is that we'd have no solution
> for platforms that don't use glibc.

glib != glibc. glib is the low-level library used by GTK and GNOME
for basic data structures, character handling etc. It's LGPL AFAIK,
which would seem to rule out diredct use from a licensing perspective.

-Doug

Dennis Bjorklund kirjutas T, 25.11.2003 kell 14:51:
> On Tue, 25 Nov 2003, Tatsuo Ishii wrote:
>
> > I'm tired of telling that Unicode is not that perfect.

Of course not, but neither is the current multibyte with only marginal
support for unicode (many people actually need upper()/lower() )

> Another gottcha
> > with Unicode is the UTF-8 encoding (currently we use) consumes 3
> > bytes for each Kanji character, while other encodings consume only 2
> > bytes.

I think that for *storage* we should use SCSU (the Standard Compression
Scheme for Unicode).

> IMO 3/2 storage ratio could not be neglected for database use.

SCSU should solve that (actually it should use less than 2 bytes char
for encoding any single language)

> The rest of the world seems to select unicode as the way to handle
> different languages in the UI of programs. For example gnome supports
> nothing but unicode. How is that handled in your country? I know that you
> are tired of people who don't understand how difficult it is for you, but
> I really would like to know. Is gnome not used over there because of this?
>
> About storing data in the database, I would expect it to work with any
> encoding, just like I would expect pg to be able to store images in any
> format.
>
> I'll try to not mention unicode near you in the feature :-)

---------------
Hannu

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:

> 2. Reimplement gettext to use 1. and allow switching of language and
> encoding at run-time.
>
> 3. Implement Unicode collation algorithm and character classification
> routines that are aware of 1. Use that in place of system locale
> routines.

This sounds like you want to completely reimplement all of the locale handling
provided by the OS? That seems like a dead-end approach to me. There's no way
your handling will ever be as complete or as well optimized as some OS's.

Better to find ways to use the OS gettext and locale handling on platforms
that provide good interfaces. On platforms that don't provide good interfaces
either don't support the features or use some third party library to provide
a good implementation.

The only thing you really need in the database is a second parameter on all
the collation functions like strxfrm(col,locale) etc. Then functional indexes
take care of almost everything.

The only advantage to adding locales per-column and/or per-index is the
notational simplicity. Queries could do simple standard expressions and not
have to worry about calling strxfrm or other locale-specific functions all the
time. I'm not sure it's worth the complexity of having to deal with
"WHERE x>y" where x and y are in different locales though.

--
greg

On Tue, Nov 25, 2003 at 08:40:57PM +0900, Tatsuo Ishii wrote:
> > On Tue, 25 Nov 2003, Peter Eisentraut wrote:
> >
> > I've always thought unicode was enough to even represent Japanese. Then
> > the client encoding can be something else that we can convert to. In any
> > way, the encoding of the message catalog has to be known to the system so
> > it can be converted to the correct encoding for the client.
>
> I'm tired of telling that Unicode is not that perfect.

Maybe it should be explained what the problems really are,
instead of saying it "isn't perfect"?

From what I understand there is only a problem converting from
the "legacy" encoding to unicode, and the other way around, and
no problem if you stop doing the conversion.

The conversion problem is because what in an encoding is only
represented by 1 character can be several characters in unicode.

Some examples people might understand are:
- µ: In iso 8859-1 it's char 0xB5. In unicode it can be U+00B5 (micro
sign) or U+03BC (greek letter small mu)
- Å: ISO 8859-1: 0xC5. Unicode U+00C5 (latin capital letter a
with ring above) or U+212B (angstrom sign)
- The ohm sign vs the greek letter omega.
- Quotation marks: You have left double quote, right double
quote, and a few others.

> Another gottcha
> with Unicode is the UTF-8 encoding (currently we use) consumes 3
> bytes for each Kanji character, while other encodings consume only 2
> bytes. IMO 3/2 storage ratio could not be neglected for database use.

You can encode unicode in different ways, and UTF-8 is only one
of them. Is there a problem with using UCS-2 (except that it
would require more storage for ASCII)?

Kurt

Greg Stark writes:

> This sounds like you want to completely reimplement all of the locale handling
> provided by the OS? That seems like a dead-end approach to me. There's no way
> your handling will ever be as complete or as well optimized as some OS's.

Actually, I'm pretty sure it will be more complete. About the
optimization, we'll have to see.

> Better to find ways to use the OS gettext and locale handling on platforms
> that provide good interfaces.

There are no such platforms to my knowledge. The exception is some
version of glibc that provides undocumented interfaces to functionality
that is rumoured to do something that may or may not be relevant to what
we're doing.

> On platforms that don't provide good interfaces either don't support the
> features or use some third party library to provide a good
> implementation.

There are no such libraries. I keep hearing ICU, but that is much too
bloated.

--
Peter Eisentraut peter_e(at)gmx(dot)net

Peter Eisentraut kirjutas T, 25.11.2003 kell 21:13:
> Greg Stark writes:
>
> > This sounds like you want to completely reimplement all of the locale handling
> > provided by the OS? That seems like a dead-end approach to me. There's no way
> > your handling will ever be as complete or as well optimized as some OS's.
>
> Actually, I'm pretty sure it will be more complete. About the
> optimization, we'll have to see.
>
> > Better to find ways to use the OS gettext and locale handling on platforms
> > that provide good interfaces.
>
> There are no such platforms to my knowledge.

Unless you consider ICU (http://oss.software.ibm.com/icu/) as a
"platform" ;)

We will hardly ever be more complete than it.

> There are no such libraries. I keep hearing ICU, but that is much too
> bloated.

At least it is kind of "standard" and also something what will be
maintained for foreseeable future, it also has a compatible license and
is available on all platforms of interest to postgresql.

And I am not sure that this "bloat" will affect us too much unless we
want to start maintaining a parallel copy - glibc is much more bloated
than ICU .

But if you insist on rolling your own library, you can always use ICU to
write regression test to compare yours with ...

-------------
Hannu

Kurt Roeckx <Q(at)ping(dot)be> writes:
> You can encode unicode in different ways, and UTF-8 is only one
> of them. Is there a problem with using UCS-2 (except that it
> would require more storage for ASCII)?

UCS-2 is impractical without some *extremely* wide-ranging changes in
the backend. To take just the most obvious point, doesn't it require
allowing embedded zero bytes in text strings?

regards, tom lane

> About storing data in the database, I would expect it to work with any
> encoding, just like I would expect pg to be able to store images in any
> format.

What's stopping us supporting the other Unicode encodings, eg. UCS-16
which could save Japansese storage space.

Chris

Greg Stark <gsstark(at)mit(dot)edu> writes:
> The only advantage to adding locales per-column and/or per-index is the
> notational simplicity.

Well, actually, the reason we are interested in doing it is the SQL spec
demands it.

regards, tom lane

On Tue, Nov 25, 2003 at 04:19:05PM -0500, Tom Lane wrote:
>
> UCS-2 is impractical without some *extremely* wide-ranging changes in
> the backend. To take just the most obvious point, doesn't it require
> allowing embedded zero bytes in text strings?

If you're going to use unicode in the rest of the backend, you'll
have to be able to deal with them anyway. You can't use normal C
string functions.

Kurt