Re: verifying unicode locale support

Holger Klawitter <lists(at)klawitter(dot)de> writes:
> In order to avoid interaction with gcc, cat and others else I've written a
> new program, reading from a file.

After setting up the test case and duplicating your problem, I realized
I was being dense :-( ... this is a well-known issue. Need more
caffeine before answering bug reports obviously ...

The problem is that PG's upper() and lower() functions are based on
the C library's <ctype.h> functions (toupper() and tolower()), which of
course only work for single-byte character sets. So they cannot work on
UTF8 data.

There has been some talk of rewriting these functions to use the
<wctype.h> API where available, but no one's actually stepped up to the
plate and done it. IIRC the main sticking point was figuring out how to
get from whatever character encoding the database is using into the wide
character set representation the C library wants. There doesn't seem to
be a portable way of discovering exactly what the wchar encoding is
supposed to be for the current locale setting.

If you're interested in trying to fix this, check the pgsql-hackers
archives for the previous discussions. Searching for "wctype" would
probably find the relevant threads.

If you just want to get your work done, I'd suggest adopting a
single-byte encoding such as Latin1 for the database.

regards, tom lane