Re: Unicode string literals versus the world

On Thu, Apr 16, 2009 at 06:34:06PM +0300, Marko Kreen wrote:
> Which hints that you can aswell enter the pairs directly: \uxx\uxx.
> If I'd be language designer, I would not see any reason to disallow it.
>
> And anyway, at least mono seems to support it:
>
> using System;
> public class HelloWorld {
> public static void Main() {
> Console.WriteLine("<\uD800\uDF02>\n");
> }
> }
>
> It will output single UTF8 character. I think this should settle it.

I don't have any .net stuff installed so can't test; but C# is defined
to use UTF-16 as its internal representation so it would make sense if
the above gets treated as a single character internally. However, if it
used any other encoding the above should be treated as an error.

> The de-facto about Postgres is stdstr=off. Even if not, E'' strings
> are still better for various things, so it would be good if they also
> aquired unicode-capabilities.

OK, this seems independent of the U&'lit' discussion that started the
thread. Note that PG already supports UTF8; if you want the character
I've been using in my examples up-thread, you can do:

SELECT E'\xF0\x90\x8C\x82';

I have a feeling that this is predicated on the server_encoding being
set to "utf8" and this can only be done at database creation time.
Another alternative would be to use the convert_from function, i.e:

SELECT convert_from(E'\xF0\x90\x8C\x82', 'UTF8');

Never had to do this though, so there may be better options available.

> Python's internal representation is *not* UTF-16, but plain UCS2/UCS4,
> that is - plain 16 or 32-bit values. Seems your python is compiled with
> UCS2, not UCS4.

Cool, I didn't know that. I believe mine is UCS4 as I can do:

ord(u'\U00010302')

and I get 66306 back rather than an error.

> As I understand, in UCS2 mode it simply takes surrogate
> values as-is.

UCS2 doesn't have surrogate pairs, or at least I believe it's considered
a bug if you don't get an error when you present it with one.

> From ord() docs:
>
> If a unicode argument is given and Python was built with UCS2 Unicode,
> then the character’s code point must be in the range [0..65535]
> inclusive; otherwise the string length is two, and a TypeError will
> be raised.
>
> So only in UCS4 mode it detects surrogates and converts them to internal
> representation. (Which in Postgres case would be UTF8.)

I think you mean UTF-16 instead of UCS4; but otherwise, yes.

> Or perhaps it is partially UTF16 aware - eg. I/O routines do unterstand
> UTF16 but low-level string routines do not:
>
> print "<%s>" % u'\uD800\uDF02'
>
> seems to handle it properly.

Yes, I get this as well. It's all a bit weird, which is why I was
asking if "this a bug in Python, my understanding, or something else".

When I do:

python <<EOF | hexdump -C
print u"\uD800\uDF02"
EOF

to see what it's doing I get an error which I'm not expecting, hence I
think it's probably my understanding.

--
Sam http://samason.me.uk/