pgsql: Add Unicode support in PL/Python PL/Python now accepts Unicode

Log Message:
-----------
Add Unicode support in PL/Python

PL/Python now accepts Unicode objects where it previously only accepted string
objects (for example, as return value). Unicode objects are converted to the
PostgreSQL server encoding as necessary.

This change is also necessary for future Python 3 support, which treats all
strings as Unicode objects.

Since this removes the error conditions that the plpython_unicode test file
tested for, the alternative result files are no longer necessary.

Modified Files:
--------------
pgsql/src/pl/plpython:
plpython.c (r1.128 -> r1.129)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/plpython.c?r1=1.128&r2=1.129)
pgsql/src/pl/plpython/expected:
README (r1.1 -> r1.2)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/expected/README?r1=1.1&r2=1.2)
plpython_trigger.out (r1.4 -> r1.5)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/expected/plpython_trigger.out?r1=1.4&r2=1.5)
plpython_unicode.out (r1.2 -> r1.3)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/expected/plpython_unicode.out?r1=1.2&r2=1.3)
pgsql/src/pl/plpython/sql:
plpython_trigger.sql (r1.3 -> r1.4)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/sql/plpython_trigger.sql?r1=1.3&r2=1.4)
plpython_unicode.sql (r1.1 -> r1.2)
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/sql/plpython_unicode.sql?r1=1.1&r2=1.2)

Removed Files:
-------------
pgsql/src/pl/plpython/expected:
plpython_unicode_2.out
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/expected/plpython_unicode_2.out)
plpython_unicode_3.out
(http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/src/pl/plpython/expected/plpython_unicode_3.out)

petere(at)postgresql(dot)org (Peter Eisentraut) writes:
> Log Message:
> -----------
> Add Unicode support in PL/Python

The buildfarm seems quite unhappy with this patch.

regards, tom lane

On lör, 2009-09-12 at 21:34 -0400, Tom Lane wrote:
> petere(at)postgresql(dot)org (Peter Eisentraut) writes:
> > Log Message:
> > -----------
> > Add Unicode support in PL/Python
>
> The buildfarm seems quite unhappy with this patch.

I am completely unable to reproduce the buildfarm failures, even with
different Python versions and the exact configure options that the
buildfarm instances use. Does anyone have an affected build and wants
to work through this with me?

Peter Eisentraut wrote:
> On lör, 2009-09-12 at 21:34 -0400, Tom Lane wrote:
>
>> petere(at)postgresql(dot)org (Peter Eisentraut) writes:
>>
>>> Log Message:
>>> -----------
>>> Add Unicode support in PL/Python
>>>
>> The buildfarm seems quite unhappy with this patch.
>>
>
> I am completely unable to reproduce the buildfarm failures, even with
> different Python versions and the exact configure options that the
> buildfarm instances use. Does anyone have an affected build and wants
> to work through this with me?
>
>

I am going out shortly, but if you send me your ssh PK before I go I
will set you an account up on dungbeetle which is failing and you can
play there.

cheers

andrew

On sön, 2009-09-13 at 10:07 -0400, Andrew Dunstan wrote:
> >>> Add Unicode support in PL/Python
> >> The buildfarm seems quite unhappy with this patch.
> > I am completely unable to reproduce the buildfarm failures, even with
> > different Python versions and the exact configure options that the
> > buildfarm instances use. Does anyone have an affected build and wants
> > to work through this with me?
> I am going out shortly, but if you send me your ssh PK before I go I
> will set you an account up on dungbeetle which is failing and you can
> play there.

This is fixed now.

The reason this was not reproducible manually but showed up so
dramatically in the build farm is that almost all buildfarm machines use
SQL_ASCII as server encoding. While it's evidently good that we have
some machines testing that, we should probably also have more machines
testing the UTF8 and other code paths. Also, perhaps I'm missing
something, but it could have helped if the buildfarm logs showed the
locale and/or encoding somewhere.

Peter Eisentraut wrote:
> On sön, 2009-09-13 at 10:07 -0400, Andrew Dunstan wrote:
>
>>>>> Add Unicode support in PL/Python
>>>>>
>>>> The buildfarm seems quite unhappy with this patch.
>>>>
>>> I am completely unable to reproduce the buildfarm failures, even with
>>> different Python versions and the exact configure options that the
>>> buildfarm instances use. Does anyone have an affected build and wants
>>> to work through this with me?
>>>
>> I am going out shortly, but if you send me your ssh PK before I go I
>> will set you an account up on dungbeetle which is failing and you can
>> play there.
>>
>
> This is fixed now.
>
> The reason this was not reproducible manually but showed up so
> dramatically in the build farm is that almost all buildfarm machines use
> SQL_ASCII as server encoding. While it's evidently good that we have
> some machines testing that, we should probably also have more machines
> testing the UTF8 and other code paths. Also, perhaps I'm missing
> something, but it could have helped if the buildfarm logs showed the
> locale and/or encoding somewhere.
>
>
>

Support for various locales and encodings is a relatively recent
addition to the buildfarm, and many members have not yet updated. But
dungbeetle has, for example. And you can see tests there run both in
C/SQL_ASCII and en_US.utf8. See
<http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=dungbeetle&dt=2009-09-12%2016:44:02>
I know some of the boxes run at Sun (e.g. gothic_moth) also do checks in
several locale/encoding combinations.

cheers

andrew