Re: testing plpython3u on 9.0beta2

I received two errors (described below) in installing 9.0beta2 on Kubuntu
10.04 , RhodiumToad on IRC recommended I post them here.

I did not have a 2.x or 3.x python dev installed, but I was really only
interested in python3 via plython3u.
So...
sudo apt-get install python3-all-dev

Configure works fine...
"./configure --with-pgport=5433 --with-python --with-ossp-uuid --with-libxml
--with-libxslt --with-perl"
"make" fails while trying to build plpython, Cannot find python.h i
believe....(I sadly did not save the text of the error and have since
continued onward. But it was not finding a file while building plpython. I
believe I would have to uninstall a few things to reproduce.)
So I tried installing the 2.x dev. After running "sudo apt-get install
python-all-dev" then "make;make check;sudo make install", it all worke fine.

However, when issuing a "createlang plpython3u template1", I get
"createlang: language installation failed: ERROR: could not access file
"$libdir/plpython3": No such file or directory". But if I "createlang
plpython2u template1" first, then it will allow "createlang plpython3u
template1".

Am I doing something incorrect, or...?

(As an aside, functions created with LANGUAGE plpython3u appear to work.)

--
Chris Spotts
rfusca(at)gmail(dot)com

On Tue, Jun 22, 2010 at 10:41 PM, Chris <rfusca(at)gmail(dot)com> wrote:
> I received two errors (described below) in installing 9.0beta2 on Kubuntu
> 10.04 , RhodiumToad on IRC recommended I post them here.
> I did not have a 2.x or 3.x python dev installed, but I was really only
> interested in python3 via plython3u.
> So...
> sudo apt-get install python3-all-dev
> Configure works fine...
> "./configure --with-pgport=5433 --with-python --with-ossp-uuid --with-libxml
> --with-libxslt --with-perl"
> "make" fails while trying to build plpython, Cannot find python.h i
> believe....(I sadly did not save the text of the error and have since
> continued onward.  But it was not finding a file while building plpython.  I
> believe I would have to uninstall a few things to reproduce.)
> So I tried installing the 2.x dev.  After running "sudo apt-get install
> python-all-dev" then "make;make check;sudo make install", it all worke fine.
> However,  when issuing a "createlang plpython3u template1", I get
> "createlang: language installation failed: ERROR:  could not access file
> "$libdir/plpython3": No such file or directory".  But if I "createlang
> plpython2u template1" first, then it will allow "createlang plpython3u
> template1".
> Am I doing something incorrect, or...?
> (As an aside, functions created with LANGUAGE plpython3u appear to work.)

I can reproduce this, here. The problem seems to be that plpython
only build either plpython2.so or plython3.so, but both languages
expect a call handler called plython_call_handler. So once we load
the shared library for one language, the other language just grabs the
same call handler.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> I can reproduce this, here. The problem seems to be that plpython
> only build either plpython2.so or plython3.so, but both languages
> expect a call handler called plython_call_handler. So once we load
> the shared library for one language, the other language just grabs the
> same call handler.

We had better fix that --- I can definitely foresee installations
wanting to use both plpython2 and plpython3. This'd require a change in
the default contents of pg_pltemplate, though. Does it seem important
enough to bump catversion for?

regards, tom lane

On Wed, Jun 23, 2010 at 10:30 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> I can reproduce this, here.  The problem seems to be that plpython
>> only build either plpython2.so or plython3.so, but both languages
>> expect a call handler called plython_call_handler.  So once we load
>> the shared library for one language, the other language just grabs the
>> same call handler.
>
> We had better fix that --- I can definitely foresee installations
> wanting to use both plpython2 and plpython3.  This'd require a change in
> the default contents of pg_pltemplate, though.  Does it seem important
> enough to bump catversion for?

Yeah, I think so. As for using both plpython2 and plpython3, it looks
to me like right now you can only use one or the other. I think if
you somehow manage to install both, you're really just getting the
same one twice (I have not tested this, however).

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Wed, Jun 23, 2010 at 10:49 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Jun 23, 2010 at 10:30 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> I can reproduce this, here.  The problem seems to be that plpython
>>> only build either plpython2.so or plython3.so, but both languages
>>> expect a call handler called plython_call_handler.  So once we load
>>> the shared library for one language, the other language just grabs the
>>> same call handler.
>>
>> We had better fix that --- I can definitely foresee installations
>> wanting to use both plpython2 and plpython3.  This'd require a change in
>> the default contents of pg_pltemplate, though.  Does it seem important
>> enough to bump catversion for?
>
> Yeah, I think so.  As for using both plpython2 and plpython3, it looks
> to me like right now you can only use one or the other.  I think if
> you somehow manage to install both, you're really just getting the
> same one twice (I have not tested this, however).

So, what's the right thing to do here? Should we just fix it so that
creating the second language always fails, or should we try to make it
possible for both of them to coexist (which is probably a lot more
work)?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On ons, 2010-06-23 at 07:17 -0400, Robert Haas wrote:
> I can reproduce this, here. The problem seems to be that plpython
> only build either plpython2.so or plython3.so, but both languages
> expect a call handler called plython_call_handler. So once we load
> the shared library for one language, the other language just grabs the
> same call handler.

The problem is apparently that when CREATE LANGUAGE creates a language
from a pg_pltemplate entry, it creates the proname from the tmplhandler
name, and if it finds a fitting proname entry already, it used that one.
So when you create plpython2 first and plpython3 second, the pg_language
entries of the latter point to the pg_proc entries of the former.

If you fix that up manually (create additional pg_proc entries and fix
the pg_language entries to point there), it works better.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> The problem is apparently that when CREATE LANGUAGE creates a language
> from a pg_pltemplate entry, it creates the proname from the tmplhandler
> name, and if it finds a fitting proname entry already, it used that one.
> So when you create plpython2 first and plpython3 second, the pg_language
> entries of the latter point to the pg_proc entries of the former.

> If you fix that up manually (create additional pg_proc entries and fix
> the pg_language entries to point there), it works better.

The fix ought to be to change the function nmes used by plpython3 ...

regards, tom lane

On fre, 2010-06-25 at 10:17 -0400, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > The problem is apparently that when CREATE LANGUAGE creates a language
> > from a pg_pltemplate entry, it creates the proname from the tmplhandler
> > name, and if it finds a fitting proname entry already, it used that one.
> > So when you create plpython2 first and plpython3 second, the pg_language
> > entries of the latter point to the pg_proc entries of the former.
>
> > If you fix that up manually (create additional pg_proc entries and fix
> > the pg_language entries to point there), it works better.
>
> The fix ought to be to change the function nmes used by plpython3 ...

Right. What shall we do about the catversion?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On fre, 2010-06-25 at 10:17 -0400, Tom Lane wrote:
>> The fix ought to be to change the function nmes used by plpython3 ...

> Right. What shall we do about the catversion?

You could argue it either way. The number of beta testers with
plpython3 installations is probably very small, so I'm kinda leaning to
just changing the code without a catversion bump. OTOH, if we want to
encourage testing of pg_upgrade ...

regards, tom lane

> You could argue it either way. The number of beta testers with
> plpython3 installations is probably very small, so I'm kinda leaning to
> just changing the code without a catversion bump. OTOH, if we want to
> encourage testing of pg_upgrade ...

FWIW, the last bump has led to a lot of testing of pgupgrade.

--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com

On Wednesday 23 June 2010 16:30:54 Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > I can reproduce this, here. The problem seems to be that plpython
> > only build either plpython2.so or plython3.so, but both languages
> > expect a call handler called plython_call_handler. So once we load
> > the shared library for one language, the other language just grabs the
> > same call handler.
>
> We had better fix that --- I can definitely foresee installations
> wanting to use both plpython2 and plpython3. This'd require a change in
> the default contents of pg_pltemplate, though. Does it seem important
> enough to bump catversion for?
Has anybody actually researched if it is safe to run python2 and python3 in
the same address space? I wouldnt be surprised at all if that where
problematic.

Andres

On fre, 2010-06-25 at 23:44 +0200, Andres Freund wrote:
> Has anybody actually researched if it is safe to run python2 and
> python3 in the same address space?

You can't run plpython2 and plpython3 in the same session, because the
libraries are loaded with dlopen(RTLD_GLOBAL) (with RTLD_LOCAL it would
apparently work). But you can use them in different sessions on the
same database, for example.

On Fri, Jun 25, 2010 at 2:49 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On fre, 2010-06-25 at 10:17 -0400, Tom Lane wrote:
>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> > The problem is apparently that when CREATE LANGUAGE creates a language
>> > from a pg_pltemplate entry, it creates the proname from the tmplhandler
>> > name, and if it finds a fitting proname entry already, it used that one.
>> > So when you create plpython2 first and plpython3 second, the pg_language
>> > entries of the latter point to the pg_proc entries of the former.
>>
>> > If you fix that up manually (create additional pg_proc entries and fix
>> > the pg_language entries to point there), it works better.
>>
>> The fix ought to be to change the function nmes used by plpython3 ...
>
> Right.  What shall we do about the catversion?

We should go ahead and apply this, either with (my vote) or without
(Tom's vote) a catversion bump.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Josh Berkus wrote:
>
> > You could argue it either way. The number of beta testers with
> > plpython3 installations is probably very small, so I'm kinda leaning to
> > just changing the code without a catversion bump. OTOH, if we want to
> > encourage testing of pg_upgrade ...
>
> FWIW, the last bump has led to a lot of testing of pgupgrade.

And fixes, that will appear in 9.0 beta3. :-) Most fixes were related
to platform compile portability.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Josh Berkus wrote:
>>> You could argue it either way. The number of beta testers with
>>> plpython3 installations is probably very small, so I'm kinda leaning to
>>> just changing the code without a catversion bump. OTOH, if we want to
>>> encourage testing of pg_upgrade ...
>>
>> FWIW, the last bump has led to a lot of testing of pgupgrade.

> And fixes, that will appear in 9.0 beta3. :-) Most fixes were related
> to platform compile portability.

Well, if you think that pg_upgrade has changed materially since beta2,
that would be a good argument for getting some fresh testing for it,
which in turn argues for doing the catversion bump here.

regards, tom lane

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Josh Berkus wrote:
> >>> You could argue it either way. The number of beta testers with
> >>> plpython3 installations is probably very small, so I'm kinda leaning to
> >>> just changing the code without a catversion bump. OTOH, if we want to
> >>> encourage testing of pg_upgrade ...
> >>
> >> FWIW, the last bump has led to a lot of testing of pgupgrade.
>
> > And fixes, that will appear in 9.0 beta3. :-) Most fixes were related
> > to platform compile portability.
>
> Well, if you think that pg_upgrade has changed materially since beta2,
> that would be a good argument for getting some fresh testing for it,
> which in turn argues for doing the catversion bump here.

Attached are the changes since beta2; they are pretty minor. The good
news is I think all reporters eventually got it working. I assume using
it for beta3 would allow it work even better, and once you have use it
once, using it again is simple.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ None of us is going to be here forever. +

On fre, 2010-06-25 at 18:57 -0400, Peter Eisentraut wrote:
> On fre, 2010-06-25 at 23:44 +0200, Andres Freund wrote:
> > Has anybody actually researched if it is safe to run python2 and
> > python3 in the same address space?
>
> You can't run plpython2 and plpython3 in the same session, because the
> libraries are loaded with dlopen(RTLD_GLOBAL) (with RTLD_LOCAL it would
> apparently work). But you can use them in different sessions on the
> same database, for example.

By the way, I thought there had been some discussion about this in the
past, but I couldn't find it ...

Why are we using RTLD_GLOBAL?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Why are we using RTLD_GLOBAL?

Some guy named Eisentraut wanted it for plpython:
http://archives.postgresql.org/pgsql-hackers/2001-05/msg00563.php
http://archives.postgresql.org/pgsql-committers/2001-05/msg00283.php

Possibly that no longer applies, though. In general it seems like
our usage of loadable modules treats them all as independent objects,
so that not using RTLD_GLOBAL would be a more sensible policy. If it
won't break any of the existing PLs then I'm fine with removing it.

regards, tom lane