Re: Python 3.0 does not work with PL/Python

I have recently fixed the configure script to recognize Python 3.0. But
note that building and running PL/Python with Python 3.0 does not
actually work. It looks like several symbols have been removed or
changed. It would be good if the Python pundits around here could take
a look.

(I have found Python 3.0 to be very quick and easy to install from
source, in case your distribution doesn't have it packaged yet.)

On Thu, 2009-01-08 at 11:38 +0200, Peter Eisentraut wrote:
> I have recently fixed the configure script to recognize Python 3.0. But
> note that building and running PL/Python with Python 3.0 does not
> actually work. It looks like several symbols have been removed or
> changed. It would be good if the Python pundits around here could take
> a look.
>

Yeah I mentioned that to Bruce a couple of days ago. I am trying to fix
it. Or I should say, I am trying to get Alvaro to hold my hand while I
fix it.

Joshua D. Drake

> (I have found Python 3.0 to be very quick and easy to install from
> source, in case your distribution doesn't have it packaged yet.)
>
--
PostgreSQL
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Peter Eisentraut wrote:
> I have recently fixed the configure script to recognize Python 3.0. But
> note that building and running PL/Python with Python 3.0 does not
> actually work. It looks like several symbols have been removed or
> changed. It would be good if the Python pundits around here could take
> a look.
>
> (I have found Python 3.0 to be very quick and easy to install from
> source, in case your distribution doesn't have it packaged yet.)

Any progress on this?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Peter Eisentraut wrote:
>> I have recently fixed the configure script to recognize Python 3.0. But
>> note that building and running PL/Python with Python 3.0 does not
>> actually work. It looks like several symbols have been removed or
>> changed. It would be good if the Python pundits around here could take
>> a look.
>>
>> (I have found Python 3.0 to be very quick and easy to install from
>> source, in case your distribution doesn't have it packaged yet.)
>
> Any progress on this?

Won't happen for 8.4, it looks like.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> I have recently fixed the configure script to recognize Python 3.0. But
> note that building and running PL/Python with Python 3.0 does not
> actually work. It looks like several symbols have been removed or
> changed. It would be good if the Python pundits around here could take
> a look.

> (I have found Python 3.0 to be very quick and easy to install from
> source, in case your distribution doesn't have it packaged yet.)

I thought I would experiment with this a bit. I got past Python's
"configure; make; make install" okay, but got no further than here
with building PG:

checking for python... /home/tgl/python3.0.1/bin/python
checking for Python distutils module... ./configure: line 6946: 21044 Aborted "${PYTHON}" -c 'import distutils' 2>&-
no
configure: error: distutils module not found
$

Okay, but some research revealed that there does not exist any
working distutils for Python 3.0.1 yet:
http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-python-3/

If the language is still at the point where they're breaking fundamental
tools with each dot-release, I don't think it's really stable enough for
us to spend effort on :-(

I also found out that my favorite distro is just *starting* to think
about what it will take to migrate to Python 3, and they seem to think
that it's not going to be viable till around Fedora 13 (a year away):
https://www.redhat.com/archives/fedora-devel-list/2009-April/msg00085.html

So my conclusion is that Python 3.0 is much too wet behind the ears for
us to worry about in PG 8.4. I'd guess that we should come back to the
issue towards the end of 2009, and perhaps think about back-porting
after we have something working in 8.5.

BTW, there is some useful info here:
http://docs.python.org/3.0/howto/cporting.html
for whenever we do get around to trying to make the code work.
It looks like we'll need not-trivial code changes. But there's
no point until the language is a bit more stable.

regards, tom lane

Tom Lane wrote:

> I thought I would experiment with this a bit. I got past Python's
> "configure; make; make install" okay, but got no further than here
> with building PG:

Consequently, I have removed the Python 3.0 item from the open items
list and added a link to this thread to the TODO item that was already
there.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 4/4/09, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > I have recently fixed the configure script to recognize Python 3.0. But
> > note that building and running PL/Python with Python 3.0 does not
> > actually work. It looks like several symbols have been removed or
> > changed. It would be good if the Python pundits around here could take
> > a look.
>
> > (I have found Python 3.0 to be very quick and easy to install from
> > source, in case your distribution doesn't have it packaged yet.)
>
>
> I thought I would experiment with this a bit. I got past Python's
> "configure; make; make install" okay, but got no further than here
> with building PG:
>
> checking for python... /home/tgl/python3.0.1/bin/python
> checking for Python distutils module... ./configure: line 6946: 21044 Aborted "${PYTHON}" -c 'import distutils' 2>&-
> no
> configure: error: distutils module not found
> $
>
> Okay, but some research revealed that there does not exist any
> working distutils for Python 3.0.1 yet:
> http://regebro.wordpress.com/2009/02/01/setuptools-and-easy_install-for-python-3/
>
> If the language is still at the point where they're breaking fundamental
> tools with each dot-release, I don't think it's really stable enough for
> us to spend effort on :-(

Well, the point of 3.0 was to "break the world"...

> I also found out that my favorite distro is just *starting* to think
> about what it will take to migrate to Python 3, and they seem to think
> that it's not going to be viable till around Fedora 13 (a year away):
> https://www.redhat.com/archives/fedora-devel-list/2009-April/msg00085.html
>
> So my conclusion is that Python 3.0 is much too wet behind the ears for
> us to worry about in PG 8.4. I'd guess that we should come back to the
> issue towards the end of 2009, and perhaps think about back-porting
> after we have something working in 8.5.

It is not "wet" (the new interfaces should be stable), but it is break
from 2.x series. This means that users of PL/Python should not expect
PL/Python to automatically work with 3.0. Supporting 3.0 will be a new
feature. So it's OK to drop it from 8.4.

--
marko

Marko Kreen <markokr(at)gmail(dot)com> writes:
> On 4/4/09, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> So my conclusion is that Python 3.0 is much too wet behind the ears for
>> us to worry about in PG 8.4. I'd guess that we should come back to the
>> issue towards the end of 2009, and perhaps think about back-porting
>> after we have something working in 8.5.

> It is not "wet" (the new interfaces should be stable), but it is break
> from 2.x series.

Hm, did you read the link I cited? It's not so surprising that 3.0
should have broken distutils, but what I found distressing is that they
fixed distutils and then 3.0.1 broke it *again*. I stand by my opinion
that Python 3 isn't stable yet.

> This means that users of PL/Python should not expect PL/Python to
> automatically work with 3.0. Supporting 3.0 will be a new feature.
> So it's OK to drop it from 8.4.

One other thing that we'll have to seriously consider is whether we
should package python3 as a separate PL, so that people can keep using
their 2.x plpython functions without fear of breakage. I know that the
Fedora guys are currently debating whether to treat it that way, and
I suppose other distros are having or will soon have the same
conversation. Six months from now, there will be some precedents and
some track record for us to look at in making that choice.

regards, tom lane

On Apr 5, 2009, at 8:54 AM, Tom Lane wrote:
> Hm, did you read the link I cited? It's not so surprising that 3.0
> should have broken distutils, but what I found distressing is that
> they
> fixed distutils and then 3.0.1 broke it *again*. I stand by my
> opinion
> that Python 3 isn't stable yet.

Yeah, actually. From some of the talk I've seen on python-dev, it
sounds like 3.0.2 will be the last 3.0 release. 3.1 is in alpha, and
ready to start cleaning things up, afaict.

>> This means that users of PL/Python should not expect PL/Python to
>> automatically work with 3.0. Supporting 3.0 will be a new feature.
>> So it's OK to drop it from 8.4.
>
> One other thing that we'll have to seriously consider is whether we
> should package python3 as a separate PL, so that people can keep using
> their 2.x plpython functions without fear of breakage. I know that
> the
> Fedora guys are currently debating whether to treat it that way, and
> I suppose other distros are having or will soon have the same
> conversation. Six months from now, there will be some precedents and
> some track record for us to look at in making that choice.

I think this would be wise.

Any thoughts on the acceptability of a complete rewrite for Python 3?
I've been fiddling with a HEAD branch including the plpy code in a
github repo. (nah it dunt compile yet: bitrot and been busy with a 3.x
driver. ;)

James Pye <lists(at)jwp(dot)name> writes:
> Any thoughts on the acceptability of a complete rewrite for Python 3?

I've always thought that plpython.c was a bit on the hackish side.
If we do decide we have to make plpython2 and plpython3 separate
languages, it'd be pretty easy to just start over with a whole new
implementation for python3 ...

regards, tom lane

On Sun, Apr 5, 2009 at 7:10 PM, James Pye <lists(at)jwp(dot)name> wrote:
> Any thoughts on the acceptability of a complete rewrite for Python 3? I've
> been fiddling with a HEAD branch including the plpy code in a github repo.
> (nah it dunt compile yet: bitrot and been busy with a 3.x driver. ;)

I'd love to see this. I can't help with the C stuff, but I can try to
help test things as you go.

David Blewett

On Thu, Apr 30, 2009 at 8:30 AM, James Pye <lists(at)jwp(dot)name> wrote:
> On Apr 30, 2009, at 5:09 AM, David Blewett wrote:
>>
>> I'd love to see this.
>
> yep, once I get 0.9 of the driver out the door, I'll probably focus on this.
>
> It's the perfect time for a rewrite.. I really don't want to see the 2.x
> version ported. =\
>
>> I can't help with the C stuff, but I can try to
>> help test things as you go.
>
> I will probably be taking you up on that offer. =)
>
> I'll be dev'ing on osx with some occasional compiles/tests on fbsd, so an
> extra pair of eyes on builds/test runs would be a *huge* help..

I have access to the current version of ubuntu, centos 5, gentoo and
opensolaris. Just let me know what you'd like tested.

David Blewett

On Monday 06 April 2009 02:10:59 James Pye wrote:
> Any thoughts on the acceptability of a complete rewrite for Python 3?

http://www.joelonsoftware.com/articles/fog0000000069.html

On May 3, 2009, at 11:02 PM, Peter Eisentraut wrote:
>
> http://www.joelonsoftware.com/articles/fog0000000069.html

Good read. =)

However, complete rewrite being relative in this case:

WIP: http://github.com/jwp/postgresql-plpython3/tree/c804e693b6a0df98c0e5c465e75ba2e9014ebf37/src/pl/plpython3

That is, from pgsql's perspective it would be a complete rewrite.

From my perspective it would be a refactored version of plpy for
Python 3.x and pgsql 8.5/head. Additionally, plpy originally came from
plpythonu(7.3 or 7.4), but was *massively* refactored(read:
effectively rewritten).

I wouldn't want to start from scratch, actually.

Peter Eisentraut wrote:
> On Monday 06 April 2009 02:10:59 James Pye wrote:
> > Any thoughts on the acceptability of a complete rewrite for Python 3?
>
> http://www.joelonsoftware.com/articles/fog0000000069.html

You usually have to rewrite when you have not done refactoring as part
of development; PGDG does refactoring regularly.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian escribió:
> Peter Eisentraut wrote:
> > On Monday 06 April 2009 02:10:59 James Pye wrote:
> > > Any thoughts on the acceptability of a complete rewrite for Python 3?
> >
> > http://www.joelonsoftware.com/articles/fog0000000069.html
>
> You usually have to rewrite when you have not done refactoring as part
> of development; PGDG does refactoring regularly.

Except that plpython stagnates, save for minor hacks here and there.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Bruce Momjian escribió:
>
>> Peter Eisentraut wrote:
>>
>>> On Monday 06 April 2009 02:10:59 James Pye wrote:
>>>
>>>> Any thoughts on the acceptability of a complete rewrite for Python 3?
>>>>
>>> http://www.joelonsoftware.com/articles/fog0000000069.html
>>>
>> You usually have to rewrite when you have not done refactoring as part
>> of development; PGDG does refactoring regularly.
>>
>
> Except that plpython stagnates, save for minor hacks here and there.
>

Does Python 3 have some sort of usable sandbox that would mean we could
have a trusted plpython?

Otherwise, I'm not too keen simply to throw Python 2.x overboard until
it's no longer common on platforms people are likely to want to install
Postgres on, if that's what's implied by the original question.

cheers

andrew

No. Still no sandbox.

-Caleb

On 5/28/09 6:06 PM, "Andrew Dunstan" <andrew(at)dunslane(dot)net> wrote:

Does Python 3 have some sort of usable sandbox that would mean we could
have a trusted plpython?

Otherwise, I'm not too keen simply to throw Python 2.x overboard until
it's no longer common on platforms people are likely to want to install
Postgres on, if that's what's implied by the original question.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

On Friday 29 May 2009 03:53:17 Alvaro Herrera wrote:
> Bruce Momjian escribió:
> > Peter Eisentraut wrote:
> > > On Monday 06 April 2009 02:10:59 James Pye wrote:
> > > > Any thoughts on the acceptability of a complete rewrite for Python 3?
> > >
> > > http://www.joelonsoftware.com/articles/fog0000000069.html
> >
> > You usually have to rewrite when you have not done refactoring as part
> > of development; PGDG does refactoring regularly.
>
> Except that plpython stagnates, save for minor hacks here and there.

But that doesn't mean that there is anything wrong with it. Of course there
is, but those are isolated problems that can be fixed when desired. For
example, it might just be that those who use it don't have use of INOUT
parameters or table returns.

On Friday 29 May 2009 04:06:14 Andrew Dunstan wrote:
> Otherwise, I'm not too keen simply to throw Python 2.x overboard until
> it's no longer common on platforms people are likely to want to install
> Postgres on, if that's what's implied by the original question.

My guess is that we will need to keep around a Python 2.x version for at least
another three years, meaning two or three major PostgreSQL releases.

That also means that maintaining a separate, parallel code base for a Python 3
variant can only be acceptable if it gives major advantages.

On Thu, May 28, 2009 at 09:06:14PM -0400, Andrew Dunstan wrote:
> Does Python 3 have some sort of usable sandbox that would mean we could
> have a trusted plpython?

Not sure if people are aware of object-capability based approaches to
security. A guy called Tav has come up with some code that constrains
python (i.e. you could build a sandbox out of it) and punch holes in
it where needed (i.e. you want to be able to execute queries in the
database but otherwise not, say, touch the filesystem). The most recent
description I've found is:

http://tav.espians.com/paving-the-way-to-securing-the-python-interpreter.html

--
Sam http://samason.me.uk/

On Thu, May 28, 2009 at 9:06 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

> Does Python 3 have some sort of usable sandbox that would mean we could
> have a trusted plpython?

I brought this up last August [1]. Zope has a working sandbox that they
include in their distribution.

David Blewett

1.
http://archives.postgresql.org/message-id/9d1f8d830808041008v50104fd8p6181d5ddce85a6a@mail.gmail.com

David Blewett wrote:
> On Thu, May 28, 2009 at 9:06 PM, Andrew Dunstan <andrew(at)dunslane(dot)net
> <mailto:andrew(at)dunslane(dot)net>> wrote:
>
> Does Python 3 have some sort of usable sandbox that would mean we
> could have a trusted plpython?
>
>
> I brought this up last August [1]. Zope has a working sandbox that
> they include in their distribution.
>
>
> 1.
> http://archives.postgresql.org/message-id/9d1f8d830808041008v50104fd8p6181d5ddce85a6a@mail.gmail.com
>

How many python installations have this gadget? If the answer is "not
many" then it's not much good to us, unless someone wants to create
PL/zope-sandbox. Really, something like this should be part of a
standard python installation.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> David Blewett wrote:
>> I brought this up last August [1]. Zope has a working sandbox that
>> they include in their distribution.
>> http://archives.postgresql.org/message-id/9d1f8d830808041008v50104fd8p6181d5ddce85a6a@mail.gmail.com

> How many python installations have this gadget? If the answer is "not
> many" then it's not much good to us, unless someone wants to create
> PL/zope-sandbox. Really, something like this should be part of a
> standard python installation.

Yeah. For one thing, how much trust can you put in a security mechanism
that hasn't been accepted upstream? Given the history of this sort of
thing in Python, I'm not prepared to just assume that Zope got it right.

regards, tom lane

On Fri, 2009-05-29 at 11:12 +0300, Peter Eisentraut wrote:
> On Friday 29 May 2009 03:53:17 Alvaro Herrera wrote:
> > Bruce Momjian escribió:
> > > Peter Eisentraut wrote:
> > > > On Monday 06 April 2009 02:10:59 James Pye wrote:
> > > > > Any thoughts on the acceptability of a complete rewrite for Python 3?
> > > >
> > > > http://www.joelonsoftware.com/articles/fog0000000069.html
> > >
> > > You usually have to rewrite when you have not done refactoring as part
> > > of development; PGDG does refactoring regularly.
> >
> > Except that plpython stagnates, save for minor hacks here and there.
>
> But that doesn't mean that there is anything wrong with it. Of course there
> is, but those are isolated problems that can be fixed when desired. For
> example, it might just be that those who use it don't have use of INOUT
> parameters or table returns.

Yeah. And I _almost_ fixed those. Just did not have time to make the
final push to clean up things enough to be acceptable for patching back
into core.

And fixing those actually meant refactoring those parts into using newer
structures and functions :)

And I also think that pl/python, even for python 2.x does need lots of
refactoring in most places in order to be maintainable.

--
Hannu Krosing http://www.2ndQuadrant.com
PostgreSQL Scalability and Availability
Services, Consulting and Training

On May 29, 2009, at 1:17 AM, Peter Eisentraut wrote:
> On Friday 29 May 2009 04:06:14 Andrew Dunstan wrote:
>> Otherwise, I'm not too keen simply to throw Python 2.x overboard
>> until
>> it's no longer common on platforms people are likely to want to
>> install
>> Postgres on, if that's what's implied by the original question.
>
> My guess is that we will need to keep around a Python 2.x version
> for at least
> another three years, meaning two or three major PostgreSQL releases.

Yeah, I wasn't meaning to imply tossing 2.x out...