Python 3.1 support

Here's the patch to support Python >=3.1 with PL/Python. The
compatibility code is mostly in line with the usual 2->3 C porting
practice and is documented inline.

I needed to create an arguably weird hack to manage the regression
tests. Instead of creating a new expected file for pretty much every
test file and also for some input files (where Python syntax had
changed), a sed script creates a complete Python 3 compatible set of
input and output files. Doesn't look pretty but works quite well. If
anyone has a better idea, please let me know.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Here's the patch to support Python >=3.1 with PL/Python. The
> compatibility code is mostly in line with the usual 2->3 C porting
> practice and is documented inline.

There was considerable debate earlier about whether we wanted to treat
Python 3 as a separate PL so it could be available in parallel with
plpython 2, because of the user-level coding incompatibilities. It
looks like this patch simply ignores that problem. What is going to
happen to plpython functions that depend on 2.x behavior?

regards, tom lane

On Nov 12, 2009, at 12:54 PM, Peter Eisentraut wrote:

> Here's the patch to support Python >=3.1 with PL/Python.

:\

I was hoping to be able to use Python 3 to draw a clear distinction between plpython and the would-be "plpython3" that I've been working on. I understand that you're not in favor of a brand new implementation for Python 3. Despite my dislike for that position(well, it would seem to be in opposition to my initiative, so naturally =), I don't entirely disagree with your rationale[wrt doing things more incrementally]. For me, plpython has never been what I would call a pleasure to use, and many of the gripes that I have with it are, IMO, entrenched far enough into the implementation that any efforts to change it would(should? =) cause unacceptable breakage in user applications(?). Well, as far as additional Python interfaces are concerned, a lot of redundant functionality, but that's not even the half of it.

[I was hoping to get to a status message this weekend,
but it seems like I should follow-up here. =]

So here's where I'm at:
--
Mostly documentation improvements since I last pinged -hackers.
Still, *sigh*, filling in documentation and fighting bugs as I go.
Currently resolving a bug instantiating MD arrays from nested lists.
Once I'm "finished" with the docs, I'm going to start looking for refcount leaks.
No major additions or changes are planned, but I have been making some minor additions as I write more docs.

Overview/Features:
http://wiki.postgresql.org/wiki/WIP:plpython3
Documentation:
http://python.projects.postgresql.org/pldocs/plpython3.html
git repo[see the plpython3 branch]:
http://git.postgresql.org/gitweb?p=plpython3.git;a=summary

Most of the documented interfaces have tests. I only have two platforms at my disposal, so I do fear that this will not "just work" on all of PG's supported platforms. Specifically, I've ran the tests on freebsd/amd64 and Mac10.6/intel(of course 10.5 as well for some earlier revisions). [err, actually, it's been a while since I ran the tests on freebsd.]
--

plpython3 is turning out to be kinda beefy(~974K diff[eh, there is some fluff in there]), and I can't say that I've seen much interest in it, so I can't really blame anyone if -hackers ends up taking a pass on it. (python3 is too far away for most folk to care? folk are content with plpython?)

eh, cheers, either way. =)

On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > Here's the patch to support Python >=3.1 with PL/Python. The
> > compatibility code is mostly in line with the usual 2->3 C porting
> > practice and is documented inline.
>
> There was considerable debate earlier about whether we wanted to treat
> Python 3 as a separate PL so it could be available in parallel with
> plpython 2, because of the user-level coding incompatibilities. It
> looks like this patch simply ignores that problem.

Exactly how to package that is something to be determined by the
packagers, and we can give them the support they need. But first you
need code that works with Python 3, which is what this patch does.

> What is going to happen to plpython functions that depend on 2.x behavior?

The porting path from 2 to 3 is pretty well established. You first port
to 2.6, then remove all the old features, then move to 3.x. This is not
something we have to reinvent. The only significant differences that
you can't use in 2.6 without future imports are unicode literals and the
print function, both of which are not in common use in PL/Python.

On tor, 2009-11-12 at 18:42 -0700, James Pye wrote:
> For me, plpython has never been what I would call a pleasure to use,
> and many of the gripes that I have with it are, IMO, entrenched far
> enough into the implementation that any efforts to change it
> would(should? =) cause unacceptable breakage in user applications(?).

Has this list of gripes ever been brought up and discussed in this
forum?

On Nov 13, 2009, at 4:47 AM, Peter Eisentraut wrote:
> Has this list of gripes ever been brought up and discussed in this
> forum?

Some are TODOs, so in part by other people. Some were briefly touched on in the recent past discussions(around the time that I announced the WIP). Native typing vs conversion, function fragments vs function modules.
I don't think native typing has seen any actual discussion, but I do recall mentioning it as something that I wanted to do(implicitly griped?).

...

There is a difference in the situation from the discussion before. Prior, it was, "I would like to implement a new PL for Python 3 with these features", and now, it is, "I have implemented a new PL for Python 3 with these features".
Simply, -hackers can choose among moving forward with Python 3 support in plpython or going with "plpython3" or even both, I suppose(?). Naturally, I'm biased toward something that involves plpython3, so I don't think I can(should?) be of much help to -hackers as a Python & PG user in any such decision. Of course, excepting the provision of justifications for my implementation/design choices...

I would really love to see some input from Python users.

I certainly don't want to waste time trying to get something into pgsql that Python users don't want.

[here's a gripe that I haven't brought up as I think it is a matter of taste]

I find (plpython) trigger functions to be a bit odd. I think it's the way in which manipulation/suppression decisions are made in BEFORE ROW triggers(return "OK", "SKIP", etc).. [label this as opinion at this point as I have yet to be able to nail down what, specifically, is "wrong" or un-pythonic about them.]

Also, having distinct entry points to handle trigger events helps reduce potential errors by forcing the user to explicitly state the events that the trigger function can handle. Currently, in plpython, users *should* do sanity checks.

Function modules opened the door for implementing this in a natural way, multiple functions(entry points) in the function module.

http://python.projects.postgresql.org/pldocs/plpython3-programming.html#PLPYTHON3-FUNCTIONS-TRIGGERS

On fre, 2009-11-13 at 11:27 -0700, James Pye wrote:
> Some are TODOs, so in part by other people. Some were briefly touched
> on in the recent past discussions(around the time that I announced the
> WIP). Native typing vs conversion, function fragments vs function
> modules.

I'm of course only one user, but these two features don't excite me at
all, and certainly not enough to go through the pain of dealing with a
second implementation.

On Nov 15, 2009, at 6:37 AM, Peter Eisentraut wrote:

> but these two features don't excite me at all,

hrm.. at all?

I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...

AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.

OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)

[looking at the PL/Python todo list..]

Excepting DB-API and trusted, I believe all the current PL/Python TODOs are fulfilled or N/A in plpython3... ugh, the docs are not yet complete, but I like to think of them as "better" anyways. :P

> the pain of dealing with a second implementation.

What pain are you anticipating? Maintenance?

On sön, 2009-11-15 at 18:39 -0700, James Pye wrote:
> I can see how function modules might look like a half-step backwards from function fragments at first, but the benefits of a *natural* initialization section (the module body) was enough to convince me. The added value on the PL developer's side was also compelling. Tracebacks were trivial to implement, and there is no need to munge the function's source. It seemed like a win all around...

The question is whether it helps the user, not the implementer. As far
as I can tell, it just creates more typing for no benefit whatsoever.
Also, it's inconsistent with normal Python script files and with other
PLs.

> AFA native typing is concerned, I think the flexibility and potential it offers is useful, no? Already, plpython3 provides properties on PG's datetime types to access the date_part()'s of the object.
>
> OTOH, for folk who primarily use the PL to access functionality in Python modules(bindings), native typing may be of no direct utility as they will likely need to convert anyways. (If that's your common use-case, then the absence of interest in native typing is quite understandable.)

Right, if I use PL/Python, I do it because I want to use Python. I
don't need another PostgreSQL implementation on top of Python. The
maintenance effort required to keep those two consistent aside.

Again, I'm only one user. But so far I haven't seen anyone else speak
up here, and clearly accepting this for inclusion will need nontrivial
convincing.

> > the pain of dealing with a second implementation.
>
> What pain are you anticipating? Maintenance?

Right.

> Again, I'm only one user.  But so far I haven't seen anyone else speak
> up here, and clearly accepting this for inclusion will need nontrivial
> convincing.

Well, FWIW, I am excited about better type integration.

Also, I am a little skeptical about this patch. I am sorry if this has
already been discussed, but would this mean that I need to choose
whether pl/python is built against Python 2.* or Python 3.*?

-Nathan

Nathan Boley <npboley(at)gmail(dot)com> writes:
> Also, I am a little skeptical about this patch. I am sorry if this has
> already been discussed, but would this mean that I need to choose
> whether pl/python is built against Python 2.* or Python 3.*?

Yes. That's exactly what I was complaining about upthread. I'm not
a Python user, but from what I can gather of the 2-to-3 changes,
having to choose one at package build time is going to be a disaster.

regards, tom lane

On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
> Nathan Boley <npboley(at)gmail(dot)com> writes:
> > Also, I am a little skeptical about this patch. I am sorry if this has
> > already been discussed, but would this mean that I need to choose
> > whether pl/python is built against Python 2.* or Python 3.*?
>
> Yes. That's exactly what I was complaining about upthread. I'm not
> a Python user, but from what I can gather of the 2-to-3 changes,
> having to choose one at package build time is going to be a disaster.
>

Agreed. We really need to have a plpython and plpython3. Heck this would
play nicely too because we support backward compatibility but also
upward version differences.

Joshua D. Drake

> regards, tom lane
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
>> Yes. That's exactly what I was complaining about upthread. I'm not
>> a Python user, but from what I can gather of the 2-to-3 changes,
>> having to choose one at package build time is going to be a disaster.

> Agreed. We really need to have a plpython and plpython3.

Peter was concerned about duplicative maintenance effort, but what I
think this patch shows is that (at least for the near future) both
could be built from a single source file. What we need is configure
and makefile support to do that.

regards, tom lane

On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
> >> Yes. That's exactly what I was complaining about upthread. I'm not
> >> a Python user, but from what I can gather of the 2-to-3 changes,
> >> having to choose one at package build time is going to be a disaster.
>
> > Agreed. We really need to have a plpython and plpython3.
>
> Peter was concerned about duplicative maintenance effort, but what I
> think this patch shows is that (at least for the near future) both
> could be built from a single source file. What we need is configure
> and makefile support to do that.

Ahh, so we would have:

--enable-plpython2=/usr/bin/python2
--enable-plpython3=/usr/bin/python3

?

That seems reasonable if we can run both. Although I wonder if longer
term (2.x is going to be support a long time) we will end up with
frustration within the single source file trying to keep things
straight.

Joshua D. Drake

>
> regards, tom lane
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> On Wed, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
>> Peter was concerned about duplicative maintenance effort, but what I
>> think this patch shows is that (at least for the near future) both
>> could be built from a single source file.

> That seems reasonable if we can run both. Although I wonder if longer
> term (2.x is going to be support a long time) we will end up with
> frustration within the single source file trying to keep things
> straight.

Once it gets to the point where it's more trouble to keep them together
than not, we can split the source. But judging from this patch, a
single source file is the ticket for the moment.

regards, tom lane

> Here's the patch to support Python >=3.1 with PL/Python.  The
> compatibility code is mostly in line with the usual 2->3 C porting
> practice and is documented inline.

I took a cursory look at this patch and, while the logic seems sound
and roughly in line with the suggested python porting procedure, I'm
not quite certain what this implies for potential future patches.

For instance, if I wanted to write a type converter for bytea -> the
python 3 byte type would the expectation be that I ensure that it
works in Python 2? Or is an ifdef that ignores it in the case of
Python 2 OK, and we can just put a note in the docs.

Also, how far back do we want to maintain 2.x compatibility? 2.0? If I
wanted to submit a patch that makes use of the list sort method, do I
need to ensure that it can either use the cmp arguments or a key
argument?

What if I wanted to implement a set returning function that made use
of an iterators next() method. Would I just put ifdefs around the code
or a preprocessor definition that defines NEXT as next() for Python
2.x and __next__() for 3.x?

I guess that my first impression is that Python broke compatibility
for a reason, and that either plpython can't evolve, or it will
quickly become impossible to maintain. That being said, I mostly buy
the maintenance arguments from the previous discussion, but if we want
to have plpython and plpython3, a bunch of defines and ifdefs does not
seem like the best way to do this.

Would a better approach be to maintain compatibility layer? ie
plython_compat.h/c
plython2.c
plython3.c

Then patches that apply to a python3 can be applied to plython3.c and
any changed function can be ripped out of plython_compat and moved
into plpython2.

I'm sorry to snipe from the sidelines like this. If we didn't expect
plpython to evolve then this patch seems like the correct approach,
but there is clearly some desire to expand plpython and following this
path seems like it will end in a much more painful split in the future
or a necessary rewrite.

If there is some consensus that this is the best approach, then I will
do a more comprehensive review.

-Nathan

On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote:
> The question is whether it helps the user, not the implementer.

Sure, but do you have a patch waiting to implement tracebacks?

I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack.

What helped the implementer here would help the user.

> As far
> as I can tell, it just creates more typing for no benefit whatsoever.

"def main(*args):" is annoying, but not entirely lamentable...
It's explicit, as well(no need to document munging that occurs behind the scenes).

Also, compare the cases where you need to cache some initialized data:

if 'key' not in SD:
...
SD['key'] = my_newly_initialized_data
...

With function modules, SD is not needed as you have your module globals to keep your locally cached data in:

...
data = my_newly_initialized_data

def main(*args):
...

> Also, it's inconsistent with normal Python script files

Hinges on whether "normal" is actually normal.
I often use the __name__ convention in script files myself:

if __name__ == '__main__':
main(...)

That is, using that convention, the script can be import'd and used without executing the "script functionality". (It has proven to be very handy a few times now)

> and with other PLs.

I don't understand why that's a significant enough interest to note.

> I don't need another PostgreSQL implementation on top of Python.

Indeed, and I do understand that. That is, I have removed some features with that very thought in mind. (OTOH, I consider the date_part properties on datetime types to be special: too likely useful.)

[tho, "PostgreSQL implementation"? I think I understand what you were getting at, but..]

> The maintenance effort required to keep those two consistent aside.

I don't think there are many consistency issues here.
What did you have in mind?

> Again, I'm only one user. But so far I haven't seen anyone else speak up here, and clearly accepting this for inclusion will need nontrivial convincing.

Agreed. It would seem quite doomed.

At this point, I'm not going to try getting it into PG. (apparent futility and such)

On Nov 18, 2009, at 1:36 PM, James Pye wrote:
> At this point, I'm not going to try getting it into PG. (apparent futility and such)

ugh, on second thought, I think I've written a bit too much code to stop now. I'm going to get plpython3 as far as I can and submit it to the next commitfest.

On ons, 2009-11-18 at 09:48 -0800, Joshua D. Drake wrote:
> Although I wonder if longer
> term (2.x is going to be support a long time) we will end up with
> frustration within the single source file trying to keep things
> straight.

There are five million Python modules with C code out there with the
same problem. Considerable effort has been put in by Python upstream to
make the effort manageable. No one in their right mind is going to
create two separate source files just because in the future the mythical
differences will be too big, when clearly the effort is going into a
direction to reduce the differences.

If you look into the source file, there is already special code for
Python 2.2, 2.3, 2.4, 2.5, 2.6, and now 3.1. The chunk for 3.1 is a bit
bigger, but only a bit, and well, that's why it's 3.x and not 2.x. No
one has ever suggested, we might need to support Python 2.2 for a long
time, let's create a separate source file.

I agree, there will probably need to be some configuration/build support
on top of this, but that's something we should work out independently of
how to manage the source file.

On ons, 2009-11-18 at 08:43 -0800, Nathan Boley wrote:
> > Again, I'm only one user. But so far I haven't seen anyone else speak
> > up here, and clearly accepting this for inclusion will need nontrivial
> > convincing.
>
> Well, FWIW, I am excited about better type integration.

Let's clarify, as there are two different models being proposed here.
The first approach, which is currently implemented (and some patches
pending), is to convert a PostgreSQL type to the "nearest" Python type.
For example, text to string, int to int, array to list, timestamp to
datetime.datetime, etc. The other approach, which is what James Pye's
new implementation proposes (as I understand it), is to convert
PostgreSQL types into specially made Python objects, such as
Postgres.types.record or Postgres.types.timestamp.

> Also, I am a little skeptical about this patch. I am sorry if this has
> already been discussed, but would this mean that I need to choose
> whether pl/python is built against Python 2.* or Python 3.*?

Yeah, see later discussion about how to resolve this. But I think in
practice, unless you use lots of print statements in your stored
procedures (?!?), this problem is exaggerated.

On ons, 2009-11-18 at 11:32 -0800, Nathan Boley wrote:
> I took a cursory look at this patch and, while the logic seems sound
> and roughly in line with the suggested python porting procedure, I'm
> not quite certain what this implies for potential future patches.
>
> For instance, if I wanted to write a type converter for bytea -> the
> python 3 byte type would the expectation be that I ensure that it
> works in Python 2? Or is an ifdef that ignores it in the case of
> Python 2 OK, and we can just put a note in the docs.

Note that this is already implemented. The main point of the patch is
to provide a small compatibility layer so that these kinds of issues are
practically nonexistent. The fact that you didn't notice might prove
that the patch does its job. ;-)

> Also, how far back do we want to maintain 2.x compatibility? 2.0?

We handle this on an ad hoc basis. We currently support Python 2.2 and
later, and this cutoff exists -- this is my interpretation of history --
because 2.2 introduced iterators and no one bothered(?) to put ifdefs
around the code in PL/Python that provides iterator support. Over the
years, we will probably drop support for other older Python versions,
but there is no process or plan for that. Right now, the support for
Python 2.2 is about three lines, so it's not a bother, but when someone
comes and implements a major feature that, say, requires Python 2.3, we
can probably drop 2.2. But when the major feature requires 2.6, we
probably don't want to drop 2.5 quite yet at this time. It's a judgment
call.

> If I
> wanted to submit a patch that makes use of the list sort method, do I
> need to ensure that it can either use the cmp arguments or a key
> argument?

Any patch one is likely to submit will be a C patch, not a Python patch.
But anyway, the "key" argument was introduced in Python 2.4, and so we'd
have to come to a decision in the community about whether Python 2.3
support is worth keeping versus the value of that new feature. See above.

But anyway, this problem has nothing to do with my patch; it has already
existed in the same form forever.

> What if I wanted to implement a set returning function that made use
> of an iterators next() method. Would I just put ifdefs around the code
> or a preprocessor definition that defines NEXT as next() for Python
> 2.x and __next__() for 3.x?

Again, you would likely submit a C patch, and the iterator API is the
same between 2.x and 3.x.

> I guess that my first impression is that Python broke compatibility
> for a reason, and that either plpython can't evolve, or it will
> quickly become impossible to maintain.

I think this is an exaggeration of reality. Python 3 removed deprecated
features. There is a perfectly good migration path that covers most
code: Switch to Python 2.6, switch to the new features, remove the old
features, switch to Python 3.x. This applies both on the Python and the
C level. They did not break compatibility with the intention of making
every module author out there reimplement their thing from scratch.
Otherwise Python 2.6 would make very little sense at all.

Take a look at an example closer to home: PostgreSQL breaks C API
compatibility in almost every major release. We do this to remove cruft
and support new features. The intent is not to make Slony and PostGIS
and all the other modules reimplement their product from scratch every
time. They put in a few ifdefs, sometimes they complain about it ;-),
and then the problem is solved.

> That being said, I mostly buy
> the maintenance arguments from the previous discussion, but if we want
> to have plpython and plpython3, a bunch of defines and ifdefs does not
> seem like the best way to do this.

These ifdefs were not my idea. They are in some cases directly and in
some cases in spirit from the Python 2.6 header files, so they are the
official way to do this.

> Would a better approach be to maintain compatibility layer? ie
> plython_compat.h/c
> plython2.c
> plython3.c
>
> Then patches that apply to a python3 can be applied to plython3.c and
> any changed function can be ripped out of plython_compat and moved
> into plpython2.

As I tried to explain above, we have always had a rolling feature model
of sorts, even across various Python 2.x versions. If you want to try
it out, you could take the current source and split it up into
plpython22.c, plpython23.c, etc. and see if that becomes useful.

On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote:
> The other approach, which is what James Pye's
> new implementation proposes (as I understand it), is to convert
> PostgreSQL types into specially made Python objects, such as
> Postgres.types.record or Postgres.types.timestamp.

Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object?

One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial.

On ons, 2009-11-18 at 12:28 -0500, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > On Wed, 2009-11-18 at 12:06 -0500, Tom Lane wrote:
> >> Yes. That's exactly what I was complaining about upthread. I'm not
> >> a Python user, but from what I can gather of the 2-to-3 changes,
> >> having to choose one at package build time is going to be a disaster.
>
> > Agreed. We really need to have a plpython and plpython3.
>
> Peter was concerned about duplicative maintenance effort, but what I
> think this patch shows is that (at least for the near future) both
> could be built from a single source file. What we need is configure
> and makefile support to do that.

By the way, it occurred to me that having two different versions of
libpython loaded into the same process is probably not going to work
sanely. So whatever solution we come up with for the Python 3
transition, the possibilities for a jolly back-and-forth are probably
going to be quite limited.

On ons, 2009-11-18 at 13:36 -0700, James Pye wrote:
> On Nov 18, 2009, at 8:37 AM, Peter Eisentraut wrote:
> > The question is whether it helps the user, not the implementer.
>
> Sure, but do you have a patch waiting to implement tracebacks?
>
> I'd argue the reason it's never been done is due to the way procedures are currently managed in PL/Python. And *without some significant refactoring*, any patch fully implementing tracebacks is going to be a seriously ugly hack.
>
> What helped the implementer here would help the user.

But you wouldn't, for example, get away with breaking SQL (or even
improving it incompatibly) to facilitate a better elog.

> > As far
> > as I can tell, it just creates more typing for no benefit whatsoever.
>
> "def main(*args):" is annoying, but not entirely lamentable...
> It's explicit, as well(no need to document munging that occurs behind the scenes).
>
> Also, compare the cases where you need to cache some initialized data:
>
> if 'key' not in SD:
> ...
> SD['key'] = my_newly_initialized_data
> ...
>
>
> With function modules, SD is not needed as you have your module globals to keep your locally cached data in:
>
> ...
> data = my_newly_initialized_data
>
> def main(*args):
> ...

I can see that this creates other options for structuring code, but it
doesn't actually match my way of thinking. (Obviously, I'm biased, but
anyway.) I think of a PL/Python function as a Python script file stored
in the database. When you call it, arguments are passed just like a
Python script receives arguments from the shell. When Python scripts
want to share data, they might use a file (or perhaps a database server
in advanced cases) and do

if not file exists:
create the file
fill it with data

This is in my mind quite analogous to how the SD business works.

The analogy to your approach, as I understand it, would be that multiple
instances of the same script file will automatically share their global
variables. That could be quite interesting, actually, but it's not how
it works, and in most cases it's better that way.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> By the way, it occurred to me that having two different versions of
> libpython loaded into the same process is probably not going to work
> sanely.

Why not? There's no way they'd even know about each other. We tell
the loader not to make the symbols globally visible.

But in any case, my main concern here is that I don't want to have
to predetermine which python version a user of Red Hat/Fedora will
have to use. If they can only use one at a time, that's still a
good bit better than not having a choice at all.

regards, tom lane

On Nov 19, 2009, at 11:32 AM, Peter Eisentraut wrote:
> But you wouldn't, for example, get away with breaking SQL (or even
> improving it incompatibly) to facilitate a better elog.

This doesn't fit the situation.

I'm not breaking PL/Python. I'm trying to add PL/Python3. =)

> I think of a PL/Python function as a Python script file stored
> in the database.

For Python, I think that's a mistake. Python scripts are independent applications.

[tho, I think this does illuminate our perspectives...]

On tor, 2009-11-19 at 13:43 -0500, Tom Lane wrote:
> But in any case, my main concern here is that I don't want to have
> to predetermine which python version a user of Red Hat/Fedora will
> have to use. If they can only use one at a time, that's still a
> good bit better than not having a choice at all.

By the way, mod_wsgi supports Python 3 already (same patch as here, in
principle). From the Fedora wiki page, I gather that no one has really
looked into packaging that yet for Python 3, but if someone does, maybe
we can cross-inspire ourselves.

On tor, 2009-11-19 at 13:12 -0700, James Pye wrote:
> > I think of a PL/Python function as a Python script file stored
> > in the database.
>
> For Python, I think that's a mistake. Python scripts are independent applications.

Is there any precedent for the sort of behavior that you are
implementing, that is, automatic sharing of variables between
independent executions of the same source container?

On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote:
> Is there any precedent for the sort of behavior that you are
> implementing, that is, automatic sharing of variables between
> independent executions of the same source container?

import foo

# bar is a regular, def'd function.
foo.bar()

...

# even in another thread, doesn't matter..
foo.bar()

In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary).

A plpython3 function *is* a Python module.

On fre, 2009-11-20 at 01:20 -0700, James Pye wrote:
> On Nov 20, 2009, at 12:02 AM, Peter Eisentraut wrote:
> > Is there any precedent for the sort of behavior that you are
> > implementing, that is, automatic sharing of variables between
> > independent executions of the same source container?
>
> import foo
>
> # bar is a regular, def'd function.
> foo.bar()
>
> ...
>
> # even in another thread, doesn't matter..
> foo.bar()
>
>
> In either call, foo.bar()'s globals() is the same dictionary object(the foo module's dictionary).

That's not what I meant, because this is the same execution of the same
source container, with threads explicitly started somewhere. You could
do the same in a plpython function (in theory, at least).

What I mean is more like, you execute the same source file twice in a
row, and the global variables are saved for the second run.

On Nov 20, 2009, at 1:26 AM, Peter Eisentraut wrote:
> because this is the same execution

Hrm, not necessarily. foo could be imported by another, completely independent part of the program. foo is cached in sys.modules. bar() is executed and it's still the same globals(). shared.

Am 19.11.2009 18:01, schrieb James Pye:
> On Nov 19, 2009, at 3:12 AM, Peter Eisentraut wrote:
>> The other approach, which is what James Pye's
>> new implementation proposes (as I understand it), is to convert
>> PostgreSQL types into specially made Python objects, such as
>> Postgres.types.record or Postgres.types.timestamp.
>
> Convert is not a good word choice. The Datum of the parameter is stored inside a new Python object(that only holds a Datum). So more like "copied into Python memory", and associated with its respective type. Wrapped in a Python object?

Yes "wrapped" is the term commonly used for that. And I must say I like
it and I used plpy where I could.

> One cool thing about doing it this way, is that if you just pass parameters forward to a prepared statement, there's no type I/O overhead. Not a huge performance win for common cases, but if someone were passing larger arrays around, it could be quite beneficial.

Exactly and you have all the meta information about the original
postgres type. IIRC there were some thoughts of having something like
that in a DBAPI interface as well (similar for example to cx_Oracle).

Tino

On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> There was considerable debate earlier about whether we wanted to treat
> Python 3 as a separate PL so it could be available in parallel with
> plpython 2, because of the user-level coding incompatibilities. It
> looks like this patch simply ignores that problem. What is going to
> happen to plpython functions that depend on 2.x behavior?

I have a proposal for how to handle this, and a prototype patch
attached. This follows essentially what the CPython distribution itself
does, which will make this tolerably easy to follow for users.

We install plpython as plpython2.so or plpython3.so, depending on the
version used to build it. Then, plpython.so is a symlink to
plpython2.so.

We then create three language definition templates:

plpythonu -> plpython.so
plpython2u -> plpython2.so
plpython3u -> plpython3.so

In the far future we flip the default symlink to plpython3.so, maybe in
about 5 years when Python 2.x expires.

This gives the users the following options and scenarios:

- Existing users don't have to do anything, until maybe in five years
they will notice that their OS doesn't ship Python 2 anymore and they
will have to act anyway. In practice, by then they might have adjusted
their coding style to Python 2.6/2.7 norms and their functions will
migrate to 3.x without change.

- Users who know that they have heavily Python 2.x dependent code and
don't want to ever change it can make use of the plpython2u language
name, just like they should probably change their scripts to use
something like #!/usr/bin/python2.

- Users who want to dive into Python 3 can use the plpython3u language
name, which will basically keep working forever by today's standards.
Those users would probably also use #!/usr/bin/python3 or the like in
their scripts. In the far future they might like to remove the "3".

- Daredevils can change symlink manually and make plpython3.so the
default plpythonu implementation. Those people would probably also
make /usr/bin/python be version 3.

Comments?

On Fri, 2009-12-11 at 01:19 +0200, Peter Eisentraut wrote:
> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> > There was considerable debate earlier about whether we wanted to treat
> > Python 3 as a separate PL so it could be available in parallel with
> > plpython 2, because of the user-level coding incompatibilities. It
> > looks like this patch simply ignores that problem. What is going to
> > happen to plpython functions that depend on 2.x behavior?
>
> I have a proposal for how to handle this, and a prototype patch
> attached. This follows essentially what the CPython distribution itself
> does, which will make this tolerably easy to follow for users.
>
> We install plpython as plpython2.so or plpython3.so, depending on the
> version used to build it. Then, plpython.so is a symlink to
> plpython2.so.
>
> We then create three language definition templates:
>
> plpythonu -> plpython.so
> plpython2u -> plpython2.so
> plpython3u -> plpython3.so
>

>
> Comments?

Well as a Python guy... makes sense to me :)

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

I wrote:
> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> > There was considerable debate earlier about whether we wanted to treat
> > Python 3 as a separate PL so it could be available in parallel with
> > plpython 2, because of the user-level coding incompatibilities. It
> > looks like this patch simply ignores that problem. What is going to
> > happen to plpython functions that depend on 2.x behavior?
>
> I have a proposal for how to handle this, and a prototype patch
> attached. This follows essentially what the CPython distribution itself
> does, which will make this tolerably easy to follow for users.
>
> We install plpython as plpython2.so or plpython3.so, depending on the
> version used to build it. Then, plpython.so is a symlink to
> plpython2.so.

So here is the potentially final patch for this, including the original
port of plpython.c itself, build system adjustments, and documentation.

On Mon, Dec 14, 2009 at 1:42 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> I wrote:
>> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
>> > There was considerable debate earlier about whether we wanted to treat
>> > Python 3 as a separate PL so it could be available in parallel with
>> > plpython 2, because of the user-level coding incompatibilities.  It
>> > looks like this patch simply ignores that problem.  What is going to
>> > happen to plpython functions that depend on 2.x behavior?
>>
>> I have a proposal for how to handle this, and a prototype patch
>> attached.  This follows essentially what the CPython distribution itself
>> does, which will make this tolerably easy to follow for users.
>>
>> We install plpython as plpython2.so or plpython3.so, depending on the
>> version used to build it.  Then, plpython.so is a symlink to
>> plpython2.so.
>
> So here is the potentially final patch for this, including the original
> port of plpython.c itself, build system adjustments, and documentation.

I think you forgot to actually attach it...

...Robert

I wrote:
> On tor, 2009-11-12 at 16:06 -0500, Tom Lane wrote:
> > There was considerable debate earlier about whether we wanted to treat
> > Python 3 as a separate PL so it could be available in parallel with
> > plpython 2, because of the user-level coding incompatibilities. It
> > looks like this patch simply ignores that problem. What is going to
> > happen to plpython functions that depend on 2.x behavior?
>
> I have a proposal for how to handle this, and a prototype patch
> attached. This follows essentially what the CPython distribution itself
> does, which will make this tolerably easy to follow for users.
>
> We install plpython as plpython2.so or plpython3.so, depending on the
> version used to build it. Then, plpython.so is a symlink to
> plpython2.so.

So here is the potentially final patch for this, including the original
port of plpython.c itself, build system adjustments, and documentation.

Really attached this time.