Re: plpython3

Here's my latest patch. I'm going to submit to the open commitfest. (hrm, it's 189K compressed, should I just let people use git in the future?)

Known issues:

- Documentation has gaps.
- Code formatting/ws is not consistent with PG's.

Any feedback would be appreciated.

Excepting trusted and DB-API, this patch fulfills all of the current PL/Python TODOs for Python 3, and adds *many* new features(See the documentation, and the WIP wiki page).

plpython3 is intentionally incompatible with plpython as Python 3 is intentionally incompatible with Python 2. Be sure to see the recent discussion in the other Python 3 thread[1]. =)

New Docs:
http://python.projects.postgresql.org/pldocs/plpython3.html

Overview:
http://wiki.postgresql.org/wiki/WIP:plpython3

git(plpython3 branch):
http://git.postgresql.org/gitweb?p=plpython3.git;a=summary

Past threads on the subject:

http://archives.postgresql.org/pgsql-hackers/2009-05/msg01376.php
http://archives.postgresql.org/pgsql-hackers/2009-07/msg01519.php
http://archives.postgresql.org/pgsql-hackers/2009-08/msg01505.php

[1] http://archives.postgresql.org/pgsql-hackers/2009-11/msg00547.php

On Nov 19, 2009, at 5:41 PM, James Pye wrote:
> Here's my latest patch.

Fixed a lot of memory/reference leaks, added some minor features(mostly around Arrays), and filled in more documentation.

At this point, I don't have any more minor features in mind(save extending Postgres.notify when the payload patch hits), so I'm just doing finish work(improvements/clarifications to docs, message strings, and maybe some makefile work).

On Sun, Dec 13, 2009 at 5:02 PM, James Pye <lists(at)jwp(dot)name> wrote:
> On Nov 19, 2009, at 5:41 PM, James Pye wrote:
>> Here's my latest patch.
>
> Fixed a lot of memory/reference leaks, added some minor features(mostly around Arrays), and filled in more documentation.
>
> At this point, I don't have any more minor features in mind(save extending Postgres.notify when the payload patch hits), so I'm just doing finish work(improvements/clarifications to docs, message strings, and maybe some makefile work).

I'm almost afraid to write anything at all about this patch for fear
of being branded a nattering nabob of negativity (see other thread:
damage control mode) but hopefully if I'm full of it (or not) others
will write in and set me straight (or confirm my thinking, whichever
is appropriate). Anyhow, I started by reviewing the past threads on
this patch, to which the author helpfully provided links:

> Past threads on the subject:
>
> http://archives.postgresql.org/pgsql-hackers/2009-05/msg01376.php
> http://archives.postgresql.org/pgsql-hackers/2009-07/msg01519.php
> http://archives.postgresql.org/pgsql-hackers/2009-08/msg01505.php

I think we should be clear that this patch doesn't have a great deal
to do with Python 3, since Peter Eisentraut has already patched the
existing code to support Python 3. It is, rather, a reimplementation
of PL/python, and accordingly it ought to be called pl/newpython or
perhaps pl/pye-thon (sorry, couldn't resist). Peter Eisentraut has
made it pretty clear that he would prefer to see us maintain and
enhance the existing implementation rather than starting over, and
even if we did start over, it seems from the above threads that we'd
still need to maintain the existing code for quite a while (if not
forever).

So it seems to me that the threshold question for this patch is - do
we think it's a good idea to maintain two implementations of PL/python
in core?

...Robert

> So it seems to me that the threshold question for this patch is - do
> we think it's a good idea to maintain two implementations of PL/python
> in core?

Not really, no. This is why we need PGAN ;-)

If the new implementation is *better* that the existing PL/python, I
could see eventually replacing it. It wouldn't be the first time that a
rewrite exceeded the original tool.

However, I'm not in a position to judge quality.

--Josh Berkus

On Tue, 2010-01-12 at 20:06 -0800, Josh Berkus wrote:
> > So it seems to me that the threshold question for this patch is - do
> > we think it's a good idea to maintain two implementations of PL/python
> > in core?
>
> Not really, no. This is why we need PGAN ;-)
>
> If the new implementation is *better* that the existing PL/python, I
> could see eventually replacing it. It wouldn't be the first time that a
> rewrite exceeded the original tool.

I think it is important to remember that the current version of
PL/python is pretty weak compared to its counter parts (Specifically
PL/Perl). If the new version, is adequately written to community
standards and increases PL/Python's capabilities we need to seriously
consider it.

If we can address any issues with this module, let's commit it as
Pl/pythonng3 or something.

Anyway, I am +1 on reviewing this patch for viability.

I would love to never touch plPerl for advanced procedures again.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 09:47 -0800, Joshua D. Drake wrote:
> I think it is important to remember that the current version of
> PL/python is pretty weak compared to its counter parts (Specifically
> PL/Perl).

How so?

On Wed, 2010-01-13 at 19:53 +0200, Peter Eisentraut wrote:
> On ons, 2010-01-13 at 09:47 -0800, Joshua D. Drake wrote:
> > I think it is important to remember that the current version of
> > PL/python is pretty weak compared to its counter parts (Specifically
> > PL/Perl).
>
> How so?

O.k. you may have just called me on an unintentional bluff. My knowledge
of plpython is dated. I just tested some of the things (like in/out) and
they appear to work at least on 8.4).

My argument would be now, what is the benefit of the James Pye version
over our version. James can you illustrate succinctly why we should be
supporting a new version?

If there is, I am still all for it, but I am a python bigot.

Sincerely,

Joshua D. Drake

>
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

> My argument would be now, what is the benefit of the James Pye version
> over our version. James can you illustrate succinctly why we should be
> supporting a new version?
>
> If there is, I am still all for it, but I am a python bigot.

Yeah, it's just my viewpoint that we don't want 2 python procedural
languages in core. One should be in core, and one should go on
pgFoundry/PGAN. Which ever one is "better" by some clear definition
should go in core.

As I said, I have no opinion about whether Pye's PLpython should replace
the existing because I'm in no position to judge.

--Josh Berkus

On Wed, Jan 13, 2010 at 1:16 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>
>> My argument would be now, what is the benefit of the James Pye version
>> over our version. James can you illustrate succinctly why we should be
>> supporting a new version?
>>
>> If there is, I am still all for it, but I am a python bigot.
>
> Yeah, it's just my viewpoint that we don't want 2 python procedural
> languages in core.  One should be in core, and one should go on
> pgFoundry/PGAN.  Which ever one is "better" by some clear definition
> should go in core.

That was my thinking also. There are a couple of reasons to think
that throwing over the current implementation for an new one may not
be the right thing to do.

1. It's not just a rewrite, it's an incompatible rewrite that will
present significant user-visible behavioral differences. So replacing
the current implementation wholesale would produce massive breakage
for anyone actually using PL/python in production.

2. Peter Eisentraut, who has put significant time into improving
PL/python for this release (see the commit logs), and who is the ONLY
committer to work on improving PL/python for this release, has made it
clear that he prefers the current implementation.

Given these two facts, it's hard for me to see how we could decide to
REMOVE the current implementation and replace it with the new one. So
the most we could do is maintain them side by side, and then you have
to ask, why? It will actually be much easier for James Pye to
maintain his code outside core, and if it turns out to be popular and
people come back to us and say "hey, why isn't this the default?" then
we can revisit the issue. Sure, his code won't get as much exposure
that way, but it's been posted to the mailing list several times now
over a period of 8 months and nobody has said "oh, wow, this is
great". The most we've gotten is several versions of this exchange:

Somebody: We should really consider this code, the current code isn't very good.
Peter: What's wrong with it?
Somebody: <describes some problem, usually fairly vaguely>
Peter: Why can't we fix that in the existing code base?
<thread ends>

I'm not saying a complete rewrite isn't the way to go - I'm just
saying that there isn't much evidence at this point that it's really
necessary, and I don't think we want to maintain two copies of
PL/python unless we're really sure that the new implementation is an
improvement.

...Robert

On Jan 13, 2010, at 11:08 AM, Joshua D. Drake wrote:
> My argument would be now, what is the benefit of the James Pye version
> over our version. James can you illustrate succinctly why we should be
> supporting a new version?

Doing so, succinctly, is unfortunately difficult.
It is primarily a matter of comparing features, AFAICT. And, furthermore, some features may not be useful to some users.

It exposes additional functionality that should *not* be incrementally developed in plpython as it would break applications. This was the point of trying to move forward with it for Python 3.

Function Modules:
- Does away with the need for GD/SD (more natural Python environment).
- Allows tracebacks (tracebacks are useful, right?) to implemented easily.
- Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
- Helps to promote the Python tenet of being explicit.

Native Typing:
- Provides PG type introspection not available in any other PL, AFAIK.
- Improves efficiency in some cases (conversion must be _explicitly_ called for)
- MD Array support.
- Composites are a sequence and a mapping.

Other features: http://wiki.postgresql.org/wiki/WIP:plpython3

Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.

I'm not sure this fulfills your request, but, hopefully, it's a start.

On Wed, 2010-01-13 at 13:06 -0700, James William Pye wrote:
> On Jan 13, 2010, at 11:08 AM, Joshua D. Drake wrote:
> > My argument would be now, what is the benefit of the James Pye version
> > over our version. James can you illustrate succinctly why we should be
> > supporting a new version?
>
>
> Doing so, succinctly, is unfortunately difficult.
> It is primarily a matter of comparing features, AFAICT. And, furthermore, some features may not be useful to some users.
>
> It exposes additional functionality that should *not* be incrementally developed in plpython as it would break applications. This was the point of trying to move forward with it for Python 3.
>
> Function Modules:
> - Does away with the need for GD/SD (more natural Python environment).
> - Allows tracebacks (tracebacks are useful, right?) to implemented easily.
> - Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
> - Helps to promote the Python tenet of being explicit.
>
> Native Typing:
> - Provides PG type introspection not available in any other PL, AFAIK.
> - Improves efficiency in some cases (conversion must be _explicitly_ called for)
> - MD Array support.
> - Composites are a sequence and a mapping.
>
> Other features: http://wiki.postgresql.org/wiki/WIP:plpython3
>
>
> Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.
>
> I'm not sure this fulfills your request, but, hopefully, it's a start.

It does actually. Now, hackers... as a Python guy I can say these things
are truly useful, to a Python programmer trying to use Python as a
procedural language with PostgreSQL.

What do we think?

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 12:12 -0800, Joshua D. Drake wrote:
> On Wed, 2010-01-13 at 13:06 -0700, James William Pye wrote:
> > Function Modules:
> > - Does away with the need for GD/SD (more natural Python environment).
> > - Allows tracebacks (tracebacks are useful, right?) to implemented easily.
> > - Does *not* expose a bastardized variant of the language by pretending that "modules/script files" can return and yield.
> > - Helps to promote the Python tenet of being explicit.
> >
> > Native Typing:
> > - Provides PG type introspection not available in any other PL, AFAIK.
> > - Improves efficiency in some cases (conversion must be _explicitly_ called for)
> > - MD Array support.
> > - Composites are a sequence and a mapping.
> >
> > Other features: http://wiki.postgresql.org/wiki/WIP:plpython3
> >
> >
> > Aside from function modules and native typing, many of plpython3's features could be implemented incrementally. However, I had a chance to sprint and they are available now in a new implementation. I did so, rather than improving plpython, because I believe that native typing and function modules are very useful.
> >
> > I'm not sure this fulfills your request, but, hopefully, it's a start.
>
> It does actually. Now, hackers... as a Python guy I can say these things
> are truly useful, to a Python programmer trying to use Python as a
> procedural language with PostgreSQL.

The problem I'm having with this discussion is that every time someone
asks what the supposed advantages of this new Python PL are, a feature
list like the above is dumped, 75% of which is subjective and tends to
use semi-buzzwords, such that then someone else who by his own admission
isn't completely up to date on things says, sure, that sounds great.
Who wouldn't like a "more natural Python environment", "native typing"
and "efficiency", and maybe even "explicitness"? The current PL/Python
also has, arguably, a more natural Python environment, native typing,
efficiency, and explicitness. So there you go. Now what?

On Wed, 2010-01-13 at 23:27 +0200, Peter Eisentraut wrote:

> The problem I'm having with this discussion is that every time someone
> asks what the supposed advantages of this new Python PL are, a feature
> list like the above is dumped, 75% of which is subjective and tends to
> use semi-buzzwords, such that then someone else who by his own admission
> isn't completely up to date on things says, sure, that sounds great.
> Who wouldn't like a "more natural Python environment", "native typing"
> and "efficiency", and maybe even "explicitness"? The current PL/Python
> also has, arguably, a more natural Python environment, native typing,
> efficiency, and explicitness. So there you go. Now what?

Peter, are you interested in the benefit of the project, or your own
ego? Take it easy and let's see if we can be productive here.

I was just stating that James's explanation was good and looking for
more information from a reviewer. Secondly nobody (at least I am not) is
suggesting we dump the current PL/Python, so your hard work is not going
to be lost.

The only thing I am currently looking for is an objective review of the
patch based on the benefits it provides. I can tell you that if the Pye
patch provides stack trace capability and the current PL does not. That
right there is enough for me to push a +1 for review and possible
inclusion.

If you feel what James is saying is actually trite and more buzzword
compliant I welcome the opportunity to see a comparative of the current
status of PL/Python versus the Pye patch from you. If nothing else it is
a learning opportunity for all involved.

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On ons, 2010-01-13 at 13:33 -0800, Joshua D. Drake wrote:
> The only thing I am currently looking for is an objective review of the
> patch based on the benefits it provides.

Right, but I was opining that such a vague feature listing is not
adequate for that.

> I can tell you that if the Pye
> patch provides stack trace capability and the current PL does not. That
> right there is enough for me to push a +1 for review and possible
> inclusion.

That is the case.

> If you feel what James is saying is actually trite and more buzzword
> compliant I welcome the opportunity to see a comparative of the current
> status of PL/Python versus the Pye patch from you. If nothing else it is
> a learning opportunity for all involved.

There was extensive discussion about some of the design decisions
upthread, so any reviewer should review those. It may end up being a
agree-to-disagree situation.

On Jan 13, 2010, at 2:27 PM, Peter Eisentraut wrote:
> The problem I'm having with this discussion is that every time someone
> asks what the supposed advantages of this new Python PL are, a feature
> list like the above is dumped,

I agree that this is unfortunate, but how else can we to discuss the advantages? It boils down to comparing a couple feature lists, and *maybe* some implementation details. No?

> 75% of which is subjective and tends to use semi-buzzwords,

You say "semi-buzzwords", I say "names". I have to call "it" something.

> such that then someone else who by his own admission
> isn't completely up to date on things says, sure, that sounds great.

Which is why we need to get some more experienced Python users involved in this.

Well, even the mileage of inexperienced users is quite useful for detecting what level of obviousness has been achieved by the features, so I'm not trying to exclude anyone.

> The current PL/Python also has, arguably, a more natural Python environment,

No, it doesn't. GD/SD are contrived in order to compensate for the very absence of that. Additionally, "modules / script files" don't return or yield.

> native typing,

Okay, that's arguably subjective, but when I write "native typing", it's wrt PG, not Python's built-in types. If that wasn't the case, I wouldn't call it anything--save "[data] conversion" when necessary--as it wouldn't be much of a feature to clamor about.

> efficiency,

Yes, as discussed in the thread before there are trade-offs here wrt how PG data is handled. It depends pretty heavily on how the parameters / query results are used.

Although, as stated before, the difference in efficiency can be rather significant in situations where conversion to built-in Python types is *not* desired.

> and explicitness.

Hrm? What is explicit about munging the source code of the procedure to make it into a function body? Perhaps you could give some examples where plpython helps promote explicitness?

And sure, I understand that other PLs do this. That may be fine for those languages, but for Python it's not, IMO.

On Jan 13, 2010, at 12:15 PM, Robert Haas wrote:
> 1. It's not just a rewrite, it's an incompatible rewrite that will
> present significant user-visible behavioral differences. So replacing
> the current implementation wholesale would produce massive breakage
> for anyone actually using PL/python in production.

Right. That was the point of trying to leverage Python 3 to make the distinction. Most people will need to update their functions if they are moving to Python 3. And for the larger chunks of code, the hard stuff, the amount of change required is likely significant already.

<snip some>
> Given these two facts, it's hard for me to see how we could decide to
> REMOVE the current implementation and replace it with the new one. So
> the most we could do is maintain them side by side, and then you have
> to ask, why?

My original hope was that plpython would be maintained for several years to come and when Python 3 started picking up steam, we would deprecate plpython. If people still wanted to use it, they could continue using the older version of PG and/or someone could continue to maintain plpython out of core for legacy support.

<snip/maintaining out of core>
> Sure, his code won't get as much exposure that way,

=)

Try next to none. The existence of core's implementation makes competing *very* difficult, IMO. Thinking of something along the lines: "Why would I use/contribute to your implementation when core has one?" And, all I can say in response is, "Check out my features." Subsequently, they will probably weigh the added risk of choosing the loner's implementation and come to the conclusion that using core's would be safer in the long term. I can't fault that line of reasoning, so ISTM that it would be difficult to "sell".

> but it's been posted to the mailing list several times now
> over a period of 8 months and nobody has said "oh, wow, this is
> great".

Yeah. :(

In the past, one person showed interest in function modules(Stuart, see the first WIP message), and two others showed interest in native typing(Nathan and Tino). Mr. Drake has also shown some interest in this thread.

But, yes, you are correct. There has been no "wow, this is great" message.

James William Pye wrote:
> On Jan 13, 2010, at 2:27 PM, Peter Eisentraut wrote:
>
>> The problem I'm having with this discussion is that every time someone
>> asks what the supposed advantages of this new Python PL are, a feature
>> list like the above is dumped,
>>
>
> I agree that this is unfortunate, but how else can we to discuss the advantages? It boils down to comparing a couple feature lists, and *maybe* some implementation details. No?
>

Code samples. You're trying to unseat a well established incumbent here,
and you're not ever going to do that with documentation. Maybe your
plpython3 has some compelling features to it. I don't know, because even
with several thousand lines of basic Python code to my credit I cannot
understand a single one of the arguments you presented for why your
implementation is better--except agreeing that, yes, tracebacks are
useful And even on that one, I'm not going to take your word on the
superiority of your implementation. You're writing way over people's
heads here. (Doesn't help that your docs link at the bottom of
http://wiki.postgresql.org/wiki/WIP:plpython3 is broken either). If one
has to be a Python expert to understand your position, you've already lost.

Python code is easy to read though. If you'd said "here's a great
example of how Function Modules are an improvement over what you can do
with the current pl/python," that would be infinitely more useful than
the list of language trivia related to them. You should be aiming to put
Peter on the spot to respond to claims you make like "you can't do this
easily with the current implementation" after showing an elegant bit of
code.

One of the things I'm increasingly frustrated by (and don't take this
personally, this is a general comment coming more from the last CF
rather than something I mean to single you out for) is how many patch
submissions we get that don't have *compelling* examples showing their
value. Have a better programming approach to something? Show me the old
way and how the new way is better. Performance improvement? Provide a
complete, self-contained example showing how to demonstrate it. New type
of feature? Cut and paste a whole session showing how it's used, with
every single command you typed after initdb.

Basically, if a reviewer can't confirm your patch is doing something
useful in five minutes and be excited that they've watched something
interesting happen, you're decreasing the chances that your patch will
ever go anywhere dramatically. I hope that everyone submitting patches
reads http://www.depesz.com/ at least once in a while. One of the things
I really enjoy about his blog is how he shows complete working examples
of so many patches. To pick a standout recent entry,
http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/
takes "exclusion constraints"--a feature I didn't follow a bit of the
discussion about--and works through the whole feature with a series of
examples that, while still complicated, are completely self-contained
and possible to follow along until you understand how it all fits
together. Patch submitters should consider it a goal to make life that
easy for the reviewer stuck with checking their patch out.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

* Greg Smith <greg(at)2ndquadrant(dot)com> [100114 02:17]:

> One of the things I'm increasingly frustrated by (and don't take this
> personally, this is a general comment coming more from the last CF
> rather than something I mean to single you out for) is how many patch
> submissions we get that don't have *compelling* examples showing their
> value. Have a better programming approach to something? Show me the old
> way and how the new way is better. Performance improvement? Provide a
> complete, self-contained example showing how to demonstrate it. New type
> of feature? Cut and paste a whole session showing how it's used, with
> every single command you typed after initdb.

Wow, I can't agree more... I've seen *so* many patches fly by that don't
mean *anything* to me with the description sent to -hackers, until I
find out what they actually do, or could do, until I find it in my RSS
reader, via:

> I hope that everyone submitting patches
> reads http://www.depesz.com/ at least once in a while. One of the things
> I really enjoy about his blog is how he shows complete working examples
> of so many patches. To pick a standout recent entry,
> http://www.depesz.com/index.php/2010/01/03/waiting-for-8-5-exclusion-constraints/
> takes "exclusion constraints"--a feature I didn't follow a bit of the
> discussion about--and works through the whole feature with a series of
> examples that, while still complicated, are completely self-contained
> and possible to follow along until you understand how it all fits
> together.

<what he said>++

> Patch submitters should consider it a goal to make life that
> easy for the reviewer stuck with checking their patch out.

Yes, submitters, please specifically try to make Hubert's life easier,
because we *all* will appreciate it...

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

On Jan 14, 2010, at 12:17 AM, Greg Smith wrote:
> Code samples.

Okay.

> I don't know, because even with several thousand lines of basic Python code to my credit I cannot understand a single one of the arguments you presented for why your implementation is better--except agreeing that, yes, tracebacks are useful

> And even on that one, I'm not going to take your word on the superiority of your implementation.

Sure, that's what review is about. No?

> You're writing way over people's heads here.

Okay. I guess I hoped the documentation would help clarify a lot of this, and make the advantages self-evident.

On that:

> (Doesn't help that your docs link at the bottom of http://wiki.postgresql.org/wiki/WIP:plpython3 is broken either).

Ouch. Thanks, that's fixed now. Please take a look again:

http://python.projects.postgresql.org/pldocs/plpython3.html

> If one has to be a Python expert to understand your position, you've already lost.

Function modules should be pretty obvious. "native typing" is a bit more difficult as a solid understanding of PG's type system is fairly important for a firm grasp.

> Python code is easy to read though. If you'd said "here's a great example of how Function Modules are an improvement over what you can do with the current pl/python," that would be infinitely more useful than the list of language trivia related to them. You should be aiming to put Peter on the spot to respond to claims you make like "you can't do this easily with the current implementation" after showing an elegant bit of code.

Okay. So, some examples would help. The documentation is back up, so please be sure to look at the numerous examples provided therein. In addition to that, I'll try to get some contrasting examples posted as a follow-up to an earlier message. "In plpython you do X whereas in plpython3 you do Y."

Thanks.

On Thu, 2010-01-14 at 05:39 -0700, James William Pye wrote:

>
> > Python code is easy to read though. If you'd said "here's a great example of how Function Modules are an improvement over what you can do with the current pl/python," that would be infinitely more useful than the list of language trivia related to them. You should be aiming to put Peter on the spot to respond to claims you make like "you can't do this easily with the current implementation" after showing an elegant bit of code.
>
> Okay. So, some examples would help. The documentation is back up, so please be sure to look at the numerous examples provided therein. In addition to that, I'll try to get some contrasting examples posted as a follow-up to an earlier message. "In plpython you do X whereas in plpython3 you do Y."
>

The documentation is very thorough, thank you. I am still a fan of
getting this reviewed for potential inclusion but I firmly agree with
what Greg has already said.

What I would (as a non hacker) would look for is:

(1) Generalized benchmarks between plpython(core) and plpython3u

I know a lot of these are subjective, but it is still good to see if
there are any curves or points that bring the performance of either to
light.

(2) Example of the traceback facility, I know it is silly but I don't
have time to actually download head, apply the patch and test this. This
type of thing, showing debugging facilities within the function would be
killer.

(3) A distinct real world comparison where the core plpython falls down
(if it does) against the plpython3u implementation

I can't speak to your code quality, that is going to have to be someone
else.

Sincerely,

Joshua D. Drake

>
> Thanks.

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

James William Pye wrote:
> The documentation is back up, so please be sure to look at the numerous examples provided therein. In addition to that, I'll try to get some contrasting examples posted as a follow-up to an earlier message. "In plpython you do X whereas in plpython3 you do Y."
>

I had read the documentation before so it wasn't that bad that I
couldn't see it. Basically, the issue I had is that it's not really
clear which features are unique to this implementation that make it
compelling. So more targeted examples like you're considering now would
help.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

On Jan 14, 2010, at 2:03 PM, Joshua D. Drake wrote:
> What I would (as a non hacker) would look for is:
>
> (1) Generalized benchmarks between plpython(core) and plpython3u
>
> I know a lot of these are subjective, but it is still good to see if
> there are any curves or points that bring the performance of either to
> light.

I guess I could do some simple function I/O tests to identify invocation overhead(take a single parameter and return it). This should give a somewhat reasonable view of the trade-offs of "native typing" vs conversion performance-wise. One thing to keep in mind is that *three* tests would need to be done per parameter set:

1. plpython's
2. plpython3's (raw data objects/"native typing")
3. plpython3's + @pytypes

The third should show degraded performance in comparison to plpythonu's whereas the second should show improvement or near equivalence.

@pytypes is actually implemented in pure-Python, so the impact should be quite visible.

http://python.projects.postgresql.org/pldocs/plpython3-postgres-pytypes.html

I'm not sure there's anything else worth measuring. SRFs, maybe?

> (2) Example of the traceback facility, I know it is silly but I don't
> have time to actually download head, apply the patch and test this.

Well, if you ever do find some time, the *easiest* way would probably be to download a branch snapshot from git.pg.org:

http://git.postgresql.org/gitweb?p=plpython3.git;a=snapshot;h=refs/heads/plpython3;sf=tgz

It requires Python 3.1. 3.0 has been abandoned by python.org.

> This
> type of thing, showing debugging facilities within the function would be
> killer.

The test output has a *lot* of tracebacks, so I'll just copy and paste one here.

This one shows the traceback output of a chained exception.

-- suffocates a pg error, and attempts to enter a protected area
CREATE OR REPLACE FUNCTION pg_failure_suf_IFTE() RETURNS VOID LANGUAGE plpython3u AS
$python$
import Postgres

rp = Postgres.Type(Postgres.CONST['REGPROCEDUREOID'])

def main():
try:
fun = rp('nosuchfunc(int17,zzz)')
except:
# Should be valid, but the protection of
# PL_DB_IN_ERROR should keep it from getting called.
rp('pg_x_failure_suf()')
$python$;

SELECT pg_failure_suf_IFTE();
ERROR: database action attempted while in failed transaction
CONTEXT: [exception from Python]
Traceback (most recent call last):
File "public.pg_failure_suf_ifte()", line 8, in main
fun = rp('nosuchfunc(int17,zzz)')
Postgres.Exception: type "int17" does not exist
CODE: 42704

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "public.pg_failure_suf_ifte()", line 12, in main
rp('pg_x_failure_suf()')
Postgres.Exception

[public.pg_failure_suf_ifte()]

> (3) A distinct real world comparison where the core plpython falls down
> (if it does) against the plpython3u implementation

Hrm. Are you looking for something that plpython3 can do that plpython can't? Or are you looking for something where plpython makes the user work a lot harder?

On Fri, 2010-01-15 at 13:26 -0700, James William Pye wrote:
> On Jan 14, 2010, at 2:03 PM, Joshua D. Drake wrote:
> > What I would (as a non hacker) would look for is:
> >
> > (1) Generalized benchmarks between plpython(core) and plpython3u
> >
> > I know a lot of these are subjective, but it is still good to see if
> > there are any curves or points that bring the performance of either to
> > light.
>
> I guess I could do some simple function I/O tests to identify invocation overhead(take a single parameter and return it). This should give a somewhat reasonable view of the trade-offs of "native typing" vs conversion performance-wise. One thing to keep in mind is that *three* tests would need to be done per parameter set:
>
> 1. plpython's
> 2. plpython3's (raw data objects/"native typing")
> 3. plpython3's + @pytypes
>
> The third should show degraded performance in comparison to plpythonu's whereas the second should show improvement or near equivalence.
>
> @pytypes is actually implemented in pure-Python, so the impact should be quite visible.
>
> http://python.projects.postgresql.org/pldocs/plpython3-postgres-pytypes.html
>
>
> I'm not sure there's anything else worth measuring. SRFs, maybe?
>
>
> > (2) Example of the traceback facility, I know it is silly but I don't
> > have time to actually download head, apply the patch and test this.
>
> Well, if you ever do find some time, the *easiest* way would probably be to download a branch snapshot from git.pg.org:
>
> http://git.postgresql.org/gitweb?p=plpython3.git;a=snapshot;h=refs/heads/plpython3;sf=tgz
>
> It requires Python 3.1. 3.0 has been abandoned by python.org.
>
> > This
> > type of thing, showing debugging facilities within the function would be
> > killer.
>
> The test output has a *lot* of tracebacks, so I'll just copy and paste one here.
>
> This one shows the traceback output of a chained exception.
>
> -- suffocates a pg error, and attempts to enter a protected area
> CREATE OR REPLACE FUNCTION pg_failure_suf_IFTE() RETURNS VOID LANGUAGE plpython3u AS
> $python$
> import Postgres
>
> rp = Postgres.Type(Postgres.CONST['REGPROCEDUREOID'])
>
> def main():
> try:
> fun = rp('nosuchfunc(int17,zzz)')
> except:
> # Should be valid, but the protection of
> # PL_DB_IN_ERROR should keep it from getting called.
> rp('pg_x_failure_suf()')
> $python$;
>
>
> SELECT pg_failure_suf_IFTE();
> ERROR: database action attempted while in failed transaction
> CONTEXT: [exception from Python]
> Traceback (most recent call last):
> File "public.pg_failure_suf_ifte()", line 8, in main
> fun = rp('nosuchfunc(int17,zzz)')
> Postgres.Exception: type "int17" does not exist
> CODE: 42704
>
> During handling of the above exception, another exception occurred:
>
> Traceback (most recent call last):
> File "public.pg_failure_suf_ifte()", line 12, in main
> rp('pg_x_failure_suf()')
> Postgres.Exception
>
> [public.pg_failure_suf_ifte()]
>
>
> > (3) A distinct real world comparison where the core plpython falls down
> > (if it does) against the plpython3u implementation
>
> Hrm. Are you looking for something that plpython3 can do that plpython can't? Or are you looking for something where plpython makes the user work a lot harder?

I think both apply.

This is great stuff, thank you for taking the effort.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Jan 14, 2010, at 7:08 PM, Greg Smith wrote:
> So more targeted examples like you're considering now would help.

So far, I have three specific examples in mind:

The first will illustrate the advantages of function modules wrt setup code in the module body. Primarily this is about convenience. (I'm going to send this example when I send this message)

The second is a generic after trigger that does manipulation logging for some simple replication purposes. This example will illustrate one application of "native typing" as it provides direct access to a PG type's typoutput.

The third one is a fairly old plpythonu example written by Elein that exercises SD to keep state for an aggregate. I'm expecting this to be a good candidate for showing off stateful functions.

Other things I plan to cover, but nothing specific in mind yet:

Direct function calls
Internal subtransactions, "with xact():" (something plpython can't do, save calling plpgsql =)

On Jan 14, 2010, at 7:08 PM, Greg Smith wrote:
> So more targeted examples like you're considering now would help.

Here's the first example. This covers an advantage of function modules.

This is a conversion of a plpythonu function published to the wiki:

http://wiki.postgresql.org/wiki/Google_Translate

In the above link, the code is executed in the body of a Python function.
Please see plpython's documentation if you don't understand what I mean by that.

The effect of this is that every time the FUNCTION is called from PG, the import statements are ran, a new class object, UrlOpener, is created, and a new function object, translate, is created. Granted, a minor amount of overhead in this case, but the point is that in order to avoid it the author would have to use SD:

if "urlopener" in SD:
UrlOpener = SD["urlopener"]
else:
class UrlOpener(urllib.UrlOpener):
...
SD["urlopener"] = UrlOpener

While some may consider this a minor inconvenience, the problem is that *setup code is common*, so it's, at least, a rather frequent, minor inconvenience.

With function modules, users have a module body to run any necessary setup code.

Now, WRT the actual example code, I'm not suggesting that either example is ideal. Only that it should *help* identify one particular advantage of function modules.

CREATE OR REPLACE FUNCTION public.gtranslate(src text, target text, phrase text)
RETURNS text
LANGUAGE plpython3u
AS $function$
from urllib.request import URLopener
from urllib.parse import quote_plus
import json

base_uri = "http://ajax.googleapis.com/ajax/services/language/translate?"

class UrlOpener(URLopener):
version = "py-gtranslate/1.0"
urlopen = UrlOpener().open

equal_fmt = '{0}={1}'.format

@pytypes
def main(src, to, phrase):
args = (
('v', '1.0'),
('langpair', quote_plus(src + '|' + to)),
('q', quote_plus(phrase)),
)
argstring = '&'.join([equal_fmt(k,v) for (k,v) in args])

resp = urlopen(base_uri + argstring).read()
resp = json.loads(resp.decode('utf-8'))
try:
return resp['responseData']['translatedText']
except:
# should probably warn about failed translation
return phrase
$function$;

pl_regression=# SELECT gtranslate('en', 'es', 'i like coffee');
gtranslate
------------------
Me gusta el café
(1 row)

pl_regression=# SELECT gtranslate('en', 'de', 'i like coffee');
gtranslate
----------------
Ich mag Kaffee
(1 row)

On Sun, Jan 17, 2010 at 4:07 PM, James William Pye <lists(at)jwp(dot)name> wrote:
> The effect of this is that every time the FUNCTION is called from PG, the import statements are ran, a new class object, UrlOpener, is created, and a new function object, translate, is created. Granted, a minor amount of overhead in this case, but the point is that in order to avoid it the author would have to use SD:
>
> if "urlopener" in SD:
>  UrlOpener = SD["urlopener"]
> else:
>  class UrlOpener(urllib.UrlOpener):
>  ...
>  SD["urlopener"] = UrlOpener
>
> While some may consider this a minor inconvenience, the problem is that *setup code is common*, so it's, at least, a rather frequent, minor inconvenience.
>
>
> With function modules, users have a module body to run any necessary setup code.

Coming from a Python developer perspective, this is indeed an
improvement. I always thought the whole SD/GD thing was a little odd.
Doing the setup at the module level and relying on the interpreter to
keep it cached is much more "Pythonic" and is a common idiom.

David

On Jan 14, 2010, at 2:03 PM, Joshua D. Drake wrote:
> What I would (as a non hacker) would look for is:
>
> (1) Generalized benchmarks between plpython(core) and plpython3u
>
> I know a lot of these are subjective, but it is still good to see if
> there are any curves or points that bring the performance of either to
> light.

k, it was pretty much as expected. However, the surprise for me was that @pytypes didn't perform as terribly as I expected it to. I imagine it's impact may become more noticeable with more parameters, but, nonetheless, I was surprised. I didn't do any SRF tests, but the installations are still setup, so if anyone really wants to see that, it shouldn't take long to do.

Apologies ahead of time for the lack pretty graphs. =)

I used two different builds/installations of PG to test as the PL names conflict. Both were compiled with the following CFLAGS(pg_config output):

-O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv

Both PLs were built against the same build of Python(recent svn update of release31-maint):

Python 3.1.1+ (release31-maint:77585M, Jan 17 2010, 10:29:13)
[GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin

I ran each of the test files a few times against the target installation, but I only attached one of each to this message. (Primarily, multiple runs to filter out any spurious spikes.)

The source SQL and output files are attached.

rawtest.sql.out is the output for raw data objects(native typing).
pytypestest.sql.out is the output of the @pytypes test(native typing with conversion overhead).
plpythontest.sql.out is the output for core's plpython(conversion).

A few samples from the output files are included inline below.

Each volatile function is called 100,000 times from a COUNT() aggregate, and the duration is measured using psql's \timing. Most of the functions simply return the first parameter given to it. The functions are ran inside a transaction because plpython3 does some cache clears(linecache) and GC at the end of transactions.

The parameter type, if any, is indicated by the label:

noparams:

raw: 125ms
pytypes: 372ms (base overhead, it would appear)
plpython: 309ms

oneint2:

raw: 140ms
pytypes: 684ms
plpython: 750ms

oneint8:

raw: 145ms
pytypes: 676ms
plpython: 718ms

text_large:

raw: 271ms
pytypes: 2766ms
plpython: 2310ms

composite:

raw: 235ms
pytypes: 795ms (N/A, no conversion done, but takes a bit of a hit anyways)
plpython: 1654ms

On Jan 14, 2010, at 7:08 PM, Greg Smith wrote:
> So more targeted examples like you're considering now would help.

Here's the trigger example which should help reveal some of the advantages of "native typing". This is a generic trigger that constructs and logs manipulation statements for simple replication purposes.

The original plpython version is located here:

http://ar.pycon.org/common/2009/talkdata/PyCon2009/020/plpython.txt
[You'll need to scroll down to the very bottom of that page.]

There are three points in this example that need to be highlighted:

1. There is no need for a "mogrify" function (see original in the above link).
2. Attributes/columns of the records (new/old) are extracted when referenced.
3. The comparisons in after_update uses the data type's actual inequality operator.

The first point is true because "native typing" gives the user direct access to a given type's typoutput via ``str(ob)``. This makes constructing the PG string representation of a given object *much* easier--quote_nullable, and done. The original plpython example will need to be updated to compensate for any changes in conversion: arrays will now need special handling and MD arrays will not work at all. It also relies heavily on the Python object representation matching PG's; where that fails, special cases need to be implemented(composites, notably). All of that compensation performed in the original version is unnecessary in the plpython3 version.

The second point touches on the "efficiency" that was referenced in an earlier message. No cycles are spent converting the contents of a container object unless the user chooses to. Naturally, there is no advantage performance-wise if you are always converting everything.
I'd wager that with triggers, it's rare that everything needs to be converted.

The third point reveals that Postgres.Object instances--a component of native typing--use the data type's operator for inequality. It's not limited to comparisons as all available Python operators are mapped to corresponding operators in PG. For many or all primitives, there is no added value over conversion. However, this provides a lot of convenience when working with UDTs, datetime types, and geometric types.

...ISTM that the primary advantage of "native typing" is that we get to define the Python interface to a given Postgres data type.

Three files are attached:

afterlog.py - the trigger returning function
afterlog.sql - the sql exercising the TRF (creates the replica_log table as well)
afterlog.out - the contents of the replica_log table after executing afterlog.sql

To replay:

\i afterlog.py
\i afterlog.sql
SELECT * FROM replica_log;

On Wed, 2010-01-20 at 19:32 -0700, James William Pye wrote:

> Apologies ahead of time for the lack pretty graphs. =)
>
> I used two different builds/installations of PG to test as the PL names conflict. Both were compiled with the following CFLAGS(pg_config output):
>
> -O2 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -fno-strict-aliasing -fwrapv
>
> Both PLs were built against the same build of Python(recent svn update of release31-maint):
>
> Python 3.1.1+ (release31-maint:77585M, Jan 17 2010, 10:29:13)
> [GCC 4.2.1 (Apple Inc. build 5646) (dot 1)] on darwin
>

This is great stuff. Thanks for your diligence.

> I ran each of the test files a few times against the target installation, but I only attached one of each to this message. (Primarily, multiple runs to filter out any spurious spikes.)
>
> The source SQL and output files are attached.
>
> rawtest.sql.out is the output for raw data objects(native typing).
> pytypestest.sql.out is the output of the @pytypes test(native typing with conversion overhead).
> plpythontest.sql.out is the output for core's plpython(conversion).
>
> A few samples from the output files are included inline below.
>
> Each volatile function is called 100,000 times from a COUNT() aggregate, and the duration is measured using psql's \timing. Most of the functions simply return the first parameter given to it. The functions are ran inside a transaction because plpython3 does some cache clears(linecache) and GC at the end of transactions.
>
> The parameter type, if any, is indicated by the label:
>
> noparams:
>
> raw: 125ms
> pytypes: 372ms (base overhead, it would appear)
> plpython: 309ms
>
> oneint2:
>
> raw: 140ms
> pytypes: 684ms
> plpython: 750ms
>
> oneint8:
>
> raw: 145ms
> pytypes: 676ms
> plpython: 718ms
>
> text_large:
>
> raw: 271ms
> pytypes: 2766ms
> plpython: 2310ms
>
> composite:
>
> raw: 235ms
> pytypes: 795ms (N/A, no conversion done, but takes a bit of a hit anyways)
> plpython: 1654ms
>
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Sat, 2010-01-23 at 13:28 -0700, James William Pye wrote:
> On Jan 14, 2010, at 7:08 PM, Greg Smith wrote:
> > So more targeted examples like you're considering now would help.

So is there any more movement on this? Peter, what do you think? I
mean... he has put in quite a bit of effort here. How do we get this
reviewed?

I asked my guys to do it, but my C guys are very Perl centric...

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Sat, Jan 23, 2010 at 3:28 PM, James William Pye <lists(at)jwp(dot)name> wrote:
> On Jan 14, 2010, at 7:08 PM, Greg Smith wrote:
>> So more targeted examples like you're considering now would help.
>
> Here's the trigger example which should help reveal some of the advantages of "native typing". This is a generic trigger that constructs and logs manipulation statements for simple replication purposes.
>
> The original plpython version is located here:
>
>  http://ar.pycon.org/common/2009/talkdata/PyCon2009/020/plpython.txt
>  [You'll need to scroll down to the very bottom of that page.]
>
>
> There are three points in this example that need to be highlighted:
>
>  1. There is no need for a "mogrify" function (see original in the above link).
>  2. Attributes/columns of the records (new/old) are extracted when referenced.
>  3. The comparisons in after_update uses the data type's actual inequality operator.
>
> The first point is true because "native typing" gives the user direct access to a given type's typoutput via ``str(ob)``. This makes constructing the PG string representation of a given object *much* easier--quote_nullable, and done. The original plpython example will need to be updated to compensate for any changes in conversion: arrays will now need special handling and MD arrays will not work at all. It also relies heavily on the Python object representation matching PG's; where that fails, special cases need to be implemented(composites, notably). All of that compensation performed in the original version is unnecessary in the plpython3 version.
>
> The second point touches on the "efficiency" that was referenced in an earlier message. No cycles are spent converting the contents of a container object unless the user chooses to. Naturally, there is no advantage performance-wise if you are always converting everything.
> I'd wager that with triggers, it's rare that everything needs to be converted.
>
> The third point reveals that Postgres.Object instances--a component of native typing--use the data type's operator for inequality. It's not limited to comparisons as all available Python operators are mapped to corresponding operators in PG. For many or all primitives, there is no added value over conversion. However, this provides a lot of convenience when working with UDTs, datetime types, and geometric types.
>
> ...ISTM that the primary advantage of "native typing" is that we get to define the Python interface to a given Postgres data type.

When I initially read the description of this patch, I had really no
idea what it was about, and I have to say that over the last several
weeks, the subsequent posts have increased my interest substantially,
and I think it could potentially have a future in core, or in contrib.
The performance gains from native typing are particularly
interesting.

That having been said, we don't have a reviewer for this patch, and
more to the point, even if we did, I don't believe we have a COMMITTER
for this patch. The only committer who has done any significant work
on the existing PL/python implementation for this release is Peter
Eisentraut, and he's made it pretty clear that he's not interested in
this reimplementation. I personally do not know Python even as a
user, let alone as a hacker, so I'm poorly qualified to review it
myself. Even were it otherwise, I know that if I were to commit this
I would to some degree be on the hook to maintain it forever, which is
more than I really want to take on. I suspect other committers feel
similarly; or if not, they haven't chimed in on any of the relevant
threads to say so.

To recap the votes I've seen on this thread and elsewhere:

- JD is very enthusiastic about this patch
- So is the OP
- Josh Berkus and I are both dubious about having two in-core PL/pythons
- Peter Eisentraut prefers the original implementation
- Greg Smith thinks (if I'm not putting words into his mouth) that
this might be worth considering, but not for 9.0

On the basis of all of the foregoing, I don't think we can consider
this patch further for this CommitFest and will update
commitfest.postgresql.org accordingly. If the user community grows or
if one of the committers takes an interest in this down the road, I
think we could consider it for a future release.

...Robert

On Mon, 2010-02-01 at 13:20 -0500, Robert Haas wrote:

> On the basis of all of the foregoing, I don't think we can consider
> this patch further for this CommitFest and will update
> commitfest.postgresql.org accordingly. If the user community grows or
> if one of the committers takes an interest in this down the road, I
> think we could consider it for a future release.
>

I spoke with James offline about this as well. My understanding (correct
me James) is that he is working on an implementation that can be
installed via PGXS.

Joshua D. Drake

> ...Robert
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Mon, Feb 1, 2010 at 1:29 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
> On Mon, 2010-02-01 at 13:20 -0500, Robert Haas wrote:
>> On the basis of all of the foregoing, I don't think we can consider
>> this patch further for this CommitFest and will update
>> commitfest.postgresql.org accordingly.  If the user community grows or
>> if one of the committers takes an interest in this down the road, I
>> think we could consider it for a future release.
>
> I spoke with James offline about this as well. My understanding (correct
> me James) is that he is working on an implementation that can be
> installed via PGXS.

Sounds good!

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> To recap the votes I've seen on this thread and elsewhere:

> - JD is very enthusiastic about this patch
> - So is the OP
> - Josh Berkus and I are both dubious about having two in-core PL/pythons
> - Peter Eisentraut prefers the original implementation
> - Greg Smith thinks (if I'm not putting words into his mouth) that
> this might be worth considering, but not for 9.0

One other problem with accepting this to be parallel with the existing
plpython is that there's a name conflict: Peter's work to allow the
existing PL to use Python 3 has already claimed the name "plpython3".
Whether it's to be distributed in core or separately, I think something
needs to be done about that.

The first thought that comes to mind is "plpythonng", following a
tradition established by the tcl client rewrite among others ... but
that double n doesn't read very well.

regards, tom lane

Tom Lane escribió:

> The first thought that comes to mind is "plpythonng", following a
> tradition established by the tcl client rewrite among others ... but
> that double n doesn't read very well.

plpythoNG perhaps?

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Mon, Feb 1, 2010 at 2:00 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
> Tom Lane escribió:
>
>> The first thought that comes to mind is "plpythonng", following a
>> tradition established by the tcl client rewrite among others ... but
>> that double n doesn't read very well.
>
> plpythoNG perhaps?

ROFL. It's a scantily-clad implementation of PL/py...

...Robert

On Feb 1, 2010, at 10:53 AM, Tom Lane wrote:

> The first thought that comes to mind is "plpythonng", following a
> tradition established by the tcl client rewrite among others ... but
> that double n doesn't read very well.

And without it, you have a thong. Who's going to wear that?

Best,

David

On Feb 1, 2010, at 11:29 AM, Joshua D. Drake wrote:

> On Mon, 2010-02-01 at 13:20 -0500, Robert Haas wrote:
>
>> On the basis of all of the foregoing, I don't think we can consider
>> this patch further for this CommitFest and will update
>> commitfest.postgresql.org accordingly. If the user community grows or
>> if one of the committers takes an interest in this down the road, I
>> think we could consider it for a future release.
>>
>
> I spoke with James offline about this as well. My understanding (correct
> me James) is that he is working on an implementation that can be
> installed via PGXS.

yep, mostly done: http://github.com/jwp/pg-python

The tests that can pass are passing on 8.3 and 8.4 now, save optimized Cursor returns in 8.3(the materialize preferred flag).

Also made some other improvements like getting rid of the ugly `__func__.stateful = True` in favor of a decorator, @Stateful. (Thanks to Harald for a push in that direction.)

Right now, I'm trying to trim some of the easy issues[1] and getting a project web page up. I expect to be able to make a release soon, and I'll follow-up to this thread when I do.

However, the lack of release files shouldn't stop anyone from trying it out. =)

Snapshot:
http://github.com/jwp/pg-python/zipball/master

[1] http://github.com/jwp/pg-python/issues

> On the basis of all of the foregoing, I don't think we can consider
> this patch further for this CommitFest and will update
> commitfest.postgresql.org accordingly.

FWIW, I am very excited about this patch and would be happy to review
it but have been very busy over the past month. If I can promise a
review by Thursday morning could we keep it active? Hopefully, at the
very least, I can provide some useful feedback and spawn some
community interest.

I am worried that there is a bit of a chicken and an egg problem with
this patch. I code nearly exclusively in python and C, but I have
often found pl/python to be very unwieldy. For this reason I often
use pl/perl or pl/pgsql for problems that, outside of postgres, I
would always use python. From the documentation, this patch seems like
an enormous step in the right direction.

-Nathan

On Mon, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > On the basis of all of the foregoing, I don't think we can consider
> > this patch further for this CommitFest and will update
> > commitfest.postgresql.org accordingly.
>
> FWIW, I am very excited about this patch and would be happy to review
> it but have been very busy over the past month. If I can promise a
> review by Thursday morning could we keep it active? Hopefully, at the
> very least, I can provide some useful feedback and spawn some
> community interest.
>
> I am worried that there is a bit of a chicken and an egg problem with
> this patch. I code nearly exclusively in python and C, but I have
> often found pl/python to be very unwieldy. For this reason I often
> use pl/perl or pl/pgsql for problems that, outside of postgres, I
> would always use python. From the documentation, this patch seems like
> an enormous step in the right direction.

I am a +1

Joshua D. Drake

>
> -Nathan
>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On Mon, Feb 1, 2010 at 3:01 PM, Nathan Boley <npboley(at)gmail(dot)com> wrote:
>> On the basis of all of the foregoing, I don't think we can consider
>> this patch further for this CommitFest and will update
>> commitfest.postgresql.org accordingly.
>
> FWIW, I am very excited about this patch and would be happy to review
> it but have been very busy over the past month. If I can promise a
> review by Thursday morning could we keep it active? Hopefully, at the
> very least, I can provide some useful feedback and spawn some
> community interest.
>
> I am worried that there is a bit of a chicken and an egg problem with
> this patch. I code nearly exclusively in python and C, but I have
> often found pl/python to be very unwieldy.  For this reason I often
> use pl/perl or pl/pgsql for problems that, outside of postgres, I
> would always use python. From the documentation, this patch seems like
> an enormous step in the right direction.

I think it would be great for you to review it... I doubt that will
cause it to get committed for 9.0, but my doubt is no reason for you
to hold off reviewing it.

...Robert

> I think it would be great for you to review it... I doubt that will
> cause it to get committed for 9.0, but my doubt is no reason for you
> to hold off reviewing it.

I assumed so, but the pretense of a chance will probably help to motivate me :-)

I'll have something by Thursday, and then 'Returned with Feedback'
will at least be factual.

Best,
Nathan

On mån, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> I code nearly exclusively in python and C, but I have
> often found pl/python to be very unwieldy. For this reason I often
> use pl/perl or pl/pgsql for problems that, outside of postgres, I
> would always use python.

I find that curious, because much of the criticism about the current
PL/Python can be traced back to the fact that the implementation used to
be an exact copy of PL/Perl.

On Mon, 2010-02-01 at 22:35 +0200, Peter Eisentraut wrote:
> On mån, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > I code nearly exclusively in python and C, but I have
> > often found pl/python to be very unwieldy. For this reason I often
> > use pl/perl or pl/pgsql for problems that, outside of postgres, I
> > would always use python.
>
> I find that curious, because much of the criticism about the current
> PL/Python can be traced back to the fact that the implementation used to
> be an exact copy of PL/Perl.

Well my guess is, if you want to code python, you don't want to feel
like you are coding perl and thus you might as well just code perl?

Joshua D. Drake

>

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

Peter Eisentraut escribió:
> On mån, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > I code nearly exclusively in python and C, but I have
> > often found pl/python to be very unwieldy. For this reason I often
> > use pl/perl or pl/pgsql for problems that, outside of postgres, I
> > would always use python.
>
> I find that curious, because much of the criticism about the current
> PL/Python can be traced back to the fact that the implementation used to
> be an exact copy of PL/Perl.

Perhaps the problem is that PL/Perl used to be unwieldy back when
PL/Python was created. PL/Perl has definitely seen a lot more activity.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> Peter Eisentraut escribi?:
> > On m?n, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > > I code nearly exclusively in python and C, but I have
> > > often found pl/python to be very unwieldy. For this reason I often
> > > use pl/perl or pl/pgsql for problems that, outside of postgres, I
> > > would always use python.
> >
> > I find that curious, because much of the criticism about the current
> > PL/Python can be traced back to the fact that the implementation used to
> > be an exact copy of PL/Perl.
>
> Perhaps the problem is that PL/Perl used to be unwieldy back when
> PL/Python was created. PL/Perl has definitely seen a lot more activity.

I would love to know why PL/Python can't be incrementally improved like
the rest of our code.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, 2010-02-01 at 16:13 -0500, Bruce Momjian wrote:
> Alvaro Herrera wrote:
> > Peter Eisentraut escribi?:
> > > On m?n, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > > > I code nearly exclusively in python and C, but I have
> > > > often found pl/python to be very unwieldy. For this reason I often
> > > > use pl/perl or pl/pgsql for problems that, outside of postgres, I
> > > > would always use python.
> > >
> > > I find that curious, because much of the criticism about the current
> > > PL/Python can be traced back to the fact that the implementation used to
> > > be an exact copy of PL/Perl.
> >
> > Perhaps the problem is that PL/Perl used to be unwieldy back when
> > PL/Python was created. PL/Perl has definitely seen a lot more activity.
>
> I would love to know why PL/Python can't be incrementally improved like
> the rest of our code.

It has been. That is exactly what PeterE has been doing.

However, if you look at this whole thread, you will see the James has a
very different view of the implementation. One that at least appears to
be more advanced and "pythonic" than our version.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

Joshua D. Drake wrote:
> On Mon, 2010-02-01 at 16:13 -0500, Bruce Momjian wrote:
> > Alvaro Herrera wrote:
> > > Peter Eisentraut escribi?:
> > > > On m?n, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
> > > > > I code nearly exclusively in python and C, but I have
> > > > > often found pl/python to be very unwieldy. For this reason I often
> > > > > use pl/perl or pl/pgsql for problems that, outside of postgres, I
> > > > > would always use python.
> > > >
> > > > I find that curious, because much of the criticism about the current
> > > > PL/Python can be traced back to the fact that the implementation used to
> > > > be an exact copy of PL/Perl.
> > >
> > > Perhaps the problem is that PL/Perl used to be unwieldy back when
> > > PL/Python was created. PL/Perl has definitely seen a lot more activity.
> >
> > I would love to know why PL/Python can't be incrementally improved like
> > the rest of our code.
>
> It has been. That is exactly what PeterE has been doing.
>
> However, if you look at this whole thread, you will see the James has a
> very different view of the implementation. One that at least appears to
> be more advanced and "pythonic" than our version.

More "pythonic" in its internal processing or in its user interface?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Feb 1, 2010, at 2:13 PM, Bruce Momjian wrote:
> I would love to know why PL/Python can't be incrementally improved like
> the rest of our code.

AFAICT, there are two primary, perhaps identifying, parts to a PL extension: code management (compilation, execution, etc) and type I/O (conversion in most PLs). (well, aside from the language itself =)

My proposed extension chooses a different design for both of those parts.

It didn't make sense to try and incrementally change PL/Python because I would have been rewriting the whole thing anyways. Not to mention breaking user code in the process for the mentioned parts--thus the Python 3 target.

Hope this clears things up.

On Mon, 2010-02-01 at 16:31 -0500, Bruce Momjian wrote:
>
> > > I would love to know why PL/Python can't be incrementally improved like
> > > the rest of our code.
> >
> > It has been. That is exactly what PeterE has been doing.
> >
> > However, if you look at this whole thread, you will see the James has a
> > very different view of the implementation. One that at least appears to
> > be more advanced and "pythonic" than our version.
>
> More "pythonic" in its internal processing or in its user interface?

User interface and also internal processing (see the types discussion).

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
Respect is earned, not gained through arbitrary and repetitive use or Mr. or Sir.

On 2/1/10 1:39 PM, Joshua D. Drake wrote:
> On Mon, 2010-02-01 at 16:31 -0500, Bruce Momjian wrote:
>>
>>>> I would love to know why PL/Python can't be incrementally improved like
>>>> the rest of our code.
>>> It has been. That is exactly what PeterE has been doing.
>>>
>>> However, if you look at this whole thread, you will see the James has a
>>> very different view of the implementation. One that at least appears to
>>> be more advanced and "pythonic" than our version.
>> More "pythonic" in its internal processing or in its user interface?
>
> User interface and also internal processing (see the types discussion).

Yeah, from the sound of it, we should put this in pgfoundry (or
elsewhere) and have people try it out for 9.0. If the python folks love
it, we can consider adding it to core, and then we can have the argument
about whether to depreciate the older version.

--Josh Berkus

On Mon, Feb 1, 2010 at 4:30 PM, Joshua D. Drake <jd(at)commandprompt(dot)com> wrote:
> On Mon, 2010-02-01 at 16:13 -0500, Bruce Momjian wrote:
>> Alvaro Herrera wrote:
>> > Peter Eisentraut escribi?:
>> > > On m?n, 2010-02-01 at 12:01 -0800, Nathan Boley wrote:
>> > > > I code nearly exclusively in python and C, but I have
>> > > > often found pl/python to be very unwieldy.  For this reason I often
>> > > > use pl/perl or pl/pgsql for problems that, outside of postgres, I
>> > > > would always use python.
>> > >
>> > > I find that curious, because much of the criticism about the current
>> > > PL/Python can be traced back to the fact that the implementation used to
>> > > be an exact copy of PL/Perl.
>> >
>> > Perhaps the problem is that PL/Perl used to be unwieldy back when
>> > PL/Python was created.  PL/Perl has definitely seen a lot more activity.
>>
>> I would love to know why PL/Python can't be incrementally improved like
>> the rest of our code.
>
> It has been. That is exactly what PeterE has been doing.
>
> However, if you look at this whole thread, you will see the James has a
> very different view of the implementation. One that at least appears to
> be more advanced and "pythonic" than our version.

I don't know if the native typing stuff is "more advanced" than our
current code or not; that's kind of fuzzy terminology if you think
about it. It is, however, a lot different than what we do in the
existing PL/python, or, to the best of my knowledge, any of the other
PLs with, perhaps, the exception of PL/pgsql. So conceivably someone
could submit a PL/perlNG, a PL/lolcodeNG, etc. taking a similar
approach. It's worth thinking about how we feel about that.

...Robert

On Feb 1, 2010, at 1:23 PM, Nathan Boley wrote:

>> I think it would be great for you to review it... I doubt that will
>> cause it to get committed for 9.0, but my doubt is no reason for you
>> to hold off reviewing it.
>
> I assumed so, but the pretense of a chance will probably help to motivate me :-)
>
> I'll have something by Thursday, and then 'Returned with Feedback'
> will at least be factual.

I haven't updated the plpython3 branch in a while, so you may want to hit the github repo with the PGXS build: http://github.com/jwp/pg-python

...Should probably get the updated docs published too, but they are available in src/documentation as ReST files. If you have sphinx installed, running `make html` in the root project directory should build them into src/sphinx/html.

On Feb 1, 2010, at 12:18 PM, James William Pye wrote:
> Right now, I'm trying to trim some of the easy issues[1] and getting a project web page up. I expect to be able to make a release soon, and I'll follow-up to this thread when I do.

Well, I ended up doing some others things at that point in time,
but I have managed to get a release out:

http://python.projects.postgresql.org/backend/