Re: PL/Python array support

Here is a patch to support arrays in PL/Python as parameters and return
values. It converts an array parameter to a Python "list", and converts
a Python "sequence" return value back to an array.

I have settled on two implementation restrictions for the moment:

- Only supports one-dimensional arrays. (Python has no multidimensional
lists, so the semantics of this would be dubious.)

- Does not support returning arrays of composite types. (Basically too
complicated to implement right now and seemingly of limited practical
value.)

On Wed, Nov 4, 2009 at 9:02 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> Here is a patch to support arrays in PL/Python as parameters and return
> values.  It converts an array parameter to a Python "list", and converts
> a Python "sequence" return value back to an array.

This is probably a stupid question, but why would you use different
types for incoming and outgoing data flow?

...Robert

On ons, 2009-11-04 at 09:44 -0500, Robert Haas wrote:
> On Wed, Nov 4, 2009 at 9:02 AM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> > Here is a patch to support arrays in PL/Python as parameters and return
> > values. It converts an array parameter to a Python "list", and converts
> > a Python "sequence" return value back to an array.
>
> This is probably a stupid question, but why would you use different
> types for incoming and outgoing data flow?

A list is one particular kind of sequence.

See also
http://docs.python.org/reference/datamodel.html#the-standard-type-hierarchy

On ons, 2009-11-04 at 16:02 +0200, Peter Eisentraut wrote:
> Here is a patch to support arrays in PL/Python as parameters and
> return values.

Slightly updated version with fixed reference counting.

CREATE OR REPLACE FUNCTION incr(stuff int[]) RETURNS int[] AS $$
for x in stuff:
yield x+1
$$
LANGUAGE 'plpythonu';

# select incr(ARRAY[1,2,3]);
ERROR: invalid memory alloc request size 18446744073709551608
CONTEXT: while creating return value
PL/Python function "incr"

Suppose, it could be fixed by additional check in PLy_function_handler near line
947 :
if (proc->is_setof) {
...
}
else if (PyIter_Check(plrv))
{
ereport(ERROR,
(errcode(ERRCODE_DATATYPE_MISMATCH),
errmsg("returned object should be iterated"),
errdetail("PL/Python returns iterable object in non-setof returning
context")));
}

Peter Eisentraut wrote:
> On ons, 2009-11-04 at 16:02 +0200, Peter Eisentraut wrote:
>> Here is a patch to support arrays in PL/Python as parameters and
>> return values.
>
> Slightly updated version with fixed reference counting.
>
>
> ------------------------------------------------------------------------
>
>

--
Teodor Sigaev E-mail: teodor(at)sigaev(dot)ru
WWW: http://www.sigaev.ru/

On fre, 2009-11-13 at 18:46 +0300, Teodor Sigaev wrote:
> CREATE OR REPLACE FUNCTION incr(stuff int[]) RETURNS int[] AS $$
> for x in stuff:
> yield x+1
> $$
> LANGUAGE 'plpythonu';
>
> # select incr(ARRAY[1,2,3]);
> ERROR: invalid memory alloc request size 18446744073709551608
> CONTEXT: while creating return value
> PL/Python function "incr"

Fixed with additional error check and regression test. (The problem
could be more simply demonstrated by returning any non-sequence from the
function.) Thanks for catching it.

On Fri, Nov 20, 2009 at 12:00:24AM +0200, Peter Eisentraut wrote:
> On fre, 2009-11-13 at 18:46 +0300, Teodor Sigaev wrote:
> > CREATE OR REPLACE FUNCTION incr(stuff int[]) RETURNS int[] AS $$
> > for x in stuff:
> > yield x+1
> > $$
> > LANGUAGE 'plpythonu';
> >
> > # select incr(ARRAY[1,2,3]);
> > ERROR: invalid memory alloc request size 18446744073709551608
> > CONTEXT: while creating return value
> > PL/Python function "incr"
>
> Fixed with additional error check and regression test. (The problem
> could be more simply demonstrated by returning any non-sequence from the
> function.) Thanks for catching it.

I've finally gotten around to getting a review done, and basically it looks
good to me. There appears to be a problem with the tests in that the expected
output doesn't include the test_type_conversion_array_error() function
mentioned in sql/plpython_types.sql. Diff generated by the regression test is
attached. Other than that problem, though, the code looks fine to me (should I
presume to judge Peter's code? :). Aside from the problem mentioned above, the
tests work fine, and seem fairly comprehensive. Other testing I've done also
passes.

This patch doesn't include any documentation; my reading of the PL/Python docs
suggests that's probably acceptable, as the existing docs don't talk about its
array handling. That said, it might be useful to include an example, to show
for instance that identical PL/Python code could create either an array of a
type or a set of rows of that type:

5432 josh(at)josh*# create function return_set() returns setof int as $$ return
(1, 2, 3, 4, 5) $$ language plpythonu;
CREATE FUNCTION
5432 josh(at)josh*# create function return_arr() returns int[] as $$ return (1,
2, 3, 4, 5) $$ language plpythonu;
CREATE FUNCTION
5432 josh(at)josh*# select return_arr();
return_arr
-------------
{1,2,3,4,5}
(1 row)

5432 josh(at)josh*# select * from return_set();
return_set
------------
1
2
3
4
5
(5 rows)

Perhaps that's overkill, though.

--
Joshua Tolley / eggyknap
End Point Corporation
http://www.endpoint.com

On Fri, Nov 20, 2009 at 12:00:24AM +0200, Peter Eisentraut wrote:
> On fre, 2009-11-13 at 18:46 +0300, Teodor Sigaev wrote:
> > CREATE OR REPLACE FUNCTION incr(stuff int[]) RETURNS int[] AS $$
> > for x in stuff:
> > yield x+1
> > $$
> > LANGUAGE 'plpythonu';
> >
> > # select incr(ARRAY[1,2,3]);
> > ERROR: invalid memory alloc request size 18446744073709551608
> > CONTEXT: while creating return value
> > PL/Python function "incr"
>
> Fixed with additional error check and regression test. (The problem
> could be more simply demonstrated by returning any non-sequence from the
> function.) Thanks for catching it.

My last email claimed that the regression test needed some additional changes
to its expected output, and further claimed that it had the regression test's
diff attached. As was helpfully pointed out off-list, it actually wasn't
attached. Trying again..

-- Josh

On tis, 2009-12-01 at 20:53 -0700, Joshua Tolley wrote:
> This patch doesn't include any documentation; my reading of the PL/Python docs
> suggests that's probably acceptable, as the existing docs don't talk about its
> array handling. That said, it might be useful to include an example, to show
> for instance that identical PL/Python code could create either an array of a
> type or a set of rows of that type:

I added a bit of documentation like that. Thanks.