Re: [COMMITTERS] pgsql: PL/Python: Convert numeric to Decimal

PL/Python: Convert numeric to Decimal

The old implementation converted PostgreSQL numeric to Python float,
which was always considered a shortcoming. Now numeric is converted to
the Python Decimal object. Either the external cdecimal module or the
standard library decimal module are supported.

From: Szymon Guz <mabewlun(at)gmail(dot)com>
From: Ronan Dunklau <rdunklau(at)gmail(dot)com>
Reviewed-by: Steve Singer <steve(at)ssinger(dot)info>

Branch
------
master

Details
-------
http://git.postgresql.org/pg/commitdiff/7919398bac8bacd75ec5d763ce8b15ffaaa3e071

Modified Files
--------------
doc/src/sgml/plpython.sgml | 23 +++++++++---
src/pl/plpython/expected/plpython_types.out | 49 +++++++++++++++++++++----
src/pl/plpython/expected/plpython_types_3.out | 49 +++++++++++++++++++++----
src/pl/plpython/plpy_typeio.c | 42 ++++++++++++++++-----
src/pl/plpython/sql/plpython_types.sql | 9 ++++-
5 files changed, 138 insertions(+), 34 deletions(-)

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> PL/Python: Convert numeric to Decimal

Assorted buildfarm members don't like this patch.

regards, tom lane

On Sat, Jul 6, 2013 at 2:39 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> PL/Python: Convert numeric to Decimal
>
> Assorted buildfarm members don't like this patch.

Do you have failure details?

This is probably an attempt to operate decimals vs floats.

Ie: Decimal('3.0') > 0 works, but Decimal('3.0') > 1.3 doesn't
(decimal is explicitly forbidden from operating on floats, some design
decision that can only be disabled in 3.3).

On 07/06/2013 01:52 AM, Claudio Freire wrote:
> On Sat, Jul 6, 2013 at 2:39 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>>> PL/Python: Convert numeric to Decimal
>> Assorted buildfarm members don't like this patch.
>
> Do you have failure details?
>
> This is probably an attempt to operate decimals vs floats.
>
> Ie: Decimal('3.0') > 0 works, but Decimal('3.0') > 1.3 doesn't
> (decimal is explicitly forbidden from operating on floats, some design
> decision that can only be disabled in 3.3).
>
>

Instead of speculating, you can actually see for yourself. The dashboard
is at <http://www.pgbuildfarm.org/cgi-bin/show_status.pl> Pick one of
the machines failing at PLCheck-C and click its 'Details' link. Then
scroll down a bit and you'll see what is failing.

cheers

andrew

Look at that:

return x
$$ LANGUAGE plpythonu;
SELECT * FROM test_type_conversion_numeric(100);
! INFO: (Decimal('100'), 'Decimal')
CONTEXT: PL/Python function "test_type_conversion_numeric"
test_type_conversion_numeric
------------------------------
--- 219,225 ----
return x
$$ LANGUAGE plpythonu;
SELECT * FROM test_type_conversion_numeric(100);
! INFO: (Decimal("100"), 'Decimal')
CONTEXT: PL/Python function "test_type_conversion_numeric"
test_type_conversion_numeric
------------------------------

" instead of '

All the more reason to use as_tuple

On Sat, Jul 6, 2013 at 9:16 AM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
> On 07/06/2013 01:52 AM, Claudio Freire wrote:
>>
>> On Sat, Jul 6, 2013 at 2:39 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>
>>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>>>>
>>>> PL/Python: Convert numeric to Decimal
>>>
>>> Assorted buildfarm members don't like this patch.
>>
>>
>> Do you have failure details?
>>
>> This is probably an attempt to operate decimals vs floats.
>>
>> Ie: Decimal('3.0') > 0 works, but Decimal('3.0') > 1.3 doesn't
>> (decimal is explicitly forbidden from operating on floats, some design
>> decision that can only be disabled in 3.3).
>>
>>
>
>
> Instead of speculating, you can actually see for yourself. The dashboard is
> at <http://www.pgbuildfarm.org/cgi-bin/show_status.pl> Pick one of the
> machines failing at PLCheck-C and click its 'Details' link. Then scroll down
> a bit and you'll see what is failing.
>
> cheers
>
> andrew
>

On 6 July 2013 17:58, Claudio Freire <klaussfreire(at)gmail(dot)com> wrote:

> Look at that:
>
> return x
> $$ LANGUAGE plpythonu;
> SELECT * FROM test_type_conversion_numeric(100);
> ! INFO: (Decimal('100'), 'Decimal')
> CONTEXT: PL/Python function "test_type_conversion_numeric"
> test_type_conversion_numeric
> ------------------------------
> --- 219,225 ----
> return x
> $$ LANGUAGE plpythonu;
> SELECT * FROM test_type_conversion_numeric(100);
> ! INFO: (Decimal("100"), 'Decimal')
> CONTEXT: PL/Python function "test_type_conversion_numeric"
> test_type_conversion_numeric
> ------------------------------
>
> " instead of '
>
> All the more reason to use as_tuple
>
>
>
> On Sat, Jul 6, 2013 at 9:16 AM, Andrew Dunstan <andrew(at)dunslane(dot)net>
> wrote:
> >
> > On 07/06/2013 01:52 AM, Claudio Freire wrote:
> >>
> >> On Sat, Jul 6, 2013 at 2:39 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >>>
> >>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> >>>>
> >>>> PL/Python: Convert numeric to Decimal
> >>>
> >>> Assorted buildfarm members don't like this patch.
> >>
> >>
> >> Do you have failure details?
> >>
> >> This is probably an attempt to operate decimals vs floats.
> >>
> >> Ie: Decimal('3.0') > 0 works, but Decimal('3.0') > 1.3 doesn't
> >> (decimal is explicitly forbidden from operating on floats, some design
> >> decision that can only be disabled in 3.3).
> >>
> >>
> >
> >
> > Instead of speculating, you can actually see for yourself. The dashboard
> is
> > at <http://www.pgbuildfarm.org/cgi-bin/show_status.pl> Pick one of the
> > machines failing at PLCheck-C and click its 'Details' link. Then scroll
> down
> > a bit and you'll see what is failing.
> >
> > cheers
> >
> > andrew
> >
>
>
Hi,
I've modifled the tests to check the numeric->decimal conversion some other
way. They check now conversion to float/int and to string, and also tuple
values.

I've checked that on decimal and cdecimal on python 2.7 and 3.3. The
outputs are the same regardles the Python and decimal versions.

thanks,
Szymon

On Sun, 2013-07-07 at 02:01 -0300, Claudio Freire wrote:
> You really want to test more than just the str. The patch contains
> casts to int and float, which is something existing PLPython code will
> be doing, so it's good to test it still can be done with decimal.

Let's remember that we are regression testing PL/Python here, not
Python. The functionality of PL/Python is to convert a numeric to
Decimal, and that's what we are testing. What Python can or cannot do
with that is not our job to test.

> Existing python code will also expect the number to be a float, and
> will try to operate against other floats. That's not going to work
> anymore, that has to go into release notes.

Right, this will be listed in the release notes under upgrade caveats.

On Sun, 2013-07-07 at 17:21 +0200, Szymon Guz wrote:
> I think that these tests are much better, so they should go into
> trunk.
> As for Python 2.5 I think we could modify the code and makefile (with
> additional documentation info) so the decimal code wouldn't be
> compiled
> with python 2.5.

I'd welcome updated tests, if you want to work on that. But they would
need to work uniformly for Python 2.4 through 3.3.

On 7 July 2013 21:35, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:

> On Sun, 2013-07-07 at 17:21 +0200, Szymon Guz wrote:
> > I think that these tests are much better, so they should go into
> > trunk.
> > As for Python 2.5 I think we could modify the code and makefile (with
> > additional documentation info) so the decimal code wouldn't be
> > compiled
> > with python 2.5.
>
> I'd welcome updated tests, if you want to work on that. But they would
> need to work uniformly for Python 2.4 through 3.3.
>
>
>
Well... I don't know what to do and which solution is better.

This patch works, but the tests are not working on some old machines.

This patch works, but changes the plpython functions, so I assume that it
will provide errors to some existing functions. I've noticed yesterday that
you cannot run code like `Decimal(10) - float(10)`. So if a function
accepts a numeric parameter 'x', which currently is converted to float,
then the code like `x - float(10)` currently works, and will not work after
this change.

Introducing decimal.Decimal also breaks python earlier than 2.4, as the
decimal module has been introduced in 2.4. We could use the old conversion
for versions before 2.4, and the new for 2.4 and newer. Do we want it to
work like this? Do we want to have different behaviour for different python
versions? I'm not sure if anyone still uses Python 2.3, but I've already
realised that the patch breaks all the functions for 2.3 which use numeric
argument.

I assume that the patch will be rolled back, if it the tests don't work on
some machines, right?

szymon

On Sun, Jul 7, 2013 at 4:28 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On Sun, 2013-07-07 at 02:01 -0300, Claudio Freire wrote:
>> You really want to test more than just the str. The patch contains
>> casts to int and float, which is something existing PLPython code will
>> be doing, so it's good to test it still can be done with decimal.
>
> Let's remember that we are regression testing PL/Python here, not
> Python. The functionality of PL/Python is to convert a numeric to
> Decimal, and that's what we are testing. What Python can or cannot do
> with that is not our job to test.

I was just proposing to test behavior likely to be expected by
PL/Python functions, but you probably know better than I.