Re: Performance problem in PLPgSQL

Lists: pgsql-hackers
From: Marc Cousin <cousinmarc(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Performance problem in PLPgSQL
Date: 2013-07-23 10:02:38
Message-ID: 51EE54BE.3090606@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi,

I've been trying to diagnose a severe performance regression we've been
having in one of our plpgsql procedures.

The example below is of course extremely simplified, and obviously not
what we are really doing in the database, but it exhibits the slowdown
between 9.1.9 and 9.2.4.

So here is the example:

create table table_test_int (col_01 int);
create table table_test_numeric (col_01 numeric);

CREATE OR REPLACE FUNCTION public.test_insert(nb integer)
RETURNS text
LANGUAGE plpgsql
AS $function$
DECLARE
time_start timestamp;
time_stop timestamp;
tmp_numeric numeric;
BEGIN

time_start :=clock_timestamp();
FOR i IN 1..nb LOOP
INSERT INTO table_test_int(col_01) VALUES (i);
END LOOP;
time_stop :=clock_timestamp();
RAISE NOTICE 'time for int:%',time_stop-time_start;

time_start :=clock_timestamp();
FOR i IN 1..nb LOOP
INSERT INTO table_test_numeric(col_01) VALUES (i);
END LOOP;
time_stop :=clock_timestamp();
RAISE NOTICE 'time for numeric:%',time_stop-time_start;

time_start :=clock_timestamp();
FOR i IN 1..nb LOOP
INSERT INTO table_test_numeric(col_01) VALUES (i::numeric);
END LOOP;
time_stop :=clock_timestamp();
RAISE NOTICE 'time for numeric, casted:%',time_stop-time_start;

time_start :=clock_timestamp();
FOR i IN 1..nb LOOP
tmp_numeric:=cast(i as numeric);
INSERT INTO table_test_numeric(col_01) VALUES (tmp_numeric);
END LOOP;
time_stop :=clock_timestamp();
RAISE NOTICE 'time for numeric with tmp variable:%',time_stop-time_start;

RETURN 1;
END;
$function$
;

It just inserts nb records in a loop in 4 different maneers:
- Directly in an int field
- Then in a numeric field (that's where we're having problems)
- Then in the same numeric field, but trying a cast (it doesn't change a
thing)
- Then tries with an intermediary temp variable of numeric type (which
solves the problem).

Here are the runtimes (tables were truncated beforehand):

9.1.9:
select test_insert(1000000);
NOTICE: time for int:00:00:09.526009
NOTICE: time for numeric:00:00:10.557126
NOTICE: time for numeric, casted:00:00:10.821369
NOTICE: time for numeric with tmp variable:00:00:10.850847

9.2.4:
select test_insert(1000000);
NOTICE: time for int:00:00:09.477044
NOTICE: time for numeric:00:00:24.757032 <----
NOTICE: time for numeric, casted:00:00:24.791016 <----
NOTICE: time for numeric with tmp variable:00:00:10.89332

I really don't know exactly where the problem comes from… but it's been
hurting a function very badly (there are several of these static queries
with types mismatch). And of course, the problem is not limited to
numeric… text has the exact same problem.

Regards,

Marc


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Marc Cousin <cousinmarc(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-07-23 14:00:18
Message-ID: 352.1374588018@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
> The example below is of course extremely simplified, and obviously not
> what we are really doing in the database, but it exhibits the slowdown
> between 9.1.9 and 9.2.4.

Hm. Some experimentation shows that the plan cache is failing to switch
to a generic plan, but I'm not sure why the cast would have anything to
do with that ...

regards, tom lane


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Marc Cousin <cousinmarc(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-07-23 22:53:25
Message-ID: 28356.1374620005@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

I wrote:
> Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
>> The example below is of course extremely simplified, and obviously not
>> what we are really doing in the database, but it exhibits the slowdown
>> between 9.1.9 and 9.2.4.

> Hm. Some experimentation shows that the plan cache is failing to switch
> to a generic plan, but I'm not sure why the cast would have anything to
> do with that ...

Hah, I see why:

(gdb) s
1009 if (plansource->generic_cost < avg_custom_cost * 1.1)
(gdb) p plansource->generic_cost
$18 = 0.012500000000000001
(gdb) p avg_custom_cost
$19 = 0.01
(gdb) p avg_custom_cost * 1.1
$20 = 0.011000000000000001

That is, the estimated cost of the custom plan is just the evaluation
time for a simple constant, while the estimated cost of the generic plan
includes a charge for evaluation of int4_numeric(). That makes the
latter more than ten percent more expensive, and since this logic isn't
considering anything else at all (particularly not the cost of
planning), it thinks that makes the custom plan worth picking.

We've been around on this before, but not yet thought of a reasonable
way to estimate planning cost, let alone compare it to estimated
execution costs. Up to now I hadn't thought that was a particularly
urgent issue, but this example shows it's worth worrying about.

One thing that was suggested in the previous discussion is to base the
planning cost estimate on the length of the rangetable. We could
do something trivial like add "10 * (length(plan->rangetable) + 1)"
in this comparison.

Another thing that would fix this particular issue, though perhaps not
related ones, is to start charging something nonzero for ModifyTable
nodes, say along the lines of one seq_page_cost per inserted/modified
row. That would knock the total estimated cost for an INSERT up enough
that the ten percent threshold wouldn't be exceeded.

Thoughts?

regards, tom lane


From: Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, pgsql-hackers(at)postgresql(dot)org
Subject: Failure to use generic plans (was: Re: Performance problem in PLPgSQL)
Date: 2013-07-23 23:47:24
Message-ID: 0e6d70debaf899bfbf37edcdfbd03e21@news-out.riddles.org.uk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

As failures to use a generic plan goes, that one's fairly tame. I've
seen much worse. For example:

PREPARE foo(integer[]) AS SELECT * FROM complexview WHERE id = ANY ($1);

where the caller typically supplies 1-5 array elements (or any number
less than 10, because generic parameter arrays are assumed to have 10
elements). This one can be a massive performance regression between
9.1 and 9.2; the first guy who mentioned this on IRC was getting a 40x
slowdown (~20ms planning time vs. 0.5ms execution time).

--
Andrew (irc:RhodiumToad)


From: Amit Kapila <amit(dot)kapila(at)huawei(dot)com>
To: "'Tom Lane'" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "'Marc Cousin'" <cousinmarc(at)gmail(dot)com>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-07-24 06:07:46
Message-ID: 000101ce8834$1e7ba660$5b72f320$@kapila@huawei.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wednesday, July 24, 2013 4:23 AM Tom Lane wrote:
> I wrote:
> > Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
> >> The example below is of course extremely simplified, and obviously
> not
> >> what we are really doing in the database, but it exhibits the
> slowdown
> >> between 9.1.9 and 9.2.4.
>
> > Hm. Some experimentation shows that the plan cache is failing to
> switch
> > to a generic plan, but I'm not sure why the cast would have anything
> to
> > do with that ...
>
> Hah, I see why:
>
> (gdb) s
> 1009 if (plansource->generic_cost < avg_custom_cost * 1.1)
> (gdb) p plansource->generic_cost
> $18 = 0.012500000000000001
> (gdb) p avg_custom_cost
> $19 = 0.01
> (gdb) p avg_custom_cost * 1.1
> $20 = 0.011000000000000001
>
> That is, the estimated cost of the custom plan is just the evaluation
> time for a simple constant, while the estimated cost of the generic
> plan
> includes a charge for evaluation of int4_numeric(). That makes the
> latter more than ten percent more expensive, and since this logic isn't
> considering anything else at all (particularly not the cost of
> planning), it thinks that makes the custom plan worth picking.
>
> We've been around on this before, but not yet thought of a reasonable
> way to estimate planning cost, let alone compare it to estimated
> execution costs. Up to now I hadn't thought that was a particularly
> urgent issue, but this example shows it's worth worrying about.
>
> One thing that was suggested in the previous discussion is to base the
> planning cost estimate on the length of the rangetable. We could
> do something trivial like add "10 * (length(plan->rangetable) + 1)"
> in this comparison.
>
> Another thing that would fix this particular issue, though perhaps not
> related ones, is to start charging something nonzero for ModifyTable
> nodes, say along the lines of one seq_page_cost per inserted/modified
> row. That would knock the total estimated cost for an INSERT up enough
> that the ten percent threshold wouldn't be exceeded.

Shouldn't it consider new value of boundparam to decide whether a new custom
plan is needed,
as that can be one of the main reason why it would need different plan.

Current behavior is either it will choose generic plan or build a new custom
plan with new parameters based on
Choose_custom_plan().

Shouldn't the behavior of this be as below:
a. choose generic plan
b. choose one of existing custom plan
c. create new custom plan

The choice can be made based on the new value of bound parameter.

With Regards,
Amit Kapila.


From: Amit Kapila <amit(dot)kapila(at)huawei(dot)com>
To: "'Tom Lane'" <tgl(at)sss(dot)pgh(dot)pa(dot)us>, "'Marc Cousin'" <cousinmarc(at)gmail(dot)com>
Cc: <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-07-24 09:04:34
Message-ID: 000b01ce884c$d17a0550$746e0ff0$@kapila@huawei.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wednesday, July 24, 2013 11:38 AM Amit Kapila wrote:
> On Wednesday, July 24, 2013 4:23 AM Tom Lane wrote:
> > I wrote:
> > > Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
> > >> The example below is of course extremely simplified, and obviously
> > not
> > >> what we are really doing in the database, but it exhibits the
> > slowdown
> > >> between 9.1.9 and 9.2.4.
> >
> > > Hm. Some experimentation shows that the plan cache is failing to
> > switch
> > > to a generic plan, but I'm not sure why the cast would have
> anything
> > to
> > > do with that ...
> >
> > Hah, I see why:
> >
> > (gdb) s
> > 1009 if (plansource->generic_cost < avg_custom_cost * 1.1)
> > (gdb) p plansource->generic_cost
> > $18 = 0.012500000000000001
> > (gdb) p avg_custom_cost
> > $19 = 0.01
> > (gdb) p avg_custom_cost * 1.1
> > $20 = 0.011000000000000001
> >
> > That is, the estimated cost of the custom plan is just the evaluation
> > time for a simple constant, while the estimated cost of the generic
> > plan
> > includes a charge for evaluation of int4_numeric(). That makes the
> > latter more than ten percent more expensive, and since this logic
> isn't
> > considering anything else at all (particularly not the cost of
> > planning), it thinks that makes the custom plan worth picking.
> >
> > We've been around on this before, but not yet thought of a reasonable
> > way to estimate planning cost, let alone compare it to estimated
> > execution costs. Up to now I hadn't thought that was a particularly
> > urgent issue, but this example shows it's worth worrying about.
> >
> > One thing that was suggested in the previous discussion is to base
> the
> > planning cost estimate on the length of the rangetable. We could
> > do something trivial like add "10 * (length(plan->rangetable) + 1)"
> > in this comparison.
> >
> > Another thing that would fix this particular issue, though perhaps
> not
> > related ones, is to start charging something nonzero for ModifyTable
> > nodes, say along the lines of one seq_page_cost per inserted/modified
> > row. That would knock the total estimated cost for an INSERT up
> enough
> > that the ten percent threshold wouldn't be exceeded.
>
> Shouldn't it consider new value of boundparam to decide whether a new
> custom
> plan is needed,
> as that can be one of the main reason why it would need different plan.
>
> Current behavior is either it will choose generic plan or build a new
> custom
> plan with new parameters based on
> Choose_custom_plan().
>
> Shouldn't the behavior of this be as below:
> a. choose generic plan
> b. choose one of existing custom plan
> c. create new custom plan
>
> The choice can be made based on the new value of bound parameter.

For the case of Insert where no scan is involved (as in this case), do we
anytime need different plan?
Can we always use generic plan for such cases?

With Regards,
Amit Kapila.


From: Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>
To: Marc Cousin <cousinmarc(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-23 18:10:26
Message-ID: CAAY+2jbjLuHrv7W8hzffwVnv5cLA1nbs_7ChVy3fdEiijepyTA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

> It just inserts nb records in a loop in 4 different maneers:
> - Directly in an int field
> - Then in a numeric field (that's where we're having problems)
> - Then in the same numeric field, but trying a cast (it doesn't change a
> thing)
> - Then tries with an intermediary temp variable of numeric type (which
> solves the problem).
>
>
> Here are the runtimes (tables were truncated beforehand):
>
> 9.1.9:
> select test_insert(1000000);
> NOTICE: time for int:00:00:09.526009
> NOTICE: time for numeric:00:00:10.557126
> NOTICE: time for numeric, casted:00:00:10.821369
> NOTICE: time for numeric with tmp variable:00:00:10.850847
>
>
> 9.2.4:
> select test_insert(1000000);
> NOTICE: time for int:00:00:09.477044
> NOTICE: time for numeric:00:00:24.757032 <----
> NOTICE: time for numeric, casted:00:00:24.791016 <----
> NOTICE: time for numeric with tmp variable:00:00:10.89332
>
>
> I really don't know exactly where the problem comes from… but it's been
> hurting a function very badly (there are several of these static queries
> with types mismatch). And of course, the problem is not limited to
> numeric… text has the exact same problem.
>
> Regards,
>
> Marc
>
>
I got the same problem today. Unfortunately, we need to rollback to 9.1 in
our production site. Of course the team needed to make better tests before
go to production. Of course they really need to write better functions in
PL/pgSQL, but this problem was a really "no go" for us.

Just don't let this gotcha gone in our to do.

--
Regards,
Fábio Telles Rodriguez
blog: http:// <http://www.midstorm.org/~telles/>s<http://tellesr.wordpress.com/>
avepoint.blog.br
e-mail / gtalk / MSN: fabio(dot)telles(at)gmail(dot)com
Skype: fabio_telles

Timbira - The Brazilian Postgres Company
http://www.timbira.com.br


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>
Cc: Marc Cousin <cousinmarc(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-23 18:45:05
Message-ID: CAFj8pRAPn3erFuung=CAbk2wUOHVetRLkuiwiVzLQeyJX3Y6HA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

2013/8/23 Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>

>
> It just inserts nb records in a loop in 4 different maneers:
>> - Directly in an int field
>> - Then in a numeric field (that's where we're having problems)
>> - Then in the same numeric field, but trying a cast (it doesn't change a
>> thing)
>> - Then tries with an intermediary temp variable of numeric type (which
>> solves the problem).
>>
>>
>> Here are the runtimes (tables were truncated beforehand):
>>
>> 9.1.9:
>> select test_insert(1000000);
>> NOTICE: time for int:00:00:09.526009
>> NOTICE: time for numeric:00:00:10.557126
>> NOTICE: time for numeric, casted:00:00:10.821369
>> NOTICE: time for numeric with tmp variable:00:00:10.850847
>>
>>
>> 9.2.4:
>> select test_insert(1000000);
>> NOTICE: time for int:00:00:09.477044
>> NOTICE: time for numeric:00:00:24.757032 <----
>> NOTICE: time for numeric, casted:00:00:24.791016 <----
>> NOTICE: time for numeric with tmp variable:00:00:10.89332
>>
>>
>> I really don't know exactly where the problem comes from… but it's been
>> hurting a function very badly (there are several of these static queries
>> with types mismatch). And of course, the problem is not limited to
>> numeric… text has the exact same problem.
>>
>> Regards,
>>
>> Marc
>>
>>
> I got the same problem today. Unfortunately, we need to rollback to 9.1 in
> our production site. Of course the team needed to make better tests before
> go to production. Of course they really need to write better functions in
> PL/pgSQL, but this problem was a really "no go" for us.
>

please, can you send a self explained test

this issue should be fixed, and we need a examples.

>
> Just don't let this gotcha gone in our to do.
>
> --
> Regards,
> Fábio Telles Rodriguez
> blog: http:// <http://www.midstorm.org/~telles/>s<http://tellesr.wordpress.com/>
> avepoint.blog.br
> e-mail / gtalk / MSN: fabio(dot)telles(at)gmail(dot)com
> Skype: fabio_telles
>
> Timbira - The Brazilian Postgres Company
> http://www.timbira.com.br
>


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
Cc: Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>, Marc Cousin <cousinmarc(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-23 21:55:00
Message-ID: 23431.1377294900@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
> please, can you send a self explained test
> this issue should be fixed, and we need a examples.

We already had a perfectly good example at the beginning of this thread.
What's missing is a decision on how we ought to approximate the cost of
planning (relative to execution costs).

As I mentioned upthread, it doesn't seem unreasonable to me to do
something quick-and-dirty based on the length of the plan's rangetable.
Pretty nearly anything would fix these specific situations where the
estimated execution cost is negligible. It's possible that there are
more complicated cases where we'll need a more accurate estimate, but
we've not seen an example of that yet.

My previous suggestion was to estimate planning cost as
10 * (length(plan->rangetable) + 1)
but on reflection it ought to be scaled by one of the cpu cost constants,
so perhaps
1000 * cpu_operator_cost * (length(plan->rangetable) + 1)
which'd mean a custom plan has to be estimated to save a minimum of
about 5 cost units (more if more than 1 table is used) before it'll
be chosen. I'm tempted to make the multiplier be 10000 not 1000,
but it seems better to be conservative about changing the behavior
until we see how well this works in practice.

Objections, better ideas?

regards, tom lane


From: Marc Cousin <cousinmarc(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>, Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-24 17:19:52
Message-ID: 5218EB38.3010304@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 23/08/2013 23:55, Tom Lane wrote:
> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
>> please, can you send a self explained test
>> this issue should be fixed, and we need a examples.
> We already had a perfectly good example at the beginning of this thread.
> What's missing is a decision on how we ought to approximate the cost of
> planning (relative to execution costs).
>
> As I mentioned upthread, it doesn't seem unreasonable to me to do
> something quick-and-dirty based on the length of the plan's rangetable.
> Pretty nearly anything would fix these specific situations where the
> estimated execution cost is negligible. It's possible that there are
> more complicated cases where we'll need a more accurate estimate, but
> we've not seen an example of that yet.
>
> My previous suggestion was to estimate planning cost as
> 10 * (length(plan->rangetable) + 1)
> but on reflection it ought to be scaled by one of the cpu cost constants,
> so perhaps
> 1000 * cpu_operator_cost * (length(plan->rangetable) + 1)
> which'd mean a custom plan has to be estimated to save a minimum of
> about 5 cost units (more if more than 1 table is used) before it'll
> be chosen. I'm tempted to make the multiplier be 10000 not 1000,
> but it seems better to be conservative about changing the behavior
> until we see how well this works in practice.
>
> Objections, better ideas?
>
> regards, tom lane
No better idea as far as I'm concerned, of course :)

But it is a bit tricky to understand what is going on when you get
hit by it, and using a very approximated cost of the planning time
seems the most logical to me. So I'm all for this solution.


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Marc Cousin <cousinmarc(at)gmail(dot)com>
Cc: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>, Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-24 19:16:53
Message-ID: 7518.1377371813@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
> On 23/08/2013 23:55, Tom Lane wrote:
>> My previous suggestion was to estimate planning cost as
>> 10 * (length(plan->rangetable) + 1)
>> but on reflection it ought to be scaled by one of the cpu cost constants,
>> so perhaps
>> 1000 * cpu_operator_cost * (length(plan->rangetable) + 1)
>> which'd mean a custom plan has to be estimated to save a minimum of
>> about 5 cost units (more if more than 1 table is used) before it'll
>> be chosen. I'm tempted to make the multiplier be 10000 not 1000,
>> but it seems better to be conservative about changing the behavior
>> until we see how well this works in practice.
>>
>> Objections, better ideas?

> No better idea as far as I'm concerned, of course :)

> But it is a bit tricky to understand what is going on when you get
> hit by it, and using a very approximated cost of the planning time
> seems the most logical to me. So I'm all for this solution.

I've pushed a patch along this line. I verified it fixes your original
example, but maybe you could try it on your real application?
http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=005f583ba4e6d4d19b62959ef8e70a3da4d188a5

regards, tom lane


From: Marc Cousin <cousinmarc(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>, Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-25 07:22:16
Message-ID: 5219B0A8.2040701@gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 24/08/2013 21:16, Tom Lane wrote:
> Marc Cousin <cousinmarc(at)gmail(dot)com> writes:
>> On 23/08/2013 23:55, Tom Lane wrote:
>>> My previous suggestion was to estimate planning cost as
>>> 10 * (length(plan->rangetable) + 1)
>>> but on reflection it ought to be scaled by one of the cpu cost constants,
>>> so perhaps
>>> 1000 * cpu_operator_cost * (length(plan->rangetable) + 1)
>>> which'd mean a custom plan has to be estimated to save a minimum of
>>> about 5 cost units (more if more than 1 table is used) before it'll
>>> be chosen. I'm tempted to make the multiplier be 10000 not 1000,
>>> but it seems better to be conservative about changing the behavior
>>> until we see how well this works in practice.
>>>
>>> Objections, better ideas?
>> No better idea as far as I'm concerned, of course :)
>> But it is a bit tricky to understand what is going on when you get
>> hit by it, and using a very approximated cost of the planning time
>> seems the most logical to me. So I'm all for this solution.
> I've pushed a patch along this line. I verified it fixes your original
> example, but maybe you could try it on your real application?
> http://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=005f583ba4e6d4d19b62959ef8e70a3da4d188a5
>
> regards, tom lane
I think that won't be possible :(

It's one of those environments where you have to ask lots of permissions
before doing anything. I'll do my best to have them do a test with this
patch.

Thanks a lot.

Marc


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Fábio Telles Rodriguez <fabio(dot)telles(at)gmail(dot)com>, Marc Cousin <cousinmarc(at)gmail(dot)com>, PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-08-25 18:27:08
Message-ID: CAFj8pRDq24w_efgPcE_KCs_GtyvAP0F-AhXdXPeRRa1rBcEN1Q@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

2013/8/23 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>

> Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> writes:
> > please, can you send a self explained test
> > this issue should be fixed, and we need a examples.
>
> We already had a perfectly good example at the beginning of this thread.
> What's missing is a decision on how we ought to approximate the cost of
> planning (relative to execution costs).
>
> As I mentioned upthread, it doesn't seem unreasonable to me to do
> something quick-and-dirty based on the length of the plan's rangetable.
> Pretty nearly anything would fix these specific situations where the
> estimated execution cost is negligible. It's possible that there are
> more complicated cases where we'll need a more accurate estimate, but
> we've not seen an example of that yet.
>
> My previous suggestion was to estimate planning cost as
> 10 * (length(plan->rangetable) + 1)
> but on reflection it ought to be scaled by one of the cpu cost constants,
> so perhaps
> 1000 * cpu_operator_cost * (length(plan->rangetable) + 1)
> which'd mean a custom plan has to be estimated to save a minimum of
> about 5 cost units (more if more than 1 table is used) before it'll
> be chosen. I'm tempted to make the multiplier be 10000 not 1000,
> but it seems better to be conservative about changing the behavior
> until we see how well this works in practice.
>
> Objections, better ideas?
>

I am thinking so this problem is little bit more complex and using only
this formula is too much simplification - although it is big step forward.

* first 5 queries uses a custom plan - it means so lot of procedures uses
custom plan for ever (if are executed without pooling, because almost all
functions with SQL are not called twice in one connect ) - and there are
really only a few reports related to prepared statements or PL/pgSQL
performance - so it can demonstrates so planning in PostgreSQL is relative
fast process and probably we don't be afraid of more wide using custom
plans. Custom plans has a nice a secondary effect - it solve a problems
with predicates in form: field = some_constant OR field IS NULL without any
special support in planner. But it sometimes 6. query can be slow, because
a generic plan is used.

where we can expect a performance problems?

* frequently fast simple statements:

** INSERT INTO table, { UPDATE | DELETE | SELECT } WHERE PK= const - these
queries can use a generic plan directly

* planer expensive queries with fast result - these queries can use a
generic plans too, with some logic as you describe.

In other cases probably using a custom plans doesn't do a performance
issue, we can use it directly.

What I see, a prepared plans (prepared statements) are used now more due
protection against SQL injection than due saving a planner time - and badly
using a generic plan is more worse than repeated planning.

P.S. Can be magic constant 5 (using custom plans) controlled via GUC? Then
we can have very good control for some special using where default
mechanism fails (0 .. use a generic plans ever, -1 use a generic plan newer)

Regards

Pavel

> regards, tom lane
>


From: dlight <avinfo79(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-09-18 11:47:58
Message-ID: 1379504878339-5771405.post@n5.nabble.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

So if I run function 10000 with varible inside the query in one session it's
replan each time.

But if whant to teling postgres do not doing this, what shoud i do?

We have more than 10000 runs in one session with varible inside sql. And
have big performance problem in 9.2 and 9.3.

Here is my tests.
http://postgresql.1045698.n5.nabble.com/function-execute-on-v-9-2-slow-down-tp5767170p5771403.html

We want change pg version but can't doing that because of performance
regression in 9.2 and 9.3.:(

--
View this message in context: http://postgresql.1045698.n5.nabble.com/Performance-problem-in-PLPgSQL-tp5764796p5771405.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.


From: Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>
To: dlight <avinfo79(at)gmail(dot)com>
Cc: PostgreSQL Hackers <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Performance problem in PLPgSQL
Date: 2013-09-18 15:19:05
Message-ID: CAFj8pRAup734osQ5r4nNYUJ2V9Rc_V-UFRKV56_RbaUXBJfdVA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hello

can you try this patch (commit)
https://github.com/postgres/postgres/commit/a5f11e24a4d1afb213c780812a3df14c04d7f845#diff-fc73a24ee7d0692c2a0c639870223d70?

Regards

Pavel

2013/9/18 dlight <avinfo79(at)gmail(dot)com>

> So if I run function 10000 with varible inside the query in one session
> it's
> replan each time.
>
> But if whant to teling postgres do not doing this, what shoud i do?
>
> We have more than 10000 runs in one session with varible inside sql. And
> have big performance problem in 9.2 and 9.3.
>
> Here is my tests.
>
> http://postgresql.1045698.n5.nabble.com/function-execute-on-v-9-2-slow-down-tp5767170p5771403.html
>
> We want change pg version but can't doing that because of performance
> regression in 9.2 and 9.3.:(
>
>
>
>
>
> --
> View this message in context:
> http://postgresql.1045698.n5.nabble.com/Performance-problem-in-PLPgSQL-tp5764796p5771405.html
> Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>


From: Andrew Gierth <andrew(at)tao11(dot)riddles(dot)org(dot)uk>
To: dlight <avinfo79(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-09-18 15:47:16
Message-ID: 87vc1yxhij.fsf@news-spur.riddles.org.uk
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

>>>>> "dlight" == dlight <avinfo79(at)gmail(dot)com> writes:

dlight> So if I run function 10000 with varible inside the query in
dlight> one session it's replan each time.

dlight> But if whant to teling postgres do not doing this, what shoud
dlight> i do?

dlight> We have more than 10000 runs in one session with varible
dlight> inside sql. And have big performance problem in 9.2 and 9.3.

dlight> Here is my tests.
dlight> http://postgresql.1045698.n5.nabble.com/function-execute-on-v-9-2-slow-down-tp5767170p5771403.html

One interesting question is: why on earth is that query taking ~500ms
just to plan?

(Second question is, what are the plans and costs for the generic and
custom plans being generated, but that's harder to determine)

--
Andrew (irc:RhodiumToad)


From: dlight <avinfo79(at)gmail(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Re: Performance problem in PLPgSQL
Date: 2013-09-19 16:30:05
Message-ID: 1379608205065-5771639.post@n5.nabble.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

We intsall postgresql 9.3.0 server from FreeBSD ports
http://svnweb.freebsd.org/ports/head/databases/postgresql93-server/ the
administrator says that he already contains this patch.

--
View this message in context: http://postgresql.1045698.n5.nabble.com/Performance-problem-in-PLPgSQL-tp5764796p5771639.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.