Re: enable_joinremoval

Lists: pgsql-hackers
From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: enable_joinremoval
Date: 2010-03-29 08:33:50
Message-ID: 1269851630.3684.3636.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers


We have planner method parameters for many important parts of the
planner. We don't have any parameter for join removal, AFAIK.

The first question I get asked is "can I see the tables it removed?".
This has been asked of me 3 times now this year, always first question.
Same issue applies to constraint exclusion.

But we *can* do this with constraint_exclusion, simply by turning it off
and checking the two outputs.

So I think we need a parameter for join removal also.

I don't want to turn it off, but I think we need a way for people to
check themselves that the removal of the joins is not an error. I
foresee many false bug reports along the lines of "optimizer ate my join
and I want it back".

--
Simon Riggs www.2ndQuadrant.com


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 13:37:33
Message-ID: 603c8f071003290637q14d431f7j90afabd6fd2ec952@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 4:33 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>
> We have planner method parameters for many important parts of the
> planner. We don't have any parameter for join removal, AFAIK.
>
> The first question I get asked is "can I see the tables it removed?".
> This has been asked of me 3 times now this year, always first question.
> Same issue applies to constraint exclusion.
>
> But we *can* do this with constraint_exclusion, simply by turning it off
> and checking the two outputs.
>
> So I think we need a parameter for join removal also.
>
> I don't want to turn it off, but I think we need a way for people to
> check themselves that the removal of the joins is not an error. I
> foresee many false bug reports along the lines of "optimizer ate my join
> and I want it back".

I had this in my original patch but Tom wanted it taken out. In many
cases it's not horribly difficult to work around because you can do
SELECT * FROM ... instead of your original select list, but there
might be some cases with multiple levels of views where it isn't that
easy. I think it would be good to add this back.

An even worse problem I've been noticing is that there is no easy way
to determine whether the planner's new penchant for inserting
Materialize notes in all sorts of fun and exciting places is in fact
improving performance or not. I'm not sure there's even a difficult
way. I really, really think we need to do something out this - the
current situation is really quite horrible.

...Robert


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 14:36:14
Message-ID: 12679.1269873374@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Mar 29, 2010 at 4:33 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> So I think we need a parameter for join removal also.

> I had this in my original patch but Tom wanted it taken out.

And I still don't want it. We are NOT going in the direction of adding
an enable_ knob for every single planner activity --- do you have the
faintest idea how many there would be? We have such knobs for a small
number of cases where it's arguable that the action might be the wrong
thing for a particular query. Join removal, if applicable, can't
possibly be the wrong choice; it's better than every other join strategy.

regards, tom lane


From: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 14:42:19
Message-ID: 20100329144219.GA3925@alvh.no-ip.org
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Tom Lane escribió:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > On Mon, Mar 29, 2010 at 4:33 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> >> So I think we need a parameter for join removal also.
>
> > I had this in my original patch but Tom wanted it taken out.
>
> And I still don't want it. We are NOT going in the direction of adding
> an enable_ knob for every single planner activity --- do you have the
> faintest idea how many there would be? We have such knobs for a small
> number of cases where it's arguable that the action might be the wrong
> thing for a particular query. Join removal, if applicable, can't
> possibly be the wrong choice; it's better than every other join strategy.

It seems that what's really needed is some debug output to be able to
find out what it did.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:01:40
Message-ID: 13178.1269874900@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> It seems that what's really needed is some debug output to be able to
> find out what it did.

Isn't EXPLAIN good enough?

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:20:21
Message-ID: 603c8f071003290820h3553d223le3b5228f789608f9@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 10:36 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Mon, Mar 29, 2010 at 4:33 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>> So I think we need a parameter for join removal also.
>
>> I had this in my original patch but Tom wanted it taken out.
>
> And I still don't want it.  We are NOT going in the direction of adding
> an enable_ knob for every single planner activity --- do you have the
> faintest idea how many there would be?  We have such knobs for a small
> number of cases where it's arguable that the action might be the wrong
> thing for a particular query.  Join removal, if applicable, can't
> possibly be the wrong choice; it's better than every other join strategy.

As Tom Lane would put it, you're attacking a straw man. No one has
proposed adding a planner knob for "every single planner activity".
What Simon and I have proposed is adding some planner knobs that are
virtually parallel to the existing ones, which are useful, and your
argument (or rather, your assertion) against that is apparently that
somehow even thought the OTHER ones are useful, these almost identical
ones for cases that aren't covered by the existing knobs will be
unuseful.

The argument that we don't need a knob to control join removal because
it has to always be the best strategy presuposes that enable_seqscan
exists because a sequential scan might not be the best strategy, or
that enable_hashjoin exists because a hash join might not be the best
strategy, which is emphatically nonsense. If that were the purpose of
those knobs, that would by definition make them planner hints, albeit
incredibly poorly designed ones, and we would be having discussions
about the best way to turn them into more useful planner hints, as by
allowing them to apply to only certain portions of the query tree
and/or changing them from booleans to floats so that you could vary
the relative level of discouragement given to any particular planner
method, rather than only allowing zero and infinity. Of course, we
are NOT having those conversations because that ISN'T the purpose of
those knobs. Rather, their purpose, at least AIUI, is to allow the
user to see what the planner would have done had it not had those
strategies as its disposal - which is just as legitimate for join
removal or materialization as it is for hash join or merge join. If
someone wants to ask a question like "how much does join removal speed
up this query?" or "how much does this material node (that didn't
exist in 8.4) speed up or slow down this query?", there is going to be
no easy way for them to understand that without these knobs.

And for the record, I believe I find it rather amusing that you're
asking me if I "have the faintest idea how many there would be". I
venture to say that after yourself I might be the person who knows
this code best. I know how many there will be, if I get my way, and
that number is two. The query planner is a great piece of code but it
is not so transparently simple that it doesn't need debugging or
instrumentation, and "why did the planner do X" has got to be one of
our top ten most-frequently asked questions. Failing to provide a few
trivially simple debugging tools for the small number of major new
planner features added in 9.0 is essentially pointless cruelty and
will result in an endless series of unanswerable questions on
-performance, particularly where materialization is concerned, since
for join removal there is at least a workaround (maybe not such an
easy one to use in all cases, but it exists - just select all the
columns).

...Robert


From: Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>
To: Alvaro Herrera <alvherre(at)commandprompt(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:23:40
Message-ID: 3073cc9b1003290823g52615893j8183d6e70fb510d6@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 10:42 AM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
>
> It seems that what's really needed is some debug output to be able to
> find out what it did.
>
>

+1, i was preparing an env for testing this but in the while i was
thinking how can i know what happens... counting the tables in the
explain analyze? it's possible but not ideal

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:27:26
Message-ID: 13795.1269876446@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> And for the record, I believe I find it rather amusing that you're
> asking me if I "have the faintest idea how many there would be".

Actually that was directed more at Simon.

> I venture to say that after yourself I might be the person who knows
> this code best. I know how many there will be, if I get my way, and
> that number is two.

If you're speaking of adding a switch for the materialize-insertion
behavior, I didn't object to that; I agree that turning that off might
be an interesting thing to do. But I remain of the opinion that a
switch to disable join removal is just useless code and user-visible
complexity.

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:29:25
Message-ID: 603c8f071003290829r1f41609cs1d5c35b19fcb47bc@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 11:23 AM, Jaime Casanova
<jcasanov(at)systemguards(dot)com(dot)ec> wrote:
> On Mon, Mar 29, 2010 at 10:42 AM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
>>
>> It seems that what's really needed is some debug output to be able to
>> find out what it did.
>
> +1, i was preparing an env for testing this but in the while i was
> thinking how can i know what happens... counting the tables in the
> explain analyze? it's possible but not ideal

I don't actually see much value in making EXPLAIN show which tables
were removed. The usefulness of doing that presumes that the user is
asking a particular and very specific question, namely, exactly which
tables got removed?, or perhaps, did this particular table get
removed? The latter question is not too hard to figure out anyway -
just look for that table in the query. The former is a little harder
but if you really need to know you can always look at which tables
WERE referenced and work backwards from there, or write a little
script to post-process the JSON, XML, or YAML output. Of course
sometimes it would be handy to have but if we start adding too many
bells and whistles to the EXPLAIN output it will get cluttered and
unuseful.

I think the value of an enable_joinremoval switch is that it lets
someone throw the switch and see how the plan changes. It might
change in a very simple way or it might have a dramatic effect on the
whole plan.

...Robert


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:32:04
Message-ID: 603c8f071003290832u23130f2x84ffa459e312269c@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 11:27 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> And for the record, I believe I find it rather amusing that you're
>> asking me if I "have the faintest idea how many there would be".
>
> Actually that was directed more at Simon.
>
>> I venture to say that after yourself I might be the person who knows
>> this code best.  I know how many there will be, if I get my way, and
>> that number is two.
>
> If you're speaking of adding a switch for the materialize-insertion
> behavior, I didn't object to that; I agree that turning that off might
> be an interesting thing to do.  But I remain of the opinion that a
> switch to disable join removal is just useless code and user-visible
> complexity.

OK, I'll write a patch for that; and a consensus emerges that we
should also have enable_joinremoval, then I will add that as well. I
think the only argument for NOT having enable_joinremoval is that you
can always modify the query to say SELECT * rather than some more
specific SELECT list, but I think when there are several levels of
views involved it may not be so simple - you'll have to define
temporary versions of all the intermediate views, which is possibly
somewhat painful and certainly error-prone.

...Robert


From: Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Alvaro Herrera <alvherre(at)commandprompt(dot)com>, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:42:33
Message-ID: 3073cc9b1003290842k6a5e1986t9bf1992a816c8547@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 11:29 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Mon, Mar 29, 2010 at 11:23 AM, Jaime Casanova
> <jcasanov(at)systemguards(dot)com(dot)ec> wrote:
>> On Mon, Mar 29, 2010 at 10:42 AM, Alvaro Herrera
>> <alvherre(at)commandprompt(dot)com> wrote:
>>>
>>> It seems that what's really needed is some debug output to be able to
>>> find out what it did.
>>
>> +1, i was preparing an env for testing this but in the while i was
>> thinking how can i know what happens... counting the tables in the
>> explain analyze? it's possible but not ideal
>
> I don't actually see much value in making EXPLAIN show which tables
> were removed.

no, i was thinking in something like a DEBUG message indicating that
that rel is being removed... maybe a DEBUG1 message

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:46:55
Message-ID: 14201.1269877615@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> OK, I'll write a patch for that; and a consensus emerges that we
> should also have enable_joinremoval, then I will add that as well. I
> think the only argument for NOT having enable_joinremoval is that you
> can always modify the query to say SELECT * rather than some more
> specific SELECT list,

Uh, no, the argument for not having enable_joinremoval is that it's
useless.

In particular, I categorically deny the argument that putting it in will
reduce user confusion. If anyone is confused because EXPLAIN shows that
some table isn't getting joined to, you think that the fact that
somewhere in the manual is a mention of enable_joinremoval will
un-confuse them? If they knew that switch was there or what it did,
they wouldn't be confused to begin with.

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 15:55:03
Message-ID: 603c8f071003290855x27d922f7kd85df5f922b136c9@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 11:46 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> OK, I'll write a patch for that; and a consensus emerges that we
>> should also have enable_joinremoval, then I will add that as well.  I
>> think the only argument for NOT having enable_joinremoval is that you
>> can always modify the query to say SELECT * rather than some more
>> specific SELECT list,
>
> Uh, no, the argument for not having enable_joinremoval is that it's
> useless.
>
> In particular, I categorically deny the argument that putting it in will
> reduce user confusion.  If anyone is confused because EXPLAIN shows that
> some table isn't getting joined to, you think that the fact that
> somewhere in the manual is a mention of enable_joinremoval will
> un-confuse them?  If they knew that switch was there or what it did,
> they wouldn't be confused to begin with.

Uh, wow, no, it doesn't make any sense from that point of view. What
I think the use case is is seeing how join removal changed the plan.
It could work out that the plan WITH join removal is significantly
different from the plan WITHOUT join removal. Most of the time that
won't be the case - the join will just get snipped out. But suppose
we're joining A to B and then to C and then to D, and that's the
actual join order. Suppose further that C is most cheaply done as a
hash join. It seems just barely possible to think that if the join to
C actually doesn't need to be done at all, then the join to D might be
done via some other method, because with the disappearance of the join
to C the join to D will receive its left input in some kind of sorted
order.

...Robert


From: Greg Smith <greg(at)2ndquadrant(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 16:03:09
Message-ID: 4BB0CF3D.8090701@2ndquadrant.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas wrote:
> The query planner is a great piece of code but it
> is not so transparently simple that it doesn't need debugging or
> instrumentation, and "why did the planner do X" has got to be one of
> our top ten most-frequently asked questions.

Debugging and instrumentation are two slightly different issues. There
is a lot more instrumentation needed in the query optimizer before
people have better odds of understanding what's going on in this part of
the database. Recent features like pg_stat_statements and auto_explain
are just the first round of what people really want here. Now that we
can get the explain data out in usable formats (XML, JSON, YAML) for a
tool to manage them, the thing at the top of my list in this area for
9.1 is to track down the rumored patch that exports information about
the rejected plans considered and get that comitted. That always seems
what I want to look at for answering the question "why this plan instead
of what I was expecting?"

Stepping away from that, from the debugging perspective it seems one way
to answer the question "is this unexpected behavior being caused by the
new join removal code or not?" is to provide a way to toggle it off and
see what changes. Much like enable_seqscan, just because we don't ever
want people to use it in production doesn't necessarily mean it's a bad
idea to expose it.

Also, given that this is a complicated feature, I think it's reasonable
to ask whether allowing it to be turned off is the right thing just from
the pragmatic basis that it provides a, ahem, backup plan in case
there's unexpected difficulty with it in the field.

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.us


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Greg Smith <greg(at)2ndquadrant(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 16:10:16
Message-ID: 603c8f071003290910l310c0e18v6b8986a22c6c4836@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 12:03 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Robert Haas wrote:
>>
>> The query planner is a great piece of code but it
>> is not so transparently simple that it doesn't need debugging or
>> instrumentation, and "why did the planner do X" has got to be one of
>> our top ten most-frequently asked questions.
>
> Debugging and instrumentation are two slightly different issues.

Yeah, you're right. This is debugging, not instrumentation.

> There is a
> lot more instrumentation needed in the query optimizer before people have
> better odds of understanding what's going on in this part of the database.
>  Recent features like pg_stat_statements and auto_explain are just the first
> round of what people really want here.  Now that we can get the explain data
> out in usable formats (XML, JSON, YAML) for a tool to manage them, the thing
> at the top of my list in this area for 9.1 is to track down the rumored
> patch that exports information about the rejected plans considered and get
> that comitted.  That always seems what I want to look at for answering the
> question "why this plan instead of what I was expecting?"

Having looked at that patch, I am skeptical of it, but we can
certainly take a fresh look.

> Stepping away from that, from the debugging perspective it seems one way to
> answer the question "is this unexpected behavior being caused by the new
> join removal code or not?" is to provide a way to toggle it off and see what
> changes.  Much like enable_seqscan, just because we don't ever want people
> to use it in production doesn't necessarily mean it's a bad idea to expose
> it.
>
> Also, given that this is a complicated feature, I think it's reasonable to
> ask whether allowing it to be turned off is the right thing just from the
> pragmatic basis that it provides a, ahem, backup plan in case there's
> unexpected difficulty with it in the field.

Yep.

...Robert


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Greg Smith <greg(at)2ndquadrant(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 16:17:54
Message-ID: 14825.1269879474@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Mon, Mar 29, 2010 at 12:03 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>> at the top of my list in this area for 9.1 is to track down the rumored
>> patch that exports information about the rejected plans considered and get
>> that comitted. That always seems what I want to look at for answering the
>> question "why this plan instead of what I was expecting?"

> Having looked at that patch, I am skeptical of it, but we can
> certainly take a fresh look.

The problem with this line of thought is that it imagines you can look
at worked-out alternative plans. You can't, because the planner doesn't
pursue rejected alternatives that far (and you'd not want to wait long
enough for it to do so...)

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Greg Smith <greg(at)2ndquadrant(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 16:29:11
Message-ID: 603c8f071003290929x3d36be21t848d2f7244c0cf5b@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 12:17 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Mon, Mar 29, 2010 at 12:03 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
>>> at the top of my list in this area for 9.1 is to track down the rumored
>>> patch that exports information about the rejected plans considered and get
>>> that comitted.  That always seems what I want to look at for answering the
>>> question "why this plan instead of what I was expecting?"
>
>> Having looked at that patch, I am skeptical of it, but we can
>> certainly take a fresh look.
>
> The problem with this line of thought is that it imagines you can look
> at worked-out alternative plans.  You can't, because the planner doesn't
> pursue rejected alternatives that far (and you'd not want to wait long
> enough for it to do so...)

Right. And that's not what the patch did. But a detailed discussion
of this topic should be (a) conducted on a separate thread and (b)
occur after we've all refamiliarized ourselves with it.

...Robert


From: Greg Smith <greg(at)2ndquadrant(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 16:33:00
Message-ID: 4BB0D63C.90408@2ndquadrant.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Tom Lane wrote:
> The problem with this line of thought is that it imagines you can look
> at worked-out alternative plans. You can't, because the planner doesn't
> pursue rejected alternatives that far (and you'd not want to wait long
> enough for it to do so...)
>

Not on any production system, sure. I know plenty of people who would
gladly let a rejected plan enumerator run for *a day* on their
development box if it let them figure out exactly why the costing on the
plan they expected ended up higher than the plan they actually get.
While I know you don't run into this, regular people can easily spend a
week on one such problem without gaining even that much insight, given
the current level of instrumentation and diagnostic tools available.
"Read the source" and "ask Tom" are both effective ways to resolve that
but have their limits. (Not because of you, of course--my bigger
problem are people who just can't share their plans with the lists for
privacy or security reasons)

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.us


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 17:23:13
Message-ID: 1269883393.3684.3941.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 11:27 -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > And for the record, I believe I find it rather amusing that you're
> > asking me if I "have the faintest idea how many there would be".
>
> Actually that was directed more at Simon.

I think you should drop the personal comments. They have no place here.

--
Simon Riggs www.2ndQuadrant.com


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 17:26:59
Message-ID: 1269883619.3684.3954.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 09:37 -0400, Robert Haas wrote:

> In many
> cases it's not horribly difficult to work around because you can do
> SELECT * FROM ... instead of your original select list, but there
> might be some cases with multiple levels of views where it isn't that
> easy. I think it would be good to add this back.

People have asked me how they would know whether join removal has
worked. If there is a workaround that works for most cases, I think that
might be all we need. Do you have an example where that doesn't work, so
we can judge how common that would be?

--
Simon Riggs www.2ndQuadrant.com


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 18:17:55
Message-ID: 1269886675.3684.4064.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 11:46 -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > OK, I'll write a patch for that; and a consensus emerges that we
> > should also have enable_joinremoval, then I will add that as well. I
> > think the only argument for NOT having enable_joinremoval is that you
> > can always modify the query to say SELECT * rather than some more
> > specific SELECT list,
>
> Uh, no, the argument for not having enable_joinremoval is that it's
> useless.
>
> In particular, I categorically deny the argument that putting it in will
> reduce user confusion. If anyone is confused because EXPLAIN shows that
> some table isn't getting joined to, you think that the fact that
> somewhere in the manual is a mention of enable_joinremoval will
> un-confuse them? If they knew that switch was there or what it did,
> they wouldn't be confused to begin with.

You're not addressing the original point. I have been asked how would
users know which tables have been removed and whether there is a way of
checking that. That is not a request for a tuning feature, or something
to reduce user confusion. If you don't like "enable_joinremoval" that's
fine but it would be good to answer the original request with an
alternative proposal.

--
Simon Riggs www.2ndQuadrant.com


From: Alex Hunsaker <badalex(at)gmail(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 19:31:23
Message-ID: 34d269d41003291231w539d07f2i3ea0357c9030e03@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> You're not addressing the original point. I have been asked how would
> users know which tables have been removed and whether there is a way of
> checking that.

Uhh... they wont be in the explain output... Seems simple enough.


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Alex Hunsaker <badalex(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 19:41:01
Message-ID: 1269891661.3684.4246.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 13:31 -0600, Alex Hunsaker wrote:
> On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > You're not addressing the original point. I have been asked how would
> > users know which tables have been removed and whether there is a way of
> > checking that.
>
> Uhh... they wont be in the explain output... Seems simple enough.

That is exactly what I replied, though nobody felt that was a great
answer.

--
Simon Riggs www.2ndQuadrant.com


From: Alex Hunsaker <badalex(at)gmail(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 19:54:57
Message-ID: 34d269d41003291254p1e42b3a8n8cab1f1df04252db@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 13:41, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> Uhh... they wont be in the explain output...  Seems simple enough.
>
> That is exactly what I replied, though nobody felt that was a great
> answer.

Maybe I missed it, but why exactly do they care? About the only
reason I can think *i* would care is: If I was running the same SQL
on multiple database products (like mysql) and thought "Sweet now I
can use this new feature to cleanup my sql so it runs better on
product X or version of postgres Y". Is there some other reason other
than it would be _cool_ to know? Or is it FUD that it might be buggy
and so they wish to be able to turn it off?

It comes to mind you can probably do this with an plannerhook
(whatever happened to those hooks to auto create/recommend indexes?)


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Simon Riggs <simon(at)2ndQuadrant(dot)com>
Cc: Alex Hunsaker <badalex(at)gmail(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 19:55:28
Message-ID: 19848.1269892528@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
> On Mon, 2010-03-29 at 13:31 -0600, Alex Hunsaker wrote:
>> On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>> You're not addressing the original point. I have been asked how would
>>> users know which tables have been removed and whether there is a way of
>>> checking that.
>>
>> Uhh... they wont be in the explain output... Seems simple enough.

> That is exactly what I replied, though nobody felt that was a great
> answer.

Who complained about that exactly? It seems like a perfectly
appropriate answer to me.

regards, tom lane


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, Alex Hunsaker <badalex(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 19:57:41
Message-ID: 603c8f071003291257i4964b1cn4531e70a6a3e0a47@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 3:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
>> On Mon, 2010-03-29 at 13:31 -0600, Alex Hunsaker wrote:
>>> On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>>> You're not addressing the original point. I have been asked how would
>>>> users know which tables have been removed and whether there is a way of
>>>> checking that.
>>>
>>> Uhh... they wont be in the explain output...  Seems simple enough.
>
>> That is exactly what I replied, though nobody felt that was a great
>> answer.
>
> Who complained about that exactly?  It seems like a perfectly
> appropriate answer to me.

Jaime Casanova.

...Robert


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Alex Hunsaker <badalex(at)gmail(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 20:11:35
Message-ID: 1269893495.3684.4310.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 15:55 -0400, Tom Lane wrote:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
> > On Mon, 2010-03-29 at 13:31 -0600, Alex Hunsaker wrote:
> >> On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> >>> You're not addressing the original point. I have been asked how would
> >>> users know which tables have been removed and whether there is a way of
> >>> checking that.
> >>
> >> Uhh... they wont be in the explain output... Seems simple enough.
>
> > That is exactly what I replied, though nobody felt that was a great
> > answer.
>
> Who complained about that exactly? It seems like a perfectly
> appropriate answer to me.

I'm relaying feedback from others not on this list. People expect me to
do this. I shouldn't need to name them for us to accept the feedback,
nor should there be doubt that I relay this accurately (why else would I
raise the subject?!?). If it comes from me, I say so.

--
Simon Riggs www.2ndQuadrant.com


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Alex Hunsaker <badalex(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 20:13:16
Message-ID: 1269893596.3684.4316.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, 2010-03-29 at 13:54 -0600, Alex Hunsaker wrote:

> whatever happened to those hooks to auto create/recommend indexes?

You haven't published them yet??

--
Simon Riggs www.2ndQuadrant.com


From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Alex Hunsaker <badalex(at)gmail(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, Robert Haas <robertmhaas(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 20:17:41
Message-ID: 20331.1269893861@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Alex Hunsaker <badalex(at)gmail(dot)com> writes:
> It comes to mind you can probably do this with an plannerhook
> (whatever happened to those hooks to auto create/recommend indexes?)

Uh, we made some hooks to let someone build an *external* module that
would do such a thing. Whether anybody got anywhere with it I dunno.

regards, tom lane


From: Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>, Simon Riggs <simon(at)2ndquadrant(dot)com>, Alex Hunsaker <badalex(at)gmail(dot)com>, pgsql-hackers(at)postgresql(dot)org
Subject: Re: enable_joinremoval
Date: 2010-03-29 21:56:55
Message-ID: 3073cc9b1003291456o316622e4vd0e566efe266292d@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Mar 29, 2010 at 3:57 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Mon, Mar 29, 2010 at 3:55 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
>>> On Mon, 2010-03-29 at 13:31 -0600, Alex Hunsaker wrote:
>>>> On Mon, Mar 29, 2010 at 12:17, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>>>> You're not addressing the original point. I have been asked how would
>>>>> users know which tables have been removed and whether there is a way of
>>>>> checking that.
>>>>
>>>> Uhh... they wont be in the explain output...  Seems simple enough.
>>
>>> That is exactly what I replied, though nobody felt that was a great
>>> answer.
>>
>> Who complained about that exactly?  It seems like a perfectly
>> appropriate answer to me.
>
> Jaime Casanova.
>

i'm just finishing to read an 347 rows EXPLAIN ANALYZE, so yes i
prefer a DEBUG message than to hunt missing rels

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157