Re: get_fn_expr_variadic considered harmful

In bug #9817 there's a complaint that the planner fails to consider
these expressions equivalent:
foo('a'::text, 'b'::text)
foo(variadic array['a'::text, 'b'::text])
when foo() is declared as taking variadic text[].

Such cases worked okay before 9.3, the reason being that the use of
VARIADIC disappeared after parsing, so that the two calls were in fact
identical. However, once we added FuncExpr.funcvariadic, they're not
identical anymore.

I thought for a bit that we could fix this easily by having equal()
disregard funcvariadic; there is precedent for that, since it ignores
other fields that are just for display purposes and have no semantic
impact, such as CoercionForm and location.

Unfortunately, funcvariadic *does* have semantic impact on a few
functions that use get_fn_expr_variadic, such as format(). Since
the planner has no good way to know which ones those are, it cannot
safely ignore funcvariadic while matching expressions.

In short, commit 75b39e790 broke this rather badly, and I don't see
any easy way out.

We could possibly salvage something by redefining funcvariadic as only
being true if VARIADIC was used *and* the function is VARIADIC ANY,
so that it returns to not being different for semantically-equivalent
cases. This would be a bit messy, since it would not un-break the
behavior for any already stored rules or indexes in 9.3 databases.
But I'm not sure there is any good way to make the problem magically
go away in 9.3 databases.

Thoughts?

regards, tom lane

On Tue, Apr 1, 2014 at 12:52 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> In bug #9817 there's a complaint that the planner fails to consider
> these expressions equivalent:
> foo('a'::text, 'b'::text)
> foo(variadic array['a'::text, 'b'::text])
> when foo() is declared as taking variadic text[].

My first reaction to this was "who cares? after all, the user should
just write the expression the same way both times and then they won't
have this problem". But after going and looking at the bug report I
see that the user wrote it the first way consistently, but pg_dump
blithely rewrote it to the second way. I'm disinclined to view that
as a planner problem; it seems to me to be a pg_dump or ruleutils bug.
If those two things don't have the same parse representation, then
pg_dump has no business treating them as equivalent - even if we were
to put enough smarts into the planner to paper over that
non-equivalence.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Tue, Apr 1, 2014 at 12:52 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> In bug #9817 there's a complaint that the planner fails to consider
>> these expressions equivalent:
>> foo('a'::text, 'b'::text)
>> foo(variadic array['a'::text, 'b'::text])
>> when foo() is declared as taking variadic text[].

> My first reaction to this was "who cares? after all, the user should
> just write the expression the same way both times and then they won't
> have this problem". But after going and looking at the bug report I
> see that the user wrote it the first way consistently, but pg_dump
> blithely rewrote it to the second way. I'm disinclined to view that
> as a planner problem; it seems to me to be a pg_dump or ruleutils bug.
> If those two things don't have the same parse representation, then
> pg_dump has no business treating them as equivalent - even if we were
> to put enough smarts into the planner to paper over that
> non-equivalence.

The point is that they *were* equivalent before 9.3, and so ruleutils
was entirely within its rights to not worry about which way it dumped
the expression; indeed, it couldn't, because the information was not
there as to which way the call had been written originally. I do not
think it's appropriate to blame ruleutils for taking advantage of this
equivalence, because more than likely user applications have too.

Or in other words, what I wrote above is a more general statement of the
problem than what was complained of in bug #9817 ... but if we just hack
ruleutils to dump the cases differently, we will fail to fix the more
general problem. So we can still expect future bug reports about that,
because it worked as-expected for years before 9.3.

There's also the point that even if we changed ruleutils' behavior
now, this would not fix existing dump files that have considered the
two forms interchangeable ever since VARIADIC existed. And we
generally try hard to not break existing dump files. To be even
more to the point: what you propose is incapable of fixing the precise
problem stated in the bug report, because it's complaining about a
dump taken from 9.1, and there is *no* way to make 9.1 produce a
dump that only uses VARIADIC if the original call did. It hasn't
got the information. Even using a newer version of pg_dump wouldn't
help that.

regards, tom lane

On 2014-04-01 12:52:54 -0400, Tom Lane wrote:
> We could possibly salvage something by redefining funcvariadic as only
> being true if VARIADIC was used *and* the function is VARIADIC ANY,
> so that it returns to not being different for semantically-equivalent
> cases. This would be a bit messy, since it would not un-break the
> behavior for any already stored rules or indexes in 9.3 databases.
> But I'm not sure there is any good way to make the problem magically
> go away in 9.3 databases.

It's pretty damn ugly, but if we're going for magic in around those
edges, we could just force the new behaviour in readfuncs.c. IIUC all
the neccessary data for it is there.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2014-04-01 12:52:54 -0400, Tom Lane wrote:
>> We could possibly salvage something by redefining funcvariadic as only
>> being true if VARIADIC was used *and* the function is VARIADIC ANY,
>> so that it returns to not being different for semantically-equivalent
>> cases. This would be a bit messy, since it would not un-break the
>> behavior for any already stored rules or indexes in 9.3 databases.
>> But I'm not sure there is any good way to make the problem magically
>> go away in 9.3 databases.

> It's pretty damn ugly, but if we're going for magic in around those
> edges, we could just force the new behaviour in readfuncs.c. IIUC all
> the neccessary data for it is there.

I don't want either readfuncs or equalfuncs going in for catalog lookups,
which is what they'd have to do to fix it at that level (the key point
being they'd have to find out whether the called function is declared as
VARIADIC ANY). Too much risk of unpleasant side effects if we do that.
The parser, on the other hand, has ready access to the function's
parameter list when building a FuncExpr.

regards, tom lane

On Tue, Apr 1, 2014 at 2:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Tue, Apr 1, 2014 at 12:52 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> In bug #9817 there's a complaint that the planner fails to consider
>>> these expressions equivalent:
>>> foo('a'::text, 'b'::text)
>>> foo(variadic array['a'::text, 'b'::text])
>>> when foo() is declared as taking variadic text[].
>
>> My first reaction to this was "who cares? after all, the user should
>> just write the expression the same way both times and then they won't
>> have this problem". But after going and looking at the bug report I
>> see that the user wrote it the first way consistently, but pg_dump
>> blithely rewrote it to the second way. I'm disinclined to view that
>> as a planner problem; it seems to me to be a pg_dump or ruleutils bug.
>> If those two things don't have the same parse representation, then
>> pg_dump has no business treating them as equivalent - even if we were
>> to put enough smarts into the planner to paper over that
>> non-equivalence.
>
> The point is that they *were* equivalent before 9.3, and so ruleutils
> was entirely within its rights to not worry about which way it dumped
> the expression; indeed, it couldn't, because the information was not
> there as to which way the call had been written originally. I do not
> think it's appropriate to blame ruleutils for taking advantage of this
> equivalence, because more than likely user applications have too.

Sure, it was reasonable at the time, certainly. But they're not
equivalent any more. Either they've got to become equivalent again,
or you can't dump one as the other. I'm happy with either one, but
the first rule of correct dumping is that what you dump out must, on
reload, produce an equivalent database.

> Or in other words, what I wrote above is a more general statement of the
> problem than what was complained of in bug #9817 ... but if we just hack
> ruleutils to dump the cases differently, we will fail to fix the more
> general problem. So we can still expect future bug reports about that,
> because it worked as-expected for years before 9.3.
>
> There's also the point that even if we changed ruleutils' behavior
> now, this would not fix existing dump files that have considered the
> two forms interchangeable ever since VARIADIC existed. And we
> generally try hard to not break existing dump files. To be even
> more to the point: what you propose is incapable of fixing the precise
> problem stated in the bug report, because it's complaining about a
> dump taken from 9.1, and there is *no* way to make 9.1 produce a
> dump that only uses VARIADIC if the original call did. It hasn't
> got the information. Even using a newer version of pg_dump wouldn't
> help that.

Well, that argues for the choice of trying to make them equivalent
again, I suppose, but it sounds like there are some nasty edge cases
that won't easily be filed down. I think your idea of redefining
funcvariadic to be true only for VARIADIC ANY is probably a promising
approach to that solution, but as you say it leaves some problems
unsolved.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Tue, Apr 1, 2014 at 2:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> There's also the point that even if we changed ruleutils' behavior
>> now, this would not fix existing dump files that have considered the
>> two forms interchangeable ever since VARIADIC existed. And we
>> generally try hard to not break existing dump files. To be even
>> more to the point: what you propose is incapable of fixing the precise
>> problem stated in the bug report, because it's complaining about a
>> dump taken from 9.1, and there is *no* way to make 9.1 produce a
>> dump that only uses VARIADIC if the original call did. It hasn't
>> got the information. Even using a newer version of pg_dump wouldn't
>> help that.

> Well, that argues for the choice of trying to make them equivalent
> again, I suppose, but it sounds like there are some nasty edge cases
> that won't easily be filed down. I think your idea of redefining
> funcvariadic to be true only for VARIADIC ANY is probably a promising
> approach to that solution, but as you say it leaves some problems
> unsolved.

I think what we'll have to do is tell complainants to recreate any
affected indexes or rules after installing 9.3.5. Given the relatively
small number of complaints, I don't think it's worth working harder,
nor taking risks like inserting catalog lookups into readfuncs.c.

regards, tom lane

I wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Well, that argues for the choice of trying to make them equivalent
>> again, I suppose, but it sounds like there are some nasty edge cases
>> that won't easily be filed down. I think your idea of redefining
>> funcvariadic to be true only for VARIADIC ANY is probably a promising
>> approach to that solution, but as you say it leaves some problems
>> unsolved.

> I think what we'll have to do is tell complainants to recreate any
> affected indexes or rules after installing 9.3.5. Given the relatively
> small number of complaints, I don't think it's worth working harder,
> nor taking risks like inserting catalog lookups into readfuncs.c.

After some thought, it seems to me that the best solution is to redefine
funcvariadic as meaning "the last actual argument is a variadic array",
which means it will always be true for a VARIADIC non-ANY function.
In HEAD, that leads to a nice patch that actually simplifies the logic
in ruleutils.c, as attached. In 9.3, we will not be able to assume that
funcvariadic has that meaning, so we'll have to use the existing
decompilation logic (which basically ignores funcvariadic unless it's a
VARIADIC ANY function). I've not made that version of the patch yet but
it should be pretty straightforward.

regards, tom lane