Re: Custom Scan APIs (Re: Custom Plan node)

On Mon, Apr 14, 2014 at 4:43 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
>> [ assorted comments about custom-scan patch, but particularly ]
>
>> * The prune hook makes me feel very uneasy. It seems weirdly specific
>> implementation detail, made stranger by the otherwise lack of data
>> maintenance API calls. Calling that for every dirty page sounds like
>> an issue and my patch rejection indicator is flashing red around that.
>
> Yeah. After a fast review of the custom-scan and cache-scan patches, it
> seems to me that my original fears are largely confirmed: the custom scan
> patch is not going to be sufficient to allow development of any truly new
> plan type. Yeah, you can plug in some new execution node types, but
> actually doing anything interesting is going to require patching other
> parts of the system. Are we going to say to all comers, "sure, we'll put
> a hook call anywhere you like, just ask"? I can't see this as being the
> way to go.

Without prejudice to the rest of what you said, this argument doesn't
hold much water with me. I mean, anything that our extensibility
mechanism doesn't support today will require new hooks, but does that
mean we're never going to add any more hooks? I sure hope not. When
hooks are proposed here, we evaluate on them on their merits and
attempt to judge the likelihood that a hook in a particular place will
be useful, but generally we're not averse to adding them, and as long
as the paths aren't too performance-critical, I don't think we should
be averse to adding them.

We have a great system today for letting people add new data types and
things of that sort, but anything that penetrates more deeply into the
heart of the system pretty much can't be done; this is why various
companies, such as our respective employers, have developed and
maintained forks of the PostgreSQL code base instead of just hooking
in to the existing code. We probably can't solve that problem
completely, but that doesn't mean we should throw in the towel.

And in particular, I think it's pretty normal that a new facility like
custom scans might create additional demand for new hooks. If
something was completely impossible before, and the new facility makes
it almost-possible, then why shouldn't someone ask for a hook there?
A prune hook probably has no business in the custom scan patch proper,
but whether it's a good idea or a bad one should be decided on the
merits.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company