Re: bg worker: patch 1 of 6 - permanent process

On Wed, Sep 15, 2010 at 2:28 PM, Markus Wanner <markus(at)bluegap(dot)ch> wrote:
>> I guess the real issue here is whether it's possible to, and whether
>> you're interested in, extracting a committable subset of this work,
>> and if so what that subset should look like.
>
> Well, as it doesn't currently provide any real benefit for autovacuum, it
> depends on how much hackers like to prepare for something like Postgres-R or
> parallel querying.

I think that the bar for committing to another in-core replication
solution right now is probably fairly high. I am pretty doubtful that
our current architecture is going to get us to the full feature set
we'd eventually like to have - multi-master, partial replication, etc.
But we're not ever going to have ten replication solutions in core,
so we need to think pretty carefully about what we accept. That
conversation probably needs to start from the other end - is the
overall architecture correct for us? - before we get down to specific
patches. On the other hand, I'm very interested in laying the
groundwork for parallel query, and I think there are probably a number
of bits of architecture both from this project and Postgres-XC, that
could be valuable contributions to PostgreSQL; however, in neither
case do I expect them to be accepted without significant modification.

>> There's sort of a
>> chicken-and-egg problem with large patches; if you present them as one
>> giant monolithic patch, they're too large to review.  But if you break
>> them down into smaller patches, it doesn't really fix the problem
>> unless the pieces have independent value.
>
> I don't quite get what you are trying to say here. I splited the bgworker
> projet from Postgres-R into 6 separate patches. Are you saying that's too
> few or too much?

I'm saying it's hard to think about committing any of them because
they aren't really independent of each other or of other parts of
Postgres-R.

I feel like there is an antagonistic thread to this conversation, and
some others that we've had. I hope I'm misreading that, because it's
not my intent to piss you off. I'm just offering my honest feedback.
Your mileage may vary; others may feel differently; none of it is
personal.

>>> Also note that it would re-introduce some of the costs we try to avoid
>>> with
>>> keeping the connected bgworker around.
>>
>> How?
>
> I'm talking about the cost of connecting to a database (and disconnecting),
> most probably flushing caches, and very probably some kind of re-registering
> with the coordinator. Most of what a normal backend does at startup. About
> the only thing you'd save here is the fork() and very basic process setup. I
> really doubt that's worth the effort.
>
> Having some more idle processes around doesn't hurt that much and solves the
> problem, I think.

OK, I think I understand what you're trying to say now. I guess I
feel like the ideal architecture for any sort of solution that needs a
pool of workers would be to keep around the workers that most recently
proved to be useful. Upon needing a new worker, you look for one
that's available and already bound to the correct database. If you
find one, you assign him to the new task. If not, you find the one
that's been idle longest and either (a) kill him off and start a new
one that is bound to the correct database or, even better, (b) tell
him to flush his caches and rebind to the correct database.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company