Re: Parallell Optimizer

On Fri, Jun 7, 2013 at 3:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Fri, Jun 7, 2013 at 2:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> I don't think that bgworkers as currently implemented make this any more
>>> practical than it was before. The communication overhead with a
>>> separate process would swamp any benefit in most cases.
>
>> I'm baffled by your statement that the communication overhead would be
>> too high. What IPC mechanism are you presuming, and why would it be
>> any more expensive in PostgreSQL than in any other database (a number
>> of which do have parallel query execution)?
>
> Well, right at the moment we don't have *any* IPC mechanism that would
> work, so the cost is infinity. But the real issues here are the same
> as we touched on in the recent round of discussions about parallelism:
> you'd have to export snapshots, locks, etc to another process before it
> could start taking over any planning work for you, and once you did have
> all the context shared, there would still be a tremendous amount of
> two-way communication needed, because the parallelizable units of work
> are not very large. There's too much work yet to be done before this is
> remotely practical.

Well, I'm not as pessimistic as all that, but I agree there's a good
deal of work to be done. I don't really see why the units of
parallelizable work can't be large. Indeed, I'd argue that if they're
not, we've missed the boat.

> As for other databases, I suspect that ones that have parallel execution
> are probably doing it with a thread model not a process model.

Yeah, maybe. Maybe I'm a hopeless optimist - though I'm rarely
accused of that - I actually think that our process model is going to
work out fairly well for us here. It's true that there is a bunch of
state that needs to be copied around or shared to make this work. But
it's also true that with a thread model, we'd have to explicitly
arrange NOT to share all the things we DIDN'T want shared. Honestly,
that sounds harder.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company