Re: Parallel Sort

From: Noah Misch <noah(at)leadboat(dot)com>
To: Claudio Freire <klaussfreire(at)gmail(dot)com>
Cc: Kohei KaiGai <kaigai(at)kaigai(dot)gr(dot)jp>, PgHacker <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Parallel Sort
Date: 2013-05-15 18:04:37
Message-ID: 20130515180437.GA234183@tornado.leadboat.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

On Tue, May 14, 2013 at 12:15:24PM -0300, Claudio Freire wrote:
> You know what would be a low-hanging fruit that I've been thinking
> would benefit many of my own queries?
>
> "Parallel" sequential scan nodes. Even if there's no real parallelism
> involved, when a query has to scan the same table at multiple nodes,
> if it's big, it would be worth parallelizing the scans to transform
> them into synchro scans.
>
> I have absolutely no idea how this would work easily without forked
> workers, because the scans might be buried in more complex execution
> trees. But still, it's worth considering, that parallelizing may
> benefit more than core usage.
>
> If execution nodes could be paused at arbitrary points, a "parallel
> scan" node could pause one branch that has consumed the circular
> buffer, letting another branches consume their part, and thus
> "parallelizing" branch execution. But this would be perhaps more
> complex than simply forking.

Execution nodes do pause between every output tuple, at least nominally.
Still, given the architecture of our executor and the planner work to
implement such a thing, I would not classify it as low-hanging fruit. It
would primarily apply to a plan with independent sequential scans of the same
large (relative to total memory) relation. I'm sure that comes up, but it
doesn't strike me as typical.

Thanks,
nm

--
Noah Misch
EnterpriseDB http://www.enterprisedb.com

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andrew Dunstan 2013-05-15 18:05:25 Re: commit fest schedule for 9.4
Previous Message Josh Berkus 2013-05-15 18:00:50 Re: commit fest schedule for 9.4