Allowing parallel pg_restore from pipe

From: Timothy Garnett <tgarnett(at)panjiva(dot)com>
To: pgsql-hackers(at)postgresql(dot)org
Subject: Allowing parallel pg_restore from pipe
Date: 2013-04-23 23:53:39
Message-ID: CAPcyiQ2MNGr21Jvp1ZJLwdtTUeoWgvGHWby1RTnbLkhnALcTDA@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi All,

Currently the -j option to pg_restore, which allows for parallelization in
the restore, can only be used if the input file is a regular file and not,
for ex., a pipe. However this is a pretty common occurrence for us
(usually in the form of pg_dump | pg_restore to copy an individual database
or some tables thereof from one machine to another). While there's no good
way to parallelize the data load steps when reading from a pipe, the index
and constraint building can still be parallelized and as they are generally
CPU bound on our machines we've found quite a bit of speedup from doing so.

Attached is two diffs off of the REL9_2_4 tag that I've been using. The
first is a simple change that serially loads the data section before
handing off the remainder of the restore to the existing parallelized
restore code (the .ALT. diff). The second which gets more parallelization
but is a bit more of a change uses the existing dependency analysis code to
allow index building etc. to occur in parallel with data loading. The data
loading tasks are still performed serially in the main thread, but non-data
loading tasks are scheduled in parallel as their dependencies are satisfied
(with the caveat that the main thread can only dispatch new tasks between
data loads).

Anyways, the question is if people think this is generally useful. If so I
can clean up the preferred choice a bit and rebase it off of master, etc.

Tim

Attachment Content-Type Size
0003-patch-pg_restore-to-allow-parallel-restore-when-the.ALT.patch application/octet-stream 5.5 KB
0003-patch-pg_restore-to-allow-parallel-restore-when-the-.patch application/octet-stream 13.9 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Jeff Davis 2013-04-24 00:10:44 Re: Enabling Checksums
Previous Message Alvaro Herrera 2013-04-23 21:56:34 Re: 9.3 Beta1 status report