Re: directory archive format for pg_dump

On 12/16/2010 03:52 PM, Tom Lane wrote:
> Andrew Dunstan<andrew(at)dunslane(dot)net> writes:
>> On 12/16/2010 03:13 PM, Robert Haas wrote:
>>> So how bad would it be if we committed this new format without support
>>> for splitting large relations into multiple files, or with some stub
>>> support that never actually gets used, and fixed this later? Because
>>> this is starting to sound like a bigger project than I think we ought
>>> to be requiring for this patch.
>> I don't think we have to have that in the first go at all. Parallel dump
>> could be extremely useful without it. I haven't looked closely, but I
>> assume there will still be an archive version recorded somewhere. When
>> we change the archive format, bump the version number.
> Sure, but it's worth thinking about the feature now. If there are
> format tweaks to be made, it might be less painful to make them now
> instead of later, even if actual support for the feature isn't there.
> (I agree I don't want to try to implement it just yet.)
>
>

Yeah, OK. Well, time is getting short but (hand waving wildly) I think
we could probably get by with just adding a member to the TOC for the
section number of the entry (set it to 0 for non TABLE DATA TOC
entries). The section number could be built into the file name in
directory format. For now that number would always be 1 for TABLE DATA
members.

This has intriguing possibilities for parallel restore of custom format
dumps too. It could be very useful to be able to restore a single table
in parallel, if we had more than one TABLE DATA member per table.

I'm deliberately just addressing infrastructure issues rather than how
we actually generate multiple sections of data for a single table
(especially if we want to do that in parallel).

cheers

andrew