Re: pg_basebackup failed to back up large file

On Tue, Jun 3, 2014 at 6:38 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > Yeah, that is a clear advantage of that method. Didn't read up on pax
> > format backwards compatibility, does it have some trick to achieve
> > something similar?
>
> I didn't read the fine print but it sounded like the extended header
> would look like a separate file entry to a non-aware tar implementation,
> which would write it out as a file and then get totally confused when
> the length specified in the overlength file's entry didn't match the
> amount of data following. So it's a nice solution for some properties
> but doesn't fail-soft for file length. Not clear that there's any way
> to achieve that though.
>

Well, there is no way to make it fail completely soft on the client side I
think. But at least we should make sure that our implementation doesn't go
do something really bad if you were to upgrade your server but not the
client. And make sure that we test with at least GNU and BSD tar to see
they don't break things badly either.

Another thought is we could make pg_basebackup simply skip any files that
> exceed RELSEG_SIZE, on the principle that you don't really need/want
> enormous log files to get copied anyhow. We'd still need the pax
> extension if the user had configured large RELSEG_SIZE, but having a
> compatible tar could be documented as a requirement of doing that.
>

I think going all the way to pax is the proper long-term thing to do, at
least if we can confirm it works in the main tar implementations.

For backpatchable that seems more reasonable. It doesn't work today, and we
just need to document that it doesn't, with larger RELSEG_SIZE. And then
fix it properly for the future.

Not sure we want to go change the file format even for 9.4 at this time, it
seems we're quite overdue for that - so I think that's a 9.5 feature.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/