Re: pg_basebackup failed to back up large file

On Tue, Jun 3, 2014 at 5:12 PM, Andres Freund <andres(at)2ndquadrant(dot)com>
wrote:

> On 2014-06-03 11:04:58 -0400, Tom Lane wrote:
> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > > On Tue, Jun 3, 2014 at 4:40 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > >> There's a far bigger problem there, which is if we're afraid that
> > >> current_len_left might exceed 4GB then what is it exactly that
> guarantees
> > >> it'll fit in an 11-digit field?
> >
> > > Well, we will only write 11 digits in there, that's when we read it.
> But
> > > print_val() on the server side should probably have an overflow check
> > > there, which it doesn't. It's going to write some strange values int
> here
> > > if it overflows..
> >
> > My point is that having backups crash on an overflow doesn't really seem
> > acceptable. IMO we need to reconsider the basebackup protocol and make
> > sure we don't *need* to put values over 4GB into this field. Where's the
> > requirement coming from anyway --- surely all files in PGDATA ought to be
> > 1GB max?
>
> Fujii's example was logfiles in pg_log. But we allow to change the
> segment size via a configure flag, so we should support that or remove
> the ability to change the segment size...
>
>
With Fujii's fix, the new limit is 8GB, per the tar standard.

To get around that we could use the "star extension" or whatever it's
actually called (implemented by both gnu and bsd tar) , to maintain
compatibility.

Of course, a quick fix would be to just reject files >8GB - that may be
what we have to do backpatchable.

IIRC we discussed this at the original tmie and said it would not be needed
because such files shouldnt' be there - clearly we forgot to consider
logfiles..

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/