Re: Streaming base backups

On 01/05/2011 02:54 PM, Magnus Hagander wrote:
[..]
> Some remaining thoughts and must-dos:
>
> * Compression: Do we want to be able to compress the backups server-side? Or
> defer that to whenever we get compression in libpq? (you can still tunnel it
> through for example SSH to get compression if you want to) My thinking is
> defer it.
> * Compression: We could still implement compression of the tar files in
> pg_streamrecv (probably easier, possibly more useful?)

hmm compression would be nice but I don't think it is required for this
initial implementation.

> * Windows support (need to implement readlink)
> * Tar code is copied from pg_dump and modified. Should we try to factor it out
> into port/? There are changes in the middle of it so it can't be done with
> the current calling points, it would need a refactor. I think it's not worth
> it, given how simple it is.
>
> Improvements I want to add, but that aren't required for basic operation:
>
> * Stefan mentiond it might be useful to put some
> posix_fadvise(POSIX_FADV_DONTNEED)
> in the process that streams all the files out. Seems useful, as long as that
> doesn't kick them out of the cache *completely*, for other backends as well.
> Do we know if that is the case?

well my main concern is that a basebackup done that way might blew up
the buffercache of the OS causing temporary performance issues.
This might be more serious with an in-core solution than with what
people use now because a number of backup software and tools (like some
of the commercial backup solutions) employ various tricks to avoid that.
One interesting tidbit i found was:

http://insights.oetiker.ch/linux/fadvise/

which is very Linux specific but interesting nevertheless...

Stefan