Re: Using pg_rewind for differential backup

Can we tweak pg_rewind to take differential backups in PostgreSQL?

I was wondering can we hack the pg_rewind code to print the details of the
file which have been modified compared to a target server.
The list output can then be used for taking differential backups.

Or perhaps we can add an option/switch in pg_rewind --action

--action=print ---> would print the files which have changed
--action=sync ---> would sync them
--action=copy ---> with this option I can specify an additional optino
--target-dir where I can copy the files which have changed blocks

pg_rewind-
https://github.com/vmware/pg_rewind

Best Regards,

*Sameer Kumar | Database Consultant*

*ASHNIK PTE. LTD.*

101 Cecil Street, #11-11 Tong Eng Building, Singapore 069533

M: *+65 8110 0350 <%2B65%208110%200350>* T: +65 6438 3504 | www.ashnik.com

*[image: icons]*

[image: Email patch] <http://www.ashnik.com/>

This email may contain confidential, privileged or copyright material and
is solely for the use of the intended recipient(s).

On Thu, Nov 27, 2014 at 9:39 PM, Sameer Kumar <sameer(dot)kumar(at)ashnik(dot)com>
wrote:

> Can we tweak pg_rewind to take differential backups in PostgreSQL?
> I was wondering can we hack the pg_rewind code to print the details of the
> file which have been modified compared to a target server. The list output
> can then be used for taking differential backups.
>
> Or perhaps we can add an option/switch in pg_rewind --action
>
> --action=print ---> would print the files which have changed
> --action=sync ---> would sync them
> --action=copy ---> with this option I can specify an additional optino
> --target-dir where I can copy the files which have changed
>

This discussion is not really adapted on hackers as pg_rewind is not
included in Postgres core code. Please let's discuss your proposal there.
Btw, pg_rewind is not aimed to be used as a tool for a backup facility. You
may find easier to use existing backup solutions instead, or help out with
an in-core solution. There was actually not so long ago a patch to
implement in-core differential backups, with a part called profiles able to
fetch back the list of files modified since a given LSN point (only an
improvement of the core feature though as that's not mandatory to make the
machinery work). The patch has not been added to any of the recent commit
fests though.
Regards,
--
Michael

On 11/28/2014 09:30 AM, Michael Paquier wrote:
> On Thu, Nov 27, 2014 at 9:39 PM, Sameer Kumar <sameer(dot)kumar(at)ashnik(dot)com>
> wrote:
>
>> Can we tweak pg_rewind to take differential backups in PostgreSQL?
>> I was wondering can we hack the pg_rewind code to print the details of the
>> file which have been modified compared to a target server. The list output
>> can then be used for taking differential backups.
>>
>> Or perhaps we can add an option/switch in pg_rewind --action
>>
>> --action=print ---> would print the files which have changed
>> --action=sync ---> would sync them
>> --action=copy ---> with this option I can specify an additional optino
>> --target-dir where I can copy the files which have changed
>>
>
> This discussion is not really adapted on hackers as pg_rewind is not
> included in Postgres core code. Please let's discuss your proposal there.
> Btw, pg_rewind is not aimed to be used as a tool for a backup facility. You
> may find easier to use existing backup solutions instead, or help out with
> an in-core solution.

It also would be quite straightforward to write a separate tool to do
just that. Would be better than conflating pg_rewind with this. You
could use pg_rewind as the basis for it - it's under the same license as
PostgreSQL.

- Heikki

On Fri, Nov 28, 2014 at 2:49 AM, Heikki Linnakangas
<hlinnakangas(at)vmware(dot)com> wrote:
> It also would be quite straightforward to write a separate tool to do just
> that. Would be better than conflating pg_rewind with this. You could use
> pg_rewind as the basis for it - it's under the same license as PostgreSQL.

If we had such a tool in core, would that completely solve the
differential backup problem, or would more be needed?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Dec 3, 2014 at 1:32 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Fri, Nov 28, 2014 at 2:49 AM, Heikki Linnakangas
> <hlinnakangas(at)vmware(dot)com> wrote:
> > It also would be quite straightforward to write a separate tool to do
> just
> > that. Would be better than conflating pg_rewind with this. You could use
> > pg_rewind as the basis for it - it's under the same license as
> PostgreSQL.
>
> ​Agree. I would want to work on that.​

> If we had such a tool in core, would that completely solve the
> differential backup problem, or would more be needed?

​In my opinion​ yes.

​
> ​This discussion is not really adapted on hackers as pg_rewind is not
> included in Postgres core code. Please let's discuss your proposal there.​

​Got it​. But given the above, can we keep this discussion thread here?

Best Regards,

*Sameer Kumar | Database Consultant*

*ASHNIK PTE. LTD.*

101 Cecil Street, #11-11 Tong Eng Building, Singapore 069533

M: *+65 8110 0350* T: +65 6438 3504 | www.ashnik.com

*[image: icons]*

[image: Email patch] <http://www.ashnik.com/>

This email may contain confidential, privileged or copyright material and
is solely for the use of the intended recipient(s).