[v9.2] Start new timeline for PITR

Folks,

The nice people at VMware, where I work, have come up with a small
patch to allow PITR to create a new timeline. This is useful in cases
where you're using filesystem snapshots of $PGDATA which may be old.

PFA a patch implementing and documenting same :)

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On 6/9/11 4:51 PM, David Fetter wrote:
> Folks,
>
> The nice people at VMware, where I work, have come up with a small
> patch to allow PITR to create a new timeline. This is useful in cases
> where you're using filesystem snapshots of $PGDATA which may be old.
>
> PFA a patch implementing and documenting same :)

Can you explain here in email what the specific goals and expected
behavior of the option are?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

David Fetter <david(at)fetter(dot)org> writes:
> The nice people at VMware, where I work, have come up with a small
> patch to allow PITR to create a new timeline. This is useful in cases
> where you're using filesystem snapshots of $PGDATA which may be old.

Huh? We already start a new timeline when doing a non-crash-recovery
replay scenario.

The code looks pretty confused too, which makes it difficult to
reverse-engineer what your point is.

regards, tom lane

On Thu, Jun 9, 2011 at 8:59 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> David Fetter <david(at)fetter(dot)org> writes:
>> The nice people at VMware, where I work, have come up with a small
>> patch to allow PITR to create a new timeline.  This is useful in cases
>> where you're using filesystem snapshots of $PGDATA which may be old.
>
> Huh?  We already start a new timeline when doing a non-crash-recovery
> replay scenario.
>
> The code looks pretty confused too, which makes it difficult to
> reverse-engineer what your point is.

I am guessing that they are taking a filesystem snapshot, and then
using that to fire up PG. So to PG it looks like a crash recovery,
but they want a new timeline anyway.

<waves hands>

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Jun 10, 2011 at 01:20:25AM -0400, Robert Haas wrote:
> On Thu, Jun 9, 2011 at 8:59 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > David Fetter <david(at)fetter(dot)org> writes:
> >> The nice people at VMware, where I work, have come up with a small
> >> patch to allow PITR to create a new timeline.  This is useful in cases
> >> where you're using filesystem snapshots of $PGDATA which may be old.
> >
> > Huh?  We already start a new timeline when doing a non-crash-recovery
> > replay scenario.
> >
> > The code looks pretty confused too, which makes it difficult to
> > reverse-engineer what your point is.
>
> I am guessing that they are taking a filesystem snapshot, and then
> using that to fire up PG. So to PG it looks like a crash recovery,
> but they want a new timeline anyway.
>
> <waves hands>

That's pretty much it. More detail:

Let's imagine we're taking filesystem snapshots each day by whatever
means. We're also archiving xlogs, but only have space for 48 hours'
worth. Now we want to recover to 3 days ago, but there are no WALs
from that time, so we do a crash recovery from the filesystem
snapshot. Doing continuous archiving from this conflicts with the
existing WALs, which we solve by creating a new timeline.

This also allows subsequent PITR to other times on the original
timeline.

Josh B pointed out that since this option to true conflicts with
another option, having both should prevent recovery from even
starting, and I'll work up a patch for this tonight or at latest
tomorrow.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David,

> Let's imagine we're taking filesystem snapshots each day by whatever
> means. We're also archiving xlogs, but only have space for 48 hours'
> worth. Now we want to recover to 3 days ago, but there are no WALs
> from that time, so we do a crash recovery from the filesystem
> snapshot. Doing continuous archiving from this conflicts with the
> existing WALs, which we solve by creating a new timeline.

How is this different from just changing the recovery_command?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

David,

> Let's imagine we're taking filesystem snapshots each day by whatever
> means. We're also archiving xlogs, but only have space for 48 hours'
> worth. Now we want to recover to 3 days ago, but there are no WALs
> from that time, so we do a crash recovery from the filesystem
> snapshot. Doing continuous archiving from this conflicts with the
> existing WALs, which we solve by creating a new timeline.

How is this different from just changing the recovery_command?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> Let's imagine we're taking filesystem snapshots each day by whatever
>> means.  We're also archiving xlogs, but only have space for 48 hours'
>> worth.  Now we want to recover to 3 days ago, but there are no WALs
>> from that time, so we do a crash recovery from the filesystem
>> snapshot.  Doing continuous archiving from this conflicts with the
>> existing WALs, which we solve by creating a new timeline.
>
> How is this different from just changing the recovery_command?

*scratches head*

How is it the same?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 10.06.2011 22:34, Robert Haas wrote:
> On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus<josh(at)agliodbs(dot)com> wrote:
>>> Let's imagine we're taking filesystem snapshots each day by whatever
>>> means. We're also archiving xlogs, but only have space for 48 hours'
>>> worth. Now we want to recover to 3 days ago, but there are no WALs
>>> from that time, so we do a crash recovery from the filesystem
>>> snapshot. Doing continuous archiving from this conflicts with the
>>> existing WALs, which we solve by creating a new timeline.
>>
>> How is this different from just changing the recovery_command?
>
> *scratches head*
>
> How is it the same?

Creating a dummy recovery.conf with bogus recovery_command would do the
trick.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On 6/10/11 12:34 PM, Robert Haas wrote:
> On Fri, Jun 10, 2011 at 2:53 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>>> Let's imagine we're taking filesystem snapshots each day by whatever
>>> means. We're also archiving xlogs, but only have space for 48 hours'
>>> worth. Now we want to recover to 3 days ago, but there are no WALs
>>> from that time, so we do a crash recovery from the filesystem
>>> snapshot. Doing continuous archiving from this conflicts with the
>>> existing WALs, which we solve by creating a new timeline.
>>
>> How is this different from just changing the recovery_command?
>
> *scratches head*
>
> How is it the same?

Well, presumably I can just change recovery_command to recover from an
empty directory. Then the PITR copy will just come up as soon as it
finishes processing local snapshot WAL, and it'll start its own
timeline. How is this different from what the patch does?

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Fri, Jun 10, 2011 at 11:30 AM, David Fetter <david(at)fetter(dot)org> wrote:
>
> This also allows subsequent PITR to other times on the original
> timeline.
>
> Josh B pointed out that since this option to true conflicts with
> another option, having both should prevent recovery from even
> starting, and I'll work up a patch for this tonight or at latest
> tomorrow.
>

Hi,

Are you still working on this? should we expect a new patch?

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación

On Fri, Jun 17, 2011 at 09:57:13AM -0500, Jaime Casanova wrote:
> On Fri, Jun 10, 2011 at 11:30 AM, David Fetter <david(at)fetter(dot)org> wrote:
> >
> > This also allows subsequent PITR to other times on the original
> > timeline.
> >
> > Josh B pointed out that since this option to true conflicts with
> > another option, having both should prevent recovery from even
> > starting, and I'll work up a patch for this tonight or at latest
> > tomorrow.
>
> Hi,
>
> Are you still working on this? should we expect a new patch?

Yes, sorry about that. I let work get on top of me. Will try for a
new patch this evening.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Fri, Jun 17, 2011 at 1:54 PM, David Fetter <david(at)fetter(dot)org> wrote:
> On Fri, Jun 17, 2011 at 09:57:13AM -0500, Jaime Casanova wrote:
>>
>> Are you still working on this? should we expect a new patch?
>
> Yes, sorry about that.  I let work get on top of me.  Will try for a
> new patch this evening.
>

ok... i will wait it to review... just in advance, i really don't like
this name "create_new_timeline"... it will drive confusion

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte 24x7 y capacitación