Re: Named restore points

Lists: pgsql-hackers
From: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Named restore points
Date: 2011-01-14 20:41:39
Message-ID: AANLkTinq0kKGv3oR7jxfZ2rxdfBOJyzafPam2=j11AdU@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Hi,

Here is a patch that implements "named restore points".

It allows DBAs to specify an exact point to which they can recover
but that point will have a name, so they have a better control of when
they want to stop recovery (ie: DBA's won't depend of remember
specific times, dates and such).

This adds a new function: pg_create_restore_point(text) (i'm not
wedded with the name so if someone wants to suggest something better,
that's fine with me), a new xlog record and a new recovery_target
parameter in recovery.conf

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL

Attachment Content-Type Size
named_restore_points.patch text/x-patch 19.8 KB

From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-14 22:18:45
Message-ID: 12810.1295043525@sss.pgh.pa.us
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Jaime Casanova <jaime(at)2ndquadrant(dot)com> writes:
> Here is a patch that implements "named restore points".

> It allows DBAs to specify an exact point to which they can recover
> but that point will have a name, so they have a better control of when
> they want to stop recovery (ie: DBA's won't depend of remember
> specific times, dates and such).

> This adds a new function: pg_create_restore_point(text) (i'm not
> wedded with the name so if someone wants to suggest something better,
> that's fine with me), a new xlog record and a new recovery_target
> parameter in recovery.conf

This seems like it's a lot of mechanism for an awfully small use-case.
How often are people actually going to have the foresight to know that
"right now" is when they would want to restore to later? And is it
really any easier to use a label for that than a timestamp? You're
still going to need to keep track of which label means what.

regards, tom lane


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-14 22:42:57
Message-ID: 4D30D171.3040009@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Em 14-01-2011 17:41, Jaime Casanova escreveu:
> Here is a patch that implements "named restore points".
>
Nice feature. I only read the provided documentation and it seems inconsistent
to allow name, time, and xid at recovery_target_name because (i) someone could
name the recovery point as '1234567' (xid) or '2011-01-14' (I use this format
a lot) and (ii) if the suffix name is *_name* it shouldn't allow xid and time.
IMHO, recovery_target_name should allow only names.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-14 22:50:20
Message-ID: AANLkTimVVDEKB2CwywXUmD8b0=4nA5bALskb-dT4NEhE@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jan 14, 2011 at 5:42 PM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> Em 14-01-2011 17:41, Jaime Casanova escreveu:
>>
>> Here is a patch that implements "named restore points".
>>
> Nice feature. I only read the provided documentation and it seems
> inconsistent to allow name, time, and xid at recovery_target_name

it only allow names, but those names could be anything

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-14 23:44:48
Message-ID: AANLkTikkt2rQ_6v3zyZT3EXG3aukEzbhTtn6N4hdyFce@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jan 14, 2011 at 3:41 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
> Here is a patch that implements "named restore points".
>
> It allows DBAs to specify  an exact point to which they can recover
> but that point will have a name, so they have a better control of when
> they want to stop recovery (ie: DBA's won't depend of remember
> specific times, dates and such).
>
> This adds a new function: pg_create_restore_point(text) (i'm not
> wedded with the name so if someone wants to suggest something better,
> that's fine with me), a new xlog record and a new recovery_target
> parameter in recovery.conf

Neat.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-15 01:09:32
Message-ID: 1295053772.18426.5.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, 2011-01-14 at 17:18 -0500, Tom Lane wrote:
> Jaime Casanova <jaime(at)2ndquadrant(dot)com> writes:
> > Here is a patch that implements "named restore points".
>
> > It allows DBAs to specify an exact point to which they can recover
> > but that point will have a name, so they have a better control of when
> > they want to stop recovery (ie: DBA's won't depend of remember
> > specific times, dates and such).
>
> > This adds a new function: pg_create_restore_point(text) (i'm not
> > wedded with the name so if someone wants to suggest something better,
> > that's fine with me), a new xlog record and a new recovery_target
> > parameter in recovery.conf
>
> This seems like it's a lot of mechanism for an awfully small use-case.
> How often are people actually going to have the foresight to know that
> "right now" is when they would want to restore to later? And is it
> really any easier to use a label for that than a timestamp? You're
> still going to need to keep track of which label means what.

I think its the other way around. In order to know what time to restore
to, you have to keep an external list of times when interesting things
happened. This gives you a way of putting that metadata into the log
stream so everything you need is in one place.

You can put a restore point in before or after any major activity, so
you can restore the database if that fails.

e.g. 'daily backup 2001/1/11', 'reference data update 2011/2/5',
'pg_upgrade', etc..

--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-15 01:33:11
Message-ID: 4D30F957.8070809@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Em 14-01-2011 19:50, Jaime Casanova escreveu:
> On Fri, Jan 14, 2011 at 5:42 PM, Euler Taveira de Oliveira
> <euler(at)timbira(dot)com> wrote:
>> Em 14-01-2011 17:41, Jaime Casanova escreveu:
>>>
>>> Here is a patch that implements "named restore points".
>>>
>> Nice feature. I only read the provided documentation and it seems
>> inconsistent to allow name, time, and xid at recovery_target_name
>
> it only allow names, but those names could be anything
>
OK. I will review your patch at the beginning of the week.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-15 01:40:27
Message-ID: AANLkTikrDfdXutkdf2eTZ8NidRsj08_diP3Qn99-2F2e@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jan 14, 2011 at 8:33 PM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
>
> OK. I will review your patch at the beginning of the week.
>

thanks

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-01-30 19:47:57
Message-ID: AANLkTi=z1auPvvMbhtcN8m8GEoiX_7RQesZd7EHTXiMG@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Jan 14, 2011 at 8:33 PM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> OK. I will review your patch at the beginning of the week.

Euler, are you still planning to review this? We're running out of time.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-01 15:02:52
Message-ID: 4D48209C.7050109@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Em 14-01-2011 17:41, Jaime Casanova escreveu:
> Here is a patch that implements "named restore points".
>
Sorry, I was swamped with work. :(

Your patch no longer applied so I rebased it and slightly modified it. Review
is below...

+ The default is to recover to the end of the WAL log.
+ The precise stopping point is also influenced by
+ <xref linkend="recovery-target-inclusive">.
+ </para>

This isn't valid. recovery_target_name are not influenced by
recovery_target_inclusive. Sentence removed.

+ static char recoveryStopNamedRestorePoint[MAXFNAMELEN];

Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking
at code...] It seems to be used for backup label too so it is not so
inappropriate.

+ typedef struct xl_named_restore_points
+ {
+ TimestampTz xtime;
+ char name[MAXFNAMELEN];
+ } xl_named_restore_points;
+

I prefixed those struct members so it won't get confused elsewhere.

+ else if (recoveryTarget == RECOVERY_TARGET_NAME)
+ snprintf(buffer, sizeof(buffer),
+ "%s%u\t%s\t%s named restore point %s\n",
+ (srcfd < 0) ? "" : "\n",
+ parentTLI,
+ xlogfname,
+ recoveryStopAfter ? "after" : "before",
+ recoveryStopNamedRestorePoint);

It doesn't matter if it is after or before the restore point. After/Before
only make sense when we're dealing with transaction or time. Removed.

else if (strcmp(item->name, "recovery_target_xid") == 0)
{
+ /*
+ * if recovery_target_name specified, then this overrides
+ * recovery_target_xid
+ */
+ if (recoveryTarget == RECOVERY_TARGET_NAME)
+ continue;
+

IMHO the right recovery precedence is xid -> name -> time. If you're
specifying xid that's because you know what you are doing. Name takes
precedence over time because it is easier to remember a name than a time. I
implemented this order in the updated patch.

+ recoveryTargetName = pstrdup(item->value);

I also added a check for long names.

+ if ((record->xl_rmid == RM_XLOG_ID) && (record_info == XLOG_RESTORE_POINT))
+ couldStop = true;
+
+ if (!couldStop)
+ return false;
+

I reworked this code path because it seems confusing.

+ recordNamedRestorePoint = (xl_named_restore_points *) XLogRecGetData(record);
+ recordXtime = recordNamedRestorePoint->xtime;

Why don't you store the named restore point here too? You will need it a few
lines below.

+ char name[MAXFNAMELEN];
+
+ memcpy(&xlrec, rec, sizeof(xl_named_restore_points));
+ strncpy(name, xlrec.name, MAXFNAMELEN);

Is it really necessary? I removed it.

+ Datum
+ pg_create_restore_point(PG_FUNCTION_ARGS)
+ {

You should have added a check for long restore point names. Added in the
updated patch.

+ ereport(NOTICE,
+ (errmsg("WAL archiving is not enabled; you must ensure that
WAL segments are copied through other means for restore points to be usefull
for you")));
+

Sentence was rewritten as "WAL archiving is not enabled; you must ensure that
WAL segments are copied through other means to recover up to named restore point".

Finally, this is a nice feature iif we have a way to know what named restore
points are available. DBAs need to take note of this list (that is not good)
and the lazy ones will have a hard time to recover the right name (possibly
with a xlog dump tool).

So how could we store this information? Perhaps a file in
$PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
location). Also it must have a way to transmit the restore_label when we add
another restore point. I didn't implement this part (Jaime?) and it seems as
important as the new xlog record type that is in the patch. It seems
complicate but I don't have ideas. Anyone? The restore point names could be
obtained by querying a function (say, pg_restore_point_names or
pg_restore_point_list).

Someone could argue that this feature could be reached if we store label and
WAL location in a file (say restore_label). This mechanism doesn't need a new
WAL record but the downside is that if we lost restore_label we are dead. I'm
not in favor of this approach because it seems too fragile.

I will mark this patch waiting on author because of those open issues.

This patch needs to bump catalog version because of the new function. I'm not
sure if the new record type requires bumping the xlog magic number.

I'm attaching the updated patch and two scripts that I used to play with the
patch.

--
Euler Taveira de Oliveira
http://www.timbira.com/

Attachment Content-Type Size
a.sh application/x-sh 1.9 KB
b.sh application/x-sh 886 bytes
nrp.diff.gz application/x-gzip 5.4 KB

From: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-05 02:15:04
Message-ID: AANLkTimu31NsrYnHoFSfpKdN4GW-_8L1s90dUUgQNjoB@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Tue, Feb 1, 2011 at 10:02 AM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> Em 14-01-2011 17:41, Jaime Casanova escreveu:
>>
>> Here is a patch that implements "named restore points".
>>
> Sorry, I was swamped with work. :(
>
> Your patch no longer applied so I rebased it and slightly modified it.
> Review is below...
>

Hi,

Thanks for the review, i've been without internet connection for 4
days so i haven't seen the review until now...

> +         The default is to recover to the end of the WAL log.
> +         The precise stopping point is also influenced by
> +         <xref linkend="recovery-target-inclusive">.
> +        </para>
>
> This isn't valid. recovery_target_name are not influenced by
> recovery_target_inclusive. Sentence removed.
>

good point! docs are boring so i was in automatic mode ;)

> + static char recoveryStopNamedRestorePoint[MAXFNAMELEN];
>
> Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking
> at code...] It seems to be used for backup label too so it is not so
> inappropriate.
>

right, i used it because it is used for backup label

>
> +       else if (recoveryTarget == RECOVERY_TARGET_NAME)
> +               snprintf(buffer, sizeof(buffer),
> +                                "%s%u\t%s\t%s named restore point %s\n",
> +                                (srcfd < 0) ? "" : "\n",
> +                                parentTLI,
> +                                xlogfname,
> +                                recoveryStopAfter ? "after" : "before",
> +                                recoveryStopNamedRestorePoint);
>
> It doesn't matter if it is after or before the restore point. After/Before
> only make sense when we're dealing with transaction or time. Removed.
>

you're right

>                else if (strcmp(item->name, "recovery_target_xid") == 0)
>                {
> +                       /*
> +                        * if recovery_target_name specified, then this
> overrides
> +                        * recovery_target_xid
> +                        */
> +                       if (recoveryTarget == RECOVERY_TARGET_NAME)
> +                               continue;
> +
>
> IMHO the right recovery precedence is xid -> name -> time. If you're
> specifying xid that's because you know what you are doing. Name takes
> precedence over time because it is easier to remember a name than a time. I
> implemented this order in the updated patch.
>

actually i was expecting to hear opinions about this and i agree with you

> +                       recoveryTargetName = pstrdup(item->value);
>
> I also added a check for long names.
>

ok

> +       if ((record->xl_rmid == RM_XLOG_ID) && (record_info ==
> XLOG_RESTORE_POINT))
> +               couldStop = true;
> +
> +       if (!couldStop)
> +               return false;
> +
>
> I reworked this code path because it seems confusing.
>

it is... it was the result of debugging an stupid error on my side...

> +               recordNamedRestorePoint = (xl_named_restore_points *)
> XLogRecGetData(record);
> +               recordXtime = recordNamedRestorePoint->xtime;
>
> Why don't you store the named restore point here too? You will need it a few
> lines below.
>

don't remember, will see

>
> + Datum
> + pg_create_restore_point(PG_FUNCTION_ARGS)
> + {
>
> You should have added a check for long restore point names. Added in the
> updated patch.
>

ok

> +         ereport(NOTICE,
> +                  (errmsg("WAL archiving is not enabled; you must ensure
> that WAL segments are copied through other means for restore points to be
> usefull for you")));
> +
>
> Sentence was rewritten as "WAL archiving is not enabled; you must ensure
> that WAL segments are copied through other means to recover up to named
> restore point".
>

sounds better, thanks

> Finally, this is a nice feature iif we have a way to know what named restore
> points are available. DBAs need to take note of this list (that is not good)
> and the lazy ones will have a hard time to recover the right name (possibly
> with a xlog dump tool).
>
> So how could we store this information? Perhaps a file in
> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
> location). Also it must have a way to transmit the restore_label when we add
> another restore point. I didn't implement this part (Jaime?) and it seems as
> important as the new xlog record type that is in the patch. It seems
> complicate but I don't have ideas. Anyone? The restore point names could be
> obtained by querying a function (say, pg_restore_point_names or
> pg_restore_point_list).
>

IMHO, probably the best answer is a tool to retrieve that info... the
problem is that a "restore_label" file should be closely attached to
the WAL segment where the named restore point is... and a sql function
won't say anything about named restore points that are in archived WAL
segments...

>
> I will mark this patch waiting on author because of those open issues.
>
> I'm attaching the updated patch and two scripts that I used to play with the
> patch.
>

ok, i will see you're reviewed version later today

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 03:59:03
Message-ID: AANLkTinJ-2fPjEM7jj91RkU5DPnRqwWwK59JZK1eNgsC@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
> ok, i will see you're reviewed version later today

This patch is still marked as "Needs Review" in the CommitFest
application, but I'm thinking perhaps it should be changed to Ready
for Committer? Are there any open issues?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


From: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 07:05:56
Message-ID: AANLkTinp2i1mL1=do1G=PK2pzJ6u2un-w=LkHWQKgXYY@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Mon, Feb 7, 2011 at 10:59 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Feb 4, 2011 at 9:15 PM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
>> ok, i will see you're reviewed version later today
>
> This patch is still marked as "Needs Review" in the CommitFest
> application, but I'm thinking perhaps it should be changed to Ready
> for Committer?  Are there any open issues?
>

only things i can found are:

> + static char recoveryStopNamedRestorePoint[MAXFNAMELEN];
>
> Is MAXFNAMELEN appropriate? AFAICS it is used for file name length. [Looking
> at code...] It seems to be used for backup label too so it is not so
> inappropriate.
>

which is just a question about if MAXFNAMELEN is the right length to use

and

> Finally, this is a nice feature iif we have a way to know what named restore
> points are available. DBAs need to take note of this list (that is not good)
> and the lazy ones will have a hard time to recover the right name (possibly
> with a xlog dump tool).
>
> So how could we store this information? Perhaps a file in
> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
> location). Also it must have a way to transmit the restore_label when we add
> another restore point. I didn't implement this part (Jaime?) and it seems as
> important as the new xlog record type that is in the patch. It seems
> complicate but I don't have ideas. Anyone? The restore point names could be
> obtained by querying a function (say, pg_restore_point_names or
> pg_restore_point_list).
>

i still think this should be a separate tool or a dba written list,
the reason beign that with sql function we were not able track restore
points in archived segments... if you like i can try to build a simple
tool for this but don't think that is a showstopper, even without that
the feature is useful IMHO at least

--
Jaime Casanova         www.2ndQuadrant.com
Professional PostgreSQL: Soporte y capacitación de PostgreSQL


From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 13:05:24
Message-ID: AANLkTinHW53TGn=v5Z8Y-yZnRAr3Qn=tx5ex-VGFr_Nb@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Tue, Feb 8, 2011 at 2:05 AM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
>> Finally, this is a nice feature iif we have a way to know what named restore
>> points are available. DBAs need to take note of this list (that is not good)
>> and the lazy ones will have a hard time to recover the right name (possibly
>> with a xlog dump tool).
>>
>> So how could we store this information? Perhaps a file in
>> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
>> location). Also it must have a way to transmit the restore_label when we add
>> another restore point. I didn't implement this part (Jaime?) and it seems as
>> important as the new xlog record type that is in the patch. It seems
>> complicate but I don't have ideas. Anyone? The restore point names could be
>> obtained by querying a function (say, pg_restore_point_names or
>> pg_restore_point_list).
>>
>
> i still think this should be a separate tool or a dba written list,

I agree. Keeping track of where you've set named restore points is
not going to be a problem with a simple solution. Which restore
points are available is going to depend on which base backup you
restored and what WAL files you stuffed into pg_xlog.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Robert Haas <robertmhaas(at)gmail(dot)com>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, Euler Taveira de Oliveira <euler(at)timbira(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 14:00:27
Message-ID: 1297173627.1770.7778.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Tue, 2011-02-08 at 08:05 -0500, Robert Haas wrote:
> On Tue, Feb 8, 2011 at 2:05 AM, Jaime Casanova <jaime(at)2ndquadrant(dot)com> wrote:
> >> Finally, this is a nice feature iif we have a way to know what named restore
> >> points are available. DBAs need to take note of this list (that is not good)
> >> and the lazy ones will have a hard time to recover the right name (possibly
> >> with a xlog dump tool).
> >>
> >> So how could we store this information? Perhaps a file in
> >> $PGDATA/pg_xlog/restore_label that contains the label (and possibly the WAL
> >> location). Also it must have a way to transmit the restore_label when we add
> >> another restore point. I didn't implement this part (Jaime?) and it seems as
> >> important as the new xlog record type that is in the patch. It seems
> >> complicate but I don't have ideas. Anyone? The restore point names could be
> >> obtained by querying a function (say, pg_restore_point_names or
> >> pg_restore_point_list).
> >>
> >
> > i still think this should be a separate tool or a dba written list,
>
> I agree. Keeping track of where you've set named restore points is
> not going to be a problem with a simple solution. Which restore
> points are available is going to depend on which base backup you
> restored and what WAL files you stuffed into pg_xlog.

Yeah agreed. No need for restore_label

--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Jaime Casanova <jaime(at)2ndquadrant(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 14:05:15
Message-ID: 1297173915.1770.7791.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Fri, 2011-02-04 at 21:15 -0500, Jaime Casanova wrote:
> >
> > + else if (recoveryTarget == RECOVERY_TARGET_NAME)
> > + snprintf(buffer, sizeof(buffer),
> > + "%s%u\t%s\t%s named restore point %
> s\n",
> > + (srcfd < 0) ? "" : "\n",
> > + parentTLI,
> > + xlogfname,
> > + recoveryStopAfter ? "after" :
> "before",
> > + recoveryStopNamedRestorePoint);
> >
> > It doesn't matter if it is after or before the restore point.
> After/Before
> > only make sense when we're dealing with transaction or time.
> Removed.
> >
>
> you're right

Not sure I understand the comment "only make sense when we're dealing
with transaction or time." Why?

At present, I think the ability to stop before/after a named restore
point should be put back.

--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services


From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Simon Riggs <simon(at)2ndQuadrant(dot)com>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 17:07:29
Message-ID: 4D517851.2050608@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Em 08-02-2011 11:05, Simon Riggs escreveu:
> On Fri, 2011-02-04 at 21:15 -0500, Jaime Casanova wrote:
>>>
>>> + else if (recoveryTarget == RECOVERY_TARGET_NAME)
>>> + snprintf(buffer, sizeof(buffer),
>>> + "%s%u\t%s\t%s named restore point %
>> s\n",
>>> + (srcfd< 0) ? "" : "\n",
>>> + parentTLI,
>>> + xlogfname,
>>> + recoveryStopAfter ? "after" :
>> "before",
>>> + recoveryStopNamedRestorePoint);
>>>
>>> It doesn't matter if it is after or before the restore point.
>> After/Before
>>> only make sense when we're dealing with transaction or time.
>> Removed.
>>>
>>
>> you're right
>
> Not sure I understand the comment "only make sense when we're dealing
> with transaction or time." Why?
>
Because named restore point is a noop xlog record; besides, transaction and
time involves xlog records that contain data.

--
Euler Taveira de Oliveira
http://www.timbira.com/


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 17:14:23
Message-ID: 1297185263.1770.8372.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote:

> > Not sure I understand the comment "only make sense when we're dealing
> > with transaction or time." Why?
> >
> Because named restore point is a noop xlog record; besides, transaction and
> time involves xlog records that contain data.

Thank you. How obvious!

--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services


From: Simon Riggs <simon(at)2ndQuadrant(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 19:53:02
Message-ID: 1297194782.1770.8891.camel@ebony
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote:

> Because named restore point is a noop xlog record; besides, transaction and
> time involves xlog records that contain data.

Committed. Thanks for the patch and the review.

I changed the patch to require wal_level > minimal, rather than
archive_mode = on.

--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services


From: Thom Brown <thom(at)linux(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-08 20:35:08
Message-ID: AANLkTi=ZsjfsPJO-Y-fTVSMSfyHf91OocC8h=kx5SY2T@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On 8 February 2011 19:53, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Tue, 2011-02-08 at 14:07 -0300, Euler Taveira de Oliveira wrote:
>
>> Because named restore point is a noop xlog record; besides, transaction and
>> time involves xlog records that contain data.
>
> Committed. Thanks for the patch and the review.
>
> I changed the patch to require wal_level > minimal, rather than
> archive_mode = on.

This could do with a bit more documentation about usage. Below the
Backup Control Functions table
(http://developer.postgresql.org/pgdocs/postgres/functions-admin.html#FUNCTIONS-ADMIN-BACKUP-TABLE),
each function has a paragraph detailing what it does.

Also, I notice you can easily write over a label. The case I'm
thinking of is someone in psql creating a named restore point, then
later on, they go in again, accidentally cursor up and select the
previous statement and create it again. Would this mean that the
previous label is lost, or would it be the case that any subsequent
duplicate labels would have no effect unless the WAL files with the
original label in were consumed? In either case, a note in the docs
about this would be useful.

And I don't see these label creations getting logged either. Could we
output that to the log because at least then users can grep the
directory for labels, and, in most cases, the time they occurred?

--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935


From: Fujii Masao <masao(dot)fujii(at)gmail(dot)com>
To: Simon Riggs <simon(at)2ndquadrant(dot)com>
Cc: Euler Taveira de Oliveira <euler(at)timbira(dot)com>, Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-09 06:02:23
Message-ID: AANLkTimN6xbXGHjy=7T5zcQPxEfrP4mgaEOib6ZJgWDB@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Wed, Feb 9, 2011 at 4:53 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> Committed. Thanks for the patch and the review.

- * We also track the timestamp of the latest applied COMMIT/ABORT record
- * in XLogCtl->recoveryLastXTime, for logging purposes.
+ * We also track the timestamp of the latest applied COMMIT/ABORT/RESTORE POINT
+ * record in XLogCtl->recoveryLastXTime, for logging purposes.

Tracking the timestamp of the restore point record in recoveryLastXTime
messes up pg_last_xact_replay_timestamp which uses recoveryLastXTime.
The timestamp of the restore point is wrongly returned as that of the latest
transaction, by the function.

As far as I read the patch, I don't think that it's necessary to track the
timestamp of the restore point. The attached patch changes the code so
that it doesn't track the timestamp of the restore point.

+ if (strlen(restore_name_str) >= MAXFNAMELEN)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("value too long for restore point")));

I think that logging the maximum length of the name is useful as follows:

ERROR: value too long for restore point (max 63 characters)

So the attached patch also changes the log message that way.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Attachment Content-Type Size
restore_name_timestamp_v1.patch application/octet-stream 3.3 KB

From: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
To: Thom Brown <thom(at)linux(dot)com>
Cc: Simon Riggs <simon(at)2ndquadrant(dot)com>, Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-24 15:28:50
Message-ID: 4D667932.5010606@timbira.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

Em 08-02-2011 17:35, Thom Brown escreveu:
> This could do with a bit more documentation about usage. Below the
> Backup Control Functions table
> (http://developer.postgresql.org/pgdocs/postgres/functions-admin.html#FUNCTIONS-ADMIN-BACKUP-TABLE),
> each function has a paragraph detailing what it does.
>
I forgot to check it.

> Also, I notice you can easily write over a label. The case I'm
> thinking of is someone in psql creating a named restore point, then
> later on, they go in again, accidentally cursor up and select the
> previous statement and create it again. Would this mean that the
> previous label is lost, or would it be the case that any subsequent
> duplicate labels would have no effect unless the WAL files with the
> original label in were consumed? In either case, a note in the docs
> about this would be useful.
>
This is a limitation that I pointed out [1] but people decided to postpone
named restore point management. The first one is used as restore point. I
added it in the attached patch.

> And I don't see these label creations getting logged either. Could we
> output that to the log because at least then users can grep the
> directory for labels, and, in most cases, the time they occurred?
>
Good point. I included location instead of time; time is already supplied by
log file.

The following patch implements the Thom's suggestions.

[1] http://archives.postgresql.org/message-id/4D48209C.7050109@timbira.com

--
Euler Taveira de Oliveira
http://www.timbira.com/

Attachment Content-Type Size
nrp.diff text/x-patch 1.4 KB

From: Robert Haas <robertmhaas(at)gmail(dot)com>
To: Euler Taveira de Oliveira <euler(at)timbira(dot)com>
Cc: Thom Brown <thom(at)linux(dot)com>, Simon Riggs <simon(at)2ndquadrant(dot)com>, Jaime Casanova <jaime(at)2ndquadrant(dot)com>, PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Re: Named restore points
Date: 2011-02-25 00:02:11
Message-ID: AANLkTinxAPD-7TUEHA1Bji=dEd_X_seYftR+imRMzaU=@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Lists: pgsql-hackers

On Thu, Feb 24, 2011 at 10:28 AM, Euler Taveira de Oliveira
<euler(at)timbira(dot)com> wrote:
> The following patch implements the Thom's suggestions.
>
> [1] http://archives.postgresql.org/message-id/4D48209C.7050109@timbira.com

Committed with some additional wordsmithing.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company