Issues Outstanding for Point In Time Recovery (PITR)

Hello:

I've got the logging system to the point where I can take a shutdown
consistent copy of a system, and play forward through multiple
checkpoints. It seems to handle CREATE TABLE/DROP TABLE/TRUNCATE
properly, and things are moving forward well. Recovery to an arbitrary
point-in-time should be just as easy, but will need some administrative
interface for it.

At this point, some input would be useful on how I should handle things.

The most important questions that need answering are in sections 2 & 5,
since they impact the most other parts of the system. They will also
require good documentation for sysadmins.

Issues Outstanding for Point In Time Recovery (PITR)

$Date: 2002/07/04 14:23:37 $

$Revision: 1.4 $

J.R. Nield

(Enc: ISO 8859-15 Latin-9)

§0 - Introduction

This file is where I'm keeping track of all the issues I run into while
trying to get PITR to work properly. Hopefully it will evolve into a
description of how PITR actually works once it is implemented.

I will also try to add feedback as it comes in.

The big items so-far are:
§1 - Logging Relation file creation, truncation, and removal
This is mostly done. Can do infinte play-forward from
online logs.
§2 - Partial-Write and Bad Block detection
Need input before starting. Migration issues.
§3 - Detecting Shutdown Consistent System Recovery
Mostly done.
§4 - Interactive Play-Forward Recovery for an Entire System
Need input before starting.
§5 - Individual file consistent recovery
Need input. Semi-Major changes required.

§1 - Logging Relation file creation, truncation, and removal

§1.1 - Problem:

Without file creation in the log, we can't replay committed
transactions that create relations.

The current code assumes that any transaction reaching commit has already
ensured it's files exist, and that those files will never be removed. This
is true now, but not for log-replay from an old backup database system.

The current XLOG code silently ignores block-write requests for
non-existent files, and assumes that the transaction generating those
requests must have aborted.

Right now a crash during TRUNCATE TABLE will leave the table in an
inconsistent state (partially truncated). This would not work when doing
replay from before the last checkpoint.

§1.1.1 - CREATE DATABASE is also unlogged

This will cause the same replay problems as above.

§1.2 - Proposal:

a) Augment the SMGR code to log relation file operations, and to handle
redo requests properly. This is simple in the case of create. Drop must be
logged only IN the commit record. For truncate see (b).

The 'struct f_smgr' needs new operations 'smgr_recreate', 'smgr_reunlink',
and 'smgr_retruncate'. smgr_recreate should accept a RelFileNode instead
of a Relation.

Transactions that abort through system failure (ie. unlogged aborts)
will simply continue to leak files.

b) If TRUNCATE TABLE fails, the system must PANIC. Otherwise, the table
may be used in a future command, and a replay-recovered database may
end-up with different data than the original.

WAL must be flushed before truncate as well.

WAL does not need to be flushed before create, if we don't mind
leaking files sometimes.

c) Redo code should treat writes to non-existent files as an error.
Changes affect heap & nbtree AM's. [Check others]

d) rtree [and GiST? WTF is GiST? ] is not logged. A replay recovery of
a database should mark all the rtree indices as corrupt.
[ actually we should do that now, are we? ]

e) CREATE DATABASE must be logged properly, not use system(cp...)

§1.3 - Status:

All logged SMGR operations are now in a START_CRIT_SECTION()/
END_CRIT_SECTION() pair enclosing the XLogInsert() and the underlying fs
operations.

Code has been added to smgr and xact modules to log:
create (no XLogFlush)
truncate (XLogFlush)
pending deletes on commit record
files to delete on abort record

Code added to md.c to support redo ops

Code added to smgr for RMGR redo/desc callbacks

Code added to xact RMGR callbacks for redo/desc

Database will do infinite shutdown consistent system recovery from the
online logs, if you manually munge the control file to set state ==
DB_IN_PRODUCTION instead of DB_SHUTDOWNED.

Still need to do:
Item (c), recovery cleanup in all AM's
Item (d), logging in other index AM's
Item (e), CREATE DATABASE stuff

§2 - Partial-Write and Bad Block detection

§2.1 - Problem:

In order to protect against partial writes without logging pages
twice, we need to detect partial pages in system files and report them
to the system administrator. We also might want to be able to detect
damaged pages from other causes, like memory corruption, OS errors,
etc. or in the case where the disk doesn't report bad blocks, but
returns bad data.

We should also decide what should happen when a file is marked as
containing corrupt pages, and requires log-archive recovery from a
backup.

§2.2 - Proposal:

Add a 1 byte 'pd_flags' field to PageHeaderData, with the following
flag definitions:

PD_BLOCK_CHECKING (1)
PD_BC_METHOD_BIT (1<<1)

PageHasBlockChecking(page) ((page)->pd_flags & PD_BLOCK_CHECKING)
PageBCMethodIsCRC64(page) ((page)->pd_flags & PD_BC_METHOD_BIT)
PageBCMethodIsLSNLast(page) (!PageBCMethodIsCRC64(page))

The last 64 bits of a page are reserved for use by the block checking
code.

[ Is it worth the trouble to allow the last 8 bytes of a
page to contain data when block checking is turned off for a Page?

This proposal does not allow that. ]

If the block checking method is CRC64, then that field will contain
the CRC64 of the block computed at write time.

If the block checking method is LSNLast, then the field contains a
duplicate of the pd_lsn field.

§2.2.1 - Changes to Page handling routines

All the page handling routines need to understand that
pd_special == (pd_special - (specialSize + 8))

Change header comment in bufpage.h to reflect this.

§2.2.2 - When Reading a Page

Block corruption is detected on read in the obvious way with CRC64.

In the case of LSNLast, we check to see if pd_lsn == the lsn in the
last 64 bits of the page. If not, we assume the page is corrupt from
a partial write (although it could be something else).

IMPORTANT ASSUMPTION:
The OS/disk device will never write both the first part and
last part of a block without writing the middle as well.
This might be wrong in some cases, but at least it's fast.

§2.2.4 - GUC Variables

The user should be able to configure what method is used:

block_checking_write_method = [ checksum | torn_page_flag | none ]

Which method should be used for blocks we write?

check_blocks_on_read = [ true | false ]

When true, verify that the blocks we read are not corrupt, using
whatever method is in the block header.

When false, ignore the block checking information.

§2.3 - Status:

Waiting for input from pgsql-hackers.

Questions:

Should we allow the user to have more detailed control over
which parts of a database use block checking?

For example: use 'checksum' on all system catalogs in all databases,
'torn_page_flag' on the non-catalog parts of the production database,
and 'none' on everything else?

§3 - Detecting Shutdown Consistent System Recovery

§3.1 - Problem:

How to notice that we need to do log-replay for a system backup, when the
restored control file points to a shutdown checkpoint record that is
before the most recent checkpoint record in the log, and may point into
an archived file.

§3.2 - Proposal:

At startup, after reading the ControlFile, scan the log directory to
get the list of active log files, and find the lowest logId and
logSeg of the files. Ensure that the files cover a contiguous range
of LSN's.

There are three cases:

1) ControlFile points to the last valid checkpoint (either
checkPoint or prevCheckPoint, but one of them is the greatest
valid checkpoint record in the log stream).

2) ControlFile points to a valid checkpoint record in an active
log file, but there are more valid checkpoint records beyond
it.

3) ControlFile points to a checkpoint record that should be in the
archive logs, and is presumably valid.

Case 1 is what we handle now.

Cases 2 and 3 would result from restoring an entire system from
backup in preparation to do a play-forward recovery.

We need to:

Detect cases 2 and 3.

Alert the administrator and abort startup.
[Question: Is this always the desired behavior? We can
handle case 2 without intervention. ]

Let the administrator start a standalone backend, and
perform a play-forward recovery for the system.

§3.3 - Status:

In progress.

§4 - Interactive Play-Forward Recovery for an Entire System

Play-Forward File Recovery from a backup file must be interactive,
because not all log files that we need are necessarily in the
archive directory. It may be possible that not all the archive files
we need can even fit on disk at one time.

The system needs to be able to prompt the system administrator to feed
it more log files.

TODO: More here

§5 - Individual file consistent recovery

§5.1 - Problem:

If a file detects corruption, and we restore it from backup, how do
we know what archived files we need for recovery?

Should file corruption (partial write, bad disk block, etc.) outside
the system catalog cause us to abort the system, or should we just
take the relation or database off-line?

Given a backup file, how do we determine the point in the log
where we should start recovery for the file? What is the highest LSN
we can use that will fully recover the file?

§5.2 - Proposal:

Put a file header on each file, and update that header to the last
checkpoint LSN at least once every 'file_lsn_time_slack' minutes, or
at least once every dbsize/'file_lsn_log_slack' megabytes of log
written, where dbsize is the estimated size of the database. Have
these values be settable from the config file. These updates would be
distributed throughout the hour, or interspersed between regular
amounts of log generation.

If we have a database backup program or command, it can update the
header on the file before backup to the greatest value it can assure
to be safe.

§5.3 - Status:

Waiting for input from pgsql-hackers.

Questions:

There are alternate methods than using a file header to get a
known-good LSN lower bound for the starting point to recover a backup
file. Is this the best way?

A) The Definitions

This stuff is obtuse, but I need it here to keep track of what I'm
saying. Someday I should use it consistently in the rest of this
document.

"system" or "database system":

A collection of postgres "databases" in one $PGDATA directory,
managed by one postmaster instance at a time (and having one WAL
log, etc.)

All the files composing such a system, as a group.

"up to date" or "now" or "current" or "current LSN":

The most recent durable LSN for the system.

"block consistent copy":

When referring to a file:

A copy of a file, which may be written to during the process of
copying, but where each BLCKSZ size block is copied atomically.

When referring to multiple files (in the same system):

A copy of all the files, such that each is independently a "block
consistent copy"

"file consistent copy":

When referring to a file:

A copy of a file that is not written to between the start and end
of the copy operation.

When referring to multiple files (in the same system):

A copy of all the files, such that each is independently a "file
consistent copy"

"system consistent copy":

When referring to a file:

A copy of a file, where the entire system of which it is a member
is not written to during the copy.

When referring to multiple files (in the same system):

A copy of all the files, where the entire system of which they are
members was not written to between the start and end of the
copying of all the files, as a group.

"shutdown consistent copy":

When referring to a file:

A copy of a file, where the entire system of which it is a member
had been cleanly shutdown before the start of and for the duration
of the copy.

When referring to multiple files (in the same system):

A copy of all the files, where the entire system of which they are
members had been cleanly shutdown before the start of and for the
duration of the copying of all the files, as a group.

"consistent copy":

A block, file, system, or shutdown consistent copy.

"known-good LSN lower bound"
or "LSN lower bound"
or "LSN-LB":

When referring to a group of blocks, a file, or a group of files:

An LSN known to be old enough that no log entries before it are needed
to bring the blocks or files up-to-date.

"known-good LSN greatest lower bound"
or "LSN greatest lower bound"
or "LSN-GLB":

When referring to a group of blocks, a file, or a group of files:

The greatest possible LSN that is a known-good LSN lower bound for
the group.

"backup file":

A consistent copy of a data file used by the system, for which
we have a known-good LSN lower bound.

"optimal backup file":

A backup file, for which we have the known-good LSN greatest lower
bound.

"backup system":

"Play-Forward File Recovery" or "PFFR":

The process of bringing an individual backup file up to date.

--
J. R. Nield
jrnield(at)usol(dot)com