Re: Re: database restoration problem- data became incorrect caused by incorrect date in the server

Hi all

I am a junior DBA:

Please give an advice on how I should solve the following problem,

We are still using 7.3.4 PostgreSQL but will be upgrading very soon, database is running on CentOS 5 O/S .

Problem: - it's about the wrong data in the system:

Everything was fine until a certain process(for financial period processing) that normally generates many transactions in the database took place when the server lost the date. The date changed from 07-04-2010 to 15-08-2003...hence the process ran until it finished which leaded to the generation of the wrong data.

When we thought of restoring from a successful dump file(backup file) for the 07-04-2010 so that we could restore it to have everything before the date changed, we were not lucky because the only latest successful dump file was on the 03/04/2010.

Question:
How can we removed everything that got processed on the wrong date and have everything restored so that it's in the correct state as at 07/04/2010... I tried to search for a solution but I can't find anything.

Your help would be much appreciated

Thanks
Khangelani

Confidentiality Notice:http://ucs.co.za/conf.html

________________________________
The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes.

To add on to previous comments:

There is only one database in the server and the backup server also has incorrect data because all the wrong data replicated to the backup server.

From: pgsql-admin-owner(at)postgresql(dot)org [mailto:pgsql-admin-owner(at)postgresql(dot)org] On Behalf Of Khangelani Gama
Sent: Friday, April 09, 2010 2:22 PM
To: pgsql-admin(at)postgresql(dot)org
Subject: [ADMIN] database restoration problem- data became incorrect caused by incorrect date in the server

Hi all

I am a junior DBA:

Please give an advice on how I should solve the following problem,

We are still using 7.3.4 PostgreSQL but will be upgrading very soon, database is running on CentOS 5 O/S .

Problem: - it's about the wrong data in the system:

Everything was fine until a certain process(for financial period processing) that normally generates many transactions in the database took place when the server lost the date. The date changed from 07-04-2010 to 15-08-2003...hence the process ran until it finished which leaded to the generation of the wrong data.

When we thought of restoring from a successful dump file(backup file) for the 07-04-2010 so that we could restore it to have everything before the date changed, we were not lucky because the only latest successful dump file was on the 03/04/2010.

Question:
How can we removed everything that got processed on the wrong date and have everything restored so that it's in the correct state as at 07/04/2010... I tried to search for a solution but I can't find anything.

Your help would be much appreciated

Thanks
Khangelani

Confidentiality Notice:http://ucs.co.za/conf.html

________________________________
The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes.

________________________________
The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes.

Khangelani Gama <Khangelani(dot)Gama(at)ucs-software(dot)co(dot)za> wrote:

> We are still using 7.3.4 PostgreSQL

Do you really mean that, or is that a typo?

What sort of backups do you have? (pg_dump, file system copy, etc.)

-Kevin

Hi

I really mean it....the backups we have uses pg_dump command. The automatic backup dumps file happen daily and gets saved in a certain directory

But the issue now is that we don't have any backup file for us to restore the database but exclude every data that got create when the date in server was incorrect.

Thanks

-----Original Message-----
From: pgsql-admin-owner(at)postgresql(dot)org [mailto:pgsql-admin-owner(at)postgresql(dot)org] On Behalf Of Kevin Grittner
Sent: Friday, April 09, 2010 4:14 PM
To: pgsql-admin(at)postgresql(dot)org; Khangelani Gama
Subject: [ADMIN] Re: database restoration problem- data became incorrect caused by incorrect date in the server

Khangelani Gama <Khangelani(dot)Gama(at)ucs-software(dot)co(dot)za> wrote:

> We are still using 7.3.4 PostgreSQL

Do you really mean that, or is that a typo?

What sort of backups do you have? (pg_dump, file system copy, etc.)

-Kevin

--
Sent via pgsql-admin mailing list (pgsql-admin(at)postgresql(dot)org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin

The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes.

Khangelani Gama <Khangelani(dot)Gama(at)ucs-software(dot)co(dot)za> wrote:

> the backups we have uses pg_dump command. The automatic
> backup dumps file happen daily and gets saved in a certain
> directory

No off-site copies or archives to fall back on? If not, my ideas
are:

(1) I hope you made a file copy as soon as you found the problem.
If not, stop the database and make one immediately. If the old
versions of the rows haven't yet been vacuumed away, it might be
possible to fish some or all of the corrupted data out of the
database. This is very tricky work, though, and you might want to
contract with an expert. (I don't do that anymore, but there are
several good companies with people who do.)

(2) Are there any source documents you can fall back on to
reconstruct the data?

(3) Is there any way to "reverse" the particular calculations or
operations which mangled the data?

Once you do get upgraded, you should look into Point In Time
Recovery (PITR) backups. Had you been running this, you could have
replayed the transactions right up to the moment of corruption.
Well, if you kept a base backup from before the corruption and all
the WAL files since that base backup.

-Kevin

Thanks for the ideas

-----Original Message-----
From: Kevin Grittner [mailto:Kevin(dot)Grittner(at)wicourts(dot)gov]
Sent: Friday, April 09, 2010 8:03 PM
To: pgsql-admin(at)postgresql(dot)org; Khangelani Gama
Subject: RE: [ADMIN] Re: database restoration problem- data became incorrect caused by incorrect date in the server

Khangelani Gama <Khangelani(dot)Gama(at)ucs-software(dot)co(dot)za> wrote:

> the backups we have uses pg_dump command. The automatic
> backup dumps file happen daily and gets saved in a certain
> directory

No off-site copies or archives to fall back on? If not, my ideas
are:

(1) I hope you made a file copy as soon as you found the problem.
If not, stop the database and make one immediately. If the old
versions of the rows haven't yet been vacuumed away, it might be
possible to fish some or all of the corrupted data out of the
database. This is very tricky work, though, and you might want to
contract with an expert. (I don't do that anymore, but there are
several good companies with people who do.)

(2) Are there any source documents you can fall back on to
reconstruct the data?

(3) Is there any way to "reverse" the particular calculations or
operations which mangled the data?

Once you do get upgraded, you should look into Point In Time
Recovery (PITR) backups. Had you been running this, you could have
replayed the transactions right up to the moment of corruption.
Well, if you kept a base backup from before the corruption and all
the WAL files since that base backup.

-Kevin

The contents of and attachments to this e-mail are intended for the addressee only, and may contain the confidential information of UCS Group and/or its subsidiaries. Any review, use or dissemination thereof by anyone other than the intended addressee is prohibited. If you are not the intended addressee please notify the writer immediately and destroy the e-mail. UCS Group Limited and its subsidiaries distance themselves from and accept no liability for unauthorised use of their e-mail facilities or e-mails sent other than strictly for business purposes.