Restoring pg_dump Backup with psql Fails

PostgreSQL 8.2.4 on RedHat ES4 64-bit

Hi all,

Doing some works on a database this morning (changing it to take data from
our new ERP system) so naturally I made a backup of the database first:
pg_dump important_database > ~/backup/important-071229.sql
chmod 440 ~/backup/important-071229.sql

Of course things have screwed up (fortunately no-one is working today) and I
need to restore:
psql
ALTER DATABASE important_database RENAME TO
important_database071229;
CREATE DATABASE important_database;
\q

psql important_database < ~/backup/important-071229.sql

But I get a whole bunch of errors, for example:
<snip>
ERROR: syntax error at or near "all"
LINE 1: all orders over $400.00 - freight free Order 10 of any produ...
^
ERROR: syntax error at or near "If"
LINE 1: If stores spend over $300.00 they will receive a 5% discount...
<snip>
invalid command \.
invalid command \.
invalid command \N
invalid command \N
<snip>

Both psql and pg_dump are the correct versions:
[postgres(at)stirling ~]$ ls -lh `which pg_dump`
lrwxrwxrwx 1 root root 31 Jul 25 11:51 /usr/local/bin/pg_dump ->
/usr/local/pg-8.2.4/bin/pg_dump
[postgres(at)stirling ~]$ ls -lh `which psql`
lrwxrwxrwx 1 root root 28 Jul 25 11:51 /usr/local/bin/psql ->
/usr/local/pg-8.2.4/bin/psql
(I compile each new version of Postgres to it's own folder and data dir:
/usr/local/pg-<version>/ and /home/Postgres/pgData-<version> )

What am I doing wrong?

Cheers,
~p

Phillip Smith
IT Coordinator
Weatherbeeta P/L
8 Moncrief Rd
Nunawading, VIC, 3131
AUSTRALIA

THINK BEFORE YOU PRINT - Save paper if you don't really need to print this

*******************Confidentiality and Privilege Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official business
of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be liable
for direct, indirect or consequential loss arising from transmission of this
message or any attachments
e-mail.

"Phillip Smith" <phillip(dot)smith(at)weatherbeeta(dot)com(dot)au> writes:
> But I get a whole bunch of errors, for example:
> <snip>
> ERROR: syntax error at or near "all"
> LINE 1: all orders over $400.00 - freight free Order 10 of any produ...
> ^

Please show us the *first* few errors, not some randomly chosen ones.
The above looks like a COPY has failed and psql has started thinking
that copy-data lines are commands, but how or why that happened is
not in evidence.

regards, tom lane

Sorry Tom - I should know better :)

I've attached the first 512 lines of the output from psql. The whole output
is 3mb.

Cheers,
~p

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> Sent: Saturday, 29 December 2007 13:26
> To: Phillip Smith
> Cc: pgsql-admin(at)postgresql(dot)org
> Subject: Re: [ADMIN] Restoring pg_dump Backup with psql Fails
>
> "Phillip Smith" <phillip(dot)smith(at)weatherbeeta(dot)com(dot)au> writes:
> > But I get a whole bunch of errors, for example:
> > <snip>
> > ERROR: syntax error at or near "all"
> > LINE 1: all orders over $400.00 - freight free Order 10 of any produ...
> > ^
>
> Please show us the *first* few errors, not some randomly chosen ones.
> The above looks like a COPY has failed and psql has started thinking
> that copy-data lines are commands, but how or why that happened is
> not in evidence.
>
> regards, tom lane

THINK BEFORE YOU PRINT - Save paper if you don't really need to print this

*******************Confidentiality and Privilege Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official business
of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be liable
for direct, indirect or consequential loss arising from transmission of this
message or any attachments
e-mail.

Phillip Smith wrote:
> Sorry Tom - I should know better :)
>
> I've attached the first 512 lines of the output from psql. The whole output
> is 3mb.
>

You have several problems in this dump. One it appears that some of the
releations already exist. Did you by chance accidently create stuff
within template1 at some point?

Secondly you need to look into iconv to fix your invalid UTF8 data.

Joshua D. Drake

> Cheers,
> ~p
>
>> -----Original Message-----
>> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
>> Sent: Saturday, 29 December 2007 13:26
>> To: Phillip Smith
>> Cc: pgsql-admin(at)postgresql(dot)org
>> Subject: Re: [ADMIN] Restoring pg_dump Backup with psql Fails
>>
>> "Phillip Smith" <phillip(dot)smith(at)weatherbeeta(dot)com(dot)au> writes:
>>> But I get a whole bunch of errors, for example:
>>> <snip>
>>> ERROR: syntax error at or near "all"
>>> LINE 1: all orders over $400.00 - freight free Order 10 of any produ...
>>> ^
>> Please show us the *first* few errors, not some randomly chosen ones.
>> The above looks like a COPY has failed and psql has started thinking
>> that copy-data lines are commands, but how or why that happened is
>> not in evidence.
>>
>> regards, tom lane
>
>
> THINK BEFORE YOU PRINT - Save paper if you don't really need to print this
>
> *******************Confidentiality and Privilege Notice*******************
>
> The material contained in this message is privileged and confidential to
> the addressee. If you are not the addressee indicated in this message or
> responsible for delivery of the message to such person, you may not copy
> or deliver this message to anyone, and you should destroy it and kindly
> notify the sender by reply email.
>
> Information in this message that does not relate to the official business
> of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
> Weatherbeeta, its employees, contractors or associates shall not be liable
> for direct, indirect or consequential loss arising from transmission of this
> message or any attachments
> e-mail.
>
>
> ------------------------------------------------------------------------
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

"Phillip Smith" <phillip(dot)smith(at)weatherbeeta(dot)com(dot)au> writes:
> I've attached the first 512 lines of the output from psql. The whole output
> is 3mb.

Hmm, these are a bit troubling:

> ERROR: relation "grps" already exists
> ...
> ERROR: relation "stock" already exists

because you claimed you were restoring into an empty database ---
either you're mistaken or there is an interesting bug in pg_dump.
Please look into that more closely.

It's hard to be entirely certain, but the subsequent errors seem to
mostly flow from having pre-existing tables "grps" and "stock" that
don't match what the dump script is expecting.

There seems to be a separate encoding problem too:

> ERROR: invalid byte sequence for encoding "UTF8": 0x96
> HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
> CONTEXT: COPY dwh_cust, line 7413
> ERROR: invalid byte sequence for encoding "UTF8": 0xe86d65
> HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by "client_encoding".
> CONTEXT: COPY dwh_stk, line 323461

You didn't mention what PG version you are dumping from, but I'm
thinking it was one that wasn't very strict about encoding correctness.
You may need to clean up the dump file with iconv or some similar tool
--- look through the PG archives for lots of previous discussions about
that.

regards, tom lane

> You have several problems in this dump. One it appears that some of the
> releations already exist. Did you by chance accidently create stuff
> within template1 at some point?

I think I've found my problem:
[postgres(at)stirling backups]$ psql template1
Welcome to psql 8.2.4, the PostgreSQL interactive terminal.
template1=# \d
List of relations
Schema | Name | Type | Owner
--------+--------------+-------+----------
public | buyers | table | postgres
public | creditors | table | postgres
public | debtors | table | postgres
public | grps | table | postgres
public | po | table | postgres
public | sales_orders | table | postgres
public | stkmvt | table | postgres
public | stock | table | postgres
public | sys | table | postgres
(9 rows)

template1=#

I don't know how or when those tables were created in template1 - they're
certainly not supposed to be there or needed. Is it safe to just DROP the
tables from template1?

> Secondly you need to look into iconv to fix your invalid UTF8 data.
My bad - the old database was using SQL_ASCII encoding, but when I recreated
it I let it default to UTF8.

Re-created it again with SQL_ASCII encoding, which didn't copy the tables
from template1 and now the restore seems to be going OK :)

THINK BEFORE YOU PRINT - Save paper if you don't really need to print this

*******************Confidentiality and Privilege Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official business
of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be liable
for direct, indirect or consequential loss arising from transmission of this
message or any attachments
e-mail.

"Phillip Smith" <phillip(dot)smith(at)weatherbeeta(dot)com(dot)au> writes:
> I think I've found my problem:
> I don't know how or when those tables were created in template1 - they're
> certainly not supposed to be there or needed. Is it safe to just DROP the
> tables from template1?

Yeah, but what other user-created stuff (functions for instance) might
be in there?

You might want to drop template1 and recreate it from template0.
(There's a blow-by-blow article about doing that on techdocs, IIRC.)

regards, tom lane

> Yeah, but what other user-created stuff (functions for instance) might
> be in there?
> You might want to drop template1 and recreate it from template0.
> (There's a blow-by-blow article about doing that on techdocs, IIRC.)

I think I'll do that :)
Thanks Tom & Joshua - you've been most helpful as always,
~p

THINK BEFORE YOU PRINT - Save paper if you don't really need to print this

*******************Confidentiality and Privilege Notice*******************

The material contained in this message is privileged and confidential to
the addressee. If you are not the addressee indicated in this message or
responsible for delivery of the message to such person, you may not copy
or deliver this message to anyone, and you should destroy it and kindly
notify the sender by reply email.

Information in this message that does not relate to the official business
of Weatherbeeta must be treated as neither given nor endorsed by Weatherbeeta.
Weatherbeeta, its employees, contractors or associates shall not be liable
for direct, indirect or consequential loss arising from transmission of this
message or any attachments
e-mail.