Re: psql enhancement idea

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi everyone,

I'm just doing a load of database transfers using pg_dump. Because I changed
the schema of quite some tables I have to dump full column inserts.
The normal -D dump doesn't use a transaction, and I can see that this
certainly shouldn't be the default. Nonetheless I think it might be a nice
feature if there is a optional switch that creates a BEGIN .... COMMIT block
around the inserts of each table's data. Full column inserts are at least 10
times faster if used in a transaction. Ok I understand that the transaction
bites big chunks out of the ressources - which wouldn't matter if you do that
on a development machine with plenty of everything and the tables don't have
billions of rows.

Thought's anyone ?

UC

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: +1 650 872 2425 San Bruno, CA 94066
Cell: +1 650 302 2405 United States
Fax: +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeAhljqGXBvRToM4RAhHBAJ9pnOxF2owze5k1+qBsjg2BB39BuwCePdpJ
clYgE7K5TD7HYhhHy3JudHQ=
=Xpsr
-----END PGP SIGNATURE-----

"Uwe C. Schroeder" <uwe(at)oss4u(dot)com> writes:
> I'm just doing a load of database transfers using pg_dump. Because I changed
> the schema of quite some tables I have to dump full column inserts.

I don't think that conclusion follows from that premise. In recent
pg_dump versions (any that use a column list with COPY, which I think is
7.3 or later) there is no fundamental disadvantage to using COPY; it
should be semantically equivalent to INSERT-with-column-list commands.

The only reason I can see for using the INSERT option anymore is if you
think that some of the row insertions might fail, and you don't want
that to stop the load altogether. If that's what you want, then you'd
certainly not want to wrap the load in BEGIN/END, so there doesn't seem
to be much point in adding an option to do it.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
> "Uwe C. Schroeder" <uwe(at)oss4u(dot)com> writes:
> > I'm just doing a load of database transfers using pg_dump. Because I
> > changed the schema of quite some tables I have to dump full column
> > inserts.
>
> I don't think that conclusion follows from that premise. In recent
> pg_dump versions (any that use a column list with COPY, which I think is
> 7.3 or later) there is no fundamental disadvantage to using COPY; it
> should be semantically equivalent to INSERT-with-column-list commands.
>
> The only reason I can see for using the INSERT option anymore is if you
> think that some of the row insertions might fail, and you don't want
> that to stop the load altogether. If that's what you want, then you'd
> certainly not want to wrap the load in BEGIN/END, so there doesn't seem
> to be much point in adding an option to do it.
>
> regards, tom lane

The reason is that I made changes to the schema, i.e. changed the ordinal
position of columns and added some columns. Since the positioning of the
columns isn't the same anymore a copy will fail. Column Inserts will leave
the new columns untouched (set them to NULL or whatever the default) but can
handle the change in order of the columns (ok, the order thing is purely
cosmetic...).
I see your point that you don't want to fail the whole import when using full
inserts - which would happen in a transaction block. My point was basically
that I assume all inserts will succeed, however I can't use copy because of
the changes but doing it in a transaction will increase the insert speed by
at least 10 (chewing up memory certainly).

As a remark: normally I make pure data backups as copy and as full column
inserts. In case of a recovery I then can easily switch to single inserts if
the copy fails - eats a bit more diskspace but increases security (besides
the slony replication I'm running on top of that...)

Uwe

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: +1 650 872 2425 San Bruno, CA 94066
Cell: +1 650 302 2405 United States
Fax: +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDF0jqGXBvRToM4RAmrkAKCoRXnlmnRpp0DI5/cFeCyx97nm1QCfWJ/A
8zSHT4NXzG8LG7GFqpPGSbE=
=c99W
-----END PGP SIGNATURE-----

"Uwe C. Schroeder" <uwe(at)oss4u(dot)com> writes:
> On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
>> I don't think that conclusion follows from that premise. In recent
>> pg_dump versions (any that use a column list with COPY, which I think is
>> 7.3 or later) there is no fundamental disadvantage to using COPY; it
>> should be semantically equivalent to INSERT-with-column-list commands.

> The reason is that I made changes to the schema, i.e. changed the ordinal
> position of columns and added some columns. Since the positioning of the
> columns isn't the same anymore a copy will fail.

No, it won't, not if you use a column list.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Thursday 21 October 2004 03:03 pm, Tom Lane wrote:
> "Uwe C. Schroeder" <uwe(at)oss4u(dot)com> writes:
> > On Thursday 21 October 2004 12:40 pm, Tom Lane wrote:
> >> I don't think that conclusion follows from that premise. In recent
> >> pg_dump versions (any that use a column list with COPY, which I think is
> >> 7.3 or later) there is no fundamental disadvantage to using COPY; it
> >> should be semantically equivalent to INSERT-with-column-list commands.
> >
> > The reason is that I made changes to the schema, i.e. changed the ordinal
> > position of columns and added some columns. Since the positioning of the
> > columns isn't the same anymore a copy will fail.
>
> No, it won't, not if you use a column list.

Ahhh - I see what you mean, thanks for that hint.

A little remark then: the pg_dump manpage (7.4.3) states

--attribute-inserts
Dump data as INSERT commands with explicit column names (INSERT
INTO table (column, ...) VALUES ...). This will make restoration
very slow, but it is necessary if you desire to rearrange the
column ordering.

Maybe that should be fixed then. I was used to do the full insert statements
(ok, I've been using postgres long before it even had SQL) - maybe a old
habit then...

Uwe

- --
Open Source Solutions 4U, LLC 2570 Fleetwood Drive
Phone: +1 650 872 2425 San Bruno, CA 94066
Cell: +1 650 302 2405 United States
Fax: +1 650 872 2417
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFBeDW4jqGXBvRToM4RAu4+AJ9gchqGNrP/IUUYy8ySMWLyP6I0JgCdG1SH
h0RLcQppb2yLaTkrG/W5cvc=
=5tEF
-----END PGP SIGNATURE-----

"Uwe C. Schroeder" <uwe(at)oss4u(dot)com> writes:
> A little remark then: the pg_dump manpage (7.4.3) states

> --attribute-inserts
> Dump data as INSERT commands with explicit column names (INSERT
> INTO table (column, ...) VALUES ...). This will make restoration
> very slow, but it is necessary if you desire to rearrange the
> column ordering.

> Maybe that should be fixed then.

Mmm, you are right, that comment is a tad out of date. Patch committed
--- thanks for pointing it out!

regards, tom lane