Re: -f <output file> option for pg_dumpall

In pgAdmin we use pg_dump's -f option to write backup files. The IO
streams are redirected to display status and errors etc. in the GUI.

In order to enhance the interface to allow backup of entire clusters as
well as role and tablespace definitions, we need to be able to get
pg_dumpall to write it's output directly to a file in the same way,
because we cannot redirect the child pg_dump IO streams (which also
means we may miss errors, but I need to think about that some more).

As far as I can see, adding a -f option to pg_dumpall should be straight
forward, the only issue being that we'd need to pass pg_dump an
additional (undocumented?) option to tell it to append to the output
file instead of writing it as normal.

Any thoughts or better ideas?

Regards, Dave.

> As far as I can see, adding a -f option to pg_dumpall should be straight
> forward, the only issue being that we'd need to pass pg_dump an
> additional (undocumented?) option to tell it to append to the output
> file instead of writing it as normal.
>
> Any thoughts or better ideas?

Yeah :) Getting rid of pg_dumpall entirely and merging it into pg_dump.
It is kind of silly that we have two different versions. Although that
may be more work than you were considering.

Joshua D. Drake

>
> Regards, Dave.
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>
--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

Dave Page wrote:
> In pgAdmin we use pg_dump's -f option to write backup files. The IO
> streams are redirected to display status and errors etc. in the GUI.
>
> In order to enhance the interface to allow backup of entire clusters as
> well as role and tablespace definitions, we need to be able to get
> pg_dumpall to write it's output directly to a file in the same way,
> because we cannot redirect the child pg_dump IO streams (which also
> means we may miss errors, but I need to think about that some more).
>
> As far as I can see, adding a -f option to pg_dumpall should be straight
> forward, the only issue being that we'd need to pass pg_dump an
> additional (undocumented?) option to tell it to append to the output
> file instead of writing it as normal.
>
> Any thoughts or better ideas?
>
>

This seems a bit like piecemeal reform. Here are some things I'd like to
see that affect this area:

. merge pg_dump and pg_dumpall (e.g. add a flag to pg_dump that says do
the lot)
. multi-db non-text dumps

And while we're about it, can we teach pg_restore to handle text dumps?
Even if it just transparently called psql it would be nice.

cheers

andrew

On Friday 05 January 2007 09:40, Andrew Dunstan wrote:
> Dave Page wrote:
> > In pgAdmin we use pg_dump's -f option to write backup files. The IO
> > streams are redirected to display status and errors etc. in the GUI.
> >
> > In order to enhance the interface to allow backup of entire clusters as
> > well as role and tablespace definitions, we need to be able to get
> > pg_dumpall to write it's output directly to a file in the same way,
> > because we cannot redirect the child pg_dump IO streams (which also
> > means we may miss errors, but I need to think about that some more).
> >
> > As far as I can see, adding a -f option to pg_dumpall should be straight
> > forward, the only issue being that we'd need to pass pg_dump an
> > additional (undocumented?) option to tell it to append to the output
> > file instead of writing it as normal.
> >
> > Any thoughts or better ideas?
>
> This seems a bit like piecemeal reform. Here are some things I'd like to
> see that affect this area:
>
> . merge pg_dump and pg_dumpall (e.g. add a flag to pg_dump that says do
> the lot)
> . multi-db non-text dumps
>
> And while we're about it, can we teach pg_restore to handle text dumps?
> Even if it just transparently called psql it would be nice.

+1 on on having pg_restore work with text dumps.

>
> cheers
>
> andrew
>
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org

Dave Page wrote:
> In pgAdmin we use pg_dump's -f option to write backup files. The IO
> streams are redirected to display status and errors etc. in the GUI.
>
> In order to enhance the interface to allow backup of entire clusters as
> well as role and tablespace definitions, we need to be able to get
> pg_dumpall to write it's output directly to a file in the same way,
> because we cannot redirect the child pg_dump IO streams (which also
> means we may miss errors, but I need to think about that some more).
>
> As far as I can see, adding a -f option to pg_dumpall should be straight
> forward, the only issue being that we'd need to pass pg_dump an
> additional (undocumented?) option to tell it to append to the output
> file instead of writing it as normal.
>
> Any thoughts or better ideas?
>
Use pgAdmin's "create script" funcion on the server.

Regards,
Andreas

On Fri, Jan 05, 2007 at 12:40:20PM -0500, Andrew Dunstan wrote:
> Dave Page wrote:
> >In pgAdmin we use pg_dump's -f option to write backup files. The IO
> >streams are redirected to display status and errors etc. in the GUI.
> >
> >In order to enhance the interface to allow backup of entire clusters as
> >well as role and tablespace definitions, we need to be able to get
> >pg_dumpall to write it's output directly to a file in the same way,
> >because we cannot redirect the child pg_dump IO streams (which also
> >means we may miss errors, but I need to think about that some more).
> >
> >As far as I can see, adding a -f option to pg_dumpall should be straight
> >forward, the only issue being that we'd need to pass pg_dump an
> >additional (undocumented?) option to tell it to append to the output
> >file instead of writing it as normal.
> >
> >Any thoughts or better ideas?
>
>
> This seems a bit like piecemeal reform. Here are some things I'd like to
> see that affect this area:
>
> . merge pg_dump and pg_dumpall (e.g. add a flag to pg_dump that says do
> the lot)
> . multi-db non-text dumps
>
> And while we're about it, can we teach pg_restore to handle text dumps?
> Even if it just transparently called psql it would be nice.

+1 on all of this :)

Cheers,
D (who can contribute code once we've agreed on what it should do)
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David Fetter wrote:
>> This seems a bit like piecemeal reform. Here are some things I'd like to
>> see that affect this area:
>>
>> . merge pg_dump and pg_dumpall (e.g. add a flag to pg_dump that says do
>> the lot)
>> . multi-db non-text dumps
>>
>> And while we're about it, can we teach pg_restore to handle text dumps?
>> Even if it just transparently called psql it would be nice.
>
> +1 on all of this :)

Hmm, well I have no interest in the latter at present, but assuming the
powers that be will allow me some time to do so, I will look at merging
pg_dump and pg_dumpall as that seems to be the way people want to go.
I'd also like to tweak the options for global objects to allow roles and
tablespaces to be dumped seperately from each other if no-one objects.

> Cheers,
> D (who can contribute code once we've agreed on what it should do)

Cool - fancy picking up the multi-db non text and pg_restore bits if I
do the former?

Regards, Dave.

Andreas Pflug wrote:
> Dave Page wrote:
>> In pgAdmin we use pg_dump's -f option to write backup files. The IO
>> streams are redirected to display status and errors etc. in the GUI.
>>
>> In order to enhance the interface to allow backup of entire clusters as
>> well as role and tablespace definitions, we need to be able to get
>> pg_dumpall to write it's output directly to a file in the same way,
>> because we cannot redirect the child pg_dump IO streams (which also
>> means we may miss errors, but I need to think about that some more).
>>
>> As far as I can see, adding a -f option to pg_dumpall should be straight
>> forward, the only issue being that we'd need to pass pg_dump an
>> additional (undocumented?) option to tell it to append to the output
>> file instead of writing it as normal.
>>
>> Any thoughts or better ideas?
>>
> Use pgAdmin's "create script" funcion on the server.

We said long ago we weren't going to replicate pg_dump functionality in
pgAdmin - a brief discussion on the pgadmin-hackers list earlier
indicates that people still feel the same way, and that time would be
better spend fixing pg_dump/pg_dumpall.

Regards, Dave.

Dave Page <dpage(at)postgresql(dot)org> writes:
> As far as I can see, adding a -f option to pg_dumpall should be straight
> forward, the only issue being that we'd need to pass pg_dump an
> additional (undocumented?) option to tell it to append to the output
> file instead of writing it as normal.

Wouldn't it be easier/better to re-point stdout at the -f file, and not
touch pg_dump at all?

regards, tom lane

Dave Page <dpage(at)postgresql(dot)org> writes:
> Hmm, well I have no interest in the latter at present, but assuming the
> powers that be will allow me some time to do so, I will look at merging
> pg_dump and pg_dumpall as that seems to be the way people want to go.

I think this will be an exercise in time-wasting, and very possibly
destabilize *both* tools. pg_dump has never been designed to reconnect
to a different database; for instance there isn't any code for resetting
all the internal state that it gathers. I think forking a separate
pg_dump for each database is a perfectly fine arrangement, and should be
left alone.

regards, tom lane

Tom Lane schrieb:
> Dave Page <dpage(at)postgresql(dot)org> writes:
>> As far as I can see, adding a -f option to pg_dumpall should be straight
>> forward, the only issue being that we'd need to pass pg_dump an
>> additional (undocumented?) option to tell it to append to the output
>> file instead of writing it as normal.
>
> Wouldn't it be easier/better to re-point stdout at the -f file, and not
> touch pg_dump at all?

Yeah, and maybe have a modifier like %n or something which would instead
write different files and replaces %n with the name of the database...

Regards
Tino

Dave Page wrote:
> In pgAdmin we use pg_dump's -f option to write backup files. The IO
> streams are redirected to display status and errors etc. in the GUI.
>
> In order to enhance the interface to allow backup of entire clusters
> as well as role and tablespace definitions, we need to be able to get
> pg_dumpall to write it's output directly to a file in the same way,
> because we cannot redirect the child pg_dump IO streams (which also
> means we may miss errors, but I need to think about that some more).

I don't understand this. Errors are sent to stderr, and the real output
is sent to stdout, where you can capture and save it. What is the
problem?

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Dave Page wrote:
>> In pgAdmin we use pg_dump's -f option to write backup files. The IO
>> streams are redirected to display status and errors etc. in the GUI.
>>
>> In order to enhance the interface to allow backup of entire clusters
>> as well as role and tablespace definitions, we need to be able to get
>> pg_dumpall to write it's output directly to a file in the same way,
>> because we cannot redirect the child pg_dump IO streams (which also
>> means we may miss errors, but I need to think about that some more).
>
> I don't understand this. Errors are sent to stderr, and the real output
> is sent to stdout, where you can capture and save it. What is the
> problem?
>

On Windows at least, I cannot redirect stdout/stderr of child processes,
only the parent (ie. pg_dumpall) that is executed by pgAdmin (the
pg_dump processes just seem to hang, and the debugger reports a deadlock
with no other useful info).

I'm sure there is a way round this but a) it'll probably mean hacking
wxWidgets or pg_dump about in ways I'm sure Vadim or Tom would dislike,
and b) it would seem reasonable to offer similar functionality in
pg_dumpall to that already in pg_dump anyway.

Regards, Dave.

On Fri, Jan 05, 2007 at 08:05:26PM +0000, Dave Page wrote:
> Andreas Pflug wrote:
> >Dave Page wrote:
> >>In pgAdmin we use pg_dump's -f option to write backup files. The IO
> >>streams are redirected to display status and errors etc. in the GUI.
> >>
> >>In order to enhance the interface to allow backup of entire clusters as
> >>well as role and tablespace definitions, we need to be able to get
> >>pg_dumpall to write it's output directly to a file in the same way,
> >>because we cannot redirect the child pg_dump IO streams (which also
> >>means we may miss errors, but I need to think about that some more).
> >>
> >>As far as I can see, adding a -f option to pg_dumpall should be straight
> >>forward, the only issue being that we'd need to pass pg_dump an
> >>additional (undocumented?) option to tell it to append to the output
> >>file instead of writing it as normal.
> >>
> >>Any thoughts or better ideas?
> >>
> >Use pgAdmin's "create script" funcion on the server.
>
> We said long ago we weren't going to replicate pg_dump functionality in
> pgAdmin - a brief discussion on the pgadmin-hackers list earlier
> indicates that people still feel the same way, and that time would be
> better spend fixing pg_dump/pg_dumpall.

It might make sense to provide a programmatic interface to pg_dump to
provide tools like pgAdmin more flexibility. But it certainly doesn't
make sense to re-create the dumping logic.

In terms of integrating pg_dumpall and pg_dump; I don't really care if
that happens, but I would like to see pg_dumpall supporting all the
options that pg_dump does (a quick diff shows that pg_dumpall is
currently missing support for -f, -F, -v, -Z, -C (which may or may not
make sense), -E, -n & -t (might not make sense), -U and -W (probably
OK).
--
Jim Nasby jim(at)nasby(dot)net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

Jim C. Nasby wrote:
>
>
> It might make sense to provide a programmatic interface to pg_dump to
> provide tools like pgAdmin more flexibility.
Are you talking about "pg_dump in a lib"? Certainly a good idea, because
it allows better integration (e.g. progress bar).
> But it certainly doesn't make sense to re-create the dumping logic.
>
> In terms of integrating pg_dumpall and pg_dump; I don't really care if
> that happens,
I can't make too much sense of integrating pg_dumpall anywhere. Dumping
a whole cluster is certainly much of a planned job, not an interactive
online one, because its output usually won't be usable except for
disaster recovery. Not much function to "re-create" here, single
exception is extracting cluster wide data, the -g option, that's why I
mentioned scripting. But apparently this didn't get into pgadmin svn any
more, so I need to retract this proposal.

Regards,
Andreas

Andreas Pflug wrote:
> Not much function to "re-create" here, single
> exception is extracting cluster wide data, the -g option, that's why I
> mentioned scripting. But apparently this didn't get into pgadmin svn any
> more, so I need to retract this proposal.

Eh? Your SCRIPT code is still there - or do you mean something else?

Regards, Dave.

Dave Page wrote:
> Andreas Pflug wrote:
>
>> Not much function to "re-create" here, single
>> exception is extracting cluster wide data, the -g option, that's why I
>> mentioned scripting. But apparently this didn't get into pgadmin svn any
>> more, so I need to retract this proposal.
>>
>
> Eh? Your SCRIPT code is still there - or do you mean something else?
>
No GetScript implementation for pgServer.

Regards,
Andreas

On Fri, 2007-01-05 at 17:52 -0500, Tom Lane wrote:
> I think this will be an exercise in time-wasting, and very possibly
> destabilize *both* tools. pg_dump has never been designed to reconnect
> to a different database; for instance there isn't any code for resetting
> all the internal state that it gathers.

That is merely an implementation issue. The question of whether pg_dump
and pg_dumpall should be separate programs is a question of design,
IMHO.

I don't think they need to be integrated any time soon, but if we were
to design pg_dump and pg_dumpall from scratch, it seems more logical to
use a single program, and therefore that is the long-term direction I
think we should head in.

-Neil

On Thu, 2007-01-11 at 14:36 -0500, Neil Conway wrote:
> I don't think they need to be integrated any time soon, but if we were
> to design pg_dump and pg_dumpall from scratch, it seems more logical to
> use a single program

On thinking about this some more, it might be useful to factor much of
pg_dump's logic for reconstructing the state of a database into a shared
library. This would make it relatively easy for developers to plug new
archive formats into the library (in addition to the present 3 archive
formats), or to make use of this functionality in other applications
that want to reconstruct the logical state of a database from the
content of the system catalogs. We could then provide a client app
implemented on top of the library that would provide similar
functionality to pg_dump.

Moving pg_dump's functionality into the backend has been suggested in
the past (and rejected for good reason), but I think this might be a
more practical method for making the pg_dump logic more easily reusable.

(While we're on the subject, this would also present a good opportunity
to make pg_dump's command-line interface a little more sane. I wonder if
pg_dump's syntax has gotten sufficiently complicated that some users
might prefer to specify behavior via a configuration file, rather than a
long string of command-line options.)

-Neil

Neil Conway wrote:
> On Thu, 2007-01-11 at 14:36 -0500, Neil Conway wrote:
>
>> I don't think they need to be integrated any time soon, but if we were
>> to design pg_dump and pg_dumpall from scratch, it seems more logical to
>> use a single program
>>
>
> On thinking about this some more, it might be useful to factor much of
> pg_dump's logic for reconstructing the state of a database into a shared
> library. This would make it relatively easy for developers to plug new
> archive formats into the library (in addition to the present 3 archive
> formats), or to make use of this functionality in other applications
> that want to reconstruct the logical state of a database from the
> content of the system catalogs. We could then provide a client app
> implemented on top of the library that would provide similar
> functionality to pg_dump.
>
> Moving pg_dump's functionality into the backend has been suggested in
> the past (and rejected for good reason), but I think this might be a
> more practical method for making the pg_dump logic more easily reusable.
>
>
>

I like this idea. For example, we might usefully map some of this to
psql \ commands, without having to replicate the underlying logic.

cheers

andrew

On Mon, Jan 15, 2007 at 10:13:16AM -0500, Andrew Dunstan wrote:
>
>
> Neil Conway wrote:
> >On Thu, 2007-01-11 at 14:36 -0500, Neil Conway wrote:
> >
> >>I don't think they need to be integrated any time soon, but if we were
> >>to design pg_dump and pg_dumpall from scratch, it seems more logical to
> >>use a single program
> >>
> >
> >On thinking about this some more, it might be useful to factor much of
> >pg_dump's logic for reconstructing the state of a database into a shared
> >library. This would make it relatively easy for developers to plug new
> >archive formats into the library (in addition to the present 3 archive
> >formats), or to make use of this functionality in other applications
> >that want to reconstruct the logical state of a database from the
> >content of the system catalogs. We could then provide a client app
> >implemented on top of the library that would provide similar
> >functionality to pg_dump.
> >
> >Moving pg_dump's functionality into the backend has been suggested in
> >the past (and rejected for good reason), but I think this might be a
> >more practical method for making the pg_dump logic more easily reusable.
> >
> >
> >
>
> I like this idea. For example, we might usefully map some of this to
> psql \ commands, without having to replicate the underlying logic.

Don't we already do this with the .psqlrc file?

--elein

elein wrote:
> On Mon, Jan 15, 2007 at 10:13:16AM -0500, Andrew Dunstan wrote:
>>
>>
>> Neil Conway wrote:
>> >On Thu, 2007-01-11 at 14:36 -0500, Neil Conway wrote:
>> >
>> >>I don't think they need to be integrated any time soon, but if we were
>> >>to design pg_dump and pg_dumpall from scratch, it seems more logical
>> to
>> >>use a single program
>> >>
>> >
>> >On thinking about this some more, it might be useful to factor much of
>> >pg_dump's logic for reconstructing the state of a database into a
>> shared
>> >library. This would make it relatively easy for developers to plug new
>> >archive formats into the library (in addition to the present 3 archive
>> >formats), or to make use of this functionality in other applications
>> >that want to reconstruct the logical state of a database from the
>> >content of the system catalogs. We could then provide a client app
>> >implemented on top of the library that would provide similar
>> >functionality to pg_dump.
>> >
>> >Moving pg_dump's functionality into the backend has been suggested in
>> >the past (and rejected for good reason), but I think this might be a
>> >more practical method for making the pg_dump logic more easily
>> reusable.
>> >
>> >
>> >
>>
>> I like this idea. For example, we might usefully map some of this to
>> psql \ commands, without having to replicate the underlying logic.
>
> Don't we already do this with the .psqlrc file?
>

No. \ commands are implemented in C code.

cheers

andrew