Re: One less footgun: deprecating pg_dump -d

Attached is a patch to fix the annoying footgun that is pg_dump -d. Myself and
many others I know have all at one time or another done this:

psql -h localhost -U greg -d postgres

pg_dump -h localhost -U greg -d postgres > dumpfile

The latter command silently succeeds, but only through the combination of -d
being an option that takes no arguments, and the fact that 'postgres' is read by
pg_dump as a separate argument indicating which database to use. Thus, your dump
file now has INSERTs when you wanted to use COPY (as you want your database
restore to take 20 minutes, not three hours).

I thought about changing -d to actually indicate the database, as in psql, but
that brings about another problem: the command above will still silently work,
but produce a dump without INSERTs. While this is good for people who meant to
leave the -d out, it's not good for people (and scripts) that DID want the -d to
work as documented. Thus, changing it will silently break those scripts (until
they try to load the schema into a non-PG database...).

The solution I came up with is to use a new letter, -I, and to deprecate -d by
having it throw an exception when used. The choice of -I seems appropriate as a
shortcut for --inserts, and (as far as I can tell) does not conflict with any
other programs (e.g. psql). Doing so will require people to rewrite any scripts
that are using -d instead of --inserts, but it seems a small price to eliminate
this nasty footgun. As a bonus, their scripts will be easier to read, as -d was
confusing at best, and hardly mnemonic.

--
Greg Sabino Mullane greg(at)endpoint(dot)com greg(at)turnstep(dot)com
End Point Corporation
PGP Key: 0x14964AC8

Greg Sabino Mullane <greg(at)endpoint(dot)com> writes:
> The solution I came up with is to use a new letter, -I, and to deprecate -d by
> having it throw an exception when used.

"Deprecate" does not mean "break".

regards, tom lane

On Mon, 2009-03-09 at 12:48 -0400, Tom Lane wrote:
> Greg Sabino Mullane <greg(at)endpoint(dot)com> writes:
> > The solution I came up with is to use a new letter, -I, and to deprecate -d by
> > having it throw an exception when used.
>
> "Deprecate" does not mean "break".

Sorry Tom. Greg is correct here although I disagree with his wording. It
should be removed and if someone passes -d it should throw an ERROR that
says something like:

ERROR: -d has been replaced by -I

Greg and I are both in the field and the field consistently uses -d in
the wrong way.

Joshua D. Drake

>
> regards, tom lane
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Sorry Tom. Greg is correct here although I disagree with his wording. It
> should be removed and if someone passes -d it should throw an ERROR that
> says something like:
> ERROR: -d has been replaced by -I

Well, if you want to break it, we can debate about the wisdom of that.
But please don't describe the patch in such a misleading way as the
current thread title.

regards, tom lane

On Mon, 2009-03-09 at 13:30 -0400, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > Sorry Tom. Greg is correct here although I disagree with his wording. It
> > should be removed and if someone passes -d it should throw an ERROR that
> > says something like:
> > ERROR: -d has been replaced by -I
>
> Well, if you want to break it, we can debate about the wisdom of that.
> But please don't describe the patch in such a misleading way as the
> current thread title.

That's fair.

Joshua D. Drake

>
> regards, tom lane
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Tom Lane wrote:
> Greg Sabino Mullane<greg(at)endpoint(dot)com> writes:
>> The solution I came up with is to use a new letter, -I, and to deprecate -d by
>> having it throw an exception when used.
>
> "Deprecate" does not mean "break".

This '-d' thing is more than just a matter of reading the documentation.
Our other command line utilities lead a person to assume (logically)
that '-d' means something completely apart from what it actually does.

I've made this mistake, so have most other sysadmins I know.

While this change may break existing scripts, the result is that future
users of Postgres will have a much less painful experience if they
accidentally try to use that option.

-selena

--
Selena Deckelmann
End Point Corporation
selena(at)endpoint(dot)com

Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> Sorry Tom. Greg is correct here although I disagree with his wording. It
>> should be removed and if someone passes -d it should throw an ERROR that
>> says something like:
>> ERROR: -d has been replaced by -I
>
> Well, if you want to break it, we can debate about the wisdom of that.
> But please don't describe the patch in such a misleading way as the
> current thread title.

+1 with breaking it, but with a better message (and let's call it
breaking, not deprecating).

Oh, and the patch contains what looks like two merge failures, I'm sure
that wasn't intentional...

//Magnus

Selena Deckelmann wrote:
> Tom Lane wrote:
>> Greg Sabino Mullane<greg(at)endpoint(dot)com> writes:
>>> ...
>>> deprecate -d by having it throw an exception when used.
>>
>> "Deprecate" does not mean "break".
> ...
> While this change may break existing scripts...less painful

Why do people want a failure rather than warning messages
being spewed to both stderr and the log files?

If someone doesn't notice warnings there, I wonder if
even throwing an exception would save them.

>>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> +1 with breaking it, but with a better message (and let's call it
> breaking, not deprecating).

Are you proposing to leave -D as is?

-Kevin

Kevin Grittner wrote:
>>>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> +1 with breaking it, but with a better message (and let's call it
>> breaking, not deprecating).
>
> Are you proposing to leave -D as is?

I was :-)

but maybe it's better to use -i and -I, and thus change them both?

//Magnus

>>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> Kevin Grittner wrote:
>> Are you proposing to leave -D as is?
>
> I was :-)
>
> but maybe it's better to use -i and -I, and thus change them both?

That's already used:

-i, --ignore-version proceed even when server version mismatches
pg_dump version

-Kevin

"Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> but maybe it's better to use -i and -I, and thus change them both?

> That's already used:

> -i, --ignore-version proceed even when server version mismatches
> pg_dump version

Proposal: drop the short forms of these two switches entirely.
Anybody who actually needs the capability can write "--inserts".

regards, tom lane

Tom Lane wrote:
> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
>
>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>
>>> but maybe it's better to use -i and -I, and thus change them both?
>>>
>
>
>> That's already used:
>>
>
>
>> -i, --ignore-version proceed even when server version mismatches
>> pg_dump version
>>
>
> Proposal: drop the short forms of these two switches entirely.
> Anybody who actually needs the capability can write "--inserts".
>
>
>

+1. I was just thinking the same thing.

cheers

andrew

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
>>
>>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> but maybe it's better to use -i and -I, and thus change them both?
>>>>
>>
>>
>>> That's already used:
>>>
>>
>>
>>> -i, --ignore-version proceed even when server version mismatches
>>> pg_dump version
>>>
>>
>> Proposal: drop the short forms of these two switches entirely.
>> Anybody who actually needs the capability can write "--inserts".
>>
>>
>>
>
> +1. I was just thinking the same thing.

+1, that sounds like a very good idea.

//Magnus

>> -i, --ignore-version proceed even when server version mismatches
>> pg_dump version
>
> Proposal: drop the short forms of these two switches entirely.
> Anybody who actually needs the capability can write "--inserts".

I thought about something like that, but that would break even more existing
scripts than the current patch, no? I'd be all for not using -I though, as that
would not break anything.

Sorry about the "deprecation" name, I withdraw that part

Magnus: Sorry about non-mergeability, I wrote this while offline...

--
Greg Sabino Mullane greg(at)endpoint(dot)com
End Point Corporation
PGP Key: 0x14964AC8

Greg Sabino Mullane wrote:
>>> -i, --ignore-version proceed even when server version mismatches
>>> pg_dump version
>> Proposal: drop the short forms of these two switches entirely.
>> Anybody who actually needs the capability can write "--inserts".
>
> I thought about something like that, but that would break even more existing
> scripts than the current patch, no? I'd be all for not using -I though, as that
> would not break anything.
>
> Sorry about the "deprecation" name, I withdraw that part
>
> Magnus: Sorry about non-mergeability, I wrote this while offline...

No, the problem is not that I get merge failures. It's that *your* merge
conflicts are included in the patch itself.

//Magnus

On Mon, 2009-03-09 at 15:48 -0400, Tom Lane wrote:
> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> > Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> >> but maybe it's better to use -i and -I, and thus change them both?
>
> > That's already used:
>
> > -i, --ignore-version proceed even when server version mismatches
> > pg_dump version
>
> Proposal: drop the short forms of these two switches entirely.
> Anybody who actually needs the capability can write "--inserts".

I could buy into that.

Joshua D. Drake

>
> regards, tom lane
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Mon, Mar 09, 2009 at 09:02:01PM +0100, Magnus Hagander wrote:
> Andrew Dunstan wrote:
> > Tom Lane wrote:
> >> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> >>
> >>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> >>>> but maybe it's better to use -i and -I, and thus change them both?
> >>
> >>> That's already used:
> >>
> >>> -i, --ignore-version proceed even when server version mismatches
> >>> pg_dump version
> >>
> >> Proposal: drop the short forms of these two switches entirely.
> >> Anybody who actually needs the capability can write "--inserts".
> >
> > +1. I was just thinking the same thing.
>
> +1, that sounds like a very good idea.

FWIW, +1 from me for removing the -i and -d options, leaving only long
versions of what they used to do.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Mon, 2009-03-09 at 21:02 +0100, Magnus Hagander wrote:
> Andrew Dunstan wrote:
> > Tom Lane wrote:
> >> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> >>
> >>> Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> >>>> but maybe it's better to use -i and -I, and thus change them both?
> >>>>
> >>
> >>
> >>> That's already used:
> >>
> >>> -i, --ignore-version proceed even when server version mismatches
> >>> pg_dump version
> >>
> >> Proposal: drop the short forms of these two switches entirely.
> >> Anybody who actually needs the capability can write "--inserts".
> >
> > +1. I was just thinking the same thing.
>
> +1, that sounds like a very good idea.

+1 good plan

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

It seemed that the majority opinion in this thread was to eliminate
the confusion by dropping the short form "d" switch in pg_dump and
pg_dumpall --- if you want that behavior you'll have to write
"--inserts".

For consistency I suggested also dropping the short form switch
"D" for "--column-inserts", although this is not absolutely necessary
in terms of eliminating a conflict.

Last chance for objections ...

regards, tom lane