Re: Renaming of pg_xlog and pg_clog

Hi all,

I am relaunching $subject as 10 development will begin soon. As far as
I know, there is agreement that we can do something here. Among the
different proposals I have found:
- pg_clog renamed to pg_commit_status, pg_xact or pg_commit
- pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Another idea from Stephen
(https://www.postgresql.org/message-id/20160826003715.GG4028@tamriel.snowman.net)
would be to put everything that is temporary and not WAL-logged into a
single place to facilitate the filtering work of backup tools.

A straight renaming would be a simple patch (including pg_upgrade
part), and if we actually do it for 10.0 it would be good to do it now
instead of in 3 months. I don't mind writing a patch for it.

Now, one of the things discussed as well was that we may want to still
keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
to not break the existing tools. Personally, I'd prefer a hard break.
That would not be complicated to fix for any tool maintainers.

Thoughts?
--
Michael

On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
> Now, one of the things discussed as well was that we may want to still
> keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
> to not break the existing tools. Personally, I'd prefer a hard break.
> That would not be complicated to fix for any tool maintainers.

I agree on a hard break, unless we get pushback from users, and even
then, they can create the symlinks themselves.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

On Fri, Aug 26, 2016 at 11:55 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
>> Now, one of the things discussed as well was that we may want to still
>> keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
>> to not break the existing tools. Personally, I'd prefer a hard break.
>> That would not be complicated to fix for any tool maintainers.
>
> I agree on a hard break, unless we get pushback from users, and even
> then, they can create the symlinks themselves.

I strongly prefer symlink approach not to break many existing tools
and scripts.

Regards,

--
Fujii Masao

* Bruce Momjian (bruce(at)momjian(dot)us) wrote:
> On Fri, Aug 26, 2016 at 11:39:29AM +0900, Michael Paquier wrote:
> > Now, one of the things discussed as well was that we may want to still
> > keep pg_xlog, and soft-link to pg_journal or whatever-the-new-name is
> > to not break the existing tools. Personally, I'd prefer a hard break.
> > That would not be complicated to fix for any tool maintainers.
>
> I agree on a hard break, unless we get pushback from users, and even
> then, they can create the symlinks themselves.

I'm in favor of the hard break generally, but I'd like David's input on
it, to be sure we're not missing something.

Thanks!

Stephen

Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+CMzgv5u6OEmxr-cbJRO+WKWQ(at)mail(dot)gmail(dot)com>
> > I agree on a hard break, unless we get pushback from users, and even
> > then, they can create the symlinks themselves.
>
> I strongly prefer symlink approach not to break many existing tools
> and scripts.

Symlinks might actually be worse than removing the directories
altogether. If your backup tool fails because the pg_xlog directory is
gone, you'll hopefully notice, but if you end up with a backup that
consists merely of a copy of a symlink named pg_xlog, you might not
notice.

Christoph

On Fri, Aug 26, 2016 at 12:44 PM, Christoph Berg <myon(at)debian(dot)org> wrote:

> Re: Fujii Masao 2016-08-26 <CAHGQGwHK2yimfLvG_WQ1Vrq2h+
> CMzgv5u6OEmxr-cbJRO+WKWQ(at)mail(dot)gmail(dot)com>
> > > I agree on a hard break, unless we get pushback from users, and even
> > > then, they can create the symlinks themselves.
> >
> > I strongly prefer symlink approach not to break many existing tools
> > and scripts.
>
> Symlinks might actually be worse than removing the directories
> altogether. If your backup tool fails because the pg_xlog directory is
> gone, you'll hopefully notice, but if you end up with a backup that
> consists merely of a copy of a symlink named pg_xlog, you might not
> notice.
>

+1. It's *much* better to cause a clean break. That way people will notice
it, and can fix it (and it will be an easy fix).

An unclean break might leave people with things that look like they work,
but don't. That's a lot more dangerous.

Same reason I'm also +1 for Stephens suggestion to put all things that
should not be in a base backup into the same directory. That may break
things now, but it will simplify things down the road. And doing it at the
same time as renaming these things makes a lot of sense, because it causes
breakage that tool-builders *have* to look at, and then they will hopefully
also notice the other change.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 26 August 2016 at 04:39, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
> Hi all,
>
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

Don't mean to be a party pooper, but what discussion and agreement are
we referring to here?

If we are going to suggest doing something we really should summarize
the reason for doing it rather than assume it is self evident, cos it
certainly isn't.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Fri, Aug 26, 2016 at 8:31 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On 26 August 2016 at 04:39, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
>> Hi all,
>>
>> I am relaunching $subject as 10 development will begin soon. As far as
>> I know, there is agreement that we can do something here. Among the
>> different proposals I have found:
>> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
>> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
> Don't mean to be a party pooper, but what discussion and agreement are
> we referring to here?
>
> If we are going to suggest doing something we really should summarize
> the reason for doing it rather than assume it is self evident, cos it
> certainly isn't.

This thread was the previous one on the matter:
http://www.postgresql.org/message-id/CAASwCXcVGma9KgEu-ESC6u928mW67noZvnawbPUSW7R7AN9UVg@mail.gmail.com

In short, with the current names, sometimes users think that pg_xlog
and pg_clog are just logs. And so it is fine to delete them to free up
space, corrupting their cluster, because they are just *logs*.
Personally I have seen that, and based on the thread I am not the only
one.
--
Michael

Hi,

On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:
> In short, with the current names, sometimes users think that pg_xlog
> and pg_clog are just logs. And so it is fine to delete them to free up
> space, corrupting their cluster, because they are just *logs*.

...and we also have "pg_logical", that includes a "log" keyword already...

Regards,

--
Devrim GÜNDÜZ
EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Twitter: @DevrimGunduz , @DevrimGunduzTR

On 26-08-2016 09:25, Devrim Gündüz wrote:
> On Fri, 2016-08-26 at 21:12 +0900, Michael Paquier wrote:
>> In short, with the current names, sometimes users think that pg_xlog
>> and pg_clog are just logs. And so it is fine to delete them to free up
>> space, corrupting their cluster, because they are just *logs*.
>
> ...and we also have "pg_logical", that includes a "log" keyword already...
>
"clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
people confusing "logical" meaning "log".

--
Euler Taveira Timbira - http://www.timbira.com.br/
PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

Euler Taveira <euler(at)timbira(dot)com(dot)br> writes:
> On 26-08-2016 09:25, Devrim Gndz wrote:
>> ...and we also have "pg_logical", that includes a "log" keyword already...

> "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
> people confusing "logical" meaning "log".

Well, I dunno; people with a weak grasp of English might have an issue
there. But I never liked that directory name anyway; my problem with
it is I read "pg_logical" and think "logical what?". Naming things using
a disconnected adjective is not good, especially one with as many
potential applications as that one has. If we're up for renaming things
under PGDATA, that one is high on my hit list.

regards, tom lane

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Euler Taveira <euler(at)timbira(dot)com(dot)br> writes:
> > On 26-08-2016 09:25, Devrim Gündüz wrote:
> >> ...and we also have "pg_logical", that includes a "log" keyword already...
>
> > "clog" and "xlog" is almost "log"; "logical" is not. I don't imagine
> > people confusing "logical" meaning "log".
>
> Well, I dunno; people with a weak grasp of English might have an issue
> there. But I never liked that directory name anyway; my problem with
> it is I read "pg_logical" and think "logical what?". Naming things using
> a disconnected adjective is not good, especially one with as many
> potential applications as that one has. If we're up for renaming things
> under PGDATA, that one is high on my hit list.

iirc, pg_logical also has both temporary and non-temporary data in it
too. That complicates things for backup utilities that are trying to
exclude all temporary files. If we actually move all the temp files
into their own directory (or tree), then we're changing what's in
pg_logical anyway, so renaming it seems like a good idea.

Thanks!

Stephen

On 08/26/2016 03:48 AM, Magnus Hagander wrote:

>
> Same reason I'm also +1 for Stephens suggestion to put all things that
> should not be in a base backup into the same directory. That may break
> things now, but it will simplify things down the road. And doing it at
> the same time as renaming these things makes a lot of sense, because it
> causes breakage that tool-builders *have* to look at, and then they will
> hopefully also notice the other change.

If this is done this fall, developers will have at least a year to fix
their utilities.

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

On 08/25/2016 07:39 PM, Michael Paquier wrote:
> Hi all,
>
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

I think the use of the pg_ prefix is redundant. Just a directory called:
wal will do (for example).

In reference to pg_xlog specifically, it should be wal. It is the Write
Ahead Log, not the Journal (although I recognize they can be
synonymous). All the documentation talks about Write Ahead Log.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd(at)commandprompt(dot)com> wrote:
>
> On 08/25/2016 07:39 PM, Michael Paquier wrote:
>>
>> Hi all,
>>
>> I am relaunching $subject as 10 development will begin soon. As far as
>> I know, there is agreement that we can do something here. Among the
>> different proposals I have found:
>> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
>> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
>
> I think the use of the pg_ prefix is redundant. Just a directory called:
wal will do (for example).
>
> In reference to pg_xlog specifically, it should be wal. It is the Write
Ahead Log, not the Journal (although I recognize they can be synonymous).
All the documentation talks about Write Ahead Log.
>

Yes, let's avoid inventing a *third* name for it, please. It's already bad
enough that we have both wal and xlog.

/Magnus

On 8/25/16 10:39 PM, Michael Paquier wrote:
> I am relaunching $subject as 10 development will begin soon. As far as
> I know, there is agreement that we can do something here. Among the
> different proposals I have found:
> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal

If we're going to do some renaming, then I suggest we do a
mini-file-system structure under $PGDATA, like

$PGDATA/etc
$PGDATA/log
$PGDATA/run (lock files etc.)
$PGDATA/tmp
$PGDATA/var

The names of all the things under "var" could still be refined, but it's
much less likely that users will confuse data with configuration or
plain logs under that scheme.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
>
> On 8/25/16 10:39 PM, Michael Paquier wrote:
> > I am relaunching $subject as 10 development will begin soon. As far as
> > I know, there is agreement that we can do something here. Among the
> > different proposals I have found:
> > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
> If we're going to do some renaming, then I suggest we do a
> mini-file-system structure under $PGDATA, like
>
> $PGDATA/etc
> $PGDATA/log
> $PGDATA/run (lock files etc.)
> $PGDATA/tmp
> $PGDATA/var
>
> The names of all the things under "var" could still be refined, but it's
> much less likely that users will confuse data with configuration or
> plain logs under that scheme

Interesting idea. I worry a bit that this might encourage distributions to
split it up into different places though, and I'm not sure we want to
encourage that..

/Magnus

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
> peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
>> If we're going to do some renaming, then I suggest we do a
>> mini-file-system structure under $PGDATA, like
>>
>> $PGDATA/etc
>> $PGDATA/log
>> $PGDATA/run (lock files etc.)
>> $PGDATA/tmp
>> $PGDATA/var
>>
>> The names of all the things under "var" could still be refined, but it's
>> much less likely that users will confuse data with configuration or
>> plain logs under that scheme

> Interesting idea. I worry a bit that this might encourage distributions to
> split it up into different places though, and I'm not sure we want to
> encourage that..

Yeah, I'm afraid that these names are not as well standardized as Peter
probably wishes they were. Also, I'd just as soon not move/rename things
that don't really need it. If, for example, we decide to move
postgresql.conf to etc/postgresql.conf, that is going to break a metric
ton of stuff that doesn't need to get broken AFAICS.

regards, tom lane

On 08/26/2016 09:28 AM, Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Aug 26, 2016 5:54 PM, "Peter Eisentraut" <
>> peter(dot)eisentraut(at)2ndquadrant(dot)com> wrote:
>>> If we're going to do some renaming, then I suggest we do a
>>> mini-file-system structure under $PGDATA, like
>>>
>>> $PGDATA/etc
>>> $PGDATA/log
>>> $PGDATA/run (lock files etc.)
>>> $PGDATA/tmp
>>> $PGDATA/var
>>>
>>> The names of all the things under "var" could still be refined, but it's
>>> much less likely that users will confuse data with configuration or
>>> plain logs under that scheme
>
>> Interesting idea. I worry a bit that this might encourage distributions to
>> split it up into different places though, and I'm not sure we want to
>> encourage that..
>
> Yeah, I'm afraid that these names are not as well standardized as Peter
> probably wishes they were. Also, I'd just as soon not move/rename things
> that don't really need it. If, for example, we decide to move
> postgresql.conf to etc/postgresql.conf, that is going to break a metric
> ton of stuff that doesn't need to get broken AFAICS.

I am not so sure that is accurate. Yes, Windows is an outlying but any
Unix person is going to easily understand etc log run. Further as Linux
is by far our most run platform (outside of possibly Windows) it will
feel right at home for the largest growing user base. Personally I
really like the idea of:

$PGDATA/etc/postgresql.conf
$PGDATA/log/Thu.log
$PGDATA/run/postmaster.pid
$PGDATA/var/base/16758

Although... wouldn't run be under var?

JD

>
> regards, tom lane
>
>

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

On 26 August 2016 at 18:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Also, I'd just as soon not move/rename things that don't really need it.

+1

Let's leave everything exactly as it is now... but put a small README
in each directory to explain why files in it shouldn't be deleted to
make space.

That helps the few people who made such mistakes, but doesn't cause
massive change as a result.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 8/26/16 11:58 AM, Magnus Hagander wrote:
> $PGDATA/etc
>> $PGDATA/log
>> $PGDATA/run (lock files etc.)
>> $PGDATA/tmp
>> $PGDATA/var
>>
>> The names of all the things under "var" could still be refined, but it's
>> much less likely that users will confuse data with configuration or
>> plain logs under that scheme
>
> Interesting idea. I worry a bit that this might encourage distributions
> to split it up into different places though, and I'm not sure we want to
> encourage that..

They already do this. This would just formalize where we think
appropriate boundaries between the pieces are.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-08-26 11:53:21 -0400, Peter Eisentraut wrote:
> On 8/25/16 10:39 PM, Michael Paquier wrote:
> > I am relaunching $subject as 10 development will begin soon. As far as
> > I know, there is agreement that we can do something here. Among the
> > different proposals I have found:
> > - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> > - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
>
> If we're going to do some renaming, then I suggest we do a
> mini-file-system structure under $PGDATA, like
>
> $PGDATA/etc
> $PGDATA/log
> $PGDATA/run (lock files etc.)
> $PGDATA/tmp
> $PGDATA/var
>
> The names of all the things under "var" could still be refined, but it's
> much less likely that users will confuse data with configuration or
> plain logs under that scheme.

Splitting of ephemeral data seems to have a benefit, the rest seems more
like rather noisy busywork to me.

On 8/26/16 12:28 PM, Tom Lane wrote:
> Also, I'd just as soon not move/rename things
> that don't really need it.

I'm just as happy with not changing anything. But if we're going to
rename stuff, let's at least think about something slightly more
comprehensive. Any rename is going to break a bunch of stuff. But if
we break it in a way that reduces the need for future discussion or
changes, it would at least be a small win in the long run.

> If, for example, we decide to move
> postgresql.conf to etc/postgresql.conf, that is going to break a metric
> ton of stuff that doesn't need to get broken AFAICS.

Sure, I'd be content with leaving those in the top-level instead of say
"etc".

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 8/26/16 4:02 PM, Joshua D. Drake wrote:
> Although... wouldn't run be under var?

Traditionally yes, but things are changing in this area, if you consider
the top-level file system of a modern Linux distribution. One reason is
that "run" is/can be blown away at reboot. This wouldn't be an entirely
useless feature for postgres: Can you tell otherwise which of the
various pid/lock/opts files you can delete if you have killed the
processes and want to eliminate any left-over state? Is postmaster.opts
a configuration file or a state file?

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-08-26 17:11:00 -0400, Peter Eisentraut wrote:
> On 8/26/16 12:28 PM, Tom Lane wrote:
> > Also, I'd just as soon not move/rename things
> > that don't really need it.
>
> I'm just as happy with not changing anything. But if we're going to
> rename stuff, let's at least think about something slightly more
> comprehensive. Any rename is going to break a bunch of stuff. But if
> we break it in a way that reduces the need for future discussion or
> changes, it would at least be a small win in the long run.

I do think there's an order of magnitude between the impact between
moving some and moving everything. And that's going to impact
cost/benefit calculations.

Moving e.g. all ephemeral files into a (possibly configurable) directory
is going to hardly impact anyone. Renaming pg_logical into something
different (FWIW, it was originally named differently...) will hopefully
impact nobody, excepting some out of date file exclusion lists possibly.

But moving config files, and even pg_xlog (which we document to be
symlinkable somewhere else) imo is different.

The other thing is that the likelihood of getting anywhere by doing
radical one-off redesigns is approximately 0.

On 8/26/16 5:20 PM, Andres Freund wrote:
> I do think there's an order of magnitude between the impact between
> moving some and moving everything. And that's going to impact
> cost/benefit calculations.
>
> Moving e.g. all ephemeral files into a (possibly configurable) directory
> is going to hardly impact anyone. Renaming pg_logical into something
> different (FWIW, it was originally named differently...) will hopefully
> impact nobody, excepting some out of date file exclusion lists possibly.
>
> But moving config files, and even pg_xlog (which we document to be
> symlinkable somewhere else) imo is different.

I agree with all that. But the subject line is specifically about
moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
then that is noted. But if we were to move it, we can think about a
good place to move it to.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
> I agree with all that. But the subject line is specifically about
> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
> then that is noted. But if we were to move it, we can think about a
> good place to move it to.

I think it's probably worth moving pg_xlog, because the benefit also
includes preventing a few users from shooting themselves somewhere
vital. That's imo much less the case for some of the other moves. But I
still don't think think a largescale reorganization is a good idea,
it'll just stall and nothing will happen.

On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>> I agree with all that. But the subject line is specifically about
>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
>> then that is noted. But if we were to move it, we can think about a
>> good place to move it to.
>
> I think it's probably worth moving pg_xlog, because the benefit also
> includes preventing a few users from shooting themselves somewhere
> vital. That's imo much less the case for some of the other moves. But I
> still don't think think a largescale reorganization is a good idea,
> it'll just stall and nothing will happen.

OK, so let's focus only on the renaming mentioned in $subject. So far
as I can see on this thread, here are the opinions of people who
clearly gave one:
- Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
David's input), Magnus
- Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
- Do nothing: Simon (add a README), Tom, Peter E

As far as I can see, there is a consensus to not rename pg_xlog to
pg_journal and avoid using a third meaning, but instead use pg_wal. I
guess that now the other renaming would be pg_clog -> pg_xact. Other
opinions? Forgot you here?
--
Michael

On 27/08/16 20:33, Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that. But the subject line is specifically about
>>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted. But if we were to move it, we can think about a
>>> good place to move it to.
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves. But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
>
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal. I
> guess that now the other renaming would be pg_clog -> pg_xact. Other
> opinions? Forgot you here?

I think if there are going to be things in pg that break software - for
good reasons, like making future usage easier at the cost an initial
sharp pain - then to do so in version '10.0.0' is very appropriate! IMHO

And better to do so in 10.0.0 (especially if closely related), rather
than 10.1.0 (or whatever the next version after that is named). So, if
other things might cause breakages, do so IN 10.0.0 - rather than hold
back - assuming that there won't be hundreds or more major breakages!!!

Cheers,
Gavin

Michael Paquier <michael(dot)paquier(at)gmail(dot)com> writes:
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

Hm, if you read me as voting against renaming pg_xlog, that wasn't
the conclusion I meant to convey. I'm against moving/renaming the
configuration files, because I think that will break a lot of users'
scripts and habits without really buying much. But I'm for consolidating
all the files that should not be copied by backup tools into one
subdirectory, and I think that while we're doing that it would be sensible
to rename pg_xlog and pg_clog to something that doesn't sound like it's
scratch data. I'm on the fence about whether pg_logical ought to get
renamed.

> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal.

Yeah, +1 for pg_wal, we do not need yet another name for that.

> I guess that now the other renaming would be pg_clog -> pg_xact.

We already have pg_subtrans, so it seems like pg_trans is an obvious
suggestion. I'm not sure whether the other precedent of pg_multixact
is a stronger one than that.

regards, tom lane

Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
> > On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
> >> I agree with all that. But the subject line is specifically about
> >> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
> >> then that is noted. But if we were to move it, we can think about a
> >> good place to move it to.
> >
> > I think it's probably worth moving pg_xlog, because the benefit also
> > includes preventing a few users from shooting themselves somewhere
> > vital. That's imo much less the case for some of the other moves. But I
> > still don't think think a largescale reorganization is a good idea,
> > it'll just stall and nothing will happen.
>
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.

I think that would make sense if we were to relocate *everything* under
PGDATA into some FHS-like subdirectory structure. But I'm against moving
the config files for previously-stated reasons, and I doubt it makes sense
to adopt an FHS-like structure only in part.

regards, tom lane

On 8/26/16 4:08 PM, Andres Freund wrote:
> Splitting of ephemeral data seems to have a benefit, the rest seems more
> like rather noisy busywork to me.

People accidentally blowing away pg_clog or pg_xlog is a pretty common
occurrence, and I don't think there's all that many tools that reference
them. I think it's well worth renaming them.

I actually like the idea of re-organizing everything too. It would be
easy to include a tool that puts symlinks in place to all the original
locations if you need that.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461

On 29 Aug 2016 12:10 PM, "Jim Nasby" <Jim(dot)Nasby(at)bluetreble(dot)com> wrote:
>
> On 8/26/16 4:08 PM, Andres Freund wrote:
>>
>> Splitting of ephemeral data seems to have a benefit, the rest seems more
>> like rather noisy busywork to me.
>
>
> People accidentally blowing away pg_clog or pg_xlog is a pretty common
occurrence, and I don't think there's all that many tools that reference
them. I think it's well worth renaming them.

Yeah. I've seen it in BIG production users who really should have known
better.

People won't see a README in amongst 5000 xlog segments while freaking out
about the sever being down.

I don't care if it comes as part of some greater reorg or not but I'll be
really annoyed if scope creep lands up killing the original proposal to
just rename these dirs. I think that a simple rename should be done first.
Then if some greater reorg is to be done it can be done shortly after. The
only people that'll upset are folks tracking early 10.0 dev and they'll be
aware it's coming.

On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
<craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
> I don't care if it comes as part of some greater reorg or not but I'll be
> really annoyed if scope creep lands up killing the original proposal to just
> rename these dirs. I think that a simple rename should be done first. Then
> if some greater reorg is to be done it can be done shortly after. The only
> people that'll upset are folks tracking early 10.0 dev and they'll be aware
> it's coming.

Okay, so let's do it. Attached are two patches:
- 0001 renames pg_clog to pg_trans. I have let clog.c with its current
name, as well as its structures. That's the mechanical patch, the ony
interesting part being in pg_upgrade.
- 0002 renames pg_xlog to pg_wal.
--
Michael

On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com> wrote:

> On 8/26/16 4:08 PM, Andres Freund wrote:
>
>> Splitting of ephemeral data seems to have a benefit, the rest seems more
>> like rather noisy busywork to me.
>>
>
> People accidentally blowing away pg_clog or pg_xlog is a pretty common
> occurrence, and I don't think there's all that many tools that reference
> them. I think it's well worth renaming them.
>

Pretty sure every single backup tool or script out there is referencing
pg_xlog. If it's not, then it's broken...

pg_clog is a different story of course.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 29 August 2016 at 14:30, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>> I don't care if it comes as part of some greater reorg or not but I'll be
>> really annoyed if scope creep lands up killing the original proposal to just
>> rename these dirs. I think that a simple rename should be done first. Then
>> if some greater reorg is to be done it can be done shortly after. The only
>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>> it's coming.
>
> Okay, so let's do it. Attached are two patches:
> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
> name, as well as its structures. That's the mechanical patch, the ony
> interesting part being in pg_upgrade.
> - 0002 renames pg_xlog to pg_wal.

Is there any expectation that a 10.0 pg_basebackup should work on a
9.x server, or fail to work gracefully? There doesn't look to be any
version specific handling of the rename there.

Otherwise looks good.

Doesn't upset 'make check' or the TAP recovery suite.

Works:

src/test/regress/tmp_check/data $ ls
base pg_commit_ts pg_hba.conf pg_logical pg_notify
pg_serial pg_stat pg_subtrans pg_trans PG_VERSION
postgresql.auto.conf postmaster.opts
global pg_dynshmem pg_ident.conf pg_multixact pg_replslot
pg_snapshots pg_stat_tmp pg_tblspc pg_twophase pg_wal
postgresql.conf

It leaves pg_xlogfilename etc with original names as it IMO should.
There's no need to break more than we have to and it's still the xlog.

The documentation on backup/restore might benefit from a note saying
that pg_wal was named pg_xlog prior to 10.0 so tools intended to work
on older versions should check PG_VERSION. Though in practice most
people who write new tools will target 10.0+ and people maintaining
older tools will know, so it's not a big deal.

I don't know if the renaming in XLogFileRead of XLOG_FROM_PG_XLOG =>
XLOG_FROM_PG_WAL is really necessary, but tend to think it's good
since that define explicitly refers to the directory name, not
transaction logs in general.

The patch does not update the translations. I wonder if it's worth
doing so to save translators the hassle by sed'ing the following
lines:

- errhint("The database server
will regularly poll the pg_xlog subdirectory to check for files placed
there.")));
+ errhint("The database server
will regularly poll the pg_wal subdirectory to check for files placed
there.")));

- (errmsg("could not open directory
\"%s\": %m", "pg_xlog")));
+ (errmsg("could not open directory
\"%s\": %m", "pg_wal")));

with

sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po

?

src/backend/access/transam/README should probably have a note about the rename.

Looks like changes in pg_upgrade for clog are a bit more than the
mechanical changes elsewhere, but seem sensible to me. I haven't yet
done a test pg_upgrade run.

--
--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 27/08/16 18:50, Tom Lane wrote:
> Michael Paquier <michael(dot)paquier(at)gmail(dot)com> writes:
>> OK, so let's focus only on the renaming mentioned in $subject. So far
>> as I can see on this thread, here are the opinions of people who
>> clearly gave one:
>> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
>> David's input), Magnus
>> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
>> - Do nothing: Simon (add a README), Tom, Peter E
>
> I'm against moving/renaming the
> configuration files, because I think that will break a lot of users'
> scripts and habits without really buying much.

I don't agree with this part, mainly because there is significant number
of installations (everything that uses debian/ubuntu scripts) that
already don't have configuration files inside the data directory.

On the other matters:
+1 for renaming pg_xlog to pg_wal and pg_clog to pg_xact/trans (don't
care really which one)

And +1 for renaming pg_logical to something more reasonable.

--
Petr Jelinek http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
<craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
> On 29 August 2016 at 14:30, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>>> I don't care if it comes as part of some greater reorg or not but I'll be
>>> really annoyed if scope creep lands up killing the original proposal to just
>>> rename these dirs. I think that a simple rename should be done first. Then
>>> if some greater reorg is to be done it can be done shortly after. The only
>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>>> it's coming.
>>
>> Okay, so let's do it. Attached are two patches:
>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>> name, as well as its structures. That's the mechanical patch, the ony
>> interesting part being in pg_upgrade.
>> - 0002 renames pg_xlog to pg_wal.
>
> Is there any expectation that a 10.0 pg_basebackup should work on a
> 9.x server, or fail to work gracefully? There doesn't look to be any
> version specific handling of the rename there.

Oops. Per the docs:
pg_basebackup works with servers of the same or an older major
version, down to 9.1. However, WAL streaming mode (-X stream) only
works with server version 9.3 and later, and tar format mode
(--format=tar) of the current version only works with server version
9.5 or later.

So we need to do a bit better than what the patch proposes, but that's
actually just tweaking the things with pg_xlog/pg_wal depending on the
version of the target server.

> The patch does not update the translations. I wonder if it's worth
> doing so to save translators the hassle by sed'ing the following
> lines:
> sed -i 's/\<pg_xlog\>/pg_wal/g' src/backend/po/*.po
> ?

Yes I was wondering about that... But concluded that normally the
translation updates would just do it. It is not complicated to update
if need be anyway.

> src/backend/access/transam/README should probably have a note about the rename.

Good idea.

> Looks like changes in pg_upgrade for clog are a bit more than the
> mechanical changes elsewhere, but seem sensible to me. I haven't yet
> done a test pg_upgrade run.

I have tested that FWIW using 10devel -> 10devel, 9.5 -> 10devel, 9.4
-> 10devel. There are versions of 10devel using pg_xlog and others
pg_wal if this patch is introduced. I am aware of the fact that
pg_upgrade supports upgrades for the same major version though it
seems to me that we would not want to support this scenario, which is
why it would be good to get this patch at the beginning of the dev
cycle.
--
Michael

On 29 August 2016 at 20:28, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
> On Mon, Aug 29, 2016 at 5:28 PM, Craig Ringer
> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>> On 29 August 2016 at 14:30, Michael Paquier <michael(dot)paquier(at)gmail(dot)com> wrote:
>>> On Mon, Aug 29, 2016 at 2:36 PM, Craig Ringer
>>> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>>>> I don't care if it comes as part of some greater reorg or not but I'll be
>>>> really annoyed if scope creep lands up killing the original proposal to just
>>>> rename these dirs. I think that a simple rename should be done first. Then
>>>> if some greater reorg is to be done it can be done shortly after. The only
>>>> people that'll upset are folks tracking early 10.0 dev and they'll be aware
>>>> it's coming.
>>>
>>> Okay, so let's do it. Attached are two patches:
>>> - 0001 renames pg_clog to pg_trans. I have let clog.c with its current
>>> name, as well as its structures. That's the mechanical patch, the ony
>>> interesting part being in pg_upgrade.
>>> - 0002 renames pg_xlog to pg_wal.
>>
>> Is there any expectation that a 10.0 pg_basebackup should work on a
>> 9.x server, or fail to work gracefully? There doesn't look to be any
>> version specific handling of the rename there.
>
> Oops. Per the docs:
> pg_basebackup works with servers of the same or an older major
> version, down to 9.1. However, WAL streaming mode (-X stream) only
> works with server version 9.3 and later, and tar format mode
> (--format=tar) of the current version only works with server version
> 9.5 or later.
>
> So we need to do a bit better than what the patch proposes, but that's
> actually just tweaking the things with pg_xlog/pg_wal depending on the
> version of the target server.

Cool. I'll mark as waiting on author pending that.

It'll be good to get this footgun put away.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Craig Ringer wrote:

> People won't see a README in amongst 5000 xlog segments while
> freaking out about the sever being down.

Maybe they're more likely to google "pg_xlog".
BTW, renaming it will not help with respect to that, as
there's a pretty good corpus on knowledge linked to that
particular keyword.

The first google results of "pg_xlog" are, for me:

- Solving pg_xlog out of disk space problem on Postgres
- PostgreSQL: Documentation: 9.1: WAL Internals
- Why is my pg_xlog directory so huge? - PostgreSQL
- Database Soup: Don't delete pg_xlog
...

That's spot-on. For each person deleting stuff in pg_xlog and then
regretting it, how many look it up in the above results and avoid
making the mistake? Will they find comparable results if the
directory is "wal" ?

Aside from that, we might also question how much of the excuse
"pg_xlog looked like it was just deletable logs" is a lie made up
after the fact, because anybody wrecking a database is not against
deflecting a bit of the blame to the software, that's human.

The truth being that they took the gamble that postgres
will somehow recover from the deletion, at the risk of possibly
loosing the latest transactions. That doesn't necessarily look
so bad when current transactions are failing anyway for lack of disk
space, users are yelling at you, and you're frantically searching for
anything to delete in the FS to come back online quickly.
Personally I'm quite skeptical of the *name* of the directory
playing that much of a role in this scenario.

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

On 08/27/2016 11:11 AM, Tom Lane wrote:
> Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
>> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
>> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
>
> I think that would make sense if we were to relocate *everything* under
> PGDATA into some FHS-like subdirectory structure. But I'm against moving
> the config files for previously-stated reasons, and I doubt it makes sense
> to adopt an FHS-like structure only in part.

I think that is a very reasonable suggestion.

Also as a note to the idea that we make break things for external user
space; the next version being v10 is the exact time to do that.

JD

>
> regards, tom lane
>
>

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

On 08/29/2016 12:04 AM, Magnus Hagander wrote:
> On Mon, Aug 29, 2016 at 2:45 AM, Jim Nasby <Jim(dot)Nasby(at)bluetreble(dot)com
> <mailto:Jim(dot)Nasby(at)bluetreble(dot)com>> wrote:
>
> On 8/26/16 4:08 PM, Andres Freund wrote:
>
> Splitting of ephemeral data seems to have a benefit, the rest
> seems more
> like rather noisy busywork to me.
>
>
> People accidentally blowing away pg_clog or pg_xlog is a pretty
> common occurrence, and I don't think there's all that many tools
> that reference them. I think it's well worth renaming them.
>
>
> Pretty sure every single backup tool or script out there is referencing
> pg_xlog. If it's not, then it's broken...

No, not really. Consider a filesytem backup using archiving and base
backups. It doesn't care one lick about pg_xlog. And I guarantee you
that there are tons of people running a backup like that considering the
same script would work all the way back to 8.2 (.1?).

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

On 08/29/2016 06:42 AM, Daniel Verite wrote:
>
> Aside from that, we might also question how much of the excuse
> "pg_xlog looked like it was just deletable logs" is a lie made up
> after the fact, because anybody wrecking a database is not against
> deflecting a bit of the blame to the software, that's human.
>
> The truth being that they took the gamble that postgres
> will somehow recover from the deletion, at the risk of possibly
> loosing the latest transactions. That doesn't necessarily look
> so bad when current transactions are failing anyway for lack of disk
> space, users are yelling at you, and you're frantically searching for
> anything to delete in the FS to come back online quickly.
> Personally I'm quite skeptical of the *name* of the directory
> playing that much of a role in this scenario.

Oh it makes perfect sense.

User who isn't a postgres/database person installs X software. X
software uses PostgreSQL and you have read on the intertubes about how
Postgres is awesome. So you install it, get everything up and running
from the README.md on Github. You walk away.

A year later, after it becomes crucial to whatever it is you do the
system crashes. You have no idea what is going on, you just set this up
on some recycled server or VM. You log in, see that all the space and
you find that you are using a ton of disk space. You look around for
anything you can delete. You find a directory called pg_xlog, it says
log, junior ignorant, don't want to be a sysadmin 101 says, "delete logs".

Boom, crushed server. Let us not forget that by far the number of
PostgreSQL users we have, have never done ANYTHING with postgres except
make it so something like Drupal can connect to it.

JD

>
> Best regards,
>

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Also as a note to the idea that we make break things for external user
> space; the next version being v10 is the exact time to do that.

Let's please drop this meme that "v10 is a great time to break things".
We don't want this to be any worse than any other major-version upgrade.
If we throw thirty different major incompatibilities in at once, we're
going to be hearing about how painful it was for the next decade, even if
any one of them individually would have been manageable. Or, if we make
the pain factor too high, users will simply not upgrade, and we'll be
faced with demands that we support 9.6 forever.

regards, tom lane

On 8/27/16 4:33 AM, Michael Paquier wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that. But the subject line is specifically about
>>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted. But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves. But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
>
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E
>
> As far as I can see, there is a consensus to not rename pg_xlog to
> pg_journal and avoid using a third meaning, but instead use pg_wal. I
> guess that now the other renaming would be pg_clog -> pg_xact. Other
> opinions? Forgot you here?

I'm definitely +1 for a hard break (internal links are a pain). Ideally
I'd would like to see:

pg_xlog -> pg_wal
pg_clog -> pg_xact
pg_logical -> pg_replgcl

pg_logical is a special case because it contains both ephemeral and
persistent files. I'd like to see the temporary files in
pg_tmp/pg_replgcl (or whatever it gets renamed to) but that means that
pg_tmp must be on the same device to get atomic renames. It would be
enough if pg_replgcl had a pgsql_tmp subdirectory so temp files are
excluded from backups with the current logic.

I'm also in favor of leaving configuration files where they are because
these are the files that are most likely to be checked/manipulated in
various user scripts (other than pg_xlog) of course.

As Stephen mentioned I would like to see purely ephemeral data moved
into its own directory. Maybe $PGDATA/pg_tmp?

pg_subtrans -> pg_tmp/pg_subxact
pg_stat_tamp -> pg_tmp/pg_stat
pg_dynshmem -> pg_tmp/pg_dynshmem (or pg_shmem?)
pg_notify -> pg_tmp/pg_notify
pg_replslot -> pg_tmp/pg_repslot
pg_serial -> pg_tmp/pg_serial
pg_snapshots -> pg_tmp/pg_snapshot

It would be helpful if we had consistent temp directory naming
everywhere so it's clear what to exclude when it is not practical to put
files in the root pg_tmp. As such I would rename pgsql_tmp to pg_tmp in
base and pg_tblspc/*/*/.

--
-David
david(at)pgmasters(dot)net

Hi,

On 2016-08-29 11:18:38 -0400, David Steele wrote:
> pg_logical -> pg_replgcl

That seems considerably worse.

> pg_replslot -> pg_tmp/pg_repslot

That's most certainly not ephemeral. Just because something isn't
generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Greetings,

Andres Freund

Andres Freund <andres(at)anarazel(dot)de> writes:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_replslot -> pg_tmp/pg_repslot

> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Do we need to account for both of those concepts explicitly?

The original point here was to simplify figuring out what needed to be
copied in base backups. I'm not sure whether we really need a marker
for what's going to be flushed at restart.

regards, tom lane

On 8/29/16 11:35 AM, Andres Freund wrote:
> On 2016-08-29 11:18:38 -0400, David Steele wrote:
>> pg_logical -> pg_replgcl
>
> That seems considerably worse.

Fair enough. I was just trying to throw something out there to get rid
of the "log" in logical.

>> pg_replslot -> pg_tmp/pg_repslot
>
> That's most certainly not ephemeral. Just because something isn't
> generally appropriate on a standby, doesn't, by far, mean it's ephemeral.

Yes, ephemeral was a poor choice of words. I really meant "can be
excluded from backups".

--
-David
david(at)pgmasters(dot)net

On 2016-08-29 12:07:51 -0400, David Steele wrote:
> >> pg_replslot -> pg_tmp/pg_repslot
> >
> > That's most certainly not ephemeral. Just because something isn't
> > generally appropriate on a standby, doesn't, by far, mean it's ephemeral.
>
> Yes, ephemeral was a poor choice of words. I really meant "can be
> excluded from backups".

Then it most certainly can't be called pg_tmp, that'll just invite
people to rm -rf it. And then they'll be screwed.

On 08/29/2016 08:07 AM, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> Also as a note to the idea that we make break things for external user
>> space; the next version being v10 is the exact time to do that.
>
> Let's please drop this meme that "v10 is a great time to break things".
> We don't want this to be any worse than any other major-version upgrade.

The moment you break backward compatibility, it will be worse. We are
talking about breaking backward compatibility. So let's just accept it
as it is, there is a mentality about a major jump. A major jump
(9.6->10) is *the* perfect time to make world changing, changes.

> If we throw thirty different major incompatibilities in at once, we're
> going to be hearing about how painful it was for the next decade, even if
> any one of them individually would have been manageable. Or, if we make
> the pain factor too high, users will simply not upgrade, and we'll be
> faced with demands that we support 9.6 forever.

Whiners always find a reason to whine.

Let's be on two feet here. I am not saying we should jump to using json
notation for our next release. I am simply stating that any largish
(even if it is a small patch) changes to expected behavior should be
done with care. We have a window because no matter how much you yell, I
yell, Magnus yells, or anybody else yells; the telling story will be,
"10.0 is a huge jump from 9.6". Most people *WOULD NOT CARE* if we only
changed one thing. They care that we jumped 4 releases. That type of
communication implies BIG CHANGES.

Whether you like it or not. Whether I like it or not.

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

On Sat, Aug 27, 2016 at 5:33 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> On Sat, Aug 27, 2016 at 6:34 AM, Andres Freund <andres(at)anarazel(dot)de> wrote:
>> On 2016-08-26 17:31:14 -0400, Peter Eisentraut wrote:
>>> I agree with all that. But the subject line is specifically about
>>> moving pg_xlog. So if your opinion is that we shouldn't move pg_xlog,
>>> then that is noted. But if we were to move it, we can think about a
>>> good place to move it to.
>>
>> I think it's probably worth moving pg_xlog, because the benefit also
>> includes preventing a few users from shooting themselves somewhere
>> vital. That's imo much less the case for some of the other moves. But I
>> still don't think think a largescale reorganization is a good idea,
>> it'll just stall and nothing will happen.
>
> OK, so let's focus only on the renaming mentioned in $subject. So far
> as I can see on this thread, here are the opinions of people who
> clearly gave one:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus
> - Rename them, hard break not OK: Fujii-san (perhaps do nothing?)
> - Do nothing: Simon (add a README), Tom, Peter E

I vote for "do nothing".
First of all, I can't have much hope for that renaming the directories really
prevents "careless" users from wrongly deleting the important files.

Regards,

--
Fujii Masao

On Sat, Aug 27, 2016 at 2:03 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> - Rename them, hard break is OK: Michael P, Bruce, Stephen (depends on
> David's input), Magnus

I'm in favor of this. But I don't like Peter's proposal to use a more
complicated structure. As for the new names, how about pg_wal and
pg_xact? I don't think "pg_trans" is as good; is it transactional or
transient or transport or transitive or what?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Joshua D. Drake wrote:

> You log in, see that all the space and you find that you are using a
> ton of disk space. You look around for anything you can delete. You
> find a directory called pg_xlog, it says log, junior ignorant, don't
> want to be a sysadmin 101 says, "delete logs".

Yes, it happens. I don't deny the problem, but I'm wondering
about the wishful thinking we're possibly falling into here
concerning the solution.

Let's imagine that pg_xlog is named wal instead.
How does that help our user with the disk space problem?
Does that point to a path of resolution? I don't see it.
What do we think that user's next move will be?
After all, WAL means Write Ahead *Log*.

On the other hand, by decommissioning pg_xlog as a name,
that makes it obsolete in all presentations, tutorials, docs
that refer to that directory, and there are many of them.
There are years of confusion ahead with questions like
"where is that pg_xlog directory that I'm supposed to
monitor, or move into a different partition, etc...",
for a benefit that remains to be seen.

Best regards,
--
Daniel Vérité
PostgreSQL-powered mailer: http://www.manitou-mail.org
Twitter: @DanielVerite

On 08/29/2016 10:00 AM, Daniel Verite wrote:

> Let's imagine that pg_xlog is named wal instead.
> How does that help our user with the disk space problem?
> Does that point to a path of resolution? I don't see it.
> What do we think that user's next move will be?
> After all, WAL means Write Ahead *Log*.

If, they look it up (which they would likely have to) they will see it
isn't removable. :D That is the point I am making. If it is called xlog
the brain says "logs". If it says wal the brain says, "What is wal?"

At that point they have to look it up and then if they still delete it;
well that is what emergency rates are for :P

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.
Unless otherwise stated, opinions are my own.

On Fri, Aug 26, 2016 at 05:14:36PM +0200, Magnus Hagander wrote:
> On Aug 26, 2016 5:13 PM, "Joshua D. Drake" <jd(at)commandprompt(dot)com> wrote:
> >
> > On 08/25/2016 07:39 PM, Michael Paquier wrote:
> >>
> >> Hi all,
> >>
> >> I am relaunching $subject as 10 development will begin soon. As far as
> >> I know, there is agreement that we can do something here. Among the
> >> different proposals I have found:
> >> - pg_clog renamed to pg_commit_status, pg_xact or pg_commit
> >> - pg_xlog renamed to pg_xjournal, pg_wal or pg_journal
> >
> >
> > I think the use of the pg_ prefix is redundant. Just a directory called: wal
> will do (for example).
> >
> > In reference to pg_xlog specifically, it should be wal. It is the Write Ahead
> Log, not the Journal (although I recognize they can be synonymous). All the
> documentation talks about Write Ahead Log.
> >
>
> Yes, let's avoid inventing a *third* name for it, please. It's already bad
> enough that we have both wal and xlog.

As far as removing 'pg_', consider that many users place pg_xlog on a
different device, so using "wal" would not be clear it was related to
Postgres. Perhaps we can have initdb --xlogdir use pg_wal, and maybe
document this suggestion.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

On 8/27/16 1:11 PM, Tom Lane wrote:
> Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
>> I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
>> to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
>
> I think that would make sense if we were to relocate *everything* under
> PGDATA into some FHS-like subdirectory structure. But I'm against moving
> the config files for previously-stated reasons, and I doubt it makes sense
> to adopt an FHS-like structure only in part.

What if we left symlinks for the config files? Or perhaps even better,
provide a tool that will create them for people that actually need them.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461

On 2016-08-29 19:27:29 -0500, Jim Nasby wrote:
> On 8/27/16 1:11 PM, Tom Lane wrote:
> > Alvaro Herrera <alvherre(at)2ndquadrant(dot)com> writes:
> > > I'm for renaming too, but I'd go with Peter E's suggestion: move pg_xlog
> > > to something like $PGDATA/var/wal or $PGDATA/srv/wal or something like that.
> >
> > I think that would make sense if we were to relocate *everything* under
> > PGDATA into some FHS-like subdirectory structure. But I'm against moving
> > the config files for previously-stated reasons, and I doubt it makes sense
> > to adopt an FHS-like structure only in part.
>
> What if we left symlinks for the config files? Or perhaps even better,
> provide a tool that will create them for people that actually need
> them.

See the thread around
http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de

On 8/29/16 12:54 PM, Robert Haas wrote:
> As for the new names, how about pg_wal and
> pg_xact? I don't think "pg_trans" is as good; is it transactional or
> transient or transport or transitive or what?

pg_transaction_status? (or pg_xact_status if you want to keep it short)

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On 8/28/16 8:45 PM, Jim Nasby wrote:
> People accidentally blowing away pg_clog or pg_xlog is a pretty common
> occurrence, and I don't think there's all that many tools that reference
> them. I think it's well worth renaming them.

I think a related problem is that the default log directory is called
"pg_log", which is very similar to those other two. There is no quick
way to tell their meaning apart.

I would consider changing the default from "pg_log" to "log". Then we'd
also be at the point where we can say, everything starting with "pg_" is
internal, don't touch it.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
<craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
> Cool. I'll mark as waiting on author pending that.
>
> It'll be good to get this footgun put away.

OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
stack as that's the one making no discussion it seems: a lot of people
like pg_wal. I have added as well handling for the renaming in
pg_basebackup by using PQserverVersion. One thing to note is that a
connection needs to be made to the target server *before* creating the
soft link of pg_xlog/pg_wal because we need to know the version of the
target server. pg_upgrade is handled as well. And that's all in 0001.

Patch 0002 does pg_clog -> pg_trans, "trans" standing for
"transaction". Switching to pg_trans_status or pg_xact_status is not
that complicated to change anyway...
--
Michael

On 8/29/16 7:34 PM, Andres Freund wrote:
>> What if we left symlinks for the config files? Or perhaps even better,
>> > provide a tool that will create them for people that actually need
>> > them.
> See the thread around
> http://archives.postgresql.org/message-id/20160826104446.n3cif4m7modslkrs%40msg.df7cb.de

Right, but I was referring only to the config files. AFAIK it should be
safe for those to be symlinks.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461

On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>> Cool. I'll mark as waiting on author pending that.
>>
>> It'll be good to get this footgun put away.
>
> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
> stack as that's the one making no discussion it seems: a lot of people
> like pg_wal. I have added as well handling for the renaming in
> pg_basebackup by using PQserverVersion. One thing to note is that a
> connection needs to be made to the target server *before* creating the
> soft link of pg_xlog/pg_wal because we need to know the version of the
> target server. pg_upgrade is handled as well. And that's all in 0001.
>
> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
> "transaction". Switching to pg_trans_status or pg_xact_status is not
> that complicated to change anyway...

Any input to offer for those patches? If there is nothing happening, I
guess that the best move is just to move it to next CF. At least I can
see that the flow would be to get those renamings done.
--
Michael

On Thu, Sep 29, 2016 at 9:17 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> On Tue, Aug 30, 2016 at 11:04 AM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
>> On Mon, Aug 29, 2016 at 9:39 PM, Craig Ringer
>> <craig(dot)ringer(at)2ndquadrant(dot)com> wrote:
>>> Cool. I'll mark as waiting on author pending that.
>>>
>>> It'll be good to get this footgun put away.
>>
>> OK, so done. I have put the renaming of pg_xlog to pg_wal on top patch
>> stack as that's the one making no discussion it seems: a lot of people
>> like pg_wal. I have added as well handling for the renaming in
>> pg_basebackup by using PQserverVersion. One thing to note is that a
>> connection needs to be made to the target server *before* creating the
>> soft link of pg_xlog/pg_wal because we need to know the version of the
>> target server. pg_upgrade is handled as well. And that's all in 0001.
>>
>> Patch 0002 does pg_clog -> pg_trans, "trans" standing for
>> "transaction". Switching to pg_trans_status or pg_xact_status is not
>> that complicated to change anyway...
>
> Any input to offer for those patches? If there is nothing happening, I
> guess that the best move is just to move it to next CF. At least I can
> see that the flow would be to get those renamings done.

+1 for pg_xlog -> pg_wal.

Of the existing suggestions, would like to add my vote for the
following renames, matching pg_multixact:

pg_clog -> pg_xact
pg_subtrans -> pg_subxact

If longer names are on the table, I would consider expanding all three of those:

pg_clog -> pg_transaction
pg_subtrans -> pg_subtransaction
pg_multixact -> pg_multitransaction

They sound eminently non-deletable.

--
Thomas Munro
http://www.enterprisedb.com

On Fri, Sep 30, 2016 at 1:32 PM, Thomas Munro
<thomas(dot)munro(at)enterprisedb(dot)com> wrote:
> +1 for pg_xlog -> pg_wal.

This is clearly the winner here and...

> Of the existing suggestions, would like to add my vote for the
> following renames, matching pg_multixact:
>
> pg_clog -> pg_xact
> pg_subtrans -> pg_subxact
>
> If longer names are on the table, I would consider expanding all three of those:
>
> pg_clog -> pg_transaction

This one sounds like the consensus by what I am reading upthread.

> pg_subtrans -> pg_subtransaction
> pg_multixact -> pg_multitransaction
>
> They sound eminently non-deletable.

Well the original complain was that users tend to remove the entries
that include *log. I am not sure if it is worth worrying about doing
that much.

Anyway, those patches have rotten because of the commits of yesterday
for initdb and basebackup.c (stuff I wrote and/or reviewed actually).
So attached is a rebased set. 0001 does the work for pg_wal, 0002 for
pg_transaction.

As there have been no reviews at code level, I am moving that to the next CF.
--
Michael

On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> As there have been no reviews at code level, I am moving that to the next CF.

Code review of 0001:

I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
I submit that either PQserverVersion(conn) >= 100000 or
PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
I vote for the first style. So in pg_basebackup.c I'd do:

#define MINIMUM_VERSION_FOR_PG_WAL 100000
...
int version = PQserverVersion(conn);
...
if (version >= MINIMUM_VERSION_FOR_PG_WAL)
/* do whatever */

Also, for this sort of thing:

+ if (!conn)
+ /* Error message already written in GetConnection() */
+ exit(1);

...because of the comment, I'd add braces around this.

Tom changed the error-reporting framework for pg_upgrade in
f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
rebasing over that. I haven't checked what the new conventions are,
but obviously you'll want to try to be consistent with them.

Other than those minor nitpicks, I think this looks OK. I'm not sure
we have consensus on 0002, but 0001 seems to be popular with far more
people than not.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Re: Michael Paquier 2016-09-30 <CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ(at)mail(dot)gmail(dot)com>

"pg_trans" is used in two places:

> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned

> - /* copy old commit logs to new data dir */
> - copy_subdir_files("pg_clog");
> + /*
> + * Copy old commit logs to new data dir. pg_clog has been renamed to
> + * pg_trans in post-10 clusters.

(Fwiw, I'd prefer something shorter than the imho clumsy
pg_transaction. "pg_xact" sounded very fine for me, it also combines
nicely with pg_subxact and the existing pg_multixact.)

Christoph

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style.

+1, that's the way most existing tests of this sort are written.

(Right at the moment, there's kind of a lot of zeroes there, but
once we get to version 11 it'll be less bad.)

regards, tom lane

On Mon, Oct 3, 2016 at 10:00 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Sep 30, 2016 at 1:45 AM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
>> As there have been no reviews at code level, I am moving that to the next CF.
>
> Code review of 0001:
>
> I think the tests for PQserverVersion(conn) / 100 >= 1000 are strange.
> I submit that either PQserverVersion(conn) >= 100000 or
> PQserverVersion(conn) / 10000 >= 10 is an easier-to-understand test.
> I vote for the first style. So in pg_basebackup.c I'd do:
>
> #define MINIMUM_VERSION_FOR_PG_WAL 100000
> ...
> int version = PQserverVersion(conn);
> ...
> if (version >= MINIMUM_VERSION_FOR_PG_WAL)
> /* do whatever */

I have adopted the existing style of BaseBackup() in last versions,
but I don't mind as well doing the way you suggest, without changing
the stuff in BaseBackup() though, we'd still want a pg_basebackup
compiled with 10.1 to work with 10.2 or newer minor versions of 10.

> Also, for this sort of thing:
>
> + if (!conn)
> + /* Error message already written in GetConnection() */
> + exit(1);
>
> ...because of the comment, I'd add braces around this.

Fixed. That was done without brackets on HEAD, but you are right that
is not PG-like. (There are other places in src/bin/pg_basebackup doing
the same thing, but I have not modified them to not create more diff
noise).

> Tom changed the error-reporting framework for pg_upgrade in
> f002ed2b8e45286fe73a36e119cba2016ea0de19, so you'll need to do some
> rebasing over that. I haven't checked what the new conventions are,
> but obviously you'll want to try to be consistent with them.

strerror(errno) needs to be used instead of getErrorText() in the
context of this patch.

> Other than those minor nitpicks, I think this looks OK. I'm not sure
> we have consensus on 0002, but 0001 seems to be popular with far more
> people than not.

Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
pg_clog...

Re: Michael Paquier 2016-09-30
<CAB7nPqR=SZNo_=B1ukwCiNn7aWDcw_dV0z-LG4ys9WF1N4a=uQ(at)mail(dot)gmail(dot)com>
> "pg_trans" is used in two places:
> -pg_clog records the commit status for each transaction that has been assigned
> +pg_trans records the commit status for each transaction that has been assigned

> - /* copy old commit logs to new data dir */
> - copy_subdir_files("pg_clog");
> + /*
> + * Copy old commit logs to new data dir. pg_clog has been renamed to
> + * pg_trans in post-10 clusters.

> (Fwiw, I'd prefer something shorter than the imho clumsy
> pg_transaction. "pg_xact" sounded very fine for me, it also combines
> nicely with pg_subxact and the existing pg_multixact.)

Thanks, fixed those.

So this is still open for votes. Here are the candidates and who
voiced for what:
- pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
- pg_xact: David S, Robert
- pg_trans: Tom
- pg_transaction_status: Peter E.

Attached is a new patch set. I have re-done tests with pg_upgrade with
9.5 -> HEAD and 9.6 -> HEAD, and pg_basebackup for symlink handling.
Actually in the latter case I have found a bug in 0001 introduced by
bc34223b because fsync_pgdata() is not aware of the version of the
server it is trying to fsync(). I think that the better way to address
that is to extend fsync_pgdata in file_utils.c with the server version
to fsync() like attached.
--
Michael

On 10/4/16 1:48 AM, Michael Paquier wrote:

> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

Christoph also prefers pg_xact [1].

--
-David
david(at)pgmasters(dot)net

[1]
https://www.postgresql.org/message-id/20161003141959.qhvd6roesj4kpgww@msg.df7cb.de

On Tue, Oct 4, 2016 at 1:48 AM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> Yes, pg_wal is fine as new name in replacement of pg_xlog. Now for the
> pg_clog...

Of course the irony here is that "WAL" stands for "Write Ahead Log".
So we're renaming a directly that has "log" in the name (pg_xlog) to a
directory that has an "l" in the name which stands for "log" (pg_wal).
Whee!

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 10/4/16 1:48 AM, Michael Paquier wrote:
> So this is still open for votes. Here are the candidates and who
> voiced for what:
> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> - pg_xact: David S, Robert
> - pg_trans: Tom
> - pg_transaction_status: Peter E.

I think this would all be a lot easier if we just moved all the pg_*
directories one directory down. We did this with "global" many years
ago and it worked out well. We didn't actually have to change the file
names, the top-level directory became cleaner, and no one was messing
around with the files anymore. Adjusting external tools also becomes
easier. There is so much lore out there about clog and xlog; it would
be annoying to break with that.

I can see a reason for not doing that with pg_xlog, because that is a
bit of an external interface, with initdb -X and such, and pg_wal is a
pretty good alternative name. But no one deals with these other
directories directly, so just move them out of the way.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Peter Eisentraut <peter(dot)eisentraut(at)2ndquadrant(dot)com> writes:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
>> So this is still open for votes. Here are the candidates and who
>> voiced for what:
>> - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> - pg_xact: David S, Robert
>> - pg_trans: Tom
>> - pg_transaction_status: Peter E.

> I think this would all be a lot easier if we just moved all the pg_*
> directories one directory down.

The main problem we're trying to fix here is people thinking that
something with "log" in the name contains discardable data. Just
relocating the directory without renaming it won't improve that.

The relocation aspect was meant to address another problem, which
is making it easier to distinguish which parts of the tree should
be copied by tools like pg_basebackup. But that's not what Michael
is asking for votes on.

regards, tom lane

* David Steele (david(at)pgmasters(dot)net) wrote:
> On 10/4/16 1:48 AM, Michael Paquier wrote:
>
> > So this is still open for votes. Here are the candidates and who
> > voiced for what:
> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
> > - pg_xact: David S, Robert
> > - pg_trans: Tom
> > - pg_transaction_status: Peter E.
>
> Christoph also prefers pg_xact [1].

Since we're asking for votes, I also prefer pg_xact over the other
options listed here.

Thanks!

Stephen

On Thu, Oct 13, 2016 at 12:31 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> * David Steele (david(at)pgmasters(dot)net) wrote:
>> On 10/4/16 1:48 AM, Michael Paquier wrote:
>>
>> > So this is still open for votes. Here are the candidates and who
>> > voiced for what:
>> > - pg_transaction: Michael P, Thomas M. => Current 0002 is doing that.
>> > - pg_xact: David S, Robert
>> > - pg_trans: Tom
>> > - pg_transaction_status: Peter E.
>>
>> Christoph also prefers pg_xact [1].
>
> Since we're asking for votes, I also prefer pg_xact over the other
> options listed here.

OK. I can live with that as well. Attached are three patches. The
pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
pg_clog -> pg_xact move. Only one survivor to be chosen among the last
two ones.
--
Michael

On 10/12/16 11:22 AM, Tom Lane wrote:
> The main problem we're trying to fix here is people thinking that
> something with "log" in the name contains discardable data. Just
> relocating the directory without renaming it won't improve that.

I think it would help if we moved it to something like
"internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
out of sight.

We have a tool called pg_xlogdump in the standard installation. initdb
has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
the xlog directory would make this all a bit confusing, unless we're
prepared to rename the programs and options as well.

--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

* Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
> On 10/12/16 11:22 AM, Tom Lane wrote:
> > The main problem we're trying to fix here is people thinking that
> > something with "log" in the name contains discardable data. Just
> > relocating the directory without renaming it won't improve that.
>
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> out of sight.

I disagree that this will materially help with the issue.

> We have a tool called pg_xlogdump in the standard installation. initdb
> has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.

pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
should be terribly concerned about either leaving it named as-is or
renaming it. I agree that we should consider adding alternative names
to the options for initdb and pg_basebackup.

Thanks!

Stephen

Re: Stephen Frost 2016-10-14 <20161014113523(dot)GZ13284(at)tamriel(dot)snowman(dot)net>
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > out of sight.
>
> I disagree that this will materially help with the issue.

And internal/base and internal/global and internal/pg_... because
these shouldn't be touched by the users either.

I don't think this would lead anywhere.

Christoph

* Christoph Berg (myon(at)debian(dot)org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523(dot)GZ13284(at)tamriel(dot)snowman(dot)net>
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > > out of sight.
> >
> > I disagree that this will materially help with the issue.
>
> And internal/base and internal/global and internal/pg_... because
> these shouldn't be touched by the users either.
>
> I don't think this would lead anywhere.

It'd probably be easier to move the things that are *not* PG internal
(eg: config files, et al) *out* of the data directory and into somewhere
sensible, like /etc ...

Oh, wait ...

Thanks!

Stephen

On 10/14/16 9:06 AM, Stephen Frost wrote:
>> > And internal/base and internal/global and internal/pg_... because
>> > these shouldn't be touched by the users either.
>> >
>> > I don't think this would lead anywhere.
> It'd probably be easier to move the things that are *not* PG internal
> (eg: config files, et al) *out* of the data directory and into somewhere
> sensible, like /etc ...

I do think it would be an improvement to segregate things users are
expected to touch (*.conf and pg_log are what come to mind) from
everything else, which could easily be done by moving everything else to
an internal/ directory. I agree that's not much of an improvement for
pg_[cx]log, but we could create internal/ as well as rename some things.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
855-TREBLE2 (855-873-2532) mobile: 512-569-9461

* Jim Nasby (Jim(dot)Nasby(at)BlueTreble(dot)com) wrote:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
> >>> And internal/base and internal/global and internal/pg_... because
> >>> these shouldn't be touched by the users either.
> >>>
> >>> I don't think this would lead anywhere.
> >It'd probably be easier to move the things that are *not* PG internal
> >(eg: config files, et al) *out* of the data directory and into somewhere
> >sensible, like /etc ...
>
> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything
> else to an internal/ directory. I agree that's not much of an
> improvement for pg_[cx]log, but we could create internal/ as well as
> rename some things.

Apologis, I should have made it clear what I was getting at.

The Debian/Ubuntu packaging already does segregate out the *.conf and
pg_log into other locations based on the FHS. On a Debian-based system,
there really shouldn't be anything that the user is expected to touch
under /var/lib (where the PG data directory lives).

Thanks!

Stephen

Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> writes:
> On 10/14/16 9:06 AM, Stephen Frost wrote:
>> It'd probably be easier to move the things that are *not* PG internal
>> (eg: config files, et al) *out* of the data directory and into somewhere
>> sensible, like /etc ...

> I do think it would be an improvement to segregate things users are
> expected to touch (*.conf and pg_log are what come to mind) from
> everything else, which could easily be done by moving everything else to
> an internal/ directory. I agree that's not much of an improvement for
> pg_[cx]log, but we could create internal/ as well as rename some things.

I can't get excited about that at all. It will just get in the way of
the actually useful end goal we had for directory restructuring, which
was to separate data to be copied by pg_basebackup from data not to be
copied by it. And in reality, people who don't understand that the
contents of PGDATA should be treated as "hands off unless documented
otherwise" are not going to be any less likely to shoot themselves in
the foot just because there's a directory named "internal" or
"here_be_dragons" or "keep_out_this_means_you" in the way.

I'd also suggest that an extra level of directory search to get at
database data files is not free.

regards, tom lane

On Fri, Oct 14, 2016 at 9:56 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Jim Nasby <Jim(dot)Nasby(at)BlueTreble(dot)com> writes:
> > On 10/14/16 9:06 AM, Stephen Frost wrote:
> >> It'd probably be easier to move the things that are *not* PG internal
> >> (eg: config files, et al) *out* of the data directory and into somewhere
> >> sensible, like /etc ...
>
> > I do think it would be an improvement to segregate things users are
> > expected to touch (*.conf and pg_log are what come to mind) from
> > everything else, which could easily be done by moving everything else to
> > an internal/ directory. I agree that's not much of an improvement for
> > pg_[cx]log, but we could create internal/ as well as rename some things.
>
> I can't get excited about that at all. It will just get in the way of
> the actually useful end goal we had for directory restructuring, which
> was to separate data to be copied by pg_basebackup from data not to be
> copied by it. And in reality, people who don't understand that the
> contents of PGDATA should be treated as "hands off unless documented
> otherwise" are not going to be any less likely to shoot themselves in
> the foot just because there's a directory named "internal" or
> "here_be_dragons" or "keep_out_this_means_you" in the way.

+1. I think we should move the don't-include-in-backup files to a separate
directory for *technical* reasons.

I don't think moving files that are already supposed to be internal into a
directory called internal is going to help much, an din particular not for
xlog. If we don't rename it, the problem will remain.

And for the "don't touch" part, if anything we should move the config files
into a subdirectory of PGDATA and not the other way around.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:

> * Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
> > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > The main problem we're trying to fix here is people thinking that
> > > something with "log" in the name contains discardable data. Just
> > > relocating the directory without renaming it won't improve that.
> >
> > I think it would help if we moved it to something like
> > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > out of sight.
>
> I disagree that this will materially help with the issue.
>
> > We have a tool called pg_xlogdump in the standard installation. initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
>
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it. I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.
>

Ugh. Changing the names of those options are probably going to break a
*lot* of things, a lot more than changing the names of the directories. Is
it really worth doing that? Especially if we are even more clear that
people should not be touching those directories in the first place.

Those are just the tip of the iceberg. We do use the term xlog in a lot of
places (almost as many as wal, but that's a different problem)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

14.10.2016, 07:38, Peter Eisentraut kirjoitti:
> On 10/12/16 11:22 AM, Tom Lane wrote:
>> The main problem we're trying to fix here is people thinking that
>> something with "log" in the name contains discardable data. Just
>> relocating the directory without renaming it won't improve that.
>
> I think it would help if we moved it to something like
> "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> out of sight.
>
> We have a tool called pg_xlogdump in the standard installation. initdb
> has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> the xlog directory would make this all a bit confusing, unless we're
> prepared to rename the programs and options as well.

pg_receivexlog should probably be renamed, seeing how we have
pg_recvlogical perhaps pg_recvwal would work?

The --xlog, -x, --xlog-method and -X flags for pg_basebackup are a bit
confusing as it is. Perhaps they can be kept around as deprecated
aliases for a new --wal stream/fetch switch: I don't think we need new
--wal and --wal-method switches.

pg_resetxlog should probably be called something different than just
plain pg_resetwal to make it obvious that running it will cause data loss.

pg_xlogdump is a developer tool, users shouldn't care; it's hard enough
to use as it is as it doesn't do anything useful when you try to point
it to a recycled wal file.

All in all, I think this is a good opportunity to clarify what the tools
are actually supposed to do and who should be running them. As an
author of a backup tool utilizing some of these tools & options I don't
think renaming commands and/or arguments is a big deal -- we have to
deal with a bunch of changes for every new major version anyway.

/ Oskari

Re: Stephen Frost 2016-10-14 <20161014113523(dot)GZ13284(at)tamriel(dot)snowman(dot)net>
> > We have a tool called pg_xlogdump in the standard installation. initdb
> > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > the xlog directory would make this all a bit confusing, unless we're
> > prepared to rename the programs and options as well.
>
> pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> should be terribly concerned about either leaving it named as-is or
> renaming it. I agree that we should consider adding alternative names
> to the options for initdb and pg_basebackup.

There's actually another instance of "rename so people shoot their
feet less often" here: pg_resetxlog, which is a user-facing tool.
Folks on #postgresql have repeatedly been joking that it should rather
be named pg_eatmydata, given the number of "howtos" on the web that
make use of it. Maybe this is the chance to find a less
innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Christoph

* Magnus Hagander (magnus(at)hagander(dot)net) wrote:
> On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > * Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
> > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > The main problem we're trying to fix here is people thinking that
> > > > something with "log" in the name contains discardable data. Just
> > > > relocating the directory without renaming it won't improve that.
> > >
> > > I think it would help if we moved it to something like
> > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > > out of sight.
> >
> > I disagree that this will materially help with the issue.
> >
> > > We have a tool called pg_xlogdump in the standard installation. initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it. I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
>
> Ugh. Changing the names of those options are probably going to break a
> *lot* of things, a lot more than changing the names of the directories. Is
> it really worth doing that? Especially if we are even more clear that
> people should not be touching those directories in the first place.

I would point out that I said "adding alternative names", not
"change/remove the existing options."

That said, while I like the general idea of "make everything referring
to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
way", I'm not sure that it's a really pressing issue or that changing
the name of the directory should drive those other changes.

> Those are just the tip of the iceberg. We do use the term xlog in a lot of
> places (almost as many as wal, but that's a different problem)

True, and yes, we should consider the places that already talk about
"wal", but, overall, I believe we can take this one step at a time and
don't have to do everything all at once simply because we're changing
the directory name. We're not changing what it *is*, after all.

Thanks!

Stephen

* Christoph Berg (myon(at)debian(dot)org) wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523(dot)GZ13284(at)tamriel(dot)snowman(dot)net>
> > > We have a tool called pg_xlogdump in the standard installation. initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it. I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
>
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.

That is *not* a user-facing tool.

> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

Not sure on the best approach here, but I agree that it's a problem that
people think pg_resetxlog is some kind of recovery tool (hint: it's
not).

Thanks!

Stephen

On Fri, Oct 14, 2016 at 11:48 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:

> * Magnus Hagander (magnus(at)hagander(dot)net) wrote:
> > On Fri, Oct 14, 2016 at 4:35 AM, Stephen Frost <sfrost(at)snowman(dot)net>
> wrote:
> > > * Peter Eisentraut (peter(dot)eisentraut(at)2ndquadrant(dot)com) wrote:
> > > > On 10/12/16 11:22 AM, Tom Lane wrote:
> > > > > The main problem we're trying to fix here is people thinking that
> > > > > something with "log" in the name contains discardable data. Just
> > > > > relocating the directory without renaming it won't improve that.
> > > >
> > > > I think it would help if we moved it to something like
> > > > "internal/pg_xlog" and "internal/pg_clog". Keep the name but move it
> > > > out of sight.
> > >
> > > I disagree that this will materially help with the issue.
> > >
> > > > We have a tool called pg_xlogdump in the standard installation.
> initdb
> > > > has an option --xlogdir, pg_basebackup has --xlog and others.
> Renaming
> > > > the xlog directory would make this all a bit confusing, unless we're
> > > > prepared to rename the programs and options as well.
> > >
> > > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > > should be terribly concerned about either leaving it named as-is or
> > > renaming it. I agree that we should consider adding alternative names
> > > to the options for initdb and pg_basebackup.
> >
> > Ugh. Changing the names of those options are probably going to break a
> > *lot* of things, a lot more than changing the names of the directories.
> Is
> > it really worth doing that? Especially if we are even more clear that
> > people should not be touching those directories in the first place.
>
> I would point out that I said "adding alternative names", not
> "change/remove the existing options."
>

Oh, I missed that. Thanks for clarifying :)

> That said, while I like the general idea of "make everything referring
> to transaction log/write-ahead xlog/xlog/wal/whatever refer in the same
> way", I'm not sure that it's a really pressing issue or that changing
> the name of the directory should drive those other changes.
>

Agreed.

> > Those are just the tip of the iceberg. We do use the term xlog in a lot
> of
> > places (almost as many as wal, but that's a different problem)
>
> True, and yes, we should consider the places that already talk about
> "wal", but, overall, I believe we can take this one step at a time and
> don't have to do everything all at once simply because we're changing
> the directory name. We're not changing what it *is*, after all.
>

Yeah, I think if we aim to fix everything at once, we'll get nothing done.
That old saying of not letting perfection getting in the way of progress
applies.

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On Fri, Oct 14, 2016 at 07:19:02PM +0200, Christoph Berg wrote:
> Re: Stephen Frost 2016-10-14 <20161014113523(dot)GZ13284(at)tamriel(dot)snowman(dot)net>
> > > We have a tool called pg_xlogdump in the standard installation. initdb
> > > has an option --xlogdir, pg_basebackup has --xlog and others. Renaming
> > > the xlog directory would make this all a bit confusing, unless we're
> > > prepared to rename the programs and options as well.
> >
> > pg_xlogdump is not a user-facing tool, frankly, so I don't believe we
> > should be terribly concerned about either leaving it named as-is or
> > renaming it. I agree that we should consider adding alternative names
> > to the options for initdb and pg_basebackup.
>
> There's actually another instance of "rename so people shoot their
> feet less often" here: pg_resetxlog, which is a user-facing tool.
> Folks on #postgresql have repeatedly been joking that it should rather
> be named pg_eatmydata, given the number of "howtos" on the web that
> make use of it. Maybe this is the chance to find a less
> innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)

I wonder how many of the uses of pg_resetxlog were caused by mistakenly
removing the pg_xlog directory. My point is renaming pg_xlog might
reduce mistake use of pg_resetxlog.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

Re: Bruce Momjian 2016-10-19 <20161018220909(dot)GA11661(at)momjian(dot)us>
> > There's actually another instance of "rename so people shoot their
> > feet less often" here: pg_resetxlog, which is a user-facing tool.
> > Folks on #postgresql have repeatedly been joking that it should rather
> > be named pg_eatmydata, given the number of "howtos" on the web that
> > make use of it. Maybe this is the chance to find a less
> > innocent-sounding name. (Or add a mandatory --rewind-my-data switch.)
>
> I wonder how many of the uses of pg_resetxlog were caused by mistakenly
> removing the pg_xlog directory. My point is renaming pg_xlog might
> reduce mistake use of pg_resetxlog.

I don't think there's much of a connection. There are people who
clean up disk space by removing everything that sounds like log files.
For those people renaming the directories makes sense so they don't
have that idea.

There are other people who have random database startup problems, ask
google, and end up with applying some pg_resetxlog advice (that
doesn't necessarily fit their problem). For those people renaming
pg_resetxlog to something that sounds more like "it will break your
data, use as last resort only" might make sense. (Though I don't have
a good suggestion, and the cost of renaming utilities is higher than
renaming some internal directory names.)

The question would now be if there's people who used pg_resetxlog
because they thought it freed up disk space, and if renaming either
would have prevented that. I'm less sure about that.

(tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
alternative.)

Christoph

On Wed, Oct 19, 2016 at 7:05 AM, Christoph Berg <myon(at)debian(dot)org> wrote:
> (tl;dr: rename pg_xlog yes, rename pg_resetxlog only if we have a good
> alternative.)

I'm amused by the idea of a TL;DR in parentheses at the very bottom of
the email, but maybe I'm just easily amused.

One idea would be to rename pg_resetxlog to pg_resetwal. I think
that's actually an improvement. The "x" in "xlog" is not a clear
abbreviation for "write-ahead", especially when we also use "x" in
other contexts to mean "transaction" - think "xid". Given our current
practices, we could justifiably rename pg_clog to pg_xlog -- after
all, that's where we log the status of the xacts. Ugh.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
<michael(dot)paquier(at)gmail(dot)com> wrote:
> OK. I can live with that as well. Attached are three patches. The
> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> two ones.

Committed 0001.

To be honest, I don't really like either pg_transaction or pg_xact.
Neither name captures the fact that what we're really tracking here is
the transaction *status*. pg_xact is slightly worse because it's a
poor abbreviation for transaction, but I think the argument against
even pg_transaction is similar to the one Tom previously levied
against pg_logical - viz. "logical what?". The transaction themselves
are not stored in the directory, just the commit status. The fact
that commit status is saved is the source of the "c" in "clog".

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> One idea would be to rename pg_resetxlog to pg_resetwal. I think
> that's actually an improvement.

This would fit in as part of a general plan to s/xlog/wal/g throughout
our user-visible names and documentation. Which seems like a good idea
to me; there's no need to expose two different terms for the same thing.

(I don't feel a great need to unify the terminology in the code, though.
Just in stuff users see.)

regards, tom lane

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
> > OK. I can live with that as well. Attached are three patches. The
> > pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
> > pg_clog -> pg_xact move. Only one survivor to be chosen among the last
> > two ones.
>
> Committed 0001.

Glad to see that happen, finally.

> To be honest, I don't really like either pg_transaction or pg_xact.

> Neither name captures the fact that what we're really tracking here is
> the transaction *status*. pg_xact is slightly worse because it's a
> poor abbreviation for transaction, but I think the argument against
> even pg_transaction is similar to the one Tom previously levied
> against pg_logical - viz. "logical what?". The transaction themselves
> are not stored in the directory, just the commit status. The fact
> that commit status is saved is the source of the "c" in "clog".

This really needs to move forward also.

When it comes to the name, I tend to think of 'pg_xact' as saying "this
is where we persist info we need to keep about transactions." Today
that's just the commit status info, but I could imagine that there
might, someday, be other things that go in there. "pg_multixact" is
an example of something quite similar but does have more than just one
"thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
bring them all under one consistent naming scheme.

Thanks!

Stephen

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > One idea would be to rename pg_resetxlog to pg_resetwal. I think
> > that's actually an improvement.
>
> This would fit in as part of a general plan to s/xlog/wal/g throughout
> our user-visible names and documentation. Which seems like a good idea
> to me; there's no need to expose two different terms for the same thing.
>
> (I don't feel a great need to unify the terminology in the code, though.
> Just in stuff users see.)

+1 on the general change of xlog -> wal.

That said, I'd also like to see a --force or similar option or mechanism
put in place to reduce the risk of users trashing their system because
they think pg_resetwal is "safe." ("It's just gonna reset things to make
the database start again, should be fine.").

pg_destroydb almost seems like a better choice, though I suppose
'pg_clearwal' would be more acceptable. Doesn't have quite the same
impact though.

Not sure on the best answer here, but it's definitely foot-gun that some
users have ended up using on themselves with depressing regularity.

Thanks!

Stephen

On 10/20/2016 09:12 AM, Stephen Frost wrote:
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:

> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").
>
> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> impact though.

pg_dropwal

Users won't *drop* things they shouldn't on purpose (usually) but they
will reset and will clear them. Destroydb isn't anymore accurate because
it doesn't destroy it. Instead it makes it so I can log in again and see
my data.

(Yes we all know the real implications with it but from a DUH user
perspective...)

Sincerely,

JD

--
Command Prompt, Inc. http://the.postgres.company/
+1-503-667-4564
PostgreSQL Centered full stack support, consulting and development.
Everyone appreciates your honesty, until you are honest with them.

On Thu, Oct 20, 2016 at 12:05 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>> To be honest, I don't really like either pg_transaction or pg_xact.
>
>> Neither name captures the fact that what we're really tracking here is
>> the transaction *status*. pg_xact is slightly worse because it's a
>> poor abbreviation for transaction, but I think the argument against
>> even pg_transaction is similar to the one Tom previously levied
>> against pg_logical - viz. "logical what?". The transaction themselves
>> are not stored in the directory, just the commit status. The fact
>> that commit status is saved is the source of the "c" in "clog".
>
> This really needs to move forward also.
>
> When it comes to the name, I tend to think of 'pg_xact' as saying "this
> is where we persist info we need to keep about transactions." Today
> that's just the commit status info, but I could imagine that there
> might, someday, be other things that go in there. "pg_multixact" is
> an example of something quite similar but does have more than just one
> "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> bring them all under one consistent naming scheme.

I don't dispute the fact that you tend to think of it that way, but I
think it's a real stretch to say that "pg_xact" is a clear name from
the point of view of the uninitiated. Now, maybe the point is to be a
little bit deliberately unclear, but "xact" for "transaction" is not a
lot better than "xlog" for "write-ahead log". It's just arbitrary
abbreviations we made up and you either know what they mean or you
don't. We could call it "pg_xkcd" and we wouldn't be removing much in
the way of clarity.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> That said, I'd also like to see a --force or similar option or mechanism
> put in place to reduce the risk of users trashing their system because
> they think pg_resetwal is "safe." ("It's just gonna reset things to make
> the database start again, should be fine.").

You know we already have that, right?

> pg_destroydb almost seems like a better choice, though I suppose
> 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> impact though.
>
> Not sure on the best answer here, but it's definitely foot-gun that some
> users have ended up using on themselves with depressing regularity.

Just to provide some perspective from the other side of this, I
handled a severity-1 escalation a few weeks ago where a user basically
called up and explained their situation and I told them based on all
of the facts and circumstances presented that running pg_resetxlog was
going to be the best way forward and they said that was pretty much
what they were expecting to hear, and did so. I think there's far too
much anti-pg_resetxlog bias on this list. The situation where you
need to use it is not common in general, but it's pretty common in
support cases that reach me. I don't think I'd be exaggerating if I
said that 25% of the customer escalations I handle personally require
the use pg_resetxlog. Of course, that's because by the time the
ticket gets to me, it's typically the case that all of the easy, neat
solutions have already been ruled out.

My experience is that users who are faced with this situation
typically understand that they are in a bad situation and when I
explain to them --- in a clear and direct way but without the sort of
hysterical exaggeration sometimes seen on this mailing list --- what
the consequences are, they are not surprised by those consequences and
they are willing to accept them because they know that the
alternatives are worse. For example, consider a customer with a
mostly read-only 20TB database. If that user runs pg_resetxlog, they
can be back up in 5 minutes and it is likely (though of course not
certain) that corruption will be minimal. If they restore from
backup, they will be down for many more hours. After pg_resetxlog,
they can restore from backup on another system or do the
dump-and-restore which I invariably recommend while they are up. For
many customers, this is a good trade-off. The decision, because of
the risks associated with it, is usually passed up from the DBA I'm
working with to some business leader. If that business leader feels
that benefits of being up immediately rather than many hours later are
sufficient to justify the risk of a possibly-corrupted database,
running pg_resetxlog is a perfectly reasonable decision.

I have not yet had one DBA or business leader call back and say "hey,
that thing where we ran pg_resetxlog? you should have discouraged us
more, because that was a disaster". Now maybe those disasters have
happened and I am just not aware of them, but I think we should all
take a deep breath and remind ourselves that the database exists in
service to the business and not the other way around. My job -- when
doing support -- is to tell you what your options are and what
consequences they may have, good and bad -- not to tell you how to run
your company. Telling users that pg_resetxlog will "destroy your
database" is over-the-top hyperbole. It's a sharp tool with risks and
benefits and it deserves to be presented exactly that way.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> > is where we persist info we need to keep about transactions." Today
> > that's just the commit status info, but I could imagine that there
> > might, someday, be other things that go in there. "pg_multixact" is
> > an example of something quite similar but does have more than just one
> > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> > bring them all under one consistent naming scheme.
>
> I don't dispute the fact that you tend to think of it that way, but I
> think it's a real stretch to say that "pg_xact" is a clear name from
> the point of view of the uninitiated. Now, maybe the point is to be a
> little bit deliberately unclear, but "xact" for "transaction" is not a
> lot better than "xlog" for "write-ahead log". It's just arbitrary
> abbreviations we made up and you either know what they mean or you
> don't. We could call it "pg_xkcd" and we wouldn't be removing much in
> the way of clarity.

What is your suggestion for a name? If you have none, I suggest we use
"pg_xact".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
>> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
>> > is where we persist info we need to keep about transactions." Today
>> > that's just the commit status info, but I could imagine that there
>> > might, someday, be other things that go in there. "pg_multixact" is
>> > an example of something quite similar but does have more than just one
>> > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
>> > bring them all under one consistent naming scheme.
>>
>> I don't dispute the fact that you tend to think of it that way, but I
>> think it's a real stretch to say that "pg_xact" is a clear name from
>> the point of view of the uninitiated. Now, maybe the point is to be a
>> little bit deliberately unclear, but "xact" for "transaction" is not a
>> lot better than "xlog" for "write-ahead log". It's just arbitrary
>> abbreviations we made up and you either know what they mean or you
>> don't. We could call it "pg_xkcd" and we wouldn't be removing much in
>> the way of clarity.
>
> What is your suggestion for a name? If you have none, I suggest we use
> "pg_xact".

I'm not sure. pg_transaction_status would be clear, but it's long.
Is pg_xact actually better than pg_clog?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Is pg_xact actually better than pg_clog?

Yes, because it doesn't contain the three letters "log".

We have the two precedents "pg_subtrans" and "pg_multixact", so
unless we want to get into renaming those too, I think "pg_trans"
and "pg_xact" are really the only options worth considering.

Personally I'd go for "pg_trans", but it's only a weak preference.

regards, tom lane

On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Is pg_xact actually better than pg_clog?
>
> Yes, because it doesn't contain the three letters "log".

I figured somebody was going to say that.

> We have the two precedents "pg_subtrans" and "pg_multixact", so
> unless we want to get into renaming those too, I think "pg_trans"
> and "pg_xact" are really the only options worth considering.
>
> Personally I'd go for "pg_trans", but it's only a weak preference.

Heaven forfend we actually use enough characters to make it self-documenting.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>> unless we want to get into renaming those too, I think "pg_trans"
>> and "pg_xact" are really the only options worth considering.
>>
>> Personally I'd go for "pg_trans", but it's only a weak preference.

> Heaven forfend we actually use enough characters to make it self-documenting.

$ ls $PGDATA
PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf
base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf
global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts
pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid
pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/

I don't see one single one of those subdirectory names that I'd call
self-documenting. Are you proposing we rename them all with carpal-
tunnel-syndrome-promoting names?

There's certainly some case to be made for renaming at least one of
"pg_subtrans" and "pg_multixact" so that these three similarly-purposed
subdirectories can all have similar names. But I think on the whole
that's (a) fixing what ain't broken, and (b) making it even more unlikely
that we'll ever get to consensus on changing anything. We've managed to
agree that we need to change the names ending in "log"; let's do that
and be happy that we've removed one foot-gun from the system.

regards, tom lane

On Thu, Oct 20, 2016 at 02:02:27PM -0400, Robert Haas wrote:
> On Thu, Oct 20, 2016 at 1:39 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > On Thu, Oct 20, 2016 at 12:29:47PM -0400, Robert Haas wrote:
> >> > When it comes to the name, I tend to think of 'pg_xact' as saying "this
> >> > is where we persist info we need to keep about transactions." Today
> >> > that's just the commit status info, but I could imagine that there
> >> > might, someday, be other things that go in there. "pg_multixact" is
> >> > an example of something quite similar but does have more than just one
> >> > "thing." Also, using "pg_xact" and then "pg_subxact" and "pg_multixact"
> >> > bring them all under one consistent naming scheme.
> >>
> >> I don't dispute the fact that you tend to think of it that way, but I
> >> think it's a real stretch to say that "pg_xact" is a clear name from
> >> the point of view of the uninitiated. Now, maybe the point is to be a
> >> little bit deliberately unclear, but "xact" for "transaction" is not a
> >> lot better than "xlog" for "write-ahead log". It's just arbitrary
> >> abbreviations we made up and you either know what they mean or you
> >> don't. We could call it "pg_xkcd" and we wouldn't be removing much in
> >> the way of clarity.
> >
> > What is your suggestion for a name? If you have none, I suggest we use
> > "pg_xact".
>
> I'm not sure. pg_transaction_status would be clear, but it's long.
> Is pg_xact actually better than pg_clog?

Uh, yeah, no "log". Wasn't that the point?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

On Thu, Oct 20, 2016 at 2:23 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> We have the two precedents "pg_subtrans" and "pg_multixact", so
>>> unless we want to get into renaming those too, I think "pg_trans"
>>> and "pg_xact" are really the only options worth considering.
>>>
>>> Personally I'd go for "pg_trans", but it's only a weak preference.
>
>> Heaven forfend we actually use enough characters to make it self-documenting.
>
> $ ls $PGDATA
> PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf
> base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf
> global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts
> pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid
> pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/
>
> I don't see one single one of those subdirectory names that I'd call
> self-documenting. Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

No. Are you proposing that self-documenting names are a bad thing
rather than a good thing?

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names. But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything. We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

I agree that there is an overwhelming consensus in favor of getting
"log" out of the names, but I do not agree that the only two possible
alternative names are "pg_trans" and "pg_xact", which strikes me as
being akin to saying that the only two options for dinner tonight are
overripe peaches and lunch meats a week past the sell-by date. If I
had to pick only between those two, I suppose I'd go with "pg_xact"
which is clear enough to someone familiar with PostgreSQL internals,
as opposed to "pg_trans" which I don't think will be clear even to
those people. But I don't like either one very much.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> >> unless we want to get into renaming those too, I think "pg_trans"
> >> and "pg_xact" are really the only options worth considering.
> >>
> >> Personally I'd go for "pg_trans", but it's only a weak preference.
>
> > Heaven forfend we actually use enough characters to make it self-documenting.
>
> $ ls $PGDATA
> PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf
> base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf
> global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts
> pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid
> pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/
>
> I don't see one single one of those subdirectory names that I'd call
> self-documenting.

That's a problem we should do something about, even if we can't do it
by renaming these all in one go. At the very least, we can do this
for any new names.

> Are you proposing we rename them all with carpal-
> tunnel-syndrome-promoting names?

Are you saying that people are getting carpal tunnel syndrome from
hitting the tab key, which has been standard for completion in shells
for decades? I'm pretty sure that doesn't actually happen.

> There's certainly some case to be made for renaming at least one of
> "pg_subtrans" and "pg_multixact" so that these three similarly-purposed
> subdirectories can all have similar names. But I think on the whole
> that's (a) fixing what ain't broken, and (b) making it even more unlikely
> that we'll ever get to consensus on changing anything. We've managed to
> agree that we need to change the names ending in "log"; let's do that
> and be happy that we've removed one foot-gun from the system.

Removing foot guns, un-sexy as it may be from a developer's
perspective, is very useful work.

Best,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

* David Fetter (david(at)fetter(dot)org) wrote:
> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
> > Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > > On Thu, Oct 20, 2016 at 2:09 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > >> We have the two precedents "pg_subtrans" and "pg_multixact", so
> > >> unless we want to get into renaming those too, I think "pg_trans"
> > >> and "pg_xact" are really the only options worth considering.
> > >>
> > >> Personally I'd go for "pg_trans", but it's only a weak preference.
> >
> > > Heaven forfend we actually use enough characters to make it self-documenting.
> >
> > $ ls $PGDATA
> > PG_VERSION pg_dynshmem/ pg_notify/ pg_stat_tmp/ postgresql.auto.conf
> > base/ pg_hba.conf pg_replslot/ pg_subtrans/ postgresql.conf
> > global/ pg_ident.conf pg_serial/ pg_tblspc/ postmaster.opts
> > pg_clog/ pg_logical/ pg_snapshots/ pg_twophase/ postmaster.pid
> > pg_commit_ts/ pg_multixact/ pg_stat/ pg_wal/
> >
> > I don't see one single one of those subdirectory names that I'd call
> > self-documenting.
>
> That's a problem we should do something about, even if we can't do it
> by renaming these all in one go. At the very least, we can do this
> for any new names.

I disagree that excessivly long names for files or directories are
useful and should be fully self-documenting. We should name our
directories with useful hints at what they are used for that remind
those who are knowledgable where things live. Secondary to that is
using an approach to naming which avoid implying anything about the
directories to those who are *not* knowledgable, which leads us to the
current discussion regarding the removal of directories with 'log' in
the name.

Individuals who are not knowledgable in this area are not going to get
any more benefit from a directory named 'pg_trans' or
'pg_transaction_status' than one named 'pg_clog' or 'pg_xact' and
therefore this whole line of reasoning is a red herring.

Thanks!

Stephen

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> * David Fetter (david(at)fetter(dot)org) wrote:
>> On Thu, Oct 20, 2016 at 02:23:32PM -0400, Tom Lane wrote:
>>> I don't see one single one of those subdirectory names that I'd call
>>> self-documenting.

>> That's a problem we should do something about, even if we can't do it
>> by renaming these all in one go. At the very least, we can do this
>> for any new names.

> I disagree that excessivly long names for files or directories are
> useful and should be fully self-documenting.

I'm mostly with Stephen on this. As the names stand, they encourage
people to go look at the documentation,
https://www.postgresql.org/docs/devel/static/storage-file-layout.html
which will provide more information than you'd ever get out of any
reasonable directory name.

Having said that, I still don't like "pg_logical", but I suppose
renaming it would have more downsides than upsides.

regards, tom lane

On Fri, Oct 21, 2016 at 12:35 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Oct 12, 2016 at 10:22 PM, Michael Paquier
> <michael(dot)paquier(at)gmail(dot)com> wrote:
>> OK. I can live with that as well. Attached are three patches. The
>> pg_xlog -> pg_wal move, the pg_clog -> pg_transaction move, and the
>> pg_clog -> pg_xact move. Only one survivor to be chosen among the last
>> two ones.
>
> Committed 0001.

Thanks!
--
Michael

On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I'm mostly with Stephen on this. As the names stand, they encourage
> people to go look at the documentation,
> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> which will provide more information than you'd ever get out of any
> reasonable directory name.

Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
would encourage that even more strongly. But I don't think that
proposal can be taken seriously. Giving things meaningful names is a
good practice in almost every case.

> Having said that, I still don't like "pg_logical", but I suppose
> renaming it would have more downsides than upsides.

Remind me what your beef is?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Fri, Oct 21, 2016 at 10:03 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I'm mostly with Stephen on this. As the names stand, they encourage
>> people to go look at the documentation,
>> https://www.postgresql.org/docs/devel/static/storage-file-layout.html
>> which will provide more information than you'd ever get out of any
>> reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly. But I don't think that
> proposal can be taken seriously. Giving things meaningful names is a
> good practice in almost every case.

Moving on with the topic of this thread... I would think that
"pg_xact" is what we are moving to rename pg_clog.

On top of that, after reading the thread, here are the options and
binaries that could be renamed for consistency with the renaming of
pg_xlog:
- pg_xlogdump -> pg_waldump
- pg_resetxlog -> pg_resetwal
- pg_receivexlog -> pg_receivewal
- initdb --xlogdir -> --waldir
- pg_basebackup --xlogmethod --xlogdir -> --walmethod --waldir
That's quite a number, and each change is trivial. Previous options
would be better kept for backward-compatibility. Personally, I see no
urge in changing all that and I am fine with just renaming the *log
directories of PGDATA for this thread. But as the point has been
raised, here are all things that could be done.
--
Michael

On Thu, Oct 20, 2016 at 6:03 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> On Thu, Oct 20, 2016 at 3:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > I'm mostly with Stephen on this. As the names stand, they encourage
> > people to go look at the documentation,
> > https://www.postgresql.org/docs/devel/static/storage-file-layout.html
> > which will provide more information than you'd ever get out of any
> > reasonable directory name.
>
> Well, we could change them all to pg_a, pg_b, pg_c, pg_d, ... which
> would encourage that even more strongly. But I don't think that
> proposal can be taken seriously. Giving things meaningful names is a
> good practice in almost every case.
>

Those don't have the virtue of being at least somewhat ​
m
nemonic
​ like pg_xact.

I'll toss my lot in with Steven's and Tom's on this.

​I have no problem continuing keeping with historical precedent ​and
allowing mnemonic abbreviations in our directory and file names at this
point.

David J.

​

On 10/21/16 3:12 AM, David G. Johnston wrote:

> I have no problem continuing keeping with historical precedent ​and
> allowing mnemonic abbreviations in our directory and file names at this
> point.

I'm still in favor of pg_xact. A search of the 9.6 docs brings up a
number of hits for "xact": pg_last_xact_replay_timestamp(),
pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also
numerous column names that have "xact" in them.

It's not just an arcane developer term when it shows up in a number of
our user-facing functions/columns.

--
-David
david(at)pgmasters(dot)net

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > That said, I'd also like to see a --force or similar option or mechanism
> > put in place to reduce the risk of users trashing their system because
> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
> > the database start again, should be fine.").
>
> You know we already have that, right?

Yes, but I was meaning an option which would be required to make
pg_resetxlog actually *do* anything. In other words, I'd rather have it
report some info back to the user, if it's run without the
'--really-force' or what-have-you option, and only proceed with
clearing the WAL or rewriting pg_control when '--really-force' is used.

> > pg_destroydb almost seems like a better choice, though I suppose
> > 'pg_clearwal' would be more acceptable. Doesn't have quite the same
> > impact though.
> >
> > Not sure on the best answer here, but it's definitely foot-gun that some
> > users have ended up using on themselves with depressing regularity.
>
> Just to provide some perspective from the other side of this, I
[...]

I wasn't suggesting that we remove the capability. There are certainly
use-cases for it, but, unfortunately, I've seen a number of cases where
users simply google'd an error that they got back when trying to start
PG and found someone saying "well, I got this error, but then I ran
pg_resetxlog, and now the database starts up again."

It likely doesn't help that the top links tend to be to mailing list
archives where pg_resetxlog was brought up.

Thanks!

Stephen

On Fri, Oct 21, 2016 at 9:47 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
>> On Thu, Oct 20, 2016 at 12:12 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
>> > That said, I'd also like to see a --force or similar option or mechanism
>> > put in place to reduce the risk of users trashing their system because
>> > they think pg_resetwal is "safe." ("It's just gonna reset things to make
>> > the database start again, should be fine.").
>>
>> You know we already have that, right?
>
> Yes, but I was meaning an option which would be required to make
> pg_resetxlog actually *do* anything. In other words, I'd rather have it
> report some info back to the user, if it's run without the
> '--really-force' or what-have-you option, and only proceed with
> clearing the WAL or rewriting pg_control when '--really-force' is used.

I don't think that the problem is that people are accidentally typing
"pg_resetxlog $PGDATA" and pressing return. They're typing that on
purpose, and if you change the sequence of characters required to get
that effect, they'll just type the new thing instead. The problem is
that they don't understand when it's appropriate to use that command
and what the consequences are.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> I don't think that the problem is that people are accidentally typing
> "pg_resetxlog $PGDATA" and pressing return. They're typing that on
> purpose, and if you change the sequence of characters required to get
> that effect, they'll just type the new thing instead. The problem is
> that they don't understand when it's appropriate to use that command
> and what the consequences are.

I agree that they don't understand, and that's why I believe that making
the command name, or a required option, a bit more ominous would make
them pause and realize that maybe they want to consider other options
first.

This is not exactly unheard of- apt-get requires an entire phrase be to
be entered when you're asking it to do something extremely dangerous
(remove an essential package):

---------------
root(at)ionith:~# apt-get remove login
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
login
WARNING: The following essential packages will be removed.
This should NOT be done unless you know exactly what you are doing!
login
0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
After this operation, 1,212 kB disk space will be freed.
You are about to do something potentially harmful.
To continue type in the phrase 'Yes, do as I say!'
?]
---------------

My thought would be to make pg_resetxlog do something more along the
lines of what pg_control does and have it, by default, just investigate
the state of things and complain about problems and then have it include
an option to actually reset things with an appropriately ominous name.
The goal there being, primairly, to give a way for users to get
information about why PG isn't starting or what it is complaining about,
with some additional information about what happens if they forcibly
reset the xlog, noting that it could very likely cause corruption, etc.

Thanks!

Stephen

David Steele wrote:
> On 10/21/16 3:12 AM, David G. Johnston wrote:
>
> > I have no problem continuing keeping with historical precedent ​and
> > allowing mnemonic abbreviations in our directory and file names at this
> > point.
>
> I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number
> of hits for "xact": pg_last_xact_replay_timestamp(),
> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous
> column names that have "xact" in them.

I'm +1 on pg_clog -> pg_xact.

Also +1 to renaming pg_subtrans to pg_subxact.

In general, +1 to short mnemonic meaningful names. -1 to overly long
names, -1 to meaningless names.

Regarding pg_receivexlog I think I'd propose pg_recvwal rather than
pg_receivewal, because of pg_recvlogical.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
<alvherre(at)2ndquadrant(dot)com> wrote:
> David Steele wrote:
>> On 10/21/16 3:12 AM, David G. Johnston wrote:
>>
>> > I have no problem continuing keeping with historical precedent and
>> > allowing mnemonic abbreviations in our directory and file names at this
>> > point.
>>
>> I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number
>> of hits for "xact": pg_last_xact_replay_timestamp(),
>> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
>> pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous
>> column names that have "xact" in them.
>
> I'm +1 on pg_clog -> pg_xact.

So let's say that's the winner then.

> Also +1 to renaming pg_subtrans to pg_subxact.

Nice suggestion, good naming for consistency with the rest.
--
Michael

On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
> <alvherre(at)2ndquadrant(dot)com> wrote:
> > David Steele wrote:
> >> On 10/21/16 3:12 AM, David G. Johnston wrote:
> >>
> >> > I have no problem continuing keeping with historical precedent and
> >> > allowing mnemonic abbreviations in our directory and file names at this
> >> > point.
> >>
> >> I'm still in favor of pg_xact. A search of the 9.6 docs brings up a number
> >> of hits for "xact": pg_last_xact_replay_timestamp(),
> >> pg_advisory_xact_lock(), pg_advisory_xact_lock_shared(),
> >> pg_last_committed_xact(), pg_prepared_xacts(), etc. There are also numerous
> >> column names that have "xact" in them.
> >
> > I'm +1 on pg_clog -> pg_xact.
>
> So let's say that's the winner then.
>
> > Also +1 to renaming pg_subtrans to pg_subxact.
>
> Nice suggestion, good naming for consistency with the rest.

Agreed.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

On 10/22/16 6:58 PM, Bruce Momjian wrote:
> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>
>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>
>> Nice suggestion, good naming for consistency with the rest.
>
> Agreed.

+1

--
-David
david(at)pgmasters(dot)net

On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> WARNING: The following essential packages will be removed.
> This should NOT be done unless you know exactly what you are doing!
> login
> 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> After this operation, 1,212 kB disk space will be freed.
> You are about to do something potentially harmful.
> To continue type in the phrase 'Yes, do as I say!'

Another case was:
glxgears -iacknowledgethatthistoolisnotabenchmark

Which was required to get it to print the FPS for a while. The problem
-- and also the advantage -- of this is that it's scriptable. That
means people can still put it in recipe books and scripts and others
can copy it without being aware what it does or even that they're
doing it.

I think the apt-get behaviour was specifically designed to ensure it
couldn't easily be put into a script which I would have said was
desirable -- except I suspect there are situations where Postgres
database scripts need to do a resetxlog. I'm not sure I can think of
any examples offhand but I wouldn't be too surprised.

--
greg

On 10/22/2016 06:00 PM, David Steele wrote:
> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>>
>>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>>
>>> Nice suggestion, good naming for consistency with the rest.
>>
>> Agreed.
>
> +1

+1 from me, too.
--
Vik Fearing +33 6 46 75 15 36
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

On Sun, Oct 23, 2016 at 5:18 PM, Vik Fearing <vik(at)2ndquadrant(dot)fr> wrote:
> On 10/22/2016 06:00 PM, David Steele wrote:
>> On 10/22/16 6:58 PM, Bruce Momjian wrote:
>>> On Sat, Oct 22, 2016 at 07:33:56AM +0900, Michael Paquier wrote:
>>>> On Sat, Oct 22, 2016 at 4:29 AM, Alvaro Herrera
>>>>
>>>>> Also +1 to renaming pg_subtrans to pg_subxact.
>>>>
>>>> Nice suggestion, good naming for consistency with the rest.
>>>
>>> Agreed.
>>
>> +1
>
> +1 from me, too.

OK, glad to see that we are reaching a conclusion to this thread.
Attached are two patches. 0001 renames pg_clog to pg_xact. I found a
comment in xact.c that was not yet updated. And 0002 renames
pg_subtrans to pg_subxact.
--
Michael

On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
> On Fri, Oct 21, 2016 at 9:03 PM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> > WARNING: The following essential packages will be removed.
> > This should NOT be done unless you know exactly what you are doing!
> > login
> > 0 upgraded, 0 newly installed, 1 to remove and 71 not upgraded.
> > After this operation, 1,212 kB disk space will be freed.
> > You are about to do something potentially harmful.
> > To continue type in the phrase 'Yes, do as I say!'
>
> Another case was:
> glxgears -iacknowledgethatthistoolisnotabenchmark
>
> Which was required to get it to print the FPS for a while. The problem
> -- and also the advantage -- of this is that it's scriptable. That
> means people can still put it in recipe books and scripts and others
> can copy it without being aware what it does or even that they're
> doing it.
>
> I think the apt-get behaviour was specifically designed to ensure it
> couldn't easily be put into a script which I would have said was
> desirable -- except I suspect there are situations where Postgres
> database scripts need to do a resetxlog. I'm not sure I can think of
> any examples offhand but I wouldn't be too surprised.

Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +

Bruce Momjian wrote:
> On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:

> > I think the apt-get behaviour was specifically designed to ensure it
> > couldn't easily be put into a script which I would have said was
> > desirable -- except I suspect there are situations where Postgres
> > database scripts need to do a resetxlog. I'm not sure I can think of
> > any examples offhand but I wouldn't be too surprised.
>
> Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.

There is one difference though, which is that the really destructive
use of pg_resetxlog is the one that removes pg_xlog files. The other
uses that simply set flags in the control file are not as bad -- you can
simply go back to what the original value was. I think we would only
need the obnoxious string requirement in the most dangerous mode.

Alternatively, we could separate the tool in two different executables.

--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

On Mon, Oct 24, 2016 at 11:59:42AM -0300, Alvaro Herrera wrote:
> Bruce Momjian wrote:
> > On Sat, Oct 22, 2016 at 11:18:28PM +0300, Greg Stark wrote:
>
> > > I think the apt-get behaviour was specifically designed to ensure it
> > > couldn't easily be put into a script which I would have said was
> > > desirable -- except I suspect there are situations where Postgres
> > > database scripts need to do a resetxlog. I'm not sure I can think of
> > > any examples offhand but I wouldn't be too surprised.
> >
> > Yes, pg_upgrade has eight calls to pg_resetxlog to set various value.
>
> There is one difference though, which is that the really destructive
> use of pg_resetxlog is the one that removes pg_xlog files. The other
> uses that simply set flags in the control file are not as bad -- you can
> simply go back to what the original value was. I think we would only
> need the obnoxious string requirement in the most dangerous mode.
>
> Alternatively, we could separate the tool in two different executables.

OK, but we are going need to document that this special flag is only
needed for certain option uses.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ As you are, so once was I. As I am, so you will be. +
+ Ancient Roman grave inscription +