Re: master in standby mode croaks

I discovered tonight that if you shut down a server, create
recovery.conf with standby_mode = 'on', and start it back up again,
you get this:

LOG: database system was shut down at 2010-03-30 22:34:09 EDT
LOG: entering standby mode
FATAL: recovery connections cannot start because the
recovery_connections parameter is disabled on the WAL source server
LOG: startup process (PID 22980) exited with exit code 1
LOG: aborting startup due to startup process failure

Now, you might certainly argue that this is a stupid thing to do (my
motivation was to test some stuff) but certainly it's fair to say that
error message is darn misleading, since in fact recovery_connections
was NOT disabled. I believe this is the same "start up from a shut
down checkpoint" problem that's been discussed previously so I won't
belabor the point other than to say that I still think we need to fix
this.

...Robert

On Tue, 2010-03-30 at 22:40 -0400, Robert Haas wrote:
> I discovered tonight that if you shut down a server, create
> recovery.conf with standby_mode = 'on', and start it back up again,
> you get this:
>
> LOG: database system was shut down at 2010-03-30 22:34:09 EDT
> LOG: entering standby mode
> FATAL: recovery connections cannot start because the
> recovery_connections parameter is disabled on the WAL source server
> LOG: startup process (PID 22980) exited with exit code 1
> LOG: aborting startup due to startup process failure
>
> Now, you might certainly argue that this is a stupid thing to do (my
> motivation was to test some stuff) but certainly it's fair to say that
> error message is darn misleading, since in fact recovery_connections
> was NOT disabled. I believe this is the same "start up from a shut
> down checkpoint" problem that's been discussed previously so I won't
> belabor the point other than to say that

I don't think it is the same thing at all. This is a separate error and
should be rejected as such.

> I still think we need to fix this.

Agreed, as a separate issue.

--
Simon Riggs www.2ndQuadrant.com

On Apr 1, 2010, at 7:06 PM, Simon Riggs <simon(at)2ndQuadrant(dot)com> wrote:
> On Tue, 2010-03-30 at 22:40 -0400, Robert Haas wrote:
>> I discovered tonight that if you shut down a server, create
>> recovery.conf with standby_mode = 'on', and start it back up again,
>> you get this:
>>
>> LOG: database system was shut down at 2010-03-30 22:34:09 EDT
>> LOG: entering standby mode
>> FATAL: recovery connections cannot start because the
>> recovery_connections parameter is disabled on the WAL source server
>> LOG: startup process (PID 22980) exited with exit code 1
>> LOG: aborting startup due to startup process failure
>>
>> Now, you might certainly argue that this is a stupid thing to do (my
>> motivation was to test some stuff) but certainly it's fair to say
>> that
>> error message is darn misleading, since in fact recovery_connections
>> was NOT disabled. I believe this is the same "start up from a shut
>> down checkpoint" problem that's been discussed previously so I won't
>> belabor the point other than to say that
>
> I don't think it is the same thing at all. This is a separate error
> and
> should be rejected as such.
>
>> I still think we need to fix this.
>
> Agreed, as a separate issue.

OK, fair enough. I admit I didn't investigate what was causing this.

...Robert

On Fri, 2010-04-02 at 04:51 -0400, Robert Haas wrote:
> On Apr 1, 2010, at 7:06 PM, Simon Riggs <simon(at)2ndQuadrant(dot)com> wrote:
> > On Tue, 2010-03-30 at 22:40 -0400, Robert Haas wrote:
> >> I discovered tonight that if you shut down a server, create
> >> recovery.conf with standby_mode = 'on', and start it back up again,
> >> you get this:
> >>
> >> LOG: database system was shut down at 2010-03-30 22:34:09 EDT
> >> LOG: entering standby mode
> >> FATAL: recovery connections cannot start because the
> >> recovery_connections parameter is disabled on the WAL source server
> >> LOG: startup process (PID 22980) exited with exit code 1
> >> LOG: aborting startup due to startup process failure
> >>
> >> Now, you might certainly argue that this is a stupid thing to do (my
> >> motivation was to test some stuff) but certainly it's fair to say
> >> that
> >> error message is darn misleading, since in fact recovery_connections
> >> was NOT disabled. I believe this is the same "start up from a shut
> >> down checkpoint" problem that's been discussed previously so I won't
> >> belabor the point other than to say that
> >
> > I don't think it is the same thing at all. This is a separate error
> > and
> > should be rejected as such.

I can't duplicate this error based upon what you have said.

With just standby_mode = 'on' the standby just waits forever, with a ps
message set to
postgres: startup process waiting for 000000010000000000000000

That's not very good, but it isn't the error you describe.

--
Simon Riggs www.2ndQuadrant.com

On Fri, Apr 2, 2010 at 5:36 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> I can't duplicate this error based upon what you have said.

I fooled around with this some more and I think I know what's going
on. The error message I received was:

recovery connections cannot start because the recovery_connections
parameter is disabled on the WAL source server

This is generated when !checkPoint.XLogStandbyInfoMode. That, in
turn, is set on the master to the results of XLogStandbyInfoActive(),
which is defined as XLogRequestRecoveryConnections && XLogIsNeeded().
XLogIsNeeded() is defined as XLogArchivingActive() || (max_wal_senders
> 0), and XLogArchivingActive() is defined as XLogArchiveMode. So
when you expand it all out, this error message gets triggered when the
following condition does not hold on the master:

XLogRequestRecoveryConnections && (XLogArchiveMode || (max_wal_senders > 0))

So this can fail in either of two ways: (1)
XLogRequestRecoveryConnections (aka recovery_connections) might be
false, which is the situation described in the error message, or (2)
XLogArchiveMode (archive_mode) might be false and at the same time
max_wal_senders might be zero. As it happens, the default
configuration of the system is recovery_connections = true,
archive_mode = false, max_wal_senders = 0, so with an out-of-the-box
config it fails for the reason that isn't the one described in the
error message.

One possible approach here is to improve the error message, but it
seems to me that having the ability of Hot Standby to run on the slave
partially controlled by three different GUCs is awfully complicated.
I think the root of the problem here is that recovery_connections
controls one behavior on the primary (whether or not we WAL-log
certain information needed for HS) and a completely unrelated behavior
on the standby (whether or not we try to allow read-only backends into
the system). In 8.4 and prior, it was always the job of archive_mode
to decide whether WAL-logging was needed. Maybe we should go back to
that and make it an enum:

wal_mode = {standby | archive | off}

...Robert

On Sat, 2010-04-10 at 09:02 -0400, Robert Haas wrote:

> So this can fail in either of two ways

If I understand this correctly, it is unconvincing as a failure mode
since it doesn't follow any of the documented procedures for creating a
standby. There are many ways to screw up that ignore the manual, which
is why the manual exists.

If you can show a full test case, with failure, then I'll follow it
through.

--
Simon Riggs www.2ndQuadrant.com

On Wed, Apr 14, 2010 at 4:21 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Sat, 2010-04-10 at 09:02 -0400, Robert Haas wrote:
>
>> So this can fail in either of two ways
>
> If I understand this correctly, it is unconvincing as a failure mode
> since it doesn't follow any of the documented procedures for creating a
> standby. There are many ways to screw up that ignore the manual, which
> is why the manual exists.
>
> If you can show a full test case, with failure, then I'll follow it
> through.

Huh? If I had done everything correctly, of course I wouldn't have
gotten an error message at all. Surely the point is that if I do
something wrong, I should get an error message that describes what I
actually did wrong rather than an error message telling me that I did
something wrong which I clearly did not do.

The recent patch to allow starting from a shutdown checkpoint means
that a standby can be created by shutting down the master and taking a
filesystem-level snapshot of the cluster directly, creating
recovery.conf, and firing it up again. Anyone who does that with the
default postgresql.conf, though, is going to get a message telling
them that they need to change a setting which is already set
correctly.

...Robert

On Wed, 2010-04-14 at 07:07 -0400, Robert Haas wrote:
> On Wed, Apr 14, 2010 at 4:21 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > On Sat, 2010-04-10 at 09:02 -0400, Robert Haas wrote:
> >
> >> So this can fail in either of two ways
> >
> > If I understand this correctly, it is unconvincing as a failure mode
> > since it doesn't follow any of the documented procedures for creating a
> > standby. There are many ways to screw up that ignore the manual, which
> > is why the manual exists.
> >
> > If you can show a full test case, with failure, then I'll follow it
> > through.
>
> Huh? If I had done everything correctly, of course I wouldn't have
> gotten an error message at all. Surely the point is that if I do
> something wrong, I should get an error message that describes what I
> actually did wrong rather than an error message telling me that I did
> something wrong which I clearly did not do.

I will change the error message.

> The recent patch to allow starting from a shutdown checkpoint means
> that a standby can be created by shutting down the master and taking a
> filesystem-level snapshot of the cluster directly, creating
> recovery.conf, and firing it up again. Anyone who does that with the
> default postgresql.conf, though, is going to get a message telling
> them that they need to change a setting which is already set
> correctly.

Why would they do that? I would never claim this supports all use cases,
just the sensible ones.

--
Simon Riggs www.2ndQuadrant.com

On Wed, Apr 14, 2010 at 7:52 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, 2010-04-14 at 07:07 -0400, Robert Haas wrote:
>> On Wed, Apr 14, 2010 at 4:21 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> > On Sat, 2010-04-10 at 09:02 -0400, Robert Haas wrote:
>> >
>> >> So this can fail in either of two ways
>> >
>> > If I understand this correctly, it is unconvincing as a failure mode
>> > since it doesn't follow any of the documented procedures for creating a
>> > standby. There are many ways to screw up that ignore the manual, which
>> > is why the manual exists.
>> >
>> > If you can show a full test case, with failure, then I'll follow it
>> > through.
>>
>> Huh?  If I had done everything correctly, of course I wouldn't have
>> gotten an error message at all.  Surely the point is that if I do
>> something wrong, I should get an error message that describes what I
>> actually did wrong rather than an error message telling me that I did
>> something wrong which I clearly did not do.
>
> I will change the error message.

I gave a good deal of thought to trying to figure out a cleaner
solution to this problem than just changing the error message and
failed. So let's change the error message. Of course I'm not quite
sure what we should change it TO, given that the situation is the
result of an interaction between three different GUCs and we have no
way to distinguish which one(s) are the problem.

...Robert

On Sat, 2010-04-17 at 17:44 -0400, Robert Haas wrote:

> > I will change the error message.
>
> I gave a good deal of thought to trying to figure out a cleaner
> solution to this problem than just changing the error message and
> failed. So let's change the error message. Of course I'm not quite
> sure what we should change it TO, given that the situation is the
> result of an interaction between three different GUCs and we have no
> way to distinguish which one(s) are the problem.

"You need all three" covers it.

--
Simon Riggs www.2ndQuadrant.com

On Sat, Apr 17, 2010 at 6:41 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Sat, 2010-04-17 at 17:44 -0400, Robert Haas wrote:
>
>> > I will change the error message.
>>
>> I gave a good deal of thought to trying to figure out a cleaner
>> solution to this problem than just changing the error message and
>> failed.  So let's change the error message.  Of course I'm not quite
>> sure what we should change it TO, given that the situation is the
>> result of an interaction between three different GUCs and we have no
>> way to distinguish which one(s) are the problem.
>
> "You need all three" covers it.

Actually you need standby_connections and either archive_mode=on or
max_wal_senders>0, I think.

...Robert

On Sun, Apr 18, 2010 at 7:52 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Sat, Apr 17, 2010 at 6:41 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> On Sat, 2010-04-17 at 17:44 -0400, Robert Haas wrote:
>>
>>> > I will change the error message.
>>>
>>> I gave a good deal of thought to trying to figure out a cleaner
>>> solution to this problem than just changing the error message and
>>> failed.  So let's change the error message.  Of course I'm not quite
>>> sure what we should change it TO, given that the situation is the
>>> result of an interaction between three different GUCs and we have no
>>> way to distinguish which one(s) are the problem.
>>
>> "You need all three" covers it.
>
> Actually you need standby_connections and either archive_mode=on or
> max_wal_senders>0, I think.

Right.

First of all, I wonder why the latter two need to affect the decision of
whether additional information is written to WAL for HS. How about just
removing XLogIsNeeded() condition from XLogStandbyInfoActive()?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Sun, Apr 18, 2010 at 9:58 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Sun, Apr 18, 2010 at 7:52 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Sat, Apr 17, 2010 at 6:41 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>>> On Sat, 2010-04-17 at 17:44 -0400, Robert Haas wrote:
>>>
>>>> > I will change the error message.
>>>>
>>>> I gave a good deal of thought to trying to figure out a cleaner
>>>> solution to this problem than just changing the error message and
>>>> failed.  So let's change the error message.  Of course I'm not quite
>>>> sure what we should change it TO, given that the situation is the
>>>> result of an interaction between three different GUCs and we have no
>>>> way to distinguish which one(s) are the problem.
>>>
>>> "You need all three" covers it.
>>
>> Actually you need standby_connections and either archive_mode=on or
>> max_wal_senders>0, I think.
>
> Right.
>
> First of all, I wonder why the latter two need to affect the decision of
> whether additional information is written to WAL for HS. How about just
> removing XLogIsNeeded() condition from XLogStandbyInfoActive()?

Bad idea, I think. If XLogIsNeeded() is returning false and
XLogStandbyInfoActive() is returning true, the resulting WAL will
still be unusable for HS, at least AIUI.

...Robert

On Mon, Apr 19, 2010 at 11:04 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> First of all, I wonder why the latter two need to affect the decision of
>> whether additional information is written to WAL for HS. How about just
>> removing XLogIsNeeded() condition from XLogStandbyInfoActive()?
>
> Bad idea, I think.  If XLogIsNeeded() is returning false and
> XLogStandbyInfoActive() is returning true, the resulting WAL will
> still be unusable for HS, at least AIUI.

Probably No. Such a WAL will be usable for HS unless an unlogged
operation (e.g., CLUSTER, CREATE TABLE AS SELECT, etc) happens.
I think that the occurrence of an unlogged operation rather than
XLogIsNeeded() itself must be checked in the standby, it's already
been being checked. So just removing XLogIsNeeded() condition makes
sense to me.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Apr 19, 2010 at 5:31 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Mon, Apr 19, 2010 at 11:04 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> First of all, I wonder why the latter two need to affect the decision of
>>> whether additional information is written to WAL for HS. How about just
>>> removing XLogIsNeeded() condition from XLogStandbyInfoActive()?
>>
>> Bad idea, I think.  If XLogIsNeeded() is returning false and
>> XLogStandbyInfoActive() is returning true, the resulting WAL will
>> still be unusable for HS, at least AIUI.
>
> Probably No. Such a WAL will be usable for HS unless an unlogged
> operation (e.g., CLUSTER, CREATE TABLE AS SELECT, etc) happens.
> I think that the occurrence of an unlogged operation rather than
> XLogIsNeeded() itself must be checked in the standby, it's already
> been being checked. So just removing XLogIsNeeded() condition makes
> sense to me.

I think that's a bad idea. Currently we have three possible types of
WAL-logging:

- just enough for crash recovery (archive_mode=off and max_wal_senders=0)
- enough for log-shipping replication (archive_mode=on or
max_wal_senders>0, but recovery_connections=off)
- enough for log-shipping replication + hot standby (archive_mode=on
or max_wal_senders>0, plus recovery_connections=on)

I'm not eager to add a fourth category where hot standby works unless
you do any of the things that break log-streaming in general. That
seems hopelessly fragile and also fairly pointless.

...Robert