Re: Parameter name standby_mode

We want to teach people that Hot Standby and Streaming Replication are
two different features. However, Streaming Replication calls its main
parameter "standby_mode" which reminds more of Hot Standby than of
Streaming Replication.

People could also run a warm standby without streaming replication,
which would result in a standby that has standby_mode = 'off'.

I found the parameter name confusing and I'd vote for changing its name.

Joachim

Joachim Wieland wrote:
> We want to teach people that Hot Standby and Streaming Replication are
> two different features.

I'm not sure about that, actually. Now that they're both in the tree,
they work nicely together and many users will think of them as one.

> However, Streaming Replication calls its main
> parameter "standby_mode" which reminds more of Hot Standby than of
> Streaming Replication.
>
> People could also run a warm standby without streaming replication,
> which would result in a standby that has standby_mode = 'off'.

If they want to implement the warm standby using the (new) built-in
logic to keep retrying restore_command, they would set
standby_mode='on'. standby_mode='on' doesn't imply streaming replication.

If you want to use pg_standby or similar tools, then you would indeed
set standby_mode='off', but I think that makes sense because you're
implementing the standby functionality outside the server in that case.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Feb 10, 2010 at 12:16 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> If they want to implement the warm standby using the (new) built-in
> logic to keep retrying restore_command, they would set
> standby_mode='on'. standby_mode='on' doesn't imply streaming replication.
>
> If you want to use pg_standby or similar tools, then you would indeed
> set standby_mode='off', but I think that makes sense because you're
> implementing the standby functionality outside the server in that case.

Okay, got it now with your explanations.

For some reason it didn't work before with standby_mode = 'on' (it
does now) and the warning "FATAL: sorry, too many standbys already"
gave me a first suspicion that SR is the only use case for this. Then
I checked the docs and there it said "If this parameter is on, the
streaming replication is enabled". I understand now what it does and
that it is a prerequisite but that there is also a non-SR use case...
So the name is okay for me :-)

Thanks again,
Joachim

On Wed, 2010-02-10 at 13:16 +0200, Heikki Linnakangas wrote:

> If they want to implement the warm standby using the (new) built-in
> logic to keep retrying restore_command, they would set
> standby_mode='on'. standby_mode='on' doesn't imply streaming replication.

The docs say "If this parameter is on, the streaming replication is
enabled". So who is wrong?

ISTM that Joachim's viewpoint is right and that most people will be
confused about this.

I think we need something named more intuitively. Something that better
describes what action (i.e. a verb) will occur when this is set.

Suggestions: streaming_replication = on
We may need to split out various complexities into multiple parameters,
or have valued parameters, e.g. standby_mode = REPLICA.

--
Simon Riggs www.2ndQuadrant.com

Simon Riggs wrote:
> The docs say "If this parameter is on, the streaming replication is
> enabled". So who is wrong?

The docs.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Feb 10, 2010 at 8:16 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> If they want to implement the warm standby using the (new) built-in
> logic to keep retrying restore_command, they would set
> standby_mode='on'. standby_mode='on' doesn't imply streaming replication.

But if we fail in restoring the archived WAL file, "standby_mode = on"
*always* tries to start streaming replication.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao wrote:
> On Wed, Feb 10, 2010 at 8:16 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> If they want to implement the warm standby using the (new) built-in
>> logic to keep retrying restore_command, they would set
>> standby_mode='on'. standby_mode='on' doesn't imply streaming replication.
>
> But if we fail in restoring the archived WAL file, "standby_mode = on"
> *always* tries to start streaming replication.

Hmm, somehow I thought it doesn't if you don't set primary_conninfo. I
think that's the way it should work, ie. if primary_conninfo is not set,
don't launch walreceiver but just keep trying to restore from the archive.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, Feb 12, 2010 at 3:19 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Fujii Masao wrote:
>> On Wed, Feb 10, 2010 at 8:16 PM, Heikki Linnakangas
>> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>>> If they want to implement the warm standby using the (new) built-in
>>> logic to keep retrying restore_command, they would set
>>> standby_mode='on'. standby_mode='on' doesn't imply streaming replication.
>>
>> But if we fail in restoring the archived WAL file, "standby_mode = on"
>> *always* tries to start streaming replication.
>
> Hmm, somehow I thought it doesn't if you don't set primary_conninfo. I
> think that's the way it should work, ie. if primary_conninfo is not set,
> don't launch walreceiver but just keep trying to restore from the archive.

Yeah, even if primary_conninfo is not given, the standby tries to invoke
walreceiver by using the another connection settings (environment variables
or defaults). This is intentional behavior, and would make the setup of SR
easier. So I'd like to leave it be.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Feb 12, 2010 at 7:28 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> Yeah, even if primary_conninfo is not given, the standby tries to invoke
> walreceiver by using the another connection settings (environment variables
> or defaults). This is intentional behavior, and would make the setup of SR
> easier. So I'd like to leave it be.

On the other hand, if it has to use defaults for the target host/port,
chances are high that either it connects to the wrong host/port or
that SR is just not wanted :-)

Whoever sets up SR will also take the effort to configure
primary_conninfo and will have a different primary than the default -
which I think is just the standby itself, no?

Joachim

Fujii Masao wrote:
> On Fri, Feb 12, 2010 at 3:19 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> Fujii Masao wrote:
>>> But if we fail in restoring the archived WAL file, "standby_mode = on"
>>> *always* tries to start streaming replication.
>> Hmm, somehow I thought it doesn't if you don't set primary_conninfo. I
>> think that's the way it should work, ie. if primary_conninfo is not set,
>> don't launch walreceiver but just keep trying to restore from the archive.
>
> Yeah, even if primary_conninfo is not given, the standby tries to invoke
> walreceiver by using the another connection settings (environment variables
> or defaults). This is intentional behavior, and would make the setup of SR
> easier. So I'd like to leave it be.

You could do primary_conninfo='' for that.

Maybe we should have two options, "streaming_mode='on'" and
"primary_conninfo='...'".

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, Feb 12, 2010 at 4:04 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Fujii Masao wrote:
>> On Fri, Feb 12, 2010 at 3:19 PM, Heikki Linnakangas
>> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>>> Fujii Masao wrote:
>>>> But if we fail in restoring the archived WAL file, "standby_mode = on"
>>>> *always* tries to start streaming replication.
>>> Hmm, somehow I thought it doesn't if you don't set primary_conninfo. I
>>> think that's the way it should work, ie. if primary_conninfo is not set,
>>> don't launch walreceiver but just keep trying to restore from the archive.
>>
>> Yeah, even if primary_conninfo is not given, the standby tries to invoke
>> walreceiver by using the another connection settings (environment variables
>> or defaults). This is intentional behavior, and would make the setup of SR
>> easier. So I'd like to leave it be.
>
> You could do primary_conninfo='' for that.
>
> Maybe we should have two options, "streaming_mode='on'" and
> "primary_conninfo='...'".

It looks better for me to extend the "standby_mode":
For example, standby_mode = 'streaming' or 'archive'.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao wrote:
> On Fri, Feb 12, 2010 at 4:04 PM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> Fujii Masao wrote:
>>> On Fri, Feb 12, 2010 at 3:19 PM, Heikki Linnakangas
>>> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>>>> Fujii Masao wrote:
>>>>> But if we fail in restoring the archived WAL file, "standby_mode = on"
>>>>> *always* tries to start streaming replication.
>>>> Hmm, somehow I thought it doesn't if you don't set primary_conninfo. I
>>>> think that's the way it should work, ie. if primary_conninfo is not set,
>>>> don't launch walreceiver but just keep trying to restore from the archive.
>>> Yeah, even if primary_conninfo is not given, the standby tries to invoke
>>> walreceiver by using the another connection settings (environment variables
>>> or defaults). This is intentional behavior, and would make the setup of SR
>>> easier. So I'd like to leave it be.
>> You could do primary_conninfo='' for that.
>>
>> Maybe we should have two options, "streaming_mode='on'" and
>> "primary_conninfo='...'".
>
> It looks better for me to extend the "standby_mode":
> For example, standby_mode = 'streaming' or 'archive'.

There's yet another mode that would be useful with hot standby: start up
the standby, but don't poll the archive and don't try to connect to the
master. Kind of 'paused' mode. Simon had functions to do that and more
in the original hot standby patch.

I've been thinking that this would work with just the three options we
have now:

standby_mode (true/false) controls whether the server keeps retrying
until trigger file is found (if trigger_file is set), rather than finish
recovery.

primary_conninfo (string) specifies a connection string to use to
connect to the master. If not given, don't try to connect.

restore_command (string) specifies a command to use to restore a file
from archive. If not given, don't try to restore files from archive.

I think this is pretty coherent and easy to explain, and makes all the
combinations restoring files from archive/streaming possible. But if
someone comes up with an even better scheme, I'm all ears.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Joachim Wieland wrote:
> On Fri, Feb 12, 2010 at 7:28 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> Yeah, even if primary_conninfo is not given, the standby tries to invoke
>> walreceiver by using the another connection settings (environment variables
>> or defaults). This is intentional behavior, and would make the setup of SR
>> easier. So I'd like to leave it be.
>
> On the other hand, if it has to use defaults for the target host/port,
> chances are high that either it connects to the wrong host/port or
> that SR is just not wanted :-)

Agreed. I've changed it now so that if primary_conninfo is not set, it
doesn't try to establish a streaming connection. If you want to get the
connection information from environment variables, you can use
primary_conninfo=''.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, Feb 12, 2010 at 4:59 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Agreed. I've changed it now so that if primary_conninfo is not set, it
> doesn't try to establish a streaming connection. If you want to get the
> connection information from environment variables, you can use
> primary_conninfo=''.

OK, you win. I would live with primary_conninfo=''.

And you need to change the document, recovery.conf.sample and
so on.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Feb 12, 2010 at 8:59 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Agreed. I've changed it now so that if primary_conninfo is not set, it
> doesn't try to establish a streaming connection. If you want to get the
> connection information from environment variables, you can use
> primary_conninfo=''.

Why not just remove the default:

If no primary_conninfo variable is set explicitly in the configuration
file, check the environment variables. If the environment variable is
not set, don't try to establish a connection.

?

Joachim

Joachim Wieland wrote:
> On Fri, Feb 12, 2010 at 8:59 AM, Heikki Linnakangas
> <heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
>> Agreed. I've changed it now so that if primary_conninfo is not set, it
>> doesn't try to establish a streaming connection. If you want to get the
>> connection information from environment variables, you can use
>> primary_conninfo=''.
>
> Why not just remove the default:
>
> If no primary_conninfo variable is set explicitly in the configuration
> file, check the environment variables. If the environment variable is
> not set, don't try to establish a connection.

The environment variables in question are the libpq environment
variables like PGHOST, PGPORT. The server shouldn't need to know about
them. Besides, there'd still be the corner case that you really want to
use the built-in defaults, ie. connect to a server running in the same
host at the default port, so you'd not set any environment variables either.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> There's yet another mode that would be useful with hot standby: start up
> the standby, but don't poll the archive and don't try to connect to the
> master. Kind of 'paused' mode. Simon had functions to do that and more
> in the original hot standby patch.

And having the pause/resume functions would lower the need for perfect
conflict resolution too. When you want to run this huge reporting query
set and not get interrupted, pause the standby. Afterward, resume it.

Of course, while paused, it's not a good HA standby anymore, but you
just did pause it, so you're not surprised, right?

> I've been thinking that this would work with just the three options we
> have now:

I like that, because it exposes exactly the code logic, and it is not
complex enough to merit being hidden from the users. Also, you depend on
understanding how the server really works to setup a trustworthy HA
solution, so exposing the very used concepts is a win.

> primary_conninfo (string) specifies a connection string to use to
> connect to the master. If not given, don't try to connect.

Would it be possible to expose that at the SQL level, so that you can
easily check in scripts what master you're a slave of? Think nagios
cascading alerts or topology graphs, etc.

Regards,
--
dim

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> Joachim Wieland wrote:
>> If no primary_conninfo variable is set explicitly in the configuration
>> file, check the environment variables. If the environment variable is
>> not set, don't try to establish a connection.

> The environment variables in question are the libpq environment
> variables like PGHOST, PGPORT. The server shouldn't need to know about
> them.

Even more to the point is that some of them, like PGPORT, are highly
likely to be set in a server's environment to point to the server
itself. It would be extremely dangerous to automatically try to start
replication just because we find those set. In fact, I would argue that
we should fix things so that any such variables inherited from the
server environment are intentionally *NOT* used for making SR
connections.

regards, tom lane

On Fri, Feb 12, 2010 at 11:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Even more to the point is that some of them, like PGPORT, are highly
> likely to be set in a server's environment to point to the server
> itself.  It would be extremely dangerous to automatically try to start
> replication just because we find those set.  In fact, I would argue that
> we should fix things so that any such variables inherited from the
> server environment are intentionally *NOT* used for making SR
> connections.

There are many environment variables which libpq automatically uses.
Which variables should not be used for SR connection? All?

If both primary_conninfo and environment variables are not given,
the default value (e.g., port = 5432) is automatically used for SR
connection. Is this OK? or NG as well as the environment variables?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Fri, Feb 12, 2010 at 4:59 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Joachim Wieland wrote:
>> On Fri, Feb 12, 2010 at 7:28 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> Yeah, even if primary_conninfo is not given, the standby tries to invoke
>>> walreceiver by using the another connection settings (environment variables
>>> or defaults). This is intentional behavior, and would make the setup of SR
>>> easier. So I'd like to leave it be.
>>
>> On the other hand, if it has to use defaults for the target host/port,
>> chances are high that either it connects to the wrong host/port or
>> that SR is just not wanted :-)
>
> Agreed. I've changed it now so that if primary_conninfo is not set, it
> doesn't try to establish a streaming connection. If you want to get the
> connection information from environment variables, you can use
> primary_conninfo=''.

If standby_mode is enabled, and neither primary_conninfo nor restore_command
are set, the standby would get stuck. How about forbidding (i.e., causing a
FATAL message) this wrong setting?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Feb 24, 2010 at 2:18 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> If standby_mode is enabled, and neither primary_conninfo nor restore_command
> are set, the standby would get stuck. How about forbidding (i.e., causing a
> FATAL message) this wrong setting?

Here is the patch which forbids that wrong setting of recovery.conf.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Mar 3, 2010 at 9:41 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Wed, Feb 24, 2010 at 2:18 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> If standby_mode is enabled, and neither primary_conninfo nor restore_command
>> are set, the standby would get stuck. How about forbidding (i.e., causing a
>> FATAL message) this wrong setting?
>
> Here is the patch which forbids that wrong setting of recovery.conf.

I think that this patch should be applied. Otherwise, if you wrongly
set neither primary_conninfo nor restore_command in recovery.conf,
the standby server would do nothing and get stuck because it doesn't
know where to retrieve the WAL files from. Banning the incorrect
setting makes sense to me.

Does anyone commit the patch? Does anyone have a say?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, Mar 30, 2010 at 12:26 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Wed, Mar 3, 2010 at 9:41 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> On Wed, Feb 24, 2010 at 2:18 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>>> If standby_mode is enabled, and neither primary_conninfo nor restore_command
>>> are set, the standby would get stuck. How about forbidding (i.e., causing a
>>> FATAL message) this wrong setting?
>>
>> Here is the patch which forbids that wrong setting of recovery.conf.
>
> I think that this patch should be applied. Otherwise, if you wrongly
> set neither primary_conninfo nor restore_command in recovery.conf,
> the standby server would do nothing and get stuck because it doesn't
> know where to retrieve the WAL files from. Banning the incorrect
> setting makes sense to me.
>
> Does anyone commit the patch? Does anyone have a say?

I just tested this and it seems to just sit there doing this over and
over again:

LOG: record with zero length at 0/3006B28

I'm not sure that we should forbid this configuration, but the current
behavior doesn't seem right either. ISTM that, in the absence of a
way to get any more WAL, it would be reasonable for the standby server
to just start up and sit there in recovery mode but without actually
advancing recovery, but the repeated log messages are pretty annoying.
If we're connected in streaming mode and there is no activity on the
primary, we don't emit logs of this type, so it doesn't seem like we
should do that if there is no primary either.

A related question is... do we ever reload recovery.conf? I tried
adding the setting to recovery.conf and doing pg_ctl reload, and it
says that it's "reloading configuration files", but doesn't pick up
the new setting. :-(

...Robert

On Wed, Mar 31, 2010 at 12:21 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> I just tested this and it seems to just sit there doing this over and
> over again:
>
> LOG:  record with zero length at 0/3006B28
>
> I'm not sure that we should forbid this configuration, but the current
> behavior doesn't seem right either.  ISTM that, in the absence of a
> way to get any more WAL, it would be reasonable for the standby server
> to just start up and sit there in recovery mode but without actually
> advancing recovery, but the repeated log messages are pretty annoying.

I'm concerned about that the configuration might prevent the standby
from accepting connection from a client because it cannot get the WAL
for making the database consistent. So that configuration seems to be
reasonable only when starting the standby from the already-consistent
database or with enough WAL files in pg_xlog. But it seems to me that
the standby often starts from the inconsistent database without enough
WAL in pg_xlog.

> A related question is... do we ever reload recovery.conf?  I tried
> adding the setting to recovery.conf and doing pg_ctl reload, and it
> says that it's "reloading configuration files", but doesn't pick up
> the new setting.  :-(

recovery.conf cannot be reloaded while the server is running. This
restriction should be removed in the future release, I think.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Mar 31, 2010 at 1:47 AM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Wed, Mar 31, 2010 at 12:21 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I just tested this and it seems to just sit there doing this over and
>> over again:
>>
>> LOG:  record with zero length at 0/3006B28
>>
>> I'm not sure that we should forbid this configuration, but the current
>> behavior doesn't seem right either.  ISTM that, in the absence of a
>> way to get any more WAL, it would be reasonable for the standby server
>> to just start up and sit there in recovery mode but without actually
>> advancing recovery, but the repeated log messages are pretty annoying.
>
> I'm concerned about that the configuration might prevent the standby
> from accepting connection from a client because it cannot get the WAL
> for making the database consistent. So that configuration seems to be
> reasonable only when starting the standby from the already-consistent
> database or with enough WAL files in pg_xlog. But it seems to me that
> the standby often starts from the inconsistent database without enough
> WAL in pg_xlog.

Agreed. I think if the server starts up in standby mode and it is an
inconsistent state with no source of WAL, then the startup process
should exit with a suitable error message, which AIUI will result in
the whole server shutting down. However if there is no source of WAL
but the server is in a consistent state, then I think we should allow
it to start up as a read-only standby.

Now, an interesting question is - if the server is in this state, and
somebody manually drops more WAL into pg_xlog, what happens? And what
happens in the similar case where primary_conninfo is set but we can't
connect to the master at the moment, and someone drops a pile of WAL
on us?

>> A related question is... do we ever reload recovery.conf?  I tried
>> adding the setting to recovery.conf and doing pg_ctl reload, and it
>> says that it's "reloading configuration files", but doesn't pick up
>> the new setting.  :-(
>
> recovery.conf cannot be reloaded while the server is running. This
> restriction should be removed in the future release, I think.

Yes. If we don't already have a TODO for that, we should definitely
add one. I found myself annoyed by this several times last night. I
kept having to restart the master, too, first to fix archive_mode and
then to fix max_wal_senders. It's far too late to start tinkering
with this stuff now but I am pretty confident there will be a huge
sigh of collective relief out there if we can relax some of these
restrictions for 9.1. Nobody likes having to shut down the server,
even if it's just for a few seconds.

...Robert

Robert Haas wrote:
> Agreed. I think if the server starts up in standby mode and it is an
> inconsistent state with no source of WAL, then the startup process
> should exit with a suitable error message, which AIUI will result in
> the whole server shutting down. However if there is no source of WAL
> but the server is in a consistent state, then I think we should allow
> it to start up as a read-only standby.
>
> Now, an interesting question is - if the server is in this state, and
> somebody manually drops more WAL into pg_xlog, what happens? And what
> happens in the similar case where primary_conninfo is set but we can't
> connect to the master at the moment, and someone drops a pile of WAL
> on us?

With the recent changes to the retry logic
(http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
they will be replayed. Even if neither primary_conninfo or
restore_command is given, the server will still keep polling pg_xlog,
and if you copy a WAL file to standby's pg_xlog directory, it will be
replayed and recovery will make progress.

I wouldn't recommend setting up a standby server like that, but it's not
totally unreasonable. So the standby always has a potential source of
WAL, pg_xlog.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Mar 31, 2010 at 4:54 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Robert Haas wrote:
>> Agreed.  I think if the server starts up in standby mode and it is an
>> inconsistent state with no source of WAL, then the startup process
>> should exit with a suitable error message, which AIUI will result in
>> the whole server shutting down.  However if there is no source of WAL
>> but the server is in a consistent state, then I think we should allow
>> it to start up as a read-only standby.
>>
>> Now, an interesting question is - if the server is in this state, and
>> somebody manually drops more WAL into pg_xlog, what happens? And what
>> happens in the similar case where primary_conninfo is set but we can't
>> connect to the master at the moment, and someone drops a pile of WAL
>> on us?
>
> With the recent changes to the retry logic
> (http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
> they will be replayed. Even if neither primary_conninfo or
> restore_command is given, the server will still keep polling pg_xlog,
> and if you copy a WAL file to standby's pg_xlog directory, it will be
> replayed and recovery will make progress.
>
> I wouldn't recommend setting up a standby server like that, but it's not
> totally unreasonable. So the standby always has a potential source of
> WAL, pg_xlog.

OK.

Is it reasonable to think that we can find a way to make it not print
the duplicate messages over and over again?

LOG: record with zero length at 0/3006B28

Maybe only print that if the location has advanced since the last such message?

Should we make it shut down if it can't immediately read enough WAL to
get to a consistent state, or just figure it's the user's job to fix
it?

...Robert

Robert Haas wrote:
> Is it reasonable to think that we can find a way to make it not print
> the duplicate messages over and over again?
>
> LOG: record with zero length at 0/3006B28
>
> Maybe only print that if the location has advanced since the last such message?

Yeah, seems reasonable.

> Should we make it shut down if it can't immediately read enough WAL to
> get to a consistent state, or just figure it's the user's job to fix
> it?

I'd say no. In testing, I have done this many times:

pg_start_backup()
copy data directory to server
create recovery.conf
Start standby server.
pg_stop_backup()

The standby doesn't reach consistency before it sees the end-of-backup
record written by pg_stop_backup(), but it does replay up to the last
WAL segment, and connect to the master.

Not sure if that's useful in real life, but there could be situations
where restore_command isn't totally reliable, for example, and it's good
to keep trying.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Mar 31, 2010 at 5:23 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Robert Haas wrote:
>> Is it reasonable to think that we can find a way to make it not print
>> the duplicate messages over and over again?
>>
>> LOG:  record with zero length at 0/3006B28
>>
>> Maybe only print that if the location has advanced since the last such message?
>
> Yeah, seems reasonable.
>
>> Should we make it shut down if it can't immediately read enough WAL to
>> get to a consistent state, or just figure it's the user's job to fix
>> it?
>
> I'd say no. In testing, I have done this many times:
>
> pg_start_backup()
> copy data directory to server
> create recovery.conf
> Start standby server.
> pg_stop_backup()
>
> The standby doesn't reach consistency before it sees the end-of-backup
> record written by pg_stop_backup(), but it does replay up to the last
> WAL segment, and connect to the master.
>
> Not sure if that's useful in real life, but there could be situations
> where restore_command isn't totally reliable, for example, and it's good
> to keep trying.

I was only thinking of doing it in the case where there's no
primary_conninfo or restore_command.

...Robert

On Thu, Apr 1, 2010 at 6:04 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> I wouldn't recommend setting up a standby server like that, but it's not
>> totally unreasonable. So the standby always has a potential source of
>> WAL, pg_xlog.
>
> OK.

OK, too. I turn down the patch.

> Is it reasonable to think that we can find a way to make it not print
> the duplicate messages over and over again?
>
> LOG:  record with zero length at 0/3006B28
>
> Maybe only print that if the location has advanced since the last such message?

Agreed. But what log message is repeated depends on the situation.
So message without any location might be output. BTW, In my testing,
the following message was repeated.

LOG: invalid magic number 0000 in log file 0, segment 14, offset 9617408

> Should we make it shut down if it can't immediately read enough WAL to
> get to a consistent state, or just figure it's the user's job to fix
> it?

I think that it's difficult for the user to fix it. So I agree to shut
down the server in that case, i.e., throw a FATAL when an invalid WAL
record is found and recovery hasn't reached the safe starting point
even if neither primary_conninfo nor restore_command is given.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, Mar 31, 2010 at 9:01 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> Agreed. But what log message is repeated depends on the situation.
> So message without any location might be output. BTW, In my testing,
> the following message was repeated.
>
>    LOG:  invalid magic number 0000 in log file 0, segment 14, offset 9617408

Yeah, that's a pain in the neck. We need to think about a way to
avoid any of these messages repeating. Not sure how, off the top of
my head.

>> Should we make it shut down if it can't immediately read enough WAL to
>> get to a consistent state, or just figure it's the user's job to fix
>> it?
>
> I think that it's difficult for the user to fix it. So I agree to shut
> down the server in that case, i.e., throw a FATAL when an invalid WAL
> record is found and recovery hasn't reached the safe starting point
> even if neither primary_conninfo nor restore_command is given.

I think that's reasonable. It's not like this should cause any
problem for the user: they can add the missing WAL while the server is
down just as well as they could if it were up, and Hot Standby isn't
going to come up anyway. But I could possibly be persuaded to change
my mind on this one, if someone feels strongly otherwise.

...Robert

On Wed, 2010-03-31 at 23:54 +0300, Heikki Linnakangas wrote:
> Robert Haas wrote:
> > Agreed. I think if the server starts up in standby mode and it is an
> > inconsistent state with no source of WAL, then the startup process
> > should exit with a suitable error message, which AIUI will result in
> > the whole server shutting down. However if there is no source of WAL
> > but the server is in a consistent state, then I think we should allow
> > it to start up as a read-only standby.
> >
> > Now, an interesting question is - if the server is in this state, and
> > somebody manually drops more WAL into pg_xlog, what happens? And what
> > happens in the similar case where primary_conninfo is set but we can't
> > connect to the master at the moment, and someone drops a pile of WAL
> > on us?
>
> With the recent changes to the retry logic
> (http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
> they will be replayed. Even if neither primary_conninfo or
> restore_command is given, the server will still keep polling pg_xlog,
> and if you copy a WAL file to standby's pg_xlog directory, it will be
> replayed and recovery will make progress.
>
> I wouldn't recommend setting up a standby server like that, but it's not
> totally unreasonable. So the standby always has a potential source of
> WAL, pg_xlog.

I have inadvertently made it impossible to specify
standby_mode && (!primary_conninfo && !restore_command)

I did that because Robert had separately to this thread reported a hang,
caused by this specification. I have verified this.

pg_xlog is a *potential* source of WAL, but if the files requested are
not present then the server just sits and waits with *no* messages. That
is unacceptable, IMHO.

What should we do now?

--
Simon Riggs www.2ndQuadrant.com

On Mon, Feb 15, 2010 at 3:45 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> On Fri, Feb 12, 2010 at 11:46 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Even more to the point is that some of them, like PGPORT, are highly
>> likely to be set in a server's environment to point to the server
>> itself.  It would be extremely dangerous to automatically try to start
>> replication just because we find those set.  In fact, I would argue that
>> we should fix things so that any such variables inherited from the
>> server environment are intentionally *NOT* used for making SR
>> connections.

This Tom's complaint is listed as a TODO item. How should we treat this?

I'm leaning toward postponing the item to v9.1 or later. Currently the
server during recovery doesn't accept the replication connection. So
it's not so dangerous for walreceiver to use the environment variables
which might point to the server itself, I think. That connection is
always refused.

Let us revisit this issue when we allow the standby server to accept the
replication connection from another standby? And I think that we should
prevent the standby from accepting the connection from its walreceiver,
rather than prevent the standby from using the environment variables.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Mon, Apr 5, 2010 at 4:46 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, 2010-03-31 at 23:54 +0300, Heikki Linnakangas wrote:
>> Robert Haas wrote:
>> > Agreed.  I think if the server starts up in standby mode and it is an
>> > inconsistent state with no source of WAL, then the startup process
>> > should exit with a suitable error message, which AIUI will result in
>> > the whole server shutting down.  However if there is no source of WAL
>> > but the server is in a consistent state, then I think we should allow
>> > it to start up as a read-only standby.
>> >
>> > Now, an interesting question is - if the server is in this state, and
>> > somebody manually drops more WAL into pg_xlog, what happens? And what
>> > happens in the similar case where primary_conninfo is set but we can't
>> > connect to the master at the moment, and someone drops a pile of WAL
>> > on us?
>>
>> With the recent changes to the retry logic
>> (http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
>> they will be replayed. Even if neither primary_conninfo or
>> restore_command is given, the server will still keep polling pg_xlog,
>> and if you copy a WAL file to standby's pg_xlog directory, it will be
>> replayed and recovery will make progress.
>>
>> I wouldn't recommend setting up a standby server like that, but it's not
>> totally unreasonable. So the standby always has a potential source of
>> WAL, pg_xlog.
>
> I have inadvertently made it impossible to specify
>   standby_mode && (!primary_conninfo && !restore_command)
>
> I did that because Robert had separately to this thread reported a hang,
> caused by this specification. I have verified this.

I don't remember reporting this (or maybe you meant the other Robert);
but there are so many threads on this topic that it's hard to keep
track of them all. Can you refresh my memory?

> pg_xlog is a *potential* source of WAL, but if the files requested are
> not present then the server just sits and waits with *no* messages. That
> is unacceptable, IMHO.
>
> What should we do now?

Well, actually, what it does for me is sits there and prints the last
xlog location over and over again every 2s. I'd actually like to get
to "sits and waits with no messages", but it's not clear how to do
that.

...Robert

On Mon, 2010-04-05 at 07:11 -0400, Robert Haas wrote:
> On Mon, Apr 5, 2010 at 4:46 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > On Wed, 2010-03-31 at 23:54 +0300, Heikki Linnakangas wrote:
> >> Robert Haas wrote:
> >> > Agreed. I think if the server starts up in standby mode and it is an
> >> > inconsistent state with no source of WAL, then the startup process
> >> > should exit with a suitable error message, which AIUI will result in
> >> > the whole server shutting down. However if there is no source of WAL
> >> > but the server is in a consistent state, then I think we should allow
> >> > it to start up as a read-only standby.
> >> >
> >> > Now, an interesting question is - if the server is in this state, and
> >> > somebody manually drops more WAL into pg_xlog, what happens? And what
> >> > happens in the similar case where primary_conninfo is set but we can't
> >> > connect to the master at the moment, and someone drops a pile of WAL
> >> > on us?
> >>
> >> With the recent changes to the retry logic
> >> (http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
> >> they will be replayed. Even if neither primary_conninfo or
> >> restore_command is given, the server will still keep polling pg_xlog,
> >> and if you copy a WAL file to standby's pg_xlog directory, it will be
> >> replayed and recovery will make progress.
> >>
> >> I wouldn't recommend setting up a standby server like that, but it's not
> >> totally unreasonable. So the standby always has a potential source of
> >> WAL, pg_xlog.
> >
> > I have inadvertently made it impossible to specify
> > standby_mode && (!primary_conninfo && !restore_command)
> >
> > I did that because Robert had separately to this thread reported a hang,
> > caused by this specification. I have verified this.
>
> I don't remember reporting this (or maybe you meant the other Robert);
> but there are so many threads on this topic that it's hard to keep
> track of them all. Can you refresh my memory?
>
> > pg_xlog is a *potential* source of WAL, but if the files requested are
> > not present then the server just sits and waits with *no* messages. That
> > is unacceptable, IMHO.
> >
> > What should we do now?
>
> Well, actually, what it does for me is sits there and prints the last
> xlog location over and over again every 2s. I'd actually like to get
> to "sits and waits with no messages", but it's not clear how to do
> that.

That's exactly the opposite of your report. Thread you started, on
hackers, in last week or so.

It's not clear to me *why* you would want it to sit there doing nothing,
and even if that has a purpose, saying nothing at all is not useful.
(Note that it cannot enter Hot Standby mode even in that state).

--
Simon Riggs www.2ndQuadrant.com

On Mon, Apr 5, 2010 at 7:34 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Mon, 2010-04-05 at 07:11 -0400, Robert Haas wrote:
>> On Mon, Apr 5, 2010 at 4:46 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
>> > On Wed, 2010-03-31 at 23:54 +0300, Heikki Linnakangas wrote:
>> >> Robert Haas wrote:
>> >> > Agreed.  I think if the server starts up in standby mode and it is an
>> >> > inconsistent state with no source of WAL, then the startup process
>> >> > should exit with a suitable error message, which AIUI will result in
>> >> > the whole server shutting down.  However if there is no source of WAL
>> >> > but the server is in a consistent state, then I think we should allow
>> >> > it to start up as a read-only standby.
>> >> >
>> >> > Now, an interesting question is - if the server is in this state, and
>> >> > somebody manually drops more WAL into pg_xlog, what happens? And what
>> >> > happens in the similar case where primary_conninfo is set but we can't
>> >> > connect to the master at the moment, and someone drops a pile of WAL
>> >> > on us?
>> >>
>> >> With the recent changes to the retry logic
>> >> (http://archives.postgresql.org/pgsql-committers/2010-03/msg00356.php),
>> >> they will be replayed. Even if neither primary_conninfo or
>> >> restore_command is given, the server will still keep polling pg_xlog,
>> >> and if you copy a WAL file to standby's pg_xlog directory, it will be
>> >> replayed and recovery will make progress.
>> >>
>> >> I wouldn't recommend setting up a standby server like that, but it's not
>> >> totally unreasonable. So the standby always has a potential source of
>> >> WAL, pg_xlog.
>> >
>> > I have inadvertently made it impossible to specify
>> >   standby_mode && (!primary_conninfo && !restore_command)
>> >
>> > I did that because Robert had separately to this thread reported a hang,
>> > caused by this specification. I have verified this.
>>
>> I don't remember reporting this (or maybe you meant the other Robert);
>> but there are so many threads on this topic that it's hard to keep
>> track of them all.  Can you refresh my memory?
>>
>> > pg_xlog is a *potential* source of WAL, but if the files requested are
>> > not present then the server just sits and waits with *no* messages. That
>> > is unacceptable, IMHO.
>> >
>> > What should we do now?
>>
>> Well, actually, what it does for me is sits there and prints the last
>> xlog location over and over again every 2s.  I'd actually like to get
>> to "sits and waits with no messages", but it's not clear how to do
>> that.
>
> That's exactly the opposite of your report. Thread you started, on
> hackers, in last week or so.

Which thread? What was the subject line? The only thing I remember
saying about this was:

http://archives.postgresql.org/pgsql-hackers/2010-03/msg01247.php

> It's not clear to me *why* you would want it to sit there doing nothing,
> and even if that has a purpose, saying nothing at all is not useful.
> (Note that it cannot enter Hot Standby mode even in that state).

Actually it can, if the database state is consistent. Anyway, this
was already discussed upthread... feel free to put in your $0.02.

...Robert

On Mon, 2010-04-05 at 18:03 +0900, Fujii Masao wrote:

> I'm leaning toward postponing the item to v9.1 or later.

If you want to defer anything, then I'd like to get a summary of what
you are thinking of deferring and why that is acceptable. Right now
there are lots of unfinished items and no movement on them. Yes, I'm
unhappy about that.

My feeling is that we should only default on "port" as part of the
primary_conninfo. All other settings are required or we should reject
streaming. Replication connections should be explicit.

--
Simon Riggs www.2ndQuadrant.com