walreceiver settings Re: Streaming Replication patch for CommitFest 2009-09

Hi,

On Fri, Sep 18, 2009 at 7:34 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
> This approach is OK if the stand-alone walreceiver is treated steadily
> by the startup process like a child process under postmaster:
>
> * Handling of some interrupts: SIGHUP, SIGTERM?, SIGINT, SIGQUIT...
>   For example, the startup process would need to rethrow walreceiver
>   the interrupt from postmaster.
>
> * Communication with other child processes: stats collector? syslogger?...
>   For example, the log message generated by walreceiver should also
>   be collected by syslogger if requested.

Also we should consider how to give a GUC parameter to the stand-alone
walreceiver. In the initial patch, since walreceiver was a child process of
postmaster, it could easily get any GUC parameter. But it's not so easy
to give a GUC parameter to a stand-alone program.

I think that at least the following parameters should affect walreceiver:

* wal_sync_method
I want walreceiver to use fdatasync instead of fsync for performance
improvement. And other DBA might want to choose another method.

* fsync
I'm not surprised if someone wants to disable fsync in the standby.

* some parameters for logging
I think that the log messages generated by walreceiver should also be
treated as well as the other postgres messages. For example, I'd like
to specify log_line_prefix also for walreceiver.

There are some approaches to give a GUC parameter to walreceiver.
Which is the best?

1) Give a parameter as a command-line argument of the stand-alone
walreceiver. This is a straightforward approach, but wouldn't cover
a reload of parameter.

2) Give a parameter via pipe between the startup process and walreceiver.

3) Change walreceiver to read a configuration file. The problem is that
the command-line argument of postmaster doesn't affect walreceiver.
The combination of 1) and 3) might be required.

4) Change walreceiver back to a child process of postmaster.

Do you have any other better approach?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

Fujii Masao wrote:
> On Fri, Sep 18, 2009 at 7:34 PM, Fujii Masao <masao(dot)fujii(at)gmail(dot)com> wrote:
>> This approach is OK if the stand-alone walreceiver is treated steadily
>> by the startup process like a child process under postmaster:
>>
>> * Handling of some interrupts: SIGHUP, SIGTERM?, SIGINT, SIGQUIT...
>> For example, the startup process would need to rethrow walreceiver
>> the interrupt from postmaster.
>>
>> * Communication with other child processes: stats collector? syslogger?...
>> For example, the log message generated by walreceiver should also
>> be collected by syslogger if requested.
>
> Also we should consider how to give a GUC parameter to the stand-alone
> walreceiver. In the initial patch, since walreceiver was a child process of
> postmaster, it could easily get any GUC parameter. But it's not so easy
> to give a GUC parameter to a stand-alone program.

Yes, good point.

> * some parameters for logging
> I think that the log messages generated by walreceiver should also be
> treated as well as the other postgres messages. For example, I'd like
> to specify log_line_prefix also for walreceiver.

Hmm, agreed, although we already have the same problem with
archive_command, and pg_standby in particular. I could live with that
for now.

The startup process could capture stderr from walreceiver and forward it
with elog(LOG).

> There are some approaches to give a GUC parameter to walreceiver.
> Which is the best?
>
> 1) Give a parameter as a command-line argument of the stand-alone
> walreceiver. This is a straightforward approach, but wouldn't cover
> a reload of parameter.

The startup process could kill and restart walreceiver to reload. If
reloading is really required, that is. Which GUC parameters are we
concerned about? The ones related to logging you mentioned, but if we
handle logging via a pipe to the startup process, that won't be an issue.

> 2) Give a parameter via pipe between the startup process and walreceiver.
>
> 3) Change walreceiver to read a configuration file. The problem is that
> the command-line argument of postmaster doesn't affect walreceiver.
> The combination of 1) and 3) might be required.

Sounds complicated..

One option that you might well want to change on the fly is the
connection info string in recovery.conf. Neither of the above really
cater for that, unless we make walreceiver read recovery.conf as well. I
think we should keep walreceiver quite dumb.

> 4) Change walreceiver back to a child process of postmaster.

Yeah, that's not out of the question either.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Hi,

On Mon, Sep 21, 2009 at 1:55 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> The startup process could capture stderr from walreceiver and forward it
> with elog(LOG).

The startup process should obtain also the message level in some way
(pipe?), and control the messages according to it. It's confusing that
every messages are output with LOG level.

> The startup process could kill and restart walreceiver to reload. If
> reloading is really required, that is.

I think that it's confusing that SIGHUP kills a process. If walreceiver is
being monitored, the monitoring tool might raise a false alert.

> Which GUC parameters are we
> concerned about? The ones related to logging you mentioned, but if we
> handle logging via a pipe to the startup process, that won't be an issue.

wal_sync_method and fsync. At least I'd like to use fdatasync instead of
fsync for performance improvement.

> Sounds complicated..
>
> One option that you might well want to change on the fly is the
> connection info string in recovery.conf. Neither of the above really
> cater for that, unless we make walreceiver read recovery.conf as well. I
> think we should keep walreceiver quite dumb.

Agreed.

>> 4) Change walreceiver back to a child process of postmaster.
>
> Yeah, that's not out of the question either.

I like this simplest approach. But, as you pointed out, in the original
patch, the way to launch walreceiver is not robust. We need to add
some codes using examples from autovacuum launcher and worker.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center