beta3 & the open items list

It would be nice to get beta3 out the door sooner rather than later,
but I sort of feel like we're not ready yet. In fact, we seem to be a
bit stalled. The open items list currently lists four items.

1. max_standby_delay. Tom has committed to getting this done, but has
been tied up with non-PostgreSQL related work for the last few weeks.

2. infinite repeat of warning message in standby. Heikki changed the
code so this isn't a tight loop any more, which is an improvement, but
we've discussed the fact that retrying forever may not be the best
behavior.

http://archives.postgresql.org/pgsql-hackers/2010-06/msg00806.php
http://archives.postgresql.org/pgsql-hackers/2010-06/msg00838.php

I am not clear, however, on how difficult it is to implement the
proposed behavior, and I'm not sure Heikki's on board with the
proposed change.

3. supply alternate hstore operator for equals-greater in preparation
for later user in function parameter assignment. There's some work
left to be done here but it's pretty minor. Mostly we're arguing
about whether to call the hstore slice operator +> or & or % or %> --
I've written three patches to rename it so far (to three different
alternative names), one of which I committed, and there's still
ongoing discussion as to whether to rename it again and/or remove it.
Aside from that, we need to deal with the singleton-hstore constructor
(text => text); I believe the consensus there is to remove the
operator in favor of the underlying hstore(text, text) function and
backpatch that function name into the back-branches to facilitate
writing hstore code that is portable across major PostgreSQL releases.

4. Streaming Replication needs to detect death of master. We need
some sort of keep-alive, here. Whether it's at the TCP level (as
advocated by Tom Lane and others) or at the protocol level (as
advocated by Greg Stark) is something that we have yet to decide; once
it's decided, someone will need to do it...

It would be nice if we could make a final push to get these issues
resolved and another beta out the door before the end of the month...

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:

> 4. Streaming Replication needs to detect death of master. We need
> some sort of keep-alive, here. Whether it's at the TCP level (as
> advocated by Tom Lane and others) or at the protocol level (as
> advocated by Greg Stark) is something that we have yet to decide; once
> it's decided, someone will need to do it...

TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
humbly suggest we *not* be pedantic and implement something practical
and less prone to variables outside the control of Pg.

Sincerely,

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering

On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> 4. Streaming Replication needs to detect death of master.  We need
> some sort of keep-alive, here.  Whether it's at the TCP level (as
> advocated by Tom Lane and others) or at the protocol level (as
> advocated by Greg Stark) is something that we have yet to decide; once
> it's decided, someone will need to do it...

This sounds like a useful feature but I don't see why it's not 9.1
material. The status quo is that the expected usage pattern is manual
failover. As long as the slave responds to manual intervention when in
this state I don't think this is a blocking issue. Monitoring and
automatic failover are clearly things we plan to add features to
handle better in the future.

--
greg

On Sat, Jun 19, 2010 at 2:46 PM, Greg Stark <gsstark(at)mit(dot)edu> wrote:
> On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> 4. Streaming Replication needs to detect death of master.  We need
>> some sort of keep-alive, here.  Whether it's at the TCP level (as
>> advocated by Tom Lane and others) or at the protocol level (as
>> advocated by Greg Stark) is something that we have yet to decide; once
>> it's decided, someone will need to do it...
>
> This sounds like a useful feature but I don't see why it's not 9.1
> material. The status quo is that the expected usage pattern is manual
> failover. As long as the slave responds to manual intervention when in
> this state I don't think this is a blocking issue. Monitoring and
> automatic failover are clearly things we plan to add features to
> handle better in the future.

Right now, if the SR master reboots unexpectedly (say, power plug pull
and restart), the slave never notices. It just sits there forever
waiting for the next byte of data from the master to arrive (which it
never will). You have to manually restart the server or hit
walreceiver with a SIGTERM to get it to start streaming agian. I
guess we could decide we're just not going to deal with that, but it
seems like a fairly large misfeature to me.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Right now, if the SR master reboots unexpectedly (say, power plug pull
> and restart), the slave never notices. It just sits there forever
> waiting for the next byte of data from the master to arrive (which it
> never will).

This is nonsense --- the slave's kernel *will* eventually notice that
the TCP connection is dead, and tell walreceiver so. I don't doubt
that the standard TCP timeout is longer than people want to wait for
that, but claiming that it will never happen is simply wrong.

I think that enabling slave-side TCP keepalives and control of the
keepalive timeout parameters is probably sufficient for 9.0 here.

regards, tom lane

On Saturday 19 June 2010 18:05:34 Joshua D. Drake wrote:
> On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:
> > 4. Streaming Replication needs to detect death of master. We need
> > some sort of keep-alive, here. Whether it's at the TCP level (as
> > advocated by Tom Lane and others) or at the protocol level (as
> > advocated by Greg Stark) is something that we have yet to decide; once
> > it's decided, someone will need to do it...
>
> TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
> humbly suggest we *not* be pedantic and implement something practical
> and less prone to variables outside the control of Pg.
And has the huge advantage of being implementable in about 5 lines of C
(setsockopt + error checking). Considering what time in the release cycle this
is...

Andres

On 06/19/2010 09:13 PM, Tom Lane wrote:
> Robert Haas<robertmhaas(at)gmail(dot)com> writes:
>> Right now, if the SR master reboots unexpectedly (say, power plug pull
>> and restart), the slave never notices. It just sits there forever
>> waiting for the next byte of data from the master to arrive (which it
>> never will).
>
> This is nonsense --- the slave's kernel *will* eventually notice that
> the TCP connection is dead, and tell walreceiver so. I don't doubt
> that the standard TCP timeout is longer than people want to wait for
> that, but claiming that it will never happen is simply wrong.
>
> I think that enabling slave-side TCP keepalives and control of the
> keepalive timeout parameters is probably sufficient for 9.0 here.

yeah I would agree - we do have tcp keepalive code in the backend for a
while now and adding that to libpq as well just seems like an easy
enough fix at this time in the release cycle.

Stefan

On Jun 19, 2010, at 21:13 , Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Right now, if the SR master reboots unexpectedly (say, power plug pull
>> and restart), the slave never notices. It just sits there forever
>> waiting for the next byte of data from the master to arrive (which it
>> never will).
>
> This is nonsense --- the slave's kernel *will* eventually notice that
> the TCP connection is dead, and tell walreceiver so. I don't doubt
> that the standard TCP timeout is longer than people want to wait for
> that, but claiming that it will never happen is simply wrong.

No, Robert is correct AFAIK. If you're *waiting* for data, TCP generates no traffic (expect with keepalive enabled). From the slave's kernel POV, a dead master is therefore indistinguishable from a inactive master.

Things are different from a sender's POV, though. Since sent data is ACK'ed by the receiving end, the TCP stack can (and does) detect a broken connection.

best regards,
Florian Pflug

On Sat, 2010-06-19 at 14:53 -0400, Robert Haas wrote:
> On Sat, Jun 19, 2010 at 2:46 PM, Greg Stark <gsstark(at)mit(dot)edu> wrote:
> > On Sat, Jun 19, 2010 at 2:43 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >> 4. Streaming Replication needs to detect death of master. We need
> >> some sort of keep-alive, here. Whether it's at the TCP level (as
> >> advocated by Tom Lane and others) or at the protocol level (as
> >> advocated by Greg Stark) is something that we have yet to decide; once
> >> it's decided, someone will need to do it...
> >
> > This sounds like a useful feature but I don't see why it's not 9.1
> > material. The status quo is that the expected usage pattern is manual
> > failover. As long as the slave responds to manual intervention when in
> > this state I don't think this is a blocking issue. Monitoring and
> > automatic failover are clearly things we plan to add features to
> > handle better in the future.
>
> Right now, if the SR master reboots unexpectedly (say, power plug pull
> and restart), the slave never notices. It just sits there forever
> waiting for the next byte of data from the master to arrive (which it
> never will). You have to manually restart the server or hit
> walreceiver with a SIGTERM to get it to start streaming agian. I
> guess we could decide we're just not going to deal with that, but it
> seems like a fairly large misfeature to me.

Are you saying it doesn't respond to a trigger file any any point? That
would be a problem.

Sounds like we should have a pg_restart_walreceiver() function. We
shouldn't be encouraging people to send signals to backends, its too
easy to get wrong.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Training and Services

Florian Pflug <fgp(at)phlo(dot)org> writes:
> On Jun 19, 2010, at 21:13 , Tom Lane wrote:
>> This is nonsense --- the slave's kernel *will* eventually notice that
>> the TCP connection is dead, and tell walreceiver so. I don't doubt
>> that the standard TCP timeout is longer than people want to wait for
>> that, but claiming that it will never happen is simply wrong.

> No, Robert is correct AFAIK. If you're *waiting* for data, TCP
> generates no traffic (expect with keepalive enabled).

Mph. I was thinking that keepalive was on by default with a very long
interval, but I see this isn't so. However, if we enable keepalive,
then it's irrelevant to the point anyway. Nobody's produced any
evidence that keepalive is an unsuitable solution.

regards, tom lane

On Saturday 19 June 2010 18:05:34 Joshua D. Drake wrote:
> On Sat, 2010-06-19 at 09:43 -0400, Robert Haas wrote:
> > 4. Streaming Replication needs to detect death of master. We need
> > some sort of keep-alive, here. Whether it's at the TCP level (as
> > advocated by Tom Lane and others) or at the protocol level (as
> > advocated by Greg Stark) is something that we have yet to decide; once
> > it's decided, someone will need to do it...
>
> TCP involves unknowns, such as firewalls, vpn routers and ssh tunnels. I
> humbly suggest we *not* be pedantic and implement something practical
> and less prone to variables outside the control of Pg.
>
> Sincerely,
>++++ +
> Joshua D. Drake

On Jun 20, 2010, at 7:18 , Tom Lane wrote:
> Florian Pflug <fgp(at)phlo(dot)org> writes:
>> On Jun 19, 2010, at 21:13 , Tom Lane wrote:
>>> This is nonsense --- the slave's kernel *will* eventually notice that
>>> the TCP connection is dead, and tell walreceiver so. I don't doubt
>>> that the standard TCP timeout is longer than people want to wait for
>>> that, but claiming that it will never happen is simply wrong.
>
>> No, Robert is correct AFAIK. If you're *waiting* for data, TCP
>> generates no traffic (expect with keepalive enabled).
>
> Mph. I was thinking that keepalive was on by default with a very long
> interval, but I see this isn't so. However, if we enable keepalive,
> then it's irrelevant to the point anyway. Nobody's produced any
> evidence that keepalive is an unsuitable solution.

Yeah, I agree. Just enabling keepalive should suffice for 9.0.

BTW, the postmaster already enables keepalive on incoming connections in StreamConnection() - presumably to prevent crashed clients from occupying a backend process forever. So there's even a clear precedent for doing so, and proof that it doesn't cause any harm.

best regards,
Florian Pflug

On 19 June 2010 14:43, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> It would be nice if we could make a final push to get these issues
> resolved and another beta out the door before the end of the month...

So should we expect beta3 imminently, or are these issues still outstanding?

Thanks

Thom