Re: Synchronous replication & Hot standby patches

Hi,

Could you please let me know what are the outstanding features that are still to be developed in the respective patches?

I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.

Thanks,
Niranjan

K, Niranjan (NSN - IN/Bangalore) wrote:
> Hi,
>
> Could you please let me know what are the outstanding features
> that are still to be developed in the respective patches?
>
> I'am currently referring the wiki: "Todo and Claim" for NTT and
> for HotStandby, i see that almost all issues are closed. Are
> there any features / refactoring / bugs still need to be fixed.

Heikki is reviewing the hot standby code; we are trying to figure out
how to address the GIN patch, and the SE-Linux patch has gone through
several modifications and is waiting on someone to review it again.
The hash patch also is waiting for performance testing, I think.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Hi,

On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
<niranjan(dot)k(at)nsn(dot)com> wrote:
> Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
>
> I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.

At least I'm planning to work on the following two items of Synch Rep for v8.5.
Of course, Synch Rep works fine without these features.

- Address the problem which the signals cannot interrupt poll() and select()
on some platforms.
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00438.php

- Add new feature which transfers all WAL records via the direct connection
between the primary and the standby. In other words, get rid of
file-based log shipping part from the patch.
http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:

> On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
> <niranjan(dot)k(at)nsn(dot)com> wrote:
> > Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
> >
> > I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
>
> At least I'm planning to work on the following two items of Synch Rep for v8.5.
> Of course, Synch Rep works fine without these features.

> - Add new feature which transfers all WAL records via the direct connection
> between the primary and the standby. In other words, get rid of
> file-based log shipping part from the patch.
> http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com

Please bear in mind my strong objection to this. Attempting to transfer
all data via a single connection destroys VLDB usage of this feature. So
for me its just additional code for ease-of-use in the simplest case,
not code replacement.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
> On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
>
>> On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
>> <niranjan(dot)k(at)nsn(dot)com> wrote:
>>> Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
>>>
>>> I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
>> At least I'm planning to work on the following two items of Synch Rep for v8.5.
>> Of course, Synch Rep works fine without these features.
>
>> - Add new feature which transfers all WAL records via the direct connection
>> between the primary and the standby. In other words, get rid of
>> file-based log shipping part from the patch.
>> http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com
>
> Please bear in mind my strong objection to this. Attempting to transfer
> all data via a single connection destroys VLDB usage of this feature. So
> for me its just additional code for ease-of-use in the simplest case,
> not code replacement.

Presumably we'll keep the capability to restore from a backup and
restore from WAL archive as well, when those are available. Keeping that
capability shouldn't add many lines of code.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Tue, 2009-02-24 at 17:36 +0000, Simon Riggs wrote:
> On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
>
> > On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
> > <niranjan(dot)k(at)nsn(dot)com> wrote:
> > > Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
> > >
> > > I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
> >
> > At least I'm planning to work on the following two items of Synch Rep for v8.5.
> > Of course, Synch Rep works fine without these features.
>
> > - Add new feature which transfers all WAL records via the direct connection
> > between the primary and the standby. In other words, get rid of
> > file-based log shipping part from the patch.
> > http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com
>
> Please bear in mind my strong objection to this. Attempting to transfer
> all data via a single connection destroys VLDB usage of this feature. So
> for me its just additional code for ease-of-use in the simplest case,
> not code replacement.

Well VLDB is like 2% of what we need. If the above will remove all the
B.S. currently associated with actually doing PITR (rsync, scp, nfs,
pg_standby pick your poison) then I am all for it.

Log shipping should be:

I am master, my slave is here.
I am slave, I understand my master is here.
Here is our mutual authentication love token.
Let congress begin.

Anything more and we are being difficult for the sake of being
difficult.

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:

> Well VLDB is like 2% of what we need.

I am against removing an existing capability that is important to some
users. We shouldn't need to debate the exact percentage of users that
would be affected, or how to count them.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Tue, 2009-02-24 at 18:48 +0000, Simon Riggs wrote:
> On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
>
> > Well VLDB is like 2% of what we need.
>
> I am against removing an existing capability that is important to some
> users. We shouldn't need to debate the exact percentage of users that
> would be affected, or how to count them.

See Heikki's comment.

Joshua D. Drake

>
> --
> Simon Riggs www.2ndQuadrant.com
> PostgreSQL Training, Services and Support
>
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Simon Riggs wrote:
> On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
>
>
>> Well VLDB is like 2% of what we need.
>>
>
> I am against removing an existing capability that is important to some
> users. We shouldn't need to debate the exact percentage of users that
> would be affected, or how to count them.
>
>

Perhaps so, but I would hope you would support what Heikki and others
have been talking about as an option for replication. The 2% shouldn't
hold back the remaining 98%.

cheers

andrew

>>> Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> Simon Riggs wrote:
>> On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
>>
>>> Well VLDB is like 2% of what we need.
>>
>> I am against removing an existing capability that is important to
>> some users. We shouldn't need to debate the exact percentage of
>> users that would be affected, or how to count them.
>
> Perhaps so, but I would hope you would support what Heikki and
> others have been talking about as an option for replication. The 2%
> shouldn't hold back the remaining 98%.

Right. I was made a bit nervous by Joshua's comments, but somewhat
reassured by his reference back to Heikki's comments. If we can make
common cases simple to implement, that's great, as long as we don't
lose functionality needed to cover the more complex cases.

>>> Anything more and we are being difficult for the sake of being
>>> difficult.

Even in context, that came off as a bit user-hostile. It probably
wasn't meant that way, but it sounded provocative to me. We do have
better reasons than that for what I mentioned (in simplified form)
here:

http://archives.postgresql.org/pgsql-hackers/2009-01/msg00740.php

If the point was that we should not require anything more for
configuring a common, simple case then I'd agree.

-Kevin

Hi,

On Wed, Feb 25, 2009 at 2:37 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> Presumably we'll keep the capability to restore from a backup and restore
> from WAL archive as well, when those are available. Keeping that capability
> shouldn't add many lines of code.

Yes, I assume that only missing WAL files (probably including .backup file)
which are not in archive area of the standby server need to be transferred.
If there are many missing files, we can also copy them via multiple connection
before starting the standby.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, 2009-02-24 at 13:25 -0600, Kevin Grittner wrote:

> Right. I was made a bit nervous by Joshua's comments, but somewhat
> reassured by his reference back to Heikki's comments. If we can make
> common cases simple to implement, that's great, as long as we don't
> lose functionality needed to cover the more complex cases.

Right. That is my only point. I am not looking to "dumb down"
postgresql. I am saying that the "default" solution should be that
simple.

Autovacuum is the same way for example. The "default" solution works for
95% of the users out there without having to tweak any knob. The knobs
are there *if* they break out of the 95%.

>
> >>> Anything more and we are being difficult for the sake of being
> >>> difficult.
>
> Even in context, that came off as a bit user-hostile. It probably
> wasn't meant that way, but it sounded provocative to me.

Sorry I am in rare form today. I wasn't trying to be provocative at all
but was trying to be succinctly clear on what my belief about this is.

>
> If the point was that we should not require anything more for
> configuring a common, simple case then I'd agree.

Right.

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Tue, 2009-02-24 at 13:53 -0500, Andrew Dunstan wrote:
>
> Simon Riggs wrote:
> > On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
> >
> >
> >> Well VLDB is like 2% of what we need.
> >>
> >
> > I am against removing an existing capability that is important to some
> > users. We shouldn't need to debate the exact percentage of users that
> > would be affected, or how to count them.
> >

> Perhaps so, but I would hope you would support what Heikki and others
> have been talking about as an option for replication. The 2% shouldn't
> hold back the remaining 98%.

So far, everything has been couched in terms of remove the way it is now
and put in its place something "better". Heikki and Josh have said that
or similar, as has Robert Haas on another thread, and Fujii-san
specifically said "get rid of" the existing functionality. I am
completely against the removal of an existing capability that is
critically important to many users.

If we can add new functionality that is a nice-to-have for a large
number of people without removing a feature that is critical to many
users, bring it on. If we can't do that, then I would oppose.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Hi,

On Wed, Feb 25, 2009 at 5:08 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> So far, everything has been couched in terms of remove the way it is now
> and put in its place something "better". Heikki and Josh have said that
> or similar, as has Robert Haas on another thread, and Fujii-san
> specifically said "get rid of" the existing functionality. I am
> completely against the removal of an existing capability that is
> critically important to many users.

Though my description may be confusing, I don't mean to remove
any existing features unless necessary. I just said get rid of
file-based log shipping part only from my patch.

> If we can add new functionality that is a nice-to-have for a large
> number of people without removing a feature that is critical to many
> users, bring it on. If we can't do that, then I would oppose.

Yes, I also think so.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, Feb 24, 2009 at 3:08 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Tue, 2009-02-24 at 13:53 -0500, Andrew Dunstan wrote:
>> Simon Riggs wrote:
>> > On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
>> >> Well VLDB is like 2% of what we need.
>> > I am against removing an existing capability that is important to some
>> > users. We shouldn't need to debate the exact percentage of users that
>> > would be affected, or how to count them.
>> Perhaps so, but I would hope you would support what Heikki and others
>> have been talking about as an option for replication. The 2% shouldn't
>> hold back the remaining 98%.
> So far, everything has been couched in terms of remove the way it is now
> and put in its place something "better". Heikki and Josh have said that
> or similar, as has Robert Haas on another thread, and Fujii-san
> specifically said "get rid of" the existing functionality. I am
> completely against the removal of an existing capability that is
> critically important to many users.

I didn't think I had proposed any such thing, although maybe I'm just
not remembering. I'm pretty confused as to what the current thread is
all about.

It seems to me that in previous discussions of Streaming Replication,
Heikki put forward the proposition that the standby server should be
able to connect to the primary and stream not only newly-generated WAL
but also, if necessary, a base backup. As I recall, he argued that
without this functionality Streaming Replication would be far too
difficult to administer for the majority of users. Assuming I'm
representing his position more or less accurately, I completely agree
with it. I don't object to providing other mechanisms as well, but if
it's not about as simple as pointing the secondary at the primary and
saying "go", it's probably more complicated than I want to mess around
with.

...Robert

Hi,

On Wed, Feb 25, 2009 at 6:52 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> It seems to me that in previous discussions of Streaming Replication,
> Heikki put forward the proposition that the standby server should be
> able to connect to the primary and stream not only newly-generated WAL
> but also, if necessary, a base backup. As I recall, he argued that
> without this functionality Streaming Replication would be far too
> difficult to administer for the majority of users.

The automatic transfer of a base backup is useful, but not essential.
So, I'm not planning to work on it until synch rep patch is committed
at least. IIRC, nobody argues that such a feature is indispensable
also to the first step. We should develop synch rep in stages?

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Tue, 2009-02-24 at 16:52 -0500, Robert Haas wrote:

> I didn't think I had proposed any such thing, although maybe I'm just
> not remembering. I'm pretty confused as to what the current thread is
> all about.

http://archives.postgresql.org/pgsql-hackers/2009-01/msg00978.php

I don't think anyone who argued in favour of removal of existing system
was aware that we'd lose anything as a result. I think everybody
supports the easier-if-possible sentiment that Heikki was expressing; I
just don't want to let that be seen as agreement to remove, by default,
at a later time.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Bruce Momjian wrote:
> K, Niranjan (NSN - IN/Bangalore) wrote:
>> Hi,
>>
>> Could you please let me know what are the outstanding features
>> that are still to be developed in the respective patches?
>>
>> I'am currently referring the wiki: "Todo and Claim" for NTT and
>> for HotStandby, i see that almost all issues are closed. Are
>> there any features / refactoring / bugs still need to be fixed.
>
> Heikki is reviewing the hot standby code; we are trying to figure out
> how to address the GIN patch, and the SE-Linux patch has gone through
> several modifications and is waiting on someone to review it again.

I have updated my patch set a few times to fix bugs since I submitted
the revision without row-level facilities (r1522) at this Feb.06.
Was it unconfortable for you? These changes are just only bugfixes,
so it is not necessary whole of patches again.

Or, does the "several modifications" mean the separation of row-level
facilities and so on?

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

On Tue, Feb 24, 2009 at 5:58 PM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Tue, 2009-02-24 at 16:52 -0500, Robert Haas wrote:
>
>> I didn't think I had proposed any such thing, although maybe I'm just
>> not remembering.  I'm pretty confused as to what the current thread is
>> all about.
>
> http://archives.postgresql.org/pgsql-hackers/2009-01/msg00978.php

I still don't see where I suggested removing anything. What Heikki
suggested, and I agreed with, was adding something: integrated base
backup.

> I don't think anyone who argued in favour of removal of existing system
> was aware that we'd lose anything as a result. I think everybody
> supports the easier-if-possible sentiment that Heikki was expressing; I
> just don't want to let that be seen as agreement to remove, by default,
> at a later time.

I'm still totally unclear as to what you think anyone might, at some
point in the future, propose to remove.

I think the more relevant question right now is whether the work Fujii
Masao is planning to do for 8.5 is reponsive to the following comment
from Heikki:

# IMHO, the synchronous replication isn't in such good shape, I'm
afraid. I've said
# this before, but I'm not happy with the "built from spare parts"
nature of it. You
# shouldn't have to configure an archive, file-based log shipping using rsync or
# whatever, and pg_standby. All that is in addition to the direct
connection between
# master and slave. The slave really should be able to just connect to
the master, and
# download all the WAL it needs directly. That's a huge usability
issue if left as is,
# but requires very large architectural changes to fix.

...Robert

Robert Haas wrote:
> I think the more relevant question right now is whether the work Fujii
> Masao is planning to do for 8.5 is reponsive to the following comment
> from Heikki:
>
> # IMHO, the synchronous replication isn't in such good shape, I'm
> afraid. I've said
> # this before, but I'm not happy with the "built from spare parts"
> nature of it. You
> # shouldn't have to configure an archive, file-based log shipping using rsync or
> # whatever, and pg_standby. All that is in addition to the direct
> connection between
> # master and slave. The slave really should be able to just connect to
> the master, and
> # download all the WAL it needs directly. That's a huge usability
> issue if left as is,
> # but requires very large architectural changes to fix.

I believe so, see second bullet point in:

http://archives.postgresql.org/message-id/3f0b79eb0902240751t13231593g17fbef70664d4444@mail.gmail.com

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, 2009-02-25 at 22:45 +0200, Heikki Linnakangas wrote:
> Robert Haas wrote:
> > I think the more relevant question right now is whether the work Fujii
> > Masao is planning to do for 8.5 is reponsive to the following comment
> > from Heikki:
> >
> > # IMHO, the synchronous replication isn't in such good shape, I'm
> > afraid. I've said
> > # this before, but I'm not happy with the "built from spare parts"
> > nature of it. You
> > # shouldn't have to configure an archive, file-based log shipping using rsync or
> > # whatever, and pg_standby. All that is in addition to the direct
> > connection between
> > # master and slave. The slave really should be able to just connect to
> > the master, and
> > # download all the WAL it needs directly. That's a huge usability
> > issue if left as is,
> > # but requires very large architectural changes to fix.
>
> I believe so, see second bullet point in:
>
> http://archives.postgresql.org/message-id/3f0b79eb0902240751t13231593g17fbef70664d4444@mail.gmail.com

That is exactly what I am against. Note the words "get rid of".

This prevents parallel data transfer, use of split mirrors and various
other techniques. It sounds neater, but it implies removal of useful
features.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
> On Wed, 2009-02-25 at 22:45 +0200, Heikki Linnakangas wrote:
>
>> Robert Haas wrote:
>>
>>> I think the more relevant question right now is whether the work Fujii
>>> Masao is planning to do for 8.5 is reponsive to the following comment
>>> from Heikki:
>>>
>>> # IMHO, the synchronous replication isn't in such good shape, I'm
>>> afraid. I've said
>>> # this before, but I'm not happy with the "built from spare parts"
>>> nature of it. You
>>> # shouldn't have to configure an archive, file-based log shipping using rsync or
>>> # whatever, and pg_standby. All that is in addition to the direct
>>> connection between
>>> # master and slave. The slave really should be able to just connect to
>>> the master, and
>>> # download all the WAL it needs directly. That's a huge usability
>>> issue if left as is,
>>> # but requires very large architectural changes to fix.
>>>
>> I believe so, see second bullet point in:
>>
>> http://archives.postgresql.org/message-id/3f0b79eb0902240751t13231593g17fbef70664d4444@mail.gmail.com
>>
>
> That is exactly what I am against. Note the words "get rid of".
>
> This prevents parallel data transfer, use of split mirrors and various
> other techniques. It sounds neater, but it implies removal of useful
> features.
>

OK, so let's assume that we'll provide an extra facility that doesn't
take anything away but which provides for close to zero config setup for
the simple case. Frankly, that's what the vast majority of people want,
in my experience.

cheers

andrew

Hi,

On Thu, Feb 26, 2009 at 6:03 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> That is exactly what I am against. Note the words "get rid of".
>
> This prevents parallel data transfer, use of split mirrors and various
> other techniques. It sounds neater, but it implies removal of useful
> features.

OK, ISTM that my description was confusing you, so I removed that statement
from the TODO item on wiki.
http://wiki.postgresql.org/wiki/NTT%27s_Development_Projects#Todo_and_Claim

Again, I'm not planning to get rid of any existing capabilities unless
necessary.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center

On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
> Hi,
>
> On Thu, Feb 26, 2009 at 6:03 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > That is exactly what I am against. Note the words "get rid of".
> >
> > This prevents parallel data transfer, use of split mirrors and various
> > other techniques. It sounds neater, but it implies removal of useful
> > features.
>
> OK, ISTM that my description was confusing you, so I removed that statement
> from the TODO item on wiki.
> http://wiki.postgresql.org/wiki/NTT%27s_Development_Projects#Todo_and_Claim
>
> Again, I'm not planning to get rid of any existing capabilities

Good

> unless necessary.

That is not a caveat I will accept, a priori.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, 2009-02-25 at 16:11 -0500, Andrew Dunstan wrote:

> OK, so let's assume that we'll provide an extra facility that doesn't
> take anything away but which provides for close to zero config setup for
> the simple case. Frankly, that's what the vast majority of people want,
> in my experience.

My experience also. Very much agreed.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Wed, Feb 25, 2009 at 3:45 PM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:
> I believe so, see second bullet point in:
>
> http://archives.postgresql.org/message-id/3f0b79eb0902240751t13231593g17fbef70664d4444@mail.gmail.com

Cool.

...Robert

Simon Riggs wrote:
>
> On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
> > Hi,
> >
> > On Thu, Feb 26, 2009 at 6:03 AM, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> > > That is exactly what I am against. Note the words "get rid of".
> > >
> > > This prevents parallel data transfer, use of split mirrors and various
> > > other techniques. It sounds neater, but it implies removal of useful
> > > features.
> >
> > OK, ISTM that my description was confusing you, so I removed that statement
> > from the TODO item on wiki.
> > http://wiki.postgresql.org/wiki/NTT%27s_Development_Projects#Todo_and_Claim
> >
> > Again, I'm not planning to get rid of any existing capabilities
>
> Good
>
> > unless necessary.
>
> That is not a caveat I will accept, a priori.

What does "accept" mean above? Are you the sole acceptor for this
feature? That is surprising to me.

You can say you would vote against it but your wording above seems
overly controlling.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> > On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
> > >
> > > Again, I'm not planning to get rid of any existing capabilities
> >
> > Good
> >
> > > unless necessary.
> >
> > That is not a caveat I will accept, a priori.
>
> What does "accept" mean above? Are you the sole acceptor for this
> feature? That is surprising to me.
>
> You can say you would vote against it but your wording above seems
> overly controlling.

I would hope my words carry the same weight as others when people speak
of what can and cannot be included, when backed by reasonable and
logical technical reasons.

Some things are important, some not, and I've done my best to use words
that indicate my views on that.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
>
> On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote:
> > Simon Riggs wrote:
> > > On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
> > > >
> > > > Again, I'm not planning to get rid of any existing capabilities
> > >
> > > Good
> > >
> > > > unless necessary.
> > >
> > > That is not a caveat I will accept, a priori.
> >
> > What does "accept" mean above? Are you the sole acceptor for this
> > feature? That is surprising to me.
> >
> > You can say you would vote against it but your wording above seems
> > overly controlling.
>
> I would hope my words carry the same weight as others when people speak
> of what can and cannot be included, when backed by reasonable and
> logical technical reasons.
>
> Some things are important, some not, and I've done my best to use words
> that indicate my views on that.

I think you need to work on improving your word choice then.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

>>> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Simon Riggs wrote:
>> On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote:
>> > Simon Riggs wrote:
>> > > On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
>> > > >
>> > > > Again, I'm not planning to get rid of any existing
>> > > > capabilities
>> > >
>> > > Good
>> > >
>> > > > unless necessary.
>> > >
>> > > That is not a caveat I will accept, a priori.
>> >
>> > What does "accept" mean above? Are you the sole acceptor for
>> > this feature? That is surprising to me.
>> >
>> > You can say you would vote against it but your wording above
>> > seems overly controlling.
>>
>> I would hope my words carry the same weight as others when people
>> speak of what can and cannot be included, when backed by reasonable
>> and logical technical reasons.
>>
>> Some things are important, some not, and I've done my best to use
>> words that indicate my views on that.
>
> I think you need to work on improving your word choice then.

Anything which breaks existing techniques for log shipping, PITR
recovery, or warm standby would present a significant obstacle to
adoption in our environment. Based on off-list emails received when I
mentioned what we do, I know we're not alone.

Some comments have made me nervous on this count, but responses to
questions about it have generally been reassuring. The "unless
necessary" seemed a little ominous. I won't get too excited unless
someone thinks a change is necessary and I see the nature of the
change.

As long as there is some way to ship base backups and WAL files to
multiple targets, not all of which are running any PostgreSQL
software, and the database can be restored from such copies, we're not
looking a more than a few weeks of staff time to adjust our scripts
and procedures.

-Kevin

On Thu, 2009-02-26 at 11:02 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> >
> > On Wed, 2009-02-25 at 17:50 -0500, Bruce Momjian wrote:
> > > Simon Riggs wrote:
> > > > On Thu, 2009-02-26 at 06:15 +0900, Fujii Masao wrote:
> > > > >
> > > > > Again, I'm not planning to get rid of any existing capabilities
> > > >
> > > > Good
> > > >
> > > > > unless necessary.
> > > >
> > > > That is not a caveat I will accept, a priori.
> > >
> > > What does "accept" mean above? Are you the sole acceptor for this
> > > feature? That is surprising to me.
> > >
> > > You can say you would vote against it but your wording above seems
> > > overly controlling.
> >
> > I would hope my words carry the same weight as others when people speak
> > of what can and cannot be included, when backed by reasonable and
> > logical technical reasons.
> >
> > Some things are important, some not, and I've done my best to use words
> > that indicate my views on that.
>
> I think you need to work on improving your word choice then.

I vote against this proposal.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Tue, 2009-02-24 at 10:34 -0800, Joshua D. Drake wrote:
> On Tue, 2009-02-24 at 17:36 +0000, Simon Riggs wrote:
> > On Wed, 2009-02-25 at 00:51 +0900, Fujii Masao wrote:
> >
> > > On Tue, Feb 24, 2009 at 3:47 PM, K, Niranjan (NSN - IN/Bangalore)
> > > <niranjan(dot)k(at)nsn(dot)com> wrote:
> > > > Could you please let me know what are the outstanding features that are still to be developed in the respective patches?
> > > >
> > > > I'am currently referring the wiki: "Todo and Claim" for NTT and for HotStandby, i see that almost all issues are closed. Are there any features / refactoring / bugs still need to be fixed.
> > >
> > > At least I'm planning to work on the following two items of Synch Rep for v8.5.
> > > Of course, Synch Rep works fine without these features.
> >
> > > - Add new feature which transfers all WAL records via the direct connection
> > > between the primary and the standby. In other words, get rid of
> > > file-based log shipping part from the patch.
> > > http://archives.postgresql.org/message-id/496B9495.4010902@enterprisedb.com
> >
> > Please bear in mind my strong objection to this. Attempting to transfer
> > all data via a single connection destroys VLDB usage of this feature. So
> > for me its just additional code for ease-of-use in the simplest case,
> > not code replacement.
>
> Well VLDB is like 2% of what we need. If the above will remove all the
> B.S. currently associated with actually doing PITR (rsync, scp, nfs,
> pg_standby pick your poison) then I am all for it.

If you use walmgr.py, then all you need is writing a conf file and
making sure that ssh and rsync work.

Actually the best way to do Sync Rep would have been to just move to C
what walmgr.py does. That the patch could have started off from a
well-tested foundation.

> Log shipping should be:
>
> I am master, my slave is here.
> I am slave, I understand my master is here.
> Here is our mutual authentication love token.
> Let congress begin.
>
> Anything more and we are being difficult for the sake of being
> difficult.

Actually I'd leave out the first line, and start with just

- I am slave, my master accepts me, start replicationg

So there could be several slaves, of both hot standby postgresql,
wal-file-store and store-and-forward-to-many types.

--
Hannu Krosing http://www.2ndQuadrant.com
PostgreSQL Scalability and Availability
Services, Consulting and Training

On Fri, 2009-02-27 at 22:17 +0200, Hannu Krosing wrote:

> > Well VLDB is like 2% of what we need. If the above will remove all the
> > B.S. currently associated with actually doing PITR (rsync, scp, nfs,
> > pg_standby pick your poison) then I am all for it.
>
> If you use walmgr.py, then all you need is writing a conf file and
> making sure that ssh and rsync work.
>
> Actually the best way to do Sync Rep would have been to just move to C
> what walmgr.py does. That the patch could have started off from a
> well-tested foundation.
>

For sync?

> > Log shipping should be:
> >
> > I am master, my slave is here.
> > I am slave, I understand my master is here.
> > Here is our mutual authentication love token.
> > Let congress begin.
> >
> > Anything more and we are being difficult for the sake of being
> > difficult.
>
> Actually I'd leave out the first line, and start with just
>
> - I am slave, my master accepts me, start replicationg
>

Heh fair enough.

> So there could be several slaves, of both hot standby postgresql,
> wal-file-store and store-and-forward-to-many types.

THat should be optional not the default.

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Fri, 2009-02-27 at 12:21 -0800, Joshua D. Drake wrote:
> On Fri, 2009-02-27 at 22:17 +0200, Hannu Krosing wrote:
>
> > > Well VLDB is like 2% of what we need. If the above will remove all the
> > > B.S. currently associated with actually doing PITR (rsync, scp, nfs,
> > > pg_standby pick your poison) then I am all for it.
> >
> > If you use walmgr.py, then all you need is writing a conf file and
> > making sure that ssh and rsync work.
> >
> > Actually the best way to do Sync Rep would have been to just move to C
> > what walmgr.py does. That the patch could have started off from a
> > well-tested foundation.
> >
>
> For sync?

For everything up to starting sync.

It somehow feels like current patch concentrated mainly on doing the
sync part and the need to get also existing stete over automatically
became as an afterthought. That's why I'm proposing to start (at least
conceptually) from an existing, fully automated and working solution.

Currently walmgr.py is doing everything from setting up replica to
getting up-to-last-second changes to slave's disk.

> > > Log shipping should be:
> > >
> > > I am master, my slave is here.
> > > I am slave, I understand my master is here.
> > > Here is our mutual authentication love token.
> > > Let congress begin.
> > >
> > > Anything more and we are being difficult for the sake of being
> > > difficult.
> >
> > Actually I'd leave out the first line, and start with just
> >
> > - I am slave, my master accepts me, start replicationg
> >
>
> Heh fair enough.
>
> > So there could be several slaves, of both hot standby postgresql,
> > wal-file-store and store-and-forward-to-many types.
>
> That should be optional not the default.

Of course. One slave is a sub-case of "any number of slaves" :)

But doing 1-1 replica _only_ would make us laughing stock for everybody
else.

--
Hannu Krosing http://www.2ndQuadrant.com
PostgreSQL Scalability and Availability
Services, Consulting and Training

Hannu Krosing wrote:
> Currently walmgr.py is doing everything from setting up replica to
> getting up-to-last-second changes to slave's disk.
>
>

If walmgr.py and its cousins had good documentation there would possibly
be much greater acceptance of them.

cheers

andrew

Fujii,

>> Again, I'm not planning to get rid of any existing capabilities
>
> Good
>
>> unless necessary.
>
> That is not a caveat I will accept, a priori.

While Simon stated it a bit strongly, I think it's important that you
alert people if you think you have to remove existing features in order
to make easy standby possible. It's possible that features which seem
trivial to you are used extensively by people with particular failover
requirements.

--Josh

On Sat, 2009-02-28 at 23:21 -0800, Josh Berkus wrote:
> Fujii,
>
> >> Again, I'm not planning to get rid of any existing capabilities
> >
> > Good
> >
> >> unless necessary.
> >
> > That is not a caveat I will accept, a priori.
>
> While Simon stated it a bit strongly

My intention was only to be clear about how important a technical point
it was for me. My comments were aimed at avoiding a costly blind alley,
not critically towards any individual.

I am happy to apologise in case people thought my words rude.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Hi,

On Sun, Mar 1, 2009 at 4:21 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> While Simon stated it a bit strongly, I think it's important that you alert
> people if you think you have to remove existing features in order to make
> easy standby possible.

Now, I think that any existing capabilities don't need to be removed
for Synch Rep.

> It's possible that features which seem trivial to
> you are used extensively by people with particular failover requirements.

Of course, I have no intention of ignoring such people.

Regards,

--
Fujii Masao
NIPPON TELEGRAPH AND TELEPHONE CORPORATION
NTT Open Source Software Center