Re: postponing some large patches to 9.2

On Mon, Feb 7, 2011 at 3:57 PM, Chris Browne <cbbrowne(at)acm(dot)org> wrote:
> It sure looks to me like there are going to be a bunch of items that,
> based on the recognized policies, need to get deferred to 9.2, and the
> prospects for Sync Rep getting into 9.1 don't look notably good to me.
>
> It's definitely readily arguable that fairness requires that:
>
>  - Items not committable by 2011-02-15 be deferred to the 2011-Next fest
>
>   There are around 25 items right now that are sitting with [Waiting
>   for Author] and [Returned with Feedback] statuses.  They largely seem
>   like pretty fair game for "next fest."
>
>  - Large items that weren't included in the 2010-11 fest be considered
>   problematic to try to integrate into 9.1
>
>   There sure seem to be some large items in the 2011-01 fest, which I
>   thought wasn't supposed to be the case.

This discussion reveals that it's time to start making some
discussions about what can be accomplished for 9.1 and what must be
postponed to 9.2. The big ones I think we should postpone are:

- Range Types. This is a large patch which was submitted for the
first time to the last CommitFest of the cycle, and the first version
that had no open TODO items was posted yesterday, three-quarters of
the way through that last CommitFest. Some good review has been done.
While more is probably needed, I think we should feel good about
what's been accomplished and mark this one Returned with Feedback.

- ALTER EXTENSION UPGRADE. This is another large patch which was
submitted for the first time to the last CommitFest of the cycle. The
prerequisite patch to provide basic extension support has not been
committed yet, although it sounds like that will happen soon.
However, since that patch is undergoing a fair amount of surgery, it's
reasonably certain that this will require significant rebasing. I
think it would also be an overstatement to say that we have consensus
on the design. My feeling is that, unless Tom thinks that failing to
get this committed now is going to leave us with a mess later, we
should mark this one Returned with Feedback and revisit it for 9.2.

- FOR KEY LOCK tables. This patch, unfortunately, has not gotten a
lot of review. But it's a major, potentially destabilizing change
that was, like the last two, submitted for the first time to the last
CommitFest of the cycle. Even if we assume for the sake of argument
that the code is unusually good for a feature of this type, I don't
think it's the right time to commit something like this. I would
argue for putting this off until 9.2, though preferably with a bit
more review than it's gotten so far.

The other remaining "big" patches are:

- extension support for pg_upgrade. Tom is planning to have this
committed within a day or so, per latest news.

- synchronous replication. Based on some looking at this today, I am
somewhat doubtful about the possibility of me or anyone else beating
this completely into shape in time for 9.2, unless we choose to extend
the deadline by several weeks. Simon said that he would have time to
finish this in the next two weeks, but, as noted, the CommitFest is
scheduled to be over in ONE week, and it looks to me like this is
still pretty rough. However, there's a lot of good stuff in here, and
I think it might be practical to get some portion of it committed even
if we can't agree on all of it. I recommend we give this one a little
more time to shake out before giving up on it.

- SQL/MED. This patch unfortunately kind of stalled for a long time.
However, I believe that Heikki is now working actively on the core
patch, so I'm hopeful for some activity here soon.

- Writeable CTEs. Tom has promised to look at this, but doesn't seem
to be in a hurry.

- Per-column collation. This one has been lingering for a long time.
But Noah Misch recently did a pretty thorough review, and it's now
marked Ready for Committer. Peter, are you planning to commit this?

- The PL/python extravaganza. I'm not really clear where we stand
with this. There are a lot of patches here.

There are a variety of smaller patches we need to make decisions
about, too. But summarizing all of that here is going to be too much.
I'll post to some of the other threads individually.

Thoughts on these?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 11-02-07 10:37 PM, Robert Haas wrote:

> - The PL/python extravaganza. I'm not really clear where we stand
> with this. There are a lot of patches here.
>

Some of the patches have been committed a few others are ready (or
almost ready) for a committer. The table function one is the only one
in 'waiting for author'

4 of the patches haven't yet received any review. Jan Urbanski has been
pretty good about posting updated patches as the dependent patches get
updated. It would be good if a few people grabbed these. The individual
patches tend to not be that large.

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> - Range Types. This is a large patch which was submitted for the
> first time to the last CommitFest of the cycle, and the first version
> that had no open TODO items was posted yesterday, three-quarters of
> the way through that last CommitFest. Some good review has been done.
> While more is probably needed, I think we should feel good about
> what's been accomplished and mark this one Returned with Feedback.

I don't agree w/ punting Range Types. Range Types were discussed as far
back as the 2010 developer meeting, were discussed quite a bit again
starting in October and throughout the fall, and Jeff has regularly
been posting updates to it. Given how thorough Jeff is, my feeling is
that this patch is more than ready for beta. My impression is also that
it's not as invasive or destablizing as the others and while it wasn't
being posted to the previous commit fests, it was clearly being worked
on, updated, and improved.

> - synchronous replication. Based on some looking at this today, I am
> somewhat doubtful about the possibility of me or anyone else beating
> this completely into shape in time for 9.2, unless we choose to extend
> the deadline by several weeks. Simon said that he would have time to
> finish this in the next two weeks, but, as noted, the CommitFest is
> scheduled to be over in ONE week, and it looks to me like this is
> still pretty rough. However, there's a lot of good stuff in here, and
> I think it might be practical to get some portion of it committed even
> if we can't agree on all of it. I recommend we give this one a little
> more time to shake out before giving up on it.

It really would be nice to have this, but I agree that it's pretty late
in the game for it to be in the state is appears to be in. :/ It also
seems to have been stalled for the past couple of months, which doesn't
bode well for it, in my view.

Thanks,

Stephen

2011/2/8 Steve Singer <ssinger_pg(at)sympatico(dot)ca>:
> On 11-02-07 10:37 PM, Robert Haas wrote:
>
>> - The PL/python extravaganza.  I'm not really clear where we stand
>> with this.  There are a lot of patches here.
>>
>
> Some of the patches have been committed a few others are ready (or almost
> ready) for a committer.   The table function one  is the only one in
> 'waiting for author'

I didn't quite finished my reviews on pl/python patches yet, but it
seems that "don't remove argument" will be easy to review and unlikely
to have issues. "quote functions" can be committed already as-is.
"table function" should be in if Jan sends another patch soon. "custom
datatype parser" looks premature yet, though we want to give more
feedbacks about its design. I'm not sure about other patches.

> 4 of the patches haven't yet received any review. Jan Urbanski has been
> pretty good about posting updated patches as the dependent patches get
> updated.  It would be good if a few people grabbed these. The individual
> patches tend to not be that large.

I agree he did quite a lot of work well. But still some of the patches
are very easy and others is like WIP stage. I hope anyone can review
them as soon as possible.

Regards,

--
Hitoshi Harada

On 08/02/11 15:44, Hitoshi Harada wrote:
> 2011/2/8 Steve Singer <ssinger_pg(at)sympatico(dot)ca>:
>> On 11-02-07 10:37 PM, Robert Haas wrote:
>>
>>> - The PL/python extravaganza. I'm not really clear where we stand
>>> with this. There are a lot of patches here.
>>>
>>
>> Some of the patches have been committed a few others are ready (or almost
>> ready) for a committer. The table function one is the only one in
>> 'waiting for author'
>
> I didn't quite finished my reviews on pl/python patches yet, but it
> seems that "don't remove argument" will be easy to review and unlikely
> to have issues. "quote functions" can be committed already as-is.
> "table function" should be in if Jan sends another patch soon. "custom
> datatype parser" looks premature yet, though we want to give more
> feedbacks about its design. I'm not sure about other patches.

From the outstanding PL/Python patches:

* invalidate composites - it fixes a TODO item and is a general
improvement, although the case being fixed is rather rare. Should be
straightforward to review, it's small and localized.

* tracebacks - IMHO a big improvement, but should get more testing than
I gave it, as traversing Python stacks tends to be tricky. Still, it's
very localized (basically just getting a traceback string).

* table functions - I'm working on this one, should have an update today

* custom datatype parsers - more of a PoC, and because the discussion
about hstores in PL/Python died down I decided to go ahead and send a
patch implementing the last idea from that thread. I'm fine with it not
making 9.1 because this should be designed carefully, and will affect
decisions similar to those that are now being taken WRT arrays in PL/Perl.

* custom SPI exceptions - I'd really like this one to go in, because it
allows writing UPSERT-kind functions in PL/Python very easily, and it's
just a handful of lines of code

* don't remove arguments - a bugfix, really, and a very small one

So from the above, I'd say custom datatype parsers could get rejected if
noone feels like having a discussion about it for 9.1. Table functions,
custom SPI exceptions and tracebacks are niceties that if postponed to
9.2 will just mean that many features less in 9.1. The rest is bordering
on bugfixes, and I think should go in.

Cheers,
Jan

sfrost(at)snowman(dot)net (Stephen Frost) writes:
> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
>> - Range Types. This is a large patch which was submitted for the
>> first time to the last CommitFest of the cycle, and the first version
>> that had no open TODO items was posted yesterday, three-quarters of
>> the way through that last CommitFest. Some good review has been done.
>> While more is probably needed, I think we should feel good about
>> what's been accomplished and mark this one Returned with Feedback.
>
> I don't agree w/ punting Range Types. Range Types were discussed as
> far back as the 2010 developer meeting, were discussed quite a bit
> again starting in October and throughout the fall, and Jeff has
> regularly been posting updates to it. Given how thorough Jeff is, my
> feeling is that this patch is more than ready for beta. My impression
> is also that it's not as invasive or destablizing as the others and
> while it wasn't being posted to the previous commit fests, it was
> clearly being worked on, updated, and improved.

I generally mirror those thoughts. Range Types don't seem invasive or
destabilizing, and the code base has been deployed for quite some time
as an extension ("not quite contrib"). It would be disappointing to
drop this one when it is mighty close.

>> - synchronous replication. Based on some looking at this today, I am
>> somewhat doubtful about the possibility of me or anyone else beating
>> this completely into shape in time for 9.2, unless we choose to extend
>> the deadline by several weeks. Simon said that he would have time to
>> finish this in the next two weeks, but, as noted, the CommitFest is
>> scheduled to be over in ONE week, and it looks to me like this is
>> still pretty rough. However, there's a lot of good stuff in here, and
>> I think it might be practical to get some portion of it committed even
>> if we can't agree on all of it. I recommend we give this one a little
>> more time to shake out before giving up on it.
>
> It really would be nice to have this, but I agree that it's pretty late
> in the game for it to be in the state is appears to be in. :/ It also
> seems to have been stalled for the past couple of months, which doesn't
> bode well for it, in my view.

The "stall" troubles me, and doesn't bode terribly well for 9.1.
--
Rules of the Evil Overlord #39. "If I absolutely must ride into
battle, I will certainly not ride at the forefront of my Legions of
Terror, nor will I seek out my opposite number among his army."
<http://www.eviloverlord.com/>

On 11-02-08 10:07 AM, Jan Urbański wrote:
>
> * custom SPI exceptions - I'd really like this one to go in, because it
> allows writing UPSERT-kind functions in PL/Python very easily, and it's
> just a handful of lines of code
>

I will try to do a review of this one (probably tomorrow night) since
I've reviewed many of the related patches.

> * don't remove arguments - a bugfix, really, and a very small one
>
> So from the above, I'd say custom datatype parsers could get rejected if
> noone feels like having a discussion about it for 9.1. Table functions,
> custom SPI exceptions and tracebacks are niceties that if postponed to
> 9.2 will just mean that many features less in 9.1. The rest is bordering
> on bugfixes, and I think should go in.
>
> Cheers,
> Jan
>

On Tue, Feb 8, 2011 at 10:40 AM, Chris Browne <cbbrowne(at)acm(dot)org> wrote:
> sfrost(at)snowman(dot)net (Stephen Frost) writes:
>> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
>>> - Range Types.  This is a large patch which was submitted for the
>>> first time to the last CommitFest of the cycle, and the first version
>>> that had no open TODO items was posted yesterday, three-quarters of
>>> the way through that last CommitFest.  Some good review has been done.
>>>  While more is probably needed, I think we should feel good about
>>> what's been accomplished and mark this one Returned with Feedback.
>>
>> I don't agree w/ punting Range Types.  Range Types were discussed as
>> far back as the 2010 developer meeting, were discussed quite a bit
>> again starting in October and throughout the fall, and Jeff has
>> regularly been posting updates to it.  Given how thorough Jeff is, my
>> feeling is that this patch is more than ready for beta.  My impression
>> is also that it's not as invasive or destablizing as the others and
>> while it wasn't being posted to the previous commit fests, it was
>> clearly being worked on, updated, and improved.
>
> I generally mirror those thoughts.  Range Types don't seem invasive or
> destabilizing, and the code base has been deployed for quite some time
> as an extension ("not quite contrib").  It would be disappointing to
> drop this one when it is mighty close.

It's a 5400 line patch that wasn't completed until the middle of the
current CommitFest. Nobody has ever submitted a major feature patch
of that size that got done in a single CommitFest, to my recollection,
or even half that size. Compare Hot Standby or True Serializability,
both of which required basically a full development cycle. It may be
true that the idea has been kicking around for a long time, but it's
been code-complete for one week.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Mon, Feb 07, 2011 at 10:37:06PM -0500, Robert Haas wrote:
> On Mon, Feb 7, 2011 at 3:57 PM, Chris Browne <cbbrowne(at)acm(dot)org> wrote:
> > It sure looks to me like there are going to be a bunch of items that,
> > based on the recognized policies, need to get deferred to 9.2, and the
> > prospects for Sync Rep getting into 9.1 don't look notably good to me.
> >
> > It's definitely readily arguable that fairness requires that:
> >
> >  - Items not committable by 2011-02-15 be deferred to the 2011-Next fest
> >
> >   There are around 25 items right now that are sitting with [Waiting
> >   for Author] and [Returned with Feedback] statuses.  They largely seem
> >   like pretty fair game for "next fest."
> >
> >  - Large items that weren't included in the 2010-11 fest be considered
> >   problematic to try to integrate into 9.1
> >
> >   There sure seem to be some large items in the 2011-01 fest, which I
> >   thought wasn't supposed to be the case.
>
> This discussion reveals that it's time to start making some
> discussions about what can be accomplished for 9.1 and what must be
> postponed to 9.2.

Given how things worked, i.e. that people were not clear that 9.1
development had actually started, etc., I am again proposing that we
have one more CF starting March 15 to get this all cleaned up. Yes, I
know that wasn't the plan, but I also know that we're really, really
close on a whole bunch of things, some of which have been in the
offing for years at this point, and we risk giving people the
impression, if they don't already have it, that if they're not in the
"inner circle," their patches get lower priority no matter what their
merits.

> The big ones I think we should postpone are:
>
> - Range Types. This is a large patch which was submitted for the
> first time to the last CommitFest of the cycle, and the first version
> that had no open TODO items was posted yesterday, three-quarters of
> the way through that last CommitFest. Some good review has been done.
> While more is probably needed, I think we should feel good about
> what's been accomplished and mark this one Returned with Feedback.

This one's a product differentiator for PostgreSQL. We can own this
space for at least a year before it gets on any other RDBMS's roadmap.
That the work wasn't submitted until it was in pretty good shape is a
mark in Jeff's favor, not a reason to punt for another year.

> - ALTER EXTENSION UPGRADE. This is another large patch which was
> submitted for the first time to the last CommitFest of the cycle.
> The prerequisite patch to provide basic extension support has not
> been committed yet, although it sounds like that will happen soon.
> However, since that patch is undergoing a fair amount of surgery,
> it's reasonably certain that this will require significant rebasing.
> I think it would also be an overstatement to say that we have
> consensus on the design. My feeling is that, unless Tom thinks that
> failing to get this committed now is going to leave us with a mess
> later, we should mark this one Returned with Feedback and revisit it
> for 9.2.

If we're going to leave this out, we should probably pull EXTENSIONs
out entirely. Is that really where we want to go?

> - FOR KEY LOCK tables. This patch, unfortunately, has not gotten a
> lot of review. But it's a major, potentially destabilizing change
> that was, like the last two, submitted for the first time to the last
> CommitFest of the cycle. Even if we assume for the sake of argument
> that the code is unusually good for a feature of this type, I don't
> think it's the right time to commit something like this. I would
> argue for putting this off until 9.2, though preferably with a bit
> more review than it's gotten so far.
>
> The other remaining "big" patches are:
>
> - extension support for pg_upgrade. Tom is planning to have this
> committed within a day or so, per latest news.

See above.

> - synchronous replication. Based on some looking at this today, I am
> somewhat doubtful about the possibility of me or anyone else beating
> this completely into shape in time for 9.2, unless we choose to extend
> the deadline by several weeks.

+1 for extending that deadline. This is another product
differentiator, and we can have it as that for about a year if we make
sure it gets into 9.1.

> Simon said that he would have time to
> finish this in the next two weeks, but, as noted, the CommitFest is
> scheduled to be over in ONE week, and it looks to me like this is
> still pretty rough. However, there's a lot of good stuff in here, and
> I think it might be practical to get some portion of it committed even
> if we can't agree on all of it. I recommend we give this one a little
> more time to shake out before giving up on it.
>
> - SQL/MED. This patch unfortunately kind of stalled for a long time.
> However, I believe that Heikki is now working actively on the core
> patch, so I'm hopeful for some activity here soon.
>
> - Writeable CTEs. Tom has promised to look at this, but doesn't seem
> to be in a hurry.

This patch is about the worst in terms of "the inner circle" vs. our
open development model I referred to above. This, too, is a product
differentiator that we can really lead the standard with, and there's
no way to break any conceivable change to the standard with it.

What's the latest?

> - Per-column collation. This one has been lingering for a long time.
> But Noah Misch recently did a pretty thorough review, and it's now
> marked Ready for Committer. Peter, are you planning to commit this?

I think we're faced with the hard reality that we can't get a perfect
implementation of this in one go, as that would involve pulling the
"correctness" bits away from the OS-supplied libraries, etc., and into
PostgreSQL. I'm all for doing this eventually, and meanwhile getting
what's already done in.

> - The PL/python extravaganza. I'm not really clear where we stand
> with this. There are a lot of patches here.

Many of these are already a go. :)

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On tis, 2011-02-08 at 00:16 -0500, Steve Singer wrote:
> On 11-02-07 10:37 PM, Robert Haas wrote:
>
> > - The PL/python extravaganza. I'm not really clear where we stand
> > with this. There are a lot of patches here.
> >
>
> Some of the patches have been committed a few others are ready (or
> almost ready) for a committer. The table function one is the only one
> in 'waiting for author'
>
> 4 of the patches haven't yet received any review. Jan Urbanski has been
> pretty good about posting updated patches as the dependent patches get
> updated. It would be good if a few people grabbed these. The individual
> patches tend to not be that large.

I'm available to commit these if someone else can do the initial review.

On Mon, 2011-02-07 at 22:37 -0500, Robert Haas wrote:
> - Range Types. This is a large patch which was submitted for the
> first time to the last CommitFest of the cycle, and the first version
> that had no open TODO items was posted yesterday, three-quarters of
> the way through that last CommitFest. Some good review has been done.
> While more is probably needed, I think we should feel good about
> what's been accomplished and mark this one Returned with Feedback.

I submitted this clearly marked WIP, so I expected that it would likely
be pushed to 9.2.

However, I don't feel like I have the kind of feedback that will help me
get it committed next commitfest. I did get some review, and that was
helpful, but it was mostly on isolated details.

The patch is a million little decisions: names, catalog structure,
interface, representation, general usability, grammar, functionality,
etc. Without some checkpoint, the chances that everyone agrees with all
of these decisions at the beginning of the next commitfest is zero.

Is the commitfest not the right place to do this? If not, then when?

Regards,
Jeff Davis

On Tue, Feb 8, 2011 at 1:02 PM, David Fetter <david(at)fetter(dot)org> wrote:
> Given how things worked, i.e. that people were not clear that 9.1
> development had actually started, etc., I am again proposing that we
> have one more CF starting March 15 to get this all cleaned up.  Yes, I
> know that wasn't the plan, but I also know that we're really, really
> close on a whole bunch of things, some of which have been in the
> offing for years at this point, and we risk giving people the
> impression, if they don't already have it, that if they're not in the
> "inner circle," their patches get lower priority no matter what their
> merits.

I agree that we have some problems in that area - particularly with
writeable CTEs - but prolonging the schedule isn't going to fix that
problem.

I don't think that's entirely fair to people who planned their work
over the last eight months so that their patches would be committed
before the deadline. I both worked hard to make sure the stuff I
cared most about got committed in time, and passed over projects that
I could not get done in time, even though I *could* have gotten them
done given another two months. I realize there are all sorts of good
reasons why people didn't get things done on time, like, say, the need
to earn a living - but having a time frame and sticking with it is
ultimately better for the project, at least in my opinion. If we have
to slip the end of the CommitFest a little to get a few extra things
in, OK, but adding another two months to the development schedule
that's been published for most of a year is a pretty drastic change.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, 2011-02-08 at 06:57 -0500, Stephen Frost wrote:
> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> > - Range Types. This is a large patch which was submitted for the
> > first time to the last CommitFest of the cycle, and the first version
> > that had no open TODO items was posted yesterday, three-quarters of
> > the way through that last CommitFest. Some good review has been done.
> > While more is probably needed, I think we should feel good about
> > what's been accomplished and mark this one Returned with Feedback.
>
> I don't agree w/ punting Range Types. Range Types were discussed as far
> back as the 2010 developer meeting, were discussed quite a bit again
> starting in October and throughout the fall, and Jeff has regularly
> been posting updates to it. Given how thorough Jeff is, my feeling is
> that this patch is more than ready for beta.

I appreciate the sentiment, but in addition to some cleanup, any patch
like this at least requires some discussion. It's a language change
we'll be supporting for a long time.

At minimum, we're a couple hundred emails shy of a real consensus on the
naming ;)

Regards,
Jeff Davis

* Jeff Davis (pgsql(at)j-davis(dot)com) wrote:
> I appreciate the sentiment, but in addition to some cleanup, any patch
> like this at least requires some discussion. It's a language change
> we'll be supporting for a long time.

My feeling was that we have had at least some of that discussion this
past fall... It's not like you went out and developed this completely
outside of any community contact. Perhaps it's just not as
controversial as you're expecting. :)

> At minimum, we're a couple hundred emails shy of a real consensus on the
> naming ;)

*smirk.

Thanks,

Stephen

On Tue, Feb 08, 2011 at 02:04:04PM -0500, Robert Haas wrote:
> On Tue, Feb 8, 2011 at 1:02 PM, David Fetter <david(at)fetter(dot)org> wrote:
> > Given how things worked, i.e. that people were not clear that 9.1
> > development had actually started, etc., I am again proposing that we
> > have one more CF starting March 15 to get this all cleaned up.  Yes, I
> > know that wasn't the plan, but I also know that we're really, really
> > close on a whole bunch of things, some of which have been in the
> > offing for years at this point, and we risk giving people the
> > impression, if they don't already have it, that if they're not in the
> > "inner circle," their patches get lower priority no matter what their
> > merits.
>
> I agree that we have some problems in that area - particularly with
> writeable CTEs - but prolonging the schedule isn't going to fix that
> problem.

What is?

> I don't think that's entirely fair to people who planned their work
> over the last eight months so that their patches would be committed
> before the deadline. I both worked hard to make sure the stuff I
> cared most about got committed in time, and passed over projects that
> I could not get done in time, even though I *could* have gotten them
> done given another two months. I realize there are all sorts of good
> reasons why people didn't get things done on time, like, say, the need
> to earn a living - but having a time frame and sticking with it is
> ultimately better for the project, at least in my opinion. If we have
> to slip the end of the CommitFest a little to get a few extra things
> in, OK, but adding another two months to the development schedule
> that's been published for most of a year is a pretty drastic change.

This development cycle was a change from other development cycles in
that it "began" before development had closed in the previous cycle.
I will not take a position at the moment as to the wisdom of making
this change, but it's been clear from feedback from lots of
developers, even ones who'd be expected to be "in the know," that this
vital piece of information did not gotten out in time.

Let's assume for the moment that we're going with overlapping
development cycles into the future. I'd submit that given the
propagation delay of this change, the schedule for the first iteration
of this never was reasonable, and "slipping" two months is a small
price to pay for the huge flock of things we're epsilon short of
getting.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Tue, 2011-02-08 at 11:56 -0500, Robert Haas wrote:
> It's a 5400 line patch that wasn't completed until the middle of the
> current CommitFest. Nobody has ever submitted a major feature patch
> of that size that got done in a single CommitFest, to my recollection,
> or even half that size.

My concern is that, aside from code, my patch didn't make much progress
this commitfest. And the code progress was mostly me working through my
own TODO list on things like GiST support -- which didn't represent any
real decisions, it was mostly just a matter of code.

Regards,
Jeff Davis

On Tue, Feb 8, 2011 at 2:02 PM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> On Mon, 2011-02-07 at 22:37 -0500, Robert Haas wrote:
>> - Range Types.  This is a large patch which was submitted for the
>> first time to the last CommitFest of the cycle, and the first version
>> that had no open TODO items was posted yesterday, three-quarters of
>> the way through that last CommitFest.  Some good review has been done.
>>  While more is probably needed, I think we should feel good about
>> what's been accomplished and mark this one Returned with Feedback.
>
> I submitted this clearly marked WIP, so I expected that it would likely
> be pushed to 9.2.
>
> However, I don't feel like I have the kind of feedback that will help me
> get it committed next commitfest. I did get some review, and that was
> helpful, but it was mostly on isolated details.
>
> The patch is a million little decisions: names, catalog structure,
> interface, representation, general usability, grammar, functionality,
> etc. Without some checkpoint, the chances that everyone agrees with all
> of these decisions at the beginning of the next commitfest is zero.
>
> Is the commitfest not the right place to do this? If not, then when?

That's a fair question, and I do understand the difficulty. I think a
CommitFest is the right place to do that. On the other hand, as I'm
sure you realize, I'm not keen to hold up 9.1beta for a feature that
isn't going to be committed until 9.2. In the 9.0 and 9.1 cycles, it
was my observation that patches which were submitted to the first or
second CommitFest got a lot of this kind of review and went onto be
committed without a problem, usually in the next CommitFest.
Absorbing patches at the end of the cycle is a lot harder, because
everyone who has been thinking about doing something for the release
wakes up and submits it, often half-finished, often at the very last
minute. Furthermore, it takes more time to review them even if they
ARE code-complete, because it takes more time to do a real
review-to-commit than it does to tell someone "call it a whisker
instead of a potato and delete all the comments about sweet potatoes".

I really don't think that getting this into 9.2 is going to be a
problem. Given the level of interest in this patch, there will be no
problem at all finding someone to do a detailed review when we're not
quite so pressed for time, and I'm willing to put some time into it
myself when I'm not quite so pressed for time. Although it doesn't
feel like it at the moment, we have actually made great strides in
absorbing large patches. We've already absorbed true seriailizability
and SE-Linux integration (cut down basic version) this release cycle,
and it looks like we're going to absorb extensions and hopefully
SQL/MED also. Those are all big, big pieces of work *by
non-committers*, and the fact that they've sailed in with hardly a
cross word is, to me, a very good sign for the future of our
development community.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Tue, Feb 8, 2011 at 2:13 PM, David Fetter <david(at)fetter(dot)org> wrote:
>> I agree that we have some problems in that area - particularly with
>> writeable CTEs - but prolonging the schedule isn't going to fix that
>> problem.
>
> What is?

I think the best solution would probably be to find corporate sponsors
for more PostgreSQL committers, or other senior community members who
might become committers if they had more time to spend on it. The
problem is that there are basically two people who understand the
executor well enough to commit that patch: Tom, and Heikki. And Tom
doesn't really like the feature (for reasons that are totally baffling
to me, BTW). If I had three weeks to spend on that patch, it would be
in by now, and I'd know the executor a lot better too, which would
make things easier for the next guy who wants to do somethign of that
type. But I (with some exceptions) don't get paid to commit other
people's patches, so that limits the amount of time I can spend on it
to (a) time when I'm not working on something that's a higher priority
for my employer and (b) evenings and weekends. I invest as much of
that time as I can in community work (which is why I have "nerd"
tatooed on my forehead) but there's a limited amount of it, and I tend
to invest it in patches where I'm already somewhat familiar with the
relevant portion of the code, because that lets me get through more
patches, if not always the best patches.

> This development cycle was a change from other development cycles in
> that it "began" before development had closed in the previous cycle.
> I will not take a position at the moment as to the wisdom of making
> this change, but it's been clear from feedback from lots of
> developers, even ones who'd be expected to be "in the know," that this
> vital piece of information did not gotten out in time.

The schedule was posted publicly, so I'm not sure how that
communication gap happened.

> Let's assume for the moment that we're going with overlapping
> development cycles into the future.  I'd submit that given the
> propagation delay of this change, the schedule for the first iteration
> of this never was reasonable, and "slipping" two months is a small
> price to pay for the huge flock of things we're epsilon short of
> getting.

I think you're being considerably over-optimistic about how close the
remaining patches are to commit, and even more optimistic about the
effects of another two months on the quality of the release. Here's
my prediction: if we slip the schedule for two months, all the
pressure that now exists to get things wrapped up will disappear, and
we'll be back in approximately the same place two months from now. A
few more things will have been committed, but at least as many new
patches will materialize, so that the remaining volume of work is the
same as before, or even larger. We'll still end up punting the same
number of patches, but in the meantime we'll have managed to rush a
few things in that will destabilize the tree and require months of
extra beta time during which no new work will be able to be committed.

The point is this: We're not going to punt major patches because they
are close to commit but yet some arbitrary deadline has expired.
We're going to punt them because they're NOT close to commit and a
long-agreed deadline has expired. I don't think I'd get much support
if I proposed punting everything that isn't yet committed at
2011-02-15 00:00:00, but it's just a mistake to think that if we only
wait another day or another week, we'll get it all done. We've tried
that before and it usually doesn't work out. The limiting factor
isn't primarily the amount of time that it takes to write the code;
it's the amount of motivation to get it done.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

pgsql(at)j-davis(dot)com (Jeff Davis) writes:
> On Tue, 2011-02-08 at 06:57 -0500, Stephen Frost wrote:
>> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
>> > - Range Types. This is a large patch which was submitted for the
>> > first time to the last CommitFest of the cycle, and the first version
>> > that had no open TODO items was posted yesterday, three-quarters of
>> > the way through that last CommitFest. Some good review has been done.
>> > While more is probably needed, I think we should feel good about
>> > what's been accomplished and mark this one Returned with Feedback.
>>
>> I don't agree w/ punting Range Types. Range Types were discussed as far
>> back as the 2010 developer meeting, were discussed quite a bit again
>> starting in October and throughout the fall, and Jeff has regularly
>> been posting updates to it. Given how thorough Jeff is, my feeling is
>> that this patch is more than ready for beta.
>
> I appreciate the sentiment, but in addition to some cleanup, any patch
> like this at least requires some discussion. It's a language change
> we'll be supporting for a long time.
>
> At minimum, we're a couple hundred emails shy of a real consensus on the
> naming ;)

It's more than a bit sad... The RangeType change has the massive merit
of enabling some substantial development changes, where we can get rid
of whole classes of comparison clauses, and hopefully whole classes of
range errors. That was my favorite would-be feature for 9.1.

It'll take some time to get code changes into systems to use this; the
sooner the feature's in a deployable version of Postgres, the earlier
that kind of thing may start.
--
(format nil "~S(at)~S" "cbbrowne" "gmail.com")
http://www3.sympatico.ca/cbbrowne/internet.html
Colorless green ideas sleep furiously.
-- Noam Chomsky

> This discussion reveals that it's time to start making some
> discussions about what can be accomplished for 9.1 and what must be
> postponed to 9.2. The big ones I think we should postpone are:

First off, I think that this is a little premature. As others have
pointed out, the unusual schedule this development cycle indicates that
we ought to give patch authors *some* leeway, otherwise we're penalizing
patch authors who worked hard on 9.0 Beta compared to people who ignored
9.0. I don't think that extends to another commitfest, but I would be
OK with extending this commitfest by two weeks to ensure that every
feature gets a full review.

> - Range Types. This is a large patch which was submitted for the
> first time to the last CommitFest of the cycle, and the first version
> that had no open TODO items was posted yesterday, three-quarters of
> the way through that last CommitFest. Some good review has been done.
> While more is probably needed, I think we should feel good about
> what's been accomplished and mark this one Returned with Feedback.

Aside from it needing more feedback, and being personally disappointed,
I think this is inevitable. Jeff seems OK with it too.

> - ALTER EXTENSION UPGRADE. This is another large patch which was
> submitted for the first time to the last CommitFest of the cycle. The

I thought we'd *already* decided to postpone this, due to the spec still
being muddy. No?

> - FOR KEY LOCK tables. This patch, unfortunately, has not gotten a
> lot of review. But it's a major, potentially destabilizing change

I think this needs to get a full review before we make any decisions on it.

> - SQL/MED. This patch unfortunately kind of stalled for a long time.
> However, I believe that Heikki is now working actively on the core
> patch, so I'm hopeful for some activity here soon.

I'll point out that SQL/MED with File_FDW would be a majorly useful
feature, even if other FDWs didn't yet work. So a partial commit is
also a possibility.

> - Writeable CTEs. Tom has promised to look at this, but doesn't seem
> to be in a hurry.

I think it would be tremendously unfair to the author to punt this
feature unless there's fundamental flaws in it. Its development started
back in 9.0, and it's never really gotten enough review/attention.

--Josh Berkus

--
-- Josh Berkus
PostgreSQL Experts Inc.
http://www.pgexperts.com

On Tue, 2011-02-08 at 14:25 -0500, Robert Haas wrote:
> > The patch is a million little decisions: names, catalog structure,
> > interface, representation, general usability, grammar, functionality,
> > etc. Without some checkpoint, the chances that everyone agrees with all
> > of these decisions at the beginning of the next commitfest is zero.
> >
> > Is the commitfest not the right place to do this? If not, then when?
>
> That's a fair question, and I do understand the difficulty. I think a
> CommitFest is the right place to do that. On the other hand, as I'm
> sure you realize, I'm not keen to hold up 9.1beta for a feature that
> isn't going to be committed until 9.2.

I'm not asking you to hold it up. Just don't mark it "returned with
feedback" when that is not true, and a week still remains. Erik is still
looking at it, and that might generate some interesting discussion.

> ...everyone who has been thinking about doing something for the release
> wakes up and submits it, often half-finished, often at the very last
> minute.

On the flip side, if we don't provide review to WIP patches during the
3rd commitfest, how do we expect to get anything close to committable on
the 1st commitfest of the next cycle?

> Although it doesn't
> feel like it at the moment, we have actually made great strides in
> absorbing large patches.

I agree completely.

Regards,
Jeff Davis

On Tue, 2011-02-08 at 15:10 -0500, Chris Browne wrote:
> It's more than a bit sad... The RangeType change has the massive merit
> of enabling some substantial development changes, where we can get rid
> of whole classes of comparison clauses, and hopefully whole classes of
> range errors. That was my favorite would-be feature for 9.1.

I appreciate the support.

If you take the feature for a quick spin before the next commitfest,
that would be a big help. If I get it in the first commitfest of 9.2
that may mean some follow-up features, like RANGE KEYs/FKs, and maybe
even RANGE JOIN might have a chance for 9.2 as well. Or, maybe some
other features might find it useful, like partitioning or audit logs.

Regards,
Jeff Davis

pgsql(at)j-davis(dot)com (Jeff Davis) writes:
> On Tue, 2011-02-08 at 15:10 -0500, Chris Browne wrote:
>> It's more than a bit sad... The RangeType change has the massive merit
>> of enabling some substantial development changes, where we can get rid
>> of whole classes of comparison clauses, and hopefully whole classes of
>> range errors. That was my favorite would-be feature for 9.1.
>
> I appreciate the support.
>
> If you take the feature for a quick spin before the next commitfest,
> that would be a big help. If I get it in the first commitfest of 9.2
> that may mean some follow-up features, like RANGE KEYs/FKs, and maybe
> even RANGE JOIN might have a chance for 9.2 as well. Or, maybe some
> other features might find it useful, like partitioning or audit logs.

I've found my "wish item"... I wish that queries could expand ranges in
much the same fashion that BETWEEN expands into two query nodes.

That way, you can use a range to pick data from a large table, and not
revert to a Seq Scan+Filter, which is what I'm seeing for the following
sort of query:

select * from some_data where '[2010-01-01,2010-02-01)'::daterange @> whensit;
--
let name="cbbrowne" and tld="gmail.com" in String.concat "@" [name;tld];;
http://linuxfinances.info/info/lsf.html
Rules of the Evil Overlord #162. "If I steal something very important
to the hero, I will not put it on public display.

On Tue, Feb 8, 2011 at 7:58 PM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> On the flip side, if we don't provide review to WIP patches during the
> 3rd commitfest, how do we expect to get anything close to committable on
> the 1st commitfest of the next cycle?

I'm not sure exactly what you're going for here, because I don't think
I've ever proposed any special treatment of patches in the third
CommitFest, and as far as I can remember everything got reviewed
except for the two Tom promised to pick up and then sat on. But if
you were to say that WIP patches *in general* get a lot less review
than non-WIP patches, I would agree with you.

To some extent, I think that's inevitable. It's not fun to review WIP
patches. When you come across something that's screwed up, you say to
yourself - I could mention this in the review, but maybe the author
already knows about it. After all, it's WIP. In other words, it's
hard to know what you should be looking for. I've found that it's
nearly always better to post specific questions that you want to know
the answer to, rather than a patch where people have to guess what
parts you want feedback on.

Now, once the patch is code-complete, I think we should review it.
And I think we usually do a fairly good job with that, even as late as
CF3. It's really CF4 where I think we get a bit less excited about
reviewing patches that aren't going to make the release anyway, and
that's not a great thing, but as procedural defects go it seems better
than most. Yeah, there won't be a lot of big patches committed in
9.2CF1; we don't have the bandwidth to review major patches and get
betas out the door at the same time, or at least we haven't in the
past. But as long as the big patches that would have made 9.2CF1 still
make it into the release, that doesn't seem like a disaster. On the
flip side, if a few more people want to step out and help get the open
items closed, I'd be more than happy to spend the time that I would
otherwise have spent on that reviewing major feature patches for 9.2,
where there's a CommitFest in progress or not.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, 2011-02-10 at 09:46 -0500, Robert Haas wrote:
> On Tue, Feb 8, 2011 at 7:58 PM, Jeff Davis <pgsql(at)j-davis(dot)com> wrote:
> > On the flip side, if we don't provide review to WIP patches during the
> > 3rd commitfest, how do we expect to get anything close to committable on
> > the 1st commitfest of the next cycle?
>
> I'm not sure exactly what you're going for here, because I don't think
> I've ever proposed any special treatment of patches in the third
> CommitFest,

I actually meant 4th (this one). I forgot that the July one was actually
a part of the 9.1 cycle.

> But if
> you were to say that WIP patches *in general* get a lot less review
> than non-WIP patches, I would agree with you.
>
> To some extent, I think that's inevitable. It's not fun to review WIP
> patches.

Agreed, but it doesn't really apply to this situation.

There was still a week left, and the reviewer was still reviewing. So I
found it jarring when you said that it had received enough review, and
bounced it.

In my opinion, if we're going to entertain WIP patches during a
commitfest, we shouldn't bounce them early for being WIP. We can bounce
them for other causes, like "waiting on author" or "we couldn't find a
reviewer" or "we're out of time".

> I've found that it's
> nearly always better to post specific questions that you want to know
> the answer to, rather than a patch where people have to guess what
> parts you want feedback on.

Well, I've certainly posted some specific questions. I don't expect to
get an answer to all of them right away, and certainly many have been
answered -- but I didn't just throw the code out and wait.

For instance:
http://archives.postgresql.org/message-id/1297230650.27157.398.camel@jdavis

Anyway, I don't think any of this affected the patch, I was just
surprised. I'll leave it at that, because I'm sure you're busy wrapping
up this commitfest.

Regards,
Jeff Davis