Re: Getting to 9.3 beta

Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
time to start targeting a date to close it and get to 9.3 beta. I see
25 items will needing attention before we can close it:

https://commitfest.postgresql.org/action/commitfest_view?id=17

What is a reasonable timeframe to target for completion of these items?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
> time to start targeting a date to close it and get to 9.3 beta. I see
> 25 items will needing attention before we can close it:

> https://commitfest.postgresql.org/action/commitfest_view?id=17

> What is a reasonable timeframe to target for completion of these items?

TBH, once Andrew commits the JSON patch, I wouldn't have a problem with
moving all the rest to Returned With Feedback or the next CF. None of
the others seem to me to be close-to-committable (with the possible
exception of the Max LSN patch, which I've not looked at), and April is
not the time to be doing development.

Next week is going to be tied up with the back-branch releases, but
maybe we could target beta for the week after? The main gating factor
at this point really would be how quickly we could write some draft
release notes, so people know what to test.

regards, tom lane

On Fri, Mar 29, 2013 at 4:05 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
>> time to start targeting a date to close it and get to 9.3 beta. I see
>> 25 items will needing attention before we can close it:
>
>> https://commitfest.postgresql.org/action/commitfest_view?id=17
>
>> What is a reasonable timeframe to target for completion of these items?
>
> TBH, once Andrew commits the JSON patch, I wouldn't have a problem with
> moving all the rest to Returned With Feedback or the next CF. None of
> the others seem to me to be close-to-committable (with the possible
> exception of the Max LSN patch, which I've not looked at), and April is
> not the time to be doing development.

I haven't looked through too many of them. as for the one I have on
there myself, the pg_retainxlog one, i think the consensus was to turn
it into a documentation patch - and we can keep adding doc patches
into beta.

But, to the point. It might be a decent starting point to at least
move all patches that are "waiting on author" and doesn't get almost
daily status updates (or has a very clear statement from the author on
when an update can be expected). That'll cut the list down quite a bit
I think, and should make it easier to push harder on the others
perhaps.

> Next week is going to be tied up with the back-branch releases, but
> maybe we could target beta for the week after? The main gating factor
> at this point really would be how quickly we could write some draft
> release notes, so people know what to test.

I think we need to give it at least one more week before we can target
a beta. Packagers will if anything have *more* work to deal with these
backbranch releases than usual, and asking them to push another one
just one (or even two) weeks after that would be rushing it
unnecessarily...

Of course, that doesn't prevent from starting work on the release
notes meanwhile :)

--
Magnus Hagander
Me: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

On 2013-03-29 10:15:42 -0400, Bruce Momjian wrote:
> Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
> time to start targeting a date to close it and get to 9.3 beta. I see
> 25 items will needing attention before we can close it:

Very much agreed!

> https://commitfest.postgresql.org/action/commitfest_view?id=17
>
> What is a reasonable timeframe to target for completion of these items?

Here's my take on it:

- Extension templates:
Not ready yet, probably takes a bit more work till committable, last
version got in rather late.
=> move to next fest

- Performance Improvement by reducing WAL for Update Operation:
Not ready yet, some redesigns are recently made, benefits aren't all
that great
=> move to next fest

- Index based regexp search for pg_trgm:
Seems like the patch is undergoing restructuring of the regex access API
=> move to next fest

- Truncate trailing nulls from heap rows to reduce the size of the null
bitmap:
Not sure, benefits don't seem to be all that big for realistic
usecases?
=> move or reject?

- allowing privileges on untrusted languages:
Direction forward seems unclear, discussion required
=> move

- replace plugins directory with GUC
Peter has agreed to boot it to the next fest afaics
(515357F4(dot)6000307(at)gmx(dot)net)
=> move (done)

- sepgsql: name qualified default security label
has been committed by robert, updated CF

- sepgsql: db_schema:search permission and sepgsql: db_procedure:execute
permission:
Not much review afaics.
=> looks unfair, but unless some comitter (robert?) takes it
on I don't see much alternative to booting it to the next fest?

- Patch to compute Max LSN of Data Pages:
I *personally* don't really see a point in including it in postgres,
there doesn't really seem to be any real demand met by the tool
=> reject?

- Make recovery.conf parameters into GUCs:
Nice to see progress, but this seems to be too late for 9.3.
=> move to -next

- pg_retainxlog for contrib:
5102F7C9(dot)6050604(at)gmx(dot)net sums up my opinion of the utility, just
doesn't seem to be reliable enough to be distributed with postgres.
=> reject/redesign?

- REINDEX CONCURRENTLY:
Imo pretty close to being comittable and pretty useful, but it got
redesigned pretty late and it mostly had review from me and fujii and
it could use a bit more input
=> unclear

- built-in/SQL Command to edit postgresql.conf ("SET PERSISTENT" patch):
Different patches afloat, none is really ready. I personally think
Zoltan's patch is more realistic, but ...
=> move

- pg_ctl idempotent option:
looks ready to me
=> commit

- New statistics for WAL buffer dirty writes:
waiting on other for months
=> returned with feedback

- pg_stat_statements: query, session, and eviction identification:
Seems to need at least docs
=> wait for author, seems to be easy enough?

- Json API and extraction functions:
Seems to be committable
=> commit

- psql watch
Waiting on author since 4 days
=> wait or boot?

- User control over psql's error stream
Unclear, there doesn't seem to be much progress and review
=> return with feedback

- plpgsql_check_function:
Tom says (27661(dot)1364267665(at)sss(dot)pgh(dot)pa(dot)us) that even if the approach
can be aggreed uppon it needs quite a bit more work
=> move

- transforms:
I am not sure what solution is proposed for the dependency issue
between shared objects.
There also hasn't been code level review for a while...
=> unclear

- Check file parameters to psql before prompt for password:
No answer to Stephen's review in december
=> return with feedback

- pkg-config for libpq and ecpg:
Issues seem to be fixable easy enough
=> wait

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 03/29/2013 11:05 AM, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
>> time to start targeting a date to close it and get to 9.3 beta. I see
>> 25 items will needing attention before we can close it:
>> https://commitfest.postgresql.org/action/commitfest_view?id=17
>> What is a reasonable timeframe to target for completion of these items?
> TBH, once Andrew commits the JSON patch, I wouldn't have a problem with
> moving all the rest to Returned With Feedback or the next CF.

I can do that pretty much right away.

cheers

andrew

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Fri, Mar 29, 2013 at 4:05 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Next week is going to be tied up with the back-branch releases, but
>> maybe we could target beta for the week after? The main gating factor
>> at this point really would be how quickly we could write some draft
>> release notes, so people know what to test.

> I think we need to give it at least one more week before we can target
> a beta.

Sure, I was just suggesting a best-case scenario. Probably a more
realistic thing would be to target beta wrap for April 15, which would
leave an extra week or so for documentation polishing and alpha-level
testing.

> Of course, that doesn't prevent from starting work on the release
> notes meanwhile :)

Right ;-)

regards, tom lane

Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> On 2013-03-29 10:15:42 -0400, Bruce Momjian wrote:
>> What is a reasonable timeframe to target for completion of these items?

> Here's my take on it:

Thanks for annotating these! I've commented on the ones where I
come to a different conclusion:

> - replace plugins directory with GUC
> Peter has agreed to boot it to the next fest afaics
> (515357F4(dot)6000307(at)gmx(dot)net)
> => move (done)

It looks to me more like a RWF situation, since Peter has evidently lost
interest in committing this feature at all. So I marked it that way.

> - Patch to compute Max LSN of Data Pages:
> I *personally* don't really see a point in including it in postgres,
> there doesn't really seem to be any real demand met by the tool
> => reject?

Not sure. It certainly hasn't drawn that much enthusiasm on the list,
but it strikes me as the sort of thing that when you need it you really
need it. It would be interesting to hear opinions about it from people
who deal with data recovery situations on a regular basis.

> - REINDEX CONCURRENTLY:
> Imo pretty close to being comittable and pretty useful, but it got
> redesigned pretty late and it mostly had review from me and fujii and
> it could use a bit more input
> => unclear

I think this really has no hope of being bulletproof until we have
MVCC-based catalog scans. The work up to now has been good exploratory
effort, but it's not going to be committable without that
infrastructure, IMHO anyway.

> - psql watch
> Waiting on author since 4 days
> => wait or boot?

If there is agreement on the basic design of the feature, ie
"\watch [n]" as a repeating version of "\g", it seems to me that
this could be fixed and committed in a day or less. I'd be willing
to look at it sometime next week.

> - transforms:
> I am not sure what solution is proposed for the dependency issue
> between shared objects.
> There also hasn't been code level review for a while...
> => unclear

Needs to be moved; no way this is ready in the next week or two.

regards, tom lane

On Fri, Mar 29, 2013 at 11:05:46AM -0400, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
> > time to start targeting a date to close it and get to 9.3 beta. I see
> > 25 items will needing attention before we can close it:
>
> > https://commitfest.postgresql.org/action/commitfest_view?id=17
>
> > What is a reasonable timeframe to target for completion of these items?
>
> TBH, once Andrew commits the JSON patch, I wouldn't have a problem with
> moving all the rest to Returned With Feedback or the next CF. None of
> the others seem to me to be close-to-committable (with the possible
> exception of the Max LSN patch, which I've not looked at), and April is
> not the time to be doing development.
>
> Next week is going to be tied up with the back-branch releases, but
> maybe we could target beta for the week after? The main gating factor
> at this point really would be how quickly we could write some draft
> release notes, so people know what to test.

I have been clearing my calendar with plans to do the 9.3 release notes,
so I am ready to start.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On 2013-03-29 12:28:59 -0400, Tom Lane wrote:
> Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > On 2013-03-29 10:15:42 -0400, Bruce Momjian wrote:
> >> What is a reasonable timeframe to target for completion of these items?
>
> > Here's my take on it:
>
> Thanks for annotating these! I've commented on the ones where I
> come to a different conclusion:
>
> > - replace plugins directory with GUC
> > Peter has agreed to boot it to the next fest afaics
> > (515357F4(dot)6000307(at)gmx(dot)net)
> > => move (done)
>
> It looks to me more like a RWF situation, since Peter has evidently lost
> interest in committing this feature at all. So I marked it that way.

Yea, after a second reading of Peter's email I agree.

> > - Patch to compute Max LSN of Data Pages:
> > I *personally* don't really see a point in including it in postgres,
> > there doesn't really seem to be any real demand met by the tool
> > => reject?
>
> Not sure. It certainly hasn't drawn that much enthusiasm on the list,
> but it strikes me as the sort of thing that when you need it you really
> need it. It would be interesting to hear opinions about it from people
> who deal with data recovery situations on a regular basis.

I only have done so infrequently, but I guess my real problem is that I
can't see it being all that helpful in such a situation. If you've lost
pg_xlog - otherwise there doesn't seem to be much use in this - you can just use
pg_resetxlog to set an insanely high lsn and then dump & restore the
database. The only reason to use something more complex than this is if
you want to continue using the cluster which seems like a really bad
idea.
Am I missing a usecase here?

I guess it can be useful as a template utility to start hacking on for
more complex scenarios, but I don't think that warrants /contrib.

> > - REINDEX CONCURRENTLY:
> > Imo pretty close to being comittable and pretty useful, but it got
> > redesigned pretty late and it mostly had review from me and fujii and
> > it could use a bit more input
> > => unclear
>
> I think this really has no hope of being bulletproof until we have
> MVCC-based catalog scans. The work up to now has been good exploratory
> effort, but it's not going to be committable without that
> infrastructure, IMHO anyway.

I think its fair not to commit it in 9.3, the patch came late and has
undergone significant issues since then. And there are some things in it
I am not sure about yet.
But I am not sure why its only acceptable with MVCC scans? Sure, it
takes an exclusive lock for a very short time when switching the
relations, but thats still a huge step up over the status quo:
* toast indexes currently cannot be created/dropped at all. So you have
to reindex the table which will need to be locked over the whole time.
* you currently cannot replace indexes that are referenced by foreign
keys without doing manual catalog hackery which is definitely unsafe
* you cannot create constraints that use an index concurrently. Parts of
that is emulatable by creating the constraint with an existing index,
but that doesn't help e.g. for exclusion constraints.
* there's no convenient way to issue CREATE INDEX CONCURRENLTY new_index; DROP
INDEX CONCURRENTLY old_index; RENAME new_index old_index; for all indexes.

The last alone imnsho is already a good enough usecase for the patch.

Also, the part that deals with the (locked) switch of the relfilenodes
is a really small part of the patch, improving that later is not going
to throw away much.

Imo having this would have made
e.g. beb850e1d873f8920a78b9b9ee27e9f87c95592f way much less painful. We
(as in 2ndq) had to write scripts for customers to deal with this in a
remotely sane manner. And those were *ugly*.

> > - psql watch
> > Waiting on author since 4 days
> > => wait or boot?
>
> If there is agreement on the basic design of the feature, ie
> "\watch [n]" as a repeating version of "\g", it seems to me that
> this could be fixed and committed in a day or less. I'd be willing
> to look at it sometime next week.

Cool.

> > - transforms:
> > I am not sure what solution is proposed for the dependency issue
> > between shared objects.
> > There also hasn't been code level review for a while...
> > => unclear
>
> Needs to be moved; no way this is ready in the next week or two.

Good. I haven't looked at it in a long time and nobody else did, so I
just wasn't sure.

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

Hello

> - plpgsql_check_function:
> Tom says (27661(dot)1364267665(at)sss(dot)pgh(dot)pa(dot)us) that even if the approach
> can be aggreed uppon it needs quite a bit more work
> => move
>
>
Can we talk about this patch little bit more before moving to next
commitfest?

I would to have some plan to next commitfest. We have to remove some wanted
functionality - possibility to identify more issues in one run, remove
warnings, ... or we have significantly refactor plpgsql parser (two stages)

Next possibility - introduce some new API and move plpgsql_check_function
to external module, although I am thinking so this is important and very
missing functionality (and should be in core) still.

There is no reply, comments to my last update - where I rewrote output,
what was mayor Tom's objection (well specified).

Regards

Pavel

On 2013/03/30, at 2:33, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:

> On 2013-03-29 12:28:59 -0400, Tom Lane
>
>>> - REINDEX CONCURRENTLY:
>>> Imo pretty close to being comittable and pretty useful, but it got
>>> redesigned pretty late and it mostly had review from me and fujii and
>>> it could use a bit more input
>>> => unclear
>>
>> I think this really has no hope of being bulletproof until we have
>> MVCC-based catalog scans. The work up to now has been good exploratory
>> effort, but it's not going to be committable without that
>> infrastructure, IMHO anyway.
>
> I think its fair not to commit it in 9.3, the patch came late and has
> undergone significant issues since then. And there are some things in it
> I am not sure about yet.
> But I am not sure why its only acceptable with MVCC scans? Sure, it
> takes an exclusive lock for a very short time when switching the
> relations, but thats still a huge step up over the status quo:
> * toast indexes currently cannot be created/dropped at all. So you have
> to reindex the table which will need to be locked over the whole time.
> * you currently cannot replace indexes that are referenced by foreign
> keys without doing manual catalog hackery which is definitely unsafe
> * you cannot create constraints that use an index concurrently. Parts of
> that is emulatable by creating the constraint with an existing index,
> but that doesn't help e.g. for exclusion constraints.
> * there's no convenient way to issue CREATE INDEX CONCURRENLTY new_index; DROP
> INDEX CONCURRENTLY old_index; RENAME new_index old_index; for all indexes.
Thanks for the detailed explanation! Just adding that this patch needs more review and attention especially for the toast part where multiple indexes are handled.

As it looks that it is too late for 9.3, well let's move it to the next commit fest and work on that in the next release cycle.

Thanks,
--
Michael

On Fri, Mar 29, 2013 at 8:22 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> - pg_stat_statements: query, session, and eviction identification:
> Seems to need at least docs
> => wait for author, seems to be easy enough?

I would have responded by now, but recent events have unfortunately
made me put a lot of things that take longer than fifteen minutes on
hold. Still, I am watching. Like you say, it's not terribly
invasive. The under-estimation error propagation is perhaps a bit too
weird to justify (even though it's a simple implementation).

--
fdr

On Friday, March 29, 2013 11:04 PM Andres Freund wrote:
> On 2013-03-29 12:28:59 -0400, Tom Lane wrote:
> > Andres Freund <andres(at)2ndquadrant(dot)com> writes:
> > > On 2013-03-29 10:15:42 -0400, Bruce Momjian wrote:
> > >> What is a reasonable timeframe to target for completion of these
> items?
> >
> > > Here's my take on it:
> >
> > Thanks for annotating these! I've commented on the ones where I
> > come to a different conclusion:
> >
> > > - replace plugins directory with GUC
> > > Peter has agreed to boot it to the next fest afaics
> > > (515357F4(dot)6000307(at)gmx(dot)net)
> > > => move (done)
> >
> > It looks to me more like a RWF situation, since Peter has evidently
> lost
> > interest in committing this feature at all. So I marked it that way.
>
> Yea, after a second reading of Peter's email I agree.
>
> > > - Patch to compute Max LSN of Data Pages:
> > > I *personally* don't really see a point in including it in
> postgres,
> > > there doesn't really seem to be any real demand met by the tool
> > > => reject?
> >
> > Not sure. It certainly hasn't drawn that much enthusiasm on the
> list,
> > but it strikes me as the sort of thing that when you need it you
> really
> > need it. It would be interesting to hear opinions about it from
> people
> > who deal with data recovery situations on a regular basis.
>
> I only have done so infrequently, but I guess my real problem is that I
> can't see it being all that helpful in such a situation. If you've lost
> pg_xlog - otherwise there doesn't seem to be much use in this - you can
> just use
> pg_resetxlog to set an insanely high lsn and then dump & restore the
> database.

This utility can give user value that is okay to proceed.

> The only reason to use something more complex than this is if
> you want to continue using the cluster which seems like a really bad
> idea.

I agree that mostly user doesn't need to use cluster further,
but incase if he is aware that only some part of the database (particular
tablespace) has problem
and he can take dump of that part of database and redo some of lost
operations on other tablespaces.

With Regards,
Amit Kapila.

On Fri, Mar 29, 2013 at 7:22 PM, Andres Freund <andres(at)2ndquadrant(dot)com>wrote:

> - Index based regexp search for pg_trgm:
> Seems like the patch is undergoing restructuring of the regex access API
> => move to next fest
>

Last proposal of regex API by Tom seems simple enough in implementation for
me. I'm going to post reworked version today or tomorrow. I hope we can do
one more review cycle in this CF.

------
With best regards,
Alexander Korotkov.

On Fri, Mar 29, 2013 at 11:22 AM, Andres Freund <andres(at)2ndquadrant(dot)com> wrote:
> - sepgsql: db_schema:search permission and sepgsql: db_procedure:execute
> permission:
> Not much review afaics.
> => looks unfair, but unless some comitter (robert?) takes it
> on I don't see much alternative to booting it to the next fest?

I'm going to try to pick these up early next week. I suspect they
will be straightforward.

I'm sorry I've been buried in other things recently. I'm attempting
to get unburied. So far it's only sort-of working, but I keep
hoping...

...Robert

On 29 March 2013 15:05, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>> Our final 9.3 commit-fest has has exceeded the two-month mark, so it is
>> time to start targeting a date to close it and get to 9.3 beta. I see
>> 25 items will needing attention before we can close it:
>
>> https://commitfest.postgresql.org/action/commitfest_view?id=17
>
>> What is a reasonable timeframe to target for completion of these items?
>
> TBH, once Andrew commits the JSON patch, I wouldn't have a problem with
> moving all the rest to Returned With Feedback or the next CF. None of
> the others seem to me to be close-to-committable (with the possible
> exception of the Max LSN patch, which I've not looked at), and April is
> not the time to be doing development.

Agreed

I completely agree with the need to end this CF here/now/soon. As
someone who has normally tried to stretch the limit on what is
possible, I do recognise the need for this to come to an end.
(Certainly my own available time is running out.)

Thanks very much to everybody that tried to get so much into the release.

> Next week is going to be tied up with the back-branch releases, but
> maybe we could target beta for the week after? The main gating factor
> at this point really would be how quickly we could write some draft
> release notes, so people know what to test.

I think we probably need a little more time for open items discussion
and lists (not more feature commits, just docs and
I-said-I'd-change-thats). In the past that has taken a couple of weeks
to work through, so I'd guess there's a few things to sort out there.

So I suggest about 23 April for Beta1.

--
Simon Riggs http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services