Time to finalize patches for 8.4 beta

Now that we are two months into the final commit fest, it is time to
finalize all the open patches so we can target a February beta.

The two major outstanding patches are:

o SE-PostgreSQL: The author has done an outstanding job of
reworking the patch so the burden is now on the community. We have to
decide if we want to add this amount of new code to have both SQL row
permissions and SE-Linux support.

o Recovery, Replication, Hot Standby: We need a _final_ version
of any patches that are targeted for 8.4. There is so much activity in
this area I am unclear what is ready for 8.4.

I think the remaining patches can be addressed pretty easily but we need
final versions from any authors who are still adjusting them.

Let's see what we can get done in the next two weeks and reevaluate.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sat, 2009-01-03 at 22:34 -0500, Bruce Momjian wrote:

> o Recovery, Replication, Hot Standby: We need a _final_ version
> of any patches that are targeted for 8.4. There is so much activity in
> this area I am unclear what is ready for 8.4.

What do we mean by _final_? This a serious, polite question.

Hot Standby hasn't had any serious code review yet by anybody, AFAIK.

Hot Standby is not a "WIP" patch. >96-8% of the patch is identical to
the way it was on Nov 1: ready for some review comments and
testing/usability feedback. The later a review happens the less time we
have to change anything within the patch in response to comments. If
anybody is deliberately holding back on review, they need to talk to me,
directly, soon.

The main problem seems to be that all qualified developers are allocated
elsewhere, or working on their own patches.

If we want this in 8.4, then other developers or companies need to
allocate some time to assist with its inclusion. All help greatly
appreciated.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
>
> On Sat, 2009-01-03 at 22:34 -0500, Bruce Momjian wrote:
>
> > o Recovery, Replication, Hot Standby: We need a _final_ version
> > of any patches that are targeted for 8.4. There is so much activity in
> > this area I am unclear what is ready for 8.4.
>
> What do we mean by _final_? This a serious, polite question.
>
> Hot Standby hasn't had any serious code review yet by anybody, AFAIK.
>
> Hot Standby is not a "WIP" patch. >96-8% of the patch is identical to
> the way it was on Nov 1: ready for some review comments and
> testing/usability feedback. The later a review happens the less time we
> have to change anything within the patch in response to comments. If
> anybody is deliberately holding back on review, they need to talk to me,
> directly, soon.
>
> The main problem seems to be that all qualified developers are allocated
> elsewhere, or working on their own patches.
>
> If we want this in 8.4, then other developers or companies need to
> allocate some time to assist with its inclusion. All help greatly
> appreciated.

I assume hot standby is ready for review/commit. The commit fest wiki
has:

o Infrastructure changes for recovery
o Hot Standby - queries during archive recovery
o rmgr hooks and contrib/rmgr_hook
o Synchronous log-shipping replication

Are these all ready for review/commit? I assume all are except I don't
know the answer for the last one.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, 2009-01-05 at 15:05 -0500, Bruce Momjian wrote:

> I assume hot standby is ready for review/commit.

It's in good shape, yes.

With such a large patch I don't expect review->commit to be a single
iteration. Though that isn't because I think there is flaky stuff in
there, just that reviewers may prefer coding/modularisation to be
different than it is.

> The commit fest wiki
> has:
>
> o Infrastructure changes for recovery
> o Hot Standby - queries during archive recovery

This is just one patch now, though Heikki has broken the first piece off
again for review/commit.

> o rmgr hooks and contrib/rmgr_hook

Yes

> o Synchronous log-shipping replication

It is in rework. As *was*, IMHO, no. As agreed, yes, possibly.

> Are these all ready for review/commit? I assume all are except I don't
> know the answer for the last one.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Bruce,

Aside from the big patches you mentioned.

The following patches do not appear to be ready to commit based on most
recent feedback, and should probably be deferred for 8.5:

-- Column-Level permissions
-- Automatic View Update Rules
-- GIN Fast Insert
-- On-Disk Bitmap Index (Gianni says new patch in a week.)
-- Hash-Join Filter Using Bloom Filters
-- Reducing some DDL Locks to ShareLock (remaining unapplied portions)

The following patches are just waiting for one more fix, and have
recently work by authors. We should set a deadline for them:

-- B-Tree GIN (this was reviewed late, so Oleg may fix it)
-- Multi-Batch Hash Join
-- PITR Performance Improvement

These patches might be ready to commit, but need further review. I've
attempted to get reviewers for them, but none of the RRR have the right
skills/experience to do the remaining review needed (i.e. we need some
reviewers to step up from -hackers!):

-- posix_fadvise
-- parallel restore
-- Solve a problem of LC_TIME of windows

Finally, these patches need performance / field testing by a wide
variety of users:

-- posix_fadvise
-- Simple postgresql.conf wizard (or is this pushed into pgFoundry?)
-- pg_upgrade script for 8.3->8.4

On Mon, 2009-01-05 at 17:12 -0800, Josh Berkus wrote:
> -- B-Tree GIN (this was reviewed late, so Oleg may fix it)

This one is really pretty much ready. The only reason I didn't pass it
along for committer review are a couple very minor things that I thought
the authors might want to comment on. If I don't see any comments later
this week, I'll post a new patch.

Regards,
Jeff Davis

Bruce Momjian wrote:
> Now that we are two months into the final commit fest, it is time to
> finalize all the open patches so we can target a February beta.
>
> The two major outstanding patches are:
>
> o SE-PostgreSQL: The author has done an outstanding job of
> reworking the patch so the burden is now on the community. We have to
> decide if we want to add this amount of new code to have both SQL row
> permissions and SE-Linux support.

At least, this patch set has been ready for reviewing for several months,
but it also hasn't get any serious code reviews since CommitFest:May.
(No need to say, I feel grateful for some of proposals, like PGACE
framework, Row-level ACLS, simultaneous DAC&MAC, ...)

I've already prepared to rework the patch set on reviewer's comments,
and our employer allows me to pay my efforts to v8.4 development cycle
with the "highest" priority.

Again, I strongly want it to be progressed.
If necessary, I'll update my patches as soon as possible.

Thanks,

> o Recovery, Replication, Hot Standby: We need a _final_ version
> of any patches that are targeted for 8.4. There is so much activity in
> this area I am unclear what is ready for 8.4.
>
> I think the remaining patches can be addressed pretty easily but we need
> final versions from any authors who are still adjusting them.
>
> Let's see what we can get done in the next two weeks and reevaluate.

--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

On Mon, 2009-01-05 at 17:12 -0800, Josh Berkus wrote:

> -- Reducing some DDL Locks to ShareLock (remaining unapplied portions)

This just needs some testing by me, so don't kick this one out please.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:
> On Mon, 2009-01-05 at 17:12 -0800, Josh Berkus wrote:
>> -- Reducing some DDL Locks to ShareLock (remaining unapplied portions)

> This just needs some testing by me, so don't kick this one out please.

The point of the discussion is that we are not going to wait too darn
much longer for patches that are "waiting on author". If you want this
in 8.4, get that testing finished. Soon.

regards, tom lane

Dear Josh,

On Mon, Jan 05, 2009 at 05:12:27PM -0800, Josh Berkus wrote:

> -- On-Disk Bitmap Index (Gianni says new patch in a week.)

while we are still working on the issues raised on pgsql-hackers, at
this point it looks like that we will not be able to resolve all of
them in time for the 8.4 release; thus we prefer to withdraw the patch
from the current commitfest.

We will update the corresponding Wiki page as soon as possible,
although not immediately (I am experiencing some connectivity
problems).

Best regards,
Dr. Gianni Ciolli - 2ndQuadrant Italia
PostgreSQL Training, Services and Support
gianni(dot)ciolli(at)2ndquadrant(dot)it | www.2ndquadrant.it

On Sat, 2009-01-03 at 22:34 -0500, Bruce Momjian wrote:
> Now that we are two months into the final commit fest, it is time to
> finalize all the open patches so we can target a February beta.
>
> The two major outstanding patches are:
...
> o Recovery, Replication, Hot Standby: We need a _final_ version
> of any patches that are targeted for 8.4. There is so much activity in
> this area I am unclear what is ready for 8.4.
>
> I think the remaining patches can be addressed pretty easily but we need
> final versions from any authors who are still adjusting them.
>
> Let's see what we can get done in the next two weeks and reevaluate.

I'm continuing to work on Hot Standby as we speak, implementing
requested changes and fixing bugs found by myself and others.
Detailed project status is updated as it happens here
http://wiki.postgresql.org/wiki/Hot_Standby#Project_Status_Tracking

Some major refactoring was completed yesterday (after being agreed
earlier that day) and there were still other things being cleaned up.

There have been calls to complete this immediately from a number of
hackers and although I understand the urgency, blurting out a patch that
has not been through a full test cycle will hinder not help this patch.

We expect a full test cycle to be completed on Friday. I will publish a
new version each day on hackers until then, but please don't start
shouting "you haven't tested it" cos the simple truth is we won't have
*re*-tested it to the standard we all expect from PostgreSQL before then
given the nature of testing required.

I also want to reserve the right to discuss things on-list. I don't want
to be forced into discussing things secretly simply to give a good
impression. And I may even ask for more help. If you have a bug or a
question or a refactoring change, ask now - on list.

So let's follow what Bruce suggested above and wait the rest of the two
weeks and then reevaluate. Thanks.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Bruce Momjian wrote:
> Now that we are two months into the final commit fest, it is time to
> finalize all the open patches so we can target a February beta.
>
> The two major outstanding patches are:
>
> o SE-PostgreSQL: The author has done an outstanding job of
> reworking the patch so the burden is now on the community. We have to
> decide if we want to add this amount of new code to have both SQL row
> permissions and SE-Linux support.
>
> o Recovery, Replication, Hot Standby: We need a _final_ version
> of any patches that are targeted for 8.4. There is so much activity in
> this area I am unclear what is ready for 8.4.
>
> I think the remaining patches can be addressed pretty easily but we need
> final versions from any authors who are still adjusting them.
>
> Let's see what we can get done in the next two weeks and reevaluate.

From the recent pgsql-hackers, it seems me people implicitly considers
the last commit-fest is forcibly called off at end of the January.

I believe we can have an opportunity to review our proposed patches at
least, because Tom commented as follows two months ago.
http://archives.postgresql.org/message-id/25086.1227240251@sss.pgh.pa.us

Yes, I can understand the planned schedule to release v8.4 beta is now
coming and we cannot wait for them forever.
However, IIRC, SE-PostgreSQL patches have been neglected for a month
(except for comments from Alvaro Herrera) since I released a revised
version which supports simultaneous DAC and MAC at Dec.17.
I'm afraid of SE-PostgreSQL rejected for v8.4 without serious reviews
due to lack of time, if nothing is done. So, I had to review my patches
by myseld for the recent weeks and rework some items which will be
commented later.

As I said a few times, I can work to upstream it with my highest priority
(my employer also allows it for v8.4), but it is impossible by myself only.
We need all your help!

I'm sorry, if you felt above my concern uncomfortable.

Thanks,
--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>

KaiGai Kohei wrote:
> Bruce Momjian wrote:
> > Now that we are two months into the final commit fest, it is time to
> > finalize all the open patches so we can target a February beta.
> >
> > The two major outstanding patches are:
> >
> > o SE-PostgreSQL: The author has done an outstanding job of
> > reworking the patch so the burden is now on the community. We have to
> > decide if we want to add this amount of new code to have both SQL row
> > permissions and SE-Linux support.
> >
> > o Recovery, Replication, Hot Standby: We need a _final_ version
> > of any patches that are targeted for 8.4. There is so much activity in
> > this area I am unclear what is ready for 8.4.
> >
> > I think the remaining patches can be addressed pretty easily but we need
> > final versions from any authors who are still adjusting them.
> >
> > Let's see what we can get done in the next two weeks and reevaluate.
>
> From the recent pgsql-hackers, it seems me people implicitly considers
> the last commit-fest is forcibly called off at end of the January.

No, we still have many people working on patches and the commit-fest is
going to extend into February.

I am about to post to the 'announce' list saying discussion on this
patch will begin on Wednesday, at 12:00 GMT. The text is below:

---------------------------------------------------------------------------

The PostgreSQL community is considering including security enhancements
in Postgres 8.4, e.g. row-level permissions and SE-Linux security.
However, to evaluate the patch and its usefulness, we need security
experts who want to use this capability or have used it in other
databases.

The most recent version of the patch is mentioned here:

http://archives.postgresql.org/pgsql-hackers/2009-01/msg01680.php

Particularly interesting is the documentation patch:

http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1460.patch

If you know someone who is interested in these features or can help in
discussing them, please have them subscribe to pgsql-hackers here:

http://www.postgresql.org/community/lists/subscribe

Email discussion about this topic will start on Wednesday, 12:00 GMT,
and will include the subject text "SE-PostgreSQL".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On 2009-01-24, at 15:33, Bruce Momjian wrote:
>
> The PostgreSQL community is considering including security
> enhancements
> in Postgres 8.4, e.g. row-level permissions and SE-Linux security.
> However, to evaluate the patch and its usefulness, we need security
> experts who want to use this capability or have used it in other
> databases.

With that sort of features, usually what happens in other projects, is
that despite their evaluation during beta period - they are explicitly
marked and considered as 'beta', even when product reaches release
status.
Maybe PostgresSql should follow that tactic too. After all, security
stuff needs to be tested and exposed for sometime before considered
rock solid, and trustworthy.
Which of course doesn't mean, that it shouldn't start in beta's.

Bruce Momjian wrote:
> KaiGai Kohei wrote:
>> Bruce Momjian wrote:
>>> Now that we are two months into the final commit fest, it is time to
>>> finalize all the open patches so we can target a February beta.
>>>
>>> The two major outstanding patches are:
>>>
>>> o SE-PostgreSQL: The author has done an outstanding job of
>>> reworking the patch so the burden is now on the community. We have to
>>> decide if we want to add this amount of new code to have both SQL row
>>> permissions and SE-Linux support.
>>>
>>> o Recovery, Replication, Hot Standby: We need a _final_ version
>>> of any patches that are targeted for 8.4. There is so much activity in
>>> this area I am unclear what is ready for 8.4.
>>>
>>> I think the remaining patches can be addressed pretty easily but we need
>>> final versions from any authors who are still adjusting them.
>>>
>>> Let's see what we can get done in the next two weeks and reevaluate.
>> From the recent pgsql-hackers, it seems me people implicitly considers
>> the last commit-fest is forcibly called off at end of the January.
>
> No, we still have many people working on patches and the commit-fest is
> going to extend into February.

All right, I'll prepare the discussion on the list.

I know a few SELinux developers good for database also, so I would forward
the message to them. They are surely experts in security field.

Thanks,

> I am about to post to the 'announce' list saying discussion on this
> patch will begin on Wednesday, at 12:00 GMT. The text is below:
>
> ---------------------------------------------------------------------------
>
> The PostgreSQL community is considering including security enhancements
> in Postgres 8.4, e.g. row-level permissions and SE-Linux security.
> However, to evaluate the patch and its usefulness, we need security
> experts who want to use this capability or have used it in other
> databases.
>
> The most recent version of the patch is mentioned here:
>
> http://archives.postgresql.org/pgsql-hackers/2009-01/msg01680.php
>
> Particularly interesting is the documentation patch:
>
> http://sepgsql.googlecode.com/files/sepostgresql-docs-8.4devel-3-r1460.patch

Also, http://wiki.postgresql.org/wiki/SEPostgreSQL

>
> If you know someone who is interested in these features or can help in
> discussing them, please have them subscribe to pgsql-hackers here:
>
> http://www.postgresql.org/community/lists/subscribe
>
> Email discussion about this topic will start on Wednesday, 12:00 GMT,
> and will include the subject text "SE-PostgreSQL".
>
>

--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>

Grzegorz Jaskiewicz wrote:
>
> On 2009-01-24, at 15:33, Bruce Momjian wrote:
> >
> > The PostgreSQL community is considering including security
> > enhancements
> > in Postgres 8.4, e.g. row-level permissions and SE-Linux security.
> > However, to evaluate the patch and its usefulness, we need security
> > experts who want to use this capability or have used it in other
> > databases.
>
> With that sort of features, usually what happens in other projects, is
> that despite their evaluation during beta period - they are explicitly
> marked and considered as 'beta', even when product reaches release
> status.
> Maybe PostgresSql should follow that tactic too. After all, security
> stuff needs to be tested and exposed for sometime before considered
> rock solid, and trustworthy.
> Which of course doesn't mean, that it shouldn't start in beta's.

Once we include the code in an official release, it is much harder to
make changes so I would like to get to 8.4 final at least _thinking_ it
is ready, rather than trying to adjust it after the final release.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +