Re: 8.4 release planning

--On 23. Januar 2009 21:18:19 -0500 Robert Haas <robertmhaas(at)gmail(dot)com>
wrote:

> In the future, I think we should have an expectation that resubmits
> within the same CommitFest should happen within a week, and that if no
> revision is forthcoming within two weeks the patch is declared dead
> (and the submitter can add it to the next CommitFest when they
> resubmit). Don't think I'm picking on you, either: there was quite a
> bit of it this CommitFest, and it's bad, because:

yes, i'd be happier if i could have spent more time on it, being more
responsive, especially in late December. But it wasn't possible, my fault :(

You're suggestion doesn't help with the problem that (like Joshua already
mentioned) core developers are too busy with reviewing stuff during the
CommitFest. Because of this it's really hard to get the necessary time of
somebody who is able to evaluate the architecture of a new feature and
(more important) its side-effects on the whole system. Especially the last
CommitFest before Feature Freeze is becoming heavily busted with many many
interesting patches, because people want to have their WIP reviewed at
least for the upcoming release (like me). I don't see how postponing
patches would make it better?

The lesson i've learned is to post more, post often and not waiting until
the last CommitFest begins.

Bernd

> You're suggestion doesn't help with the problem that (like Joshua already
> mentioned) core developers are too busy with reviewing stuff during the
> CommitFest. Because of this it's really hard to get the necessary time of
> somebody who is able to evaluate the architecture of a new feature and (more
> important) its side-effects on the whole system. Especially the last
> CommitFest before Feature Freeze is becoming heavily busted with many many
> interesting patches, because people want to have their WIP reviewed at least
> for the upcoming release (like me). I don't see how postponing patches would
> make it better?

Hmm, well, I'm only talking about postponing patches that aren't ready
to be committed, and then only if their authors don't respond to
feedback in a timely fashion. I agree that it would be nice if the
core developers had more time to provide more feedback to more people,
but that seems like an unrelated problem, and in any event I don't
know how to solve it, other than by hoping that we'll eventually have
more core developers.

On the other hand, it's easy to draw a line from the lax criteria for
resubmitting patches to the length of this CommitFest. It now appears
that this CommitFest will be something like 3.5 months long and that
the next one will not occur before May. That means we're essentially
closed to new patches for six months, which is a really long time. To
put it another way, for every week the core team spends reworking the
existing patches, it will be another week before someone can get
feedback on any new patches submitted now. At some point that becomes
a bad trade-off for the project as a whole. Judging when exactly
you've hit that point is difficult, but I'm starting to think we may
have entered the zone of diminishing returns.

...Robert

Robert Haas wrote:
> On the other hand, it's easy to draw a line from the lax criteria for
> resubmitting patches to the length of this CommitFest. It now appears
> that this CommitFest will be something like 3.5 months long and that
> the next one will not occur before May. That means we're essentially
> closed to new patches for six months, which is a really long time. To
> put it another way, for every week the core team spends reworking the
> existing patches, it will be another week before someone can get
> feedback on any new patches submitted now. At some point that becomes
> a bad trade-off for the project as a whole. Judging when exactly
> you've hit that point is difficult, but I'm starting to think we may
> have entered the zone of diminishing returns.

Well, also consider 8.3 was released in February, so we had 9 months of
development before the last commit-fest started.

Also, one thing we have always known is that many of the complex patches
bottle up on the last commit-fest because we kind of realize they are
not going to get much easier to deal with.

As long as we are mostly busy reviewing and committing stuff, and not
bottled up on 1-2 patches, I think the time in commit fest is being well
spent.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Sun, Jan 25, 2009 at 12:28 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Well, also consider 8.3 was released in February, so we had 9 months of
> development before the last commit-fest started.

Yes - that was good. But what will happen for 8.5?

> Also, one thing we have always known is that many of the complex patches
> bottle up on the last commit-fest because we kind of realize they are
> not going to get much easier to deal with.

Yeah... I'm not sure what to do about that, but as Tom pointed out,
it has the disadvantage that all of these massive changes are getting
put into the tree just before we start beta.

...Robert

Robert Haas wrote:
> On Sun, Jan 25, 2009 at 12:28 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > Well, also consider 8.3 was released in February, so we had 9 months of
> > development before the last commit-fest started.
>
> Yes - that was good. But what will happen for 8.5?

Probably the same.

> > Also, one thing we have always known is that many of the complex patches
> > bottle up on the last commit-fest because we kind of realize they are
> > not going to get much easier to deal with.
>
> Yeah... I'm not sure what to do about that, but as Tom pointed out,
> it has the disadvantage that all of these massive changes are getting
> put into the tree just before we start beta.

Well, it is less a problem than in previous releases, so things are
getting better.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

>> Yeah... I'm not sure what to do about that, but as Tom pointed out,
>> it has the disadvantage that all of these massive changes are getting
>> put into the tree just before we start beta.
>
> Well, it is less a problem than in previous releases, so things are
> getting better.

Well, that is good.

I wonder if it would be practical to have a "FeedbackFest" sometime
between now and the release of 8.4 - not to actually commit anything,
but to do preliminary reviews of patches for 8.5 while there's still
plenty of time for them to be revised and submitted earlier in the
cycle. I would be willing to put in quite a bit of time to reviewing
other people's patches if it meant that someone knowledgeable would
take a quick look at mine. (Unfortunately my current area of interest
is the optimizer, so the pool of sufficiently knowledgeable people is
pretty small.)

In any case, thanks for your thoughts on this subject - I appreciate
your taking the time to write back.

...Robert

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Robert Haas wrote:
>> Yeah... I'm not sure what to do about that, but as Tom pointed out,
>> it has the disadvantage that all of these massive changes are getting
>> put into the tree just before we start beta.

> Well, it is less a problem than in previous releases, so things are
> getting better.

Are they? If we stay in commitfest mode until every single thing on the
list has been dealt with, I am not sure this release cycle will be any
shorter than 8.3's was.

We said back at the beginning that

>> The hope here is that we will not have enormous, previously unreviewed
>> patches landing on us at the end of October --- if that happens, we'll
>> be back in the same position we were in at 8.3 feature freeze. Although
>> this schedule allows for the final commit fest to take a good deal of
>> time, we'll reserve the right to reject patches that are too large to be
>> reviewed in a timely fashion. We want to encourage people to do
>> development of large features in an incremental fashion, with a new
>> increment landing during each commit fest.

and I'm beginning to think that we need to invoke that provision.
Particularly with regard to hot standby, which by any sane reading was
not close to being committable on 1 November (a fortiori from the fact
that it's *still* not committable despite large amounts of later work).
I'm also feeling that we are not in a position to commit SE-Postgres in
a timely fashion; which is not KaiGai-san's fault, rather that of the
community which has taken nearly zero interest in that patch.

If we want to ensure that 8.5 development opens soon, what we have to do
is reject those two patches, revert updatable views, and finish up the
other stuff (which is all small and could likely be dealt with in a week
or two). That puts us in position to go beta by perhaps mid-February
with release perhaps on May 1. If we don't, I hereby predict that 8.4
release will not happen before September. Trying to deal with those
late, large features will add *at least* one more month to commitfest
and *at least* one more month to beta (you think they'll be bug-free?).

As our new president has been reminding us, it's time to start making
hard choices.

regards, tom lane

> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.
>
> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two). That puts us in position to go beta by perhaps mid-February
> with release perhaps on May 1. If we don't, I hereby predict that 8.4
> release will not happen before September. Trying to deal with those
> late, large features will add *at least* one more month to commitfest
> and *at least* one more month to beta (you think they'll be bug-free?).

As much as I hate to say it, I agree with all of this. I think it
would be a good to see whether some subset of the Hot Standby can be
committed - perhaps the "infrastructure changes for recovery" portion.
With respect to the remaining patches, I think that anything that is
basically committable now should be committed and everything else
should be considered returned with feedback (or reverted, in the case
of updatable views). Nearly every patch that is still on the
CommitFest wiki has undergone significant changes since the CommitFest
started, and I don't think it's the purpose of CommitFest to give
people another 3 months to work the bugs out of a patch that basically
isn't finished, or to have the committers finish them in lieu of the
original submitters. And if it's really true that we will be in
feature freeze for 10 months (11/1/08-9/1/09) then I don't think
that's remotely a good idea.

I would, however, like to see us make a commitment to actually review
SE-PostgreSQL. There was some talk that we might not want to include
this feature in core at all, and if that is the case then I think it
is long past time to make that decision. Assuming that isn't the
case, then we need to get past the stage where we make occasional
comments on the overall architecture and get down to really reading
the code. I am willing to help with this but I don't have either the
time or the qualifications to do it single-handedly. To be brutally
honest, I don't care about the feature at all: the only thing I ever
do with SELinux is turn it off (row-level DAC is mildly interesting to
me). But I think that if we want to build a community of developers
around PostgreSQL, we'd better at least look at the work they submit.

...Robert

On Sun, Jan 25, 2009 at 12:06 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).

maybe what we need is to put a policy for large patches... something
like: if you submit a large patch that introduce fairly complex new
features very late in the release cycle (maybe at the last commit fest
or previous to that one) then there are no promises about committers
to review them... maybe that will enforce authors to send patches more
often and more early...

and of course, the rest of us that make some kind of review (call it:
testing or code review) should start making those reviews on large
patches (just like someone suggests)

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Tom Lane wrote:
> and I'm beginning to think that we need to invoke that provision.
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.

Yes, few feedbacks have been a concern for me, though I can understand
that reviewers/commiters had to deal with various kind of many patches
for the v8.4 development cycle. (Needless to say, I remember Tom commented
a lot on CommitFest:May and it made SE-PostgreSQL well.)
If SE-PostgreSQL is not still committable phase, I want to discuss what
code should be reworked and what items are remained.
Because it was unclear, I had to review patches by myself in this January. :(

In addition, I would like to mention about the scale of patches.
| % diffstat sepostgresql-sepgsql-8.4devel-3-r1467.patch
says,
| 110 files changed, 9813 insertions(+), 16 deletions(-), 924 modifications(!)
However, the total number of insertions under "src/backend/security" and
"src/include/security" is 8207 lines of 9813. It modifies 924 lines, but
504 lines of 924 is due to a new field of pg_attribute system catalog.
Please note that it never applies massive rough-mannered fixes all over
the core implementation, so it is informidable.

> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two). That puts us in position to go beta by perhaps mid-February
> with release perhaps on May 1. If we don't, I hereby predict that 8.4
> release will not happen before September. Trying to deal with those
> late, large features will add *at least* one more month to commitfest
> and *at least* one more month to beta (you think they'll be bug-free?).

I can understand we cannot postpone to release v8.4 forever and it is
important to see a calendar. However, I would like folks to see not
only a calendar, but a current *trend* also.

Nowaday, we have no days without looking a term such as SaaS/PaaS or
cloud-computing. It consists of various kind of web applications and
highly-integrated database system on cluster systems.
In this movement, end-users pay their attention on "availability" and
"security" most. If we can include these features in a timely fashion,
it makes PostgreSQL more attractive compared to other DBMSs.
Needless to say, I'll pay my maximum effort to reduce the additional
days, even if it is estimated one more month is necessary.

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

Robert Haas wrote:
> I would, however, like to see us make a commitment to actually review
> SE-PostgreSQL. There was some talk that we might not want to include
> this feature in core at all, and if that is the case then I think it
> is long past time to make that decision. Assuming that isn't the
> case, then we need to get past the stage where we make occasional
> comments on the overall architecture and get down to really reading
> the code. I am willing to help with this but I don't have either the
> time or the qualifications to do it single-handedly. To be brutally
> honest, I don't care about the feature at all: the only thing I ever
> do with SELinux is turn it off (row-level DAC is mildly interesting to
> me). But I think that if we want to build a community of developers
> around PostgreSQL, we'd better at least look at the work they submit.

I would like to call for SELinux folks to join reviewing SE-PostgreSQL
from the point of view of security expert. If folks in pgsql-hackers
have questions, I belive they can provide an answer to the questions.

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:

> If we want to ensure that 8.5 development opens soon, what we have to
> do is reject those two patches, revert updatable views, and finish up
> the other stuff (which is all small and could likely be dealt with in
> a week or two). That puts us in position to go beta by perhaps
> mid-February with release perhaps on May 1. If we don't, I hereby
> predict that 8.4 release will not happen before September. Trying to
> deal with those late, large features will add *at least* one more
> month to commitfest and *at least* one more month to beta (you think
> they'll be bug-free?).

> As our new president has been reminding us, it's time to start making
> hard choices.

We're clearly ahead of where we were in 8.3. Any release with big
features in it will take longer, whether you wait a year, or not.

I think the choice here is currently between features or release
schedule, but we could look at things differently and generate some
other options.

What would our users want?

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
> If we want to ensure that 8.5 development opens soon, what we have to
> do is reject those two patches, revert updatable views, and finish up
> the other stuff (which is all small and could likely be dealt with in
> a week or two). That puts us in position to go beta by perhaps
> mid-February with release perhaps on May 1. If we don't, I hereby
> predict that 8.4 release will not happen before September.

Expecting a release around September with those patches a bit
optimistic, IMHO. We already experienced that a release just after
summer is not possible. So, unless there is a way to commit SELinux+Hot
Standby+Sync Replication in the next two weeks (or so), as you wrote, we
should reject those patches and release 8.4 on time.

(Oh, personally I'd like to see all these features at 8.4, from advocacy
side).
--
Devrim GÜNDÜZ, RHCE
devrim~gunduz.org, devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org

Simon Riggs wrote:
> On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
>
>> Any release with big
> features in it will take longer, whether you wait a year, or not.

Well, big features that land early in the release cycle don't delay the
release. Just the ones that are submitted in the last commit fest.

> I think the choice here is currently between features or release
> schedule, but we could look at things differently and generate some
> other options.

Yeah, that's the tradeoff. features vs. release schedule. I don't see
how else you could look at things.

> What would our users want?

I'm sure it depends on the user. Users that are more interested in the
features we already have in the bag like window functions and
WITH-clause, will obviously prefer to release earlier without hot
standby. And users that want hot standby (or SE-postgresql) will prefer
to delay the release and have those features included.

I don't personally have an opinion, because I don't work with any
real-world databases. It's a tradeoff and both options seem good to me.
I'm going to stick around and keep reviewing Hot Standby as long as it
takes, and when it's ready I can commit it to either 8.4 or 8.5. It's
the same amount of work for me either way.

There's still a list of non-resolved issues that I estimate will take
about two weeks to address. And there's a good possibility that new
issues arise, which require yet more time.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Mon, Jan 26, 2009 at 11:35 AM, Heikki Linnakangas
<heikki(dot)linnakangas(at)enterprisedb(dot)com> wrote:

> I'm sure it depends on the user. Users that are more interested in the
> features we already have in the bag like window functions and WITH-clause,
> will obviously prefer to release earlier without hot standby. And users that
> want hot standby (or SE-postgresql) will prefer to delay the release and
> have those features included.

At LinuxLive (UK) the overwhelming majority of people I spoke to over
three days wanted hot standby and replication (preferably
multi-master, but thats another story). Window functions, recursive
queries, SE PostgreSQL, updatable views and other new features were
barely mentioned.

> There's still a list of non-resolved issues that I estimate will take about
> two weeks to address. And there's a good possibility that new issues arise,
> which require yet more time.

That doesn't seem like it's worth deferring such a useful feature for
12+ months for - especially as the work is clearly ongoing and can
happen in parallel with work on other patches.

As I've pointed out before, we're not a commercial company working for
our shareholders, we're a FOSS project working for our end users. If
we can include an important and popular feature like this at the
expense of a few weeks extra wait for the release, it seems to me that
we'll be serving our users far better overall than making a fair
percentage of them wait another 12 months for work that is more or
less complete.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Sun, Jan 25, 2009 at 12:06 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).

While I haven't follwed every detail of this patch set, I'm not quite sure I
see that as being very fair to Simon.

Simon has put a lot of time into Hot Standby and has followed the
pseudo-defacto community process from design through what he believes to be
near-completion; he can't be sure of completion until someone reviews his
work.

Honestly, I'm not trying to marginalize your effort reviewing other patches,
and I know everyone is busy. Hell, as much as I'd love to have this
feature, I too have been unable to find enough time to review Simon's
stuff. However, over the past few months, I seem to recall seeing Simon
submit countless requests for review and regardless of whether it was
completely ready to go November 1st or not, at this time I don't think
anyone but Simon has a complete view of what his patch(es) even look like.
Yet, albeit with almost no review from the committers, Simon has continually
worked through testing, revising his patches, and requesting information and
suggestions from the community.

Frankly Tom, I don't know of anyone in the community with as much experience
in the recovery code as you and Simon. So, any of the major edge case
problems will probably only be found by you regardless of how many of us
review Simon's work.

I do know that this is a feature which a large number of Postgres users
really want and were counting on being in 8.4. Looking forward, if no one
wanted to review these patches in November, and seemingly no one wants to
review them now, how can we expect this to change for 8.5? Can anyone point
out something Simon did wrong in this process?

--
Jonah H. Harris, Senior DBA
myYearbook.com

On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:

> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later
> work).

I've wasted much time in two main areas in recent months:

The first area I've wasted time on is patch review. Had I known that
some patches on the queue would not be properly reviewed and that time
was going to be a factor then I would not have wasted a single second on
patch review. My main review item has been Sync Replication, which I did
because of Core's statement that we would include that feature in this
release. It would appear that my time on that has been both pointless
and damaging to my own situation.

The second is in developing an answer to the FATAL errors problem, which
I asked a direct question on and received a very specific answer:
http://archives.postgresql.org/pgsql-hackers/2008-09/msg01809.php
I spent two weeks working out a difficult solution to that, a week
subsequently arguing with Heikki (possibly longer) about whether it was
necessary and then two weeks refactoring it back out again when you were
silent.

>From my side, I've spent time and money supporting your views and
decisions, without argument with you, though most would not see that. I
learned on Jan 12 that you'd been too busy to consider this work and
that a priori you argued for patch rejection.

It would be much simpler if I had badly screwed up or if we'd found an
awkward showstopper. That hasn't happened. In fact, I've done the full
dance so far.

I fail to see how rejecting unreviewed patches provides benefit for
users, developers or sponsors. How does de-committing from features
voted most-popular or promised by core sit with users? How will
developers react when they know that helping in patch review assists
their own demise? How will sponsors react when they see another set of
patches rejected?

If there is a problem with timing it's because we need more planning,
scheduling and prioritisation of resources. But it isn't possible to do
it retrospectively without causing problems, and those problems may be
worse than slipping a month, or three.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

> Simon has put a lot of time into Hot Standby and has followed the
> pseudo-defacto community process from design through what he believes to be
> near-completion; he can't be sure of completion until someone reviews his
> work.

I think this is a fair critique.

> Yet, albeit with almost no review from the committers, Simon has continually
> worked through testing, revising his patches, and requesting information and
> suggestions from the community.

There really was not a lot of review from mid-October through the end
of the year. That is partly because Simon was out of commission for
about three weeks and did not respond (in particular) to several
requests to separate ICfR from HS. That having been said, since this
is such an important feature to so many people, I think it would have
been better if more effort could have been put into doing what
additional reviewing was still possible. However, since the turn of
the year, it looks to me like Heikki has actually put quite a bit of
time into reviewing and responding to issues.

Still, I agree that if there's anything we should be putting our
effort into as a community right now, it's this feature. If we got
Hot Standby in the next release and everything else in the CommitFest
got bumped, I think a lot of people would consider that a good trade
(though probably not the authors of the patches that got bumped). For
example, I would much rather see Tom revert the updatable views patch
and work on this than spend the next week fixing updatable views.

At a minimum, I think the following patches from the CommitFest wiki
should be returned with feedback or rejected:

1. SE-PostgreSQL. We handled this one badly, but there's not enough
time to fix it now. 8.5.
2. rmgr hooks and contrib/rmgr_hook. Reject because Tom and Heikki
don't believe it's the right approach. Need better use cases.
3. Synchronous log-shipping replication. We handled this one well,
but it's not in good enough shape. 8.5.
4. pg_upgrade script. I haven't heard much about this in a while...
I am doubtful that it is production-quality, but maybe I'm wrong?
5. Reducing some DDL Locks to ShareLock. No activity in a long time,
no time to wait for this to be finished. 8.5.

...Robert

On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
> Simon Riggs wrote:
> > On Sun, 2009-01-25 at 12:06 -0500, Tom Lane wrote:
> >
> >> Any release with big
> > features in it will take longer, whether you wait a year, or not.
>
> Well, big features that land early in the release cycle don't delay the
> release. Just the ones that are submitted in the last commit fest.

Has that ever happened? :-)

I don't think its chance we get big patches in last commit fest. No
sponsor I ever met was happy to both sponsor and to wait. The two seem
mutually exclusive.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs wrote:
>
> On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
>
> > Well, big features that land early in the release cycle don't delay the
> > release. Just the ones that are submitted in the last commit fest.
>
> Has that ever happened? :-)
>
> I don't think its chance we get big patches in last commit fest. No
> sponsor I ever met was happy to both sponsor and to wait. The two seem
> mutually exclusive.

How come it works for Linux?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Jonah H. Harris wrote:
> Looking forward, if no one
> wanted to review these patches in November,

I did, and many others were active in the discussion too.

> and seemingly no one wants to review them now,

I do. Thank you for your appreciation :-(.

> how can we expect this to change for 8.5? Can anyone point
> out something Simon did wrong in this process?

Not really, except maybe started 6 months too late. Big patches simply
take a long time to mature.

Looking back at the timeline for hot standby, it doesn't look
unreasonable at all:

September: First discussion about the user-visible behavior, transaction
isolation etc. Big architectural decisions are made, like where that
snapshots are taken. "Infrastructure changes for recovery" patch is
reviewed, and pushed to next commitfest because of issues. Discussion
started on how to handle (heavy-weight) locks.

October: Discussion on various administration commands, how to handle
b-tree splits, and some other stuff. More discussion on how to derive
snapshots. First complete hot standby patch is submitted. A patch to
change the way subtransactions is submitted, and committed.

November: Various people test and review the patch, including Mark
Kirkwood and Pavan Deolasee. Bugs are found and fixed. Decision is made
that SIGINT should kill an idle-in-transaction transaction as well.

December: Discussion on slotids and the B-tree killed items problem.

January: Major changes; slotids eliminated, conflict resolution
mechanism rewritten. RestoreBkpBlocks refactoring committed. More bugs
in conflict resolution unearthed. Discussion on adding a GUC.

There has been steady progress, starting from basic design discussions
and decisions, moving on to implementation details. Progress never
stalled for a long time. I can see no wrongdoing on Simon's part, and
there is also no grounds to say that the community has neglected this patch.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

>>> Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Still, I agree that if there's anything we should be putting our
> effort into as a community right now, it's this feature. If we got
> Hot Standby in the next release and everything else in the
CommitFest
> got bumped, I think a lot of people would consider that a good trade

Based on discussions at Milwaukee BarCamp, improvements in replication
would do more to break down barriers to migration to PostgreSQL than
anything else.

-Kevin

Robert Haas wrote:
> At a minimum, I think the following patches from the CommitFest wiki
> should be returned with feedback or rejected:
>
> 1. SE-PostgreSQL. We handled this one badly, but there's not enough
> time to fix it now. 8.5.

Unacceptable.
Please make it clear how many items should be fixed/reworked before
rejecting it due to the lack of time, at least.
(I'm optimistic for its design and quality.)
In addition, why does it impossible to be worked in parallel?
Maybe, I cannot contribute Simon's work, but I can fix/rework
for SE-PostgreSQL in parallel, with my highest priority.

As I noted in another message, the current trend strongly
requires "availability" and "security" for the platform of
SaaS/PaaS or cloud-computing. So, we should also include
SE-PostgreSQL within v8.4. I believe it makes PostgreSQL
more widespreadly applied.

> 2. rmgr hooks and contrib/rmgr_hook. Reject because Tom and Heikki
> don't believe it's the right approach. Need better use cases.
> 3. Synchronous log-shipping replication. We handled this one well,
> but it's not in good enough shape. 8.5.

IMO, this feature is also key factor for "availablity".
If author can have enough activity, we should not give up right now...

> 4. pg_upgrade script. I haven't heard much about this in a while...
> I am doubtful that it is production-quality, but maybe I'm wrong?
> 5. Reducing some DDL Locks to ShareLock. No activity in a long time,
> no time to wait for this to be finished. 8.5.
>
> ...Robert

--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>

On Mon, 2009-01-26 at 12:05 -0300, Alvaro Herrera wrote:
> Simon Riggs wrote:
> >
> > On Mon, 2009-01-26 at 13:35 +0200, Heikki Linnakangas wrote:
> >
> > > Well, big features that land early in the release cycle don't delay the
> > > release. Just the ones that are submitted in the last commit fest.
> >
> > Has that ever happened? :-)
> >
> > I don't think its chance we get big patches in last commit fest. No
> > sponsor I ever met was happy to both sponsor and to wait. The two seem
> > mutually exclusive.
>
> How come it works for Linux?

Not sure. How come up it happens here is a more relevant question.

I can only tell you that people only get their cheque books out when
they are certain somebody else will not do it for them for free.

This happened to you also, or did you have this terrible desire for CRC
checks on data blocks quite late in release cycle? If there hadn't been
blockers you'd be being lightly barbecued on the patch queue also. :-)

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Simon Riggs <simon(at)2ndQuadrant(dot)com> writes:

> I fail to see how rejecting unreviewed patches provides benefit for
> users, developers or sponsors.

Nobody has suggested rejecting either sync replication or standby database.
The debate here is over whether to commit into 8.4 or into 8.5.

Put another way, the choice here is whether to have a half-baked delayed 8.4
release in 6 months or a polished on-time 8.5 release in 12 months. Either way
the feature ships and on a not terribly different timeline either.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's PostGIS support!

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:

> Not really, except maybe started 6 months too late. Big patches simply take a
> long time to mature.
>
> Looking back at the timeline for hot standby, it doesn't look unreasonable at
> all:
>
> September: First discussion about the user-visible behavior, transaction
> isolation etc.

Is that right? I had the impression Simon had started working on it
previously. If so then given that feature freeze was scheduled to be November
1st starting in September does seem like it was more than a bit unrealistic.

Large patches should really be targeted towards the *first* commitfest of the
cycle, not the last. Targeting the last is putting all your eggs in one basket
as far as getting everything right the first time.

Someone else asked why this works for Linux. The reason it works is because
work is *always* committed early in the release cycle. The first couple weeks
of the release cycle are the *only* time significant patches are taken.

The entire rest of the cycle is spent in feature freeze with development
happening exclusively on subsystem maintainer's private branches. When the
next release cycle begins subsystem maintainers send up patches for all the
changes that have happened since the last cycle that they think are ready for
release.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> Put another way, the choice here is whether to have a half-baked delayed 8.4
> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
> the feature ships and on a not terribly different timeline either.

This is pretty much exactly how I see it. *Hot standby is not ready*,
and committing it into 8.4 isn't going to magically make that better.
The earliest we are going to have a HS feature that I would trust my
data to is probably ten-twelve months off. The decision we need to
make now is whether that release will be called 8.4 or 8.5; in the
former case meaning that all the stuff already in 8.4 will not reach
users' hands for close to a year more.

regards, tom lane

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Gregory Stark <stark(at)enterprisedb(dot)com> writes:
>> Put another way, the choice here is whether to have a half-baked delayed 8.4
>> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
>> the feature ships and on a not terribly different timeline either.
>
> This is pretty much exactly how I see it. *Hot standby is not ready*,
> and committing it into 8.4 isn't going to magically make that better.
> The earliest we are going to have a HS feature that I would trust my
> data to is probably ten-twelve months off. The decision we need to
> make now is whether that release will be called 8.4 or 8.5; in the
> former case meaning that all the stuff already in 8.4 will not reach
> users' hands for close to a year more.

I'm not even so much worried about the stuff already in 8.4. I'm worried about
the queue for the first 8.5 commitfest. Nobody's going to be reviewing new
patches until 8.4 is out which will leave all that stuff sitting in limbo for
months. Worse, it'll take months to sort through it and nobody will be able to
work on anything which depends on stuff in that queue for, uh, 2*months.

This is stuff like sync replication, join removal, the posix_fadvise for WAL
replay, index-only scans, etc. If this stuff isn't ready to be committed until
next September we'll be back in the same boat again.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's PostGIS support!

On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Gregory Stark <stark(at)enterprisedb(dot)com> writes:
>> Put another way, the choice here is whether to have a half-baked delayed 8.4
>> release in 6 months or a polished on-time 8.5 release in 12 months. Either way
>> the feature ships and on a not terribly different timeline either.
>
> This is pretty much exactly how I see it. *Hot standby is not ready*,
> and committing it into 8.4 isn't going to magically make that better.
> The earliest we are going to have a HS feature that I would trust my
> data to is probably ten-twelve months off.

So can you give us an idea of what parts of the code are in need of
rethinking etc? I assume you've looked at it now if you can estimate
it's going to take another 10 -12 months?

I ask because I've only seen Heikki doing any in depth public review,
and he suggested a couple of weeks for the outstanding issues he's
aware of.

If there are fundamental problems which will take 10 - 12 months to
resolve to our normal standards, then I do believe 8.5 would be more
appropriate.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Heikki Linnakangas <heikki(dot)linnakangas(at)enterprisedb(dot)com> writes:
> Jonah H. Harris wrote:
>> how can we expect this to change for 8.5? Can anyone point
>> out something Simon did wrong in this process?

> Not really, except maybe started 6 months too late. Big patches simply
> take a long time to mature.

Yeah, exactly. When we decided at PgCon that we needed to accept
replication into the core, we thought that there was a realistic chance
of it being in 8.4 if full-tilt development started *immediately*.
For one reason or another nothing much happened until September.
Simon's certainly been working hard on it since then, but it just
takes longer than four months for something like this to happen.

The alternatives that we realistically have got are:
* ship 8.4 "now" (ie before the summer) without HS, plan for HS in 8.5
* delay 8.4 to late summer and ship it with an unpolished HS feature
* delay 8.4 long enough to have a stable, polished HS feature; I claim
this will mean it's not out before Christmas.

I think anyone who thinks alternatives 2 or 3 will happen faster than
sketched here is living in dreamland.

So my feeling is that prudent project management would suggest that
HS go into 8.5 near the beginning of that development cycle, rather
than trying to push it into 8.4.

regards, tom lane

On 1/26/09, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> > Put another way, the choice here is whether to have a half-baked delayed 8.4
> > release in 6 months or a polished on-time 8.5 release in 12 months. Either way
> > the feature ships and on a not terribly different timeline either.
>
>
> This is pretty much exactly how I see it. *Hot standby is not ready*,
> and committing it into 8.4 isn't going to magically make that better.
> The earliest we are going to have a HS feature that I would trust my
> data to is probably ten-twelve months off. The decision we need to
> make now is whether that release will be called 8.4 or 8.5; in the
> former case meaning that all the stuff already in 8.4 will not reach
> users' hands for close to a year more.

What about a compromise solution: release 8.4 now, then focus on
wrapping up the big ticket items that didn't make it into 8.4 into a
quick (as possible) 8.5 release. This means no fests.

merlin

On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:

> What about a compromise solution: release 8.4 now, then focus on
> wrapping up the big ticket items that didn't make it into 8.4 into a
> quick (as possible) 8.5 release. This means no fests.

That would depend on timing then. Trying to get people to upgrade to 8.4 is
going to be difficult if they're waiting on Hot Standby, which means less
in-the-field testing of the 8.4 code base until the 8.5 release. Similarly,
if we're looking at a quick 8.5 around September/October (having no commit
fests), that means it will probably be early 2011 for 8.6, which is fairly
unacceptable for the other patches currently in the queue.

--
Jonah H. Harris, Senior DBA
myYearbook.com

Dave Page <dpage(at)pgadmin(dot)org> writes:
> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> This is pretty much exactly how I see it. *Hot standby is not ready*,

> So can you give us an idea of what parts of the code are in need of
> rethinking etc? I assume you've looked at it now if you can estimate
> it's going to take another 10 -12 months?

No, I'm just estimating that based on the amount of design churn that's
still going on according to the mailing list discussions. I haven't
looked at the code at all. (If you expect me to sign off on it you can
figure it'll be a couple of months even for that to happen...)

regards, tom lane

On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Dave Page <dpage(at)pgadmin(dot)org> writes:
>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> This is pretty much exactly how I see it. *Hot standby is not ready*,
>
>> So can you give us an idea of what parts of the code are in need of
>> rethinking etc? I assume you've looked at it now if you can estimate
>> it's going to take another 10 -12 months?
>
> No, I'm just estimating that based on the amount of design churn that's
> still going on according to the mailing list discussions. I haven't
> looked at the code at all. (If you expect me to sign off on it you can
> figure it'll be a couple of months even for that to happen...)

Well there is one of the problems imho - the project is getting too
big and the patches are getting too complex for us to be able to rely
on you to sign off on every feature or major patch. We need put more
trust in the judgement of the other committers and senior developers
otherwise the project will simply bottleneck as you get more and more
overworked.

/me hopes that comes across as intended :-)

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

"Jonah H. Harris" <jonah(dot)harris(at)gmail(dot)com> writes:
> On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>> What about a compromise solution: release 8.4 now, then focus on
>> wrapping up the big ticket items that didn't make it into 8.4 into a
>> quick (as possible) 8.5 release. This means no fests.

> That would depend on timing then. Trying to get people to upgrade to 8.4 is
> going to be difficult if they're waiting on Hot Standby, which means less
> in-the-field testing of the 8.4 code base until the 8.5 release.

[ deja vu... ] Just like no one was going to bother upgrading to 8.3
because what they wanted wouldn't be there till 8.4, and the similar
claims we heard about 8.2 and 8.1 before that ...

> Similarly,
> if we're looking at a quick 8.5 around September/October (having no commit
> fests), that means it will probably be early 2011 for 8.6, which is fairly
> unacceptable for the other patches currently in the queue.

Right, one of the major considerations here is allowing other
development to get started again (and not be looking at two years wait
to see the light of day).

regards, tom lane

On 1/26/09, Jonah H. Harris <jonah(dot)harris(at)gmail(dot)com> wrote:
> On Mon, Jan 26, 2009 at 11:11 AM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
> > What about a compromise solution: release 8.4 now, then focus on
> > wrapping up the big ticket items that didn't make it into 8.4 into a
> > quick (as possible) 8.5 release. This means no fests.
>
> That would depend on timing then. Trying to get people to upgrade to 8.4 is
> going to be difficult if they're waiting on Hot Standby, which means less
> in-the-field testing of the 8.4 code base until the 8.5 release. Similarly,
> if we're looking at a quick 8.5 around September/October (having no commit
> fests), that means it will probably be early 2011 for 8.6, which is fairly
> unacceptable for the other patches currently in the queue.

I think that's baked in the cards no matter what. IMO, the issue is
that code to review is building up faster than it is getting reviewed,
so maybe a festless release is nesessary to flush out the difference
(along with the other held over patches from 8.4).

The other thing going on here is that HS is extremely important to the
project. If it was up to me, 100% effort would be geared to getting
it out as quickly as possible. I'm just looking for a way to get HS
in the hands of people as quickly as possible that is fair to
everybody. (I would actually prefer to hold 8.4 for it, but that's
starting to sound unrealistic based on Tom's comments).

merlin

On Mon, Jan 26, 2009 at 11:33 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> > That would depend on timing then. Trying to get people to upgrade to 8.4
> is
> > going to be difficult if they're waiting on Hot Standby, which means less
> > in-the-field testing of the 8.4 code base until the 8.5 release.
>
> [ deja vu... ] Just like no one was going to bother upgrading to 8.3
> because what they wanted wouldn't be there till 8.4, and the similar
> claims we heard about 8.2 and 8.1 before that ...

I'm not trying to be an alarmist, I'm just stating what I saw when I was @
EDB. Customers, especially those with large databases or small admin teams,
would definitely wait for features before upgrading. Some people waited
specifically for HOT or features that would benefit them specifically. My
only gripe with a small window between 8.4 and 8.5 was just that I believe
people would be more likely to wait until 8.5 rather than upgrading twice in
the same year. Though, as I generally like people to be using the latest
version of PG, I'd certainly be happy to be wrong on this.

> > Similarly,
> > if we're looking at a quick 8.5 around September/October (having no
> commit
> > fests), that means it will probably be early 2011 for 8.6, which is
> fairly
> > unacceptable for the other patches currently in the queue.
>
> Right, one of the major considerations here is allowing other
> development to get started again (and not be looking at two years wait
> to see the light of day).

Agreed.

--
Jonah H. Harris, Senior DBA
myYearbook.com

Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> writes:

> Dave Page <dpage(at)pgadmin(dot)org> writes:
>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> This is pretty much exactly how I see it. *Hot standby is not ready*,
>
>> So can you give us an idea of what parts of the code are in need of
>> rethinking etc? I assume you've looked at it now if you can estimate
>> it's going to take another 10 -12 months?
>
> No, I'm just estimating that based on the amount of design churn that's
> still going on according to the mailing list discussions.

That's, uh, not a very useful metric. It should argue in *favour* of the patch
being near completion that the patch is getting feedback and Simon is rapidly
taking such suggestions to heart and making changes responsively.

And incidentally most of the changes Heikki's been making have been
simplifications. Every bit which is simplified is one fewer area to find
further problems in. It's not like he's been finding problems which require
new code which in turn could have more problems.

> I haven't looked at the code at all. (If you expect me to sign off on it you
> can figure it'll be a couple of months even for that to happen...)

Well, given that you're basing your judgements on the mailing list traffic and
haven't read the code I guess we have to go on Heikki's judgement that there
are about two known weeks of work and possibly more afterwards.

The real question is how long will the beta period be with or without Hot
Standby? How many users have been pounding on it so far and how many more will
we get in beta?

If we assume it's about a month before the patch is committed, and then the
beta period is an extra month or two long (on top of what I assume will be
about a month) then we're looking at 3-4 months before 8.4 or about May.

I fear that 3-4 months will already be long enough to back up development on
8.5 and cause a big queue for the first commitfest. Frankly, at 3 months I
think feature freeze has _already_ gone on too long, I don't think we need the
doom scenario of 10-12 months before we create a serious problem again.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Dave Page wrote:
> On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Dave Page <dpage(at)pgadmin(dot)org> writes:
>>> On Mon, Jan 26, 2009 at 3:58 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> This is pretty much exactly how I see it. *Hot standby is not ready*,
>>> So can you give us an idea of what parts of the code are in need of
>>> rethinking etc? I assume you've looked at it now if you can estimate
>>> it's going to take another 10 -12 months?
>> No, I'm just estimating that based on the amount of design churn that's
>> still going on according to the mailing list discussions. I haven't
>> looked at the code at all. (If you expect me to sign off on it you can
>> figure it'll be a couple of months even for that to happen...)
>
> Well there is one of the problems imho - the project is getting too
> big and the patches are getting too complex for us to be able to rely
> on you to sign off on every feature or major patch. We need put more
> trust in the judgement of the other committers and senior developers
> otherwise the project will simply bottleneck as you get more and more
> overworked.

About another issued patch, SE-PostgreSQL has been also said
large and complex. However, I cannot believe the reputation
reflects the correct shape.

As I noted in another message, the main patch of SE-PostgreSQL
changes 110 files, inserts 9813 lines, deletes 16 lines and
modifies 924 lines. It might seem the patch is large/complex.

However, 8207 of 9813 lines are on "src/backend/security/*" and
"src/include/security/*" because of its modular design.
It means most of patch simply adds new files.
The 504 of 924 lines modifications are due to a new field in
pg_attribute system catalog, so it is obvious change.
Rest of relatively-large changes are 293 lines in copy.c and
209 lines in execMain.c.

I don't think it is too complex patch to hesiate reviewing.
So, I believe we can move it to "ready to commit" state
within reasonable term, as long as we make it progress.

Thanks,

[kaigai(at)fedora10 ~]$ diffstat sepostgresql-sepgsql-8.4devel-3-r1467.patch
configure | 113 +
configure.in | 13
src/Makefile.global.in | 1
src/backend/Makefile | 7
src/backend/access/common/heaptuple.c | 35
src/backend/access/common/reloptions.c | 22
src/backend/access/common/tupdesc.c | 12
src/backend/access/heap/heapam.c | 19
src/backend/access/heap/tuptoaster.c | 19
src/backend/bootstrap/bootparse.y | 13
src/backend/bootstrap/bootstrap.c | 8
src/backend/catalog/Makefile | 1
src/backend/catalog/aclchk.c | 2
src/backend/catalog/catalog.c | 4
src/backend/catalog/heap.c | 91 !
src/backend/catalog/index.c | 16
src/backend/catalog/pg_aggregate.c | 3
src/backend/catalog/pg_largeobject.c | 5
src/backend/catalog/pg_proc.c | 6
src/backend/catalog/toasting.c | 3
src/backend/commands/cluster.c | 11
src/backend/commands/copy.c | 293 +++!
src/backend/commands/dbcommands.c | 20
src/backend/commands/functioncmds.c | 29
src/backend/commands/lockcmds.c | 3
src/backend/commands/proclang.c | 6
src/backend/commands/tablecmds.c | 25
src/backend/commands/trigger.c | 25
src/backend/executor/execJunk.c | 6
src/backend/executor/execMain.c | 209 +++
src/backend/executor/execQual.c | 4
src/backend/executor/execScan.c | 40
src/backend/executor/execTuples.c | 19
src/backend/executor/execUtils.c | 10
src/backend/executor/functions.c | 6
src/backend/executor/nodeAgg.c | 5
src/backend/executor/nodeMergejoin.c | 2
src/backend/executor/nodeSubplan.c | 4
src/backend/executor/nodeWindowAgg.c | 4
src/backend/executor/spi.c | 4
src/backend/libpq/be-fsstubs.c | 16
src/backend/nodes/copyfuncs.c | 30
src/backend/nodes/equalfuncs.c | 22
src/backend/nodes/outfuncs.c | 34
src/backend/nodes/readfuncs.c | 43
src/backend/optimizer/plan/createplan.c | 6
src/backend/optimizer/plan/planner.c | 1
src/backend/optimizer/util/clauses.c | 5
src/backend/optimizer/util/relnode.c | 1
src/backend/parser/analyze.c | 46
src/backend/parser/gram.y | 64 !
src/backend/parser/parse_target.c | 64 !
src/backend/postmaster/postmaster.c | 43
src/backend/rewrite/rewriteHandler.c | 3
src/backend/security/Makefile | 23
src/backend/security/pgaceCommon.c | 729 ++++++++++++
src/backend/security/pgaceHooks.c | 1547 ++++++++++++++++++++++++++
src/backend/security/rowacl/rowacl.c | 721 ++++++++++++
src/backend/security/sepgsql/avc.c | 1200 ++++++++++++++++++++
src/backend/security/sepgsql/core.c | 623 ++++++++++
src/backend/security/sepgsql/hooks.c | 1019 +++++++++++++++++
src/backend/security/sepgsql/permissions.c | 785 +++++++++++++
src/backend/security/sepgsql/proxy.c | 1097 ++++++++++++++++++
src/backend/storage/file/fd.c | 7
src/backend/storage/ipc/ipci.c | 2
src/backend/tcop/fastpath.c | 2
src/backend/tcop/pquery.c | 2
src/backend/tcop/utility.c | 3
src/backend/utils/adt/acl.c | 6
src/backend/utils/adt/ri_triggers.c | 25
src/backend/utils/adt/trigfuncs.c | 11
src/backend/utils/cache/catcache.c | 32
src/backend/utils/cache/plancache.c | 12
src/backend/utils/cache/relcache.c | 38
src/backend/utils/cache/syscache.c | 40
src/backend/utils/fmgr/dfmgr.c | 10
src/backend/utils/init/postinit.c | 4
src/backend/utils/misc/guc.c | 58
src/backend/utils/misc/postgresql.conf.sample | 6
src/include/access/htup.h | 68 +
src/include/access/sysattr.h | 9
src/include/access/tupdesc.h | 2
src/include/catalog/heap.h | 11
src/include/catalog/indexing.h | 5
src/include/catalog/pg_attribute.h | 504 !!!!!!!!
src/include/catalog/pg_class.h | 2
src/include/catalog/pg_proc.h | 21
src/include/catalog/pg_proc_fn.h | 3
src/include/catalog/pg_security.h | 31
src/include/catalog/pg_type.h | 1
src/include/executor/executor.h | 11
src/include/executor/tuptable.h | 4
src/include/fmgr.h | 3
src/include/libpq/be-fsstubs.h | 3
src/include/nodes/nodes.h | 3
src/include/nodes/parsenodes.h | 17
src/include/nodes/plannodes.h | 10
src/include/nodes/relation.h | 2
src/include/nodes/security.h | 40
src/include/pg_config.h.in | 3
src/include/security/pgace.h | 181 +++
src/include/security/rowacl.h | 41
src/include/security/sepgsql.h | 241 ++++
src/include/storage/fd.h | 1
src/include/storage/lwlock.h | 1
src/include/utils/acl.h | 7
src/include/utils/catcache.h | 1
src/include/utils/errcodes.h | 7
src/include/utils/rel.h | 18
src/include/utils/syscache.h | 4
110 files changed, 9813 insertions(+), 16 deletions(-), 924 modifications(!)

--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>

Dave Page <dpage(at)pgadmin(dot)org> writes:
> On Mon, Jan 26, 2009 at 4:20 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> ... (If you expect me to sign off on it you can
>> figure it'll be a couple of months even for that to happen...)

> Well there is one of the problems imho - the project is getting too
> big and the patches are getting too complex for us to be able to rely
> on you to sign off on every feature or major patch. We need put more
> trust in the judgement of the other committers and senior developers
> otherwise the project will simply bottleneck as you get more and more
> overworked.

Well, actually, Bruce and I are pretty much leaving it to Heikki's
judgement as to when this patch will be ready to commit. He's spent
far more time on it than either of us. The reason I started this thread
is not to debate when it'll be ready but whether we are prepared to
delay 8.4 for however long it will take, realizing that optimistic
estimates are very likely to be wrong.

regards, tom lane

On Mon, Jan 26, 2009 at 11:44 AM, Jonah H. Harris
<jonah(dot)harris(at)gmail(dot)com> wrote:
> On Mon, Jan 26, 2009 at 11:33 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>
>> > That would depend on timing then. Trying to get people to upgrade to
>> > 8.4 is
>> > going to be difficult if they're waiting on Hot Standby, which means
>> > less
>> > in-the-field testing of the 8.4 code base until the 8.5 release.
>>
>> [ deja vu... ] Just like no one was going to bother upgrading to 8.3
>> because what they wanted wouldn't be there till 8.4, and the similar
>> claims we heard about 8.2 and 8.1 before that ...
>
> I'm not trying to be an alarmist, I'm just stating what I saw when I was @
> EDB. Customers, especially those with large databases or small admin teams,
> would definitely wait for features before upgrading. Some people waited
> specifically for HOT or features that would benefit them specifically. My
> only gripe with a small window between 8.4 and 8.5 was just that I believe
> people would be more likely to wait until 8.5 rather than upgrading twice in
> the same year. Though, as I generally like people to be using the latest
> version of PG, I'd certainly be happy to be wrong on this.

We've got folks working on the upgrade to 8.3; 8.4 isn't on our radar
yet, particularly in view of the fact that it's getting pretty
nebulous when 8.4 will be production-worthy :-).

It would surprise me not at all if we never got around to an 8.4
deployment, irrespective of whether we hold off another 6 months to
get more of hot standby into it or not.

Based on that metric, I suppose I ought to prefer for us to get 8.4
out the door more quickly, and start seeing work progress on the 8.5
backlog, which would presumably include the two large O/S patches
(e.g. - hot standby, SE-PG).

I suppose there are two, possibly 3 directions to consider, with
respective merits and demerits:

1. Suppose we cut things off now, and say "push that stuff to 8.5"...

- Simon and KaiGai will be justifiably angry since they *have* been
doing the right kinds of things, and were expecting to see their
material in 8.4.

- On the other hand, their work would be among the first considered
for 8.5, so that they'd not be behind the way they were for 8.4.

- We *won't* irritate the somewhat numerous set of people with patches
already awaiting
8.5,<http://wiki.postgresql.org/wiki/CommitFest_2009-First> and allow
those patches to "bit rot" in the interim.

2. Alternatively, we may press on...

- We may put off 8.4 for an extra 4-6 months.

- Everyone loses the utility of the features already committed in 8.4
for that extra period of time.

- We irritate everyone that was accepting that their contributions
would be waiting until the first 8.5 commitfest.

3. There's always the possibility of a "worst of all worlds."

- All the irritations and losses of #2

- For whatever reason, we don't get usable hot standby / SEPostgreSQL
at the end of it.

I'm getting increasingly scared of #2 and #3.

From a purely selfish perspective, neither of these features are ones
I was expecting to be using in the short or medium term (I don't
actually see a use case for SELinux, myself). I'll note that as my
bias, but I don't think that invalidates the analysis of the 3
directions.
--
http://linuxfinances.info/info/linuxdistributions.html
Robert Benchley - "Drawing on my fine command of the English
language, I said nothing."

> I think that's baked in the cards no matter what. IMO, the issue is
> that code to review is building up faster than it is getting reviewed,
> so maybe a festless release is nesessary to flush out the difference
> (along with the other held over patches from 8.4).

How is that going to help? People will still write new code in the
meanwhile and some of it may be better or more important than the
stuff that doesn't get committed to 8.4. Artificially saying that
nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't
seem to me to solve anything.

I think we've actually done a very fine job reviewing code for 8.4.
Take a look at how many patches were reviewed and committed. The
problem is that we didn't review the big ones. But at least a part of
that is due to some oversights in assigning reviewers, rather than an
absolute lack of manpower.

> The other thing going on here is that HS is extremely important to the
> project. If it was up to me, 100% effort would be geared to getting
> it out as quickly as possible. I'm just looking for a way to get HS
> in the hands of people as quickly as possible that is fair to
> everybody. (I would actually prefer to hold 8.4 for it, but that's
> starting to sound unrealistic based on Tom's comments).

+1.

...Robert

Robert Haas wrote:
>> The other thing going on here is that HS is extremely important to the
>> project. If it was up to me, 100% effort would be geared to getting
>> it out as quickly as possible. I'm just looking for a way to get HS
>> in the hands of people as quickly as possible that is fair to
>> everybody. (I would actually prefer to hold 8.4 for it, but that's
>> starting to sound unrealistic based on Tom's comments).
>
> +1.

+1 Too.

Really, the most interesting part of 8.4 for us is sync_rep and hot_standby!

Best regards,
--
Laurent COUSTET

On Mon, 2009-01-26 at 12:47 -0500, Robert Haas wrote:

> How is that going to help? People will still write new code in the
> meanwhile and some of it may be better or more important than the
> stuff that doesn't get committed to 8.4. Artificially saying that
> nothing will be allowed for 8.5 that wasn't planned for 8.4 doesn't
> seem to me to solve anything.
>
> I think we've actually done a very fine job reviewing code for 8.4.

We have. This release has been better than any other release in recent
memory. We have had more reviewers, and a more transparent process. That
is a great thing and should only stand to improve for the next release.

> Take a look at how many patches were reviewed and committed. The
> problem is that we didn't review the big ones.

I am not sure this is entirely true. SE Postgres for example has been in
constant review by Bruce and Hot Standby has been in constant review by
Heikki.

I think specifically these two patches suffer from two very distinct
problems.

1. This community in general doesn't understand SE Linux/Postgres. It
also struggles with the very Linux centric nature of it.

2. Hot Standby has two very distinct problems of its own.

A. It is a very much buzzword compliant feature.

B. It *appears* to be very much in dynamic development. The chatter
on the list alone is enough to make conservative hackers very nervous.
Myself included.

I am personally not in favor of delaying 8.4. To me Hot Standby is patch
#8762. It isn't going to make the cut because of timing. That is life.
It will be one of the first in the queue for 8.5.

Or put another way. Nobody would blink about cutting the new \dfU stuff
in favor of getting 8.4 out on time. Why are we delaying for Hot
Standby?

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

All,

So, some feedback to make this decision more difficult:

Users: care about HS more than anything else in the world. I'm
convinced that if we took a staw poll, 80% of our users would be in
favor of waiting for HS. This one feature will make more of a
difference in the number of PG users than any feature since the Windows
port. Maybe more.

on the other hand:

We held back version 4 months 7.4 for Windows, before it became apparent
that there was at least a year more work to do. That was a mistake, and
in many ways HS seems like a similar case.

SE-Linux: this patch has effectively been in development for 2 years
ourside the core process before putting it in; the forked SEPostgres is
in use in production. KaiGai has been available for 20 hours a week (or
more) to troubleshoot issues and change APIs. I really don't see what
the problem is with committing it.

pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a
moving target. I've been waiting for it to settle down so that we can
see if upgrade works. It was always true that pg_upgrade would be among
the last patches tested; we discussed this at pgCon.

==============

Regarding the Commitfests in general: we still haven't perfected this
process. There are a number of issues with it which are hampered by
technology, but a lot more by people. Here's my analysis of what's
changed over the last year:

1) having the last CF on Nov. 1 was a mistake. That put us square in
the path of the US & Christian holidays during the critical integration
phase .. which means we haven't really had 3 months of integration,
we've had *two*.

2) Having the CFs improves visibility. However, as SEPostgres shows, it
doesn't eliminate the ability of major committers to put off dealing
with large complex patches which junior reviewers can't be assigned to.
Particularly if a major reviewer "claims" a patch, but then doesn't
seem to do a lot of review.

3) I don't feel like I got a real handle on the "Round Robin Reviewers"
and got them processing small patches efficiently until the November
review. It just took several cycles to work out how to do it,
particularly given my job change this year.

Better technology would also help, such as automated tracking of patch
changes and when the last time a reviewer spoke up was. Currently, Dave
and I have been doing these things by hand and I know we missed a lot of
patches which stalled. But the main issue is (and will remain) people
and procrastination.

--Josh

On Mon, 2009-01-26 at 11:36 -0800, Josh Berkus wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world. I'm
> convinced that if we took a staw poll, 80% of our users would be in
> favor of waiting for HS. This one feature will make more of a
> difference in the number of PG users than any feature since the Windows
> port. Maybe more.

Maybe. Most of the people I talk to are more interested in Recursive
Queries and Parallel restore. Not that they don't see the benefit of HS
of course just that they already have in place solutions for that
problem.

>
> on the other hand:
>
> We held back version 4 months 7.4 for Windows, before it became apparent
> that there was at least a year more work to do. That was a mistake, and
> in many ways HS seems like a similar case.

Had to read this twice. I think you mean we held back 7.4 four 4 months.
Which I had actually forgotten.

> SE-Linux: this patch has effectively been in development for 2 years
> ourside the core process before putting it in; the forked SEPostgres is
> in use in production. KaiGai has been available for 20 hours a week (or
> more) to troubleshoot issues and change APIs. I really don't see what
> the problem is with committing it.

As I posted earlier I think it is mostly understanding of the
technology.

>
> pg_upgrade hasn't recieved a lot of testing because 8.4 has been such a
> moving target. I've been waiting for it to settle down so that we can
> see if upgrade works. It was always true that pg_upgrade would be among
> the last patches tested; we discussed this at pgCon.
>

I thought pg_upgrade was dead. I am happy to hear it isn't. When can we
see a patch that works on anything?

>
> 1) having the last CF on Nov. 1 was a mistake. That put us square in
> the path of the US & Christian holidays during the critical integration
> phase .. which means we haven't really had 3 months of integration,
> we've had *two*.
>

Its always something though. Its either Christmas or summer or school
starting or school ending or spring break etc...

> Better technology would also help, such as automated tracking of patch
> changes and when the last time a reviewer spoke up was. Currently, Dave
> and I have been doing these things by hand and I know we missed a lot of
> patches which stalled. But the main issue is (and will remain) people
> and procrastination.
>

Nod.

Sincerely,

Joshua D. Drake

> --Josh
>
>
>
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On 1/26/09, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world. I'm convinced
> that if we took a staw poll, 80% of our users would be in favor of waiting
> for HS. This one feature will make more of a difference in the number of PG
> users than any feature since the Windows port. Maybe more.
>
> on the other hand:
>
> We held back version 4 months 7.4 for Windows, before it became apparent
> that there was at least a year more work to do. That was a mistake, and in
> many ways HS seems like a similar case.

Not completely. HS is in much better shape than win32 was when it was
pulled from 7.4...the build system wasn't even in place yet nor any of
the major challenges solved (like fork/exec).

HS is working very well (Simon's ongoing work aside). I am pretty
confident based on my personal testing that it would represent the
project well if committed today.

merlin

Merlin,

> Not completely. HS is in much better shape than win32 was when it was
> pulled from 7.4...the build system wasn't even in place yet nor any of
> the major challenges solved (like fork/exec).
>
> HS is working very well (Simon's ongoing work aside). I am pretty
> confident based on my personal testing that it would represent the
> project well if committed today.

OK. I haven't tried it since early December; that's why I was just
presenting alternatives.

--Josh

On Mon, Jan 26, 2009 at 2:36 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world. I'm convinced
> that if we took a staw poll, 80% of our users would be in favor of waiting
> for HS. This one feature will make more of a difference in the number of PG
> users than any feature since the Windows port. Maybe more.
>

if we think we will need a lot of time to get this in form, there will
be no difference in the time of the release just in the numbers in
which HS comes

>
> SE-Linux: this patch has effectively been in development for 2 years
> ourside the core process before putting it in; the forked SEPostgres is in
> use in production. KaiGai has been available for 20 hours a week (or more)
> to troubleshoot issues and change APIs. I really don't see what the problem
> is with committing it.
>

it hasn't been testing by ours in different platforms (ie: ubuntu has
selinux and i want to give it a try, badly enough i have never used
selinux so this is new to me)...

nor we have any evidence that it doesn't affect to users that doesn't
have any variant of selinux (ie: windows)... the real problem here is
the base of users we enough knowledge of the tool to make some usefull
tests

> ==============
>
> Regarding the Commitfests in general: we still haven't perfected this
> process. There are a number of issues with it which are hampered by
> technology, but a lot more by people. Here's my analysis of what's changed
> over the last year:
>
> 1) having the last CF on Nov. 1 was a mistake. That put us square in the
> path of the US & Christian holidays during the critical integration phase ..
> which means we haven't really had 3 months of integration, we've had *two*.
>

+1

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

All,

>> 1) having the last CF on Nov. 1 was a mistake. That put us square in the
>> path of the US & Christian holidays during the critical integration phase ..
>> which means we haven't really had 3 months of integration, we've had *two*.

Actually, I'm thinking about this again, and made a mistake about the
mistake. The *original plan* was that we were not going to accept any
new patches for Nov-CF. Just revised patches from eariler Fests. We
didn't stick to that, which is mostly why we are still reviewing now.

--Josh

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> So, some feedback to make this decision more difficult:

> Users: care about HS more than anything else in the world.

I don't think this is correct. There are certainly a lot of users who
would like an in-core replication solution, but HS by itself is not that
--- you also need (near) real-time log shipping, which we have already
decided to punt to 8.5. That being the case, I think the argument
that HS is a must-have feature for 8.4 is actually rather weak.

> SE-Linux: this patch has effectively been in development for 2 years
> ourside the core process before putting it in; the forked SEPostgres is
> in use in production. KaiGai has been available for 20 hours a week (or
> more) to troubleshoot issues and change APIs. I really don't see what
> the problem is with committing it.

The problem, in words of one syllable, is that we are not sure we want
it. Do you see a user community clamoring for SEPostgres, or a hacker
community that is willing or able to maintain it? If KaiGai-san got run
over by a bus tomorrow, this patch would be a dead letter, because there
just isn't anyone else who is taking sufficient (any?) interest in it.
That doesn't bode well for its future viability. Compare the likely
audience for it to previous patches of roughly similar complexity,
such as integrated text search or the Windows port, and it's just not
in the ballpark.

The second problem is that we're not sure it's really the right thing,
because we have no one who is competent to review the design from a
security standpoint. But unless we get past the first problem the
second one is moot.

regards, tom lane

Merlin Moncure <mmoncure(at)gmail(dot)com> writes:

> HS is working very well (Simon's ongoing work aside). I am pretty
> confident based on my personal testing that it would represent the
> project well if committed today.

I think a lot of people weren't aware there was anybody testing this patch
other than Simon and Heikki -- I wasn't until just today. I wonder how many
more people are trying it out?

This is one of the changes for the better from the past. Though I still think
a lot more would try it out once it's committed.

Here's a thought experiment. If it was committable *today* would we be willing
to go with it and plan to release with it? Assume that it would *still* mean a
longer beta process, so it would still mean releasing in, say April instead of
February or March.

While personally I want us to switch to a mode where we commit large patches
early in the development cycle I don't believe we would refuse to commit it
today if it was ready. And I can't imagine two weeks would make the
difference either.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Get trained by Bruce Momjian - ask me about EnterpriseDB's PostgreSQL training!

On 1/26/09, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> All,
>
>>> 1) having the last CF on Nov. 1 was a mistake. That put us square in the
>>> path of the US & Christian holidays during the critical integration phase
>>> ..
>>> which means we haven't really had 3 months of integration, we've had
>>> *two*.
>
> Actually, I'm thinking about this again, and made a mistake about the
> mistake. The *original plan* was that we were not going to accept any
> new patches for Nov-CF. Just revised patches from eariler Fests. We
> didn't stick to that, which is mostly why we are still reviewing now.

I don't recall us discussing that, but it sounds like it might help next cycle.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On 1/26/09, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
>
> Merlin Moncure <mmoncure(at)gmail(dot)com> writes:
>
> > HS is working very well (Simon's ongoing work aside). I am pretty
> > confident based on my personal testing that it would represent the
> > project well if committed today.
>
>
> I think a lot of people weren't aware there was anybody testing this patch
> other than Simon and Heikki -- I wasn't until just today. I wonder how many
> more people are trying it out?
>
> This is one of the changes for the better from the past. Though I still think
> a lot more would try it out once it's committed.
>
> Here's a thought experiment. If it was committable *today* would we be willing
> to go with it and plan to release with it? Assume that it would *still* mean a
> longer beta process, so it would still mean releasing in, say April instead of
> February or March.

I think the best person to answer that is Simon. He got a small
reprieve the last time this discussion came up, but it's 'put up or
shut up time'...time's up. Is the thing ready to go?

While i'm 'pro HS', I'm also very much 'pro get 8.4 out asap'. I
think both _may_ be possible. This is probably horribly optimistic.
What I can do however is offer to summarize my own experiences testing
and invite others to do the same. I'd also like to see Simon and or
Heikki make a strong statement on where things stand _right now_ (not
in two weeks) :-)

merlin

Gregory Stark <stark(at)enterprisedb(dot)com> writes:
> Here's a thought experiment. If it was committable *today* would we be
> willing to go with it and plan to release with it? Assume that it
> would *still* mean a longer beta process, so it would still mean
> releasing in, say April instead of February or March.

"April instead of February or March"? If we cut off *all* further patch
acceptance *today*, we might conceivably be ready to release May 1.
It is not physically possible to get to beta from where we are before
mid-February, given the open issues, lack of release notes, and
already-scheduled back-branch updates that will be consuming some of
core's time this week. And I doubt anyone thinks we need less than 2
months in beta.

(Or did you mean April of 2010?)

> While personally I want us to switch to a mode where we commit large patches
> early in the development cycle I don't believe we would refuse to commit it
> today if it was ready. And I can't imagine two weeks would make the
> difference either.

Well, if it were ready, we'd not be having this discussion. The problem
is that it's not ready and no one is very sure about when it will be.

regards, tom lane

Josh Berkus wrote:
> All,
>
> So, some feedback to make this decision more difficult:
>
> Users: care about HS more than anything else in the world. I'm
> convinced that if we took a staw poll, 80% of our users would be in
> favor of waiting for HS. This one feature will make more of a
> difference in the number of PG users than any feature since the
> Windows port. Maybe more.
>
> on the other hand:
>
> We held back version 4 months 7.4 for Windows, before it became
> apparent that there was at least a year more work to do. That was a
> mistake, and in many ways HS seems like a similar case.
>

I can only confirm what Josh is saying here.
We would also assume that 80% have been waiting for Simon's work for
years. In fact, I have been dealing fulltime with PostgreSQL since 1999
and it has been a missing issue since than.
Now that we are so close to fixing this issue for so many people out
there, we should give it all the attention we have and support Simon +
team wherever we can.
I think Simon has responded to all question is almost realtime. We
should take that into consideration.
Also, Simon is focuing on a very open development model - this naturally
means a lot of mailing list traffic. Isn't this what this project is all
about?

I am in favor of giving this patch a useful timeframe for completion.
If people decide to give this patch a chance, we will definitely agree
on putting some significant manpower in here as well.
We are not the only ones who want to see that in.
We already see people saying that they delay migrations because they are
hoping for readable slaves to go in.
Also, in the past 10 years I have been tortured with "when can we have
replication" each and every day ...
I am fed up :). i cannot hear it anymore ("... but MySQL has
replication" *aaaaaaaaaaaaaaaaaaaaaaaaaargh*).

best regards,

hans

--
--
Cybertec Schönig & Schönig GmbH
Professional PostgreSQL Consulting, Support, Training
Gröhrmühlgasse 26, A-2700 Wiener Neustadt
Web: www.postgresql-support.de

On Mon, 2009-01-26 at 20:12 +0000, Gregory Stark wrote:
> Merlin Moncure <mmoncure(at)gmail(dot)com> writes:
>
> > HS is working very well (Simon's ongoing work aside). I am pretty
> > confident based on my personal testing that it would represent the
> > project well if committed today.
>
> I think a lot of people weren't aware there was anybody testing this patch
> other than Simon and Heikki -- I wasn't until just today. I wonder how many
> more people are trying it out?

OK, so maybe its been unclear: Gianni Ciolli has been leading a test
team on this, with other people swapping in and out as they go to work
on other things. There's been multiple test machines running various
kinds of test on different OS. Gianni and I have done many midnight
sessions together on this and his help and support has been invaluable.
If some bugs have got passed that test screening in recent weeks, I
think that's down to the sheer volume of tests and my own descriptions
of how things ought to work and exactly where to focus. Gianni has
located more issues than Heikki has, just.

Hannu provided a lot of early input on the problems of conflict
resolution and various alternatives.

I'm the main developer, but I don't underrate their contributions.

I've had specific bug reports from 6 people and test feedback/questions
from about another 4-5 people; Mark isn't credited because I wasn't
keeping track of them in earl stages of review.

Recent refactoring has been a rich source of regressions and those don't
go away overnight, but they don't take months either.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> So, some feedback to make this decision more difficult:
>
>> Users: care about HS more than anything else in the world.
>
> I don't think this is correct. There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5. That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.
>
>> SE-Linux: this patch has effectively been in development for 2 years
>> ourside the core process before putting it in; the forked SEPostgres is
>> in use in production. KaiGai has been available for 20 hours a week (or
>> more) to troubleshoot issues and change APIs. I really don't see what
>> the problem is with committing it.
>
> The problem, in words of one syllable, is that we are not sure we want
> it. Do you see a user community clamoring for SEPostgres, or a hacker
> community that is willing or able to maintain it? If KaiGai-san got run
> over by a bus tomorrow, this patch would be a dead letter, because there
> just isn't anyone else who is taking sufficient (any?) interest in it.
> That doesn't bode well for its future viability. Compare the likely
> audience for it to previous patches of roughly similar complexity,
> such as integrated text search or the Windows port, and it's just not
> in the ballpark.
>
> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint. But unless we get past the first problem the
> second one is moot.
>

I've never posted to this list before, but I am an SELinux upstream maintainer.

I'd just like to interject here, we (the SELinux community) are very interested
in KaiGai's work and have been looking forward to it being upstreamed for quite
some time.

While we haven't been able to analyze the patches directly to determine whether
the security goals are indeed being met we have had much discussion and
eventually community agreement on the security model being implemented. This
happened years ago and has since been merged into the SELinux reference policy
that practically all SELinux users use (distributions start with the reference
policy and add rules/modules suitable for them).

So the security model has been looked at, though not the implementation and we
do have a community of developers, users and customers interested in this work.

Joshua Brindle

On Mon, 2009-01-26 at 15:47 -0500, Merlin Moncure wrote:

> I'd also like to see Simon and or
> Heikki make a strong statement on where things stand _right now_ (not
> in two weeks) :-)

Well, we just found 2 bugs over the weekend, one of which is a
regression from refactoring.

The second bug is being fixed as part of requested refactoring from
Heikki, discussed today.

I've insisted on full visibility, so all known bugs have been recorded
on Wiki as soon as they are reported, even internally located ones. Most
have been fixed on same day found. Check out the Wiki, any day.

I still have not completed Prepared Transaction support, but if I had
then it just would have slowed down the earlier rework. This is probably
the only hanging offence. Please remember that Heikki's rework proposals
were still under discussion only 2 weeks ago.

Personally, I've very happy with the code as stands, generally, but
Heikki may wish to move a few things around yet. The rework has meant
I've reread most of my own code, so I have found and fixed many of my
own bugs and rewritten comments to better explain things. I found,
reported and fixed the issue of drop tablespace for example. If cynical,
I could have hidden that and waited a month for people to find it.

Going Beta will reveal further feedback on usability, so we might expect
a few tweaks there as people moan. We'll see.

I won't put a time limit on this since people will just add their own
fudge factors to that and we'll be no nearer to a true figure. Without
everybody breathing down my neck it might be 2-3 days, but I'm spending
time reading email and trying to keep my patches alive.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

On 1/26/09 4:28 PM, "Joshua Brindle" <method(at)manicmethod(dot)com> wrote:

> Tom Lane wrote:
>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
<snip>
>>> SE-Linux: this patch has effectively been in development for 2 years
>>> ourside the core process before putting it in; the forked SEPostgres is
>>> in use in production. KaiGai has been available for 20 hours a week (or
>>> more) to troubleshoot issues and change APIs. I really don't see what
>>> the problem is with committing it.
>>
>> The problem, in words of one syllable, is that we are not sure we want
>> it. Do you see a user community clamoring for SEPostgres, or a hacker
>> community that is willing or able to maintain it? If KaiGai-san got run
>> over by a bus tomorrow, this patch would be a dead letter, because there
>> just isn't anyone else who is taking sufficient (any?) interest in it.
>> That doesn't bode well for its future viability. Compare the likely
>> audience for it to previous patches of roughly similar complexity,
>> such as integrated text search or the Windows port, and it's just not
>> in the ballpark.
>>
>> The second problem is that we're not sure it's really the right thing,
>> because we have no one who is competent to review the design from a
>> security standpoint. But unless we get past the first problem the
>> second one is moot.
>>
>
>
> I've never posted to this list before, but I am an SELinux upstream
> maintainer.
>
> I'd just like to interject here, we (the SELinux community) are very
> interested
> in KaiGai's work and have been looking forward to it being upstreamed for
> quite
> some time.
>
> While we haven't been able to analyze the patches directly to determine
> whether
> the security goals are indeed being met we have had much discussion and
> eventually community agreement on the security model being implemented. This
> happened years ago and has since been merged into the SELinux reference policy
> that practically all SELinux users use (distributions start with the reference
> policy and add rules/modules suitable for them).
>
> So the security model has been looked at, though not the implementation and we
> do have a community of developers, users and customers interested in this
> work.
>
I'd just like to echo Josh's sentiments. I'm also active in the SELinux
community, and have been involved in several developments that really needed
a database with mandatory access control mechanisms. Unfortunately, these
developments have all had maintenance requirements that precluded using
KaiGai's code as it was outside not in a commercial distribution. We've been
waiting anxiously for it to be merged upstream.

Additionally, I've talked to many other end users that really want to deploy
LAPP stacks with these security features. They often came to us looking for
us to help them build such systems, but we've had to turn them away as there
was no supported way to build it.

Thanks,
Chad Sellers

On Monday 26 January 2009 2:12:02 pm Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> > So, some feedback to make this decision more difficult:
> >
> > Users: care about HS more than anything else in the world.
>
> I don't think this is correct. There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5.
Precisely. Thank you.

> That being the case, I think the argument that HS is a must-have feature for
8.4 is actually rather weak.
I am just one person, representing one company...but I do agree, fwiw.

Joshua,

> So the security model has been looked at, though not the implementation
> and we do have a community of developers, users and customers interested
> in this work.

Can you please take a look at it ASAP, then? In the next week, we will
probably decide on whether or not to defer SEPostgres until 8.5. The
fact that we haven't gotten a sign-off from any security expert anywhere
is leaning the whole community towards "defer".

--Josh

On Mon, 2009-01-26 at 15:49 -0500, Tom Lane wrote:

> The problem is that it's not ready and no one is very sure about when
> it will be.

With respect, I've done more than any other developer has done to give
you and the community full information on the patch as it develops. I'm
sorry you don't believe what I tell you, but if you don't you should see
the patch for yourself or ask the person reviewing it.

The Wiki has been there for 4 months now, updated daily for last 8 weeks
apart from illness.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Gregory Stark wrote:
>
> I think a lot of people weren't aware there was anybody testing this patch
> other than Simon and Heikki -- I wasn't until just today. I wonder how many
> more people are trying it out?

I've been running the patch (I think since Jan 5) on a couple dev instances
that were used primarily to test if our software worked with what we expected
to turn into 8.4. I can't say I've really been testing the patch specifically,
but rather testing my workplace's software against a version with this patch,
and hadn't noticed any problems.

Gregory Stark wrote:
> I think a lot of people weren't aware there was anybody testing this patch
> ...I wonder how many more people are trying it out?

I think I have an idea to improve this aspect for future commit fests.

For a long time at each of my workplaces I've been running a development
instance against CVS-HEAD just to make sure our software is more future-proof
against up-and-coming releases. We run this system with -enable-debug, asserts,
etc, and accept that it's just a development system not expected to be totally
stable.

If it were just as easy for us to pull from a
"all 'pending-patches' for-commit-fest-nov that pass regression tests"
branch, I'd happily pull from that instead.

I realize in the current system (emailed patches), this would be a horrible
pain to maintain such a branch; but perhaps some of the burden could be
pushed down to the patch submitters (asking them to merge their own changes
into this merged branch). And I hate bringing up the version control
flame war again; but git really would make this easier. If all patches
were on their own branches; the painful merges into this shared branch
would be rare, as the source control system would remember the painful
parts of the merges.

Josh Berkus wrote:
> Joshua,
>
>> So the security model has been looked at, though not the
>> implementation and we do have a community of developers, users and
>> customers interested in this work.
>
> Can you please take a look at it ASAP, then? In the next week, we will
> probably decide on whether or not to defer SEPostgres until 8.5. The
> fact that we haven't gotten a sign-off from any security expert anywhere
> is leaning the whole community towards "defer".
>

Yes, I will look at them to the extent I am able. As I am not familiar with the
postgresql codebase I won't be able to assert the correctness of the hook
placement (that is, where the security functions are called with respect to the
data they are protecting being accessed). The postgresql community should be
more familiar with the hook call sites and hopefully can assist there.

I should be able to handle the security backend and determining whether it
matches the security model we agreed on, but the hook placement is just as
important since a misplaced or missing hook will allow access that should not be
granted.

Joshua Brindle

Ron Mayer <rm_pg(at)cheapcomplexdevices(dot)com> writes:

> I realize in the current system (emailed patches), this would be a horrible
> pain to maintain such a branch; but perhaps some of the burden could be
> pushed down to the patch submitters (asking them to merge their own changes
> into this merged branch).

I've considered maintaining such a repository a few times and dismissed it
when I realized how much work it would be to maintain.

> And I hate bringing up the version control flame war again; but git really
> would make this easier. If all patches were on their own branches; the
> painful merges into this shared branch would be rare, as the source control
> system would remember the painful parts of the merges.

We have git repositories, I still think maintaining a merged tree with dozens
of patches would be a lot of work.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's On-Demand Production Tuning

Joshua Brindle <method(at)manicmethod(dot)com> writes:

> Yes, I will look at them to the extent I am able. As I am not familiar with the
> postgresql codebase I won't be able to assert the correctness of the hook
> placement (that is, where the security functions are called with respect to the
> data they are protecting being accessed). The postgresql community should be
> more familiar with the hook call sites and hopefully can assist there.

I would suggest looking at the documentation (which I assume the patch does
update). If it's not clear that things are in order from the documentation
then either it needs better documentation or something's wrong...

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> > Users: care about HS more than anything else in the world.
>
> I don't think this is correct. There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5. That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.

HS would still be really nice, and I'm a bit concerned that we'll end up
in 8.5 with a similar discussion along the lines of "well, we've only
just committed HS, why don't we release 8.5 with that and wait on
replication till 8.6". I agree that more folks are after replication
than HS, but there is still a user base for it.

> > SE-Linux: this patch has effectively been in development for 2 years
> > ourside the core process before putting it in; the forked SEPostgres is
> > in use in production. KaiGai has been available for 20 hours a week (or
> > more) to troubleshoot issues and change APIs. I really don't see what
> > the problem is with committing it.
>
> The problem, in words of one syllable, is that we are not sure we want
> it. Do you see a user community clamoring for SEPostgres, or a hacker
> community that is willing or able to maintain it? If KaiGai-san got run
> over by a bus tomorrow, this patch would be a dead letter, because there
> just isn't anyone else who is taking sufficient (any?) interest in it.

I'm certainly interested in it and would like to see it in core. Would
I use it immediately? No, because I've so far avoided having to have it
for my systems. I don't expect that to last forever though, at some
point I'll have to implement a system which requires the seperation
provided by SELinux or move to something like Solaris Trusted Extensions
and Oracle Label Security.

> That doesn't bode well for its future viability. Compare the likely
> audience for it to previous patches of roughly similar complexity,
> such as integrated text search or the Windows port, and it's just not
> in the ballpark.

No, it doesn't have as large a user base as the Windows port or
integrated text search. On the other hand, there *are* users out there,
and hackers, who are willing and interested in it for PostgreSQL because
it would give them an alternative to the de-facto standards. Perhaps
it's just me, but historically it seems like there's also been a fair
bit of aid given to trusted/SE implementations by various organizations
who need it, both in terms of funding for others to work on it and in
actual direct development. I realize this is a trivially defeated POV,
but I really feel that 'if you build it, they will come' in this case.

> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint. But unless we get past the first problem the
> second one is moot.

I think the database design for SELinux is pretty well defined.. The
implementation needs to be reviewed, but I think there's few who can
do that, unfortunately.

Thanks,

Stephen

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>> The problem, in words of one syllable, is that we are not sure we want
>> it. Do you see a user community clamoring for SEPostgres, or a hacker
>> community that is willing or able to maintain it?

> No, it doesn't have as large a user base as the Windows port or
> integrated text search. On the other hand, there *are* users out there,
> and hackers, who are willing and interested in it for PostgreSQL because
> it would give them an alternative to the de-facto standards.

Then why has *nobody* stepped up to review the design, much less the
whole patch? The plain truth is that no one appears to care enough to
expend any real effort. But this patch is far too large and invasive
to accept on the basis that only one guy understands it and will/might
continue to maintain it.

I'll risk being rude to make my point: those who want SEPostgres in core
need to put up or shut up. Now, not at some future time. We need
people to sign off that this patch implements the features they want
(not "sounds roughly like some vague future need I might have") and does
so correctly. An incorrect security feature is considerably worse than
useless. And once it's in core we aren't going to have a whole lot of
elbow room to change the definition later.

regards, tom lane

Tom Lane wrote:
> The problem, in words of one syllable, is that we are not sure we want
> it. Do you see a user community clamoring for SEPostgres, or a hacker

This is a chicken-and-egg type of problem.

Security-conscious users, applications, hackers, and customers will
flock towards whichever database product leads in that area.

If some hypothetical database has only minimal security features, I
imagine few security experts would spend a lot of time with the database.

> The second problem is that we're not sure it's really the right thing,
> because we have no one who is competent to review the design from a
> security standpoint. But unless we get past the first problem the
> second one is moot.

Are we underestimating Kaigai Kohei? I seem to see him credited on
the NSA's SELinux pages:
http://www.nsa.gov/research/selinux/contrib.shtml

and it seems his patches there related to postgresql were pretty widely
discussed on the SELinux lists:
http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

On Monday 26 January 2009 6:31:48 pm Ron Mayer wrote:
> Tom Lane wrote:
[...snip...]
> > The second problem is that we're not sure it's really the right thing,
> > because we have no one who is competent to review the design from a
> > security standpoint. But unless we get past the first problem the
> > second one is moot.
>
> Are we underestimating Kaigai Kohei? I seem to see him credited on
> the NSA's SELinux pages:
> http://www.nsa.gov/research/selinux/contrib.shtml
>
> and it seems his patches there related to postgresql were pretty widely
> discussed on the SELinux lists:
> http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

I'm pretty sure that when he says "review" that it is implied to be a "peer
review"

On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:

> Then why has *nobody* stepped up to review the design, much less the
> whole patch? The plain truth is that no one appears to care enough to
> expend any real effort. But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

It was only today that I saw an announcement go out to our announce list
to try and get people to pop their heads up and help. It looks like
today is the day that someone actually reached out to the SELinux folks
and asked for help. We really aren't very good at reaching out for help.
We have a tendency to stick to our little pods, in this case -hackers.

>
> I'll risk being rude to make my point: those who want SEPostgres in core
> need to put up or shut up. Now, not at some future time. We need
> people to sign off that this patch implements the features they want
> (not "sounds roughly like some vague future need I might have") and does
> so correctly. An incorrect security feature is considerably worse than
> useless. And once it's in core we aren't going to have a whole lot of
> elbow room to change the definition later.

I think this is a fair statement to make. I would also note that at
least one other SELinux person has offered today to review at least the
SE portions of this. I quote:

"
Yes, I will look at them to the extent I am able. As I am not familiar
with the postgresql codebase I won't be able to assert the correctness
of the hook placement (that is, where the security functions are called
with respect to the data they are protecting being accessed). The
postgresql community should be more familiar with the hook call sites
and hopefully can assist there.

I should be able to handle the security backend and determining whether
it matches the security model we agreed on, but the hook placement is
just as important since a misplaced or missing hook will allow access
that should not be granted.

Joshua Brindle
"

I do think its important to recognize that SEPostgres is a specialty
feature. Like full disjunctions was, it is a feature that very, very few
will use but those that do really do want and will very much make use of
it.

If I put on my pragmatic helmet it is a high cost, low benefit feature
for the vast majority of our community.

If I put on my advocacy hat it is a, "Hah take that you little blue
mammal!" and frankly "There is no big O in this equation... if you want
this kind of power you gotta ride the elephant!".

There is a strategic element though. Adding features for the too smart
for their own good lures the too smart for their own good to use (and
enhance) a product. This could help us in the long run.

Sincerely,

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Ron Mayer <rm_pg(at)cheapcomplexdevices(dot)com> writes:
> Tom Lane wrote:
>> The second problem is that we're not sure it's really the right thing,
>> because we have no one who is competent to review the design from a
>> security standpoint.

> Are we underestimating Kaigai Kohei?

Perhaps he walks on water, but still I'd like to have more than one
person who has confidence that this design and implementation are correct.

> and it seems his patches there related to postgresql were pretty widely
> discussed on the SELinux lists:
> http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163

Well, a quick look through that thread shows a lot of discussion of the
selinux policy code that's in the patch, which is good as far as it goes
because for sure there's no one in *this* list who understands a line of
that stuff. But to be blunt there's no evidence there that anyone in
that discussion has heard of a foreign key, much less understands why
it might be an issue for this patch. I see a lot of reasoning by
analogy to X servers, and little if any database-specific knowledge.

Mind you, I'd like nothing better than to have some NSA database
security experts (I'm sure there are some) show up here and tell us that
this design is good, secure, and useful --- and why. But right now we
have no evidence for that proposition. And we really need to understand
*why* it's a useful design and what the critical security issues are,
because otherwise we are 100% certain to break it in future maintenance
(even granting the improbable supposition that there are no bugs in the
patch today).

regards, tom lane

Tom Lane wrote:
> Ron Mayer <rm_pg(at)cheapcomplexdevices(dot)com> writes:
>> Are we underestimating Kaigai Kohei?
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.

Totally fair. I know I'm totally unqualified to review his buoyancy on
water, though.

>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>> http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
>
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff.

Totally fair too.

> Mind you, I'd like nothing better than to have some NSA database
> security experts (I'm sure there are some) show up here and tell us that
> this design is good, secure, and useful --- and why. But right now we

What's the right way for us to ask them? No doubt there are some,
but how do we expect them to find & join our email list? If we
wanted more feedback would it make sense for someone who can speak
for the project to call them and ask if they'd be interested
in getting involved?

* Ron Mayer (rm_pg(at)cheapcomplexdevices(dot)com) wrote:
> What's the right way for us to ask them? No doubt there are some,
> but how do we expect them to find & join our email list? If we
> wanted more feedback would it make sense for someone who can speak
> for the project to call them and ask if they'd be interested
> in getting involved?

Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to
-announce (with some additional links to this discussion) to the mailing
list up thread, to which I've also subscribed.

Thanks,

Stephen

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
>> Then why has *nobody* stepped up to review the design, much less the
>> whole patch?

> It was only today that I saw an announcement go out to our announce list
> to try and get people to pop their heads up and help. It looks like
> today is the day that someone actually reached out to the SELinux folks
> and asked for help.

Hmm, you think selinux people read pgsql-announce? But seriously,
we *have* been trying to get people's attention for this patch, both
inside and outside the postgres community, for well over a year now.
The lack of response has been depressing and (IMHO) telling. Nowhere
have we gotten anything more concrete than "ooh, that's cool ... maybe
I might use it someday, but I can't be bothered right now".

regards, tom lane

On Mon, 2009-01-26 at 20:27 -0500, Stephen Frost wrote:
> * Ron Mayer (rm_pg(at)cheapcomplexdevices(dot)com) wrote:
> > What's the right way for us to ask them? No doubt there are some,
> > but how do we expect them to find & join our email list? If we
> > wanted more feedback would it make sense for someone who can speak
> > for the project to call them and ask if they'd be interested
> > in getting involved?
>
> Sorry if it wasn't the 'right way', but I've forwarded Bruce's post to
> -announce (with some additional links to this discussion) to the mailing
> list up thread, to which I've also subscribed.

As have I. Perhaps some humble requests for help that are directed at
the pin point will help.

Joshua D. Drake

>
> Thanks,
>
> Stephen
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Tom Lane wrote:
> Hmm, you think selinux people read pgsql-announce? But seriously,
> we *have* been trying to get people's attention for this patch, both
> inside and outside the postgres community, for well over a year now.
> The lack of response has been depressing and (IMHO) telling. Nowhere
> have we gotten anything more concrete than "ooh, that's cool ... maybe
> I might use it someday, but I can't be bothered right now".

Ah! Then yes, that does say something about the lack of interest.
It wasn't obvious to me that people were reaching out beyond these lists.

* Ron Mayer (rm_pg(at)cheapcomplexdevices(dot)com) wrote:
> Tom Lane wrote:
> > Hmm, you think selinux people read pgsql-announce? But seriously,
> > we *have* been trying to get people's attention for this patch, both
> > inside and outside the postgres community, for well over a year now.
> > The lack of response has been depressing and (IMHO) telling. Nowhere
> > have we gotten anything more concrete than "ooh, that's cool ... maybe
> > I might use it someday, but I can't be bothered right now".
>
> Ah! Then yes, that does say something about the lack of interest.
> It wasn't obvious to me that people were reaching out beyond these lists.

Where were we reaching outside the postgres community..? Just curious
what other SELinux or related lists are out there. As mentioned up
thread, there was interest for a couple of months this past summer on
the NSA SELinux mailing list.

Thanks,

Stephen

On Mon, 2009-01-26 at 20:28 -0500, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
> >> Then why has *nobody* stepped up to review the design, much less the
> >> whole patch?
>
> > It was only today that I saw an announcement go out to our announce list
> > to try and get people to pop their heads up and help. It looks like
> > today is the day that someone actually reached out to the SELinux folks
> > and asked for help.
>
> Hmm, you think selinux people read pgsql-announce?

*shrug* they very well might if they are also postgres people.

> But seriously,
> we *have* been trying to get people's attention for this patch, both
> inside and outside the postgres community, for well over a year now.
> The lack of response has been depressing and (IMHO) telling. Nowhere
> have we gotten anything more concrete than "ooh, that's cool ... maybe
> I might use it someday, but I can't be bothered right now".

Well that is certainly unfortunate. If the very community that invented
the tech isn't willing to help.... :(

Joshua D. Drake

>
> regards, tom lane
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Tom Lane wrote:
> Ron Mayer <rm_pg(at)cheapcomplexdevices(dot)com> writes:
>> Tom Lane wrote:
>>> The second problem is that we're not sure it's really the right thing,
>>> because we have no one who is competent to review the design from a
>>> security standpoint.
>
>> Are we underestimating Kaigai Kohei?
>
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.
>
>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>> http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
>
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff. But to be blunt there's no evidence there that anyone in
> that discussion has heard of a foreign key, much less understands why
> it might be an issue for this patch. I see a lot of reasoning by
> analogy to X servers, and little if any database-specific knowledge.
>

http://marc.info/?l=selinux&m=115762285013528&w=2

Is the original discussion thread for the security model used in the
sepostgresql work. Hopefully you'll see some of the evidence you speak of there.

For some additional resources there is a good chapter on database security
(specifically multilevel databases) in Pfleeger's Security in Computing:
http://www.amazon.com/Security-Computing-Second-Charles-Pfleeger/dp/0133374866/ref=si3_rdr_bb_product

> Mind you, I'd like nothing better than to have some NSA database
> security experts (I'm sure there are some) show up here and tell us that
> this design is good, secure, and useful --- and why. But right now we
> have no evidence for that proposition. And we really need to understand
> *why* it's a useful design and what the critical security issues are,
> because otherwise we are 100% certain to break it in future maintenance
> (even granting the improbable supposition that there are no bugs in the
> patch today).
>

This capability puts PostgreSQL in a unique position. Not only is the security
model more fine grained than any previous trusted database but it also allows
interesting things like trusted stored procedures that can access data (and do
any necessary fuzzing, modifying or filtering) that the client himself cannot read.

It also integrates well with the labeled networking supplied by SELinux, as
outlined on KaiGai's wiki page. There can be multiple clients running on one
machine (such as a web server) that have different levels of database access, in
a more fine grained and flexible way than using different database credentials
allows.

I'm happy to discuss this at length if it will help the patches get upstreamed.

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> * Ron Mayer (rm_pg(at)cheapcomplexdevices(dot)com) wrote:
>> Ah! Then yes, that does say something about the lack of interest.
>> It wasn't obvious to me that people were reaching out beyond these lists.

> Where were we reaching outside the postgres community..?

Well, I've been trying to get Red Hat to interest their NSA contacts in
it, and Bruce and Josh B have been making efforts via EDB and Sun that
I don't have details about, but nothing much has come of any of that.
Maybe waving a red flag in front of the selinux list will have some
effect.

regards, tom lane

Tom Lane wrote:
> Stephen Frost <sfrost(at)snowman(dot)net> writes:
>> * Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
>>> The problem, in words of one syllable, is that we are not sure we want
>>> it. Do you see a user community clamoring for SEPostgres, or a hacker
>>> community that is willing or able to maintain it?
>
>> No, it doesn't have as large a user base as the Windows port or
>> integrated text search. On the other hand, there *are* users out there,
>> and hackers, who are willing and interested in it for PostgreSQL because
>> it would give them an alternative to the de-facto standards.
>
> Then why has *nobody* stepped up to review the design, much less the
> whole patch? The plain truth is that no one appears to care enough to
> expend any real effort. But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

The matter we're currently faced can be called as like a disconnection
between OSS communities.
At least, as several folks introduced in this thread, security focused
people are strongly waiting for SE-PostgreSQL feature upstreamed.
However, we have a wall to be overed, if they join to review the patches,
because most of security experts are not database experts (familiar to
its internal architectures).

In addition, I have hesitated to involve security experts due to the
discussion will need deep knowledge about its internal architectures.
But I think Bruce's suggestion is whorthwhile. At least, it is a case
we need cross-community discussion.

> I'll risk being rude to make my point: those who want SEPostgres in core
> need to put up or shut up. Now, not at some future time. We need
> people to sign off that this patch implements the features they want
> (not "sounds roughly like some vague future need I might have") and does
> so correctly. An incorrect security feature is considerably worse than
> useless. And once it's in core we aren't going to have a whole lot of
> elbow room to change the definition later.

At least, the security design of SE-PostgreSQL has been accepted for
two years in SELinux community. An evidence is its upstreamed security
policy (reference policy) contains rules for SE-PostgreSQL.

http://oss.tresys.com/repos/refpolicy/trunk/policy/modules/services/postgresql.te

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

Jaime Casanova wrote:
>> SE-Linux: this patch has effectively been in development for 2 years
>> ourside the core process before putting it in; the forked SEPostgres is in
>> use in production. KaiGai has been available for 20 hours a week (or more)
>> to troubleshoot issues and change APIs. I really don't see what the problem
>> is with committing it.
>>
>
> it hasn't been testing by ours in different platforms (ie: ubuntu has
> selinux and i want to give it a try, badly enough i have never used
> selinux so this is new to me)...
>
> nor we have any evidence that it doesn't affect to users that doesn't
> have any variant of selinux (ie: windows)... the real problem here is
> the base of users we enough knowledge of the tool to make some usefull
> tests

It is obvious, if you see the code.
When SELinux is disabled by platform or GUC option, security hooks
works nothing.

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> Well, I've been trying to get Red Hat to interest their NSA contacts in
> it, and Bruce and Josh B have been making efforts via EDB and Sun that
> I don't have details about, but nothing much has come of any of that.

It's certainly frustrating to hear that Red Hat, who have been pretty
big proponents of SELinux, not showing interest in having it supported
in PostgreSQL.. :/

> Maybe waving a red flag in front of the selinux list will have some
> effect.

I certainly hope so. I'm planning to review the patch and documentation
and set up an environment and run it through its paces this week.

Thanks,

Stephen

Joshua Brindle wrote:
> While we haven't been able to analyze the patches directly to
> determine whether the security goals are indeed being met we
> have had much discussion and eventually community agreement on
> the security model being implemented. This happened years ago
> and has since been merged into the SELinux reference policy that
> practically all SELinux users use (distributions start with the
> reference policy and add rules/modules suitable for them).
>
> So the security model has been looked at, though not the
> implementation and we do have a community of developers, users
> and customers interested in this work.

That is good to know.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:
> Ron Mayer <rm_pg(at)cheapcomplexdevices(dot)com> writes:
>> Tom Lane wrote:
>>> The second problem is that we're not sure it's really the right thing,
>>> because we have no one who is competent to review the design from a
>>> security standpoint.
>
>> Are we underestimating Kaigai Kohei?
>
> Perhaps he walks on water, but still I'd like to have more than one
> person who has confidence that this design and implementation are correct.
>
>> and it seems his patches there related to postgresql were pretty widely
>> discussed on the SELinux lists:
>> http://www.nsa.gov/research/selinux/list-archive/0805/index.shtml#26163
>
> Well, a quick look through that thread shows a lot of discussion of the
> selinux policy code that's in the patch, which is good as far as it goes
> because for sure there's no one in *this* list who understands a line of
> that stuff. But to be blunt there's no evidence there that anyone in
> that discussion has heard of a foreign key, much less understands why
> it might be an issue for this patch. I see a lot of reasoning by
> analogy to X servers, and little if any database-specific knowledge.

I believe they understand the issues related to covert channels (via PK/FK)
and polyinstantiation, though it is not talked on the thread.
(NOTE: The origin of poluinstantiation is previous security research!)
Yes, there is indeed no evidence. If so, it is a good idea to ask
their opinion with SELinux folks (expect for me?).

As I noted before several times, there are several classes in security
requirements. Historically, TSCEC (Trusted Computer System Evaluation
Criteria, Orange-Book) is a representative evaluation criteria for
security features in IT producets. Now it is inherited to ISO/IEC15408
called as CC (Common Criteria).
We can also consider CC as a set of security functional requirements,
and it defined various kind of requirements.

An interaction between foreign keys and invisible rows is a kind of
covert channel issue. Indeed, it has a possibility users to infer
existance of invisible tuples.
In ISO/IEC15408, FDP_IFF (Information Flow Function, Functionality
of Data Protection) section describes about related requirements.
It defines various classes of requirements. One highest stuff requires
to eliminate all the information-flows via covert-channel, but others
does not require to eliminate at all, or does not mention about it.
Here, it is important what requirement are choosen depends on a
sort of evaluated product, environment, estimated-threat and so on.

NOTE: Oracle Label Security does not care about covert-channels,
because it may also compound for FK/PK issues and row-level
security.
http://www.commoncriteriaportal.org/files/epfiles/20080306_0402b.pdf

In the previous discussion, we (including me) decided that SE-PostgreSQL
also does not care about the FK/PK issues, because polyinstantiation feature
needs unacceptable widespread reworks on PostgreSQL, so I added an explicit
text in the documentation to notice its restriction for end-users.

At least, earlier commercial database (Oracle) does not care, and it can
follow ISO/IEC15408 manner, so I think it is fair enough approach.
It is a summary of previous discussion.

Joshua, Chad, please comment anything, if my understanding was incorrect.

BTW, I have not walked on water yet.

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:

>
> BTW, I have not walked on water yet.
>

I have but I always end up wet. :)

Joshua D. Drake

> Thanks,
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

> -----Original Message-----
> From: pgsql-hackers-owner(at)postgresql(dot)org [mailto:pgsql-hackers-
> owner(at)postgresql(dot)org] On Behalf Of Joshua D. Drake
> Sent: Monday, January 26, 2009 7:42 PM
> To: KaiGai Kohei
> Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure;
> Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd
> Helmle; Peter Eisentraut; pgsql-hackers(at)postgresql(dot)org
> Subject: Re: [HACKERS] 8.4 release planning
>
> On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:
>
> >
> > BTW, I have not walked on water yet.
> >
>
> I have but I always end up wet. :)

I find that it helps to freeze the water first.
;-)

Dann Corbit wrote:
>> -----Original Message-----
>> From: pgsql-hackers-owner(at)postgresql(dot)org [mailto:pgsql-hackers-
>> owner(at)postgresql(dot)org] On Behalf Of Joshua D. Drake
>> Sent: Monday, January 26, 2009 7:42 PM
>> To: KaiGai Kohei
>> Cc: Tom Lane; Ron Mayer; Josh Berkus; Robert Haas; Merlin Moncure;
>> Jonah H. Harris; Gregory Stark; Simon Riggs; Bruce Momjian; Bernd
>> Helmle; Peter Eisentraut; pgsql-hackers(at)postgresql(dot)org
>> Subject: Re: [HACKERS] 8.4 release planning
>>
>> On Tue, 2009-01-27 at 12:30 +0900, KaiGai Kohei wrote:
>>
>>> BTW, I have not walked on water yet.
>>>
>> I have but I always end up wet. :)
>
> I find that it helps to freeze the water first.
> ;-)

I'm sorry for the improper humor.
Shall we return to the discussion?
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

Joshua Brindle <method(at)manicmethod(dot)com> writes:
> http://marc.info/?l=selinux&m=115762285013528&w=2
> Is the original discussion thread for the security model used in the
> sepostgresql work. Hopefully you'll see some of the evidence you speak of there.

Thanks for the link. I took a look through that thread and saw a lot of
discussion about issues like how to relate the database-side and
client-side permissions, which is all good stuff but mostly outside my
purview as a database geek. I didn't find anything about the stuff that
is really bothering me, which I think can be broken down into two main
categories:

1. Silently filtering out rows according to an arbitrary security policy
can break a bunch of fundamental SQL semantics, the most obvious being
foreign key constraints --- an application might be able to see a
dependent row that apparently has no referenced row, or might get an
update or delete failure for a row that is unreferenced as far as it can
see. Things get worse if an application can insert, update or delete
rows that it can't select. The only answer I've been able to get about
what SEPostgres will do about that is "we really don't care that we're
breaking SQL semantics". I don't find that to be a satisfactory answer.
The security-geek reason why not is that it represents a huge
information leak. The database-geek reason why not is that this will
permanently destroy our ability to do a lot of important optimizations,
eg join removal on the basis of foreign key constraints. (There are
probably other good reasons, but that one will do for starters.)
Perhaps this is fixable by constraining what a security policy is
allowed to do, or in some other way that I don't know about, but I've
seen no serious discussion about that.

2. I don't understand where to draw the dividing line between database
system accesses (which can't be security constrained, at least not
without breaking things entirely --- eg it will do you little good to
imagine that you can hide rows in pg_security from the
security-enforcement code ;-)) and user accesses that should be
security-constrained. I am certain that the line is muddied beyond
usability in the current system; there are a lot of user-exposed
functions that are making use of the same infrastructure that core
system accesses do. As an example, some of the people who think they
want this feature would like to use it to hide the bodies of their
user-defined functions from other people whom they don't wish to see
their code. But pg_get_functiondef() uses the same catcache
infrastructure as the code that would be called on to actually execute
their function, so there is no reasonable place to prevent the function
body from being exposed through that inquiry function or others of its
ilk. This problem gets rapidly worse when you consider that Postgres is
designed to be a very extensible system. It's not clear how to classify
add-on code, and it is clear that any of it could unintentionally
introduce a security hole. The only solution I can see is "we stop
guaranteeing that SEPostgres is good for anything the moment you load
even one extension module", and that isn't a very satisfactory answer
either. Even accepting such a restriction, there's too much code in
core Postgres to let anyone feel very good about keeping the core free
of security leaks.

There are some other problems, like the rather frightening thought that
a database dump might silently omit critical data (remember pg_dump is
just another client). But I think the two categories above cover the
issues that are making me seriously leery of this patch.

regards, tom lane

OK, time for me to chime in.

I think the outstanding commit-fest items can be broken down into four
sections:

o Log streaming
o Hot standby
o SE-PostgreSQL
o Others

I think we all agree that log streaming is not ready for 8.4, and that
delaying for this feature would lead to an indeterminate delay. I have
already talked to the NTT folks, and as painful as it is, I think they
have accepted this --- many thanks to them.

Hot standby seems to be having a lot of code churn. I am not sure if it
is because the patch is being polished for completion or if lots of new
problems are being discovered and fixed. Tom and I have both given up
trying to track this patch. Fortunately, Heikki has been following it
closely, so I think Heikki is going to make the final decision if hot
standby is ready for 8.4 (because he is going to have to stand behind it
as a committer).

SE-PostgreSQL has been in steady development for a year so this is the
time to decide about it. My feeling is if we don't accept it now, we
are never going to have SE-Linux or row-level security. The next week
should show us the right direction when we start discussion on
Wednesday, noon GMT.

The other patches are 1-2 weeks work and will be dealt with like patches
from previous commit-fests.

FYI, I think delaying a major release to get feature X always going to
be counter-productive because it requires restarting development with no
known completion time, and they typical behavior is "just 2 more weeks",
"just two more weeks", over and over again, so no one knows how much
time they have to develop stuff and things stall badly.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

All,

FWIW, I'll comment that what we're seeing here is nothing new. We have:

--One invasive patch which everybody (myself included) procrastinated on
reviewing even though we got it early, and:

--One "must have" big complex patch which arrived very late in the
development cycle.

These are our two reasons for every release delay I can think of. In
prior releases, HOT, PITR, Windows, etc., all presented the same issues.

--Josh Berkus

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> FWIW, I'll comment that what we're seeing here is nothing new.

Certainly the Hot Standby situation is the same old song, different verse.

(I'm personally of the opinion that the project has usually been better
served when we decided not to postpone a release to wait for a specific
feature.)

SEPostgres seems qualitatively different to me, though. I think PG
people have avoided reviewing it because (a) they weren't interested in
it and (b) they knew they were unqualified to review it.

Meanwhile it's emerging that the selinux people don't feel qualified to
review it either. I'm not quite sure what to do about that. But "throw
it in there on faith" doesn't sound like an appealing answer, and I've
got no idea how long it will take to work out a non-faith-based answer.

regards, tom lane

On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> > FWIW, I'll comment that what we're seeing here is nothing new.

> Meanwhile it's emerging that the selinux people don't feel qualified to
> review it either. I'm not quite sure what to do about that. But "throw
> it in there on faith" doesn't sound like an appealing answer, and I've
> got no idea how long it will take to work out a non-faith-based answer.
>

O.k. maybe it is time to consider something non-traditional. What about
two 8.4 releases?

8.4-stable
8.4-experimental

stable is everything that stable is. PostgreSQL at its best.

experimental is the day after we branch. Same catalog version but
contains say SEPostgres and Hot Standby etc...

8.4-experimental gives people a stable same version compatible version
to test at their leisure. We support it until 8.5 comes out. At 8.5,
those features are merged (or ripped out) into 8.5-stable.

Yes I know this is non-standard for our community but maybe it is time
to crank it up a notch?

Sincerely,

Joshua D. Drake

> regards, tom lane
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

> SEPostgres seems qualitatively different to me, though. I think PG
> people have avoided reviewing it because (a) they weren't interested in
> it and (b) they knew they were unqualified to review it.

I think that you are off-base here. As I've pointed out previously,
nobody was ever ASSIGNED to review SE-PostgreSQL.

http://archives.postgresql.org/message-id/603c8f070901111132m2a595ec0g677762c1fff587c0@mail.gmail.com

The reviewing that happened during this CommitFest did not happen on
the basis of who was interested in which patches. There was a bit of
that, but for the most part people reviewed the patches that they were
asked to review. I assumed (am I the only one?) that the REASON why
we were not asked to review SE-PostgreSQL or Hot Standby is because
the committers were planning to do that themselves due to the
complexity of the patches.

Now, apparently, that assumption was totally wrong. But this doesn't
seem complicated to me. If we bump SE-PostgreSQL to the next
CommitFest and assign reviewers, they will review it. Maybe their
review will not be 100% perfect, but that is why we have committers.
If we continue to NOT assign reviewers, reviewers are unlikely to
crawl out of the woodwork, just as they (mostly) didn't crawl out of
the woodwork for any other patches (HS/SR being a notable exception).

...Robert

Robert,

> The reviewing that happened during this CommitFest did not happen on
> the basis of who was interested in which patches. There was a bit of
> that, but for the most part people reviewed the patches that they were
> asked to review. I assumed (am I the only one?) that the REASON why
> we were not asked to review SE-PostgreSQL or Hot Standby is because
> the committers were planning to do that themselves due to the
> complexity of the patches.

Actually, I did assign someone to do a build and specification review.
But yes, I expected that the code review would *have* to be done by a
long-term committer. I pretty much assume that of anything over 300 lines.

The idea behind having new reviewers take on all the small patches, was,
of course, to give the main committers more time with patches like
SEPostgres. It worked with other stuff (like Windowing and CTE).

--Josh

>> The reviewing that happened during this CommitFest did not happen on
>> the basis of who was interested in which patches. There was a bit of
>> that, but for the most part people reviewed the patches that they were
>> asked to review. I assumed (am I the only one?) that the REASON why
>> we were not asked to review SE-PostgreSQL or Hot Standby is because
>> the committers were planning to do that themselves due to the
>> complexity of the patches.
>
> Actually, I did assign someone to do a build and specification review. But
> yes, I expected that the code review would *have* to be done by a long-term
> committer. I pretty much assume that of anything over 300 lines.
>
> The idea behind having new reviewers take on all the small patches, was, of
> course, to give the main committers more time with patches like SEPostgres.
> It worked with other stuff (like Windowing and CTE).

Right, so, then I'm not sure why Tom is taking the lack of review as a
sign of lack of interest.

...Robert

2009/1/27 Joshua D. Drake <jd(at)commandprompt(dot)com>:
> On Mon, 2009-01-26 at 23:39 -0500, Tom Lane wrote:
>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> > FWIW, I'll comment that what we're seeing here is nothing new.
>
>> Meanwhile it's emerging that the selinux people don't feel qualified to
>> review it either. I'm not quite sure what to do about that. But "throw
>> it in there on faith" doesn't sound like an appealing answer, and I've
>> got no idea how long it will take to work out a non-faith-based answer.
>>
>
> O.k. maybe it is time to consider something non-traditional. What about
> two 8.4 releases?
>
> 8.4-stable
> 8.4-experimental
>
> stable is everything that stable is. PostgreSQL at its best.
>

I dislike this idea - it's same like short processed 8.5 - that is
more simple. Actually current base 8.4 has lot of features, enough for
people, so it could be released. 8.5 should be implemented in shorted
cycle - only one commitfest, that is enough (+3 month) for well
completing SE and replication patches. In czech, big users started
testing and using 8.3, and propably skip 8.4. For small projects
should be 8.4 available early.

regards
Pavel Stehule

> experimental is the day after we branch. Same catalog version but
> contains say SEPostgres and Hot Standby etc...
>
> 8.4-experimental gives people a stable same version compatible version
> to test at their leisure. We support it until 8.5 comes out. At 8.5,
> those features are merged (or ripped out) into 8.5-stable.
>
> Yes I know this is non-standard for our community but maybe it is time
> to crank it up a notch?
>
> Sincerely,
>
> Joshua D. Drake
>
>
>
>> regards, tom lane
>>
> --
> PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
> Consulting, Development, Support, Training
> 503-667-4564 - http://www.commandprompt.com/
> The PostgreSQL Company, serving since 1997
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

On Mon, Jan 26, 2009 at 6:07 PM, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
>> I realize in the current system (emailed patches), this would be a horrible
>> pain to maintain such a branch; but perhaps some of the burden could be
>> pushed down to the patch submitters (asking them to merge their own changes
>> into this merged branch).
>
> I've considered maintaining such a repository a few times and dismissed it
> when I realized how much work it would be to maintain.

I don't think it would be too bad if everyone were using git. And in
fact, if people weren't using git, but we had some kind of a system
that could (1) return a list of all of the current patches and (2)
return for any given patch the message-ids of the messages wherein the
various versions of the patch were submitted, it would be harder, but
possibly managable. You might have trouble with really big patches
creating lots of merge conflicts, but even if you just merged all the
smaller patches into one tree and push it out for people to test
against, that might still have some real value if a decent number of
people did testing against that tree.

I think that it would probably be pretty easy to write a webapp to
replace the CommitFest web page that basically did the same thing but
with a bit more structure around it - with database tables like
"commitfest", "patch", "patch_version", "patch_comment", and
"patch_review". I think I might even be willing to write such a
webapp if someone would be willing to provide the infrastructure. The
CommitFest web page was really useful this time around, but it's not
conducive to any kind of automated pull.

...Robert

On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
> so it could be released. 8.5 should be implemented in shorted
> cycle - only one commitfest, that is enough (+3 month) for well
> completing SE and replication patches.

we tried this before (8.2 to 8.3 i think), the idea was that the next
release should be in 6 months... we release at least 6 months later...

ATM that a new release cycle starts new patch will arrive and there
will be no way to get the shorted release in time...

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

2009/1/27 Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>:
> On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>> so it could be released. 8.5 should be implemented in shorted
>> cycle - only one commitfest, that is enough (+3 month) for well
>> completing SE and replication patches.
>
> we tried this before (8.2 to 8.3 i think), the idea was that the next
> release should be in 6 months... we release at least 6 months later...
>
> ATM that a new release cycle starts new patch will arrive and there
> will be no way to get the shorted release in time...
>

I remember it. Solution is - don't accept new patches for next commitfest.

Pavel

> --
> Atentamente,
> Jaime Casanova
> Soporte y capacitación de PostgreSQL
> Asesoría y desarrollo de sistemas
> Guayaquil - Ecuador
> Cel. +59387171157
>

Pavel Stehule wrote:
> 2009/1/27 Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>:
>> On Tue, Jan 27, 2009 at 12:40 AM, Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com> wrote:
>>> so it could be released. 8.5 should be implemented in shorted
>>> cycle - only one commitfest, that is enough (+3 month) for well
>>> completing SE and replication patches.
>> we tried this before (8.2 to 8.3 i think), the idea was that the next
>> release should be in 6 months... we release at least 6 months later...
>>
>> ATM that a new release cycle starts new patch will arrive and there
>> will be no way to get the shorted release in time...
>
> I remember it. Solution is - don't accept new patches for next commitfest.

I don't think that'll work. People will still keep writing patches. Or
if they don't, that's even worse! Either people will work on patches
that they're interested in, or they'll go away and do something else.
Only very few will drop their pet projects for the common good and help
with the review instead.

We can adjust the length of the release cycle by adjusting the number of
commitfests. I think the normal ~ 1 year cycle is quite optimal, though.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Sorry for long description.

Tom Lane wrote:
> Joshua Brindle <method(at)manicmethod(dot)com> writes:
>> http://marc.info/?l=selinux&m=115762285013528&w=2
>> Is the original discussion thread for the security model used in the
>> sepostgresql work. Hopefully you'll see some of the evidence you speak of there.
>
> Thanks for the link. I took a look through that thread and saw a lot of
> discussion about issues like how to relate the database-side and
> client-side permissions, which is all good stuff but mostly outside my
> purview as a database geek. I didn't find anything about the stuff that
> is really bothering me, which I think can be broken down into two main
> categories:
>
> 1. Silently filtering out rows according to an arbitrary security policy
> can break a bunch of fundamental SQL semantics, the most obvious being
> foreign key constraints --- an application might be able to see a
> dependent row that apparently has no referenced row, or might get an
> update or delete failure for a row that is unreferenced as far as it can
> see. Things get worse if an application can insert, update or delete
> rows that it can't select. The only answer I've been able to get about
> what SEPostgres will do about that is "we really don't care that we're
> breaking SQL semantics". I don't find that to be a satisfactory answer.

As I repeated it several times, it is a "covert channel" issue.
An important thing is what the limilation is well documented and
allows end-users to choose the option.
As I summarized in another message, it is not a must-requirement
for security evaluation criteria (ISO/IEC15408, CC).
The criteria allows to include a feature to eliminate covert channels,
but it also allows not to include ones to eliminate them.
http://wiki.postgresql.org/wiki/SEPostgreSQL#Covert_channels

For example, Oracle Label Security which is certified as EAL4+ class
and also provides row-level mandatory access controls, but it does not
care about covert channels. This fact is documented as "Security Target",
so end-users can know that elimination of covert channel is over spec
for the product.
They can choose the product with their own responsibility.

At first, we should understand is individual security features have
its own targets, coverages and so on.
I assumes the target of SE-PostgreSQL is enterprise class users
who are interested in web application security like a cloud-computing
platform, similar to Oracle's one.

> The security-geek reason why not is that it represents a huge
> information leak. The database-geek reason why not is that this will
> permanently destroy our ability to do a lot of important optimizations,
> eg join removal on the basis of foreign key constraints. (There are
> probably other good reasons, but that one will do for starters.)
> Perhaps this is fixable by constraining what a security policy is
> allowed to do, or in some other way that I don't know about, but I've
> seen no serious discussion about that.

IIRC, we had discussions refering to ISO/IEC15408 a few times at the
past. I believe it was serious discussions.
Polyinstantiation technology might help the situation, but we decided
not to port it on PostgreSQL due to widespread unacceptable changes.

I make clear it again.
It is an over spec for SE-PostgreSQL to eliminate all the covert channels,
however, we documented the limitation for end-users, and they can choose
SE-PostgreSQL with their own responsibility.

> 2. I don't understand where to draw the dividing line between database
> system accesses (which can't be security constrained, at least not
> without breaking things entirely --- eg it will do you little good to
> imagine that you can hide rows in pg_security from the
> security-enforcement code ;-)) and user accesses that should be
> security-constrained.

Please consider the symmetrical architecture between OS and DBMS.
A process accesses resources managed by OS via system calls, and
SELinux acquire the system calls and applies its decision making.
In other hand, a client accesses database object managed by DBMS
via SQL queries, and SE-PostgreSQL applies its decision come from
security policy of SELinux.

Please note that SELinux cannot apply its security policy on
accesses come from in-kernel code in principle, although it
enables to control kernel-loadable-modules.
In same manner, SE-PostgreSQL cannot apply its access controls
on internal database system accesses.

It come from characteristics of a reference monitor which applies
its security policy for all the required accesses, but internal
steps are exceptions.

For example, if SELinux allows a malicious one to load a kernel
loadable module which hijacks filesystems, any other users have
a possibility to invoke the malicious code indirectly which
can bypass SELinux's checks.

> I am certain that the line is muddied beyond
> usability in the current system; there are a lot of user-exposed
> functions that are making use of the same infrastructure that core
> system accesses do. As an example, some of the people who think they
> want this feature would like to use it to hide the bodies of their
> user-defined functions from other people whom they don't wish to see
> their code. But pg_get_functiondef() uses the same catcache
> infrastructure as the code that would be called on to actually execute
> their function, so there is no reasonable place to prevent the function
> body from being exposed through that inquiry function or others of its
> ilk. This problem gets rapidly worse when you consider that Postgres is
> designed to be a very extensible system. It's not clear how to classify
> add-on code, and it is clear that any of it could unintentionally
> introduce a security hole.

In this case, security administrator should not allow unprivileged
users to invoke pg_get_functiondef(). User invokes pg_get_functiondef()
via SQL queries, but pg_get_functiondef() refers system cache via
its internal process.

For add-on module, SE-PostgreSQL checks whether the client has
db_database:{install_module} privilege on the target database
and installed module (on the filesystem).

I would like to make clear I have never said it is possible to
prevent malicious accesses from system internal entities.

As you mentioned before, it is not fundamentally possible.
Do you remember a previous discussion you suggested to remove
the pgaceAllowOverridePlannerHook() hook and I agreed soon?

> The only solution I can see is "we stop
> guaranteeing that SEPostgres is good for anything the moment you load
> even one extension module", and that isn't a very satisfactory answer
> either. Even accepting such a restriction, there's too much code in
> core Postgres to let anyone feel very good about keeping the core free
> of security leaks.

Forgive me, it seems to me you concern about SE-PostgreSQL is not
a magic-ballet of security. However, we never guarantee it enables
to protect malicious anything. Any security features have its merits
and limitation, needless to say. SELinux is not also perfect.
(At least, who can prove it is perfect?)

Information-Security has a long history more than 2,000 years since
the period of Caesar-cipher, but mankind has not see perfect security
yet. Thus, we need to improve it step by step.

I would like you to understand not negligible number of users are
waiting for SE-PostgreSQL feature, even if these limitations.

> There are some other problems, like the rather frightening thought that
> a database dump might silently omit critical data (remember pg_dump is
> just another client). But I think the two categories above cover the
> issues that are making me seriously leery of this patch.

As I documented, a client runs pg_dump/pg_restore should have enough
privileges on whole of the databases.

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

Bruce Momjian wrote:
> OK, time for me to chime in.
>
> I think the outstanding commit-fest items can be broken down into four
> sections:
>
> o Log streaming
> o Hot standby
> o SE-PostgreSQL
> o Others

- snip -

> SE-PostgreSQL has been in steady development for a year so this is the
> time to decide about it. My feeling is if we don't accept it now, we
> are never going to have SE-Linux or row-level security. The next week
> should show us the right direction when we start discussion on
> Wednesday, noon GMT.

Hmm...?
This conditional punishment of death seems to me not a reasonable one.
If we can found a matter as a result of discussion, which is impossible
to fix within reasonable term, I'll agree it being postponed to v8.5.
However, why is the punishment of death necessary here?

Thanks,
--
OSS Platform Development Division, NEC
KaiGai Kohei <kaigai(at)ak(dot)jp(dot)nec(dot)com>

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> The idea behind having new reviewers take on all the small patches, was,
> of course, to give the main committers more time with patches like
> SEPostgres. It worked with other stuff (like Windowing and CTE).

Huh? There were certainly non-committer reviewers for the window
functions and CTE patches.

But the larger point here is that SEPostgres has been around for nearly
two years, and has been submitted into every 8.4 commit fest not just
this last one, and has drawn no noticeable amount of enthusiasm from the
pghacker community. It's not just a matter of "we haven't gotten to it
yet"; AFAICS no one has wanted to get to it.

regards, tom lane

On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> So, some feedback to make this decision more difficult:
>
>> Users: care about HS more than anything else in the world.
>
> I don't think this is correct. There are certainly a lot of users who
> would like an in-core replication solution, but HS by itself is not that
> --- you also need (near) real-time log shipping, which we have already
> decided to punt to 8.5. That being the case, I think the argument
> that HS is a must-have feature for 8.4 is actually rather weak.

I don't buy that. Sure, sync-rep would be the icing on the cake, but
HS with a small archive_timeout (even of the order of 10 or 15
minutes) would have been extremely useful on a number of systems I
used to run.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:

> Then why has *nobody* stepped up to review the design, much less the
> whole patch? The plain truth is that no one appears to care enough to
> expend any real effort.

I've spent some time looking at it and have made all the comments I
wished to make. The design seems clear and fit for purpose, having read
KaiGai's excellent Wiki description of how it all fits together and also
read some PDF links Bruce sent out.

But I've not had time to look at the whole patch and my contacts have
not had sufficient time to do anything meaningful with it either.

If we can minimise the impact on normal running and it doesn't have any
implications for robustness, it should be OK. Surely we should give it a
quick review to see if it has any gotchas. If not, and KaiGai is willing
to commit to supporting it, then should be good to go. KaiGai isn't a
home hacker, he's a lead developer for a major multinational, so we
should be able to take his word if he says he will continue to
contribute fixes if problems are found. If we don't commit to him and
his company then they won't commit to us either.

The process works like this: software gets developed, then it gets
certified. If its not certified, then Undercover Elephant will not be
used by the secret people. We can't answer the "will it be certified?"
question objectively yet. If we have someone willing to write the
software and put it forward for certification then we should trust that
it probably will pass certification and if it doesn't we will see
further patches to allow that to happen.

--
Simon Riggs www.2ndQuadrant.com
PostgreSQL Training, Services and Support

Dave Page wrote:
> On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>>
>>> So, some feedback to make this decision more difficult:
>>>
>>> Users: care about HS more than anything else in the world.
>>>
>> I don't think this is correct. There are certainly a lot of users who
>> would like an in-core replication solution, but HS by itself is not that
>> --- you also need (near) real-time log shipping, which we have already
>> decided to punt to 8.5. That being the case, I think the argument
>> that HS is a must-have feature for 8.4 is actually rather weak.
>>
>
> I don't buy that. Sure, sync-rep would be the icing on the cake, but
> HS with a small archive_timeout (even of the order of 10 or 15
> minutes) would have been extremely useful on a number of systems I
> used to run.
>
>
>
+1

I have customers who want exactly this - a simple to administer,
query-able slave that does DDL transparently and is up to date within a
controllable time frame. Bluntly, it looks like a killer feature.

regards

Mark

On Jan 27, 2009, at 2:41 AM, Mark Kirkwood wrote:

> Dave Page wrote:
>> On Mon, Jan 26, 2009 at 8:12 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>
>>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>>>
>>>> So, some feedback to make this decision more difficult:
>>>> Users: care about HS more than anything else in the world.
>>>>
>>> I don't think this is correct. There are certainly a lot of users
>>> who
>>> would like an in-core replication solution, but HS by itself is
>>> not that
>>> --- you also need (near) real-time log shipping, which we have
>>> already
>>> decided to punt to 8.5. That being the case, I think the argument
>>> that HS is a must-have feature for 8.4 is actually rather weak.
>>>
>>
>> I don't buy that. Sure, sync-rep would be the icing on the cake, but
>> HS with a small archive_timeout (even of the order of 10 or 15
>> minutes) would have been extremely useful on a number of systems I
>> used to run.
>>
>>
>>
> +1
>
> I have customers who want exactly this - a simple to administer,
> query-able slave that does DDL transparently and is up to date
> within a controllable time frame. Bluntly, it looks like a killer
> feature.
>
> regards

+1

So, I am just a lurker here. I mostly follow hackers to find out if
any new features are coming out that will make it worth upgrading, and
to keep up on any backwards compatibly changes that I should be aware
of. I am on 8.1 and it performs well and no features added since then
have seemed worth downing the whole system to do the upgrade for.
However, a simple to administer, query-able slave that does DDL
transparently and is up to date within a controllable time frame is
something that would undoubtably make it worth the upgrade. Whatever
version this feature makes it into will probably be the one I will
upgrade to.

Of course this is just one developer giving you anecdotal evidence and
there are obviously many concerns other than just how in demand it is,
but I just wanted to register my vote that this is a very sought after
feature and it is hard for me to imagine a situation (especially for a
24x7 web application) where having an easy to admin hot standby server
wouldn't help your local DBA sleep better at night.

Rick

* Tom Lane (tgl(at)sss(dot)pgh(dot)pa(dot)us) wrote:
> SEPostgres seems qualitatively different to me, though. I think PG
> people have avoided reviewing it because (a) they weren't interested in
> it and (b) they knew they were unqualified to review it.
>
> Meanwhile it's emerging that the selinux people don't feel qualified to
> review it either. I'm not quite sure what to do about that. But "throw
> it in there on faith" doesn't sound like an appealing answer, and I've
> got no idea how long it will take to work out a non-faith-based answer.

Erm, I have to say here that this strikes me as rather unfair. Perhaps
I'm wrong, but I suspect KaiGai feels pretty good about the patch and
his qualifications in both the PG realm and the SELinux realm. He's
asking the PG folks to review it because that's the process that the PG
community (through the CommitFest, etc) has laid out for getting a patch
included upstream. I'm confident KaiGai isn't going to just disappear
into the ether if the patch is committed.

Sure, it'd be nice if 4 or 5 other SELinux developers got in and
understood the PG code well enough to implement such a patch, but I
think the combination of KaiGai (overall), a seperate SELinux hacker
(for the security design and SELinux side of it), and a PG committer
(for where the hooks are placed and how), reviewing the patch and being
comfortable with it is quite sufficient for a high quality result.

Thanks,

Stephen

Simon Riggs wrote:
> On Mon, 2009-01-26 at 19:21 -0500, Tom Lane wrote:
>
>> Then why has *nobody* stepped up to review the design, much less the
>> whole patch? The plain truth is that no one appears to care enough to
>> expend any real effort.
>
> I've spent some time looking at it and have made all the comments I
> wished to make. The design seems clear and fit for purpose, having read
> KaiGai's excellent Wiki description of how it all fits together and also
> read some PDF links Bruce sent out.

Thanks for your comment, although you also have a tough work.

> But I've not had time to look at the whole patch and my contacts have
> not had sufficient time to do anything meaningful with it either.
>
> If we can minimise the impact on normal running and it doesn't have any
> implications for robustness, it should be OK. Surely we should give it a
> quick review to see if it has any gotchas. If not, and KaiGai is willing
> to commit to supporting it, then should be good to go. KaiGai isn't a
> home hacker, he's a lead developer for a major multinational, so we
> should be able to take his word if he says he will continue to
> contribute fixes if problems are found. If we don't commit to him and
> his company then they won't commit to us either.

Needless to say, I will continue to support the feature.
I cannot understand why is it necessary to disappear from here.

At least, a binary with "--enable-selinux" passes all regression
test with/without "pgace_feature=selinux".
The benchmark results I have is a bit legacy, so it is necessary
to record it again, but I don't think it gives significant
implications on normal running (pgace_feature=none).
(Yes, it indeed gives us performance loss with selinux-enabled,
but we assume performance is not the first priority in this case.)

> The process works like this: software gets developed, then it gets
> certified. If its not certified, then Undercover Elephant will not be
> used by the secret people. We can't answer the "will it be certified?"
> question objectively yet. If we have someone willing to write the
> software and put it forward for certification then we should trust that
> it probably will pass certification and if it doesn't we will see
> further patches to allow that to happen.

--
KaiGai Kohei <kaigai(at)kaigai(dot)gr(dot)jp>

On Tuesday 27 January 2009 00:42:32 Ron Mayer wrote:
> If it were just as easy for us to pull from a
> "all 'pending-patches' for-commit-fest-nov that pass regression tests"
> branch, I'd happily pull from that instead.

Considering that most patches don't come with regression tests, this would
accomplish very little. And even those patches that did come with regression
tests (e.g., updatable views) need a design analysis much more than running
an automated test suite. Ultimately, it does come down to human work.

Robert Haas escribió:

> I think that it would probably be pretty easy to write a webapp to
> replace the CommitFest web page that basically did the same thing but
> with a bit more structure around it - with database tables like
> "commitfest", "patch", "patch_version", "patch_comment", and
> "patch_review". I think I might even be willing to write such a
> webapp if someone would be willing to provide the infrastructure. The
> CommitFest web page was really useful this time around, but it's not
> conducive to any kind of automated pull.

Hey, if you're willing to do it, we're certainly accepting proposals.
The current wiki-based CommitFest is supposed to be just a stop-gap. It
was started not only to support 8.4 development, but also as a test of
the Commitfest idea itself. This has proven so successful that it's
clear we should be going somewhere with it.

As for somewhere to host it, we certainly have some servers; not tons,
but probably enough. Some of them even have Postgres running on it.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:

> As for somewhere to host it, we certainly have some servers; not tons,
> but probably enough. Some of them even have Postgres running on it.

We can certainly host an app under postgresql.org. The bigger issue
will be speccing it to meet the requirements of the community without
getting bogged down in bike shedding.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Dave Page wrote:
> On Tue, Jan 27, 2009 at 1:42 PM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
>
>> As for somewhere to host it, we certainly have some servers; not tons,
>> but probably enough. Some of them even have Postgres running on it.
>
> We can certainly host an app under postgresql.org. The bigger issue
> will be speccing it to meet the requirements of the community without
> getting bogged down in bike shedding.

I have started some very trivial work around this a while ago with the
intent to get something simple up and working before too much bike
shedding is done. I'll contact Robert off-list to discuss that. If
somebody else - who actively works with what we have now!! - is
interested in that discussion, let me know.

Will obviously take it on-list before any decisions are made. So far I'm
just talking about discussing a prototype.

//Magnus

On 1/27/09, Alvaro Herrera <alvherre(at)commandprompt(dot)com> wrote:
> Robert Haas escribió:
> > I think that it would probably be pretty easy to write a webapp to
> > replace the CommitFest web page that basically did the same thing but
> > with a bit more structure around it - with database tables like
> > "commitfest", "patch", "patch_version", "patch_comment", and
> > "patch_review". I think I might even be willing to write such a
> > webapp if someone would be willing to provide the infrastructure. The
> > CommitFest web page was really useful this time around, but it's not
> > conducive to any kind of automated pull.
>
> Hey, if you're willing to do it, we're certainly accepting proposals.
> The current wiki-based CommitFest is supposed to be just a stop-gap. It
> was started not only to support 8.4 development, but also as a test of
> the Commitfest idea itself. This has proven so successful that it's
> clear we should be going somewhere with it.
>
> As for somewhere to host it, we certainly have some servers; not tons,
> but probably enough. Some of them even have Postgres running on it.

Such app already exists:

http://ozlabs.org/~jk/projects/patchwork/

So it's a matter of just setting it up.

--
marko

On Sunday 25 January 2009 19:06:50 Tom Lane wrote:
> Particularly with regard to hot standby, which by any sane reading was
> not close to being committable on 1 November (a fortiori from the fact
> that it's *still* not committable despite large amounts of later work).
> I'm also feeling that we are not in a position to commit SE-Postgres in
> a timely fashion; which is not KaiGai-san's fault, rather that of the
> community which has taken nearly zero interest in that patch.
>
> If we want to ensure that 8.5 development opens soon, what we have to do
> is reject those two patches, revert updatable views, and finish up the
> other stuff (which is all small and could likely be dealt with in a week
> or two).

Updatable views is reverted. I agree that we should reject the rest and
prepare a release.

Bruce Momjian píše v po 26. 01. 2009 v 23:02 -0500:
> OK, time for me to chime in.
>
> I think the outstanding commit-fest items can be broken down into four
> sections:
>
> o Log streaming
> o Hot standby
> o SE-PostgreSQL
> o Others

You omit pg_upgrade. Does it mean that this project is already killed
for 8.4?

Thanks Zdenek

On Tuesday 27 January 2009 02:21:41 Tom Lane wrote:
> Then why has *nobody* stepped up to review the design, much less the
> whole patch?  The plain truth is that no one appears to care enough to
> expend any real effort.  But this patch is far too large and invasive
> to accept on the basis that only one guy understands it and will/might
> continue to maintain it.

As one of the earlier reviewers, I think the design is OK, but the way the
implementation is presented was not acceptable, and very little has been
accomplished in terms of reacting to our comments. For example, where is the
SQL row security feature, which should have been designed, implemented, and
committed separately, in the opinion of most commentaries.