Re: MySQL Compatibility WAS: 8.5 release timetable, again

I posted a note about a week ago which drew far less commentary than I
expected, regarding the timetable for the release of 8.5.

http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php

Perhaps this is already being discussed on -core, but if so the
conclusions haven't been shared publicly. We started the first
CommitFest for 8.5 on July 15, 2009. I believe that there is general
consensus on a two-month cycle for CommitFests, which means that the
second one will start on September 15, 2009, and the third one on
November 15, 2009. The open question is whether there will be a
fourth or a fifth CommitFest, and what that will do to the time line
for the final release.

During the 8.4 development cycle, we started the last CommitFest on
November 1, 2008 and released 8 months later on July 1, 2009. I would
like to think that we could shave a bit of time off of that this time
around, because last time the final CommitFest took approximately 4
months. I found that to be pretty ridiculous and I think that I was
not the only one. By the end, there were not too many patches left
and most of those had been thoroughly reviewed, so there was no
obvious role for non-committers to play. There were also many patches
that went for weeks, sometimes over a month, without being updated or
bounced. I think we can improve this a lot this time around, by
applying the same rules to the last CommitFest that we did to the
first one: patches must be updated in a timely fashion. If they
can't, it's a sign that either the patch author is too busy to work on
the patch, or the revisions needed are too extensive for a single
CommitFest. Either way, we move on.

I am a lot less clear on what we can do to shorten the time we spent
in beta. In some ways, the time we spent in beta for 8.4 seems not to
have been long enough. Several critical bugs introduced by the
infrastructure-changes-for-recovery patch were not fixed until just
before release, and a number of planner and other significant bugs
have been found since the release and will be fixed in 8.4.1. On the
other hand, from my point of view, there was nothing to do during
beta. Most of the open items required first and foremost a decision
about the best way to fix them, and those decisions really need to be
made by (or at least with the consent of) the committers. If we have
a substantial open items list again for 8.5, we need to find a way to
formalize the handling of that list so that it can be dealt with
efficiently and the work distributed among as many people as are
willing and able to help. Writing release notes also seemed to take a
lot of time, perhaps partly because we didn't know until the very end
whether certain large patches were going to be committed.

All of the above having been said, I think it's too much to hope that
the beta/release cycle is going to get drastically shorter than it was
for 8.4. So I think if we decide on three CommitFests, the timetable
will end up going something like this:

2009-11-15 Last CommitFest Begins
2009-12-15 Last CommitFest Ends
2010-01-01 Alpha
2010-03-01 Beta
2010-05-01 Release

This schedule assumes that instead of having 8 months between
start-of-last-CommitFest and release, we'll have five and a half.
Assuming that we can avoid the temptation to let the last CommitFest
drag on and on and on, that seems achievable.

If we go with four CommitFests, we'll probably end up release 45-60
days later, in mid June or start of July. If we go with five
CommitFests, we'll probably be looking at September.

Personally, I favor the more aggressive schedule of just three
CommitFests, because if something goes wrong and the schedule does
slip, we should still be able to get the release out the door next
year by the same time we did this year. If things go as planned,
we'll have the release out the door in time for PGCon. If things go
truly excellently and we manage to get the release out before 1-May,
then we have that much more time for 8.6 development. On the other
hand, if we plan for four Commitfests, we'll be looking at releasing
the same time next year that we did this year *if* everything goes as
planned. If anything slips, we'll release even later - and that also
gets into the summer time, when more people are away.

But, some other decision that has consensus is fine too. The most
important thing is to MAKE a decision before the passage of time makes
one for us.

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> I posted a note about a week ago which drew far less commentary than I
> expected, regarding the timetable for the release of 8.5.

> http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php

> Perhaps this is already being discussed on -core, but if so the
> conclusions haven't been shared publicly.

Core hasn't discussed 8.5 schedule since the discussions that Peter
summarized in the message you cited above. I share your concern that
"release in time for PGCon" isn't very realistic if we don't get more
aggressive about schedule. On the other hand, we didn't get all that
much done in this fest. If we cut back to a three-fest schedule
I think we may succeed in releasing 8.5 early, but only because there
is nothing interesting in it :-(. Dunno where the right balance is.

regards, tom lane

On Sun, Aug 23, 2009 at 1:57 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> I posted a note about a week ago which drew far less commentary than I
>> expected, regarding the timetable for the release of 8.5.
>
>> http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php
>
>> Perhaps this is already being discussed on -core, but if so the
>> conclusions haven't been shared publicly.
>
> Core hasn't discussed 8.5 schedule since the discussions that Peter
> summarized in the message you cited above.  I share your concern that
> "release in time for PGCon" isn't very realistic if we don't get more
> aggressive about schedule.  On the other hand, we didn't get all that
> much done in this fest.  If we cut back to a three-fest schedule
> I think we may succeed in releasing 8.5 early, but only because there
> is nothing interesting in it :-(.  Dunno where the right balance is.

Here is my thinking on that point. We have several major features
underway for the September CommitFest, including at least Hot Standby
(Simon Riggs), Streaming Replication (Fujii Masao), and Index-Only
Scans (Heikki). At the July CommitFest, none of these patches were in
a state where they could be seriously reviewed. Simon Riggs and Fujii
Masao have both committed to produce reviewable versions by 9/15, and
it looks like Heikki will hit that date too, if not beat it.

I am assuming that at least Hot Standby and Streaming Replication will
likely require two CommitFests to go from the point where they are
seriously reviewable to actual commit. So if they hit the 9/15 date,
they should make 8.5 even with just three CommitFests. If they don't
hit the 9/15 date, then a 3-CommitFest cycle will probably be too
short for them to make it in. But if we schedule a fourth CommitFest
in January in the hopes of seeing one of those patches committed, then
ISTM we're basically speculating that the patch authors will not hit
the 9/15 date but that they will hit an 11/15 date.

To me, that's an entirely arbitrary and unfounded speculation. On the
one hand, both patch authors have said they will hit 9/15, so why
should we second-guess them? On the other hand, neither of those
patches seems to have made a great deal of progress in the last 7
months, so it's unclear that tacking a few more months onto the
schedule will help anything. Furthermore, even if we're right that
four CommitFests is the right number to land one of those patches
(rather than 3 or 5 or 6 or 7), we're then talking about committing a
major feature in the very last CommitFest for this release. We tried
that for 8.4 and it wasn't a stunning success.

To some degree, what this boils down to is that you can have
time-based releases or feature-based releases, but not both. And the
problem with feature-based releases in a community environment is that
there is no guarantee that patches will be delivered when promised.
None of the people developing these major features work for the
community and we do not have the ability to control any of their
schedules. So saying that we're going to wait for them to be ready is
potentially saying that we're going to wait forever. No one is
proposing that, but we are sort of trying to read the tea leaves and
try to guess when they might be ready and tailor the schedule to it.

To me, it seems to make more sense to just decide we're going to ship
the release on a time line that works for the project overall. If we
get some new features in, good. If they slip a bit past the deadline,
well, at least that should hopefully mean that they get committed
right at the beginning of the 8.6 cycle. If they slip WAY past the
deadline, bummer, but at least we didn't hold up the release to no
benefit. I don't think there's anything wrong with having a release
that has many small improvements but no major new features, and I
think that is the worst that will happen.

...Robert

On sön, 2009-08-23 at 09:37 -0400, Robert Haas wrote:
> To some degree, what this boils down to is that you can have
> time-based releases or feature-based releases, but not both.

Sure. But some people are trying to introduce another subvariant: The
conference-circuit-based releases. ;-) It sounds attractive, but it
shouldn't trump all other concerns. Consider this instead: Nothing to
do during beta? Write your conference slides! ;-)

I suggest going with four commit fests. Three is too short. We already
started the first one early, which didn't give those involved in the
release any time to prepare some patches for it. So with three fests
you'd only give the major developers 8 weeks to code something for a
yearly release.

On Mon, Aug 24, 2009 at 7:39 AM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
> On sön, 2009-08-23 at 09:37 -0400, Robert Haas wrote:
>> To some degree, what this boils down to is that you can have
>> time-based releases or feature-based releases, but not both.
>
> Sure.  But some people are trying to introduce another subvariant: The
> conference-circuit-based releases. ;-)  It sounds attractive, but it
> shouldn't trump all other concerns.  Consider this instead:  Nothing to
> do during beta?  Write your conference slides! ;-)

Oh, gee, what am I speaking on? :-)

The main thing that I (can't speak for anyone else) like about getting
a release out in time for PGCon is that it happens at about the same
time every year, and I kind of like the idea of shooting for a yearly
cycle. I'd be just as happy to release every year on Christmas day,
but it's harder to get people to bundle a release around then.

> I suggest going with four commit fests.  Three is too short.  We already
> started the first one early, which didn't give those involved in the
> release any time to prepare some patches for it.  So with three fests
> you'd only give the major developers 8 weeks to code something for a
> yearly release.

Well, that's a good point, but 12 weeks out of a 14-month release
cycle is only marginally better than 8 weeks out of a 12-month release
cycle. If we really want to give people more time to write patches,
we need to figure out how to speed up the time from the end of the
last CommitFest until release.

One thing that I had thought of proposing is that we branch the tree
when we go to 8.5-beta and hold the first 8.6 CommitFest at that time
- or, failing that, that we hold a DontCommitFest at that time, where
we review all of the patches just as we would for a regular
CommitFest, but without the committing part. At least for 8.4, it
didn't seem like much was happening during beta, at least not much
that was discussed on -hackers or could be distributed across the
community.

I actually don't think that's the ideal solution, though. It's just
papering around the problem that the release takes too long and the
burden falls on too few people. But we haven't yet come up with any
good idea to address that problem.

At any rate, I'm OK with 4 CommitFests if that is the consensus, but
so far we only have 3 people weighing in on this thread, which is not
a consensus for anything.

...Robert

peter_e(at)gmx(dot)net (Peter Eisentraut) writes:
> I suggest going with four commit fests. Three is too short. We already
> started the first one early, which didn't give those involved in the
> release any time to prepare some patches for it. So with three fests
> you'd only give the major developers 8 weeks to code something for a
> yearly release.

Partial counter-argument...

A large portion of the patches in CommitFest #1 represented items that
had been deferred from 8.4. So...

a) Many of these patches came in with ~6 months of preparation time

b) People were always free to start work earlier than CommitFest #1

c) If something requires a *lot* of work, then it may be that it
gets deferred so that it comes in as part of CommitFest #1 for
8.6, with the very same characteristics as in a)...

I do agree that trying to force coordination with a specific conference
in Ottawa seems like a very peculiar sort of forced scheduling.
--
select 'cbbrowne' || '@' || 'ca.afilias.info';
Christopher Browne
"Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock
phasers on the Heffalump, Piglet, meet me in transporter room three"

Christopher Browne <cbbrowne(at)ca(dot)afilias(dot)info> writes:
> I do agree that trying to force coordination with a specific conference
> in Ottawa seems like a very peculiar sort of forced scheduling.

Well, PGCon is just a convenient concrete target. The real point here
is that we're trying to get the release cycle to end with a release in
the spring. We've grown tired of releasing in the fall and invariably
finding the date slipping into holiday season, so we'd like to try
releasing near the other equinox. (Neither solstice is good, since
you are up against either holidays or vacations taking away the time
of the people who need to be doing the work.)

regards, tom lane

On Mon, Aug 24, 2009 at 5:55 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Christopher Browne <cbbrowne(at)ca(dot)afilias(dot)info> writes:
>> I do agree that trying to force coordination with a specific conference
>> in Ottawa seems like a very peculiar sort of forced scheduling.
>
> Well, PGCon is just a convenient concrete target.  The real point here
> is that we're trying to get the release cycle to end with a release in
> the spring.

Yeah, conference-based releases is just a proxy for time-based
releases. It's nice to have something to be happy about at the
conference too. And it's a convenient time to start talking about the
next release when you're all face-to-face.

--
greg
http://mit.edu/~gsstark/resume.pdf

All,

> Yeah, conference-based releases is just a proxy for time-based
> releases. It's nice to have something to be happy about at the
> conference too. And it's a convenient time to start talking about the
> next release when you're all face-to-face.

On the one hand:

I'd say that we go for the 3-CF release. I think we need to prove that
we can do a time-based release once before a lot of people on this list
will believe in it.

If we do 4 CFs, we're in danger of still being in beta in late May ...
and once conference and vacation season start, things get a lot slower.
Mind you, it's possible that we can shorten the final CF and beta this
time, but I wouldn't want to count on it.

If we *can* shorten them, then 8.6 can have 4CFs. But we won't know
until after we've done it.

On the other hand:

I think if we do another release without Standby/replication, we'll
start to lose a lot of users. People are waiting on that, and a lot of
folks were expecting it in 8.4.

Therefore: I think, 3CFs, but we go all-out to get Standby/Replication
into 8.5 in the next month. So, every committer/major hacker on this
list should pitch in to get those features done.

So, is there someone here who could be helping with HS/SR and isn't?
Why not?

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

On Mon, Aug 24, 2009 at 1:48 PM, Josh Berkus<josh(at)agliodbs(dot)com> wrote:
> Therefore: I think, 3CFs, but we go all-out to get Standby/Replication
> into 8.5 in the next month.  So, every committer/major hacker on this
> list should pitch in to get those features done.
>
> So, is there someone here who could be helping with HS/SR and isn't?
> Why not?

Unfortunately, neither Simon nor Fujii Masao is publishing a git
repository of their work or regular updates to their patch set.
Therefore, no one else can contribute code, and in fact no one else
can even provide a review, because there is nothing to look at. I
don't think that there would be any problem getting these patches
reviewed/committed outside of the regular CommitFest cycle, but
there's simply no place to start.

I have volunteered to work on Hot Standby and have merged it up to CVS
HEAD several times, and cleaned out a bit of other cruft, but the
scope of the task appears to exceed the time I have available. It
seems to me that other people could clone my git repo (or that branch)
and submit patches against it (or ask me to pull from a branch that
they publish), but will Simon incorporate those patches (assuming that
they don't suck) into his next version, or ignore them? It's not
really clear.

As I've said before, I am presently of the opinion that Streaming
Replication has little chance of making it into 8.5. This opinion is
vulnerable to contrary evidence, like a new version of the patch
showing up that shows massive progress. But the patch was bounced
from CF 2009-07 for a whole series of architectural problems which
have to be addressed before we can even get to implementation details,
bugs, documentation, etc. Hot Standby is in better shape but amount
of code cleanup needed is substantial and there is also quite a bit of
'git diff master | grep XXX' that needs to be gone through.

...Robert

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> So, is there someone here who could be helping with HS/SR and isn't?
> Why not?

You mean, other than Simon's hands-off attitude?

regards, tom lane

Josh Berkus wrote:
> All,
>
> > Yeah, conference-based releases is just a proxy for time-based
> > releases. It's nice to have something to be happy about at the
> > conference too. And it's a convenient time to start talking about the
> > next release when you're all face-to-face.
>
> On the one hand:
>
> I'd say that we go for the 3-CF release. I think we need to prove that
> we can do a time-based release once before a lot of people on this list
> will believe in it.
>
> If we do 4 CFs, we're in danger of still being in beta in late May ...
> and once conference and vacation season start, things get a lot slower.
> Mind you, it's possible that we can shorten the final CF and beta this
> time, but I wouldn't want to count on it.
>
> If we *can* shorten them, then 8.6 can have 4CFs. But we won't know
> until after we've done it.
>
> On the other hand:
>
> I think if we do another release without Standby/replication, we'll
> start to lose a lot of users. People are waiting on that, and a lot of
> folks were expecting it in 8.4.

That is a slightly alarmist. Who are we going to lose these users to?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Robert Haas wrote:
> As I've said before, I am presently of the opinion that Streaming
> Replication has little chance of making it into 8.5. This opinion is
> vulnerable to contrary evidence, like a new version of the patch
> showing up that shows massive progress. But the patch was bounced
> from CF 2009-07 for a whole series of architectural problems which
> have to be addressed before we can even get to implementation details,
> bugs, documentation, etc. Hot Standby is in better shape but amount
> of code cleanup needed is substantial and there is also quite a bit of
> 'git diff master | grep XXX' that needs to be gone through.

I agree. I think it is unlikely we will have anything ready to commit
for Streaming Replication or Hot Standby for the next commit-fest in
mid-September, and if we go for a 3-CF (commit fest) release, that gives
us only one final CF to get those features accepted, again unlikely. We
are either going to need to go to a 4-CF release, change the way we are
developing these patches, or both to get either in 8.5.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, Aug 24, 2009 at 4:56 PM, Bruce Momjian<bruce(at)momjian(dot)us> wrote:
> Robert Haas wrote:
>> As I've said before, I am presently of the opinion that Streaming
>> Replication has little chance of making it into 8.5.  This opinion is
>> vulnerable to contrary evidence, like a new version of the patch
>> showing up that shows massive progress.  But the patch was bounced
>> from CF 2009-07 for a whole series of architectural problems which
>> have to be addressed before we can even get to implementation details,
>> bugs, documentation, etc.  Hot Standby is in better shape but amount
>> of code cleanup needed is substantial and there is also quite a bit of
>> 'git diff master | grep XXX' that needs to be gone through.
>
> I agree.  I think it is unlikely we will have anything ready to commit
> for Streaming Replication or Hot Standby for the next commit-fest in
> mid-September, and if we go for a 3-CF (commit fest) release, that gives
> us only one final CF to get those features accepted, again unlikely.  We
> are either going to need to go to a 4-CF release, change the way we are
> developing these patches, or both to get either in 8.5.

I don't think a 4-CF release is going to help. It's just going to be
2 more months before everything else that has been done gets released.
Call me a pessimist if you will, but zero times an arbitrary number
of CommitFests is still zero.

The only solution here is to get more people working on these patches.
I have volunteered to work on HS and would also be willing to work on
SR. Work can be reviewing or actual code. But I cannot work on a
patch I cannot see, and neither can anyone else.

...Robert

On Monday 24 August 2009 3:51:31 pm Bruce Momjian wrote:
> > folks were expecting it in 8.4.
>
> That is a slightly alarmist. Who are we going to lose these users to?
the insane asylum?

Robert Haas wrote:
> On Mon, Aug 24, 2009 at 4:56 PM, Bruce Momjian<bruce(at)momjian(dot)us> wrote:
> > Robert Haas wrote:
> >> As I've said before, I am presently of the opinion that Streaming
> >> Replication has little chance of making it into 8.5. ?This opinion is
> >> vulnerable to contrary evidence, like a new version of the patch
> >> showing up that shows massive progress. ?But the patch was bounced
> >> from CF 2009-07 for a whole series of architectural problems which
> >> have to be addressed before we can even get to implementation details,
> >> bugs, documentation, etc. ?Hot Standby is in better shape but amount
> >> of code cleanup needed is substantial and there is also quite a bit of
> >> 'git diff master | grep XXX' that needs to be gone through.
> >
> > I agree. ?I think it is unlikely we will have anything ready to commit
> > for Streaming Replication or Hot Standby for the next commit-fest in
> > mid-September, and if we go for a 3-CF (commit fest) release, that gives
> > us only one final CF to get those features accepted, again unlikely. ?We
> > are either going to need to go to a 4-CF release, change the way we are
> > developing these patches, or both to get either in 8.5.
>
> I don't think a 4-CF release is going to help. It's just going to be
> 2 more months before everything else that has been done gets released.
> Call me a pessimist if you will, but zero times an arbitrary number
> of CommitFests is still zero.
>
> The only solution here is to get more people working on these patches.
> I have volunteered to work on HS and would also be willing to work on
> SR. Work can be reviewing or actual code. But I cannot work on a
> patch I cannot see, and neither can anyone else.

Agreed, so we fall back to "change the way we are developing these
patches". I am hesistant to jump into managing these patches until they
get their shot by their original authors in the September CF, but
managing them starting in mid-October will probably be too late for them
to get into 8.5.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, Aug 24, 2009 at 04:51:31PM -0400, Bruce Momjian wrote:
> Josh Berkus wrote:
> > All,
> >
> > > Yeah, conference-based releases is just a proxy for time-based
> > > releases. It's nice to have something to be happy about at the
> > > conference too. And it's a convenient time to start talking
> > > about the next release when you're all face-to-face.
> >
> > On the one hand:
> >
> > I'd say that we go for the 3-CF release. I think we need to prove
> > that we can do a time-based release once before a lot of people on
> > this list will believe in it.
> >
> > If we do 4 CFs, we're in danger of still being in beta in late May
> > ... and once conference and vacation season start, things get a
> > lot slower. Mind you, it's possible that we can shorten the final
> > CF and beta this time, but I wouldn't want to count on it.
> >
> > If we *can* shorten them, then 8.6 can have 4CFs. But we won't
> > know until after we've done it.
> >
> > On the other hand:
> >
> > I think if we do another release without Standby/replication,
> > we'll start to lose a lot of users. People are waiting on that,
> > and a lot of folks were expecting it in 8.4.
>
> That is a slightly alarmist. Who are we going to lose these users
> to?

Sadly, to one of the MySQL forks. This is one of those cases (cf. the
current thread on -advocacy) where pointy-hair-friendliness can really
help or hurt us.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

>> I think if we do another release without Standby/replication, we'll
>> start to lose a lot of users. People are waiting on that, and a lot of
>> folks were expecting it in 8.4.
>
> That is a slightly alarmist. Who are we going to lose these users to?

Drizzle. MySQL forks. CouchDB. Any database which has replication
which you don't need a professional DBA to understand. Whether or not
it works.

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> That is a slightly alarmist. Who are we going to lose these users to?

> Drizzle. MySQL forks. CouchDB. Any database which has replication
> which you don't need a professional DBA to understand. Whether or not
> it works.

You haven't explained why we'd lose such folk next year when we haven't
lost them already. MySQL has had replication (or at least has checked
off the bullet point ;-)) for years. I'd be seriously surprised if any
of the forks will offer significantly better replication than is there
now, so the competitive situation is not changing in that regard.

It is true that we're missing a chance to pull some folks away while
the situation on that side of the fence is so messy. But I don't see
our situation getting worse because of that, just not getting better.

regards, tom lane

On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> >> That is a slightly alarmist. Who are we going to lose these
> >> users to?
>
> > Drizzle. MySQL forks. CouchDB. Any database which has
> > replication which you don't need a professional DBA to understand.
> > Whether or not it works.
>
> You haven't explained why we'd lose such folk next year when we
> haven't lost them already. MySQL has had replication (or at least
> has checked off the bullet point ;-)) for years. I'd be seriously
> surprised if any of the forks will offer significantly better
> replication than is there now, so the competitive situation is not
> changing in that regard.
>
> It is true that we're missing a chance to pull some folks away while
> the situation on that side of the fence is so messy. But I don't
> see our situation getting worse because of that, just not getting
> better.

"Not getting better," isn't a situation to be dismissed lightly. In
FLOSS, as I've seen it, a project whose adoption isn't growing is
dying.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Mon, Aug 24, 2009 at 10:15 PM, David Fetter<david(at)fetter(dot)org> wrote:
> On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote:
>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>> >> That is a slightly alarmist.  Who are we going to lose these
>> >> users to?
>>
>> > Drizzle.  MySQL forks.  CouchDB.  Any database which has
>> > replication which you don't need a professional DBA to understand.
>> > Whether or not it works.
>>
>> You haven't explained why we'd lose such folk next year when we
>> haven't lost them already.  MySQL has had replication (or at least
>> has checked off the bullet point ;-)) for years.  I'd be seriously
>> surprised if any of the forks will offer significantly better
>> replication than is there now, so the competitive situation is not
>> changing in that regard.
>>
>> It is true that we're missing a chance to pull some folks away while
>> the situation on that side of the fence is so messy.  But I don't
>> see our situation getting worse because of that, just not getting
>> better.
>
> "Not getting better," isn't a situation to be dismissed lightly.  In
> FLOSS, as I've seen it, a project whose adoption isn't growing is
> dying.

You may be right, but I'm not sure that it has much to do with the
ostensible topic of this thread. The reason why these features have
not already been committed is because they are not done. The question
of why they are not done and/or why more people aren't working on them
has been asked and answered. Figuring out some concrete steps that we
can take to address those issues is probably a better use of time than
trying to define whether these features are really important or
really, really important.

...Robert

On Aug 24, 2009, at 9:46 PM, Robert Haas wrote:

> On Mon, Aug 24, 2009 at 10:15 PM, David Fetter<david(at)fetter(dot)org>
> wrote:
>> On Mon, Aug 24, 2009 at 08:02:31PM -0400, Tom Lane wrote:
>>> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>>>>> That is a slightly alarmist. Who are we going to lose these
>>>>> users to?
>>>
>>>> Drizzle. MySQL forks. CouchDB. Any database which has
>>>> replication which you don't need a professional DBA to understand.
>>>> Whether or not it works.
>>>
>>> You haven't explained why we'd lose such folk next year when we
>>> haven't lost them already. MySQL has had replication (or at least
>>> has checked off the bullet point ;-)) for years. I'd be seriously
>>> surprised if any of the forks will offer significantly better
>>> replication than is there now, so the competitive situation is not
>>> changing in that regard.
>>>
>>> It is true that we're missing a chance to pull some folks away while
>>> the situation on that side of the fence is so messy. But I don't
>>> see our situation getting worse because of that, just not getting
>>> better.

One possible reason that replication is more critical now than it was
a year ago is the rise in cloud computing. The ability to fire up
instances on demand is much more useful when some of those boxes can
be database servers and those databases servers can replicate the
primary database and start doing something useful. As far as I can
tell this one feature alone is the one thing that makes it hard to
convince people to migrate away from Mysql despite it's demonstrable
inferiority in many other areas. Postgres seems to be winning
mindshare as the "real" and reliable database of choice for people who
are serious about their data. But for many, many businesses (many of
whom are really not that serious about their data) easy to set up
replication is just too big of a draw, such that you can't get them to
consider anything without it.

I don't know if current postgres users are really going to switch over
existing projects that were built on postgres, but for new apps
running on EC2 or similar I would not be surprised to see people
choosing mysql over postgres solely on this one issue. Databases
scalability is becoming and issue for more and more businesses and
others are filling in the gap. If postgres could combine it's current
deserved reputation for having a robust feature set, standards
compliance, high performance, reliability, stability, etc, with easy
to use replication it would be be a slam dunk, no-brainer decision to
go with postgres on just about anything.

Just my 2 cents.

Rick

P.S. I don't actually use mysql anywhere but I know many who do and
replication is always the sticking point.

Le mercredi 26 août 2009 à 01:36 -0600, Rick Gigger a écrit :
> One possible reason that replication is more critical now than it
> was
> a year ago is the rise in cloud computing. The ability to fire up
> instances on demand is much more useful when some of those boxes can
> be database servers and those databases servers can replicate the
> primary database and start doing something useful. As far as I can
> tell this one feature alone is the one thing that makes it hard to
> convince people to migrate away from Mysql despite it's demonstrable
> inferiority in many other areas.

I think you should have a deep look at
these two manuals that I wrote for Drupal:

Guidelines for writing MySQL and PostgreSQL compliant SQL
http://drupal.org/node/555514

and

Quidelines for writing SQL efficient code:
http://drupal.org/node/559302

I have been using PostgreSQL since 1998. I took part in the development
of pgAdmin 1 and pgAdmin 2. I am very proud of the PostgreSQL community,
but I think it should fix some important issues in the domain of SQL
compliance and compatibility.

When reading developers comments on Drupal web site, it seems that there
is deep misunderstanding between developers and SQL databases. I would
say that 1% of developers know database technology. For example, most
Drupal developers think that an INNER JOIN is faster than a LEFT JOIN.

The reality of facts is that developers will not even test PostgreSQL
and stop using it after the first SQL warning or error.

So I would recommend focussing on usability.

Then you can work on replication and materilized views. You probably
know that a cloud requires several computers. With materialized view, a
single computer can perform 100 times better. I see plenty of of
possibilities to improve speed using materialized views.

But first and firstly ... focus on usability. Then make a pre-release of
a PostgreSQL 8.4.2 release or so ... We need to wipe out this MySQL
issue once for all.

If there is a compat MySQL mode or functions, then include it in core.
This is too important for PostgreSQL success.

Why MySQL usability is achieved add materialized views and MySQL is dead
in the performance statistics, beaten 99% by PostgreSQL.

Kind regards,
Jean-Michel

Jean-Michel Pouré<jm(at)poure(dot)com> wrote:
> focus on usability.

It's not clear to me what you feel is needed. That could mean many
things....

-Kevin

Le mercredi 26 août 2009 à 09:30 -0500, Kevin Grittner a écrit :
> It's not clear to me what you feel is needed. That could mean many
> things....

Dear Kevin,

I rarely post on Hackers, so I will try to explain:
* I use PostgreSQL since 1998.
* I took part in the development of pgAdmin 1&2.
* I love PostgreSQL and I believe MySQL sucks. Recently I was forced to
use MySQL for Kdenlive.org project and the database sometimes stops
responding sending nothing or crap. I believe that if you use MySQL in
your company for a paid work, you can die of a heart attack.

But at the same time:
* I rewrote very long and tedious queries from PHPBB. Maybe 100 of them.
They are now part of PhpBB3. I drove all queries below 30 ms and this
enables PhpBB to scale easily. I consider this is my work.
* I think Drupal queries presently have performance problems. If I
wanted, I would be able to drive down Drupal web site, using a
collection of simple queries on projects, forum and comments. But I
don't of course.

This is why I wrote http://drupal.org/node/559302 and
http://drupal.org/node/555514

Everytime I try a new Drupal module under PostgreSQL, I run into tiny
SQL problems ranging from error to performance drop. The most
problematic problem is http://drupal.org/node/559986

To fix a problem, I need to open a thread on Drupal web site and wait
for the maintainer to discuss and commit.

To give an example, Drupal includes a query optimizer written in PHP,
which sometimes adds "DISTINCT" to queries. In the forum, some
incredible query fetches 22000 rows, copies the rows to an arrays and
then computes the array. This allows to display previous and next
message.

But we are not going to change the world of MySQL users, which believe
they know SQL programming and in reality are complete beginners, who
like to boast about farms and replication, just like Windows users like
to collect Adobe products on DVDs and discuss with friends about them.

IMHO for what I know from the porting work (I worked heavily on PHPBB3
and now Drupal), the first goal is to achieve compatibility with issues
mentioned there: http://drupal.org/node/555514 and add mysql compat
functions in PostgreSQL core without breaking existing code.

Then I can insure that 99% of MySQL compatibility problems will be
behind.

When this is achieve, we will be able to start education of developers.
And this will take another decade.

To win over MySQL, the best is to work on materialized views. There are
very good articles available from hackers. Why not port to C.
Materialized which which update when the data is needed would be
perfect.

Then we can convince Drupal hackers to add views in the schema. The
trick would be that MySQL would support normal views, whereas we would
also support materialized. We can do the same with nearly all available
frameworks: PhpBB, etc ...

Web apps are 95% of PostgreSQL possible users.

Kind regards,
Jean-Michel

Jean-Michel Pouré wrote:
>
> Everytime I try a new Drupal module under PostgreSQL, I run into tiny
> SQL problems ranging from error to performance drop. The most
> problematic problem is http://drupal.org/node/559986
>

I strongly suspect this post badly mis-diagnoses the problem.

>
> IMHO for what I know from the porting work (I worked heavily on PHPBB3
> and now Drupal), the first goal is to achieve compatibility with issues
> mentioned there: http://drupal.org/node/555514 and add mysql compat
> functions in PostgreSQL core without breaking existing code.
>

That might be your goal, but it's not the community's goal, I believe.
There are already external projects for compatibility libraries. You are
never going to achieve 100% compatibility.

>
> To win over MySQL, the best is to work on materialized views. There are
> very good articles available from hackers. Why not port to C.
> Materialized which which update when the data is needed would be
> perfect.
>

IIRC some work was being done on materialised views.

>
> Web apps are 95% of PostgreSQL possible users.
>

Most applications these days have a web front end. But that doesn't mean
the database needs to be terribly aware of them. To the database, a web
server is just another client.

cheers

andrew

On Aug 26, 2009, at 11:18 , Jean-Michel Pouré wrote:

> Web apps are 95% of PostgreSQL possible users.

Where does this figure come from?

Michael Glaesemann
grzm seespotcode net

Jean-Michel Pouré<jm(at)poure(dot)com> wrote:
> Kevin Grittner a écrit :
>> It's not clear to me what you feel is needed.

> http://drupal.org/node/559302

These look like performance issues.

> http://drupal.org/node/555514

These are MySQL compatibility issues.

So when you talk about focusing on usablility improvements you mean
that priority should be given to supporting MySQL-specific syntax
extensions and ensuring that there are no queries where the MySQL
optimizer comes up with a more efficient plan than PostgreSQL?

One concern I have is that you don't mention PostgreSQL configuration
in your performance advice, and I seem to remember you said that you
didn't tune your postgresql.conf file beyond boosting the
shared_buffers setting. If that's true, you might be somewhat
surprised with the performance improvements if you tweak just a few
other settings.

You might want to see what suggestions you get from:

http://pgfoundry.org/projects/configurator/

-Kevin

Robert Haas wrote:
> I am assuming that at least Hot Standby and Streaming Replication will
> likely require two CommitFests to go from the point where they are
> seriously reviewable to actual commit. So if they hit the 9/15 date,
> they should make 8.5 even with just three CommitFests. If they don't
> hit the 9/15 date, then a 3-CommitFest cycle will probably be too
> short for them to make it in. But if we schedule a fourth CommitFest
> in January in the hopes of seeing one of those patches committed, then
> ISTM we're basically speculating that the patch authors will not hit
> the 9/15 date but that they will hit an 11/15 date.

My concern is not just with those features, but with the whole ratio of
the window for new work to the total development cycle. That ratio keeps
going down and the time the tree is effectively frozen to new features
keeps going up. I'd like to see us keep the tree open as long as
possible but be much more ruthless about chopping off things that aren't
ready at the end. That way we can quickly get to a beta and get on with
the next cycle. I realise the idea is that significant features must be
submitted by the penultimate CF, but I'm not too sure how well that's
going to work in practice. That just seems like we're relabelling things
rather than a fundamental change. At the very least my vote goes for
four CFs rather than three.

cheers

andrew

On Wed, Aug 26, 2009 at 12:07 PM, Andrew Dunstan<andrew(at)dunslane(dot)net> wrote:
> Robert Haas wrote:
>>
>> I am assuming that at least Hot Standby and Streaming Replication will
>> likely require two CommitFests to go from the point where they are
>> seriously reviewable to actual commit.  So if they hit the 9/15 date,
>> they should make 8.5 even with just three CommitFests.  If they don't
>> hit the 9/15 date, then a 3-CommitFest cycle will probably be too
>> short for them to make it in.  But if we schedule a fourth CommitFest
>> in January in the hopes of seeing one of those patches committed, then
>> ISTM we're basically speculating that the patch authors will not hit
>> the 9/15 date but that they will hit an 11/15 date.
>
> My concern is not just with those features, but with the whole ratio of the
> window for new work to the total development cycle. That ratio keeps going
> down and the time the tree is effectively frozen to new features keeps going
> up.

I think that's a very valid concern. Against that, if release cycles
become very long, then features can hit the tree more of the time, but
they don't get into a released version for ages.

> I'd like to see us keep the tree open as long as possible but be much
> more ruthless about chopping off things that aren't ready at the end. That
> way we can quickly get to a beta and get on with the next cycle

I'm happy to assist with that, but recall that even after we ended CF
2008-11 another four months went by before release. That's a whole
lotta time for the tree to be closed right there.

> I realise
> the idea is that significant features must be submitted by the penultimate
> CF, but I'm not too sure how well that's going to work in practice. That
> just seems like we're relabelling things rather than a fundamental change.
> At the very least my vote goes for four CFs rather than three.

Fair enough, more votes are good.

...Robert

Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

> the window for new work to the total development cycle. That ratio
> keeps going down and the time the tree is effectively frozen to new
> features keeps going up. I'd like to see us keep the tree open as
> long as possible but be much more ruthless about chopping off things
> that aren't ready at the end. That way we can quickly get to a beta
> and get on with the next cycle. I realise the idea is that
> significant features must be submitted by the penultimate CF, but
> I'm not too sure how well that's going to work in practice. That
> just seems like we're relabelling things rather than a fundamental
> change. At the very least my vote goes for four CFs rather than
> three.

Unless the community can reduce the time between the start of the last
commit-fest and the release, you're limited to an average of four
months of programming time per year for new features (assuming that
people are observing the rules about what they should be doing during
commit-fests and beta testing). If you want to move the next release
back into Spring rather than Summer (which is the season in which 8.4
was released -- at least of those of us in the Northern Hemisphere),
you would need to shorten that to three months for this release.

Unless...

Both the ruthless cutting of anything not totally ready at the end of
a commit-fest, *and* reducing the time from the end of the last
commit-fest to release would be needed to get that up to five months
per year. We obviously don't want less testing during the beta cycle,
but delaying the release while the release notes are developed at the
end of the cycle seems like an obvious target for improvement. I'd
bet there are others, though I don't know what they are....

-Kevin

> Robert Haas wrote:
>> I am assuming that at least Hot Standby and Streaming Replication will
>> likely require two CommitFests to go from the point where they are
>> seriously reviewable to actual commit.

FWIW, I think that both HS and SR are special cases: if we ever see
reviewable patches for them, people will probably be willing to work
on them outside the CommitFest framework. We shouldn't be setting
the schedule with the idea that those will only be dealt with in CFs.
To my mind the CF process is for dealing with "run of the mill" patches.

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> My concern is not just with those features, but with the whole ratio of
> the window for new work to the total development cycle. That ratio keeps
> going down and the time the tree is effectively frozen to new features
> keeps going up.

Yup. This is a huge problem and we need to deal with it somehow. At
the same time, I'm worried that our beta testing process is already
inadequate. We've found several rather embarrassing bugs in 8.4, for
instance, things that should have been found in beta IMO. Shortening
beta or encouraging people to start next-cycle development instead of
testing doesn't seem like a wise move. You can't just develop all
the time, you have to test & debug too ...

regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>> My concern is not just with those features, but with the whole ratio of
>> the window for new work to the total development cycle. That ratio keeps
>> going down and the time the tree is effectively frozen to new features
>> keeps going up.
>>
>
> Yup. This is a huge problem and we need to deal with it somehow. At
> the same time, I'm worried that our beta testing process is already
> inadequate. We've found several rather embarrassing bugs in 8.4, for
> instance, things that should have been found in beta IMO. Shortening
> beta or encouraging people to start next-cycle development instead of
> testing doesn't seem like a wise move. You can't just develop all
> the time, you have to test & debug too ...
>
>
>

Perhaps some more formalised beta program would be useful. I have at
least one client who could probably be persuaded to devote some
resources to Beta testing. Maybe we need a Beta Program co-ordinator or
some such animal. I suspect that plenty of possible beta testers don't
even know when we go beta.

cheers

andrew

Andrew Dunstan escribió:

> Perhaps some more formalised beta program would be useful. I have at
> least one client who could probably be persuaded to devote some
> resources to Beta testing. Maybe we need a Beta Program co-ordinator
> or some such animal. I suspect that plenty of possible beta testers
> don't even know when we go beta.

This seems a good idea. Possibly pushing the betas more aggresively to
current users would make them tested not only by PG hackers ...

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

> Perhaps some more formalised beta program would be useful. I have
> at least one client who could probably be persuaded to devote some
> resources to Beta testing. Maybe we need a Beta Program co-ordinator
> or some such animal. I suspect that plenty of possible beta testers
> don't even know when we go beta.

In some shops it would be necessary to have a date set months in
advance of the start of the beta to have any chance of getting
managers to assign hardware and staff for a really good test. I
usually have to work in tests on my own time on whatever hardware
happens to be in standby status, when I can get it. Hitting such a
date would seem to require a significant change from prior releases;
although the just-completed commit-fest could be taken as evidence
that such a thing is possible.

-Kevin

On Wed, Aug 26, 2009 at 1:01 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Yup.  This is a huge problem and we need to deal with it somehow.  At
> the same time, I'm worried that our beta testing process is already
> inadequate.  We've found several rather embarrassing bugs in 8.4, for
> instance, things that should have been found in beta IMO.  Shortening
> beta or encouraging people to start next-cycle development instead of
> testing doesn't seem like a wise move.  You can't just develop all
> the time, you have to test & debug too ...

Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
doesn't necessarily buy you much, either. I'm good at focused
activity - but there was nothing focused about 8.4 beta that I could
see. Maybe we need some kind of TestFest process.

...Robert

Dear Kevin

> So when you talk about focusing on usablility improvements you mean
> that priority should be given to supporting MySQL-specific syntax
> extensions and ensuring that there are no queries where the MySQL
> optimizer comes up with a more efficient plan than PostgreSQL?

Yes. PostgreSQL should be able to run MySQL code quoted here:

This is a prerequisite for people to be willing to test and adopt
PostgreSQL. People are not willing to debug frameworks like Drupal and
port them to PostgreSQL. We are quite alone and lost.

> One concern I have is that you don't mention PostgreSQL configuration
> in your performance advice, and I seem to remember you said that you
> didn't tune your postgresql.conf file beyond boosting the
> shared_buffers setting. If that's true, you might be somewhat
> surprised with the performance improvements if you tweak just a few
> other settings.

shared_buffer 24M.

Kind regards,
Jean-Michel

Alvaro Herrera wrote:
> This seems a good idea. Possibly pushing the betas more aggresively to
> current users would make them tested not only by PG hackers ...

Isn't this the purpose of the new alpha releases, at lease to some extent.

"Matthew T. O'Connor" <matthew(at)zeut(dot)net> writes:
> Alvaro Herrera wrote:
>> This seems a good idea. Possibly pushing the betas more aggresively to
>> current users would make them tested not only by PG hackers ...

> Isn't this the purpose of the new alpha releases, at lease to some extent.

The alpha releases as currently constituted are practically the exact
opposite of what's being suggested here :-(. We are pushing them out
without very much advertisement, and certainly without any formal
program to encourage significant testing. That's more or less forced by
the decision that alpha releases should be low-overhead, but I think
we're unlikely to get wide-ranging test coverage that way.

regards, tom lane

2009/8/26 Jean-Michel Pouré <jm(at)poure(dot)com>:
> Dear Kevin
>
>> So when you talk about focusing on usablility improvements you mean
>> that priority should be given to supporting MySQL-specific syntax
>> extensions and ensuring that there are no queries where the MySQL
>> optimizer comes up with a more efficient plan than PostgreSQL?
>
> Yes. PostgreSQL should be able to run MySQL code quoted here:
>
> This is a prerequisite for people to be willing to test and adopt
> PostgreSQL. People are not willing to debug frameworks like Drupal and
> port them to PostgreSQL. We are quite alone and lost.

Er, so we should debug Drupal for them? I find it difficult to
believe that's the best use of our time.

>> One concern I have is that you don't mention PostgreSQL configuration
>> in your performance advice, and I seem to remember you said that you
>> didn't tune your postgresql.conf file beyond boosting the
>> shared_buffers setting.  If that's true, you might be somewhat
>> surprised with the performance improvements if you tweak just a few
>> other settings.
>
> shared_buffer 24M.

That doesn't actually respond to anything he said in that paragraph.

...Robert

>> So when you talk about focusing on usablility improvements you mean
>> that priority should be given to supporting MySQL-specific syntax
>> extensions and ensuring that there are no queries where the MySQL
>> optimizer comes up with a more efficient plan than PostgreSQL?

Well, I'd be interested in this just because I *always* want to improve
the optimizer. I think these cases demonstrate places we could improve
our optimizer still further, just as looking for the same cases with
Oracle or SQL Server does.

> Yes. PostgreSQL should be able to run MySQL code quoted here:
>
> This is a prerequisite for people to be willing to test and adopt
> PostgreSQL. People are not willing to debug frameworks like Drupal and
> port them to PostgreSQL. We are quite alone and lost.

First, off, the evidence is against that; people are doing the work to
port things. And creating new projects which are based on Postgres.

Second, we're not going to support MySQL's *bugs* and *bad design
decisions* which is what lazy developers actually want; they want
something exactly the same as MySQL, including bugs. If they want that,
they can use MySQL. We are not MySQL, and trying to out-MySQL MySQL is
stupid, just as it would be to copy Oracle exactly.

Now, there are things which MySQL does better which we should fix,
because they are real problems for our users who already like
PostgreSQL. These include simple replication, upgrade in place, driver
maintenance, covering indexes, MERGE, etc. But we'll do these because
they make *Postgres* better, not because MySQL has them.

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

On Wed, Aug 26, 2009 at 02:46:43PM -0400, Tom Lane wrote:
> "Matthew T. O'Connor" <matthew(at)zeut(dot)net> writes:
> > Alvaro Herrera wrote:
> >> This seems a good idea. Possibly pushing the betas more
> >> aggresively to current users would make them tested not only by
> >> PG hackers ...
>
> > Isn't this the purpose of the new alpha releases, at lease to some
> > extent.
>
> The alpha releases as currently constituted are practically the
> exact opposite of what's being suggested here :-(. We are pushing
> them out without very much advertisement, and certainly without any
> formal program to encourage significant testing. That's more or
> less forced by the decision that alpha releases should be
> low-overhead, but I think we're unlikely to get wide-ranging test
> coverage that way.

How would you recommend that this change?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david(at)fetter(dot)org> writes:
> On Wed, Aug 26, 2009 at 02:46:43PM -0400, Tom Lane wrote:
>> The alpha releases as currently constituted are practically the
>> exact opposite of what's being suggested here :-(. We are pushing
>> them out without very much advertisement, and certainly without any
>> formal program to encourage significant testing. That's more or
>> less forced by the decision that alpha releases should be
>> low-overhead, but I think we're unlikely to get wide-ranging test
>> coverage that way.

> How would you recommend that this change?

As far as the alpha releases go, I wouldn't --- I see no evidence that
we have the manpower to formalize them any more than they are now.
I do like the idea of trying to reach out to more beta testers and
manage that phase more aggressively. Maybe if we can make something
happen there, it would be possible to do the same for alphas later.

regards, tom lane

On Wednesday 26 August 2009 12:17:25 pm Josh Berkus wrote:

> > Yes. PostgreSQL should be able to run MySQL code quoted here:
> >
> > This is a prerequisite for people to be willing to test and adopt
> > PostgreSQL. People are not willing to debug frameworks like Drupal and
> > port them to PostgreSQL. We are quite alone and lost.
>
> First, off, the evidence is against that; people are doing the work to
> port things. And creating new projects which are based on Postgres.

Yeah the above is pretty much a non starter anymore. We out rank MySQL on
several polls in terms of popularity. Our channel is growing whereas the MySQL
one is shrinking. Our community isn't splintered and even projects like Drupal
are making it a specific point to fix the original mysql centric design.

> Now, there are things which MySQL does better which we should fix,
> because they are real problems for our users who already like
> PostgreSQL. These include simple replication, upgrade in place, driver
> maintenance, covering indexes, MERGE, etc. But we'll do these because
> they make *Postgres* better, not because MySQL has them.

Agreed.

Joshua D. Drake

--
PostgreSQL.org Major Contributor
Command Prompt, Inc : 503-667-4564 - http://www.commandprompt.com/
Since 1997, Consulting, Development, Support, Training

> Second, we're not going to support MySQL's *bugs* and *bad design
> decisions* which is what lazy developers actually want; they want
> something exactly the same as MySQL, including bugs. If they want
> that,
> they can use MySQL. We are not MySQL, and trying to out-MySQL MySQL
> is
> stupid, just as it would be to copy Oracle exactly.

I understand what you mean.

To tell you how lazy MySQL people are is my last experience in the
Drupal world. In short, on my devel server, Drupal previous/next link
display SQL script returns 21.000 rows.

Of course, it should return only two rows.

The 21.000 rows are then processed step by step by a PHP script.

I open a bug and one of Drupal core developers answers:
"Jean-Michel, this is a friendly warning, please change your behavior.
This is getting really annoying."

In fact, this core developer does not like the way I try to explain how
to use databases. I wrote two tutorials:

http://drupal.org/node/559302
and
http://drupal.org/node/555514

The truth is that Drupal core developers do not believe fixing the
prev/next link script is important. They don't care for SQL and don't
understand the relationship between SQL queries and CPU cycles.

Read this page: http://drupal.org/project/prev_next

This performance issue was known for several months and these MySQL
developers did not even fixed it.

Finally, I escalated to the founder of the Drupal community to ask him
to integrate a proper SQL Previous/Next script into Drupal.

Their last answer: "Holy crap jmpoure, this is not how the community
works. We don't beg to Dries."

Read: http://drupal.org/node/559424

> Now, there are things which MySQL does better which we should fix,
> because they are real problems for our users who already like
> PostgreSQL. These include simple replication, upgrade in place,
> driver
> maintenance, covering indexes, MERGE, etc. But we'll do these because
> they make *Postgres* better, not because MySQL has them.

After reading my story, I hope we can agree that noone is going to port
any MySQL code to PostgreSQL ever. This demands too much intellectual
efforts. Many people will migrate from DB2 and Oracle to PostgreSQL. But
no MySQL developer is going to use PostgreSQL if he needs to modify SQL
queries. I don't want to be offensive, but I really believe it.

So we should support a minimal set of MySQL SQL instructions.

After several years of porting MySQL code to PostrgeSQL, I believe that
this limited list is enough: http://drupal.org/node/555514

This is quite a straightforward need. Without this list of issues,
PostgreSQL may never be able to run popular products developed under
MySQL. Think of all commercial and free software projects. The impact of
MySQLisms are huge. I can only compare it to Windows vs. GNU/Linux or
FreeBSD. This is what comes in mind first.

We are not leaving in a perfect world and there no reason to achieve
perfectness. So let's support this list, please:
http://drupal.org/node/555514

Kind regards,
Jean-Michel

2009/8/26 Jean-Michel Pouré <jm(at)poure(dot)com>:
> After reading my story, I hope we can agree that noone is going to port
> any MySQL code to PostgreSQL ever. This demands too much intellectual

Surely this is a complete overgeneralization...

...Robert

Jean-Michel Pour wrote:
> To tell you how lazy MySQL people are is my last experience in the
> Drupal world. In short, on my devel server, Drupal previous/next link
> display SQL script returns 21.000 rows.

Yes, we have seen this too. We have always targeted serious database
developers, and those wanting to be serious. It is difficult to cater
to MySQL eccentricities and maintain a serious database offering.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Jean-Michel Pouré wrote:

> We are not leaving in a perfect world and there no reason to achieve
> perfectness. So let's support this list, please:
> http://drupal.org/node/555514

Have you tried Drupal 7? It's said to have many of these corrected.
Maybe you should stop wasting your time with 6.x.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

2009/8/26 Jean-Michel Pouré <jm(at)poure(dot)com>:
> After reading my story, I hope we can agree that noone is going to port
> any MySQL code to PostgreSQL ever. This demands too much intellectual
> efforts. Many people will migrate from DB2 and Oracle to PostgreSQL. But
> no MySQL developer is going to use PostgreSQL if he needs to modify SQL
> queries. I don't want to be offensive, but I really believe it.
>
> So we should support a minimal set of MySQL SQL instructions.
>
> After several years of porting MySQL code to PostrgeSQL, I believe that
> this limited list is enough: http://drupal.org/node/555514
>
> This is quite a straightforward need. Without this list of issues,
> PostgreSQL may never be able to run popular products developed under
> MySQL. Think of all commercial and free software projects. The impact of
> MySQLisms are huge. I can only compare it to Windows vs. GNU/Linux or
> FreeBSD. This is what comes in mind first.
>
> We are not leaving in a perfect world and there no reason to achieve
> perfectness. So let's support this list, please:
> http://drupal.org/node/555514
>

I'm starting to see how you rubbed the Drupal people the wrong way......

Most of the items on your list are misdiagnosed problems or are
features that we think are bugs in MySQL. You would get further asking
questions and for suggestions than you would insisting that you know
better and Postgres should change its interface to match MySQL.

With your current approach you're likely to get dismissed out of hand,
not unlike what I can well believe happened in the Drupal world. That
would be unfortunate because I think there are 2 or 3 real
improvements hidden in your list.

--
greg
http://mit.edu/~gsstark/resume.pdf

On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
> doesn't necessarily buy you much, either. I'm good at focused
> activity - but there was nothing focused about 8.4 beta that I could
> see. Maybe we need some kind of TestFest process.

Yeah, exactly. I can't imagine end users would know what to do during
beta. Even assuming that you have release notes at the beginning of
beta, you can't expect people to go through every item and do a formal
test for it. Surely it's been tested before, else it would not be in
the release, right?

Le mercredi 26 août 2009 à 22:06 +0100, Greg Stark a écrit :
> With your current approach you're likely to get dismissed out of hand,
> not unlike what I can well believe happened in the Drupal world.
This is the case.

> That
> would be unfortunate because I think there are 2 or 3 real
> improvements hidden in your list.

Then explain I don't have your skills.

Thanks.

Le mercredi 26 août 2009 à 16:56 -0400, Alvaro Herrera a écrit :
> Have you tried Drupal 7? It's said to have many of these corrected.
> Maybe you should stop wasting your time with 6.x.

I am running a large community on the Net and people would like to
migrate our framework to Drupal. We agreed to.

I already assembled 30 modules to develop a complete service.

But it does not scale very well as my testings show that queries are
slow. I wrote SQL guides to inform the Drupal community. It seems that
Drupal developers do not make a relationship between database and CPU
time or memory usage. Therefore Drupal PHP cache is filled with SQL
queries. At it demands more and more memory.

I am probably going to migrate to Drupal 6.x in a few days and I will
fix queries by hand in case of problems. This is what happened when we
migrated to PhpBB.

Bye, Jean-Michel

Tom, all,

> As far as the alpha releases go, I wouldn't --- I see no evidence that
> we have the manpower to formalize them any more than they are now.
> I do like the idea of trying to reach out to more beta testers and
> manage that phase more aggressively. Maybe if we can make something
> happen there, it would be possible to do the same for alphas later.

Well, we need a concrete list of things to beta test if we want people
to do this in any organized fashion. It's easy enough for us to say "we
need beta testers"; but without telling our community *what* to test,
people just download, compile, do a select * from table, and say "it works".

Once we have a list, we can launch a simple app/wiki page where people
can report results of their tests.

Without all that, the task is too amorphous for us to expect much from
the community-at-large.

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

Josh Berkus wrote:
> Tom, all,
>
>
>> As far as the alpha releases go, I wouldn't --- I see no evidence that
>> we have the manpower to formalize them any more than they are now.
>> I do like the idea of trying to reach out to more beta testers and
>> manage that phase more aggressively. Maybe if we can make something
>> happen there, it would be possible to do the same for alphas later.
>>
>
> Well, we need a concrete list of things to beta test if we want people
> to do this in any organized fashion. It's easy enough for us to say "we
> need beta testers"; but without telling our community *what* to test,
> people just download, compile, do a select * from table, and say "it works".
>
> Once we have a list, we can launch a simple app/wiki page where people
> can report results of their tests.
>
> Without all that, the task is too amorphous for us to expect much from
> the community-at-large.
>
>

Actually, what I had in mind was getting people to run their
applications etc. in some non-production environment on the beta. I
talked to a client today and he said "sure, we have several development
environments and we can put one or two on the beta and then let the
developers just do their thing on it." Testing the things we know about
is in a way less important than making sure nothing else got broken.

cheers

andrew

On Wed, Aug 26, 2009 at 5:12 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
> On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
>> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
>> doesn't necessarily buy you much, either.  I'm good at focused
>> activity - but there was nothing focused about 8.4 beta that I could
>> see.  Maybe we need some kind of TestFest process.
>
> Yeah, exactly.  I can't imagine end users would know what to do during
> beta.  Even assuming that you have release notes at the beginning of
> beta, you can't expect people to go through every item and do a formal
> test for it.  Surely it's been tested before, else it would not be in
> the release, right?

I would sure hope so. Testing features individually makes a whole lot
more sense to me than testing the release as a whole. Just trying a
bunch of random stuff and seeing if anything breaks is not a very
productive activity. On the other hand, testing individual features
is frequently very productive, but it's my understanding of the way PG
does things that that is supposed to happen before the patch is
committed.

It appears to me that most of the really nasty bugs that have been
found in 8.4.0 relate to one of the following three things, each of
which seems to be related to multiple back-branch commits.

1. SEMI/ANTI join support.
2. running the bgwriter during recovery (infrastructure changes for recovery)
3. deadman switch

Maybe some of these weren't tested well enough prior to commit? Or
perhaps they're just more significant changes and therefore likely
spots for rough edges.

I think there is a lot of merit (as Andrew suggests) in running a
production application on a beta version of the database just to see
if anything funny happens. But insisting that all PG developers stop
doing development to focus ONLY on that activity doesn't seem very
reasonable: not everyone is well-placed to do that kind of experiment,
or cares to do so. Conversely, there are many people who are NOT
developers who ARE well-placed to beta test (for example, Kevin
Grittner does a lot more testing than he does development, and I think
there are few people on this mailing list who would argue that the
quality of that testing is any less than kickass).

...Robert

Hi,

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On ons, 2009-08-26 at 14:26 -0400, Robert Haas wrote:
>> Sure, but an aimless mandate to do testing for 4 (or 8, or 12) months
>> doesn't necessarily buy you much, either. I'm good at focused
>> activity - but there was nothing focused about 8.4 beta that I could
>> see. Maybe we need some kind of TestFest process.
>
> Yeah, exactly. I can't imagine end users would know what to do during
> beta. Even assuming that you have release notes at the beginning of
> beta, you can't expect people to go through every item and do a formal
> test for it. Surely it's been tested before, else it would not be in
> the release, right?

Well we all know that developpers are really bad at testing, because
they tend to test for cases they though about while developping the code
rather than being creative and running a full application against the
overall new product.

Now, it could be that what we miss is some tool suite to enable
application people to test their full code against our new releases and
check results, performance, plans, etc.

I know about a couple of tools to get there, Tsung and Playr. And the
focus is performance testing and scalability, not that it still works.

Is the offering good enough? We might need to run some kind of tutorials
for users to be able to run large tests easily, and maybe think about
some newer tools allowing to compare logs of two application runs in two
database versions (capture all queries and result in a database, then
have a way to diff). Then beta testing would mean having a spare machine
where to run the magic regression test suite against some internal
application.

Regards,
--
dim

Tsung: http://tsung.erlang-projects.org/
Playr: https://area51.myyearbook.com/trac.cgi/wiki/Playr

PS: sql level unit testing isn't an answer here as it means the
application already have the tests when entering beta. Hard sell.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Wed, Aug 26, 2009 at 5:12 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
>> ... Surely it's been tested before, else it would not be in
>> the release, right?

> I would sure hope so. Testing features individually makes a whole lot
> more sense to me than testing the release as a whole. Just trying a
> bunch of random stuff and seeing if anything breaks is not a very
> productive activity.

I beg to disagree. New features have presumably been tested, in
isolation, by the original developer and reviewers. The things we tend
to miss are bad side-effects on corner cases and seemingly-unrelated
features. So "testing as a whole" is exactly what beta is for, to my
mind.

> I think there is a lot of merit (as Andrew suggests) in running a
> production application on a beta version of the database just to see
> if anything funny happens.

... but here we seem to be coming out at the same place anyway. Getting
people to put their existing apps onto a beta is very productive.
We have to encourage people to do more of that while it's still beta,
instead of waiting till .0 or .1 or later.

regards, tom lane

2009/8/26 Jean-Michel Pouré <jm(at)poure(dot)com>:
> Le mercredi 26 août 2009 à 22:06 +0100, Greg Stark a écrit :
>> That
>> would be unfortunate because I think there are 2 or 3 real
>> improvements hidden in your list.
>
> Then explain I don't have your skills.

What I'm suggesting is that you should take a different direction,
both with Drupal and with us. Instead of saying Postgres is wrong
because of your analysis, ask why what's going wrong and what the
right way to fix it is. So instead of telling people what to do
you're asking for their help. People are a lot more friendly when
asked for help than when being bossed around.

Your list:

> PostgreSQL and MySQL do not use the same concatenation funtions (D6 only, fixed in D7)

Personally I don't see a problem with us adding this to Postgres now
that we have variadic functions. I'm not sure why others are so
dead-set against it; it seems a lot less burdensome than some of the
Oracle compatibility stuff we have.

> PostgreSQL does CAST implicitely between int and a domain derived from int (unsolved)

This is misdiagnosed. It's not clear where the problem lies if there's
a problem but it likely has nothing to do with the cast. If you posted
with information and asked for help people would suggest things to
test to get more information and eventually find the best way to fix
it.

> PostgreSQL does not allow DELETE on JOINS
> PostgreSQL does not allow UPDATE or DELETE on multiple tables (not solved)

These are a open issues. I would like to have better support here but
it's a major feature, not just a minor compatibility issue. Again if
you posted the queries and asked for suggestions people would help you
rewrite the queries.

> PostgreSQL does not allow mutiple ROW inserts (PostgreSQL 8.2 + only)

As you point out this was added to Postgres 3 years ago

> PostgreSQL does not allow nested ORDER BY

I don't think this is something we're interested in doing. The grammar
is hard enough to maintain as it is.

> PostgreSQL does not automatically cast data between BOOLEAN and INT
> PostgreSQL does not automatically cast data between INT and VARCHAR/CHAR

These are things we've gone out of our way to NOT do. At some cost
too. Being loose here makes it easy to miss errors in your SQL.

> PostgreSQL requires all non-aggregated fields to be present in the GROUP BY clause

As I explained at length we could do something here but it would be a
major feature and it would still not be compatible with MySQL unless
you happen to be using it in a particular (common) way. For more
general uses you have to use DISTINCT ON.

> PostgreSQL requires single quotes and not double-quotes

There's no way in the world we would switch this. We're an SQL engine
and we parse the SQL language which uses quotes in a certain way. If
we switched this we would be parsing some other very different
language and everything else written in SQL (including, say, internal
queries for referential integrity checks or psql tab completion or
pg_dump) would break.

--
greg
http://mit.edu/~gsstark/resume.pdf

2009/8/26 Greg Stark <gsstark(at)mit(dot)edu>:
>> PostgreSQL does not automatically cast data between BOOLEAN and INT
>> PostgreSQL does not automatically cast data between INT and VARCHAR/CHAR
>
> These are things we've gone out of our way to NOT do. At some cost
> too. Being loose here makes it easy to miss errors in your SQL.

Actually it always bothered me that we don't have implicit casts from
integer->boolean. I can't see any ambiguity or unintentional effects
this would cause problems with. Am I missing something?

--
greg
http://mit.edu/~gsstark/resume.pdf

Greg Stark wrote:
> > PostgreSQL and MySQL do not use the same concatenation funtions (D6 only, fixed in D7)
>
> Personally I don't see a problem with us adding this to Postgres now
> that we have variadic functions. I'm not sure why others are so
> dead-set against it; it seems a lot less burdensome than some of the
> Oracle compatibility stuff we have.

The Oracle functions are there to add functionality that can't be easily
performed using ANSI syntax; this is not true of concact() and ||.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Greg Stark <gsstark(at)mit(dot)edu> wrote:

> Actually it always bothered me that we don't have implicit casts
> from integer->boolean. I can't see any ambiguity or unintentional
> effects this would cause problems with. Am I missing something?

I'd be at least a little bit concerned about how such automatic
casting to boolean might interact with bug #3822:

http://archives.postgresql.org/pgsql-bugs/2007-12/msg00145.php

-Kevin

On Wed, Aug 26, 2009 at 11:15 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> ... but here we seem to be coming out at the same place anyway.  Getting
> people to put their existing apps onto a beta is very productive.
> We have to encourage people to do more of that while it's still beta,
> instead of waiting till .0 or .1 or later.

+1 for the goal being to get users to test their applications on beta.

I also wonder if we've adopted the wrong strategy with betas by
stopping development during them. It seems to be the worst of both
worlds in that both developers and users are unhappy.

Perhaps we should fork 8.6 right away after 8.5.beta is released (I'm
assuming *after* any open issues are closed) and start a commitfest
with any pending patches. While we do that call for users to test
8.5.beta with their applications and wait a fixed amount of time for
any bugs to turn up. That lets us have a long beta period for users to
test on without stopping development.

--
greg
http://mit.edu/~gsstark/resume.pdf

On Thu, Aug 27, 2009 at 12:03 AM, Dimitri
Fontaine<dfontaine(at)hi-media(dot)com> wrote:
> Is the offering good enough? We might need to run some kind of tutorials
> for users to be able to run large tests easily, and maybe think about
> some newer tools allowing to compare logs of two application runs in two
> database versions (capture all queries and result in a database, then
> have a way to diff). Then beta testing would mean having a spare machine
> where to run the magic regression test suite against some internal
> application.

Well, people may recall I spent a lot of time testing 8.3 before and
during beta. The important words are "a lot of time", probably one
month full time spread on 3 months to find *only* 3 problems Tom,
Alvarro and Andrew fixed: yes one month for only 3 problems
identified, reported, discussed and fixed.
The problem isn't to connect your application to the database - it's
the easy part: if you have a large one, you probably won't see the
anomalies.
The application I used for my tests is displaying every SQL query at
the bottom of the page with the time spent executing the query, I was
switching from the 8.1 site to the 8.3 site to check everything
manually.
I also got all the urls of this application (more than one million),
and use a load test tool to load each page and pgFouine to grab any
error from the PostgreSQL logs.

Even with these information and this work, I'm pretty sure I would
have missed a join problem which would have returned 2 or 3 more rows
even with the time I spent working on it.

My plan at the time was to develop an application which would parse
the query logs from the server, replay the queries on 2 PG servers
with both versions and report me the anomalies (difference in the
number of rows, difference in the content, queries slower than with
the old version).

I haven't had the opportunity to work on the 8.4 beta test due to my
daily (and often nightly) job work load but the idea is still there
and IMHO it's really necessary if we want to be able to detect the
problems we only discovered after the release.

--
Guillaume

Jean-Michel,

> The truth is that Drupal core developers do not believe fixing the
> prev/next link script is important. They don't care for SQL and don't
> understand the relationship between SQL queries and CPU cycles.

I doubt this would be an accurate description of all Drupal developers.
The Drupal developers I've met have been very intelligent; most don't
know much about SQL database, but then I know zero JavaScript so we're even.

The answer to this is to be educational and not confrontational. And
submit patches ... to a current development version and not an old version.

> After reading my story, I hope we can agree that noone is going to port
> any MySQL code to PostgreSQL ever.

Again, the evidence is against you.

> So we should support a minimal set of MySQL SQL instructions.

In a loadable module, sure. Patches?

> After several years of porting MySQL code to PostrgeSQL, I believe that
> this limited list is enough: http://drupal.org/node/555514

There's other stuff as well. Also, see the updates people have made to
your list.

> can only compare it to Windows vs. GNU/Linux or
> FreeBSD. This is what comes in mind first.

Well, I don't use Windows either. So that's a pretty weak argument with
me ...

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

On ons, 2009-08-26 at 18:15 -0400, Tom Lane wrote:
> > I think there is a lot of merit (as Andrew suggests) in running a
> > production application on a beta version of the database just to see
> > if anything funny happens.
>
> ... but here we seem to be coming out at the same place anyway. Getting
> people to put their existing apps onto a beta is very productive.
> We have to encourage people to do more of that while it's still beta,
> instead of waiting till .0 or .1 or later.

I think people should be running their applications' system tests on top
of the new PostgreSQL. Just installing the application, clicking three
buttons, and I-don't-have-more-time-than-this helps a little but not
much.

Of course many people won't have system tests, which is why this process
is a problem.

To pick up a current example: "Drupal system tests pass with 8.5betaN"
is nice and useful. "I ran our app on 8.5betaN and didn't see any
issues" is interesting, but ultimately doesn't help much.

Much of the delay and uncertainty during beta in my mind comes from the
situation that we wait for negative results and don't trust the release
until we have seen and fixed enough of them. Instead of waiting for
concrete, positive results and producing the release with confidence.

Peter Eisentraut wrote:
> Much of the delay and uncertainty during beta in my mind comes from the
> situation that we wait for negative results and don't trust the release
> until we have seen and fixed enough of them. Instead of waiting for
> concrete, positive results and producing the release with confidence.

Yep, that is our dilemma.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Aug 26, 2009, at 8:17 AM, Jean-Michel Pouré wrote:

> Le mercredi 26 août 2009 à 01:36 -0600, Rick Gigger a écrit :
>> One possible reason that replication is more critical now than it
>> was
>> a year ago is the rise in cloud computing. The ability to fire up
>> instances on demand is much more useful when some of those boxes can
>> be database servers and those databases servers can replicate the
>> primary database and start doing something useful. As far as I can
>> tell this one feature alone is the one thing that makes it hard to
>> convince people to migrate away from Mysql despite it's demonstrable
>> inferiority in many other areas.
>
> I think you should have a deep look at
> these two manuals that I wrote for Drupal:
>
> Guidelines for writing MySQL and PostgreSQL compliant SQL
> http://drupal.org/node/555514
>
> and
>
> Quidelines for writing SQL efficient code:
> http://drupal.org/node/559302
>
> I have been using PostgreSQL since 1998. I took part in the
> development
> of pgAdmin 1 and pgAdmin 2. I am very proud of the PostgreSQL
> community,
> but I think it should fix some important issues in the domain of SQL
> compliance and compatibility.
>
> When reading developers comments on Drupal web site, it seems that
> there
> is deep misunderstanding between developers and SQL databases. I would
> say that 1% of developers know database technology. For example, most
> Drupal developers think that an INNER JOIN is faster than a LEFT JOIN.
>
> The reality of facts is that developers will not even test PostgreSQL
> and stop using it after the first SQL warning or error.
>
> So I would recommend focussing on usability.
>
> Then you can work on replication and materilized views. You probably
> know that a cloud requires several computers. With materialized
> view, a
> single computer can perform 100 times better. I see plenty of of
> possibilities to improve speed using materialized views.
>
> But first and firstly ... focus on usability. Then make a pre-
> release of
> a PostgreSQL 8.4.2 release or so ... We need to wipe out this MySQL
> issue once for all.
>
> If there is a compat MySQL mode or functions, then include it in core.
> This is too important for PostgreSQL success.
>
> Why MySQL usability is achieved add materialized views and MySQL is
> dead
> in the performance statistics, beaten 99% by PostgreSQL.

This may be your experience and maybe there are stats to back this
up. I was simply saying, that in my experience I have consulted with
companies using cloud computing services (ie EC2) and mysql. They are
performance conscious. When they have to fire up more boxes, they pay
for it immediately. When they ran into problems getting good
performance out of mysql it was very easy to show them how to get
better performance using postgres. (Often this was just: do the same
thing in postgres, and look, it's faster!). But they also rely on
replication to be able to scale. And without it they just weren't
interested.

Porting any project has it's own set of issues, I was speaking to the
case where people are evaluating databases for a new project. I was
not however trying to make any kind of statement as too how important
it is as compared to any other specific feature.

I was just trying to say that in my experience current trends indicate
that having easy to set up replication is getting more important over
time, not less, and not the same. Other features may be more
important. Getting it right is certainly more important that getting
it soon (for reasonable values of "soon" and "right" of course IMHO).
The experience of others my be totally different, but that is mine.

Just wanted to clarify what I was actually trying to say because this
response seems to indicate that I didn't make certain things clear.

- Rick

Greg Stark <gsstark(at)mit(dot)edu> writes:
> Actually it always bothered me that we don't have implicit casts from
> integer->boolean. I can't see any ambiguity or unintentional effects
> this would cause problems with. Am I missing something?

Personally, as an old Pascal-lover, I always thought that C's failure
to distinguish between int and boolean was the single biggest design
mistake in the language. I'm very glad that SQL doesn't make that
mistake, and I don't want to go against the standard to introduce it.

regards, tom lane

* Andrew Dunstan (andrew(at)dunslane(dot)net) wrote:
> Actually, what I had in mind was getting people to run their
> applications etc. in some non-production environment on the beta. I
> talked to a client today and he said "sure, we have several development
> environments and we can put one or two on the beta and then let the
> developers just do their thing on it." Testing the things we know about
> is in a way less important than making sure nothing else got broken.

I agree entirely with Andrew here- what we need are a set of users who
would be willing to run their actual applications against a beta release
in a testing environment. The Beta-Mom position would be working with
some list of users who've volunteered to do that; prodding them when a
new beta comes out, poking them for feedback, working with them on
issues they run into, etc.

The other possible group of users are those who are interested and
willing to beta-test actual new external-facing features. That'd be
great to have as well, but I don't believe is as important. New
features having bugs are a much smaller impact, overall, than bugs which
have been introduced in existing code-paths due to changes.

Thanks,

Stephen

On Thu, Aug 27, 2009 at 1:24 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Greg Stark <gsstark(at)mit(dot)edu> writes:
>> Actually it always bothered me that we don't have implicit casts from
>> integer->boolean. I can't see any ambiguity or unintentional effects
>> this would cause problems with. Am I missing something?
>
> Personally, as an old Pascal-lover, I always thought that C's failure
> to distinguish between int and boolean was the single biggest design
> mistake in the language.  I'm very glad that SQL doesn't make that
> mistake, and I don't want to go against the standard to introduce it.

I'm sure I can think of bigger flaws in C than that :)

I tend to think SQL has more in common with lisp than either of those,
perhaps because of the tinge of functional programming style.

But I think you're generalizing my suggestion to the point of building
a straw man to say "failure to distinguish". You could argue that
using a boolean in places where integers are expected could be
confusing or dangerous. But using other data types where boolean
values are expected is perfectly reasonable and safe -- especially in
cases like integer where people do expect it to work and the behaviour
is very predictable.

--
greg
http://mit.edu/~gsstark/resume.pdf

Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
>>> That is a slightly alarmist. Who are we going to lose these users to?
>
>> Drizzle. MySQL forks. CouchDB. Any database which has replication
>> which you don't need a professional DBA to understand. Whether or not
>> it works.
>
> You haven't explained why we'd lose such folk next year when we haven't
> lost them already. MySQL has had replication (or at least has checked
> off the bullet point ;-)) for years.

I think it's a slow but ongoing stream of organizations that are
switching away using logic similar to the thoughts outlined here:

http://archives.postgresql.org/pgsql-hackers/2008-05/msg00955.php
"...switched their bugzilla from Postgres to MySQL because the
admins didn't want to deal with Slony any more. People want simple."

MySQL may not have caught postgres in a number of ways yet, but
it's good enough now for many of the things it wasn't good enough
for earlier. And if it's good enough and easier, it's easy to switch.

On Thu, Aug 27, 2009 at 2:30 AM, Stephen Frost<sfrost(at)snowman(dot)net> wrote:
> I agree entirely with Andrew here- what we need are a set of users who
> would be willing to run their actual applications against a beta release
> in a testing environment.  The Beta-Mom position would be working with
> some list of users who've volunteered to do that; prodding them when a
> new beta comes out, poking them for feedback, working with them on
> issues they run into, etc.

I don't think we need the Beta-Mom to work with them on issues they
run into. We want to hear those things on the mailing lists.

But I do think the idea of having someone being Beta-Mom is a good
idea because of this simple point: right now we don't know if we have
any beta testers. All we hear about is when they run into problems.
What we want to know is how many people have run their applications
and *not* had problems.

So I see the Beta-Mom as being in charge of:

Gathering a list of volunteers willing to report results
For each volunteer getting a list of features they tested
For each volunteer getting a "yay" or "nay" report

Ideally we would want to know that people have run their application
frameworks for a certain length of time, with slave databases set up,
with slony set up, tested failover, with various pluggable langauges,
with various contrib modules, with large objects, with full text
search, etc.

Once we have a reasonable number of positive reports covering a wide
range of heavy-duty configuration then we can start to base our
confidence with going ahead with a release on those positive reports
rather than simply on the amount of time that's passed without a
problem.

--
greg
http://mit.edu/~gsstark/resume.pdf

Tom Lane wrote:
> Greg Stark <gsstark(at)mit(dot)edu> writes:
>> Actually it always bothered me that we don't have implicit casts from
>> integer->boolean. I can't see any ambiguity or unintentional effects
>> this would cause problems with. Am I missing something?
>
> Personally, as an old Pascal-lover, I always thought that C's failure
> to distinguish between int and boolean was the single biggest design
> mistake in the language. I'm very glad that SQL doesn't make that
> mistake, and I don't want to go against the standard to introduce it.

Then you should love Python where everything non-empty is regarded True
in boolean context ;-)

Cheers
Tino

Dear all,

Just a quick message about postgresql.conf auto-configuration.

When MySQL users test PostgreSQL, they load their data and run simple
SQL queries. If postgresql.conf is configured with default values, it
may produce slow results.

Would there be a way for postgresql.conf to auto configure?

This is how MySQL works. Nothing is configurable. And people love it. So
why not implement a simple auto configuration mechanism which would set
basic parameters.

Example
autotune=true;
autotune_profile='web site';
autotune_profile='dedicated SQL server';

...

Kind regards,
Jean-Michel

Tom Lane írta:
> Greg Stark <gsstark(at)mit(dot)edu> writes:
>
>> Actually it always bothered me that we don't have implicit casts from
>> integer->boolean. I can't see any ambiguity or unintentional effects
>> this would cause problems with. Am I missing something?
>>
>
> Personally, as an old Pascal-lover, I always thought that C's failure
> to distinguish between int and boolean was the single biggest design
> mistake in the language.

On the other hand, the first two programming languages I have learnt
was Sinclair ZX Spectrum BASIC and Z80 assembly,
I welcomed C being so near to assembly... :-)

> I'm very glad that SQL doesn't make that
> mistake, and I don't want to go against the standard to introduce it.
>

In that we agree. SQL is not C.

Best regards,
Zoltán Böszörményi

--
Bible has answers for everything. Proof:
"But let your communication be, Yea, yea; Nay, nay: for whatsoever is more
than these cometh of evil." (Matthew 5:37) - basics of digital technology.
"May your kingdom come" - superficial description of plate tectonics

----------------------------------
Zoltán Böszörményi
Cybertec Schönig & Schönig GmbH
http://www.postgresql.at/

On Wed, Aug 26, 2009 at 7:39 PM, Bruce Momjian<bruce(at)momjian(dot)us> wrote:
> Peter Eisentraut wrote:
>> Much of the delay and uncertainty during beta in my mind comes from the
>> situation that we wait for negative results and don't trust the release
>> until we have seen and fixed enough of them.  Instead of waiting for
>> concrete, positive results and producing the release with confidence.
>
> Yep, that is our dilemma.

To get positive results in which you can have confidence, you have to
know that the testing which was done actually did a reasonably good
job exercising the code in a way that would have flushed out bugs, had
any been present. That sounds a lot like the definition of a
regression test suite. Of course, we have that already, but it's
nowhere near comprehensive. Maybe we should be looking at an expanded
test suite that runs on a time scale of hours rather than seconds.
Actually, didn't Peter talk about something like this at PGCon?

I don't think that any test suite is going to eliminate the need for
beta-testing. But if we could say that we had a regression test suite
which covered X% of our code, and it passed on all Y platforms tested,
that would certainly be a confidence booster, especially for large
values of X. What we're basically doing now is hoping that
beta-testers come up with some novel tests, and that's a bit
hit-or-miss.

Part of the question, of course, is how to build up such a regression
test suite. One way to do it is try to add a test every time you fix
a bug, such that the new test fails on the unpatched code and passes
with the fix... or we could have a expanded-regression-test only
commitfest during beta, or something. I think people would be willing
to put in some work on this. I certainly would. I run the regression
tests frequently during development but unless I'm making system
catalog changes they rarely catch anything. I'm not prepared to write
the whole thing myself, but I would definitely be willing to develop
and contribute some tests.

...Robert

Le mercredi 26 août 2009 à 15:51 -0700, Josh Berkus a écrit :
> I doubt this would be an accurate description of all Drupal
> developers.

My opinion was :

Before adding replication to PostgreSQL, it would be better to support a
basic set of MySQL syntax seems relevant:

DELETE FROM table1, table2 FROM table1 INNER JOIN table2 ON ...
UPDATE table1, table1 SET table1.foo=1, table2.bar=3 WHERE table1.foo=2
INNER JOIN table2 ...,
etc ...

There is not much to add to PostgreSQL, but it seems relevant.

Otherwise, replicating some MySQL SQL syntax will not work.

As you know, people willing to use PostgreSQL replication are possibly
already MySQL replication users. So if they test and PostgreSQL fails,
this is too bad.

Drupal was only an example, I did not mean to criticize all Drupal
developers. Just to say they focus on PHP and not SQL. They don't have
time to port MySQL code. Besides, they are very nice people.

So I apologize for this short (false) focus.

Bye, Jean-Michel

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> Maybe we should be looking at an expanded test suite that runs on a
> time scale of hours rather than seconds.

> if we could say that we had a regression test suite which covered X%
> of our code, and it passed on all Y platforms tested, that would
> certainly be a confidence booster, especially for large values of X.

> Part of the question, of course, is how to build up such a
> regression test suite.

Aren't there code coverage monitoring tools that could be run during
regression tests? Sure it would take some time to review the results
and fashion tests to exercise chunks of code which were missed, but at
least we could quantify X and try to make incremental progress on
increasing it....

-Kevin

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> ... That sounds a lot like the definition of a
> regression test suite. Of course, we have that already, but it's
> nowhere near comprehensive. Maybe we should be looking at an expanded
> test suite that runs on a time scale of hours rather than seconds.

mysql's got one of those, and I haven't noticed that it's kept their
defect rate down any. Hour-long regression suites are the sort of
thing developers won't run. Worse, regression suites are necessarily
designed to exercise only 100%-reproducible behavior.

> I don't think that any test suite is going to eliminate the need for
> beta-testing.

Precisely...

What I'd like to see is some sort of test mechanism for WAL recovery.
What I've done sometimes in the past (and recently had to fix the tests
to re-enable) is to kill -9 a backend immediately after running the
regression tests, let the system replay the WAL for the tests, and then
take a pg_dump and compare that to the dump gotten after a conventional
run. However this is quite haphazard since (a) the regression tests
aren't especially designed to exercise all of the WAL logic, and (b)
pg_dump might not show the effects of some problems, particularly not
corruption in non-system indexes. It would be worth the trouble to
create a more specific test methodology.

In short: merely making the tests bigger doesn't impress me in the
least. Focused testing on areas we aren't covering at all could be
worth the trouble.

regards, tom lane

On Thu, Aug 27, 2009 at 10:11 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> ... That sounds a lot like the definition of a
>> regression test suite.  Of course, we have that already, but it's
>> nowhere near comprehensive.  Maybe we should be looking at an expanded
>> test suite that runs on a time scale of hours rather than seconds.
>
> mysql's got one of those, and I haven't noticed that it's kept their
> defect rate down any.  Hour-long regression suites are the sort of
> thing developers won't run.

Well, I'll run them. And I bet we could get volunteers to provide
machines to run them every night, too, against CVS HEAD. This has
been discussed before, and I wasn't the one who suggested it. I can't
speak to the value (or lack thereof) of mysql's regression test suite
as I know nothing about it, but even if it is completely worthless
that does not prove that a worthwhile test suite can't be constructed.

> Worse, regression suites are necessarily
> designed to exercise only 100%-reproducible behavior.

That is true, but our current testing methodology (hoping the
beta-testers find the bugs) seems not to be completely satisfactory
either. Which brings us to:

>> I don't think that any test suite is going to eliminate the need for
>> beta-testing.
>
> Precisely...
>
> What I'd like to see is some sort of test mechanism for WAL recovery.
> What I've done sometimes in the past (and recently had to fix the tests
> to re-enable) is to kill -9 a backend immediately after running the
> regression tests, let the system replay the WAL for the tests, and then
> take a pg_dump and compare that to the dump gotten after a conventional
> run.  However this is quite haphazard since (a) the regression tests
> aren't especially designed to exercise all of the WAL logic, and (b)
> pg_dump might not show the effects of some problems, particularly not
> corruption in non-system indexes.  It would be worth the trouble to
> create a more specific test methodology.

Yep. I was thinking about this as an area for possible improvement.
I don't immediately have a brilliant idea how to do it. I did have
the idea of creating a loadable C function which would panic the
database. This could be used to crash the database at a particular
point (even in mid-query, with sufficient creativity).

I think we would certainly need some more powerful way of checking
pass/failure than exact-text comparisons on SQL query results. Being
a perl guy, the first thing that occurs to me is to write some kind of
test harness in perl that can issue SQL queries as well as do other
things, but I don't have an exact design mapped out in my head, and
I'm sure there are other viable approaches.

> In short: merely making the tests bigger doesn't impress me in the
> least.  Focused testing on areas we aren't covering at all could be
> worth the trouble.

Well, I wasn't suggesting adding a lot more testing of things that
we're already testing. I was assuming that we would craft the
additional tests to hit areas that we are not now covering well. My
point here is only to what Peter said upthread: we want to be able to
get positive results rather than waiting for "enough" negative results
(whatever that means). To get positive results, you must have a test
suite. While letting beta testers test whatever they want has some
value, testing things we think might be likely hiding places for bugs
(such as WAL recovery) has merit, too. Making those tests
well-organized and easily repeatable is, IMHO, a Good Thing.

...Robert

andrew(at)dunslane(dot)net (Andrew Dunstan) writes:
> Actually, what I had in mind was getting people to run their
> applications etc. in some non-production environment on the beta. I
> talked to a client today and he said "sure, we have several
> development environments and we can put one or two on the beta and
> then let the developers just do their thing on it." Testing the things
> we know about is in a way less important than making sure nothing else
> got broken.

I've gotten the DB work on one of our applications to the point where
there's a meaningful set of DB tests that can be run in automated
fashion. As a result, I rotate between PG versions periodically; every
couple weeks, I recompile HEAD, and run a test against it to make sure I
don't see any regressions.

It would be insane to think about deploying on some 8.5-alpha version,
but it's nice to have something I can run in ~5 minutes that exercises a
fair bit of at least vaguely realistic functionality.
--
"cbbrowne","@","ca.afilias.info"
Christopher Browne
"Bother," said Pooh, "Eeyore, ready two photon torpedoes and lock
phasers on the Heffalump, Piglet, meet me in transporter room three"

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> Well, I wasn't suggesting adding a lot more testing of things that
> we're already testing. I was assuming that we would craft the
> additional tests to hit areas that we are not now covering well. My
> point here is only to what Peter said upthread: we want to be able to
> get positive results rather than waiting for "enough" negative results
> (whatever that means). To get positive results, you must have a test
> suite. While letting beta testers test whatever they want has some
> value, testing things we think might be likely hiding places for bugs
> (such as WAL recovery) has merit, too. Making those tests
> well-organized and easily repeatable is, IMHO, a Good Thing.

The problem here is the "easily repeatable" bit. Almost by definition,
easily repeatable tests don't find hard-to-reproduce problems. I don't
mean to suggest that they're without value, but they are no substitute
for beta testers doing unpredictable things.

regards, tom lane

On Thu, Aug 27, 2009 at 3:11 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> Precisely...
>
> What I'd like to see is some sort of test mechanism for WAL recovery.
> What I've done sometimes in the past (and recently had to fix the tests
> to re-enable) is to kill -9 a backend immediately after running the
> regression tests, let the system replay the WAL for the tests, and then
> take a pg_dump and compare that to the dump gotten after a conventional
> run.  However this is quite haphazard since (a) the regression tests
> aren't especially designed to exercise all of the WAL logic, and (b)
> pg_dump might not show the effects of some problems, particularly not
> corruption in non-system indexes.  It would be worth the trouble to
> create a more specific test methodology.

What I've been thinking of doing is having the regression suite take a
backup after initdb and set archive mode on. when the regression suite
finishes start the backup up and replay all the WAL.

I'm not sure how to compare the databases since I don't think pg_dump
actually works here -- a lot of the data is dropped by the end of the
test. But at least that would test that replaying WAL didn't cause any
crashes.

--
greg
http://mit.edu/~gsstark/resume.pdf

On Thu, Aug 27, 2009 at 11:29:42AM -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > Well, I wasn't suggesting adding a lot more testing of things that
> > we're already testing. I was assuming that we would craft the
> > additional tests to hit areas that we are not now covering well. My
> > point here is only to what Peter said upthread: we want to be able to
> > get positive results rather than waiting for "enough" negative results
> > (whatever that means). To get positive results, you must have a test
> > suite. While letting beta testers test whatever they want has some
> > value, testing things we think might be likely hiding places for bugs
> > (such as WAL recovery) has merit, too. Making those tests
> > well-organized and easily repeatable is, IMHO, a Good Thing.
>
> The problem here is the "easily repeatable" bit. Almost by definition,
> easily repeatable tests don't find hard-to-reproduce problems. I don't
> mean to suggest that they're without value, but they are no substitute
> for beta testers doing unpredictable things.

I've wondered before about using a system emulator to snapshot the disk
on each write (I'd expect you could put some pretty low level hooks
in with qemu and gdb) and then run each snapshot in another system to
make sure that either the transaction is rolled back or committed as
appropriate. I guess this would take a while to run but may help catch
some obscure bugs. Or this an area that's well tested already?

--
Sam http://samason.me.uk/

On Thu, Aug 27, 2009 at 11:29 AM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> Well, I wasn't suggesting adding a lot more testing of things that
>> we're already testing.  I was assuming that we would craft the
>> additional tests to hit areas that we are not now covering well.  My
>> point here is only to what Peter said upthread: we want to be able to
>> get positive results rather than waiting for "enough" negative results
>> (whatever that means).  To get positive results, you must have a test
>> suite.  While letting beta testers test whatever they want has some
>> value, testing things we think might be likely hiding places for bugs
>> (such as WAL recovery) has merit, too.  Making those tests
>> well-organized and easily repeatable is, IMHO, a Good Thing.
>
> The problem here is the "easily repeatable" bit.  Almost by definition,
> easily repeatable tests don't find hard-to-reproduce problems.  I don't
> mean to suggest that they're without value, but they are no substitute
> for beta testers doing unpredictable things.

I think you're overstating the case, but I don't want to argue the
point, particularly. What I do want to do is find a way to address
the problem described in the last sentence of this email:

http://archives.postgresql.org/pgsql-hackers/2009-08/msg01614.php

Both committers and non-committers are currently suffering from the
fact that there is not a lot of time in the year which is set aside
for development, i.e. neither CommitFest-time nor beta-time. To fix
this problem, we can:

1. Make CommitFests shorter.
2. Make CommitFests less frequent.
3. Continue developing during CommitFests.
4. Make beta cycles shorter.
5. Make beta cycles less frequent (i.e. lengthen the release cycle).
6. Continue developing during beta.

I believe (1) to be completely impractical and (3) to be
self-defeating. I suspect (2) will backfire badly. That doesn't
leave us with a lot of options. We can certainly do (5), but the
downside is that features that get committed won't hit release for a
very long time. I and others have suggested a couple of possible
approaches toward (4) or (6), such as changing the way we do release
notes, adding more regression tests to give us more (not perfect)
confidence that the release is solid, and/or branching the tree during
beta. None of those ideas have gotten a single vote of confidence
from you or Bruce. What's your suggestion?

...Robert

On Thu, Aug 27, 2009 at 10:38 AM, Greg Stark<gsstark(at)mit(dot)edu> wrote:
>
> What I've been thinking of doing is having the regression suite take a
> backup after initdb and set archive mode on. when the regression suite
> finishes start the backup up and replay all the WAL.
>
> I'm not sure how to compare the databases

- execute 60 of the 121 tests (or at least those that create tables
and insert/update/delete the most data)
- crash the server and replay the WAL
- execute the rest of the tests and cross your fingers :)

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Greg Stark <gsstark(at)mit(dot)edu> writes:
> On Thu, Aug 27, 2009 at 3:11 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> ... However this is quite haphazard since (a) the regression tests
>> aren't especially designed to exercise all of the WAL logic, and (b)
>> pg_dump might not show the effects of some problems, particularly not
>> corruption in non-system indexes. It would be worth the trouble to
>> create a more specific test methodology.

> What I've been thinking of doing is having the regression suite take a
> backup after initdb and set archive mode on. when the regression suite
> finishes start the backup up and replay all the WAL.

> I'm not sure how to compare the databases since I don't think pg_dump
> actually works here -- a lot of the data is dropped by the end of the
> test.

Yeah, that's another problem with using the existing tests for this
purpose --- a lot of possibly-useful stuff isn't kept around to the end.
And the desire to keep the test modules independent limits the amount of
interaction between them too. I really think we'd need a bespoke set of
tests to get very far with this.

This reminds me that pg_dump/pg_restore is another large pile of code
that receives no formalized testing whatsoever ...

regards, tom lane

2009/8/27 Jean-Michel Pouré <jm(at)poure(dot)com>:
>
> Otherwise, replicating some MySQL SQL syntax will not work.
>
> As you know, people willing to use PostgreSQL replication are possibly
> already MySQL replication users. So if they test and PostgreSQL fails,
> this is too bad.
>

yeah! but some times the reason MySQL is not failing on those cases is
because it's broken...
Consider this one:
http://archives.postgresql.org/pgsql-general/2005-12/msg00487.php

we don't want to copy the bugs nor the bad designs decisions from
MySQL, no that everything from MySQL is bad but i would be scary if we
start supporting every single piece of code MySQL accepts

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

2009/8/27 Jaime Casanova <jcasanov(at)systemguards(dot)com(dot)ec>:
> 2009/8/27 Jean-Michel Pouré <jm(at)poure(dot)com>:
>>
>> Otherwise, replicating some MySQL SQL syntax will not work.
>>
>> As you know, people willing to use PostgreSQL replication are possibly
>> already MySQL replication users. So if they test and PostgreSQL fails,
>> this is too bad.
>>
>
> yeah! but some times the reason MySQL is not failing on those cases is
> because it's broken...
> Consider this one:
> http://archives.postgresql.org/pgsql-general/2005-12/msg00487.php
>
> we don't want to copy the bugs nor the bad designs decisions from
> MySQL, no that everything from MySQL is bad but i would be scary if we
> start supporting every single piece of code MySQL accepts
>

And that behavior has changed to be sane in 5.0+, iirc.

--
Rob Wultsch
wultsch(at)gmail(dot)com

On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote:
> To get positive results in which you can have confidence, you have to
> know that the testing which was done actually did a reasonably good
> job exercising the code in a way that would have flushed out bugs, had
> any been present. That sounds a lot like the definition of a
> regression test suite. Of course, we have that already, but it's
> nowhere near comprehensive. Maybe we should be looking at an expanded
> test suite that runs on a time scale of hours rather than seconds.
> Actually, didn't Peter talk about something like this at PGCon?

Let's look at it this way: If I were writing a compiler, then I would
have two main test approaches. First, I would have an in-tree test
suite that compiles a bunch of example code snippets and checks that the
results are reasonable. Call that a regression test. It would be
informed by code coverage analysis and previously reported bugs.
Second, as part of my release cycle, I would have an agenda to try to
compile a large set of real programs against my new compiler version.
I would do that during the beta period. You will notice that GCC pretty
much operates that way.

We have regression tests. They could and should be expanded. That's a
developer job, and we can start working on that now. But this
discussion was about what to do during beta. And I think during beta
you want to test PostgreSQL against a large set of real applications.
But we could try to clarify how to actually do that in an organized way.

Now, if you want to improve the regression tests, I would suggest going
through the commits since 8.4beta and since 8.4.0 final release and ask
how these problems could have been prevented or caught earlier. I
suppose a test suite for WAL might be part of the answer, but a closer
analysis might be insightful.

On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
> On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote:
>> To get positive results in which you can have confidence, you have to
>> know that the testing which was done actually did a reasonably good
>> job exercising the code in a way that would have flushed out bugs, had
>> any been present.  That sounds a lot like the definition of a
>> regression test suite.  Of course, we have that already, but it's
>> nowhere near comprehensive.  Maybe we should be looking at an expanded
>> test suite that runs on a time scale of hours rather than seconds.
>> Actually, didn't Peter talk about something like this at PGCon?
>
> Let's look at it this way: If I were writing a compiler, then I would
> have two main test approaches.  First, I would have an in-tree test
> suite that compiles a bunch of example code snippets and checks that the
> results are reasonable.  Call that a regression test.  It would be
> informed by code coverage analysis and previously reported bugs.
> Second, as part of my release cycle, I would have an agenda to try to
> compile a large set of real programs against my new compiler version.
> I would do that during the beta period.  You will notice that GCC pretty
> much operates that way.
>
> We have regression tests.  They could and should be expanded.  That's a
> developer job, and we can start working on that now.  But this
> discussion was about what to do during beta.  And I think during beta
> you want to test PostgreSQL against a large set of real applications.
> But we could try to clarify how to actually do that in an organized way.
>
> Now, if you want to improve the regression tests, I would suggest going
> through the commits since 8.4beta and since 8.4.0 final release and ask
> how these problems could have been prevented or caught earlier.  I
> suppose a test suite for WAL might be part of the answer, but a closer
> analysis might be insightful.

What I want to do is address the concern about too much of any given
year being consumed by beta and CommitFest. I'm not sure I know how
to do that though.

...Robert

2009/8/27 Rob Wultsch <wultsch(at)gmail(dot)com>:
>
> And that behavior has changed to be sane in 5.0+, iirc.
>

5.0.12+ actually... that is stated in the same thread...

the point was that if we simply were saying: hey! mysql can interpret
this, make postgres do the same then we could end up with a lot of
broken stuff... just because mysql users think is wonderful to not
have to write sane code...

--
Atentamente,
Jaime Casanova
Soporte y capacitación de PostgreSQL
Asesoría y desarrollo de sistemas
Guayaquil - Ecuador
Cel. +59387171157

Hi,

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
>> We have regression tests.  They could and should be expanded.  That's a
>> developer job, and we can start working on that now.  But this
>> discussion was about what to do during beta.  And I think during beta
>> you want to test PostgreSQL against a large set of real applications.
>> But we could try to clarify how to actually do that in an organized
>> way.

Exactly, and I think that what we're missing here is a simple tool for
our users to check a new PostgreSQL release against their existing
application.

We already know how to either log all queries and analyze the log files
(CSV makes it easier, pgfouine parses them too) or to have a fe/be
protocol proxy to record application SQL traffic (tsung recorder does
that).

What we miss is a tool to run the captured queries through both versions
of PG and report any resultset mismatch, of course with a way to account
for ordering issues (but we've seen people rely on the ordering when
they don't give an order by clause, then bug the lists about it if a new
release changes it).

> What I want to do is address the concern about too much of any given
> year being consumed by beta and CommitFest. I'm not sure I know how
> to do that though.

To do this I think we *need* to provide beta tester a validator kind of
tool which they can use even without having an application unit test
suite. And a way to easily report success or failure, and in case of
success, a notion of their tests coverage (which is reduced to the list
of PG features the application exercices, but an automated way to
extract the information while running the tool would allow for hackers
friendly categories).

Anyone interrested into starting a project and coding the tool we so
much need?

Regards,
--
dim

PS: of course provided with such a tool, I'd run it for several
applications as soon as possible, which might include every alpha
release.

PPS: I'll be happy to participate into such a project, but I seem to
have a rather full plate already and a shrinking OpenSource free time,
so waiting for me to get there won't help until it's about too late.

On Sun, 2009-08-23 at 01:57 -0400, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > I posted a note about a week ago which drew far less commentary than I
> > expected, regarding the timetable for the release of 8.5.
>
> > http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php
>
> > Perhaps this is already being discussed on -core, but if so the
> > conclusions haven't been shared publicly.
>
> Core hasn't discussed 8.5 schedule since the discussions that Peter
> summarized in the message you cited above. I share your concern that
> "release in time for PGCon" isn't very realistic if we don't get more
> aggressive about schedule. On the other hand, we didn't get all that
> much done in this fest. If we cut back to a three-fest schedule
> I think we may succeed in releasing 8.5 early, but only because there
> is nothing interesting in it :-(. Dunno where the right balance is.

General comment on thread:

The level of detailed planning happening now is a change for the
community and in general I think it's a good thing. In the past we've
always said it will be shipped when it's ready, and now we seem to be
caught by our own rules. There's no need to make hard decisions now.
Let's keep some flexibility in our thinking. If the structures give us
problems, lets change the structures. The idea is the plans help the
developers, not hinder them or make it harder to include big features.

In my view it is important that I have a holistic view of what I am
doing and that means it is very difficult for others to help in a way
that doesn't merely hinder me (or Fujii-san). Speed of coding is not the
issue here, nor is the number of hands on a keyboard.

I don't share the doubts or fears expressed that the two big patches
will not make it into Postgres in this release. We now have more people
experienced in these areas of code than at any other time in our
history. We have almost-complete solutions from experienced developers.
In particular, I have faith in Fujii-san.

I would appreciate it if somebody could send out some messages of calm,
while I/we work. The time for open review will come around soon enough.

Faith and patience. Please. No need for Fawlty Towers re-runs.

--
Simon Riggs www.2ndQuadrant.com

On Thu, Aug 27, 2009 at 08:48:43PM +0100, Simon Riggs wrote:
>
> On Sun, 2009-08-23 at 01:57 -0400, Tom Lane wrote:
> > Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > > I posted a note about a week ago which drew far less commentary
> > > than I expected, regarding the timetable for the release of 8.5.
> >
> > > http://archives.postgresql.org/pgsql-hackers/2009-08/msg01256.php
> >
> > > Perhaps this is already being discussed on -core, but if so the
> > > conclusions haven't been shared publicly.
> >
> > Core hasn't discussed 8.5 schedule since the discussions that
> > Peter summarized in the message you cited above. I share your
> > concern that "release in time for PGCon" isn't very realistic if
> > we don't get more aggressive about schedule. On the other hand,
> > we didn't get all that much done in this fest. If we cut back to
> > a three-fest schedule I think we may succeed in releasing 8.5
> > early, but only because there is nothing interesting in it :-(.
> > Dunno where the right balance is.
>
> General comment on thread:
>
> The level of detailed planning happening now is a change for the
> community and in general I think it's a good thing. In the past
> we've always said it will be shipped when it's ready, and now we
> seem to be caught by our own rules. There's no need to make hard
> decisions now. Let's keep some flexibility in our thinking. If the
> structures give us problems, lets change the structures. The idea is
> the plans help the developers, not hinder them or make it harder to
> include big features.
>
> In my view it is important that I have a holistic view of what I am
> doing and that means it is very difficult for others to help in a
> way that doesn't merely hinder me (or Fujii-san). Speed of coding is
> not the issue here, nor is the number of hands on a keyboard.
>
> I don't share the doubts or fears expressed that the two big patches
> will not make it into Postgres in this release. We now have more
> people experienced in these areas of code than at any other time in
> our history. We have almost-complete solutions from experienced
> developers. In particular, I have faith in Fujii-san.
>
> I would appreciate it if somebody could send out some messages of
> calm, while I/we work. The time for open review will come around
> soon enough.

With all due respect, the time for open review is now. You have
already tried closed development several times, and it each time has
been, more or less, a spectacular failure.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Robert Haas escribió:

> What I want to do is address the concern about too much of any given
> year being consumed by beta and CommitFest. I'm not sure I know how
> to do that though.

How much time were we in beta? I thought most time was spent trying to
get to beta in the first place.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Thu, Aug 27, 2009 at 3:53 PM, David Fetter <david(at)fetter(dot)org> wrote:

> > I would appreciate it if somebody could send out some messages of
> > calm, while I/we work. The time for open review will come around
> > soon enough.
>
> With all due respect, the time for open review is now. You have
> already tried closed development several times, and it each time has
> been, more or less, a spectacular failure.

Unlike Robert and Heikki, I don't see you contributing to or assisting
Simon's work. And, while I may be wrong, I doubt that you assisted in
funding any of Simon's work on hot standby either. As such, it's my opinion
that continuing to criticize him from the sidelines is not only rude, but is
also a bad idea as it relates to his motivation in working on this feature.

--
Jonah H. Harris, Senior DBA
myYearbook.com

On Thu, Aug 27, 2009 at 3:56 PM, Alvaro
Herrera<alvherre(at)commandprompt(dot)com> wrote:
> Robert Haas escribió:
>> What I want to do is address the concern about too much of any given
>> year being consumed by beta and CommitFest.  I'm not sure I know how
>> to do that though.
>
> How much time were we in beta?  I thought most time was spent trying to
> get to beta in the first place.

[ looks ]

The final CommitFest began November 11, 2008. It closed March 25,
2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days).
8.4.0 was released July 1, 2009 (+ 77 days). The first CommitFest for
8.5 began on July 15, 2009 (+ 14 days).

http://www.postgresql.org/about/news.1074
http://wiki.postgresql.org/index.php?title=CommitFest_2008-11&action=history
http://www.postgresql.org/docs/8.4/static/release-8-4.html

In total, the tree was closed for 256 days, or 8.5 months, of which
the final CommitFest accounted for approximately 56%. Had we closed
the final CommitFest in 30 days rather than 144 days, and had
everything else taken the same amount of time, the release would have
occurred on March 9th and the first CommitFest for 8.5 would have
started on March 23rd.

Hmm... maybe that's not actually that bad. If we stuck to a similar
schedule for 8.5, but with a timely last CF, then we'd have either (3
CF):

2009-11-15 Final CommitFest Begins
2009-12-15 Final CommitFest Ends
2010-01-05 Beta
2010-03-23 Release
2010-04-06 First CommitFest for 8.6 Begins

Or (4 CF):

2010-01-15 Final CommitFest Begins
2010-02-15 Final CommitFest Ends
2010-03-08 Beta
2010-05-24 Release
2010-06-07 First CommitFest for 8.6 Begins

Of course I don't think we'd actually need to start a CommitFest quite
as quickly as we did this time, because with a shorter release cycle
there ought to be a lot less patches already accumulated by the time
we release, especially if there are clearly defined tasks for
developers to do during the beta period. On the other hand, 8.4beta
was arguably too short, since we missed some serious problems, so the
picture above may be a bit too rosy.

...Robert

On Thu, Aug 27, 2009 at 04:22:58PM -0400, Jonah H. Harris wrote:
> On Thu, Aug 27, 2009 at 3:53 PM, David Fetter <david(at)fetter(dot)org> wrote:
>
> > > I would appreciate it if somebody could send out some messages
> > > of calm, while I/we work. The time for open review will come
> > > around soon enough.
> >
> > With all due respect, the time for open review is now. You have
> > already tried closed development several times, and it each time
> > has been, more or less, a spectacular failure.
>
> Unlike Robert and Heikki, I don't see you contributing to or
> assisting Simon's work.

My assistance is of the form of actually getting it done. Simon's
work is absolutely fantastic, but only when the rest of the community
can actually help. When it shows up late, it actually hurts
everybody, most of all Simon.

> And, while I may be wrong, I doubt that you assisted in funding any
> of Simon's work on hot standby either. As such, it's my opinion
> that continuing to criticize him from the sidelines is not only
> rude, but is also a bad idea as it relates to his motivation in
> working on this feature.

If, "your past strategy has a track record of failure, go with a new
strategy, one pretty much universally adopted in PostgreSQL," will
demotivate someone, I can't help that, and I doubt it's actually true.
I'm trying to help here, and encouraging a failed strategy is not
helping.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> The final CommitFest began November 11, 2008. It closed March 25,
> 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days).

I'm not entirely clear on what was happening during the 21 days
between the end of the CommitFest and and the release of beta1. I
seem to remember Bruce saying that there were bugs being fixed, and
that it didn't make sense to release a beta with known bugs of that
magnitude, but I'm not clear on what was up with that. Did we close
the CF with known bugs open, or were these missed in the CF and found
after? Surely it shouldn't normally take three weeks to get a beta
test version to the public after we close the last CF?

Just looking for where we could pick up a few weeks more of
development in each year....

-Kevin

Robert Haas escribió:

> Of course I don't think we'd actually need to start a CommitFest quite
> as quickly as we did this time, because with a shorter release cycle
> there ought to be a lot less patches already accumulated by the time
> we release, especially if there are clearly defined tasks for
> developers to do during the beta period. On the other hand, 8.4beta
> was arguably too short, since we missed some serious problems, so the
> picture above may be a bit too rosy.

Maybe what this says is that we need to get a "pre-beta" release out as
early as possible, just after finalizing the last commitfest, and before
the "open items" and Bruce-approved release note writing are sorted out.
Probably the last alpha release will fill that role. Sadly, the greek
folk did not consider having a letter between alpha and beta :-(

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

"Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> The final CommitFest began November 11, 2008. It closed March 25,
>> 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days).

> I'm not entirely clear on what was happening during the 21 days
> between the end of the CommitFest and and the release of beta1.

Release note drafting is one part of it, but mostly it's "loose end
cleanup". Historically there have always been a pile of loose ends
to be dealt with, and the CommitFest structure doesn't really do
anything to avoid that. If you're interested, attached are all the
commits between commitfest closure (which I announced immediately
after committing the addition of contrib/btree_gin) and stamping
beta1 (which was actually several days before the date Robert gives,
because of the need for the packagers to do their thing).

It appears to me that release notes weren't actually the bottleneck this
time, though they have been in some prior cycles. Bruce committed a
first draft immediately after the commitfest closed. Rather, it was
arguing about things like \df behavior and cardinality() that took up
the time.

We could certainly have released a perhaps-less-polished beta earlier.
I think that the traditional criterion is that we don't release beta1
as long as there are any known issues that might force an initdb.
We were successful in avoiding a post-beta initdb this time, although
IIRC the majority of release cycles have had one --- so maybe you
could argue that that's not so important. It would certainly be
less important if we had working pg_migrator functionality to ease
the pain of going from beta to final.

Now that we have the alpha-release mechanism defined, some of the
pressure for a quick beta could be taken off by releasing a final
alpha right after the final commitfest closes.

regards, tom lane

2009-04-09 20:20 scrappy

* configure, configure.in, doc/bug.template,
src/include/pg_config.h.win32:
commit and tag beta1

2009-04-09 19:22 tgl

* doc/src/sgml/release.sgml: Update release notes through
yesterday; some minor wordsmithing.

2009-04-09 18:32 momjian

* doc/src/sgml/monitoring.sgml: Clarify documentation references to
pg_stat_get_blocks_fetched and pg_stat_get_blocks_hit, per
suggestion from Robert Haas.

2009-04-09 17:50 momjian

* src/tools/RELEASE_CHANGES: No more need to update FAQs.

2009-04-09 17:35 petere

* src/tools/RELEASE_CHANGES: Add URL for config.guess/sub updates

2009-04-09 17:33 petere

* config/: config.guess, config.sub: Update config.guess and
config.sub

2009-04-09 16:50 tgl

* src/timezone/data/: africa, asia, leapseconds, northamerica,
southamerica (REL8_1_STABLE), africa, asia, leapseconds,
northamerica, southamerica (REL8_3_STABLE), africa, asia,
leapseconds, northamerica, southamerica (REL8_0_STABLE), africa,
asia, leapseconds, northamerica, southamerica (REL8_2_STABLE),
africa, asia, leapseconds, northamerica, southamerica: Update time
zone data files to tzdata release 2009e: DST law changes in
Argentina/San_Luis, Cuba, Jordan (historical correction only),
Morocco, Palestine, Syria, Tunisia.

2009-04-09 15:38 petere

* src/: backend/nls.mk, backend/po/de.po, backend/po/es.po,
backend/po/fr.po, backend/po/ja.po, backend/po/nl.po,
backend/po/ru.po, backend/po/tr.po, bin/initdb/nls.mk,
bin/initdb/po/de.po, bin/initdb/po/es.po, bin/initdb/po/fr.po,
bin/initdb/po/ja.po, bin/initdb/po/ru.po, bin/pg_config/nls.mk,
bin/pg_config/po/de.po, bin/pg_config/po/fr.po,
bin/pg_config/po/ja.po, bin/pg_config/po/ru.po,
bin/pg_config/po/tr.po, bin/pg_controldata/nls.mk,
bin/pg_controldata/po/de.po, bin/pg_controldata/po/fr.po,
bin/pg_controldata/po/ja.po, bin/pg_ctl/nls.mk,
bin/pg_ctl/po/cs.po, bin/pg_ctl/po/de.po, bin/pg_ctl/po/fr.po,
bin/pg_ctl/po/ja.po, bin/pg_ctl/po/ru.po, bin/pg_dump/nls.mk,
bin/pg_dump/po/de.po, bin/pg_dump/po/fr.po, bin/pg_dump/po/ja.po,
bin/pg_dump/po/tr.po, bin/pg_resetxlog/nls.mk,
bin/pg_resetxlog/po/de.po, bin/pg_resetxlog/po/fr.po,
bin/pg_resetxlog/po/ja.po, bin/pg_resetxlog/po/ru.po,
bin/psql/nls.mk, bin/psql/po/de.po, bin/psql/po/es.po,
bin/psql/po/fr.po, bin/psql/po/ja.po, bin/psql/po/tr.po,
bin/scripts/nls.mk, bin/scripts/po/de.po, bin/scripts/po/fr.po,
bin/scripts/po/ja.po, interfaces/ecpg/ecpglib/nls.mk,
interfaces/ecpg/ecpglib/po/de.po, interfaces/ecpg/ecpglib/po/es.po,
interfaces/ecpg/ecpglib/po/fr.po, interfaces/ecpg/preproc/nls.mk,
interfaces/ecpg/preproc/po/de.po, interfaces/ecpg/preproc/po/es.po,
interfaces/ecpg/preproc/po/fr.po, interfaces/libpq/nls.mk,
interfaces/libpq/po/de.po, interfaces/libpq/po/fr.po,
interfaces/libpq/po/ja.po, pl/plperl/nls.mk, pl/plperl/po/de.po,
pl/plperl/po/es.po, pl/plperl/po/fr.po, pl/plpgsql/src/nls.mk,
pl/plpgsql/src/po/de.po, pl/plpgsql/src/po/es.po,
pl/plpgsql/src/po/fr.po, pl/plpgsql/src/po/ja.po,
pl/plpgsql/src/po/ro.po, pl/plpython/nls.mk, pl/plpython/po/de.po,
pl/plpython/po/es.po, pl/plpython/po/fr.po, pl/tcl/nls.mk,
pl/tcl/po/de.po, pl/tcl/po/es.po, pl/tcl/po/fr.po: Translation
updates for 8.4 beta

2009-04-09 15:07 tgl

* doc/src/sgml/gin.sgml: Update GIN limitations documentation to
match current reality.

2009-04-09 13:39 tgl

* doc/src/sgml/func.sgml, doc/src/sgml/release.sgml,
src/backend/utils/adt/arrayfuncs.c,
src/include/catalog/catversion.h, src/include/catalog/pg_proc.h,
src/include/utils/array.h, src/test/regress/expected/arrays.out,
src/test/regress/sql/arrays.sql: Remove SQL-compatibility function
cardinality(). It is not exactly clear how this ought to behave
for multi-dimensional arrays. Per discussion, not having it at all
seems better than having it with what might prove to be the wrong
behavior. We can always add it later when we have consensus on the
correct behavior.

2009-04-09 12:20 momjian

* doc/src/sgml/wal.sgml: Improve documentation about how checkpoint
spreads I/O activity.

2009-04-09 10:21 tgl

* src/backend/utils/misc/guc-file.l: Treat EOF like \n for
line-counting purposes in ParseConfigFile, per bug #4752. Fujii
Masao

2009-04-08 22:57 tgl

* src/: pl/plpgsql/src/pl_exec.c, pl/plpgsql/src/plpgsql.h,
test/regress/expected/plpgsql.out, test/regress/sql/plpgsql.sql:
Fix the plpgsql memory leak exhibited in bug #4677. That leak was
introduced by my patch of 2007-01-28 to use per-subtransaction
ExprContexts/EStates: since we re-prepared any expression tree when
the current subtransaction ID changed, we'd accumulate more and
more leaked expression state trees in the outermost subtransaction
if the same function was executed at multiple levels of
subtransaction nesting. To fix, go back to the previous scheme
where there was only one EState per transaction for simple plpgsql
expressions. We really only need an ExprContext per
subtransaction, not a whole EState, so it's possible to keep
prepared expression state trees in the one EState throughout the
transaction. This should be more efficient as well as not leaking
memory for cases involving lots of subtransactions.

The added regression test is the case that inspired the 2007-01-28
patch in the first place, just to make sure we didn't go backwards.
The current memory leak complaint is unfortunately hard to test
for in the regression test framework, though manual testing shows
it's fixed.

Although this is a pre-existing bug, I'm not back-patching because
I'd like to see this method get some field testing first. Consider
back-patching if it gets through 8.4beta unscathed.

2009-04-08 18:29 tgl

* doc/src/sgml/ref/psql-ref.sgml, src/bin/psql/describe.c: Remove
psql's ancient hack that suppressed functions taking or returning
cstring from the output of \df. Now that the default behavior is
to exclude all system functions, the de-cluttering rationale for
this behavior seems pretty weak; and it was always quite
confusing/unhelpful if you were actually looking for I/O functions.
(Not to mention if you were looking for encoding converters or
other cases that might take or return cstring.)

2009-04-08 18:08 tgl

* src/: backend/utils/adt/numeric.c,
test/regress/expected/numeric.out: Allow leading and trailing
spaces around NaN in numeric_in.

Sam Mason, rewritten a bit by Tom.

2009-04-08 17:51 petere

* src/: backend/executor/execQual.c, backend/utils/adt/xml.c,
include/nodes/execnodes.h, test/regress/expected/xml.out,
test/regress/expected/xml_1.out, test/regress/sql/xml.sql:
XMLATTRIBUTES() should send the attribute values through
map_sql_value_to_xml_value() instead of directly through the data
type output function. This is per SQL standard, and consistent
with XMLELEMENT().

2009-04-08 15:02 heikki

* src/bin/pg_dump/: pg_dump.c, pg_dumpall.c: Quote string literals
correctly in the new CREATE SERVER statements and binary upgrade
UPDATE statements.

2009-04-08 09:08 heikki

* src/backend/utils/init/postinit.c: Oops, mustn't call
textdomain() when compiling without --enable-nls

2009-04-08 05:50 heikki

* src/: backend/utils/init/miscinit.c,
backend/utils/init/postinit.c, backend/utils/mb/mbutils.c,
include/mb/pg_wchar.h: Tell gettext which codeset to use by calling
bind_textdomain_codeset(). We already did that on Windows, but it's
needed on other platforms too when LC_CTYPE=C. With other locales,
we enforce (or trust) that the codeset of the locale matches the
server encoding so we don't need to bind it explicitly. It should
do no harm in that case either, but I don't have full faith in the
PG encoding -> OS codeset mapping table yet. Per recent discussion
on pgsql-hackers.

2009-04-08 00:05 momjian

* src/bin/psql/tab-complete.c: Improve tab completion for \ef.

Andrew Gierth

2009-04-07 19:27 momjian

* src/backend/utils/misc/guc.c: Revert addition of units to GUC
descriptions; doesn't affect postgresql.conf.

2009-04-07 18:48 momjian

* configure, configure.in: Disable effective_io_concurrency on
Solaris because posix_fadvise() is no-op on that platform.

2009-04-07 18:22 momjian

* src/backend/utils/misc/: guc.c, postgresql.conf.sample: More GUC
units doc updates.

Euler Taveira de Oliveira

2009-04-07 17:30 momjian

* doc/src/sgml/release.sgml: Add attribution for:

Add Japanese message translations (Japan PostgreSQL Users
Group)

2009-04-07 17:29 momjian

* doc/src/sgml/release.sgml: Add release note item:

Add Japanese message translations

2009-04-07 17:20 momjian

* doc/: FAQ, FAQ_DEV: Remove FAQ and FAQ_DEV ASCII and HTML files
from CVS; now on the wiki.

Per-language files kept for transator usage.

2009-04-07 15:35 mha

* src/tools/msvc/Mkvcbuild.pm: Support Perl 5.10 and TCL 8.5 in
MSVC builds.

We should probably have a better way to do this (meaning something
not hardcoded) eventually, but this fixes the problem for 8.4.

Dave Page

2009-04-07 14:10 tgl

* contrib/pg_freespacemap/: pg_freespacemap.c (REL8_3_STABLE),
pg_freespacemap.c (REL8_2_STABLE): Fix contrib/pg_freespacemap's
underestimate of the number of pages it could find in the FSM. Per
report from Dimitri Fontaine and Andrew Gierth.

(Affects only 8.2 and 8.3 since HEAD no longer has MaxFSMPages at
all.)

2009-04-07 13:57 tgl

* contrib/pg_freespacemap/pg_freespacemap.c: Remove useless
(leftover?) extern declaration.

2009-04-07 13:49 teodor

* src/backend/access/gist/gist.c (REL7_4_STABLE): Fix 'all at one
page bug' in picksplit method of R-tree emulation. Add defense from
buggy user-defined picksplit to GiST.

2009-04-07 13:46 teodor

* src/backend/access/gist/: gistproc.c, gistutil.c (REL8_1_STABLE),
gist.c (REL8_0_STABLE): Fix 'all at one page bug' in picksplit
method of R-tree emulation. Add defense from buggy user-defined
picksplit to GiST.

2009-04-07 11:53 tgl

* contrib/fuzzystrmatch/: fuzzystrmatch.h (REL7_4_STABLE),
fuzzystrmatch.h (REL8_1_STABLE), fuzzystrmatch.h (REL8_3_STABLE),
fuzzystrmatch.h (REL8_0_STABLE), fuzzystrmatch.h (REL8_2_STABLE),
fuzzystrmatch.c: Defend against non-ASCII letters in fuzzystrmatch
code. The functions still don't behave very sanely for multibyte
encodings, but at least they won't be indexing off the ends of
static arrays.

2009-04-07 00:02 momjian

* doc/src/sgml/trigger.sgml: Add doc link to section about how to
compile triggers.

2009-04-06 20:31 tgl

* doc/src/sgml/backup.sgml, doc/src/sgml/func.sgml,
src/backend/access/transam/xlog.c,
src/backend/catalog/system_views.sql,
src/include/catalog/catversion.h, src/include/catalog/pg_proc.h:
Add an optional parameter to pg_start_backup() that specifies
whether to do the checkpoint in immediate or lazy mode. This is to
address complaints that pg_start_backup() takes a long time even
when there's no need to minimize its I/O consumption.

2009-04-06 17:00 momjian

* src/backend/utils/misc/: guc.c, postgresql.conf.sample: Add unit
documentation for various postgresql.conf settings.

2009-04-06 15:34 petere

* src/backend/utils/mb/mbutils.c: Add entry in the encoding number
to OS name table for KOI8-U.

2009-04-06 15:03 momjian

* src/backend/utils/misc/postgresql.conf.sample: Properly align
equals signs in new postgresql.conf units comments.

2009-04-06 15:00 momjian

* src/backend/utils/misc/postgresql.conf.sample: Document in
postgresql.conf that the default units for
log_min_duration_statement is milliseconds.

2009-04-06 14:40 momjian

* src/backend/utils/misc/postgresql.conf.sample: Display
postgresql.conf unit options in an easier-to-understand, 2-column
format.

2009-04-06 13:56 momjian

* doc/src/sgml/maintenance.sgml: Doc change in new patch,
stand-alone -> standalone

2009-04-06 13:55 momjian

* doc/src/sgml/maintenance.sgml: Add documentation mention of
'check_postgres.pl' in Routine Database Maintenance Tasks section.

2009-04-06 11:50 momjian

* src/bin/psql/tab-complete.c: Adjust psql tab completion for new
\d 'S' flag behavior; adjust code to be more flexible about
additional modifiers for \d commands.

2009-04-06 11:43 tgl

* doc/src/sgml/fuzzystrmatch.sgml: Document the fact that
fuzzystrmatch doesn't work in multibyte encodings.

2009-04-06 11:01 tgl

* doc/src/sgml/keywords.sgml: Correct keywords table for status of
COLLATE vs LC_COLLATE.

2009-04-06 10:47 teodor

* src/backend/access/gist/: gistproc.c, gistsplit.c
(REL8_2_STABLE): Fix 'all at one page bug' in picksplit method of
R-tree emulation. Add defense from buggy user-defined picksplit to
GiST.

2009-04-06 10:39 teodor

* src/backend/access/gist/: gistproc.c, gistsplit.c
(REL8_3_STABLE): Fix 'all at one page bug' in picksplit method of
R-tree emulation. Add defense from buggy user-defined picksplit to
GiST.

2009-04-06 10:27 teodor

* src/backend/access/gist/: gistproc.c, gistsplit.c: Fix 'all at
one page bug' in picksplit method of R-tree emulation. Add defense
from buggy user-defined picksplit to GiST.

2009-04-06 04:42 heikki

* doc/src/sgml/charset.sgml, doc/src/sgml/keywords.sgml,
doc/src/sgml/ref/create_database.sgml,
src/backend/commands/dbcommands.c, src/backend/parser/gram.y,
src/bin/pg_dump/pg_dump.c, src/bin/pg_dump/pg_dumpall.c,
src/bin/scripts/createdb.c, src/include/parser/kwlist.h,
src/interfaces/ecpg/preproc/ecpg.trailer: Rename the new CREATE
DATABASE options to set collation and ctype into LC_COLLATE and
LC_CTYPE, per discussion on pgsql-hackers.

2009-04-05 18:28 tgl

* src/: backend/utils/adt/arrayfuncs.c,
include/catalog/catversion.h, include/catalog/pg_proc.h,
include/utils/array.h: Change cardinality() into a C-code function,
instead of a SQL-language alias for array_length(v,1). The
efficiency gain here is doubtless negligible --- what I'm
interested in is making sure that if we have second thoughts about
the definition, we will not have to force a post-beta initdb to
change the implementation.

2009-04-05 16:32 tgl

* src/backend/executor/execQual.c: Make ExecInitExpr build the list
of SubPlans found in a plan tree in order of discovery, rather than
reverse order. This doesn't matter functionally (I suppose the
previous coding dates from the time when lcons was markedly cheaper
than lappend). However now that EXPLAIN is labeling subplans with
IDs that are based on order of creation, this may help produce a
slightly less surprising printout.

2009-04-05 15:59 tgl

* src/: backend/commands/explain.c, backend/nodes/copyfuncs.c,
backend/nodes/equalfuncs.c, backend/nodes/outfuncs.c,
backend/optimizer/plan/subselect.c, backend/utils/adt/ruleutils.c,
include/nodes/primnodes.h: Change EXPLAIN output so that subplans
and initplans (particularly CTEs) are individually labeled, rather
than just grouped under an "InitPlan" or "SubPlan" heading. This
in turn makes it possible for decompilation of a subplan reference
to usefully identify which subplan it's referencing. I also made
InitPlans identify which parameter symbol(s) they compute, so that
references to those parameters elsewhere in the plan tree can be
connected to the initplan that will be executed. Per a gripe from
Robert Haas about EXPLAIN output of a WITH query being inadequate,
plus some longstanding pet peeves of my own.

2009-04-05 07:32 teodor

* src/backend/access/gin/: ginget.c, ginscan.c: Fix infinite loop
while checking of partial match in pending list. Improve comments.
Now GIN-indexable operators should be strict. Per Tom's
questions/suggestions.

2009-04-05 00:19 tgl

* src/: backend/postmaster/postmaster.c, bin/initdb/initdb.c,
bin/pg_ctl/pg_ctl.c, bin/pg_dump/pg_dump.c,
bin/pg_dump/pg_dumpall.c, bin/pg_dump/pg_restore.c,
bin/psql/startup.c, bin/scripts/common.c, bin/scripts/common.h,
include/getopt_long.h, port/getopt_long.c: Remove a boatload of
useless definitions of 'int optreset'. If we are using our own
ports of getopt or getopt_long, those will define the variable for
themselves; and if not, we don't need these, because we never touch
the variable anyway.

2009-04-05 00:09 tgl

* src/include/pg_config.h.win32: I had always wondered why
pg_config.h.win32 claimed that Windows provides optreset. Current
mastodon results prove that in fact it does not; it was only
because getopt.c defined the variable anyway that things failed to
fall over.

2009-04-04 20:40 tgl

* contrib/intarray/_int.sql.in,
contrib/intarray/uninstall__int.sql, doc/src/sgml/intarray.sgml:
Remove contrib/intarray's definitions of the <@ and @> operators,
so that they don't cause confusion with the built-in anyarray
versions of those operators. Adjust the module's index opclasses
to support the built-in operators in place of the private ones.

The private implementations are still available under their
historical names @ and ~, so no functionality is lost. Some quick
testing suggests that they offer no real benefit over the core
operators, however.

Per a complaint from Rusty Conover.

2009-04-04 18:36 tgl

* src/port/getopt.c: Hmm, baiji thinks we need explicit 'extern'
here.

2009-04-04 17:55 tgl

* configure, configure.in, src/include/pg_config.h.in,
src/port/getopt.c: Make an attempt at fixing our current Solaris 11
breakage: add a configure probe for opterr (exactly like the one
for optreset) and have getopt.c define the variables only if
configure doesn't find them in libc.

2009-04-04 17:12 tgl

* src/: backend/access/common/reloptions.c,
backend/commands/define.c, backend/commands/foreigncmds.c,
backend/commands/sequence.c, backend/commands/typecmds.c,
backend/commands/view.c, backend/nodes/copyfuncs.c,
backend/nodes/equalfuncs.c, backend/nodes/makefuncs.c,
backend/nodes/outfuncs.c, backend/parser/gram.y,
backend/parser/parse_clause.c, include/commands/defrem.h,
include/foreign/foreign.h, include/nodes/makefuncs.h,
include/nodes/nodes.h, include/nodes/parsenodes.h: Remove the
recently added node types ReloptElem and OptionDefElem in favor of
adding optional namespace and action fields to DefElem. Having
three node types that do essentially the same thing bloats the code
and leads to errors of confusion, such as in yesterday's bug report
from Khee Chin.

2009-04-04 13:40 tgl

* src/: backend/commands/indexcmds.c,
backend/storage/ipc/procarray.c, include/storage/lock.h,
include/storage/procarray.h: A session that does not have any live
snapshots does not have to be waited for when we are waiting for
old snapshots to go away during a concurrent index build. In
particular, this rule lets us avoid waiting for idle-in-transaction
sessions.

This logic could be improved further if we had some way to wake up
when the session we are currently waiting for goes
idle-in-transaction. However that would be a significantly more
complex/invasive patch, so it'll have to wait for some other day.

Simon Riggs, with some improvements by Tom.

2009-04-04 00:53 tgl

* src/: backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql
(REL7_4_STABLE), backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql
(REL8_1_STABLE), backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql
(REL8_3_STABLE), backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql
(REL8_0_STABLE), backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql
(REL8_2_STABLE), backend/utils/adt/timestamp.c,
test/regress/expected/interval.out, test/regress/sql/interval.sql:
Rewrite interval_hash() so that the hashcodes are equal for values
that interval_eq() considers equal. I'm not sure how that
fundamental requirement escaped us through multiple revisions of
this hash function, but there it is; it's been wrong since
interval_hash was first written for PG 7.1. Per bug #4748 from
Roman Kononov.

Backpatch to all supported releases.

This patch changes the contents of hash indexes for interval
columns. That's no particular problem for PG 8.4, since we've
broken on-disk compatibility of hash indexes already; but it will
require a migration warning note in the next minor releases of all
existing branches: "if you have any hash indexes on columns of type
interval, REINDEX them after updating".

2009-04-03 20:45 alvherre

* src/: backend/access/common/reloptions.c,
include/access/reloptions.h: Disallow setting fillfactor for TOAST
tables.

To implement this without almost duplicating the reloption table,
treat relopt_kind as a bitmask instead of an integer value. This
decreases the range of allowed values, but it's not clear that
there's need for that much values anyway.

This patch also makes heap_reloptions explicitly a no-op for
relation kinds other than heap and TOAST tables.

Patch by ITAGAKI Takahiro with minor edits from me. (In particular
I removed the bit about adding relation kind to an error message,
which I intend to commit separately.)

2009-04-03 20:44 tgl

* src/bin/psql/describe.c: Improve obsolete comment.

2009-04-03 20:41 tgl

* src/bin/psql/describe.c: Make \dt \di and friends more consistent
about the treatment of TOAST tables and indexes; to wit, never show
either. (You can examine them with plain \d if you're really so
inclined.)

2009-04-03 20:39 tgl

* doc/src/sgml/ref/psql-ref.sgml: Minor wordsmithing on
descriptions of some \d commands.

2009-04-03 19:38 tgl

* src/bin/psql/help.c: Add missing help output for \ef option.
Andrew Gierth

2009-04-03 19:27 tgl

* src/port/: path.c (REL8_3_STABLE), path.c (REL8_2_STABLE),
path.c: Use (unsigned char) cast in argument of pg_tolower().
Maybe it works on Windows without that, but we shouldn't put bad
examples where people might copy them. Also, reformat slightly to
improve the odds that pgindent won't go nuts on this.

2009-04-03 14:17 tgl

* src/backend/storage/buffer/bufmgr.c: Add a comment documenting
the question of whether PrefetchBuffer should try to protect an
already-existing buffer from being evicted. This was left as an
open issue when the posix_fadvise patch was committed. I'm not
sure there's any evidence to justify more work in this area, but we
should have some record about it in the source code.

2009-04-03 12:59 tgl

* src/pl/plpython/: plpython.c, expected/plpython_function.out,
expected/plpython_test.out, sql/plpython_function.sql,
sql/plpython_test.sql: Defend against possible crash if a plpython
function does not specify names for its arguments. Also add a
regression test, since someone apparently changed every single
plpython test case to use only named parameters; else we'd have
noticed this sooner.

Euler Taveira de Oliveira, per a report from Alvaro

2009-04-03 07:52 mha

* src/port/: path.c (REL8_3_STABLE), path.c (REL8_2_STABLE),
path.c: Make directory name comparisons on Win32 case insensitive.

This method will not catch all different ways since the locale
handling in NTFS doesn't provide an easy way to do that, but it
will hopefully solve the most common cases causing startup problems
when the backend is found in the system PATH.

Attempts to fix bug #4694.

2009-04-02 18:44 momjian

* doc/src/sgml/config.sgml: Document that Solaris can't use
effective_io_concurrency because of an ineffective posix_fadvise().

2009-04-02 18:39 tgl

* src/: backend/executor/execQual.c, backend/executor/execUtils.c,
include/nodes/execnodes.h: Refactor ExecProject and associated
routines so that fast-path code is used for simple Var targetlist
entries all the time, even when there are other entries that are
not simple Vars. Also, ensure that we prefetch attributes (with
slot_getsomeattrs) for all Vars in the targetlist, even those
buried within expressions. In combination these changes seem to
significantly reduce the runtime for cases where tlists are mostly
but not exclusively Vars. Per my proposal of yesterday.

2009-04-02 16:59 momjian

* src/backend/: access/transam/slru.c, executor/execScan.c,
executor/nodeAgg.c, executor/nodeGroup.c, executor/nodeHash.c,
executor/nodeHashjoin.c, executor/nodeLimit.c,
executor/nodeMaterial.c, executor/nodeMergejoin.c,
executor/nodeNestloop.c, executor/nodeSetOp.c, executor/nodeSort.c,
executor/nodeSubplan.c, executor/nodeUnique.c, utils/probes.d:
Revert DTrace patch from Robert Lor

2009-04-02 16:16 tgl

* src/pl/plpgsql/src/pl_exec.c: Minor code
beautification/consolidation.

2009-04-02 15:57 momjian

* src/backend/utils/misc/guc.c: Give a better error message when
trying to change "effective_io_concurrency" on systems without
posix_fadvise().

2009-04-02 15:20 momjian

* doc/src/sgml/plpgsql.sgml, src/pl/plpgsql/src/pl_exec.c: Have
PL/pgSQL FETCH set DIAGNOSTICS ROW_COUNT.

Andrew Gierth

2009-04-02 15:14 momjian

* src/backend/: access/transam/slru.c, executor/execScan.c,
executor/nodeAgg.c, executor/nodeGroup.c, executor/nodeHash.c,
executor/nodeHashjoin.c, executor/nodeLimit.c,
executor/nodeMaterial.c, executor/nodeMergejoin.c,
executor/nodeNestloop.c, executor/nodeSetOp.c, executor/nodeSort.c,
executor/nodeSubplan.c, executor/nodeUnique.c, utils/probes.d: Add
support for additional DTrace probes.

Robert Lor

2009-04-02 14:06 teodor

* contrib/hstore/hstore_io.c (REL8_2_STABLE): Fix memory allocation
for output of hstore type. Per "maosen.zhang"
<maosen(dot)zhang(at)alibaba-inc(dot)com> report.

2009-04-02 13:57 teodor

* contrib/hstore/: hstore_io.c (REL8_3_STABLE), hstore_io.c: Fix
memory allocation for output of hstore type. Per "maosen.zhang"
<maosen(dot)zhang(at)alibaba-inc(dot)com> report.

2009-04-02 13:38 momjian

* src/bin/psql/describe.c: Do not show information_schema in \d*
commands, unless 'S' or pattern is specified.

Martin Pihlak

2009-04-02 13:30 tgl

* src/: backend/utils/mb/mbutils.c, include/mb/pg_wchar.h: Fix
SetClientEncoding() to maintain a cache of previously selected
encoding conversion functions. This allows transaction rollback to
revert to a previous client_encoding setting without doing fresh
catalog lookups. I believe that this explains and fixes the recent
report of "failed to commit client_encoding" failures.

This bug is present in 8.3.x, but it doesn't seem prudent to
back-patch the fix, at least not till it's had some time for field
testing in HEAD.

In passing, remove SetDefaultClientEncoding(), which was used
nowhere.

2009-04-02 11:15 momjian

* doc/src/sgml/ref/psql-ref.sgml, src/bin/psql/describe.c: Change
psql \d* display so 'S' _or_ a pattern include system objects.

2009-04-01 23:51 tgl

* src/backend/utils/misc/: guc.c (REL8_3_STABLE), guc.c: Fix GUC's
reports of assign_hook failure to always include the parameter
value we failed to assign, even in "can't happen" cases. Motivated
by wondering what's going on in a recent trouble report where
"failed to commit" did happen.

2009-04-01 21:16 tgl

* src/pl/plpgsql/src/: pl_exec.c (REL8_3_STABLE), pl_exec.c
(REL8_2_STABLE), pl_exec.c: plpgsql's exec_simple_cast_value()
mistakenly supposed that it could bypass casting effort whenever
the input value was NULL. However this prevents application of
not-null domain constraints in the cases that use this function, as
illustrated in bug #4741. Since this function isn't meant for use
in performance-critical paths anyway, this certainly seems like
another case of "premature optimization is the root of all evil".

Back-patch as far as 8.2; older versions made no effort to enforce
domain constraints here anyway.

2009-04-01 14:54 tgl

* src/bin/pg_dump/: pg_dump.c (REL8_3_STABLE), pg_dump.c: Improve
pg_dump's query for retrieving BLOB comments to be more efficient
when there are many blobs and not so many comments. Tamas Vincze

2009-04-01 05:17 heikki

* src/backend/utils/adt/pg_locale.c: Update comment to reflect that
LC_COLLATE and LC_CTYPE are now per-database settings.

2009-03-31 23:32 tgl

* doc/src/sgml/func.sgml: Index some array functions, per Mario
Splivalo.

2009-03-31 23:23 tgl

* src/backend/libpq/: auth.c, pg_hba.conf.sample: Remove last
references to the crypt auth method, per Andreas Scherbaum.

2009-03-31 18:54 tgl

* contrib/: pgstattuple/pgstattuple.c (REL7_4_STABLE),
pgstattuple/pgstattuple.c (REL8_1_STABLE),
pageinspect/btreefuncs.c, pageinspect/rawpage.c,
pgstattuple/pgstatindex.c, pgstattuple/pgstattuple.c
(REL8_3_STABLE), pgstattuple/pgstattuple.c (REL8_0_STABLE),
pgstattuple/pgstatindex.c, pgstattuple/pgstattuple.c
(REL8_2_STABLE), pageinspect/btreefuncs.c, pageinspect/rawpage.c,
pgstattuple/pgstatindex.c, pgstattuple/pgstattuple.c: Fix
contrib/pgstattuple and contrib/pageinspect to prevent attempts to
read temporary tables of other sessions; that is unsafe because of
the way our buffer management works. Per report from Stuart
Bishop. This is redundant with the bufmgr.c checks in HEAD, but
not at all redundant in the back branches.

2009-03-31 18:23 momjian

* doc/src/sgml/release.sgml: Remove some "Other" sections in the
release notes by putting the items at the top of their sections.

2009-03-31 18:12 tgl

* src/: backend/catalog/index.c, backend/catalog/namespace.c,
backend/catalog/toasting.c, backend/commands/analyze.c,
backend/commands/cluster.c, backend/commands/copy.c,
backend/commands/indexcmds.c, backend/commands/tablecmds.c,
backend/commands/vacuum.c, backend/optimizer/prep/prepunion.c,
backend/postmaster/autovacuum.c, backend/storage/buffer/bufmgr.c,
backend/utils/cache/relcache.c, include/utils/rel.h: Modify the
relcache to record the temp status of both local and nonlocal temp
relations; this is no more expensive than before, now that we have
pg_class.relistemp. Insert tests into bufmgr.c to prevent
attempting to fetch pages from nonlocal temp relations. This
provides a low-level defense against bugs-of-omission allowing temp
pages to be loaded into shared buffers, as in the
contrib/pgstattuple problem reported by Stuart Bishop. While at
it, tweak a bunch of places to use new relcache tests (instead of
expensive probes into pg_namespace) to detect local or nonlocal
temp tables.

2009-03-31 14:58 mha

* src/bin/initdb/: initdb.c (REL8_1_STABLE), initdb.c
(REL8_3_STABLE), initdb.c (REL8_0_STABLE), initdb.c
(REL8_2_STABLE), initdb.c: Don't crash initdb when we fail to get
the current username. Give an error message and exit instead, like
we do elsewhere...

Per report from Wez Furlong and Robert Treat.

2009-03-31 13:59 tgl

* doc/src/sgml/catalogs.sgml, src/backend/catalog/heap.c,
src/backend/utils/cache/relcache.c,
src/include/catalog/catversion.h,
src/include/catalog/pg_attribute.h, src/include/catalog/pg_class.h:
Add a "relistemp" boolean column to pg_class, which is true for
temporary relations (including a temp table's indexes and toast
table/index), and false for normal relations. For ease of
checking, this commit just adds the column and fills it correctly
--- revising the relation access machinery to use it will come
separately.

2009-03-31 01:18 heikki

* src/backend/storage/ipc/: procarray.c (REL8_1_STABLE),
procarray.c (REL8_3_STABLE), procarray.c (REL8_2_STABLE),
procarray.c: Fix a rare race condition when commit_siblings > 0 and
a transaction commits at the same instant as a new backend is
spawned. Since CountActiveBackends() doesn't hold ProcArrayLock, it
needs to be prepared for the case that a pointer at the end of the
proc array is still NULL even though numProcs says it should be
valid, since it doesn't hold ProcArrayLock. Backpatch to 8.1. 8.0
and earlier had this right, but it was broken in the split of
PGPROC and sinval shared memory arrays.

Per report and proposal by Marko Kreen.

2009-03-30 22:34 momjian

* doc/src/sgml/release.sgml: Update release note introductory
description.

2009-03-30 21:41 tgl

* doc/src/sgml/libpq.sgml, src/interfaces/libpq/exports.txt,
src/interfaces/libpq/fe-secure.c, src/interfaces/libpq/libpq-fe.h:
Add PQinitOpenSSL() function to support applications that use
libcrypto but not OpenSSL (or perhaps vice versa, if that's
possible).

Andrew Chernow, with minor editorialization by me.

2009-03-30 21:26 momjian

* doc/src/sgml/release.sgml: More new subsections in release notes.

2009-03-30 18:01 momjian

* doc/src/sgml/release.sgml: More release note changes, including a
lower level of subsections.

2009-03-30 16:32 momjian

* doc/src/sgml/release.sgml: More release note adjustments,
reordering.

2009-03-30 15:59 momjian

* doc/src/sgml/release.sgml: More release note wording
improvements; section order adjustments.

2009-03-30 14:34 momjian

* doc/src/sgml/release.sgml: Reorder release note sections.

2009-03-30 13:30 tgl

* src/backend/optimizer/plan/planner.c: Fix window function plan
generation to cope with volatile sort expressions. (Not clear how
useful these really are, but failing is no good...) Per report from
David Fetter and Robert Treat.

2009-03-30 12:15 alvherre

* doc/src/sgml/: plpython.sgml (REL8_3_STABLE), plpython.sgml:
Update URL to Python bug tracker. Backpatch to 8.3; doesn't seem
worthy of further backpatch.

2009-03-30 00:08 tgl

* src/: backend/access/common/heaptuple.c,
backend/executor/execTuples.c, include/executor/tuptable.h,
test/regress/expected/rangefuncs.out,
test/regress/sql/rangefuncs.sql (REL8_3_STABLE),
backend/access/common/heaptuple.c, backend/executor/execTuples.c,
include/executor/tuptable.h, test/regress/expected/rangefuncs.out,
test/regress/sql/rangefuncs.sql (REL8_2_STABLE),
backend/access/common/heaptuple.c, backend/executor/execTuples.c,
include/executor/tuptable.h, test/regress/expected/rangefuncs.out,
test/regress/expected/with.out, test/regress/sql/rangefuncs.sql,
test/regress/sql/with.sql: Fix an oversight in the support for
storing/retrieving "minimal tuples" in TupleTableSlots. We have
functions for retrieving a minimal tuple from a slot after storing
a regular tuple in it, or vice versa; but these were implemented by
converting the internal storage from one format to the other. The
problem with that is it invalidates any pass-by-reference Datums
that were already fetched from the slot, since they'll be pointing
into the just-freed version of the tuple. The known problem cases
involve fetching both a whole-row variable and a pass-by-reference
value from a slot that is fed from a tuplestore or tuplesort
object. The added regression tests illustrate some simple cases,
but there may be other failure scenarios traceable to the same bug.
Note that the added tests probably only fail on unpatched code if
it's built with --enable-cassert; otherwise the bug leads to
fetching from freed memory, which will not have been overwritten
without additional conditions.

Fix by allowing a slot to contain both formats simultaneously;
which turns out not to complicate the logic much at all, if
anything it seems less contorted than before.

Back-patch to 8.2, where minimal tuples were introduced.

2009-03-29 15:13 momjian

* doc/src/sgml/release.sgml: More release note markup.

2009-03-28 23:58 momjian

* doc/src/sgml/release.sgml: More release note markup.

2009-03-28 23:01 momjian

* doc/src/sgml/release.sgml: Add SGML markup for
commands/literal/application/etc in release notes; still more work
to do.

2009-03-28 18:05 momjian

* doc/src/sgml/release.sgml: Consistent 8.4 release note naming for
Itagaki Takahiro

2009-03-28 14:48 momjian

* src/interfaces/libpq/fe-secure.c: Clarify variable naming:
pq_initssllib -> pq_init_ssl_lib

2009-03-28 10:15 momjian

* doc/src/sgml/release.sgml: Update release notes to say citext is
multi-byte aware, per suggestion from patch author:

Add /contrib/citext as a case-insensitive, multibyte-capable
text data type (David Wheeler)

2009-03-27 23:26 momjian

* doc/src/sgml/ref/: alter_role.sgml, set_role.sgml: Better
document that SET ROLE does not uset ALTER ROLE SET settings;
suggested wording from Josh Berkus.

2009-03-27 21:36 momjian

* doc/src/sgml/libpq.sgml, src/interfaces/libpq/fe-secure.c: Better
document PQinitSSL(0) behavior in regards to libcrypto.

2009-03-27 20:10 tgl

* doc/src/sgml/monitoring.sgml: Add documentation of the fact that
dtrace probes evaluate their parameters even when not active.
Explain how to prevent that with an ENABLED() check.

2009-03-27 18:39 momjian

* doc/src/sgml/release.sgml: Document in release notes that NOT IN
is only for NOT EXIST clauses.

Andrew Gierth

2009-03-27 15:58 tgl

* configure, configure.in: On Solaris, we should only force use of
our own getopt(); it's okay to use the system's getopt_long(). The
previous coding was the result of a sloppy discussion that failed
to draw this distinction. The result was that PG programs don't
handle options as users of that platform expect. Per gripe from
Chuck McDevitt.

Although this is a pre-existing bug, I'm not backpatching since I
think we could do with a bit of beta testing before concluding this
is really OK.

2009-03-27 15:17 mha

* doc/src/sgml/release.sgml: Fix markup, per Devrim

2009-03-27 14:56 tgl

* src/backend/utils/adt/xml.c: Add an errdetail explaining why we
reject infinite dates and timestamps while converting to XML.
Bernd Helmle

2009-03-27 14:30 tgl

* src/: backend/executor/execQual.c, backend/executor/functions.c,
backend/executor/nodeCtescan.c,
backend/executor/nodeFunctionscan.c,
backend/executor/nodeMaterial.c, backend/executor/nodeWindowAgg.c,
backend/executor/nodeWorktablescan.c, backend/tcop/pquery.c,
backend/utils/sort/tuplestore.c, include/utils/tuplestore.h: Fix
possible failures when a tuplestore switches from in-memory to
on-disk mode while callers hold pointers to in-memory tuples. I
reported this for the case of nodeWindowAgg's primary scan tuple,
but inspection of the code shows that all of the calls in
nodeWindowAgg and nodeCtescan are at risk. For the moment, fix it
with a rather brute-force approach of copying whenever one of the
at-risk callers requests a tuple. Later we might think of some
sort of reference-count approach to reduce tuple copying.

2009-03-27 11:57 tgl

* src/backend/catalog/index.c: Teach reindex_index() to clear
pg_index.indcheckxmin when possible. Greg Stark, slightly modified
by me.

2009-03-27 10:58 heikki

* src/bin/psql/: tab-complete.c (REL8_1_STABLE), tab-complete.c
(REL8_3_STABLE), tab-complete.c (REL8_2_STABLE), tab-complete.c:
Fix tab completion of ANALYZE VERBOSE <tab>. It was previously
confused with EXPLAIN ANALYZE VERBOSE.

Greg Sabino Mullane, reformatted by myself. Backpatch to 8.1, where
the bug was introduced.

2009-03-27 08:01 mha

* doc/src/sgml/release.sgml: Clearify new SSL certificate
verification in libpq

2009-03-27 07:58 mha

* doc/src/sgml/release.sgml: Fix release notes about pg_hba changes

2009-03-26 22:25 momjian

* doc/src/sgml/release.sgml: Updated release wording, per Greg
Stark:

Previously EXPLAIN VERBOSE had output an internal
representation of the

2009-03-26 21:44 momjian

* doc/src/sgml/release.sgml: Second batch of release note fixes by
Guillaume Smet

2009-03-26 21:26 momjian

* doc/src/sgml/release.sgml: Mark Greg as the instigator of the
statistics target increase:

Increase the default value of default_statistics_target from
10 to 100
(Greg Sabino Mullane, Tom)

2009-03-26 20:45 momjian

* doc/src/sgml/release.sgml: First round of release note changes
from Guillaume Smet.

2009-03-26 20:11 momjian

* doc/src/sgml/release.sgml: Fix release note attribution:

Support the IS0 8601 time interval syntax (Tom, Kevin
Grittner)

per Ron Mayer

2009-03-26 20:08 momjian

* doc/src/sgml/release.sgml: Improved release notes interval
wording:

Made interval seconds rounding more consistent across output
formats (Ron Mayer)

Ron Mayer

2009-03-26 20:06 momjian

* doc/src/sgml/release.sgml: Remove duplicate interval (fractional
seconds) items; Ron Mayer

2009-03-26 20:01 momjian

* doc/src/sgml/release.sgml: Document that Datestyle no longer
controls interval output, per suggestion from Ron Mayer

2009-03-26 18:29 tgl

* contrib/pg_standby/: pg_standby.c (REL8_3_STABLE), pg_standby.c:
Make pg_standby's maxretries option do what one would expect.
Fujii Masao

2009-03-26 18:26 petere

* src/: backend/catalog/dependency.c, backend/catalog/pg_proc.c,
backend/catalog/pg_shdepend.c, backend/executor/execQual.c,
backend/parser/parse_func.c, backend/postmaster/bgwriter.c,
bin/pg_dump/pg_backup_archiver.c, bin/pg_dump/pg_backup_tar.c,
bin/pg_dump/pg_dump.c, bin/psql/describe.c, bin/psql/print.c,
include/c.h, interfaces/ecpg/preproc/variable.c,
pl/plpgsql/src/pl_exec.c, pl/plpython/plpython.c: Gettext plural
support

In the backend, I changed only a handful of exemplary or
important-looking instances to make use of the plural support;
there is probably more work there. For the rest of the source,
this should cover all relevant cases.

2009-03-26 16:55 tgl

* doc/src/sgml/: charset.sgml, config.sgml: Fix a couple of places
that still claimed LC_COLLATE and LC_CTYPE can't be changed after
initdb.

2009-03-26 15:24 tgl

* src/backend/commands/copy.c: Adjust phrasing of complaints about
multi-byte COPY delimiter strings. Per pgsql-hackers discussion of
2009-02-17.

2009-03-26 13:15 tgl

* src/: backend/nodes/outfuncs.c,
backend/optimizer/path/costsize.c,
backend/optimizer/plan/createplan.c,
backend/optimizer/util/pathnode.c, include/nodes/relation.h: If we
expect a hash join to be performed in multiple batches, suppress
"physical tlist" optimization on the outer relation (ie, force a
projection step to occur in its scan). This avoids storing useless
column values when the outer relation's tuples are written to
temporary batch files.

Modified version of a patch by Michael Henderson and Ramon
Lawrence.

2009-03-26 08:38 momjian

* doc/src/sgml/release.sgml: Correction: ansi-join ->anti-join.

2009-03-25 23:46 tgl

* doc/src/sgml/release.sgml: Fix markup so that 'make HISTORY'
works. A couple very minor editorial improvements.

2009-03-25 22:48 momjian

* doc/src/sgml/release.sgml: Reorder 8.4 release note sections.

2009-03-25 22:40 momjian

* doc/src/sgml/release.sgml: Re-add release notes for release
8.3.7.

2009-03-25 21:48 momjian

* doc/src/sgml/release.sgml: Adjust AS OF release notes markup.

2009-03-25 21:31 momjian

* doc/src/sgml/release.sgml: Mention release note items current as
of 2009-03-16.

2009-03-25 21:19 momjian

* doc/src/sgml/release.sgml: First version of 8.4 release notes;
markup/cleanup/reorganization still required.

Kevin Grittner escribió:
> Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>
> > The final CommitFest began November 11, 2008. It closed March 25,
> > 2009 (+ 144 days). Beta1 was released April 15, 2009 (+ 21 days).
>
> I'm not entirely clear on what was happening during the 21 days
> between the end of the CommitFest and and the release of beta1. I
> seem to remember Bruce saying that there were bugs being fixed, and
> that it didn't make sense to release a beta with known bugs of that
> magnitude, but I'm not clear on what was up with that.

That's nonsense IMHO. We should have just released a version
documenting the known bugs, and asking for people to test the rest of
the system.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Thu, Aug 27, 2009 at 03:04:20PM -0400, Robert Haas wrote:
> On Thu, Aug 27, 2009 at 2:54 PM, Peter Eisentraut<peter_e(at)gmx(dot)net> wrote:
> > On tor, 2009-08-27 at 09:58 -0400, Robert Haas wrote:
> >> To get positive results in which you can have confidence, you
> >> have to know that the testing which was done actually did a
> >> reasonably good job exercising the code in a way that would have
> >> flushed out bugs, had any been present.  That sounds a lot like
> >> the definition of a regression test suite.  Of course, we have
> >> that already, but it's nowhere near comprehensive.  Maybe we
> >> should be looking at an expanded test suite that runs on a time
> >> scale of hours rather than seconds. Actually, didn't Peter talk
> >> about something like this at PGCon?
> >
> > Let's look at it this way: If I were writing a compiler, then I
> > would have two main test approaches.  First, I would have an
> > in-tree test suite that compiles a bunch of example code snippets
> > and checks that the results are reasonable.  Call that a
> > regression test.  It would be informed by code coverage analysis
> > and previously reported bugs. Second, as part of my release
> > cycle, I would have an agenda to try to compile a large set of
> > real programs against my new compiler version. I would do that
> > during the beta period.  You will notice that GCC pretty much
> > operates that way.
> >
> > We have regression tests.  They could and should be expanded.
> >  That's a developer job, and we can start working on that now.
> >  But this discussion was about what to do during beta.  And I
> > think during beta you want to test PostgreSQL against a large set
> > of real applications. But we could try to clarify how to actually
> > do that in an organized way.
> >
> > Now, if you want to improve the regression tests, I would suggest
> > going through the commits since 8.4beta and since 8.4.0 final
> > release and ask how these problems could have been prevented or
> > caught earlier.  I suppose a test suite for WAL might be part of
> > the answer, but a closer analysis might be insightful.
>
> What I want to do is address the concern about too much of any given
> year being consumed by beta and CommitFest. I'm not sure I know how
> to do that though.

How about something in the alphas to the effect of,

Using PostgreSQL?
Have a development server to spare?
Try your application stack on alpha1!
We'd love to hear back. Functionality, performance, you name it.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:
>
> How about something in the alphas to the effect of,
>
> Using PostgreSQL?
> Have a development server to spare?
> Try your application stack on alpha1!
> We'd love to hear back. Functionality, performance, you name it.
>
>
>

I don't know of anyone who is likely to want to try out alphas in their
normal development environments. The client I approached was
specifically prepared to test beta releases that way.

cheers

andrew

Le jeudi 27 août 2009 à 14:27 -0500, Jaime Casanova a écrit :
> the point was that if we simply were saying: hey! mysql can interpret
> this, make postgres do the same then we could end up with a lot of
> broken stuff... just because mysql users think is wonderful to not
> have to write sane code...

Not MySQL in general. Only the subset which helps and seems reasonable.
Bye, JM

Simon,

> The level of detailed planning happening now is a change for the
> community and in general I think it's a good thing. In the past we've
> always said it will be shipped when it's ready, and now we seem to be
> caught by our own rules. There's no need to make hard decisions now.
> Let's keep some flexibility in our thinking. If the structures give us
> problems, lets change the structures. The idea is the plans help the
> developers, not hinder them or make it harder to include big features.

There's some very good reasons for the health of the project to have
specific release dates and stick to them. "When will it be released" is
an important question to answer, and it's far better for the developers
who *aren't* working on big features to not be elastic.

So, with that in mind: what is your statement on three versus four
commitfests? Does it make a difference to you?

--
Josh Berkus
PostgreSQL Experts Inc.
www.pgexperts.com

On Thu, Aug 27, 2009 at 6:09 PM, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
>> Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> The final CommitFest began November 11, 2008.  It closed March 25,
>>> 2009 (+ 144 days).  Beta1 was released April 15, 2009 (+ 21 days).
>
>> I'm not entirely clear on what was happening during the 21 days
>> between the end of the CommitFest and and the release of beta1.
>
> Release note drafting is one part of it, but mostly it's "loose end
> cleanup".  Historically there have always been a pile of loose ends
> to be dealt with, and the CommitFest structure doesn't really do
> anything to avoid that.  If you're interested, attached are all the
> commits between commitfest closure (which I announced immediately
> after committing the addition of contrib/btree_gin) and stamping
> beta1 (which was actually several days before the date Robert gives,
> because of the need for the packagers to do their thing).
>
> It appears to me that release notes weren't actually the bottleneck this
> time, though they have been in some prior cycles.  Bruce committed a
> first draft immediately after the commitfest closed.  Rather, it was
> arguing about things like \df behavior and cardinality() that took up
> the time.

It felt, at the time, like the release notes were holding things up,
hence the various and so-far-unsuccesful volunteering related to that
task. But it's possible that there was enough parallel activity going
on that it wasn't actually so.

> We could certainly have released a perhaps-less-polished beta earlier.
> I think that the traditional criterion is that we don't release beta1
> as long as there are any known issues that might force an initdb.
> We were successful in avoiding a post-beta initdb this time, although
> IIRC the majority of release cycles have had one --- so maybe you
> could argue that that's not so important.  It would certainly be
> less important if we had working pg_migrator functionality to ease
> the pain of going from beta to final.
>
> Now that we have the alpha-release mechanism defined, some of the
> pressure for a quick beta could be taken off by releasing a final
> alpha right after the final commitfest closes.

Definitely. Looking at it in hindsight, 3 weeks seems like a
reasonable amount of time between the end of the last CommitFest and
the start of beta. It felt long at the time, but maybe that's partly
because the CommitFest was so intermininable.

I think what a lot of this boils down to is that we need a better
system for managing the tasks that need to be completed at each stage
of the release process, and who is working on each one, and what the
status of it is, just as we do for CommitFests.

...Robert

Andrew Dunstan wrote:
> I don't know of anyone who is likely to want to try out alphas in their
> normal development environments. The client I approached was
> specifically prepared to test beta releases that way.

Perhaps end-users won't, but I think companies who develop software that
works on top of postgres will. Perhaps to make sure their existing software
continues to work; or perhaps to get a head start working with new features.
I test against CVS-head occasionally.

Josh Berkus wrote:
> There's some very good reasons for the health of the project to have
> specific release dates and stick to them.

Help me understand why?

The Linux kernel seems to do fine with a "when it is ready" cycle,
where some releases(2.6.24) take twice the time of others(2.6.28)[1,2].
I imagine it has similar stability and lack-of-data-loss requirements
as postgres does.

I understand why commercial packagers like Ubuntu - especially public
ones like Novell and Red Hat who have to forecast earnings - want to
schedule their releases.

And I can imagine they'd appreciate it if project releases aren't
too close to their release schedules (if postgres releases right
after they release they suffer from not having the current version;
if postgres releases just before, they have limited testing time).

[1] http://www.linuxfoundation.org/publications/linuxkerneldevelopment.php
[2] http://fblinux.freebase.com/view/base/fblinux/views/linux_kernel_release

> So, with that in mind: what is your statement on three versus four
> commitfests? Does it make a difference to you?

On Fri, Aug 28, 2009 at 4:39 AM, Ron Mayer<rm_pg(at)cheapcomplexdevices(dot)com> wrote:
> Josh Berkus wrote:
>> There's some very good reasons for the health of the project to have
>> specific release dates and stick to them.
>
> Help me understand why?
>
> The Linux kernel seems to do fine with a "when it is ready" cycle,
> where some releases(2.6.24) take twice the time of others(2.6.28)[1,2].
> I imagine it has similar stability and lack-of-data-loss requirements
> as postgres does.