Re: 8.2 features status

I'm not clear on why there's all this doom and gloom about how 8.2 will
be "merely" a performance-oriented release, with few new features, eg
http://archives.postgresql.org/pgsql-hackers/2006-07/msg00111.php

Certainly there's been a ton of effort spent on high-end performance
issues. But a quick troll through the CVS logs shows a fair number of
features that could be considered killer must-have things by their
respective target audiences:

multi-row VALUES, not only for INSERT but everywhere SELECT is allowed
pg_dump multiple -n and -t options, regex patterns for object names
multi-argument aggregates, including SQL2003-standard statistical aggregates
fully configurable timezone abbreviations (no more 'australian_timezones' hack)
allow full zic timezone names in datetime input values
support comparisons like "if row(new.*) is distinct from row(old.*)"
DROP ... IF EXISTS
numerous tsearch2 improvements, eg thesaurus
GIN index opclass
GRANT CONNECT ON DATABASE
support SSL Certificate Revocation List (CRL) files
plpython supports named parameters, composite-type results, more result-set options
plperl prepared queries
domain constraint checks are now applied everywhere
better psql multiline command handling
error cursor position displayed for many parse-analysis errors
standard_conforming_strings can be turned on (HUGE deal for some people)
initdb and pg_ctl can safely start from an admin account on Windows
display multiline values nicely in psql
support SQL-compliant row comparisons; they can be indexscan quals
DROP OWNED, REASSIGN OWNED for dealing with removal of a user
null elements in arrays

There are also some commits that are "mere" performance tweaks, and yet
we should not understate their importance because they could make the
difference between usability and non-usability in many applications:

lazy vacuums are ignored by other processes; improves behavior of concurrent vacuums
add index and table storage options (currently only FILLFACTOR)
stats_command_string overhead reduced to near zero, now on by default
reduce locking involved in DATABASE commands, eg CREATE DATABASE no longer blocks incoming connections
constraint exclusion works for UPDATE and DELETE
constraint exclusion works for UNION ALL views, not only inheritance trees
planner can rearrange join order for many common OUTER JOIN scenarios

And that's not counting some pretty significant submitted-but-not-yet-
reviewed patches (sure, some of these may get rejected, but they're all
open possibilities today):

online index builds
bitmap index AM
updatable views
PL plugin patch (plpgsql debugger infrastructure)
restartable recovery (allow checkpoints for a hot-standby server)
INSERT/UPDATE RETURNING

Not that there's anything wrong with a performance-oriented release
... but if you think that 8.2 is short on features, you'd better get
ready to be disappointed by every future release. There's not all
that much stuff left to do in terms of raw language "features".
(Of course the SQL committee keeps inventing a ton of new stuff every
few years, but how much of that do you really care about?)

regards, tom lane

My outlook is that it isn't a lot of _new_ things that you couldn't do
before, but rather improvements of existing functionality.

---------------------------------------------------------------------------

Tom Lane wrote:
> I'm not clear on why there's all this doom and gloom about how 8.2 will
> be "merely" a performance-oriented release, with few new features, eg
> http://archives.postgresql.org/pgsql-hackers/2006-07/msg00111.php
>
> Certainly there's been a ton of effort spent on high-end performance
> issues. But a quick troll through the CVS logs shows a fair number of
> features that could be considered killer must-have things by their
> respective target audiences:
>
> multi-row VALUES, not only for INSERT but everywhere SELECT is allowed
> pg_dump multiple -n and -t options, regex patterns for object names
> multi-argument aggregates, including SQL2003-standard statistical aggregates
> fully configurable timezone abbreviations (no more 'australian_timezones' hack)
> allow full zic timezone names in datetime input values
> support comparisons like "if row(new.*) is distinct from row(old.*)"
> DROP ... IF EXISTS
> numerous tsearch2 improvements, eg thesaurus
> GIN index opclass
> GRANT CONNECT ON DATABASE
> support SSL Certificate Revocation List (CRL) files
> plpython supports named parameters, composite-type results, more result-set options
> plperl prepared queries
> domain constraint checks are now applied everywhere
> better psql multiline command handling
> error cursor position displayed for many parse-analysis errors
> standard_conforming_strings can be turned on (HUGE deal for some people)
> initdb and pg_ctl can safely start from an admin account on Windows
> display multiline values nicely in psql
> support SQL-compliant row comparisons; they can be indexscan quals
> DROP OWNED, REASSIGN OWNED for dealing with removal of a user
> null elements in arrays
>
> There are also some commits that are "mere" performance tweaks, and yet
> we should not understate their importance because they could make the
> difference between usability and non-usability in many applications:
>
> lazy vacuums are ignored by other processes; improves behavior of concurrent vacuums
> add index and table storage options (currently only FILLFACTOR)
> stats_command_string overhead reduced to near zero, now on by default
> reduce locking involved in DATABASE commands, eg CREATE DATABASE no longer blocks incoming connections
> constraint exclusion works for UPDATE and DELETE
> constraint exclusion works for UNION ALL views, not only inheritance trees
> planner can rearrange join order for many common OUTER JOIN scenarios
>
> And that's not counting some pretty significant submitted-but-not-yet-
> reviewed patches (sure, some of these may get rejected, but they're all
> open possibilities today):
>
> online index builds
> bitmap index AM
> updatable views
> PL plugin patch (plpgsql debugger infrastructure)
> restartable recovery (allow checkpoints for a hot-standby server)
> INSERT/UPDATE RETURNING
>
> Not that there's anything wrong with a performance-oriented release
> ... but if you think that 8.2 is short on features, you'd better get
> ready to be disappointed by every future release. There's not all
> that much stuff left to do in terms of raw language "features".
> (Of course the SQL committee keeps inventing a ton of new stuff every
> few years, but how much of that do you really care about?)
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, 4 Aug 2006, Bruce Momjian wrote:

>
> My outlook is that it isn't a lot of _new_ things that you couldn't do
> before, but rather improvements of existing functionality.

It seems as though the majority of things on Tom's list are new things you
couldn't do (at all easily) before.

Gavin

Gavin Sherry wrote:
> On Fri, 4 Aug 2006, Bruce Momjian wrote:
>
> >
> > My outlook is that it isn't a lot of _new_ things that you couldn't do
> > before, but rather improvements of existing functionality.
>
> It seems as though the majority of things on Tom's list are new things you
> couldn't do (at all easily) before.

To me new things are like PITR, Win32, savepoints, two-phase commit,
partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
there in 8.2 like that?

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, 4 Aug 2006, Bruce Momjian wrote:

> Gavin Sherry wrote:
> > On Fri, 4 Aug 2006, Bruce Momjian wrote:
> >
> > >
> > > My outlook is that it isn't a lot of _new_ things that you couldn't do
> > > before, but rather improvements of existing functionality.
> >
> > It seems as though the majority of things on Tom's list are new things you
> > couldn't do (at all easily) before.
>
> To me new things are like PITR, Win32, savepoints, two-phase commit,
> partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> there in 8.2 like that?

Well, GIN and some of the unreviewed stuff (bitmaps, plpgsql debugger,
updateable views) are in the same league as the stuff in 8.0 in terms of
user demand and catching up with competitors, I think.

A lot of the things on Tom's list are new bits of functionality to things
added around 8.0 and 8.1 (major enhancements to the usability of
constraint exclusion, for example). We knew then that these needed
additional functionality to fill them out and make them useful to a wide
range of people. Ideally we'd have both at each release but reality
doesn't work like that.

Thanks,

Gavin

Gavin Sherry wrote:
> On Fri, 4 Aug 2006, Bruce Momjian wrote:
>
> > Gavin Sherry wrote:
> > > On Fri, 4 Aug 2006, Bruce Momjian wrote:
> > >
> > > >
> > > > My outlook is that it isn't a lot of _new_ things that you couldn't do
> > > > before, but rather improvements of existing functionality.
> > >
> > > It seems as though the majority of things on Tom's list are new things you
> > > couldn't do (at all easily) before.
> >
> > To me new things are like PITR, Win32, savepoints, two-phase commit,
> > partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> > there in 8.2 like that?
>
> Well, GIN and some of the unreviewed stuff (bitmaps, plpgsql debugger,
> updateable views) are in the same league as the stuff in 8.0 in terms of
> user demand and catching up with competitors, I think.
>
> A lot of the things on Tom's list are new bits of functionality to things
> added around 8.0 and 8.1 (major enhancements to the usability of
> constraint exclusion, for example). We knew then that these needed
> additional functionality to fill them out and make them useful to a wide
> range of people. Ideally we'd have both at each release but reality
> doesn't work like that.

Yes, that is my point. It is a "usability" release. Nothing wrong with
that. In fact, some people asked me if we were still doing things for
ordinary users rather than just doing enterprise functionality.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> To me new things are like PITR, Win32, savepoints, two-phase commit,
> partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> there in 8.2 like that?

[ shrug... ] Five out of your six items have no basis in the SQL spec.
So it's not clear to me what your definition of "major feature" is,
unless maybe it's "anything except what we did for 8.2". Can you
enumerate ten things you would consider comparable to the above features
that aren't done yet?

regards, tom lane

Bruce Momjian wrote:
> Gavin Sherry wrote:
>> On Fri, 4 Aug 2006, Bruce Momjian wrote:
>>
>>> My outlook is that it isn't a lot of _new_ things that you couldn't do
>>> before, but rather improvements of existing functionality.
>> It seems as though the majority of things on Tom's list are new things you
>> couldn't do (at all easily) before.
>
> To me new things are like PITR, Win32, savepoints, two-phase commit,
> partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> there in 8.2 like that?

Well to be honest, the things that are coming in 8.2 more people will
use then any of the things you just mentioned.

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > To me new things are like PITR, Win32, savepoints, two-phase commit,
> > partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> > there in 8.2 like that?
>
> [ shrug... ] Five out of your six items have no basis in the SQL spec.
> So it's not clear to me what your definition of "major feature" is,
> unless maybe it's "anything except what we did for 8.2". Can you
> enumerate ten things you would consider comparable to the above features
> that aren't done yet?

No, I cannot. I do think our missing list is shrinking. My point is
that you really couldn't easily work around the 8.0/8.1 items I listed
if they were missing, while the 8.2 items could be more easily
worked-around. Again, nothing wrong with that.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
> Bruce Momjian wrote:
> > Gavin Sherry wrote:
> >> On Fri, 4 Aug 2006, Bruce Momjian wrote:
> >>
> >>> My outlook is that it isn't a lot of _new_ things that you couldn't do
> >>> before, but rather improvements of existing functionality.
> >> It seems as though the majority of things on Tom's list are new things you
> >> couldn't do (at all easily) before.
> >
> > To me new things are like PITR, Win32, savepoints, two-phase commit,
> > partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> > there in 8.2 like that?
>
> Well to be honest, the things that are coming in 8.2 more people will
> use then any of the things you just mentioned.

Right, hence "usability", not "new enterprise features".

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

>>>> It seems as though the majority of things on Tom's list are new things you
>>>> couldn't do (at all easily) before.
>>> To me new things are like PITR, Win32, savepoints, two-phase commit,
>>> partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
>>> there in 8.2 like that?
>> Well to be honest, the things that are coming in 8.2 more people will
>> use then any of the things you just mentioned.
>
> Right, hence "usability", not "new enterprise features".

O.k. I buy that.

Joshua D. Drake

>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

+1

UPDATE/DELETE for CE are a big deal - I really wish we had INSERT too, then
we'd be able to claim "complete" support for partitioning, but this is a big
deal improvement.

- Luke

On 8/3/06 9:30 PM, "Gavin Sherry" <swm(at)linuxworld(dot)com(dot)au> wrote:

> A lot of the things on Tom's list are new bits of functionality to things
> added around 8.0 and 8.1 (major enhancements to the usability of
> constraint exclusion, for example).

Ühel kenal päeval, R, 2006-08-04 kell 00:46, kirjutas Bruce Momjian:
> Tom Lane wrote:
> > Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > > To me new things are like PITR, Win32, savepoints, two-phase commit,
> > > partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> > > there in 8.2 like that?
> >
> > [ shrug... ] Five out of your six items have no basis in the SQL spec.
> > So it's not clear to me what your definition of "major feature" is,
> > unless maybe it's "anything except what we did for 8.2". Can you
> > enumerate ten things you would consider comparable to the above features
> > that aren't done yet?
>
> No, I cannot. I do think our missing list is shrinking. My point is
> that you really couldn't easily work around the 8.0/8.1 items I listed
> if they were missing, while the 8.2 items could be more easily
> worked-around.

The workaround for missing concurrent vacuum was to design your
databases so the your small and large OLTP tables are on different
servers or that you keep a replica and vacuum your large tables there
and then switch over to it.

It is debatable if that qualifies as "more easily worked-around"

The workaround for pl/python not allowing returning records and sets was
to write an actual pl/python function to generate the data and to store
it in global ditionary GD, a set of pl/python data retrieval functions
for each postgresql data type used and a wrapper function in pl/pgsql to
call the real function and then return the the data records using the
data retrieval functions.

May be simple, but a real PITA to do for more than one function.

I guess some other new features that were missing before were as easy to
work around :)

> Again, nothing wrong with that.

Sure. Actually I think that people were able to work around missing
features in 8.0/8.1 as well. The proof being, that people *did* actually
use postgresql before 8.x , some even on win32 ;)

--
----------------
Hannu Krosing
Database Architect
Skype Technologies OÜ
Akadeemia tee 21 F, Tallinn, 12618, Estonia

Skype me: callto:hkrosing
Get Skype for free: http://www.skype.com

On Fri, Aug 04, 2006 at 12:37:10AM -0400, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > To me new things are like PITR, Win32, savepoints, two-phase
> > commit, partitioned tables, tablespaces. These are from 8.0 and
> > 8.1. What is there in 8.2 like that?
>
> [ shrug... ] Five out of your six items have no basis in the SQL
> spec. So it's not clear to me what your definition of "major
> feature" is, unless maybe it's "anything except what we did for
> 8.2". Can you enumerate ten things you would consider comparable to
> the above features that aren't done yet?

First, I'd like to say people are doing a fantastic job here. Kudos!

One huge thing missing from the "done" list is that crucial bit of
infrastructure and process that has shortened feedback loops--hence
the beta period--by weeks if not months: the build farm. It's now
smoothly integrated into the development process, and as a
consequence, we can realistically have a release each year. :)

As far as big missing features go, here's a short list:

* Splitting queries among CPUs--possibly even among machines--for OLAP
loads

* In-place upgrades (pg_upgrade)

* Several varieties of replication, which I believe we as a project
will eventually endorse and ship

* CALL

* WITH RECURSIVE

* MERGE

* Windowing functions

* On-the-fly in-line calls out to PL/your_choice without needing to
issue DDL

* Wild-eyed feral bits of the SQL standard like SQL/MED and SQL/XML

But all that leaves out the oldest, most honored Postgres tradition:

Breaking New Ground.

We're definitely not done yet. :)

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David,

On 8/3/06 11:02 PM, "David Fetter" <david(at)fetter(dot)org> wrote:

> * Splitting queries among CPUs--possibly even among machines--for OLAP
> loads
>
> * In-place upgrades (pg_upgrade)
>
> * Several varieties of replication, which I believe we as a project
> will eventually endorse and ship
>
> * CALL
>
> * WITH RECURSIVE
>
> * MERGE
>
> * Windowing functions
>
> * On-the-fly in-line calls out to PL/your_choice without needing to
> issue DDL

My ordering of this list in terms of priority is:

1) Windowing functions
2) MERGE
3) Index only access (new)
4) In-place upgrades

We already have splitting queries among CPUs and machines.

- Luke

All,

<grin> Aren't I, the marketing geek, supposed to be the one whining about
this?

Seriously, PostgreSQL has the fastest release cycle of any RDBMS project in
the world. The request I'm hearing from large production users is to release
*less* often. So I don't find it a problem that this release has less
"checklist" features than the last two did, and I don't think anyone else
will.

If all the pending features die then I might find it a stretch to write the
press release, but otherwise, no problem. And since when were we a
marketing-driven project anyway?

> * In-place upgrades (pg_upgrade)

BTW, I may get Sun to contribute an engineer for this; will get you posted.

Oh, and if it makes it, Tzadhi's FULL DISJUNCTIONS patch is newsworthy.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

David Fetter skrev:

> As far as big missing features go, here's a short list:
>
> * Windowing functions

If we are to wish for things the window functions and a proper
collation/locale support is what I miss the most.

/Dennis

On Fri, Aug 04, 2006 at 08:27:02AM +0200, Dennis Bjorklund wrote:
> If we are to wish for things the window functions and a proper
> collation/locale support is what I miss the most.

Agreed. The complaints about collation/locale support have been
continuous over the years, and it really is quite irritating in certain
situations. I got the COLLATE support as far as grammer and executor
support but got stuck on the planning and optimiser. Maybe one day I'll
get the time to really finish it off properly... (if anyone else wants
to have a shot, go for it).

One downside of a fast release cycle: fast tree drift :) But I guess
you can't really complain about that.

Oh yeah, user-defined typmod would be cool. There's some movement on
that though.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Bruce Momjian wrote:
>
> Right, hence "usability", not "new enterprise features".
>
I'm not too happy about the label "usability".

"Ok, maybe postgres gets usable finally by supporting features that
MySQL had for a long time...." a MySql guy would say.

Regards,
Andreas

On 04/08/06, Andreas Pflug <pgadmin(at)pse-consulting(dot)de> wrote:
> Bruce Momjian wrote:
> >
> > Right, hence "usability", not "new enterprise features".
> >
> I'm not too happy about the label "usability".
>
> "Ok, maybe postgres gets usable finally by supporting features that
> MySQL had for a long time...." a MySql guy would say.

I have the same feeling about the term "usability". It could
be interpreted like : PostgreSQL was not usable until now.

Best wishes,
Adrian Maier

On 8/4/06, Luke Lonergan <llonergan(at)greenplum(dot)com> wrote:
> My ordering of this list in terms of priority is:
>
> 1) Windowing functions
> 2) MERGE
> 3) Index only access (new)
> 4) In-place upgrades

And what about compression of on-disk sorting? There has been a long
thread about this idea. Is there any news about this feature? IIRC Jim
Nasby and Martijn were working on testing that and on validating it
was interesting for most of the cases.

--
Guillaume

Tom Lane wrote:

>Not that there's anything wrong with a performance-oriented release
>... but if you think that 8.2 is short on features, you'd better get
>ready to be disappointed by every future release.
>

It's a pity that some expectations have been raised about features that
we haven't seen patches for, e.g. MERGE and/or some form of UPSERT, and
recursive queries. I am not pointing fingers, but I do think we need
some way in which the community can ensure that certain goals are met,
or at least try to help if things fall in a ditch, rather than just
relying on hackers scratching whatever itch they happen to get in
splendid isolation and then trying to merge the results.

cheers

andrew

Josh Berkus wrote:

>Oh, and if it makes it, Tzadhi's FULL DISJUNCTIONS patch is newsworthy.
>
>
>

Have we seen a patch for this? I don't recall seeing one. If not it had
better get in damn fast, I guess.

cheers

andrew

On Friday 04 August 2006 02:20, Josh Berkus wrote:
> Seriously, PostgreSQL has the fastest release cycle of any RDBMS project in
> the world. The request I'm hearing from large production users is to
> release *less* often. So I don't find it a problem that this release has
> less "checklist" features than the last two did, and I don't think anyone
> else will.
>

Yes... one idea I have seen floated is that every other release should work
within the constraints of not requireing dump/reload, so that the really
nasty upgrade cycles could be spread out 2 years apart, but people could get
new features / imporvements each year if they wanted to. It sounds like a
good idea in theory, but would take some real world wrangling to achieve it.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> Have we seen a patch for this? I don't recall seeing one. If not it had
> better get in damn fast, I guess.

Yes, it was submitted the day before freeze.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris(at)enterprisedb(dot)com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

Jonah H. Harris wrote:

> On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>> Have we seen a patch for this? I don't recall seeing one. If not it had
>> better get in damn fast, I guess.
>
>
> Yes, it was submitted the day before freeze.
>
Ah. good. Probably was when my mail was down for about 12 hours around
then :-(

cheers

andrew

On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> It's a pity that some expectations have been raised about features that
> we haven't seen patches for, MERGE/UPSERT & recursive queries

Honestly, I've only had four people say it would be nice to have
hierarchical queries (one of them wasn't even a PostgreSQL user).
Almost everyone else seems to ask some variation of, "what's a
hierarchical query and why do I need it?" It's hard to get excited
about writing a patch no one sees a real need for.

When I have a choice of working on things in my spare time, I choose
what to work on based on basically two things (a) what is my interest
in it and (b) who is going to use it. Based on that, I determine
whether it's worth going through the hassle of design, development,
testing, and final documentation (my prelim docs come from design).

In short, I know a lot of people would probably use this feature after
it was there, but *very* few have shown any interest in it and a patch
for it (while in need of rewriting) has existed since 7.4.

> I am not pointing fingers, but I do think we need
> some way in which the community can ensure
> that certain goals are met, or at least try to help
> if things fall in a ditch, rather than just relying on
> hackers scratching whatever itch they happen to
> get in splendid isolation and then trying to merge
> the results.

I agree. The real problem is that we don't look at things in a
business-like, "what are we going to have in the next release"
perspective. Being as it's an OSS community, we just see what patches
come in and we apply what we choose... then pick which ones we see as
"major features" and announce them.

I'd like to be mistaken here, but I don't think we've ever really had
release planning in regards to features. We can always sit down after
8.2 is finalized and plan what are we doing for 8.3 and go from there.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris(at)enterprisedb(dot)com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

Jonah H. Harris wrote:

> On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>
>> It's a pity that some expectations have been raised about features that
>> we haven't seen patches for, MERGE/UPSERT & recursive queries
>
>
> Honestly, I've only had four people say it would be nice to have
> hierarchical queries (one of them wasn't even a PostgreSQL user).
> Almost everyone else seems to ask some variation of, "what's a
> hierarchical query and why do I need it?" It's hard to get excited
> about writing a patch no one sees a real need for.
>
> When I have a choice of working on things in my spare time, I choose
> what to work on based on basically two things (a) what is my interest
> in it and (b) who is going to use it. Based on that, I determine
> whether it's worth going through the hassle of design, development,
> testing, and final documentation (my prelim docs come from design).
>
> In short, I know a lot of people would probably use this feature after
> it was there, but *very* few have shown any interest in it and a patch
> for it (while in need of rewriting) has existed since 7.4.
>

Chicken vs. egg. Look at the number of people using nested sets and
other similar abominations to handle hierarchical data. There are whole
books written about using these kludgy techniques. I think this really
is a case of "build it and they will come".

cheers

andrew

On 8/3/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I'm not clear on why there's all this doom and gloom about how 8.2 will
> be "merely" a performance-oriented release, with few new features, eg
> http://archives.postgresql.org/pgsql-hackers/2006-07/msg00111.php
>
> Certainly there's been a ton of effort spent on high-end performance
> issues. But a quick troll through the CVS logs shows a fair number of
> features that could be considered killer must-have things by their
> respective target audiences:

i can't resist this unproductive distraction from actual work. this
is a huge release for me as it nails a lot of the features i've been
waiting literally years for. it feels a lot like the 7.4 release
where similar debates when on esp. regarding the windows port, etc.

note that even if the release had no user level features at all, it
would be better to release: the outside world likes to see the project
is still active and moving forward.

merlin

On Friday 04 August 2006 09:37, Jonah H. Harris wrote:
> On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
> > It's a pity that some expectations have been raised about features that
> > we haven't seen patches for, MERGE/UPSERT & recursive queries
>
> Honestly, I've only had four people say it would be nice to have
> hierarchical queries (one of them wasn't even a PostgreSQL user).
> Almost everyone else seems to ask some variation of, "what's a
> hierarchical query and why do I need it?"  It's hard to get excited
> about writing a patch no one sees a real need for.

Make that five. I'll bless the day I can get rid of my recursive plsql
functions.

jan

--
--------------------------------------------------------------
Jan de Visser                     jdevisser(at)digitalfairway(dot)com

                Baruk Khazad! Khazad ai-menu!
--------------------------------------------------------------

Luke Lonergan wrote:
> +1
>
> UPDATE/DELETE for CE are a big deal - I really wish we had INSERT too, then
> we'd be able to claim "complete" support for partitioning, but this is a big
> deal improvement.

I haven't be following this but.. does the above mean that if CE is
turned on and they are partitioned tables, I don't have to use a trigger
for update/delete operations?

Joshua D. Drake

>
> - Luke
>
>
> On 8/3/06 9:30 PM, "Gavin Sherry" <swm(at)linuxworld(dot)com(dot)au> wrote:
>
>> A lot of the things on Tom's list are new bits of functionality to things
>> added around 8.0 and 8.1 (major enhancements to the usability of
>> constraint exclusion, for example).
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

> * Several varieties of replication, which I believe we as a project
> will eventually endorse and ship

This one will cause confusion regardless of how much advocacy,
documentation and will power we put into it.

> * On-the-fly in-line calls out to PL/your_choice without needing to
> issue DDL

You mean something like: EXEC plperl(print $foo)?

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

> 3) Index only access (new)

Does this mean, I have hit the index and have the actual tuple data in
the index row? So I don't have to go back to the relation to get the info?

Joshua D. Drake

> We already have splitting queries among CPUs and machines.

Yes, YOU do. We don't.

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Andreas Pflug wrote:
> Bruce Momjian wrote:
>> Right, hence "usability", not "new enterprise features".
>>
> I'm not too happy about the label "usability".
>
> "Ok, maybe postgres gets usable finally by supporting features that
> MySQL had for a long time...." a MySql guy would say.

Good point...

What about "refinement"

Joshua D. Drake

>
> Regards,
> Andreas
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Fri, Aug 04, 2006 at 09:37:32AM -0400, Jonah H. Harris wrote:
> I agree. The real problem is that we don't look at things in a
> business-like, "what are we going to have in the next release"
> perspective. Being as it's an OSS community, we just see what patches
> come in and we apply what we choose... then pick which ones we see as
> "major features" and announce them.

There's a fairly standard way of getting people to do things for you:
pay them money. Thing is, people can run around making lists of thing
that would be cool, but no-one can really force anyone to do anything.

Now, there are the priveledged few who are being paid to work on
postgres. If you can convince their employers to fund things on the
list, it might help. Also, some stuff might work well as Google SoC
projects...

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Jonah H. Harris wrote:
> On 8/4/06, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> It's a pity that some expectations have been raised about features that
>> we haven't seen patches for, MERGE/UPSERT & recursive queries
>
> Honestly, I've only had four people say it would be nice to have
> hierarchical queries (one of them wasn't even a PostgreSQL user).

You obviously aren't dealing with the number of people I am :) I get
this question all the time.

> I agree. The real problem is that we don't look at things in a
> business-like, "what are we going to have in the next release"
> perspective. Being as it's an OSS community, we just see what patches
> come in and we apply what we choose... then pick which ones we see as
> "major features" and announce them.
>
> I'd like to be mistaken here, but I don't think we've ever really had
> release planning in regards to features. We can always sit down after
> 8.2 is finalized and plan what are we doing for 8.3 and go from there.

The only way we will get even a reasonable about of release planning is
for the commercial entities involved around PostgreSQL to put a lot more
resources into actually developing PostgreSQL.

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Josh,

On 8/4/06 7:47 AM, "Joshua D. Drake" <jd(at)commandprompt(dot)com> wrote:

>> 3) Index only access (new)
>
> Does this mean, I have hit the index and have the actual tuple data in
> the index row? So I don't have to go back to the relation to get the info?

Yep. Fix the visibility issue - there are a number of good ideas on how to
do it, we are in a position to bang it out now IMO.

>> We already have splitting queries among CPUs and machines.
>
> Yes, YOU do. We don't.

Details, details - redefine "We" and it's the same thing.

- Luke

Adrian Maier wrote:
> On 04/08/06, Andreas Pflug <pgadmin(at)pse-consulting(dot)de> wrote:
> > Bruce Momjian wrote:
> > >
> > > Right, hence "usability", not "new enterprise features".
> > >
> > I'm not too happy about the label "usability".
> >
> > "Ok, maybe postgres gets usable finally by supporting features that
> > MySQL had for a long time...." a MySql guy would say.
>
> I have the same feeling about the term "usability". It could
> be interpreted like : PostgreSQL was not usable until now.

_improved_ usability

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

>>> "Ok, maybe postgres gets usable finally by supporting features that
>>> MySQL had for a long time...." a MySql guy would say.
>> I have the same feeling about the term "usability". It could
>> be interpreted like : PostgreSQL was not usable until now.
>
> _improved_ usability

I still don't like it. Usability is an opinion based thing. Personally I
find MySQL confusing and illogical. However I know many people love it
for that very same reason.

I think we should drop the term usability as a selling part of the PR
and push it into further description.. Instead we should use a slightly
more expensive word (think 50 cents, not 5). :)

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>
> >Not that there's anything wrong with a performance-oriented release
> >... but if you think that 8.2 is short on features, you'd better get
> >ready to be disappointed by every future release.
> >
>
> It's a pity that some expectations have been raised about features that
> we haven't seen patches for, e.g. MERGE and/or some form of UPSERT, and
> recursive queries. I am not pointing fingers, but I do think we need
> some way in which the community can ensure that certain goals are met,
> or at least try to help if things fall in a ditch, rather than just
> relying on hackers scratching whatever itch they happen to get in
> splendid isolation and then trying to merge the results.

What we do is when people claim items, we monitor them to be sure they
get them done for the current release, or at least give it their best
try. There is not much more we can do.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
>
> >>> "Ok, maybe postgres gets usable finally by supporting features that
> >>> MySQL had for a long time...." a MySql guy would say.
> >> I have the same feeling about the term "usability". It could
> >> be interpreted like : PostgreSQL was not usable until now.
> >
> > _improved_ usability
>
> I still don't like it. Usability is an opinion based thing. Personally I
> find MySQL confusing and illogical. However I know many people love it
> for that very same reason.
>
> I think we should drop the term usability as a selling part of the PR
> and push it into further description.. Instead we should use a slightly
> more expensive word (think 50 cents, not 5). :)

Fine, I am all ears. Also, a lot of people are thinking usability
improvements aren't a big item, but they are. I had a press person ask
during the 8.0 release, "You have a lot of large features in 8.0, but
what is there for the ordinary database user?". At that time all we had
was COPY CSV. This release will have a lot of those usability
improvements.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

>> I think we should drop the term usability as a selling part of the PR
>> and push it into further description.. Instead we should use a slightly
>> more expensive word (think 50 cents, not 5). :)
>
> Fine, I am all ears. Also, a lot of people are thinking usability
> improvements aren't a big item, but they are. I had a press person ask
> during the 8.0 release, "You have a lot of large features in 8.0, but
> what is there for the ordinary database user?". At that time all we had
> was COPY CSV. This release will have a lot of those usability
> improvements.

I don't disagree with your thought process and I full agree :). I am
just saying we should spice it up a bit :)

Hmmmm....

PostgreSQL has showed an unprecedented commitment to all skill levels of
database developers. With the release of 8.2 the group has focused on
refinement and usability. Features such as ... add to the already
impressive list of enterprise class features such as ...

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Joshua D. Drake wrote:
>
> >> I think we should drop the term usability as a selling part of the PR
> >> and push it into further description.. Instead we should use a slightly
> >> more expensive word (think 50 cents, not 5). :)
> >
> > Fine, I am all ears. Also, a lot of people are thinking usability
> > improvements aren't a big item, but they are. I had a press person ask
> > during the 8.0 release, "You have a lot of large features in 8.0, but
> > what is there for the ordinary database user?". At that time all we had
> > was COPY CSV. This release will have a lot of those usability
> > improvements.
>
> I don't disagree with your thought process and I full agree :). I am
> just saying we should spice it up a bit :)
>
> Hmmmm....
>
> PostgreSQL has showed an unprecedented commitment to all skill levels of
> database developers. With the release of 8.2 the group has focused on
> refinement and usability. Features such as ... add to the already
> impressive list of enterprise class features such as ...

Sounds very good, and that was my goal, that we get a "focus" idea for
the release, I think that is it. As I remember, 8.0 was enterprise
features, and 8.1 was performance.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Merlin Moncure wrote:
> On 8/3/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I'm not clear on why there's all this doom and gloom about how 8.2 will
>> be "merely" a performance-oriented release, with few new features, eg
>> http://archives.postgresql.org/pgsql-hackers/2006-07/msg00111.php
>>
>> Certainly there's been a ton of effort spent on high-end performance
>> issues. But a quick troll through the CVS logs shows a fair number of
>> features that could be considered killer must-have things by their
>> respective target audiences:
>
> i can't resist this unproductive distraction from actual work. this
> is a huge release for me as it nails a lot of the features i've been
> waiting literally years for. it feels a lot like the 7.4 release
> where similar debates when on esp. regarding the windows port, etc.
>
> note that even if the release had no user level features at all, it
> would be better to release: the outside world likes to see the project
> is still active and moving forward.

I fully agree here - 8.2 is a release that is of more interest to us
than say 8.0 was.
For some of our existing apps 8.2 is dramatically faster due to much
better planed queries and things like 20-25% faster dump/restore cycles
due to the dramatic improvements on sorting (and therefor CREATE INDEX)
are really really cool things.
Just switching to 8.2 makes one of our (interactive-web) app feel
"blazingly fast" instead of just "ok" and that is a good thing - a very
good one in fact ...
And beside that the list tom posted is already damn impressive on it's
own - i guess there are a number of large projects that can only dream
of having a "new features" list like that.

Stefan

Bruce Momjian wrote:
> Joshua D. Drake wrote:
>>>> I think we should drop the term usability as a selling part of the PR
>>>> and push it into further description.. Instead we should use a slightly
>>>> more expensive word (think 50 cents, not 5). :)
>>> Fine, I am all ears. Also, a lot of people are thinking usability
>>> improvements aren't a big item, but they are. I had a press person ask
>>> during the 8.0 release, "You have a lot of large features in 8.0, but
>>> what is there for the ordinary database user?". At that time all we had
>>> was COPY CSV. This release will have a lot of those usability
>>> improvements.
>> I don't disagree with your thought process and I full agree :). I am
>> just saying we should spice it up a bit :)
>>
>> Hmmmm....
>>
>> PostgreSQL has showed an unprecedented commitment to all skill levels of
>> database developers. With the release of 8.2 the group has focused on
>> refinement and usability. Features such as ... add to the already
>> impressive list of enterprise class features such as ...
>
> Sounds very good, and that was my goal, that we get a "focus" idea for
> the release, I think that is it. As I remember, 8.0 was enterprise
> features, and 8.1 was performance.
>

Quick reword:

With the release of 8.2, PostgreSQL has shown an unprecedented
commitment to all skill levels of database developers. The 8.2 version
of PostgreSQL has focused on refinement and usability with features such
as ... Adding to the already impressive list of enterprise class
features such as ...

Sincerely,

Joshua D. Drake
--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Martijn van Oosterhout <kleptog(at)svana(dot)org> writes:
> On Fri, Aug 04, 2006 at 09:37:32AM -0400, Jonah H. Harris wrote:
>> I agree. The real problem is that we don't look at things in a
>> business-like, "what are we going to have in the next release"
>> perspective. Being as it's an OSS community, we just see what patches
>> come in and we apply what we choose... then pick which ones we see as
>> "major features" and announce them.

> There's a fairly standard way of getting people to do things for you:
> pay them money. Thing is, people can run around making lists of thing
> that would be cool, but no-one can really force anyone to do anything.

> Now, there are the priveledged few who are being paid to work on
> postgres. If you can convince their employers to fund things on the
> list, it might help. Also, some stuff might work well as Google SoC
> projects...

Not to be unkind, but AFAIR all the unmet expectations in this release
cycle came from commercially-sponsored developers who said they'd do X
and then didn't finish it. I don't see that "taking a business-like
approach" would improve matters at all, even if there were a way for
the project to dictate to people what they should work on, which there
surely is not. Programmers are optimists by nature and will *always*
think they can accomplish more than really gets done (cf. The Mythical
Man-Month, still on target after all these years).

If you want an easy solution, here it is: *never* tell anyone that
feature X will be in the next release until it's actually committed
to CVS. (And maybe not then ... we've backed things out before.)
The only way there were any unmet expectations here were if people
failed to distinguish "someone is working on it" from "will be done
by 8.2 for sure".

regards, tom lane

"Luke Lonergan" <llonergan(at)greenplum(dot)com> writes:
> UPDATE/DELETE for CE are a big deal - I really wish we had INSERT too,

Huh? We had INSERT working before, that's why it's not mentioned.

regards, tom lane

"Guillaume Smet" <guillaume(dot)smet(at)gmail(dot)com> writes:
> And what about compression of on-disk sorting?

That's purely a performance issue, which some people seem to want
to define as "not a new feature" ... which is not *my* view of
what's important ...

regards, tom lane

On Fri, Aug 04, 2006 at 12:40:36PM -0400, Tom Lane wrote:
> "Guillaume Smet" <guillaume(dot)smet(at)gmail(dot)com> writes:
> > And what about compression of on-disk sorting?
>
> That's purely a performance issue, which some people seem to want
> to define as "not a new feature" ... which is not *my* view of
> what's important ...
>
> regards, tom lane
>
>From my point of view, some of these new performance features are
the most important changes in the 8.2 release. We are using other
databases currently because of performance problems that are going
to be addressed in this release. If we can use PostgreSQL instead,
their are new possibilities for reliability and robustness that
our current database choice will not allow.

Ken

Luke,

> Yep. Fix the visibility issue - there are a number of good ideas on how to
> do it, we are in a position to bang it out now IMO.

Actually, a group of us discussed this at the Code Sprint in Toronto, and came
up with a plan which will also reduce row overhead on large tables. I can't
remember who was working on that though.

> >> We already have splitting queries among CPUs and machines.
> >
> > Yes, YOU do. We don't.
>
> Details, details - redefine "We" and it's the same thing.

Well, if you'll give us the schedule for open-sourcing MPP 2.0 ... ;-)

And actually, even incorporating very limited multi-threading of queries ...
such as the proposal to dispatch an I/O thread for seq scans ... would help
PostgreSQL a lot.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Joshua D. Drake wrote:

> Bruce Momjian wrote:
>
>> Joshua D. Drake wrote:
>>
>>>>> I think we should drop the term usability as a selling part of the
>>>>> PR and push it into further description.. Instead we should use a
>>>>> slightly more expensive word (think 50 cents, not 5). :)
>>>>
>>>> Fine, I am all ears. Also, a lot of people are thinking usability
>>>> improvements aren't a big item, but they are. I had a press person
>>>> ask
>>>> during the 8.0 release, "You have a lot of large features in 8.0, but
>>>> what is there for the ordinary database user?". At that time all
>>>> we had
>>>> was COPY CSV. This release will have a lot of those usability
>>>> improvements.
>>>
>>> I don't disagree with your thought process and I full agree :). I am
>>> just saying we should spice it up a bit :)
>>>
>>> Hmmmm....
>>>
>>> PostgreSQL has showed an unprecedented commitment to all skill
>>> levels of database developers. With the release of 8.2 the group has
>>> focused on refinement and usability. Features such as ... add to the
>>> already impressive list of enterprise class features such as ...
>>
>>
>> Sounds very good, and that was my goal, that we get a "focus" idea for
>> the release, I think that is it. As I remember, 8.0 was enterprise
>> features, and 8.1 was performance.
>>
>
> Quick reword:
>
>
> With the release of 8.2, PostgreSQL has shown an unprecedented
> commitment to all skill levels of database developers. The 8.2 version
> of PostgreSQL has focused on refinement and usability with features
> such as ... Adding to the already impressive list of enterprise class
> features such as ...
>
>
> Sincerely,
>
> Joshua D. Drake

I am interrested in finding out what you folks mean by usability and
refinement. How do you measure it? These seem to me to be unmeasurable
hackneyed terms with little intrinsic meaning! So could you say
something like: the New Postgresql, version 8.2, building on its
reputation of sheer power and ACID compliance, now implements feature 1,
which provides benefit 1, plus feature 2, which leads to benefit 2,
plus .... etc. thereby maximizing overall ease of use. ??

WE tend to speak of features, now is a good time to speak of benefits to
the dba, benefits for upgrading to an existing project, etc. The New
Postgresql 8,2 will increase speed of response by x %, ad hoc queries
will be x % faster, etc. I know I am being simplistic, but the benefits
need to blurt out, and be immediately recognizable as such to the
majority of players. JMUETCW. Michael

--
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.10.5/407 - Release Date: 8/3/2006

Tom Lane wrote:
> "Luke Lonergan" <llonergan(at)greenplum(dot)com> writes:
>
>>UPDATE/DELETE for CE are a big deal - I really wish we had INSERT too,
>
> Huh? We had INSERT working before, that's why it's not mentioned.

I think what Luke means, is that an INSERT into the base table of the
inheritance hierarchy with CE enabled gets automagically inserted into
the correct partition, and as far as I know that still doesn't happen --
but I've certainly been wrong before :-).

Joe

On Fri, Aug 04, 2006 at 10:52:34AM +0200, Guillaume Smet wrote:
> On 8/4/06, Luke Lonergan <llonergan(at)greenplum(dot)com> wrote:
> >My ordering of this list in terms of priority is:
> >
> >1) Windowing functions
> >2) MERGE
> >3) Index only access (new)
> >4) In-place upgrades
>
> And what about compression of on-disk sorting? There has been a long
> thread about this idea. Is there any news about this feature? IIRC Jim
> Nasby and Martijn were working on testing that and on validating it
> was interesting for most of the cases.

Well, I posted a bunch of numbers, but people wanted more proof. Which I
haven't had time to generate yet.

If someone wants to pick up the ball and run with it, do some testing of

SELECT random() AS r INTO test_table FROM generate_series(1,some_big_number);
\t
SELECT count(*) FROM (SELECT r FROM test_table ORDER BY r);

This is about the worst-case test I can think of; if it still shows
improvement here...

BTW, I suspect it was too late for this to get into 8.2 over a month
ago, as the patch that's floating around has some rather serious issue
in it that would limit usability too much... something about specific
sort modes or some such that I don't recall anymore. My intention was
to try and finish testing after 8.2 when it was more likely that someone
could look into whatever that issue was.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Thu, Aug 03, 2006 at 11:20:48PM -0700, Josh Berkus wrote:
> > * In-place upgrades (pg_upgrade)
>
> BTW, I may get Sun to contribute an engineer for this; will get you posted.

How would such a thing handle changes to page formats?
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Our database has about 700 objects (tables,views, stored functions,
types etc), we want to make a change on a view or tables, it said that there
were a lot of depended obejcts. I know that depended object must be dropped
and then created to solve this. But, It has too many dependecies to drop.

It will be nice to all, if PostgreSQL database objects has an state
(valid/invalid) like Oracle has. Depended objects states are changed to
invalid if any changes would make to the objects that are referenced by
them.

For example ;

Assuming, we have a table named "T" and a view named "W". We wanna
change on a column table "T", it will said that view "W" depended on it. If
the database would has a lot of dependecies then the problem will never be
solved. In my opinion, we let the changes on table "T" then must change the
state the view "W" to invalid.

Of course, i dont know if this is possible..

I hope this is not a nice dream :-)

Best regards

Adnan DURSUN
ASRIN Bilisim Ltd.

> I am interrested in finding out what you folks mean by usability and
> refinement. How do you measure it? These seem to me to be unmeasurable
> hackneyed terms with little intrinsic meaning!

Yep you are absolutely right. That is what press releases are all about.

> So could you say
> something like: the New Postgresql, version 8.2, building on its
> reputation of sheer power and ACID compliance, now implements feature 1,
> which provides benefit 1, plus feature 2, which leads to benefit 2,
> plus .... etc. thereby maximizing overall ease of use. ??
>
> WE tend to speak of features, now is a good time to speak of benefits to
> the dba, benefits for upgrading to an existing project, etc.

If they don't know by reading the terms, then they will want to buy
reading the 50 cent, fancy words.

> The New
> Postgresql 8,2 will increase speed of response by x %, ad hoc queries
> will be x % faster, etc. I know I am being simplistic, but the benefits
> need to blurt out, and be immediately recognizable as such to the
> majority of players. JMUETCW. Michael

Percentages never work because it depends on a specific work load,
specific design, specific hardware etc...

Joshua D. Drake

>
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Fri, Aug 04, 2006 at 12:37:10AM -0400, Tom Lane wrote:
> Bruce Momjian <bruce(at)momjian(dot)us> writes:
> > To me new things are like PITR, Win32, savepoints, two-phase commit,
> > partitioned tables, tablespaces. These are from 8.0 and 8.1. What is
> > there in 8.2 like that?
>
> [ shrug... ] Five out of your six items have no basis in the SQL spec.
> So it's not clear to me what your definition of "major feature" is,

Stuff that's "sexy". Like 2PC, partitioning, etc.

And to add to David Fetter's list without another post...

* Better interface to partitioning

* Stored procedures (ie: BEGIN; COMMIT;)
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Guys,

> I still don't like it. Usability is an opinion based thing. Personally I
> find MySQL confusing and illogical. However I know many people love it
> for that very same reason.

As the person who's leading the draft of the press release, let me say that
any "theme" discussions which happen on -hackers from here on out I'm
going to ignore. PR discussions belong on -advocacy, not this list.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

On Fri, Aug 04, 2006 at 12:03:59PM -0400, Bruce Momjian wrote:
> Andrew Dunstan wrote:
> >
> >
> > Tom Lane wrote:
> >
> > >Not that there's anything wrong with a performance-oriented release
> > >... but if you think that 8.2 is short on features, you'd better get
> > >ready to be disappointed by every future release.
> > >
> >
> > It's a pity that some expectations have been raised about features that
> > we haven't seen patches for, e.g. MERGE and/or some form of UPSERT, and
> > recursive queries. I am not pointing fingers, but I do think we need
> > some way in which the community can ensure that certain goals are met,
> > or at least try to help if things fall in a ditch, rather than just
> > relying on hackers scratching whatever itch they happen to get in
> > splendid isolation and then trying to merge the results.
>
> What we do is when people claim items, we monitor them to be sure they
> get them done for the current release, or at least give it their best
> try. There is not much more we can do.

Do we? There seems to be improvements to be had here. For example,
someone at OSCon (Josh D maybe) was saying that we'll have MERGE/UPSERT
in 8.2, but apparently we won't.

At best, people will lay claim to TODO items, and the the nebulous group
concious will vaguely remember that someone's working on it; sometimes
remembering who, sometimes even pushing to get it in for the next
release. But there's plenty of room there for things to get forgotten
about (ie: MERGE/UPSERT, which would have been a nice big feature to add
to the list in 8.2).

BTW, while I'm thinking about it, I believe INSERT ... RETURNING is in,
no?
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Joshua D. Drake wrote:

>
>> I am interrested in finding out what you folks mean by usability and
>> refinement. How do you measure it? These seem to me to be
>> unmeasurable hackneyed terms with little intrinsic meaning!
>
>
> Yep you are absolutely right. That is what press releases are all about.
>
>> So could you say something like: the New Postgresql, version 8.2,
>> building on its reputation of sheer power and ACID compliance, now
>> implements feature 1, which provides benefit 1, plus feature 2, which
>> leads to benefit 2, plus .... etc. thereby maximizing overall ease
>> of use. ??
>>
>> WE tend to speak of features, now is a good time to speak of benefits
>> to the dba, benefits for upgrading to an existing project, etc.
>
>
> If they don't know by reading the terms, then they will want to buy
> reading the 50 cent, fancy words.
>
>> The New Postgresql 8,2 will increase speed of response by x %, ad
>> hoc queries will be x % faster, etc. I know I am being simplistic,
>> but the benefits need to blurt out, and be immediately recognizable
>> as such to the majority of players. JMUETCW. Michael
>
>
> Percentages never work because it depends on a specific work load,
> specific design, specific hardware etc...
>
> Joshua D. Drake
>
>
>
>>
>>
>
>
Josh, percentages, like almost anything, do work in the right context,
in this case, that of the testimonial, something postgresql hasn't
emphasized IMHO. If ten to 20 projects were treated as real and
realistic case studies, with an in-depth description of the project, and
how the NEW Postgresql effected these projects,and these were featured
one after another, daily for an entire month, there would be major play
in the news. By talking in depth about specific projects, we relate to
the actual production users own daily experiences, especiually if % can
be translated into dollars.

For instance: "I Adapted the NEW Postgresql 8.2 in my online e-commerce
website and save $1,343.45 per month in ...."
or: "The Sheer Speed of such an ACID compliant ORDBMS as the NEW
Postgresql 8.2, especially in danling xnxnxnxn enabled our company to ..."

I am just brainstorming, I lack the db expertise, but real life stroies
are powerful creatures with a life of their own.

--
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.394 / Virus Database: 268.10.5/407 - Release Date: 8/3/2006

* Jim C. Nasby (jnasby(at)pervasive(dot)com) wrote:
> On Thu, Aug 03, 2006 at 11:20:48PM -0700, Josh Berkus wrote:
> > > * In-place upgrades (pg_upgrade)
> >
> > BTW, I may get Sun to contribute an engineer for this; will get you posted.
>
> How would such a thing handle changes to page formats?

Couldn't this be done by converting a table/partial-table at a time?
It wouldn't be something which could run while the system is live, but
it'd probably take less time than dump/restore and wouldn't require
double the disk space of the whole database... no?

Thanks,

Stephen

On 8/4/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Not to be unkind, but AFAIR all the unmet expectations in this release
> cycle came from commercially-sponsored developers who said
> they'd do X and then didn't finish it.

FYI, I am not commercially sponsered. I am a full-time employee
devoted to working on database internals, drivers, Oracle
compatibility, architecture, prototyping, and some performance stuff
for EnterpriseDB.

While I work full-time for EnterpriseDB, they have ALWAYS given me
time to work on other PostgreSQL-related things when I ask for it.
However, with the exception of the INSERT/UPDATE RETURNING patch
(which EnterpriseDB asked me to submit), the PostgreSQL community is,
and always has been, primarily my own personal time.

If I would've known a good number of people were asking for WITH
RECURSIVE (as Josh mentioned), I would've had more incentive to work
on it.

Again, the original patch for this has been out since 7.3.4, and no
one has seen fit to work on this feature in 3 years. If it's a
feature the community seriously wanted, maybe someone should've said
something. A couple of you said it would be nice to have, but I don't
recall seeing any community-oriented, "we *really* want this feature
in 8.2" (similar to other features). I would've gladly supported
anyone else who wanted to do it.

I do have a family and by the time I get to work on something at home
it's approaching 23:00; the last thing I want to do is spend a few
hours coding something no one seems to want. Similarly, it's even
harder to get motivated when the people complaining about a missing
feature are those who didn't seem to want it to begin with.

I don't know what you want unless you tell me and I've got better
things to do than play a psychic.

-Jonah

The community cannot ask anyone to work harder. What we do ask is that
if you start working on an item, let us know, and if you stop working on
it, let us know soon so others can work on it.

Also, if something is on the TODO list, the community doesn't need to
shoot signal rockets to tell people it is important. The fact it is on
the TODO list indicates it is significant, unless you are told
otherwise. Shifting blame on an incomplete feature to the community
doesn't help anyone.

---------------------------------------------------------------------------

Jonah H. Harris wrote:
> On 8/4/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Not to be unkind, but AFAIR all the unmet expectations in this release
> > cycle came from commercially-sponsored developers who said
> > they'd do X and then didn't finish it.
>
> FYI, I am not commercially sponsered. I am a full-time employee
> devoted to working on database internals, drivers, Oracle
> compatibility, architecture, prototyping, and some performance stuff
> for EnterpriseDB.
>
> While I work full-time for EnterpriseDB, they have ALWAYS given me
> time to work on other PostgreSQL-related things when I ask for it.
> However, with the exception of the INSERT/UPDATE RETURNING patch
> (which EnterpriseDB asked me to submit), the PostgreSQL community is,
> and always has been, primarily my own personal time.
>
> If I would've known a good number of people were asking for WITH
> RECURSIVE (as Josh mentioned), I would've had more incentive to work
> on it.
>
> Again, the original patch for this has been out since 7.3.4, and no
> one has seen fit to work on this feature in 3 years. If it's a
> feature the community seriously wanted, maybe someone should've said
> something. A couple of you said it would be nice to have, but I don't
> recall seeing any community-oriented, "we *really* want this feature
> in 8.2" (similar to other features). I would've gladly supported
> anyone else who wanted to do it.
>
> I do have a family and by the time I get to work on something at home
> it's approaching 23:00; the last thing I want to do is spend a few
> hours coding something no one seems to want. Similarly, it's even
> harder to get motivated when the people complaining about a missing
> feature are those who didn't seem to want it to begin with.
>
> I don't know what you want unless you tell me and I've got better
> things to do than play a psychic.
>
> -Jonah
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> BTW, while I'm thinking about it, I believe INSERT ... RETURNING is in,
> no?

There's a recently-submitted patch, but it's not been reviewed yet,
so it's premature to say "it's in". See upthread comments about
promising things in advance of them hitting CVS ...

regards, tom lane

On 8/4/06, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> Also, if something is on the TODO list, the community doesn't need to
> shoot signal rockets to tell people it is important. The fact it is on
> the TODO list indicates it is significant, unless you are told
> otherwise.

True, but stating that you want certain items in the TODO list done
for a certain release is another thing. Being on the TODO list has
nothing to do with if/when a TODO item will be worked-on or released.

> Shifting blame on an incomplete feature to the community
> doesn't help anyone.

I don't believe any one is at fault.

-Jonah

On Fri, Aug 04, 2006 at 10:27:49AM -0700, Joe Conway wrote:
> Tom Lane wrote:
> >"Luke Lonergan" <llonergan(at)greenplum(dot)com> writes:
> >
> >>UPDATE/DELETE for CE are a big deal - I really wish we had INSERT too,
> >
> >Huh? We had INSERT working before, that's why it's not mentioned.
>
> I think what Luke means, is that an INSERT into the base table of the
> inheritance hierarchy with CE enabled gets automagically inserted into
> the correct partition, and as far as I know that still doesn't happen --
> but I've certainly been wrong before :-).

Likewise, I suspect that the UPDATE/DELETE constraint elimination don't
do anything 'magical' either; merely figure out what partitions can
safely be ignored for a statement. While that's a good feature to have,
it's far short of being able to automatically 'route' data to the
correct partitions.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Fri, Aug 04, 2006 at 02:12:16PM -0400, Stephen Frost wrote:
> * Jim C. Nasby (jnasby(at)pervasive(dot)com) wrote:
> > On Thu, Aug 03, 2006 at 11:20:48PM -0700, Josh Berkus wrote:
> > > > * In-place upgrades (pg_upgrade)
> > >
> > > BTW, I may get Sun to contribute an engineer for this; will get you posted.
> >
> > How would such a thing handle changes to page formats?
>
> Couldn't this be done by converting a table/partial-table at a time?
> It wouldn't be something which could run while the system is live, but
> it'd probably take less time than dump/restore and wouldn't require
> double the disk space of the whole database... no?

True, but if you're going to go about creating code that can deal with 2
different versions of on-disk data, why not go one better: put that code
into the database itself, so that pages are converted on-the-fly as
they're dirtied. That way you have *no* downtime (or almost no, anyway).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Fri, Aug 04, 2006 at 03:18:37PM -0400, Tom Lane wrote:
> "Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> > BTW, while I'm thinking about it, I believe INSERT ... RETURNING is in,
> > no?
>
> There's a recently-submitted patch, but it's not been reviewed yet,
> so it's premature to say "it's in". See upthread comments about
> promising things in advance of them hitting CVS ...

True, but I didn't remember seeing it in your list of
yet-to-be-committed patches, so I thought I'd mention it.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

On Fri, Aug 04, 2006 at 12:03:59PM -0400, Bruce Momjian wrote:
> Andrew Dunstan wrote:
> > Tom Lane wrote:
> >
> > >Not that there's anything wrong with a performance-oriented
> > >release ... but if you think that 8.2 is short on features, you'd
> > >better get ready to be disappointed by every future release.
> >
> > It's a pity that some expectations have been raised about features
> > that we haven't seen patches for, e.g. MERGE and/or some form of
> > UPSERT, and recursive queries. I am not pointing fingers, but I do
> > think we need some way in which the community can ensure that
> > certain goals are met, or at least try to help if things fall in a
> > ditch, rather than just relying on hackers scratching whatever
> > itch they happen to get in splendid isolation and then trying to
> > merge the results.
>
> What we do is when people claim items, we monitor them to be sure
> they get them done for the current release, or at least give it
> their best try. There is not much more we can do.

While I am not going to reopen the can of worms labeled 'bug tracker',
I think it would be good to have a little more formality as far as
claiming items goes.

I'm picturing something like this:

1. Each person taking an item agrees to write at least one email each
week to -hackers detailing progress or lack of same on the item.

2. Should someone wish to relinquish a claim on a feature, there needs
to be some standard way to do a hand-off of whatever they've
done/found and announce that the feature is now available to others to
claim.

3. Should the person claiming the feature not communicate to -hackers
for some period--I'm thinking 3 weeks is about right--the item goes
back in the unclaimed pool with a message to -hackers saying that
that's what's happened.

What say?

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

Jonah H. Harris wrote:
> On 8/4/06, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > Also, if something is on the TODO list, the community doesn't need to
> > shoot signal rockets to tell people it is important. The fact it is on
> > the TODO list indicates it is significant, unless you are told
> > otherwise.
>
> True, but stating that you want certain items in the TODO list done
> for a certain release is another thing. Being on the TODO list has
> nothing to do with if/when a TODO item will be worked-on or released.

True. But with better communication, we might be able to have had
someone else work on this.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Jonah,

> If I would've known a good number of people were asking for WITH
> RECURSIVE (as Josh mentioned), I would've had more incentive to work
> on it.

You didn't ask. If you had asked, you would have got a response.

People knew you were working on it, and assumed that it would be done,
since you're a "full time" developer and a very fast programmer.

The first time I became aware that you weren't expecting to finish was the
PostgreSQL Anniversary. I expect that a lot of other people weren't
aware of it until earlier in this thread.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

> I'm picturing something like this:
>
> 1. Each person taking an item agrees to write at least one email each
> week to -hackers detailing progress or lack of same on the item.
>
> 2. Should someone wish to relinquish a claim on a feature, there needs
> to be some standard way to do a hand-off of whatever they've
> done/found and announce that the feature is now available to others to
> claim.
>
> 3. Should the person claiming the feature not communicate to -hackers
> for some period--I'm thinking 3 weeks is about right--the item goes
> back in the unclaimed pool with a message to -hackers saying that
> that's what's happened.
>
> What say?

I say that many people have been shouting on deaf ears about this topic
for a long time and that I back an idea to change our incredibility
inefficient policies on todo items.

We are like the last bastion of anarchy, refusing to except that we are
in fact a recognized and important open source project that has
responsibilities to its community.

Those responsibilities include better communication, feature tracking
and milestones...

but alas... again the walls of the canyon echo until there was nothing
but the sound of water and crickets.

Sincerely,

Joshua D. rake

>
> Cheers,
> D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

* Jim C. Nasby (jnasby(at)pervasive(dot)com) wrote:
> On Fri, Aug 04, 2006 at 02:12:16PM -0400, Stephen Frost wrote:
> > * Jim C. Nasby (jnasby(at)pervasive(dot)com) wrote:
> > > How would such a thing handle changes to page formats?
> >
> > Couldn't this be done by converting a table/partial-table at a time?
> > It wouldn't be something which could run while the system is live, but
> > it'd probably take less time than dump/restore and wouldn't require
> > double the disk space of the whole database... no?
>
> True, but if you're going to go about creating code that can deal with 2
> different versions of on-disk data, why not go one better: put that code
> into the database itself, so that pages are converted on-the-fly as
> they're dirtied. That way you have *no* downtime (or almost no, anyway).

Certainly a good idea but I'm really not sure that: a) we'd want the
core server to have essentially cruft code and b) that it'd be anywhere
near simple to make this actually work in the core system.

Supporting two versions of the page format when the only goal is
changing the formatting is quite a different thing than fully supporting
two on-disk formats throughout the backend. Additionally, if you have
something to convert the database wholesale then that really just needs
to support 2 versions at a time, if it's piecemeal then you could end up
in a situation where you have to support 3, 4, 5, however many prior
page formats existed. If not then you have to have a way to force all
the pages to be upgraded and you're unlikely to be able to do a whole
lot more with the system while that's happening, thus adding alot of
extra complication just to end up having everyone follow the
'pg_upgrade'-style path anyway...

In the end though, I think once it's been done in pg_upgrade this could
be revisited and considered and if enough people are interested in doing
it they could probably reuse some of hte pg_upgrade code to make it
happen.

Thanks,

Stephen

On Fri, Aug 04, 2006 at 07:45:56AM -0700, Joshua D. Drake wrote:
>
> >* Several varieties of replication, which I believe we as a project
> > will eventually endorse and ship
>
> This one will cause confusion regardless of how much advocacy,
> documentation and will power we put into it.

It will, but I think we'll eventually have to get there if for no
other reason than that every other DBMS on earth would have it.

> >* On-the-fly in-line calls out to PL/your_choice without needing to
> >issue DDL
>
> You mean something like: EXEC plperl(print $foo)?

Something like this:

SELECT a, b, c
FROM (
EXECUTE IMMEDIATE
LANGUAGE plperl
$$...$$
) AS (a int, b point, c text)
JOIN ...

Anyhow, the idea is to be able to call PL functionality in-line
without having to create a function in advance.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

>>> issue DDL
>> You mean something like: EXEC plperl(print $foo)?
>
> Something like this:
>
> SELECT a, b, c
> FROM (
> EXECUTE IMMEDIATE
> LANGUAGE plperl
> $$...$$
> ) AS (a int, b point, c text)
> JOIN ...
>
> Anyhow, the idea is to be able to call PL functionality in-line
> without having to create a function in advance.

Well that is certainly interesting. Opens up a whole lot of flexibility,
especially to views and such.

Seems a little unnatural though ;)

Joshua D. Drake

>
> Cheers,
> D

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Fri, Aug 04, 2006 at 01:41:42PM -0700, Joshua D. Drake wrote:
> >>>issue DDL
> >>You mean something like: EXEC plperl(print $foo)?
> >
> >Something like this:
> >
> >SELECT a, b, c
> >FROM (
> > EXECUTE IMMEDIATE
> > LANGUAGE plperl
> > $$...$$
> >) AS (a int, b point, c text)
> >JOIN ...
> >
> >Anyhow, the idea is to be able to call PL functionality in-line
> >without having to create a function in advance.
>
> Well that is certainly interesting. Opens up a whole lot of flexibility,
> especially to views and such.
>
> Seems a little unnatural though ;)

That's the table constructor version. It would be nice to call other
versions, too, like

SELECT
CALL IMMEDIATE
LANGUAGE plpython $$...$$ (a, LOWER(b))
AS wacky_python_output
FROM ...

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

On Fri, Aug 04, 2006 at 12:40:01PM -0700, David Fetter wrote:
> While I am not going to reopen the can of worms labeled 'bug tracker',
> I think it would be good to have a little more formality as far as
> claiming items goes.

Agreed.

> I'm picturing something like this:
>
> 1. Each person taking an item agrees to write at least one email each
> week to -hackers detailing progress or lack of same on the item.
>
> 2. Should someone wish to relinquish a claim on a feature, there needs
> to be some standard way to do a hand-off of whatever they've
> done/found and announce that the feature is now available to others to
> claim.
>
> 3. Should the person claiming the feature not communicate to -hackers
> for some period--I'm thinking 3 weeks is about right--the item goes
> back in the unclaimed pool with a message to -hackers saying that
> that's what's happened.
>
> What say?

It's a shame to have a person burn cycles on this, but anything would be
an improvement over what we've got now.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim C. Nasby wrote:
> On Fri, Aug 04, 2006 at 12:40:01PM -0700, David Fetter wrote:
> > While I am not going to reopen the can of worms labeled 'bug tracker',
> > I think it would be good to have a little more formality as far as
> > claiming items goes.
>
> Agreed.
>
> > I'm picturing something like this:
> >
> > 1. Each person taking an item agrees to write at least one email each
> > week to -hackers detailing progress or lack of same on the item.
> >
> > 2. Should someone wish to relinquish a claim on a feature, there needs
> > to be some standard way to do a hand-off of whatever they've
> > done/found and announce that the feature is now available to others to
> > claim.
> >
> > 3. Should the person claiming the feature not communicate to -hackers
> > for some period--I'm thinking 3 weeks is about right--the item goes
> > back in the unclaimed pool with a message to -hackers saying that
> > that's what's happened.
> >
> > What say?
>
> It's a shame to have a person burn cycles on this, but anything would be
> an improvement over what we've got now. --------

Really?

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, 2006-08-04 at 12:40 -0700, David Fetter wrote:
> While I am not going to reopen the can of worms labeled 'bug tracker',
> I think it would be good to have a little more formality as far as
> claiming items goes.

> What say?

I think this is a good plan for adding additional process overhead, and
getting essentially nothing of value in return. I'm not convinced
there's a problem in need of solving here...

-Neil

>>> What say?
>> It's a shame to have a person burn cycles on this, but anything would be
>> an improvement over what we've got now. --------
>
> Really?

I lot of this could be automated with a web app. The web app takes the
todo, a hacker signs up. Hacker takes todo. Web app reminds hacker every
6, 8 weeks to post an update or release the todo.

Every 6/8 weeks we get an automated email to pgsql-hackers that states
the current status of the todo list.
Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Fri, Aug 04, 2006 at 02:37:56PM -0700, Neil Conway wrote:
> On Fri, 2006-08-04 at 12:40 -0700, David Fetter wrote:
> > While I am not going to reopen the can of worms labeled 'bug
> > tracker', I think it would be good to have a little more formality
> > as far as claiming items goes.
>
> > What say?
>
> I think this is a good plan for adding additional process overhead,
> and getting essentially nothing of value in return. I'm not
> convinced there's a problem in need of solving here...

Perhaps you'd like to explain how big a burden on the developer it is
to send an once a week, that being what I'm proposing here.

As far as the "problem in need of solving," it's what Andrew Dunstan
referred to as "splendid isolation," which is another way of saying,
"letting the thing you've taken on gather dust while people think
you're working on it."

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

David Fetter wrote:

> On Fri, Aug 04, 2006 at 02:37:56PM -0700, Neil Conway wrote:
>> On Fri, 2006-08-04 at 12:40 -0700, David Fetter wrote:
>>> While I am not going to reopen the can of worms labeled 'bug
>>> tracker', I think it would be good to have a little more formality
>>> as far as claiming items goes.
>>> What say?
>> I think this is a good plan for adding additional process overhead,
>> and getting essentially nothing of value in return. I'm not
>> convinced there's a problem in need of solving here...
>
> Perhaps you'd like to explain how big a burden on the developer it is
> to send an once a week, that being what I'm proposing here.
>
> As far as the "problem in need of solving," it's what Andrew Dunstan
> referred to as "splendid isolation," which is another way of saying,
> "letting the thing you've taken on gather dust while people think
> you're working on it."

Well I guess the key thing is that its transparent who is working on
what feature and that someone (some projects have a new release manager
for every release) keeps tabs on these people (maybe once every few
weeks he asks how things are progressing, if the target can still be
made etc). Maybe this can even be enumerated into a percentage value
that can be published so that people who feel a given feature is very
important to them can see if things are lagging behind.

Actually the person keeping track of these feature developments does not
need to be a pgsql hacker. As a matter of fact I have been trying to do
something similar for the PHP project and I am more than willing to do
the same for PostgreSQL. I do not however think it makes sense to ask
developers to send in reports in a regular basis on their own.

regards,
Lukas

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:

> Those responsibilities include better communication, feature tracking and
> milestones...

Wow, if we had all those we could have as efficient a release-engineering
process as Mozilla!

>
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>
>> Those responsibilities include better communication, feature tracking
>> and
>> milestones...
>
> Wow, if we had all those we could have as efficient a release-engineering
> process as Mozilla!
>
>

This is not really a good argument. Might it not be possible that there is
a sweeter spot somewhere in the middle? I don't think anyone wants
something very heavy handed.

cheers

andrew

Folks,

> This is not really a good argument. Might it not be possible that there
> is a sweeter spot somewhere in the middle? I don't think anyone wants
> something very heavy handed.

Well, I think the answer is just to set something up and see if people can
use it. If we keep kibitzing about it, the grousing will go on forever.

Mind you, I'm not thinking a bug tracker, but just a task manager for the
TODOs. BTW, I've road-tested GForge's task manager and it's not adequate
for what we need.

--
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

This has been a very interesting thread, if for no other reason then
to just catalog all of the changes going into 8.2. I am going to be
changing some hardware around so I need to decide if I want to a)
change the hardware now and don't bother with 8.2, b) wait to upgrade
hardware and do the upgrade to 8.2 at the same time, c) upgrade the
hardware now and then upgrade to 8.2 when it is released.

What I do basically depends on how much benefit I'm going to get from
8.2 and whether it's worth planning my hardware upgrades around or
incurring additional downtime in order to do the postgres upgrade.
Doing a dump/reload is not a problem at all, it doesn't take that
long with the data I've got. It's just having to kick everyone off
the system and shut everything down in order to do it.

My applications are not that demanding on the DB so there isn't
anything that I "NEED" that postgres doesn't already have. Because
of this I stayed on 7.3 for way, way too long. It just didn't seem
worth the effort to do the upgrade or the additional testing against
each postgres version to make sure it didn't break my apps. But I
finally bit the bullet and upgraded straight to 8.1. I couldn't
believe what a huge performance difference it made even though I
didn't use any of the big headlining features. (ie PITR, two-phase
commit, etc). So I'm sure that even though I don't even understand
what most of the items on toms huge list are, and I certainly don't
understand where they will come into play for me, I'm sure that once
I've upgraded and I use it for a while, I'll be very glad I did.

That being said I think that Bruce has a point in that there is a
certain class of features (which seems hard to define) which existed
in 8.0 and 8.1 but not 8.2. I would define it like this:

A) There are features that PHBs would look for.
B) There are features that a casual database user would look for.
C) There are features that an experienced database user would look
for / understand
D) and then there are features that a database guru/developer of
database software would understand.

Features of class A contribute to the bullet point lists, the buzz
word checklists, etc and are easily used for marketing. For instance
if someone were to package up postgres and put it in a box you would
not have a list of "new features in 8.2!" that included "lazy vacuums
are ignored by other processes". Only an existing postgres user who
has had a setup complex enough to need a tricky vacuuming strategy
would ever even know what that meant much less care about it. It
might be life and death for many users, but it doesn't go on the back
of the box.

So if you define "major features" as class A features. In this case
major doesn't mean important or useful or difficult to implement,
just that they are the sort of features that one might be told to
look for when shopping for a database. So in terms of marketing
PITR, two phase commit, WIN32 support were very much "major" features.

If people had expectations that are not being met it could be because
8.0 and 8.1 had so many of these headlining, market friendly,
buzzword compliant "major" features. It doesn't make it any less
impressive technically or less useful for the actual users of the
database but it DOES make a difference as to how this release will be
perceived by a lot of people. Not that it's a problem, but many
people I think will see this release as less "Major" than the 8.0 or
8.1 releases. I think this is reflected in the fact that 8.0 was
picked as the version to jump up to 8.0 instead of 7.5.

I will upgrade at some point but mostly because experience has taught
me that each release of postgres is significantly better than the
last one, even if I won't see how until I actually use it in production.

That being said I think that two of the not-yet-reviewed features are
just as "major" as the "major" features from the past two releases.

1) updatable views - I won't really use this but it just seems like
one of those features that people use when doing rdbms features
comparison charts. I think that having updatable views will be
considered by the masses to be a "major features"

2) restartable recovery (allow checkpoints for a hot-standby server)
- Having the ability to have a hot standby database is a HUGE feature
in my book. Maybe I'm just biased because it's a feature that I
actually need but I think it's very big. Yes you can sort of do the
same thing or something "better" with slony but this is what I really
want (and I'm guessing that this will get used A LOT if it makes it
in). And it's "bulit in" unlike slony. And it seems much easier to
set up and maintain and less likely to have problems or complicate
things than slony. In terms of having a setup with no "single point
of failure" this goes a long way. And it builds on something that I
will want set up anyway (PITR).

Anyway this is just my $0.02 as a fairly average to low end, user of
postgres as to how this issue may be perceived by the masses.

Thanks in advance for what will certainly be another great release.

On Aug 3, 2006, at 10:46 PM, Bruce Momjian wrote:

> Tom Lane wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>> To me new things are like PITR, Win32, savepoints, two-phase commit,
>>> partitioned tables, tablespaces. These are from 8.0 and 8.1.
>>> What is
>>> there in 8.2 like that?
>>
>> [ shrug... ] Five out of your six items have no basis in the SQL
>> spec.
>> So it's not clear to me what your definition of "major feature" is,
>> unless maybe it's "anything except what we did for 8.2". Can you
>> enumerate ten things you would consider comparable to the above
>> features
>> that aren't done yet?
>
> No, I cannot. I do think our missing list is shrinking. My point is
> that you really couldn't easily work around the 8.0/8.1 items I listed
> if they were missing, while the 8.2 items could be more easily
> worked-around. Again, nothing wrong with that.
>
> --
> Bruce Momjian bruce(at)momjian(dot)us
> EnterpriseDB http://www.enterprisedb.com
>
> + If your life is a hard drive, Christ can be your backup. +
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>

Gregory Stark wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>
>> Those responsibilities include better communication, feature tracking and
>> milestones...
>
> Wow, if we had all those we could have as efficient a release-engineering
> process as Mozilla!

Thanks for the sarcasm. Anything productive to add?

How about this... We can use Trac for the TODO list. It allows for
assigning of todos, status, difficulty ratings, gives us a wiki for
collaborative features etc...

We even have one that is already pulling the pgsql repo for review.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Lukas Smith <smith(at)pooteeweet(dot)org> writes:

> David Fetter wrote:
>
> > On Fri, Aug 04, 2006 at 02:37:56PM -0700, Neil Conway wrote:
> >> On Fri, 2006-08-04 at 12:40 -0700, David Fetter wrote:
> >>> While I am not going to reopen the can of worms labeled 'bug
> >>> tracker', I think it would be good to have a little more formality
> >>> as far as claiming items goes.
> >>> What say?
>
> >> I think this is a good plan for adding additional process overhead,
> >> and getting essentially nothing of value in return. I'm not
> >> convinced there's a problem in need of solving here...
>
> > Perhaps you'd like to explain how big a burden on the developer it is
> > to send an once a week, that being what I'm proposing here.

There seems to be a lack of recognition here of how free software development
works. When people are contributing their time scratching an itch for their
own edification the LAST thing they want is to have a manager to report to.

> > As far as the "problem in need of solving," it's what Andrew Dunstan
> > referred to as "splendid isolation," which is another way of saying,
> > "letting the thing you've taken on gather dust while people think
> > you're working on it."

Really you guys are talking as if the developers that are working for the most
part on an entirely volunteer basis have some sort of responsibility to you.
They do not. If they don't feel like tell you where they're at then feel free
to ask for your money back.

Now if you think you have some tool that will make it easier for developers to
do something they honestly want to do then feel free to suggest it and make it
available. But if you want to dictate how programmers work for the gain of
others you're going to have a hard time swimming against the current.

Now not all the developers working on Postgres are unpaid volunteers. But the
ones that aren't have their own managers to report to already and their own
timelines and other responsibilities to deal with. They're not being paid to
meet yours.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

On Fri, Aug 04, 2006 at 08:44:24PM -0400, Gregory Stark wrote:
> > > Perhaps you'd like to explain how big a burden on the developer
> > > it is to send an once a week, that being what I'm proposing
> > > here.
>
> There seems to be a lack of recognition here of how free software
> development works. When people are contributing their time
> scratching an itch for their own edification the LAST thing they
> want is to have a manager to report to.

It it were--note the subjunctive--only for their own edification, that
would be true. It's not. It's from a TODO list that a community put
together, and claiming a TODO means taking an opportunity away from
one or more other parties.

> > > As far as the "problem in need of solving," it's what Andrew
> > > Dunstan referred to as "splendid isolation," which is another
> > > way of saying, "letting the thing you've taken on gather dust
> > > while people think you're working on it."
>
> Really you guys are talking as if the developers that are working
> for the most part on an entirely volunteer basis have some sort of
> responsibility to you. They do not.

When they've taken on a TODO, they most certainly have taken on a
responsibility to the entire Postgres community.

> If they don't feel like tell you where they're at then feel free to
> ask for your money back.

How do we get the time that somebody frittered away by grabbing a TODO
when others could have worked on it? Time is a much more precious
resource than money because no matter what you do, you can't get any
more of it.

> Now if you think you have some tool that will make it easier for
> developers to do something they honestly want to do then feel free
> to suggest it and make it available.

Joshua Drake has made Trac available. Perhaps you weren't here for
the extensive, drawn-out, rancorous discussions that surrounded a bug
tracking system, but they're all over the archives.

> But if you want to dictate how programmers work for the gain of
> others you're going to have a hard time swimming against the
> current.

It's for their own gain, too. People who contribute good-sized
features to PostgreSQL can generally choose their place of work and to
a large degree, their salaries, so let's not get too excited about how
this is only a matter of selfless generosity.

Cheers,
D
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
phone: +1 415 235 3778 AIM: dfetter666
Skype: davidfetter

Remember to vote!

> There seems to be a lack of recognition here of how free software
> development
> works. When people are contributing their time scratching an itch for
> their
> own edification the LAST thing they want is to have a manager to report
> to.
>

I am sick of hearing lectures on this. It is simply NOT true that free
software follows a single model of development. There are many projects
and all have their own methods, some formal and some not very formal. But
the idea that there is ONE method is simply nonsense. You really ought to
know better.

>> > As far as the "problem in need of solving," it's what Andrew Dunstan
>> > referred to as "splendid isolation," which is another way of saying,
>> > "letting the thing you've taken on gather dust while people think
>> > you're working on it."
>
> Really you guys are talking as if the developers that are working for the
> most
> part on an entirely volunteer basis have some sort of responsibility to
> you.
> They do not. If they don't feel like tell you where they're at then feel
> free
> to ask for your money back.
>
> Now if you think you have some tool that will make it easier for
> developers to
> do something they honestly want to do then feel free to suggest it and
> make it
> available. But if you want to dictate how programmers work for the gain of
> others you're going to have a hard time swimming against the current.
>
> Now not all the developers working on Postgres are unpaid volunteers. But
> the
> ones that aren't have their own managers to report to already and their
> own
> timelines and other responsibilities to deal with. They're not being paid
> to
> meet yours.
>

Really, you are suggesting that a volunteer coimmunity is incapable of
actually organising itself in any coherent fashion, and that only money
will motivate people to act with any sense of responsibility to others.
FYI I have never got a penny for the work I have done on Postgres. I would
not object to an occasional query about items I had undertaken to
implement.

Greg, you are on an utterly wrong track here. Try to look about a bit more
broadly.

cheers

andrew

On Fri, 2006-08-04 at 15:44 -0700, David Fetter wrote:
> As far as the "problem in need of solving," it's what Andrew Dunstan
> referred to as "splendid isolation," which is another way of saying,
> "letting the thing you've taken on gather dust while people think
> you're working on it."

I'm just not convinced this is a problem. We've seen *one* example of
this happening in recent times (hierarchical queries) -- but even with
better communication, it's not like there were hordes of folks
volunteering to hack on hierarchical queries who just idled because they
thought it was already being completed.

If people are interested in the status of a patch, I think it's fine for
them to email the person who's volunteered to work on it. If there
hasn't been public activity from that person in recent times, it is
probably a reasonable bet that the work has stalled. Formalizing the
process strikes me as a waste of time, and IMHO would mostly serve to
irritate the folks working on these features.

-Neil

Rick Gigger <rick(at)alpinenetworking(dot)com> writes:
> So if you define "major features" as class A features. In this case
> major doesn't mean important or useful or difficult to implement,
> just that they are the sort of features that one might be told to
> look for when shopping for a database. So in terms of marketing
> PITR, two phase commit, WIN32 support were very much "major" features.

You have a point: 8.0 and 8.1 had much more buzzword-compliant stuff
added. The truth of the matter is that a lot of that stuff was pretty
rough-edged in actual use, and now we're starting to smooth it out and
make it more readily usable. So in terms of *usable* PITR etc we're
only now getting there with 8.2. But that's not a bullet point that
impresses PHBs.

> That being said I think that two of the not-yet-reviewed features are
> just as "major" as the "major" features from the past two releases.

> 1) updatable views - I won't really use this but it just seems like
> one of those features that people use when doing rdbms features
> comparison charts.

Agreed, if this gets in it will be a Real Biggie. I de-emphasized it
in my list because I haven't looked at the patch yet and so have no
idea whether it's any good, but I fully agree it's a PHB-worthy
bullet point if it works.

> 2) restartable recovery (allow checkpoints for a hot-standby server)
> - Having the ability to have a hot standby database is a HUGE feature
> in my book.

Again, we claimed to have hot standby in 8.1, and we sort of did, it
just didn't work all that nicely. This will file down one seriously
rough edge, but is that a good marketing bullet point? Probably not.

regards, tom lane

> There seems to be a lack of recognition here of how free software development
> works. When people are contributing their time scratching an itch for their
> own edification the LAST thing they want is to have a manager to report to.

I have heard you make this argument before, and it is just is not true.
Even Debian is moving toward a more formal structure as has FreeBSD. You
seem stuck in this world where everything is still 1994 and all FOSS
software is developed in academia.

http://www.freebsd.org/doc/en_US.ISO8859-1/articles/releng/
http://lwn.net/Articles/144281/
http://developer.kde.org/documentation/other/checklist.html

>
> Now not all the developers working on Postgres are unpaid volunteers. But the
> ones that aren't have their own managers to report to already and their own
> timelines and other responsibilities to deal with. They're not being paid to
> meet yours.

No one is expecting them to. However we are expecting some communication
about it. When you start developing with a community you have a
responsibility to that community. That means you let people know if you
are not going to finish something, if you need help, if you can't help,
or if you are going to bail on a project. You should also do so with
(hopefully) the ability for someone to pick up where you left off.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

andrew(at)dunslane(dot)net writes:

> > There seems to be a lack of recognition here of how free software
> > development works. When people are contributing their time scratching an
> > itch for their own edification the LAST thing they want is to have a
> > manager to report to.
>
> I am sick of hearing lectures on this. It is simply NOT true that free
> software follows a single model of development. There are many projects
> and all have their own methods, some formal and some not very formal. But
> the idea that there is ONE method is simply nonsense. You really ought to
> know better.

Well strictly speaking I blurred a distinction in that not all free software
projects are projects are volunteer projects at all. Projects like Mozilla and
Emacs are driven by a single entity with paid developers. But excluding
projects like that can you name a single project with a "formal" development
process that it requires its developers to follow?

And I'll note that my earlier sarcasm had a point. Projects that, while they
may be free software, are driven by a single commercial entity frequently
drive away any volunteer contributors with their onerous formal development
methodologies. Mozilla in particular spent years angsting over why they were
failing to attract any community contributors.

I'm involved in a fairly large number of free software projects, and have been
involved in a great deal more in the past. Certainly all projects use various
tools to help. But where they do it's because those tools make their lives
easier. Not because it makes it easier for other people watching them.

If you phrase your argument in terms of milestones and progress updates and so
on and how those things will help other people make use of the work then
you're just going to find people continue to ignore it. The way to phrase the
argument if you want to win people over is by figuring out how your tools will
make the developers' lives easier.

--
greg

Neil Conway <neilc(at)samurai(dot)com> writes:
> If people are interested in the status of a patch, I think it's fine for
> them to email the person who's volunteered to work on it. If there
> hasn't been public activity from that person in recent times, it is
> probably a reasonable bet that the work has stalled. Formalizing the
> process strikes me as a waste of time, and IMHO would mostly serve to
> irritate the folks working on these features.

I tend to agree --- I don't see much value in trying to institute a
formalized process. We have not had that many cases where lack of
communication was a problem.

In the case at hand (hierarchical queries), I believe what happened was
that Jonah expected that he'd be able to complete the feature with a
relatively small amount of work by starting with the existing patch.
When he got into it he realized that a complete rewrite was needed, and
there was no time to get that done for 8.2. He told a few of us at the
anniversary conference, and perhaps should have been more proactive
about mentioning the problem on the mailing list, but *it would have
made no difference* as to whether hierarchical queries got into 8.2.
There was not anyone else in a position to invest the work needed in
the time available, either.

regards, tom lane

andrew(at)dunslane(dot)net writes:
> Greg, you are on an utterly wrong track here. Try to look about a bit more
> broadly.

FWIW, I tend to agree with Greg. This project has gotten to where it is
with a very loose structure, and I think that trying to impose more
structure carries a significant risk of breaking the cooperative
dynamics that have worked so well for us so far. In short, I'm not sure
that we should try to fix something that isn't clearly broken.

I don't object to someone informally polling people who have claimed a
TODO item and not produced any visible progress for awhile. But I think
anything like "thou shalt report in once a week" will merely drive
people away from publicly claiming items, if not drive them away from
doing anything at all. We've already got far too much problem with
lack of visibility, in the sense that people pop up with patches after
not having told anyone they were working on a given problem (much less
posted a preliminary design for feedback, as I desperately wish people
would do before starting to code anything). We should encourage people
to say "I'm working on X", and I fear that putting requirements on them
as soon as they say that will mostly serve to keep them from saying
anything.

regards, tom lane

On Friday 04 August 2006 13:56, mdean wrote:
> Josh, percentages, like almost anything, do work in the right context,
> in this case, that of the testimonial, something postgresql hasn't
> emphasized IMHO. If ten to 20 projects were treated as real and
> realistic case studies, with an in-depth description of the project, and
> how the NEW Postgresql effected these projects,and these were featured
> one after another, daily for an entire month, there would be major play
> in the news. By talking in depth about specific projects, we relate to
> the actual production users own daily experiences, especiually if % can
> be translated into dollars.
>

The problem has never been the idea of doing more case studies, but has been
of finding someone to do it. Writing case studies isn't exactly exciting to
you're average programmer, and in most cases you'd rather have them hacking
code anyway.

> I am just brainstorming, I lack the db expertise, but real life stroies
> are powerful creatures with a life of their own.

Sure. Can you drum us up some advertising interns who are willing to work on
case studies for us? We surely have people who are interested in
participating.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

> I tend to agree --- I don't see much value in trying to institute a
> formalized process. We have not had that many cases where lack of
> communication was a problem.

How do you know? Seriously... this comes up at least twice a year :).
There is probably a basis for it.

As I was saying on #postgresql, the current system works well for a
small group of developers. I don't think there is any arguing that.

However, there is a larger group out there, that would likely be willing
to contribute but we are a bit of a black box, or perhaps we are too
transparent?? I am not sure which.

Frankly, I don't care if we ever get a bug tracker or use trac. However
a more formalized communication process is sorely needed IMHO.

I am actually hoping that jabber.postgresql.org would help that in the
long run.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> As I was saying on #postgresql, the current system works well for a
> small group of developers. I don't think there is any arguing that.

> However, there is a larger group out there, that would likely be willing
> to contribute but we are a bit of a black box, or perhaps we are too
> transparent?? I am not sure which.

Maybe I'm too much "on the inside", but I see the problem the other way
'round: too little visibility of what's being done by people who are not
part of the inner circle of developers.

It's possible that creating a more formal structure would aid these folk
to let the rest of us know what they're doing ... but I think it's at
least as likely that a more formal structure would just drive them away.

> I am actually hoping that jabber.postgresql.org would help that in the
> long run.

Now here I think we might be on the same page. If people pop up on IRC
or jabber or any other communication method and talk about what they're
doing, that fixes the whole problem. I'm for adding anything that
provides an opportunity for people to talk to the community --- I'm not
for trying to force them to talk to the community, 'cause I don't think
that will work very well.

regards, tom lane

> I don't object to someone informally polling people who have claimed a
> TODO item and not produced any visible progress for awhile. But I think
> anything like "thou shalt report in once a week" will merely drive
> people away from publicly claiming items, if not drive them away from
> doing anything at all.

Well a weekly report is probably ridiculous but as I said in an earlier
post it would be fairly trivial to allow communication via web site and
have that site keep tabs on people every 6-8 weeks.

Heck we could even set it up to feed off of email as well.. for example:

todo_173(at)postgresql(dot)org

Hey... just letting you guys know that I am still moving forward. My
current list is:

A
B
C

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

On Friday 04 August 2006 21:19, Tom Lane wrote:
> Rick Gigger <rick(at)alpinenetworking(dot)com> writes:
> > So if you define "major features" as class A features. In this case
> > major doesn't mean important or useful or difficult to implement,
> > just that they are the sort of features that one might be told to
> > look for when shopping for a database. So in terms of marketing
> > PITR, two phase commit, WIN32 support were very much "major" features.
>
> You have a point: 8.0 and 8.1 had much more buzzword-compliant stuff
> added. The truth of the matter is that a lot of that stuff was pretty
> rough-edged in actual use, and now we're starting to smooth it out and
> make it more readily usable. So in terms of *usable* PITR etc we're
> only now getting there with 8.2. But that's not a bullet point that
> impresses PHBs.
>
> > That being said I think that two of the not-yet-reviewed features are
> > just as "major" as the "major" features from the past two releases.
> >
> > 1) updatable views - I won't really use this but it just seems like
> > one of those features that people use when doing rdbms features
> > comparison charts.
>
> Agreed, if this gets in it will be a Real Biggie. I de-emphasized it
> in my list because I haven't looked at the patch yet and so have no
> idea whether it's any good, but I fully agree it's a PHB-worthy
> bullet point if it works.
>

Hmm.. I would de-emphasize it because it doesn't give us give us anything we
couldn't do before; and really what we can do now is way above most database
systems.

> > 2) restartable recovery (allow checkpoints for a hot-standby server)
> > - Having the ability to have a hot standby database is a HUGE feature
> > in my book.
>
> Again, we claimed to have hot standby in 8.1, and we sort of did, it
> just didn't work all that nicely. This will file down one seriously
> rough edge, but is that a good marketing bullet point? Probably not.
>

So, the things I hear most non-postgresql people complain about wrt postgresql
are:

no full text indexing built in
no replication built in
no stored procedures (with a mix of wanting in db cron facility)
the planner is not smart enough (with a mix of wanting hints)
vacuum leads to unpredictable performance

Of that list, they could probably all be turned into nice marketing points
(though #4 is pretty nebulous), though I don't see any of them getting
resolved anytime soon.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

On 8/4/06, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> In the case at hand (hierarchical queries), I believe what happened
> was ...

Tom is exactly correct, this is what happened.

In the many months I've stated to be working on it, I've only had one
other person (Mark Cave-Ayland) interested in assisting with the
patch.

Based on the need to completely rewrite it, neither he nor I would
probably be able to get this done correctly for 8.2 without a fairly
good amount of design. I would gladly assist anyone that wants to
work on a new patch.

--
Jonah H. Harris, Software Architect | phone: 732.331.1300
EnterpriseDB Corporation | fax: 732.331.1301
33 Wood Ave S, 2nd Floor | jharris(at)enterprisedb(dot)com
Iselin, New Jersey 08830 | http://www.enterprisedb.com/

> I don't object to someone informally polling people who have claimed a
> TODO item and not produced any visible progress for awhile. But I think
> anything like "thou shalt report in once a week" will merely drive
> people away from publicly claiming items, if not drive them away from
> doing anything at all.

The former is much more what I had in mind than the latter. Let's do that.

cheers

andrew

Robert Treat <xzilla(at)users(dot)sourceforge(dot)net> writes:
> On Friday 04 August 2006 21:19, Tom Lane wrote:
>> Rick Gigger <rick(at)alpinenetworking(dot)com> writes:
>>> 1) updatable views - I won't really use this but it just seems like
>>> one of those features that people use when doing rdbms features
>>> comparison charts.
>>
>> Agreed, if this gets in it will be a Real Biggie.

> Hmm.. I would de-emphasize it because it doesn't give us give us anything we
> couldn't do before; and really what we can do now is way above most database
> systems.

Well, you know that and I know that, but no PHB knows that --- and
what's more important, the average journeyman database developer
probably doesn't know how to write those rules either. This feature
isn't about doing something that couldn't be done at all before; it is
about bringing a pretty useful behavior within the reach of mere mortals.

regards, tom lane

andrew(at)dunslane(dot)net writes:
>> I don't object to someone informally polling people who have claimed a
>> TODO item and not produced any visible progress for awhile. But I think
>> anything like "thou shalt report in once a week" will merely drive
>> people away from publicly claiming items, if not drive them away from
>> doing anything at all.

> The former is much more what I had in mind than the latter. Let's do that.

Like I said, no objection here. But who exactly is "we" --- ie, who's
going to do the legwork? We surely don't want multiple people pestering
the same developer ...

regards, tom lane

>
> It's possible that creating a more formal structure would aid these folk
> to let the rest of us know what they're doing ... but I think it's at
> least as likely that a more formal structure would just drive them away.
>
Well just some informal from the internal workings of CMD. Some
of this won't work with the community because of QA but...

I pretty much leave Alvaro, Alexey and Andrei (odd just realized the
three As)
alone. I watch the timeline on track to see what is going on. Once a week
I will touch base just to get the team flow going.

About once a month Alvaro will post a status to the lists that gives me
a general idea of his direction and goals. Alvaro keeps tabs on Alexey
and Andrei so that all development is done in the style of PostgreSQL.

None of this is hard and fast, and it gives the developers the cat like
feeling
they like but allows me to be comfortable with the business requirement
side.

Obviously the business side doesn't apply here, but the "checkin" type does.

One of the items I have been toying with for us (the community) is mentors.
Basically you have a green programmer, or even a seasoned one... there are
plenty lurking on the lists. Someone like Alvaro, Neil, or Andrew offers
to mentor
a particular feature such as Enums.

Basically Alvaro or Neil could offer guidance and answer questions,
maybe be
a little more readily available for that particular email address.

Alvaro does that with Alexey and Andrei... helping insure appropriate
style,
coding standards, giving feedback on particular implementations.

Yes people can get that on the lists, but if they felt they were getting
a little
more direct attention, it might help retain them to the community.

>> I am actually hoping that jabber.postgresql.org would help that in the
>> long run.
>
> Now here I think we might be on the same page. If people pop up on IRC
> or jabber or any other communication method and talk about what they're
> doing, that fixes the whole problem. I'm for adding anything that
> provides an opportunity for people to talk to the community --- I'm not
> for trying to force them to talk to the community, 'cause I don't think
> that will work very well.
Cool well that is nice to know, I didn't get a whole lot of feedback on
the idea.

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

Robert Treat wrote:
> So, the things I hear most non-postgresql people complain about wrt postgresql
> are:
>
> no full text indexing built in
> no replication built in
> no stored procedures (with a mix of wanting in db cron facility)
> the planner is not smart enough (with a mix of wanting hints)
> vacuum leads to unpredictable performance

FTI is a biggie in my mind. I know it ain't happening for 8.2, but is
the general plan to integrate TSearch2 directly into the backend?

I had a few thoughts on this issue:

The objective is to smoothly upgrade to the new version with minimal
downtime.

The different proposals as far as I can see are as follows:

Proposal A - the big one time reformatting
1) shutdown the db
2) run a command that upgrades the data directory to the new format
3) start up the new postgres version with the new data dir

Pros: only pg_upgrade (or whatever it's called) needs to know about
the old and new formats, each version of postgres knows about "it's"
format and that's it. The postgres code stays clean
cons: your database is down while the upgrade takes place. This
sucks because the people who need this are the same people who are
trying to avoid downtime. It's faster than a dump/reload but it
doesn't completely solve the problem, it just mitigates it.

Proposal B - the gradual upgrade
1) shutdown the db
2) start it back up with the new version of postgres
3) the new postgres version upgrades things in place as needed

Pros: very short downtime. only the time to shutdown the postgres
version and start up the new one
cons: postgres code gets filled with cruft. each version has to know
about the old versions on disk data format and how to upgrade it.
Until it is finished you will be left with a database that is part
old format, part new format. This could introduce bugs for people
who never needed the feature in the first place.

Here is another proposal that I haven't heard anyone else suggest.
My apologies in advance if it's obviously not workable or has already
be discussed.

Proposal C - PITR with in on the fly disk upgrades
1) setup PITR
2) run pg_upgrade on your latest backed up data directories
3) start up the new pg on that data directory in restartable
recovery / read-only / hot-standby mode
4) update the recovery log importer so that it can update the log
files on the fly as it applies them
5) failover to the hot standby as you normally would

Pros: essentially no downtime, just any incidental time needed for
the failover to occur.
cruft in postgres main codebase is mimimized. It's limited to the
log importer. All other parts of postgres are unaffected
Cons: requires another server or double the disk space on the
original server. Is this a problem for people with databases so
large that a dump reload is unacceptable?
Perhaps there are technical issues with postgres that I don't
understand that would make this too hard.
Maybe it would take to long to update each log file as it's applied
so it wouldn't be able to catch up.

Oh yeah there's another way
Proposal D - Use slony
But I figured since that's been working for a long time, if slony
solved their problem then they wouldn't be looking for something else.

I have no need for this feature as a dump reload is not a problem for
me. I've always wondered though if that was a feasible answer to
this problem. Each time it crops up people propose solutions A and B
but never C.

On Aug 4, 2006, at 1:30 PM, Jim C. Nasby wrote:

> On Fri, Aug 04, 2006 at 02:12:16PM -0400, Stephen Frost wrote:
>> * Jim C. Nasby (jnasby(at)pervasive(dot)com) wrote:
>>> On Thu, Aug 03, 2006 at 11:20:48PM -0700, Josh Berkus wrote:
>>>>> * In-place upgrades (pg_upgrade)
>>>>
>>>> BTW, I may get Sun to contribute an engineer for this; will get
>>>> you posted.
>>>
>>> How would such a thing handle changes to page formats?
>>
>> Couldn't this be done by converting a table/partial-table at a time?
>> It wouldn't be something which could run while the system is live,
>> but
>> it'd probably take less time than dump/restore and wouldn't require
>> double the disk space of the whole database... no?
>
> True, but if you're going to go about creating code that can deal
> with 2
> different versions of on-disk data, why not go one better: put that
> code
> into the database itself, so that pages are converted on-the-fly as
> they're dirtied. That way you have *no* downtime (or almost no,
> anyway).
> --
> Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
> Pervasive Software http://pervasive.com work: 512-231-6117
> vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

If people are going to start listing features they want here's some
things I think would be nice. I have no idea though if they would be
useful to anyone else:

1) hierarchical / recursive queries. I realize it's just been
discussed at length but since there was some question as to whether
or not there's demand for it so I am just weighing in that I think
there is. I have to deal with hierarchy tables all the time and I
simply have several standard methods of dealing with them depending
on the data set / format. But they all suck. I've just gotten use
to using the workarounds since there is nothing else. If you are not
hearing the screams it's just because I think it's just become a fact
of life for most people (unless you're using oracle) that you've just
got to work around it. And everyone already has some code to do this
and they've already done it everywhere it needs to be done. And as
long as you're a little bit clever you can always work around it
without taking a big performance hit. But it would sure be nice to
have next time I have to deal with a tree table.

2) PITR on a per database basis. I think this would be nice but I'm
guessing that the work involved is big and that few people really
care or need it, so it will probably never happen.

3) A further refinement of PITR where some sort of deamon ships small
log segments as they are created so that the hot standby doesn't have
to be updated in 16MB increments or have to wait for some timeout to
occur. It could always be up to the minute data.

4) All the Greenplum Bizgress MPP goodness. In reality (and I don't
know if bizgress mpp can actually do this) I'd like to have a cluster
of cheap boxes. I'd like to install postgres on all of them and
configure them in such a way that it automatically partitions and
mirrors each table so that each piece of data is always on two boxes
and large tables and indexes get divided up intelligently. Sort of
like a raid10 on the database level. This way any one box could die
and I would be fine. Enormous queries could be handled efficiently
and I could scale up by just dropping in new hardware.

Maybe greeenplum has done this. Maybe we will get their changes soon
enough, maybe not. Maybe this sort of functionality will never
happen. My guess is that all the little bit's a pieces of this will
trickle in over the next several years and this sort of setup will be
slowly converged on over time as lot's of little things come
together. Table spaces and constraint exclusion come to mind here as
things that could eventually evolve to contribute to a larger solution.

5) Somehow make it so I NEVER HAVE TO THINK ABOUT OR DEAL WITH VACUUM
AGAIN. Once I get everything set up right everything works great but
I'm sure if there's one thing I think everyone would love it would be
getting postgres to the point where you don't even need to ship
vacuumdb because there's no way the user could outsmart postgres's
attempts to do garbage collection on it's own.

6) genuine updatable views. such that you just add an updatable
keyword when you create the view and it's automagically updatable.
I'm guessing that we'll get something like that, but its real magic
will be throwing an error to tell you when you try to make a view
updatable and it can't figure out how to make the rules properly.

7) allow some way to extract the data files from a single database
and insert them into another database cluster. In many cases it
would be a lot faster to copy the datafiles across the network than
it is to dump, copy dump file, reload.

8) some sort of standard "hooks" to be used for replication. I guess
when the replication people all get their heads together and tell the
core developers what they all need something like this could evolve.

Like I said, postgres more than satisfies my "needs". I am
especially happy when you factor in the cost of the software (free),
and the quality of the community support (excellent).

And you can definitely say that the "missing" list is shrinking. But
I think of it like this. There are tiers of database functionality
that different people need:
A) Correct me if I'm wrong but as great as postgres is there are
still people out there that MUST HAVE Oracle or DB2 to get done what
they need to get done. They just do things that the others can't.
They may be expensive. They may suck to use and administer but the
simple fact is that they have features that people need that are not
offered in less expensive databases.
B) Very, very powerful databases but lack the biggest, most
complicated "enterprise" features.
C) Light weight db for taking care of the basic need to store data
and query it with sql. (some would call these "toy" databases)
D) databases which are experimental, unreliable or have other limits
that make them not practical compared with the other options

I would say that with version 7.0 postgres moved from D to C (please
don't get offended if this is way off base, I never used 6.x but I
heard it was prone to crashes, data corruption and of course there
was that pesky row size limit). It then proceeded to move up within
tier C to become the best of it's class and pushing up into level B.
With 8.0 it was firmly in level B. It was fast, efficient, powerful
and began adding lots of really, really big features like PITR,
savepoints, tablespaces, etc. Add ons like slony also allowed it to
be used in places where it otherwise wouldn't have measured up.

Now there are only a few features left in the B range and so there
are tons of situations that can be taken care of by postgres now that
were out of it's reach just a few years ago. Once those features are
all gone there will still be some very big, very difficult features
on the table that once completed will begin to remove any advantage
that the really big guys have. I'm thinking especially of #4 above
here. But they will definitely take a while.

I may have tons of details wrong here but my point is that I think
that postgres isn't just taking stuff off a big to do list, but
rather is pushing itself upwards and is now in a position to start
working on some very hard problems that once completed will put it
into a very elite class of database systems. The "missing" list for
tier B type problems is shrinking down to almost nothing and items
from the tier A missing list are starting to come into view.

Maybe I'm way off base here but that's how I see it. Postgres has
come a long, long way, but the problems ahead are bigger and meaner
than the ones behind.

On Aug 4, 2006, at 12:02 AM, David Fetter wrote:

> On Fri, Aug 04, 2006 at 12:37:10AM -0400, Tom Lane wrote:
>> Bruce Momjian <bruce(at)momjian(dot)us> writes:
>>> To me new things are like PITR, Win32, savepoints, two-phase
>>> commit, partitioned tables, tablespaces. These are from 8.0 and
>>> 8.1. What is there in 8.2 like that?
>>
>> [ shrug... ] Five out of your six items have no basis in the SQL
>> spec. So it's not clear to me what your definition of "major
>> feature" is, unless maybe it's "anything except what we did for
>> 8.2". Can you enumerate ten things you would consider comparable to
>> the above features that aren't done yet?
>
> First, I'd like to say people are doing a fantastic job here. Kudos!
>
> One huge thing missing from the "done" list is that crucial bit of
> infrastructure and process that has shortened feedback loops--hence
> the beta period--by weeks if not months: the build farm. It's now
> smoothly integrated into the development process, and as a
> consequence, we can realistically have a release each year. :)
>
> As far as big missing features go, here's a short list:
>
> * Splitting queries among CPUs--possibly even among machines--for OLAP
> loads
>
> * In-place upgrades (pg_upgrade)
>
> * Several varieties of replication, which I believe we as a project
> will eventually endorse and ship
>
> * CALL
>
> * WITH RECURSIVE
>
> * MERGE
>
> * Windowing functions
>
> * On-the-fly in-line calls out to PL/your_choice without needing to
> issue DDL
>
> * Wild-eyed feral bits of the SQL standard like SQL/MED and SQL/XML
>
> But all that leaves out the oldest, most honored Postgres tradition:
>
> Breaking New Ground.
>
> We're definitely not done yet. :)
>
> Cheers,
> D
> --
> David Fetter <david(at)fetter(dot)org> http://fetter.org/
> phone: +1 415 235 3778 AIM: dfetter666
> Skype: davidfetter
>
> Remember to vote!
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
>

Joshua D. Drake wrote:
> Frankly, I don't care if we ever get a bug tracker or use trac.
> However a more formalized communication process is sorely needed
> IMHO.

There's also supposed to be a wiki set up. There, people can try to
make up tracking lists, project management, task lists, release goals
or whatever on their own. If patterns emerge, we can "formalize" them,
but I feel this would be a good way to try things out.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

> andrew(at)dunslane(dot)net writes:
>>> I don't object to someone informally polling people who have claimed a
>>> TODO item and not produced any visible progress for awhile. But I
>>> think
>>> anything like "thou shalt report in once a week" will merely drive
>>> people away from publicly claiming items, if not drive them away from
>>> doing anything at all.
>
>> The former is much more what I had in mind than the latter. Let's do
>> that.
>
> Like I said, no objection here. But who exactly is "we" --- ie, who's
> going to do the legwork? We surely don't want multiple people pestering
> the same developer ...
>

Perl has its pumpking ... maybe we need a designated "holder of the
trunk". I see that as a Core function.

cheers

andrew

On Sat, Aug 05, 2006 at 12:19:54AM -0400, Matthew T. O'Connor wrote:
> Robert Treat wrote:
> >So, the things I hear most non-postgresql people complain about wrt
> >postgresql are:
> >
> >no full text indexing built in
>
> FTI is a biggie in my mind. I know it ain't happening for 8.2, but is
> the general plan to integrate TSearch2 directly into the backend?

When the Tsearch developers say so I think. This will be the first
major release with GIN which will form the basis of future releases of
tsearch. IIRC they have a whole list of features they still want to add
before it gets included...

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

Tom Lane wrote:
> But a quick troll through the CVS logs shows ...
>
> multi-row VALUES, not only for INSERT but everywhere SELECT is allowed ...
> multi-argument aggregates, including SQL2003-standard statistical aggregates ...
> standard_conforming_strings can be turned on (HUGE deal for some people) ...
> support SQL-compliant row comparisons; they can be indexscan quals

ISTM this could be spun as a standards-focused release as well (at least
partial implementations of a number of optional(?) SQL2003 features).

On Fri, Aug 04, 2006 at 06:25:35PM -0700, Joshua D. Drake wrote:
> I have heard you make this argument before, and it is just is not true.
> Even Debian is moving toward a more formal structure as has FreeBSD. You
> seem stuck in this world where everything is still 1994 and all FOSS
> software is developed in academia.

Debian moving towards a more formal structure? What I seeing is that
they're trying to get away from the having one person responsible for
things to working in groups. What it amounts to is simplifying the
rules to doing someone elses work. People who don't like it leave and
you hope you're left with a more efficient group.

The links you provide are mostly about handling releases. To be honest,
I think PostgreSQL's release handling is fine. But none of those
projects tackles the issue of making sure certain things get done. If
someone didn't do the work for getting GCC 4.1 working for Debian, then
no matter how much of a release goal it was, it wouldn't happen...

> That means you let people know if you
> are not going to finish something, if you need help, if you can't help,
> or if you are going to bail on a project. You should also do so with
> (hopefully) the ability for someone to pick up where you left off.

That I can agree with, but I don't think you can force it.

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> From each according to his ability. To each according to his ability to litigate.

andrew(at)dunslane(dot)net wrote:
> > andrew(at)dunslane(dot)net writes:
> >>> I don't object to someone informally polling people who have claimed a
> >>> TODO item and not produced any visible progress for awhile. But I
> >>> think
> >>> anything like "thou shalt report in once a week" will merely drive
> >>> people away from publicly claiming items, if not drive them away from
> >>> doing anything at all.
> >
> >> The former is much more what I had in mind than the latter. Let's do
> >> that.
> >
> > Like I said, no objection here. But who exactly is "we" --- ie, who's
> > going to do the legwork? We surely don't want multiple people pestering
> > the same developer ...
> >
>
> Perl has its pumpking ... maybe we need a designated "holder of the
> trunk". I see that as a Core function.

I can assure you that individual developers were contacted about
completing their items for 8.2, to the extent that some developers got
upset at me because of my insistence. If they were hired by PostgreSQL
companies and I had a relationship with their manager, their managers
were informed as well.

Jonah, who said the community wasn't clear it wanted his items
completed, was part of that group. I see no need to mention the other
people I contacted. Many of them completed their items, and Jonah
finished some of his items.

The fact is, the existing system worked as it should, though it is often
invisible. We didn't get all the features we wanted, but that isn't
because the system isn't working.

--
Bruce Momjian bruce(at)momjian(dot)us
EnterpriseDB http://www.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

> There's also supposed to be a wiki set up. There, people can try to
> make up tracking lists, project management, task lists, release goals
> or whatever on their own. If patterns emerge, we can "formalize" them,
> but I feel this would be a good way to try things out.

Well I will re-extend my offer to put up a trac site for everyone which
does contain a wiki. However, last time I offered I believe Marc was
actually going to do it.

Sincerely,

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Bruce Momjian wrote:

>I can assure you that individual developers were contacted about
>completing their items for 8.2, to the extent that some developers got
>upset at me because of my insistence. If they were hired by PostgreSQL
>companies and I had a relationship with their manager, their managers
>were informed as well.
>
>Jonah, who said the community wasn't clear it wanted his items
>completed, was part of that group. I see no need to mention the other
>people I contacted. Many of them completed their items, and Jonah
>finished some of his items.
>
>The fact is, the existing system worked as it should, though it is often
>invisible. We didn't get all the features we wanted, but that isn't
>because the system isn't working.
>
>
>

Thank you Bruce. That is good to know. Maybe the invisibility has led me
astray. I'll shut up now and see if I can actually get Enums and some
other good stuff done by this time next year. With any luck I won't be
quite as derailed as I was last cycle.

Also, I hope it's now clear at least that there are many people who want
to see recursive queries.

cheers

andrew

> The fact is, the existing system worked as it should, though it is often
> invisible. We didn't get all the features we wanted, but that isn't
> because the system isn't working.

Well that kind of comes back to my point of better communication.
Perhaps a lot of this discussion could have been avoided if the TODO had
been more... proactive?

For example:

Make postmater and postgres options distinct so the postmaster -o
option is no longer needed | PeterE | Confirmed for 8.2 | 07/20/06

I *think* it was peter that did that one, but you see my point.

Joshua D. Drake

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Martijn van Oosterhout <kleptog(at)svana(dot)org> writes:
> On Sat, Aug 05, 2006 at 12:19:54AM -0400, Matthew T. O'Connor wrote:
>> FTI is a biggie in my mind. I know it ain't happening for 8.2, but is
>> the general plan to integrate TSearch2 directly into the backend?

> When the Tsearch developers say so I think.

Yeah, that's my take too. Oleg and Teodor obviously feel it's not "done"
yet, and ISTM leaving it in contrib gives them more flexibility in a
couple of ways:
* they can make user-visible API changes without people getting as upset
as if they were changing core features;
* because it is a removable contrib module, they can (and do) offer
back-ports of newer versions to existing PG release branches.

I think some descendant of tsearch2 will eventually be in core, but
we'll wait till we're pretty certain it's feature-stable.

regards, tom lane

Neil, all:

> If people are interested in the status of a patch, I think it's fine for
> them to email the person who's volunteered to work on it.

The problem I would like to see resolved is that there is currently no
accurate way to determine who is working on a patch except by comprehensive
-hackers, -patches, and -performance archive reading. This is a little
daunting for people who just joined the community, or who are users just
wanting to know if someone is working on a feature they want.

I doubt that any TODO system would have 100% participation, and I know that it
would depend on having some non-hacker volunteers updating the information on
behalf of developers who didn't want to use it. However, I think that
getting those volunteers is entirely possible (for example, PWN is inculding
a weekly patch list and it's not much more effort to check off those patches
against a web-based TODO list). If the system reflected 70% of current
development activity, then I think it would be a big improvement over the
current "read 100% of the mail archives for three mailing lists back one year
to find out what's going on."

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Rick,

> The objective is to smoothly upgrade to the new version with minimal
> downtime.

Thanks for jumping in.

> The different proposals as far as I can see are as follows:
>
> Proposal A - the big one time reformatting
> 1) shutdown the db
> 2) run a command that upgrades the data directory to the new format
> 3) start up the new postgres version with the new data dir
>
> Pros: only pg_upgrade (or whatever it's called) needs to know about
> the old and new formats, each version of postgres knows about "it's"
> format and that's it. The postgres code stays clean
> cons: your database is down while the upgrade takes place. This
> sucks because the people who need this are the same people who are
> trying to avoid downtime. It's faster than a dump/reload but it
> doesn't completely solve the problem, it just mitigates it.

Yes, but it mitigates it *considerably.* The general idea is that doing an
in-place binary upgrade should take 80% less time, and require only 10% (or
so) extra disk space. This would be sufficient for most users with large
databases; they can afford to be down from midnight to 3 am but not to be
down for the whole weekend (as dump/reload requires for a 3tb database) nor
do they have 100% + extra disk space and 20% extra CPU on the machine (as
upgrade-by-replication requires).

Plus, there are versions (like 8.2 looks to be) where there is *no* change in
the file format. For these, pg_upgrade would need just to bootstrap the
system, swap the system tables and header files, and restart. Seconds
instead of hours. As PostgreSQL matures further, I predict that there will
be more releases with no change in the file format, making this even more
useful.

> Proposal B - the gradual upgrade
> 1) shutdown the db
> 2) start it back up with the new version of postgres
> 3) the new postgres version upgrades things in place as needed

This is impractical, because the number of version-to-version compatibility
libraries required will grow geometrically with time. We don't have to just
accomodate the last version, but the last 4+ versions.

> Proposal C - PITR with in on the fly disk upgrades
> 1) setup PITR
> 2) run pg_upgrade on your latest backed up data directories
> 3) start up the new pg on that data directory in restartable
> recovery / read-only / hot-standby mode
> 4) update the recovery log importer so that it can update the log
> files on the fly as it applies them
> 5) failover to the hot standby as you normally would

I don't think this is practical. Simon?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus wrote:

> I doubt that any TODO system would have 100% participation, and I know that it
> would depend on having some non-hacker volunteers updating the information on
> behalf of developers who didn't want to use it. However, I think that
> getting those volunteers is entirely possible (for example, PWN is inculding
> a weekly patch list and it's not much more effort to check off those patches
> against a web-based TODO list). If the system reflected 70% of current
> development activity, then I think it would be a big improvement over the
> current "read 100% of the mail archives for three mailing lists back one year
> to find out what's going on."

Yup, thats exactly what I experienced when I started a TODO list for the
PHP project. A few developers wanted to get direct access, others use me
as their proxy and then others do not care about the list at all. But it
has been very helpful to the community to hear whats going on. It has
let to some people joining the development or suggesting possible
implementation options and more importantly it has reduced the work load
of the release managers. This is not a task that requires a lot of
technical know how and is therefore a great way for some non C hacker to
contribute to the project.

regards,
Lukas

Tom Lane wrote:
> I tend to agree --- I don't see much value in trying to institute a
> formalized process.

One more problem with the formalized process of claiming features
in advance may stop what I suspect is a significant source of
contributions -- people who add features/patches for internal
work in their company and only after the fact find that they
are something they'd contribute back.

The small contribution I made (to help admins know when FSM
settings were too low by monitoring log files instead
of manual checks[1]) was done because we wanted it internally.

Only after it proved useful to us, it was mentioned to the lists.

Thanks in part to the BSD nature of postgresql, I suspect
there are many internal-and-not-yet-released useful patches
lurking around in industry. If I'm right, I'd wonder what
the advocacy guys could do to get corporations to volunteer
to contribute changes back that they've found useful
internally.

> We have not had that many cases where lack of
> communication was a problem.

One could say too much communication was the problem this time.

I get the impression people implied they'd do something on a TODO
and didn't. Arguably the project had been better off if noone
had claimed the TODO, so if another company/team/whatever needed
the feature badly, they could have worked on it themselves rather
than waiting in hope of the feature. Of course they could have
done this anyway - but if they see it on an implied roadmap document
for the next release they're more likely to wait.

Ron

[1] http://archives.postgresql.org/pgsql-patches/2005-02/msg00171.php