Re: Release Note Changes

Few proposals

- Can we say "smoothed" rather than "distributed" checkpoints?
"Smoothed checkpoints greatly reduce checkpoint I/O spikes"

- Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
change to
"Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"

I also notice that two performance features have disappeared from the
release notes. (Presumably they have been removed from source). Both of
them have changes that can be seen by users, so can't see why we would
want them removed.

- Merge Join performance has been substantially improved by ring buffer
which avoids materializing the previous sort step. (Simon, Greg)

More info, not for release notes:
The materialization of the prior sort step would generally double the
time taken for the sort, so avoiding this effectively gives a 50%
performance gain on sorts that are part of large merge joins.

- WAL file switches don't update controlfile any longer. Recovery now
refers to the last checkpoint time, which may be many minutes earlier
than time previously mentioned. (Simon, Tom)

More info, not for release notes:
WAL file switches were performed holding important LWLocks, so this
improves scalability on high end systems as well as reducing response
time spikes under heavy load on all kinds of hardware.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

On Fri, 2007-11-30 at 06:31 +0000, Simon Riggs wrote:

> I also notice that two performance features have disappeared from the
> release notes. (Presumably they have been removed from source). Both of
> them have changes that can be seen by users, so can't see why we would
> want them removed.

Wow, just realised 3 of Heikki's performance patches aren't mentioned
either:

- CheckpointStartLock removal

- I/O reduction during recovery

- Tuning of Visibility code

I'm not sure what the rationale is for not mentioning these things.
They're at least as important, if not more so, than mentioning minor
source code changes.

If people understand there aren't 13 performance improvements there are
at *least* 19+ that is a positive message to help people decide to
upgrade.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:

> If people understand there aren't 13 performance improvements there are
> at *least* 19+ that is a positive message to help people decide to
> upgrade.

Frankly I think the release notes are already too long. People who judge a
release by counting the number of items in the release notes are not worth
appeasing. Including every individual lock removed or code path optimized will
only obscure the important points on which people should be judging the
relevance of the release to them. Things like smoothing checkpoint i/o which
could be removing a show-stopper problem for them.

If they're mentioned at all a single release note bullet point saying "Many
optimizations and concurrency improvements in areas such as transaction start
and finish, checkpoint start, record visibility checking, merge join plans,
..." would suffice.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

On 11/30/07, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
>
>
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
>
> > If people understand there aren't 13 performance improvements there are
> > at *least* 19+ that is a positive message to help people decide to
> > upgrade.
>
> Frankly I think the release notes are already too long. People who judge a
> release by counting the number of items in the release notes are not worth
> appeasing. Including every individual lock removed or code path optimized
> will
> only obscure the important points on which people should be judging the
> relevance of the release to them. Things like smoothing checkpoint i/o
> which
> could be removing a show-stopper problem for them.
>
> If they're mentioned at all a single release note bullet point saying
> "Many
> optimizations and concurrency improvements in areas such as transaction
> start
> and finish, checkpoint start, record visibility checking, merge join
> plans,
> ..." would suffice.
>
> --
> Gregory Stark
> EnterpriseDB http://www.enterprisedb.com
> Ask me about EnterpriseDB's 24x7 Postgres support!
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>

i agree that release notes should not be too long, but may be there should
be (if there isn't one already) something like a "change log" where people
can find out all the changes done from the previous release, if they are
intrested ?

--
Usama Munir Dar http://linkedin.com/in/usamadar
Consultant Architect
Cell:+92 321 5020666
Skype: usamadar

On Fri, 2007-11-30 at 09:49 +0000, Gregory Stark wrote:
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
>
> > If people understand there aren't 13 performance improvements there are
> > at *least* 19+ that is a positive message to help people decide to
> > upgrade.
>
> Frankly I think the release notes are already too long.

So why do we have stuff in there that the users will never see?

We already have a release summary, so why summarise *some* of the detail
as well, but not all of it???

I see no reason to diminish yours, Heikki's or my own contributions, all
of which were in the area of performance, which people do care about.
None of the ones I mentioned were trivial patches, nor were their
effects small.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Simon Riggs wrote:
>
> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
> change to
> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
>
>
>

I think we need to qualify this, or it could be quite misleading.
perhaps add "that don't affect indexed columns" or something like that.

cheers

andrew

On Nov 30, 2007 4:49 AM, Gregory Stark <stark(at)enterprisedb(dot)com> wrote:
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > If people understand there aren't 13 performance improvements there are
> > at *least* 19+ that is a positive message to help people decide to
> > upgrade.
>
> Frankly I think the release notes are already too long. People who judge a
> release by counting the number of items in the release notes are not worth
> appeasing. Including every individual lock removed or code path optimized will
> only obscure the important points on which people should be judging the
> relevance of the release to them. Things like smoothing checkpoint i/o which
> could be removing a show-stopper problem for them.

IMO, it's probably good to include things that materially affect how
people operate the databse. An example is improvements to statistics
gathering because it eliminates a historical trade-off in configuring
the server. I agree with you regarding basic operations though.

merlin

Simon Riggs wrote:
> On Fri, 2007-11-30 at 06:31 +0000, Simon Riggs wrote:
>
>> I also notice that two performance features have disappeared from the
>> release notes. (Presumably they have been removed from source). Both of
>> them have changes that can be seen by users, so can't see why we would
>> want them removed.
>
> Wow, just realised 3 of Heikki's performance patches aren't mentioned
> either:
>
> - CheckpointStartLock removal

I don't think it's worth mentioning, given that we have the Load
Distributed Checkpoints in there. That alone will tell people that
there's been some major changes to checkpoints.

> - I/O reduction during recovery

This might be worth mentioning, since it can be quite a big difference
in the right circumstances, and it helps a bit with the scalability
problem of the recovery. Should mention that it only helps with
full_pages_writes=on. One more reason to not gamble with data integrity ;-).

> - Tuning of Visibility code

I don't think that was release notes worthy.

The release notes are quite long already...

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

"Usama Dar" <munir(dot)usama(at)gmail(dot)com> writes:
> i agree that release notes should not be too long, but may be there should
> be (if there isn't one already) something like a "change log" where people
> can find out all the changes done from the previous release, if they are
> intrested ?

The CVS history (either direct from the CVS server, or in the
pgsql-committers archives) will give you as much detail as you could
possibly want.

regards, tom lane

Greg,

> Frankly I think the release notes are already too long. People who judge a
> release by counting the number of items in the release notes are not worth
> appeasing. Including every individual lock removed or code path optimized
> will only obscure the important points on which people should be judging
> the relevance of the release to them. Things like smoothing checkpoint i/o
> which could be removing a show-stopper problem for them.

I disagree. For people who want a quick summary of the major user-facing
things changed we'll have multiple sources: (a) the announcement, (b) the
press features list, (c) the Feature-Version matrix. The Release notes
should have a *complete* list of changes.

Why? Because we don't use a bug/feature tracker. So a user trying to figure
out "hey, was my issue XXX fixed so that I should upgrade?" has *no other
source* than the Release notes to look at, except CVS history. And if we
start asking sysadmins and application vendors to read the CVS history, we're
gonna simply push them towards other DBMSes which have this information more
clearly.

If we want to shorten the release notes, then we should adopt an issue
tracker.

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Heikki,

> This might be worth mentioning, since it can be quite a big difference
> in the right circumstances, and it helps a bit with the scalability
> problem of the recovery. Should mention that it only helps with
> full_pages_writes=on. One more reason to not gamble with data integrity
> ;-).

Does this mean that recovery from logs with full_page_writes will be faster
than recovery from logs without them?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> I disagree. For people who want a quick summary of the major user-facing
> things changed we'll have multiple sources: (a) the announcement, (b) the
> press features list, (c) the Feature-Version matrix. The Release notes
> should have a *complete* list of changes.

Define "complete".

> Why? Because we don't use a bug/feature tracker. So a user trying to
> figure out "hey, was my issue XXX fixed so that I should upgrade?" has
> *no other source* than the Release notes to look at, except CVS
> history. And if we start asking sysadmins and application vendors to
> read the CVS history, we're gonna simply push them towards other
> DBMSes which have this information more clearly.

So in other words, you don't *really* want "complete".

This discussion is all about finding a suitable balance between length
and detail. Simplistic pronouncements don't help us strike that
balance.

FWIW, I tend to agree with the folks who think Bruce trimmed too much
this time. But the release notes are, and always have been, intended to
boil the CVS history down to something useful by eliminating irrelevant
detail. For the vast majority of people, the details that are being
mentioned here are indeed irrelevant. There will be some for whom they
are not. But depending on the question, almost any detail might not be
irrelevant, and at that point you have to be prepared to go check the
archives.

regards, tom lane

On Nov 30, 2007 11:07 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> > I disagree. For people who want a quick summary of the major
> user-facing
> > things changed we'll have multiple sources: (a) the announcement, (b)
> the
> > press features list, (c) the Feature-Version matrix. The Release notes
> > should have a *complete* list of changes.
>
> Define "complete".
>
> > Why? Because we don't use a bug/feature tracker. So a user trying to
> > figure out "hey, was my issue XXX fixed so that I should upgrade?" has
> > *no other source* than the Release notes to look at, except CVS
> > history. And if we start asking sysadmins and application vendors to
> > read the CVS history, we're gonna simply push them towards other
> > DBMSes which have this information more clearly.
>
> So in other words, you don't *really* want "complete".

i think he means a list meant for end users which mentions all features and
bug fixes done for that release. Your argument of go read the CVS logs is
valid, but there are just too many for someone to go through to get the
complete picture. i mean people may end up reading 1000 + logs in a worst
case scenario to find out if a bug they are interested in is fixed , and the
someone who compiled the release notes didn't think it was important enough
to make it to the notes. Going through a 5K release notes document would be
half that time, granted that over time thier ability to read through logs
quicker will improve, but thats a learning curve they have to be willing to
go trough, and not everyone will be interested to do that

if i would have to find a word to describe what we need, i would say we need
something *compendious* i.e. what is at once full in scope and brief and
concise in treatment

it is however work that someone will have to do, but it can be managed as
such that it is a by-product of the process, instead of a 'one time in the
end' job.

>
>
> This discussion is all about finding a suitable balance between length
> and detail. Simplistic pronouncements don't help us strike that
> balance.
>
> FWIW, I tend to agree with the folks who think Bruce trimmed too much
> this time. But the release notes are, and always have been, intended to
> boil the CVS history down to something useful by eliminating irrelevant
> detail. For the vast majority of people, the details that are being
> mentioned here are indeed irrelevant. There will be some for whom they
> are not. But depending on the question, almost any detail might not be
> irrelevant, and at that point you have to be prepared to go check the
> archives.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

--
Usama Munir Dar http://linkedin.com/in/usamadar
Consultant Architect
Cell:+92 321 5020666
Skype: usamadar

Josh Berkus wrote:
>> This might be worth mentioning, since it can be quite a big difference
>> in the right circumstances, and it helps a bit with the scalability
>> problem of the recovery. Should mention that it only helps with
>> full_pages_writes=on. One more reason to not gamble with data integrity
>> ;-).
>
> Does this mean that recovery from logs with full_page_writes will be faster
> than recovery from logs without them?

In general, yes. Depends a lot on how randomly the data in the WAL is
distributed, speed of reading from WAL etc.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Fri, 30 Nov 2007 06:32:10 -0500
Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

>
>
> Simon Riggs wrote:
> >
> > - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
> > change to
> > "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
> >
> >
> >
>
> I think we need to qualify this, or it could be quite misleading.
> perhaps add "that don't affect indexed columns" or something like
> that.

Heap Only Tuples (HOT) improves performance for heavy update tables
where the column being updated isn't indexed?

Seems kind of long but isn't that "exactly" what it does?

Sincerely,

Joshua D. Drake

>
> cheers
>
> andrew
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 1: if posting/reading
> through Usenet, please send an appropriate subscribe-nomail command
> to majordomo(at)postgresql(dot)org so that your message can get through to
> the mailing list cleanly
>

On Fri, 2007-11-30 at 13:07 -0500, Tom Lane wrote:

> FWIW, I tend to agree with the folks who think Bruce trimmed too much
> this time. But the release notes are, and always have been, intended to
> boil the CVS history down to something useful by eliminating irrelevant
> detail.

OK, so given everything mentioned on this thread, there are three items
that are user noticeable and so don't fall into the category of
irrelevant detail:

- Merge Join performance has been substantially improved when low number
of duplicate join keys exist on the outer side of the join (Simon, Greg)

- Large I/O reduction during recovery when full_page_writes = on
(Heikki)

- WAL file switch performance has been improved. Recovery startup now
refers to the last checkpoint time, which may be anything up to the
checkpoint_timeout interval before a database crash. (Simon, Tom)

The last one seems important to me, but I can see that might be TMI for
some people, though I feel we should document it somewhere.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Simon Riggs wrote:

[ Sorry for my delay in replying to this.]

> Few proposals
>
> - Can we say "smoothed" rather than "distributed" checkpoints?
> "Smoothed checkpoints greatly reduce checkpoint I/O spikes"

Agreed. Changed.

> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
> change to
> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"

I used the original text because it tries to explain _how_ HOT improves
performance. The item that has the descriptive text explains how the
space reuse works. A generic "improve performance" doesn't seem like an
improvement.

> I also notice that two performance features have disappeared from the
> release notes. (Presumably they have been removed from source). Both of
> them have changes that can be seen by users, so can't see why we would
> want them removed.
>
> - Merge Join performance has been substantially improved by ring buffer
> which avoids materializing the previous sort step. (Simon, Greg)
>
> More info, not for release notes:
> The materialization of the prior sort step would generally double the
> time taken for the sort, so avoiding this effectively gives a 50%
> performance gain on sorts that are part of large merge joins.
>
>
> - WAL file switches don't update controlfile any longer. Recovery now
> refers to the last checkpoint time, which may be many minutes earlier
> than time previously mentioned. (Simon, Tom)
>
> More info, not for release notes:
> WAL file switches were performed holding important LWLocks, so this
> improves scalability on high end systems as well as reducing response
> time spikes under heavy load on all kinds of hardware.

Let me give you the criteria I use for the release notes. The release
notes try to document all changes visible to the average user in a way
that is understandable to the average user.

The above items are probably neither visible (except faster) nor
understandable. Now of course we change change that criteria but that
is going to need a larger discussion.

One idea that would allow these to be included is a "geek" section of
the release notes that has items that would not be understandable by the
average user, e.g. optimizer improvements, locking improvements. It
would be kind of like "Postgres is faster in this release in some
obscure ways, and this is why". Of course the section would have to be
labeled clearly and it does open us up to the release notes being less
user-friendly.

Such a section seems to be more of a supplying a curiosity rather than
useful information, though.

I will address the issue of giving people credit for work in my next
email.

The good news is that we can keep adjusting the release notes until 8.3
final.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Simon Riggs wrote:
> On Fri, 2007-11-30 at 06:31 +0000, Simon Riggs wrote:
>
> > I also notice that two performance features have disappeared from the
> > release notes. (Presumably they have been removed from source). Both of
> > them have changes that can be seen by users, so can't see why we would
> > want them removed.
>
> Wow, just realised 3 of Heikki's performance patches aren't mentioned
> either:
>
> - CheckpointStartLock removal
>
> - I/O reduction during recovery
>
> - Tuning of Visibility code
>
> I'm not sure what the rationale is for not mentioning these things.
> They're at least as important, if not more so, than mentioning minor
> source code changes.

The source code changes are more _visible_, I think, meaning they often
require programmers to adjust their code.

> If people understand there aren't 13 performance improvements there are
> at *least* 19+ that is a positive message to help people decide to
> upgrade.

Frankly I think most people expact an X% improvement in every Postgres
release. I don't see how mentioning 19 vs. 13 items is going to change
the general understanding that you should upgrade to get better
performance.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Usama Dar wrote:
> i agree that release notes should not be too long, but may be there should
> be (if there isn't one already) something like a "change log" where people
> can find out all the changes done from the previous release, if they are
> intrested ?

Right now only the CVS logs provide more detailed information. At some
point perhaps we should have something that summarizes the CVS logs.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
>
>> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
>> change to
>> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
>>
>
> I used the original text because it tries to explain _how_ HOT improves
> performance. The item that has the descriptive text explains how the
> space reuse works. A generic "improve performance" doesn't seem like an
> improvement.
>
>
>
I still think this needs to be qualified either way. As it stands it's
quite misleading. Many update scenarios will not benefit one whit from
HOT updates.

cheers

andrew

Simon Riggs wrote:
> On Fri, 2007-11-30 at 09:49 +0000, Gregory Stark wrote:
> > "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> >
> > > If people understand there aren't 13 performance improvements there are
> > > at *least* 19+ that is a positive message to help people decide to
> > > upgrade.
> >
> > Frankly I think the release notes are already too long.
>
> So why do we have stuff in there that the users will never see?

Which release note items?

> We already have a release summary, so why summarise *some* of the detail
> as well, but not all of it???
>
> I see no reason to diminish yours, Heikki's or my own contributions, all
> of which were in the area of performance, which people do care about.
> None of the ones I mentioned were trivial patches, nor were their
> effects small.

I totally agree that we are unfair in how we give attribution in the
release notes. There is no weight given to patch difficulty and people
who produce user-invisible changes are much less likely to be mentioned
in the release notes.

I don't see any way to fix this that would not harm the release notes
themselves. As I mentioned in an earlier email the release notes are
designed to highlight user-visible changes in a user-understandable way.
The mentioning of people who wrote the patches is merely a side-effect
of that to give some credit, but it is a side-effect, not the main
reason we mention something in the release notes.

If people are concerned about the unfairness, and I understand that, the
best solution is not to add more items to the release notes to be more
fair, but to remove all names from release note items.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan wrote:
>
>
> Simon Riggs wrote:
> >
> > - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
> > change to
> > "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
> >
> >
> >
>
> I think we need to qualify this, or it could be quite misleading.
> perhaps add "that don't affect indexed columns" or something like that.

This is covered in the detailed text below the item:

<para>
Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse for
<command>UPDATE</>s (Pavan Deolasee, with ideas from many others)
</para>

<para>
<command>UPDATE</>s and <command>DELETE</>s leave dead tuples
behind, as do failed <command>INSERT</>s. Previously only
<command>VACUUM</> could reclaim space taken by dead tuples. With
<acronym>HOT</> dead tuple space can be reclaimed at the time of
<command>UPDATE</> or <command>INSERT</> if no changes are made to
indexed columns. This allows for more consistent performance.
Also, <acronym>HOT</> avoids adding duplicate index entries.

I just updated the wording so the index issue is clearer.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Josh Berkus wrote:
> Greg,
>
> > Frankly I think the release notes are already too long. People who judge a
> > release by counting the number of items in the release notes are not worth
> > appeasing. Including every individual lock removed or code path optimized
> > will only obscure the important points on which people should be judging
> > the relevance of the release to them. Things like smoothing checkpoint i/o
> > which could be removing a show-stopper problem for them.
>
> I disagree. For people who want a quick summary of the major user-facing
> things changed we'll have multiple sources: (a) the announcement, (b) the
> press features list, (c) the Feature-Version matrix. The Release notes
> should have a *complete* list of changes.
>
> Why? Because we don't use a bug/feature tracker. So a user trying to figure
> out "hey, was my issue XXX fixed so that I should upgrade?" has *no other
> source* than the Release notes to look at, except CVS history. And if we
> start asking sysadmins and application vendors to read the CVS history, we're
> gonna simply push them towards other DBMSes which have this information more
> clearly.
>
> If we want to shorten the release notes, then we should adopt an issue
> tracker.

We do mention bug fixes in the release notes if they affect more than a
few users. For rare bugs the original bug submitter is told what
release will have the fix as part of the bug fxing discussion. If it
wasn't fixed right away and became a TODO item that item is removed as
part of the release. (I just did that for 8.3.)

It is true we don't have a tracker but I have not seen a major demand
for it, or at least not enough for someone to actually do the work
required to list _all_ fixes/changes in a release. The job isn't that
hard, maybe a few days work for someone experienced.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan wrote:
>
>
> Bruce Momjian wrote:
> >
> >> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
> >> change to
> >> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
> >>
> >
> > I used the original text because it tries to explain _how_ HOT improves
> > performance. The item that has the descriptive text explains how the
> > space reuse works. A generic "improve performance" doesn't seem like an
> > improvement.
> >
> >
> >
> I still think this needs to be qualified either way. As it stands it's
> quite misleading. Many update scenarios will not benefit one whit from
> HOT updates.

Doesn't the detail description qualify it enought? The heading isn't
suppose to have all the information or it would be unreadable.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 7 Dec 2007 14:47:22 -0500 (EST)
Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> It is true we don't have a tracker but I have not seen a major demand
> for it,

Are you kidding?

> or at least not enough for someone to actually do the work
> required to list _all_ fixes/changes in a release. The job isn't that
> hard, maybe a few days work for someone experienced.

O.k. I would buy this..

Joshua D. Drake

>

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHWaVGATb/zqfZUUQRAlhoAKCNRp5eCwa71y70d8pkEKWR4eKscQCfa/4Z
UK47SO6aqelh0f9atYcyZmA=
=x8b6
-----END PGP SIGNATURE-----

On Fri, 2007-12-07 at 12:33 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> > On Fri, 2007-11-30 at 09:49 +0000, Gregory Stark wrote:
> > > "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > >
> > > > If people understand there aren't 13 performance improvements there are
> > > > at *least* 19+ that is a positive message to help people decide to
> > > > upgrade.
> > >
> > > Frankly I think the release notes are already too long.
> >
> > So why do we have stuff in there that the users will never see?

Thanks for your reply.

> Which release note items?

Most of the stuff in Source Code would fall into that category. I don't
advocate removing those items, but I don't see the argument that space
is so tight in the release notes that we have to remove important
performance items but keep those.

> > We already have a release summary, so why summarise *some* of the detail
> > as well, but not all of it???
> >
> > I see no reason to diminish yours, Heikki's or my own contributions, all
> > of which were in the area of performance, which people do care about.
> > None of the ones I mentioned were trivial patches, nor were their
> > effects small.
>
> I totally agree that we are unfair in how we give attribution in the
> release notes.

I do understand that the release notes are there to inform the user and
not directly to give credit.

Some important items have been removed from the release notes. It took
me a whole month to notice, but I did eventually notice because I'm
familiar with my own work, as well as that of people working on closely
related items. I have, when I have been aware of them, pointed out
patches produced by others that I thought were missing.

> There is no weight given to patch difficulty and people
> who produce user-invisible changes are much less likely to be mentioned
> in the release notes.

The reward system drives the outputs. If trivial feature additions are
what we reward, then that's what we'll get. That's not important right
now and discussing that is not why I started this thread.

> I don't see any way to fix this that would not harm the release notes
> themselves. As I mentioned in an earlier email the release notes are
> designed to highlight user-visible changes in a user-understandable way.
> The mentioning of people who wrote the patches is merely a side-effect
> of that to give some credit, but it is a side-effect, not the main
> reason we mention something in the release notes.

Perhaps we are talking about different things. I'm discussing whether
something is important and you seem to be imagining that I only want to
see the phrase "(Simon)" lots of times. If that was the case, it would
have been very simple to arrange, yet I seem to have elected the most
difficult route to doing that. I could easily have hoovered up a few
more trivial changes if that was my line of thinking. So it clearly
wasn't.

Maybe the importance of the patches that were removed wasn't clear
enough, so let me explain my viewpoint. On another part of this thread I
summarised the feedback from others to a list of features that were
definitely user noticeable. The list was:

- Merge Join performance has been substantially improved when low number
of duplicate join keys exist on the outer side of the join (Simon, Greg)

- Large I/O reduction during recovery when full_page_writes = on
(Heikki)

- WAL file switch performance has been improved. Recovery startup now
refers to the last checkpoint time, which may be anything up to the
checkpoint_timeout interval before a database crash. (Simon, Tom)

The last one seems important to me, but I can see that might be TMI for
some people, though I feel we should document it somewhere. The other
side of that is that many people know about those response time spikes
and they will be very keen to know their cause was identified and
removed.

Those items have resulted in important performance gains, not just a few
percentage points. The first one can be 50% or more, the second one is
100% gain and the last one reduced response time spikes on busy systems
by something like a second at switch time. I wouldn't dare to mention
these things if the effects were small, but they are massive gains.

> If people are concerned about the unfairness, and I understand that, the
> best solution is not to add more items to the release notes to be more
> fair, but to remove all names from release note items.

That makes no sense, but it would benefit people that wrote fewer
patches, I guess.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Bruce Momjian wrote:
> Andrew Dunstan wrote:
>
>> Bruce Momjian wrote:
>>
>>>
>>>
>>>> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
>>>> change to
>>>> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
>>>>
>>>>
>>> I used the original text because it tries to explain _how_ HOT improves
>>> performance. The item that has the descriptive text explains how the
>>> space reuse works. A generic "improve performance" doesn't seem like an
>>> improvement.
>>>
>>>
>>>
>>>
>> I still think this needs to be qualified either way. As it stands it's
>> quite misleading. Many update scenarios will not benefit one whit from
>> HOT updates.
>>
>
> Doesn't the detail description qualify it enought? The heading isn't
> suppose to have all the information or it would be unreadable.
>
>

If you don't want to be more specific I'd say "certain updates" or "some
updates" or something similar, just some flag to say it's not all of them.

cheers

andrew

Simon Riggs wrote:
> On Fri, 2007-12-07 at 12:33 -0500, Bruce Momjian wrote:
> > Simon Riggs wrote:
> > > On Fri, 2007-11-30 at 09:49 +0000, Gregory Stark wrote:
> > > > "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > > >
> > > > > If people understand there aren't 13 performance improvements there are
> > > > > at *least* 19+ that is a positive message to help people decide to
> > > > > upgrade.
> > > >
> > > > Frankly I think the release notes are already too long.
> > >
> > > So why do we have stuff in there that the users will never see?
>
> Thanks for your reply.
>
> > Which release note items?
>
> Most of the stuff in Source Code would fall into that category. I don't
> advocate removing those items, but I don't see the argument that space
> is so tight in the release notes that we have to remove important
> performance items but keep those.

I just looked at that section and see the need for each one. Which
items exactly?

> > > We already have a release summary, so why summarise *some* of the detail
> > > as well, but not all of it???
> > >
> > > I see no reason to diminish yours, Heikki's or my own contributions, all
> > > of which were in the area of performance, which people do care about.
> > > None of the ones I mentioned were trivial patches, nor were their
> > > effects small.
> >
> > I totally agree that we are unfair in how we give attribution in the
> > release notes.
>
> I do understand that the release notes are there to inform the user and
> not directly to give credit.
>
> Some important items have been removed from the release notes. It took
> me a whole month to notice, but I did eventually notice because I'm
> familiar with my own work, as well as that of people working on closely
> related items. I have, when I have been aware of them, pointed out
> patches produced by others that I thought were missing.
>
> > There is no weight given to patch difficulty and people
> > who produce user-invisible changes are much less likely to be mentioned
> > in the release notes.
>
> The reward system drives the outputs. If trivial feature additions are
> what we reward, then that's what we'll get. That's not important right
> now and discussing that is not why I started this thread.

Franky I don't think the release note mention is a significant part of
why people work on patches. If we remove names entirely, for example, I
don't think we would see any change in activity. The fact that this
issue hasn't come up since I started in 1996 confirms that.

> > I don't see any way to fix this that would not harm the release notes
> > themselves. As I mentioned in an earlier email the release notes are
> > designed to highlight user-visible changes in a user-understandable way.
> > The mentioning of people who wrote the patches is merely a side-effect
> > of that to give some credit, but it is a side-effect, not the main
> > reason we mention something in the release notes.
>
> Perhaps we are talking about different things. I'm discussing whether
> something is important and you seem to be imagining that I only want to
> see the phrase "(Simon)" lots of times. If that was the case, it would
> have been very simple to arrange, yet I seem to have elected the most
> difficult route to doing that. I could easily have hoovered up a few
> more trivial changes if that was my line of thinking. So it clearly
> wasn't.
>
> Maybe the importance of the patches that were removed wasn't clear
> enough, so let me explain my viewpoint. On another part of this thread I
> summarised the feedback from others to a list of features that were
> definitely user noticeable. The list was:
>
> - Merge Join performance has been substantially improved when low number
> of duplicate join keys exist on the outer side of the join (Simon, Greg)

Most users don't know if they are using mergejoins or not, nor are they
going to do anything differently now that the feature is in, so that is
why I don't see a need to mention it.

> - Large I/O reduction during recovery when full_page_writes = on
> (Heikki)

Again, a speedup, but not something that impacts people to behave
differently or see different output.

> - WAL file switch performance has been improved. Recovery startup now
> refers to the last checkpoint time, which may be anything up to the
> checkpoint_timeout interval before a database crash. (Simon, Tom)

The cput switch performance is the same issue as above. When you say
"recovery startup" are you talking about the log message?

> The last one seems important to me, but I can see that might be TMI for
> some people, though I feel we should document it somewhere. The other
> side of that is that many people know about those response time spikes
> and they will be very keen to know their cause was identified and
> removed.

True. We already mention some improvements in that area so I don't see
how mentioning something else that is kind of vague is going to help.

> Those items have resulted in important performance gains, not just a few
> percentage points. The first one can be 50% or more, the second one is
> 100% gain and the last one reduced response time spikes on busy systems
> by something like a second at switch time. I wouldn't dare to mention
> these things if the effects were small, but they are massive gains.

OK, we can add something about recovery being faster or something. We
could even tag it on to an existing item.

> > If people are concerned about the unfairness, and I understand that, the
> > best solution is not to add more items to the release notes to be more
> > fair, but to remove all names from release note items.
>
> That makes no sense, but it would benefit people that wrote fewer
> patches, I guess.

Yep, kind of illogical but it is fair.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Andrew Dunstan wrote:
> >> I still think this needs to be qualified either way. As it stands it's
> >> quite misleading. Many update scenarios will not benefit one whit from
> >> HOT updates.
> >>
> >
> > Doesn't the detail description qualify it enought? The heading isn't
> > suppose to have all the information or it would be unreadable.
> >
> >
>
> If you don't want to be more specific I'd say "certain updates" or "some
> updates" or something similar, just some flag to say it's not all of them.

Good idea. I added "most":

Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse for most
<command>UPDATE</>s (Pavan Deolasee, with ideas from many others)

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Simon Riggs wrote:
>
> [ Sorry for my delay in replying to this.]
>
>> Few proposals
>>
>> - Can we say "smoothed" rather than "distributed" checkpoints?
>> "Smoothed checkpoints greatly reduce checkpoint I/O spikes"
>
> Agreed. Changed.
>
>> - Heap-Only Tuples (HOT) accelerate space reuse for UPDATEs
>> change to
>> "Heap-Only Tuples (HOT) improve performance of frequent UPDATEs"
>
> I used the original text because it tries to explain _how_ HOT improves
> performance. The item that has the descriptive text explains how the
> space reuse works. A generic "improve performance" doesn't seem like an
> improvement.
>
>> I also notice that two performance features have disappeared from the
>> release notes. (Presumably they have been removed from source). Both of
>> them have changes that can be seen by users, so can't see why we would
>> want them removed.
>>
>> - Merge Join performance has been substantially improved by ring buffer
>> which avoids materializing the previous sort step. (Simon, Greg)
>>
>> More info, not for release notes:
>> The materialization of the prior sort step would generally double the
>> time taken for the sort, so avoiding this effectively gives a 50%
>> performance gain on sorts that are part of large merge joins.
>>
>>
>> - WAL file switches don't update controlfile any longer. Recovery now
>> refers to the last checkpoint time, which may be many minutes earlier
>> than time previously mentioned. (Simon, Tom)
>>
>> More info, not for release notes:
>> WAL file switches were performed holding important LWLocks, so this
>> improves scalability on high end systems as well as reducing response
>> time spikes under heavy load on all kinds of hardware.
>
> Let me give you the criteria I use for the release notes. The release
> notes try to document all changes visible to the average user in a way
> that is understandable to the average user.

hmm I'm not so sure about that - there are a number of changes to psql
that are not mentioned for example - like the addition of various new
backslashcommands (\dF, \dFp, \dFt ,...) as part of the main tsearch2
commit or a number of tab-complete improvements/additions (for tsearch
as well as some other stuff).
While looking at this i noted a "psql's \d command and pg_dump have been
enhanced" in the Object Manipulation Changes" Chapter which looks a bit
missplaced/useless and probably should be removed or clarified and moved
to the psql part.

Stefan

Stefan Kaltenbrunner wrote:
> > Let me give you the criteria I use for the release notes. The release
> > notes try to document all changes visible to the average user in a way
> > that is understandable to the average user.
>
> hmm I'm not so sure about that - there are a number of changes to psql
> that are not mentioned for example - like the addition of various new
> backslashcommands (\dF, \dFp, \dFt ,...) as part of the main tsearch2
> commit or a number of tab-complete improvements/additions (for tsearch
> as well as some other stuff).

Yea, but when you say you are adding tsearch there is the assumption
that psql and pg_dump and tab completion will be updated to reflect
that.

> While looking at this i noted a "psql's \d command and pg_dump have been
> enhanced" in the Object Manipulation Changes" Chapter which looks a bit
> missplaced/useless and probably should be removed or clarified and moved
> to the psql part.

I don't see that anywhere. Are you looking at the most recent version:

http://momjian.us/main/writings/pgsql/sgml/release-8-3.html

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Andrew Dunstan wrote:
>
>>>> I still think this needs to be qualified either way. As it stands it's
>>>> quite misleading. Many update scenarios will not benefit one whit from
>>>> HOT updates.
>>>>
>>>>
>>> Doesn't the detail description qualify it enought? The heading isn't
>>> suppose to have all the information or it would be unreadable.
>>>
>>>
>>>
>> If you don't want to be more specific I'd say "certain updates" or "some
>> updates" or something similar, just some flag to say it's not all of them.
>>
>
> Good idea. I added "most":
>
> Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse for most
> <command>UPDATE</>s (Pavan Deolasee, with ideas from many others)
>

But that's not true either. For example, in my current $dayjob app not
one significant update will benefit - we have an index rich environment.
You have no basis for saying "most" that I can see. We really should not
be in the hyp business in our release notes - that job belongs to the
commercial promoters ;-)

cheers

andrew

On Fri, 2007-12-07 at 16:21 -0500, Bruce Momjian wrote:
> Simon Riggs wrote:
> > Maybe the importance of the patches that were removed wasn't clear
> > enough, so let me explain my viewpoint. On another part of this thread I
> > summarised the feedback from others to a list of features that were
> > definitely user noticeable. The list was:
> >
> > - Merge Join performance has been substantially improved when low number
> > of duplicate join keys exist on the outer side of the join (Simon, Greg)
>
> Most users don't know if they are using mergejoins or not, nor are they
> going to do anything differently now that the feature is in, so that is
> why I don't see a need to mention it.

Wow, what an amazing claim. Please post that to -performance!

But if you applied that test, another half of the performance features
should go also. As would many others in various other sections. I want
them all to stay, so I won't point the finger at others. And why did we
mention outer joins were tuned in 8.2?

The release notes are already 33 pages on Firefox's Print Preview, so a
couple of extra sentences won't change anything.

> > - Large I/O reduction during recovery when full_page_writes = on
> > (Heikki)
>
> Again, a speedup, but not something that impacts people to behave
> differently or see different output.

Same comment.

> > > If people are concerned about the unfairness, and I understand that, the
> > > best solution is not to add more items to the release notes to be more
> > > fair, but to remove all names from release note items.
> >
> > That makes no sense, but it would benefit people that wrote fewer
> > patches, I guess.
>
> Yep, kind of illogical but it is fair.

If we're at the stage of saying logic is irrelevant, then I'll never
persuade you, probably of anything, ever...

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Bruce Momjian wrote:
> Stefan Kaltenbrunner wrote:
>>> Let me give you the criteria I use for the release notes. The release
>>> notes try to document all changes visible to the average user in a way
>>> that is understandable to the average user.
>> hmm I'm not so sure about that - there are a number of changes to psql
>> that are not mentioned for example - like the addition of various new
>> backslashcommands (\dF, \dFp, \dFt ,...) as part of the main tsearch2
>> commit or a number of tab-complete improvements/additions (for tsearch
>> as well as some other stuff).
>
> Yea, but when you say you are adding tsearch there is the assumption
> that psql and pg_dump and tab completion will be updated to reflect
> that.

hmm well - tab.complete support was not done for a number of new
SQL-level stuff with the main commits in the last few releases and was
mostly added by different developers (like me) either in the same
release cycle or even later.
Anyway this "implicitly included in the main commit" adds a few
interesting gotchas like having a "Build Options" section that has stuff
like "--enable-profiling" and "--with-system-tzdata" but
"--with-ossp-uuid" is in the contrib section and "--with-libxml" is
missing (mostly because implicitly assumed as part of SQL/XML) as is
"--with-gssapi"(part of the GSSAPI feature?) and "--with-libxslt (no idea ?)

>
>> While looking at this i noted a "psql's \d command and pg_dump have been
>> enhanced" in the Object Manipulation Changes" Chapter which looks a bit
>> missplaced/useless and probably should be removed or clarified and moved
>> to the psql part.
>
> I don't see that anywhere. Are you looking at the most recent version:
>
> http://momjian.us/main/writings/pgsql/sgml/release-8-3.html

hmm yeah sorry - I was looking at the beta3 version we have on the main
website seems to be fixed in the current code.

Stefan

Simon Riggs wrote:
> On Fri, 2007-12-07 at 16:21 -0500, Bruce Momjian wrote:
> > Simon Riggs wrote:
> > > Maybe the importance of the patches that were removed wasn't clear
> > > enough, so let me explain my viewpoint. On another part of this thread I
> > > summarised the feedback from others to a list of features that were
> > > definitely user noticeable. The list was:
> > >
> > > - Merge Join performance has been substantially improved when low number
> > > of duplicate join keys exist on the outer side of the join (Simon, Greg)
> >
> > Most users don't know if they are using mergejoins or not, nor are they
> > going to do anything differently now that the feature is in, so that is
> > why I don't see a need to mention it.
>
> Wow, what an amazing claim. Please post that to -performance!
>
> But if you applied that test, another half of the performance features
> should go also. As would many others in various other sections. I want
> them all to stay, so I won't point the finger at others. And why did we
> mention outer joins were tuned in 8.2?

Because outer joins are user-visible.

> The release notes are already 33 pages on Firefox's Print Preview, so a
> couple of extra sentences won't change anything.
>
> > > - Large I/O reduction during recovery when full_page_writes = on
> > > (Heikki)
> >
> > Again, a speedup, but not something that impacts people to behave
> > differently or see different output.
>
> Same comment.
>
> > > > If people are concerned about the unfairness, and I understand that, the
> > > > best solution is not to add more items to the release notes to be more
> > > > fair, but to remove all names from release note items.
> > >
> > > That makes no sense, but it would benefit people that wrote fewer
> > > patches, I guess.
> >
> > Yep, kind of illogical but it is fair.
>
> If we're at the stage of saying logic is irrelevant, then I'll never
> persuade you, probably of anything, ever...

You said above yourself you didn't want the names removed.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Stefan Kaltenbrunner wrote:
> Bruce Momjian wrote:
> > Stefan Kaltenbrunner wrote:
> >>> Let me give you the criteria I use for the release notes. The release
> >>> notes try to document all changes visible to the average user in a way
> >>> that is understandable to the average user.
> >> hmm I'm not so sure about that - there are a number of changes to psql
> >> that are not mentioned for example - like the addition of various new
> >> backslashcommands (\dF, \dFp, \dFt ,...) as part of the main tsearch2
> >> commit or a number of tab-complete improvements/additions (for tsearch
> >> as well as some other stuff).
> >
> > Yea, but when you say you are adding tsearch there is the assumption
> > that psql and pg_dump and tab completion will be updated to reflect
> > that.
>
> hmm well - tab.complete support was not done for a number of new
> SQL-level stuff with the main commits in the last few releases and was
> mostly added by different developers (like me) either in the same
> release cycle or even later.
> Anyway this "implicitly included in the main commit" adds a few
> interesting gotchas like having a "Build Options" section that has stuff
> like "--enable-profiling" and "--with-system-tzdata" but
> "--with-ossp-uuid" is in the contrib section and "--with-libxml" is
> missing (mostly because implicitly assumed as part of SQL/XML) as is
> "--with-gssapi"(part of the GSSAPI feature?) and "--with-libxslt (no idea ?)

These are new user-visible flags, not something like tab support which
you just kind of expect to work for every SQL command, and if it doesn't,
it is a bug.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Simon Riggs wrote:
> On Fri, 2007-12-07 at 16:21 -0500, Bruce Momjian wrote:
> > Simon Riggs wrote:
> > > Maybe the importance of the patches that were removed wasn't clear
> > > enough, so let me explain my viewpoint. On another part of this thread I
> > > summarised the feedback from others to a list of features that were
> > > definitely user noticeable. The list was:
> > >
> > > - Merge Join performance has been substantially improved when low number
> > > of duplicate join keys exist on the outer side of the join (Simon, Greg)
> >
> > Most users don't know if they are using mergejoins or not, nor are they
> > going to do anything differently now that the feature is in, so that is
> > why I don't see a need to mention it.
>
> Wow, what an amazing claim. Please post that to -performance!

I am talking the average user, not someone subscribed to performance.

> But if you applied that test, another half of the performance features
> should go also. As would many others in various other sections. I want
> them all to stay, so I won't point the finger at others. And why did we
> mention outer joins were tuned in 8.2?

This is starting to sound quite petty. I have already outlined the
logic I used. Now if you think the logic I am using is invalid we can
discuss it. If there are specific items I have added/removed that are
wrong based on that logic, be specific.

Frankly, based on the amount of time I might expend dealing with
everyone's "why isn't my item in there" I might suggest just removing
names completely just to avoid the discussion.

> The release notes are already 33 pages on Firefox's Print Preview, so a
> couple of extra sentences won't change anything.

I have to add entries based on some coherent logic, not based on a
request because then the release notes aren't consistent in their
limited coverage. And if I add everyone's two lines it will be 66 pages
with 33 pages of useful information.

For example, why add the mergejoin item and not the many other optimizer
improvements in every release? Is that useful information for the
average user? Couldn't we just tell them "Postgres is faster" and leave
it at that. If they are curious why they can ask or find out themselves.

If you want to collect all the optimizer items for 8.3 and put them in a
list we can link to them from the web site.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Stefan Kaltenbrunner wrote:
>> Bruce Momjian wrote:
>>> Stefan Kaltenbrunner wrote:
>>>>> Let me give you the criteria I use for the release notes. The release
>>>>> notes try to document all changes visible to the average user in a way
>>>>> that is understandable to the average user.
>>>> hmm I'm not so sure about that - there are a number of changes to psql
>>>> that are not mentioned for example - like the addition of various new
>>>> backslashcommands (\dF, \dFp, \dFt ,...) as part of the main tsearch2
>>>> commit or a number of tab-complete improvements/additions (for tsearch
>>>> as well as some other stuff).
>>> Yea, but when you say you are adding tsearch there is the assumption
>>> that psql and pg_dump and tab completion will be updated to reflect
>>> that.
>> hmm well - tab.complete support was not done for a number of new
>> SQL-level stuff with the main commits in the last few releases and was
>> mostly added by different developers (like me) either in the same
>> release cycle or even later.
>> Anyway this "implicitly included in the main commit" adds a few
>> interesting gotchas like having a "Build Options" section that has stuff
>> like "--enable-profiling" and "--with-system-tzdata" but
>> "--with-ossp-uuid" is in the contrib section and "--with-libxml" is
>> missing (mostly because implicitly assumed as part of SQL/XML) as is
>> "--with-gssapi"(part of the GSSAPI feature?) and "--with-libxslt (no idea ?)
>
> These are new user-visible flags, not something like tab support which
> you just kind of expect to work for every SQL command, and if it doesn't,
> it is a bug.

ok but to summarize that again - we have the following new
configure/buildlevel flags in 8.3 and the place their are mentioned:

--enable-profiling - "Build Options"
--with-system-tzdata - "Build Options"
--with-libxml - not mentioned
--with-gssapi - not mentioned
--with-ossp-uuid - "Contrib"
--with-libxslt - not mentioned(which should probably be fixed)

So we have at least 6 important user visible changes in our build-system
but we only dokument(in the release notes all are in the docs) two in
the "Build Options" section and one in another which seems kinda
arbitrary ...

All this seems to actually beg the question of "what is the intended use
for our release notes". It has become clear that it is not intended to
be a complete listing of changes done to the code but what is the
intended use in the current form?

Stefan

On Dec 8, 2007 3:37 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> If you want to collect all the optimizer items for 8.3 and put them in a
> list we can link to them from the web site.

Perhaps we can find a compromise by adding a global item on the
optimizer enhancements with the names of people having worked on them.
You could add a few examples in the detail of the item.
While I understand both points of view, I must admit that it's quite
weird to have all the ground work making this release far better not
mentioned in the release notes.

--
Guillaume

Bruce Momjian wrote:
> Simon Riggs wrote:
> > On Fri, 2007-12-07 at 16:21 -0500, Bruce Momjian wrote:

> > > > > If people are concerned about the unfairness, and I understand that, the
> > > > > best solution is not to add more items to the release notes to be more
> > > > > fair, but to remove all names from release note items.
> > > >
> > > > That makes no sense, but it would benefit people that wrote fewer
> > > > patches, I guess.
> > >
> > > Yep, kind of illogical but it is fair.
> >
> > If we're at the stage of saying logic is irrelevant, then I'll never
> > persuade you, probably of anything, ever...
>
> You said above yourself you didn't want the names removed.

Simon's the guy who (rightfully, IMHO) smacked me for forgetting to
credit him on a commit message. Credit is important to some people.
Let's not get in the business of annoying the people who gives their
work for free. For some, credit is the second sole reason for hacking
Postgres (first one being because it's fun). If I had had no credit,
I wouldn't have a job.

If anyone explicitely says that credit is not important to him, feel
free to remove his name from the release notes. I doubt you're going to
get many takers.

I don't think 100% fairness is a goal anyway. Just name the people who
did the work and that's enough.

That said, maybe we should additionally have a *separate* section in the
release notes to name the people who collaborated. Perhaps not just a
name list, but also what did they do.

--
Alvaro Herrera http://www.amazon.com/gp/registry/5ZYLFMCVHXC
"Endurecerse, pero jamás perder la ternura" (E. Guevara)

Guillaume Smet wrote:
> On Dec 8, 2007 3:37 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
> > If you want to collect all the optimizer items for 8.3 and put them in a
> > list we can link to them from the web site.
>
> Perhaps we can find a compromise by adding a global item on the
> optimizer enhancements with the names of people having worked on them.
> You could add a few examples in the detail of the item.
> While I understand both points of view, I must admit that it's quite
> weird to have all the ground work making this release far better not
> mentioned in the release notes.

In the past I added a generic optimizer release item, saying "Improved
optimizer handling of X, Y, Z" but Tom removed it because it was
"content-free", and he was right.

Again, what I would love to see is a detailed list of optimizer changes,
not only for 8.3 but for back branch major releases as well, that we
could link to from the release notes. I do think that information is
useful to some segment of our user base.

I just think putting optimizer items in the main release notes is going
to decrease the usability of the release notes for the majority of
users.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:
> Simon's the guy who (rightfully, IMHO) smacked me for forgetting to
> credit him on a commit message. Credit is important to some people.
> Let's not get in the business of annoying the people who gives their
> work for free. For some, credit is the second sole reason for hacking
> Postgres (first one being because it's fun). If I had had no credit,
> I wouldn't have a job.
>
> If anyone explicitly says that credit is not important to him, feel
> free to remove his name from the release notes. I doubt you're going to
> get many takers.

I don't think people are going to volunteer to remove just their name
but they might agree to remove them all. As a contributor I personally
would have no problem with that. I don't think all open source projects
have names next to release note items so naming people isn't something
that is universally followed.

I am actually a little worried that companies who sponsor developers
might some day want their company name on the release note item. I am
glad we have not had to make that decision yet. This actually
highlights a danger of having "giving credit" be a more major part of
the release notes in that companies could ask that they the company be
given credit in the release notes --- something we want to avoid. Right
now the name is mentioned merely to state who did the work, rather than
give credit.

> I don't think 100% fairness is a goal anyway. Just name the people who
> did the work and that's enough.
>
> That said, maybe we should additionally have a *separate* section in the
> release notes to name the people who collaborated. Perhaps not just a
> name list, but also what did they do.

Yes, we could do that, but clearly that section is going to be for
"credit-only". (I am reminded of banquets where they thank everyone who
helped organize the banquet.) I actually thought our developer profiles
had that function:

http://www.postgresql.org/developer/bios

but I don't think it has enough people to give "credit" for all 8.3
contributors.

Also, the release notes often just attribute items to just a first name,
so in some way that name has to be related to the developer profiles to
know who "Tom" or "Simon" is.

The bottom line is that the release notes for not for attribution, but
we do attribute if the item appears on the release notes. This seems
like the best balance, though it is unfair in failing to attribute work
to many contributors.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

> I am actually a little worried that companies who sponsor developers
> might some day want their company name on the release note item. I am
> glad we have not had to make that decision yet. This actually

O.k. I will bite :)

> highlights a danger of having "giving credit" be a more major part of
> the release notes in that companies could ask that they the company be
> given credit in the release notes --- something we want to avoid.

Why? If a company sponsored the feature why not mention them?

Note that I am not arguing one way or the other, but I find the
distinction between a individual who is a contributor and a company that
is a contributor interesting.

Sincerely,

Joshua D. Drake

Bruce Momjian wrote:

> I don't think people are going to volunteer to remove just their name
> but they might agree to remove them all. As a contributor I personally
> would have no problem with that.

I would.

> I am actually a little worried that companies who sponsor developers
> might some day want their company name on the release note item. I am
> glad we have not had to make that decision yet.

Yeah, I think companies have no place in a listing like that. Companies
don't do any work. People do. If sponsoring companies want to be in a
list, let them be on a separate list that's explicitely for sponsors.

I kinda hate the fact that people is now so used to the concept that
companies are "people" and have much of the same "rights". It is
absurd.

> Also, the release notes often just attribute items to just a first name,
> so in some way that name has to be related to the developer profiles to
> know who "Tom" or "Simon" is.

Yeah, I don't know when did that start but I would prefer that the names
would be spelled in full. On the other hand, having a first name only
is a kind of a sign that you're already an established developer. Still
I would like my last name to be there and I was considering adding it
myself.

> The bottom line is that the release notes for not for attribution, but
> we do attribute if the item appears on the release notes. This seems
> like the best balance, though it is unfair in failing to attribute work
> to many contributors.

Yeah. In my opinion, we're not really looking for 100% fairness. I
think credit is important, so if Stefan Kaltenbrunner did some psql
autocomplete work and it's not attributed, let's add a note to that
effect. There is no point in saying exactly how many lines he
contributed, nor listing the exact changes he did, though.

--
Alvaro Herrera http://www.flickr.com/photos/alvherre/
"No deja de ser humillante para una persona de ingenio saber
que no hay tonto que no le pueda enseñar algo." (Jean B. Say)

Alvaro Herrera wrote:

> Yeah. In my opinion, we're not really looking for 100% fairness. I
> think credit is important, so if Stefan Kaltenbrunner did some psql
> autocomplete work and it's not attributed, let's add a note to that
> effect. There is no point in saying exactly how many lines he
> contributed, nor listing the exact changes he did, though.

Then we should remove exactly how many lines, or listings of exact
changes any developer did.

Items of importance are subjective and to favor one over the other isn't
right (not that I am suggesting you are suggesting that).

Why not just a simple listing in general:

Alvaro Herrera - Auto vacuum work
Simon Riggs - PITR Work
Greg Stark - Tuple header work
Stefan K (not going to try and spell his name - Additions to psql
autocomplete

...

Alternatively we could use the contributors list (currently developers
list) as the definitive source and just provide a link.

Sincerely,

Joshua D. Drake

Joshua D. Drake wrote:
> Alvaro Herrera wrote:
>
>> Yeah. In my opinion, we're not really looking for 100% fairness. I
>> think credit is important, so if Stefan Kaltenbrunner did some psql
>> autocomplete work and it's not attributed, let's add a note to that
>> effect. There is no point in saying exactly how many lines he
>> contributed, nor listing the exact changes he did, though.
>
> Then we should remove exactly how many lines, or listings of exact changes
> any developer did.

We don't list that for anyone. That's precisely my point. We just put
the name of the person next to each release note item.

We don't say "worked on improving the backend" and then plaster 100
names, because we need a reasonable level of detail. But at the same
time we needn't mention that such-and-such command got tab completion on
psql, because as Bruce says, it is expected that all our commands have
it.

> Alternatively we could use the contributors list (currently developers
> list) as the definitive source and just provide a link.

I think that's a bad idea, because it doesn't keep historical
information. Furthermore it seems inactive people is removed
eventually, which doesn't happen with release notes.

--
Alvaro Herrera http://www.PlanetPostgreSQL.org/
"On the other flipper, one wrong move and we're Fatal Exceptions"
(T.U.X.: Term Unit X - http://www.thelinuxreview.com/TUX/)

On Saturday 08 December 2007 16:01, Alvaro Herrera wrote:
> > Alternatively we could use the contributors list (currently developers
> > list) as the definitive source and just provide a link.
>
> I think that's a bad idea, because it doesn't keep historical
> information. Furthermore it seems inactive people is removed
> eventually, which doesn't happen with release notes.

Unless it's been changed, I added the "past contributors" section just for the
purpose of giving a historical credit and ongoing thanks for past
contribution.

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

On Sat, 2007-12-08 at 12:46 -0500, Bruce Momjian wrote:
> Alvaro Herrera wrote:
> > Simon's the guy who (rightfully, IMHO) smacked me for forgetting to
> > credit him on a commit message. Credit is important to some people.
> > Let's not get in the business of annoying the people who gives their
> > work for free. For some, credit is the second sole reason for hacking
> > Postgres (first one being because it's fun). If I had had no credit,
> > I wouldn't have a job.
> >
> > If anyone explicitly says that credit is not important to him, feel
> > free to remove his name from the release notes. I doubt you're going to
> > get many takers.
>
> I don't think people are going to volunteer to remove just their name
> but they might agree to remove them all.

I'm not sure how a thread about a few missing items on the performance
list became a discussion about removing all credits.

The release notes are there to inform people about important features,
so if me waiving any credit on those missing items is what is required
to get them mentioned, I will happily do so.

Please add the items I mentioned upthread.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

On Sat, 2007-12-08 at 10:13 -0300, Alvaro Herrera wrote:
> If I had had no credit, I wouldn't have a job.

Agree with this 100%

I don't have a problem with mentioning sponsoring companies on the
bottom of the release notes. I think it will encourage wider sponsorship
if people do that. Probably should be optional though, allowing sponsors
that want to remain anonymous to do so.

"Simon Riggs was primarily sponsored by EnterpriseDB, with additional
sponsorship from 2ndQuadrant during the 8.3 release cycle." or something
similar. I'm very thankful to Denis Lussier in particular for allowing
me a clear schedule right up to the 8.3 feature freeze, without which
many items would not have been completed. (I'd also like to thank
Greenplum for their partial sponsorship of my work for the 8.1 and 8.2
releases).

Encouraging sponsorship is good; nobody loses from it, since everybody
gains from the software that gets written.

--
Simon Riggs
2ndQuadrant http://www.2ndQuadrant.com

Simon Riggs wrote:
> On Sat, 2007-12-08 at 10:13 -0300, Alvaro Herrera wrote:
> > If I had had no credit, I wouldn't have a job.
>
> Agree with this 100%
>
>
>
> I don't have a problem with mentioning sponsoring companies on the
> bottom of the release notes. I think it will encourage wider sponsorship
> if people do that. Probably should be optional though, allowing sponsors
> that want to remain anonymous to do so.
>
> "Simon Riggs was primarily sponsored by EnterpriseDB, with additional
> sponsorship from 2ndQuadrant during the 8.3 release cycle." or something
> similar. I'm very thankful to Denis Lussier in particular for allowing
> me a clear schedule right up to the 8.3 feature freeze, without which
> many items would not have been completed. (I'd also like to thank
> Greenplum for their partial sponsorship of my work for the 8.1 and 8.2
> releases).
>
> Encouraging sponsorship is good; nobody loses from it, since everybody
> gains from the software that gets written.

The release notes are not the place for sponsor advertizement. And
removing the names was so we could be fair and mention on one, not to
allow additional items to be added to the list.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
>
> > I am actually a little worried that companies who sponsor developers
> > might some day want their company name on the release note item. I am
> > glad we have not had to make that decision yet. This actually
>
> O.k. I will bite :)
>
>
> > highlights a danger of having "giving credit" be a more major part of
> > the release notes in that companies could ask that they the company be
> > given credit in the release notes --- something we want to avoid.
>
> Why? If a company sponsored the feature why not mention them?
>
>
> Note that I am not arguing one way or the other, but I find the
> distinction between a individual who is a contributor and a company that
> is a contributor interesting.

Individual mentions are only so we know who did the work. Company names
are for advertizement, and that doesn't belong in the release notes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:
> Yeah, I don't know when did that start but I would prefer that the names
> would be spelled in full. On the other hand, having a first name only
> is a kind of a sign that you're already an established developer. Still
> I would like my last name to be there and I was considering adding it
> myself.

The problem with full names is that right now the names are mostly from
frequent contributors, and hance one word. If we go with full names the
attribution becomes longer and we might decide we just want to remove
names completely.

When I started we were small so naming folks made sense, but now that we
are larger perhaps the name detracts from the polish of the release
notes and should be removed. Of course, if we do that, we have to start
a new thread and discuss it.

> > The bottom line is that the release notes for not for attribution, but
> > we do attribute if the item appears on the release notes. This seems
> > like the best balance, though it is unfair in failing to attribute work
> > to many contributors.
>
> Yeah. In my opinion, we're not really looking for 100% fairness. I
> think credit is important, so if Stefan Kaltenbrunner did some psql
> autocomplete work and it's not attributed, let's add a note to that
> effect. There is no point in saying exactly how many lines he
> contributed, nor listing the exact changes he did, though.

I do think we have demand for an "internals" section of the release
notes or a separate section on 8.3 internals, but it should list all
changes in 8.3 that are not in the main release notes.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Joshua D. Drake wrote:
> >
> > > I am actually a little worried that companies who sponsor developers
> > > might some day want their company name on the release note item. I am
> > > glad we have not had to make that decision yet. This actually
> >
> > O.k. I will bite :)
> >
> >
> > > highlights a danger of having "giving credit" be a more major part of
> > > the release notes in that companies could ask that they the company be
> > > given credit in the release notes --- something we want to avoid.
> >
> > Why? If a company sponsored the feature why not mention them?
> >
> >
> > Note that I am not arguing one way or the other, but I find the
> > distinction between a individual who is a contributor and a company that
> > is a contributor interesting.
>
> Individual mentions are only so we know who did the work. Company names
> are for advertisement, and that doesn't belong in the release notes.

I am thinking company names for advertisement belong in the press
release.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

"Bruce Momjian" <bruce(at)momjian(dot)us> writes:

> Joshua D. Drake wrote:
>
>> Note that I am not arguing one way or the other, but I find the
>> distinction between a individual who is a contributor and a company that
>> is a contributor interesting.
>
> Individual mentions are only so we know who did the work. Company names
> are for advertizement, and that doesn't belong in the release notes.

I think it would be kind of neat to see all the sponsored work somewhere. Only
I think it may be a bit repetitive with Redhat sponsoring most of the best
stuff... :)

As for what items should go in, I think it should be pretty clear if you put
yourself in the seat of a user trying to determine if upgrading will solve
problems with their application or allow them to do things with their
application that they couldn't before.

It's certainly possible for performance improvements to affect that. I think
"multiple space reductions, especially (but not only) for text, numeric and
other variable sized data" could easily be something which particular DBAs
would have noticed was a problem they needed to deal with.

I understand the thinking but I disagree that "various optimizations speeding
up merge sort, reducing contention at transaction start and end, ..." is
entirely content-free. I agree that nobody is really going to be specifically
saying "gee, i wish we could use postgres but merge-join is just too slow".
However as a user I find it helpful to get a kind of overview of the kinds of
invisible changes there were so I can get a feel for the magnitude of the
improvements between versions.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com
Ask me about EnterpriseDB's 24x7 Postgres support!

Gregory Stark wrote:
> "Bruce Momjian" <bruce(at)momjian(dot)us> writes:
>
>> Joshua D. Drake wrote:
>>
>>> Note that I am not arguing one way or the other, but I find the
>>> distinction between a individual who is a contributor and a company that
>>> is a contributor interesting.
>> Individual mentions are only so we know who did the work. Company names
>> are for advertizement, and that doesn't belong in the release notes.
>
> I think it would be kind of neat to see all the sponsored work somewhere. Only
> I think it may be a bit repetitive with Redhat sponsoring most of the best
> stuff... :)

Hah!

> It's certainly possible for performance improvements to affect that. I think
> "multiple space reductions, especially (but not only) for text, numeric and
> other variable sized data" could easily be something which particular DBAs
> would have noticed was a problem they needed to deal with.

Well let's take it a step back because I think you are on to something here.

Our customers will "directly" see a benefit from the changes Greg made
for 8.3 on the space reductions. Not only is a benefit for on disk
storage but it is also a direct performance benefit. When my customers
say... why does 8.3 take so much less space than 8.2, I will tell them
and further expound on it being one of the key components to their
greater performance.

Most of out customers however will not notice one of our key
contributions which was the multi-worker autovacuum. Yes it is
important. Yes we have specific customers that will benefit greatly from
it but the mass majority will not.

>
> I understand the thinking but I disagree that "various optimizations speeding
> up merge sort, reducing contention at transaction start and end, ..." is
> entirely content-free. I agree that nobody is really going to be specifically
> saying "gee, i wish we could use postgres but merge-join is just too slow".

Right.

> However as a user I find it helpful to get a kind of overview of the kinds of
> invisible changes there were so I can get a feel for the magnitude of the
> improvements between versions.
>

I agree with this as well. However, I am starting to wonder if the
"release notes" is the place for this. I am starting to think that
instead we need a white paper on the release (or each release) that is
categorized and reasonably in depth on "why" 8.3 (or 8.4 etc..).

Sincerely,

Joshua D. Drake

Gregory Stark wrote:
> I understand the thinking but I disagree that "various optimizations speeding
> up merge sort, reducing contention at transaction start and end, ..." is
> entirely content-free. I agree that nobody is really going to be specifically
> saying "gee, i wish we could use postgres but merge-join is just too slow".
> However as a user I find it helpful to get a kind of overview of the kinds of
> invisible changes there were so I can get a feel for the magnitude of the
> improvements between versions.

Agreed. You will have to convince Tom because it was mostly such items
from him in previous release that he asked to be removed.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
> > However as a user I find it helpful to get a kind of overview of the kinds of
> > invisible changes there were so I can get a feel for the magnitude of the
> > improvements between versions.
> >
>
> I agree with this as well. However, I am starting to wonder if the
> "release notes" is the place for this. I am starting to think that
> instead we need a white paper on the release (or each release) that is
> categorized and reasonably in depth on "why" 8.3 (or 8.4 etc..).

Sure, and we could perhaps put up with some gritty detail there, though
it doesn't solve the problem of people's work not being mentioned.
Seems that is a separate problem.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Joshua D. Drake wrote:
>>> However as a user I find it helpful to get a kind of overview of the kinds of
>>> invisible changes there were so I can get a feel for the magnitude of the
>>> improvements between versions.
>>>
>> I agree with this as well. However, I am starting to wonder if the
>> "release notes" is the place for this. I am starting to think that
>> instead we need a white paper on the release (or each release) that is
>> categorized and reasonably in depth on "why" 8.3 (or 8.4 etc..).
>
> Sure, and we could perhaps put up with some gritty detail there, though
> it doesn't solve the problem of people's work not being mentioned.
> Seems that is a separate problem.

I assumed the white paper would have proper attribution.

Sincerely,

Joshua D. Drake

Joshua D. Drake wrote:
> Bruce Momjian wrote:
> > Joshua D. Drake wrote:
> >>> However as a user I find it helpful to get a kind of overview of the kinds of
> >>> invisible changes there were so I can get a feel for the magnitude of the
> >>> improvements between versions.
> >>>
> >> I agree with this as well. However, I am starting to wonder if the
> >> "release notes" is the place for this. I am starting to think that
> >> instead we need a white paper on the release (or each release) that is
> >> categorized and reasonably in depth on "why" 8.3 (or 8.4 etc..).
> >
> > Sure, and we could perhaps put up with some gritty detail there, though
> > it doesn't solve the problem of people's work not being mentioned.
> > Seems that is a separate problem.
>
> I assumed the white paper would have proper attribution.

Right, but is the white paper going to be thorough to mention _all_
changes?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Based on this discussion I think it is clear the release notes chapter
needs an introductory section. This would not be for any specific
release but the release notes in general. I have come up with the
following text:

The release notes contain the significant changes for each PostgreSQL
release, with major features or migration issues often listed at the
top. The release notes do not contain changes that affect only a few
users or changes that are internal and therefore not user-visible. For
example, the optimizer is improved in almost every release, but the
improvements are usually observed by users as simply faster queries.

A complete list of all changes for a release can only be obtained
by viewing the CVS logs for each release. The committers email
list (http://archives.postgresql.org/pgsql-committers/) contains
all source code changes as well. There is also a web interface
that shows changes to specific files or directories
(http://developer.postgresql.org/cvsweb.cgi/pgsql/). (XXX SVN is
good but needs "Next" button at bottom, no branch filter, https
certificate update
https://projects.commandprompt.com/public/pgsql/log/?action=stop_on_copy&rev=&stop_rev=&mode=stop_on_copy&verbose=on).

A names appearing next to an item represents the major developer for
that item. Of course all changes involve community discussion and patch
review so each item is truely a community activity. First-name-only
entries represent established developers, while full names represent
newer contributors.

I need help with the CVS section. Do we publish full CVS logs for a
release? I like the SVN display because it groups commits but can
improvements I listed above be made?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Joshua D. Drake wrote:

>> I assumed the white paper would have proper attribution.
>
> Right, but is the white paper going to be thorough to mention _all_
> changes?
>

Hmmm good question which gets back to where we started :). My very first
thought on all of this was that we would list all notable changes but
that we wouldn't mention anyone's name.

Then, we would have a "Who contributed to this release" section that
just listed names without attribution to the specific feature. IMO, that
is the only "fair" way.

I realize that notable is subjective.

Here is the deal :). I think as long as a single person is making the
decision as to what goes and stays, there will always be friction.
Perhaps it is time for a "release team"? Odd numbers only, +1/-1 voting
etc... I don't know maybe that is too much.

Sincerely,

Joshua D. Drake

On Dec 10, 2007 10:39 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

I like the realease notes intro. You may have already picked up on
these, but a couple typos:

> A names appearing next to an item represents the major developer for
> that item. Of course all changes involve community discussion and patch
> review so each item is truely a community activity. First-name-only
> entries represent established developers, while full names represent
> newer contributors.

"A names" -> "Names", "represents" -> "represent", "developer" ->
"developers" or "developer(s)" if you prefer.

I could really go a language that doesn't distinguish between singular
and plural, but looks like we're stuck with English for the duration.

And "truely" -> "truly".

Cheers,
BJ

On Dec 8, 2007 3:42 AM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

>
>
> Bruce Momjian wrote:
> > Andrew Dunstan wrote:
> >
> >>>> I still think this needs to be qualified either way. As it stands
> it's
> >>>> quite misleading. Many update scenarios will not benefit one whit
> from
> >>>> HOT updates.
> >>>>
> >>>>
> >>> Doesn't the detail description qualify it enought? The heading isn't
> >>> suppose to have all the information or it would be unreadable.
> >>>
> >>>
> >>>
> >> If you don't want to be more specific I'd say "certain updates" or
> "some
> >> updates" or something similar, just some flag to say it's not all of
> them.
> >>
> >
> > Good idea. I added "most":
> >
> > Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse for
> most
> > <command>UPDATE</>s (Pavan Deolasee, with ideas from many others)
> >
>
> But that's not true either. For example, in my current $dayjob app not
> one significant update will benefit - we have an index rich environment.
> You have no basis for saying "most" that I can see. We really should not
> be in the hyp business in our release notes - that job belongs to the
> commercial promoters ;-)
>
>
>

I don't agree completely. HOT updates is just one significant benefit of
HOT and is constrained by the non-index column updates. But the other
major benefit of truncating the tuples to their line pointers applies to
HOT as well as COLD updates and DELETEs. This should also have
a non trivial positive impact on the performance.

There might be few scenarios where HOT may not show any improvement
such as CPU-bound applications, but I am not sure if its worth mentioning.

Thanks,
Pavan

--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

Hi,

Le lundi 10 décembre 2007, Bruce Momjian a écrit :
> Based on this discussion I think it is clear the release notes chapter
> needs an introductory section. This would not be for any specific
> release but the release notes in general.

Excellent idea, IMHO.

> I need help with the CVS section. Do we publish full CVS logs for a
> release? I like the SVN display because it groups commits but can
> improvements I listed above be made?

I tend to like the git web interface found here:
http://repo.or.cz/w/PostgreSQL.git
http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=master
http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=REL8_2_STABLE

Hope this helps,
--
dim

Brendan Jurd wrote:
> On Dec 10, 2007 10:39 AM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
> I like the realease notes intro. You may have already picked up on
> these, but a couple typos:
>
> > A names appearing next to an item represents the major developer for
> > that item. Of course all changes involve community discussion and patch
> > review so each item is truely a community activity. First-name-only
> > entries represent established developers, while full names represent
> > newer contributors.
>
> "A names" -> "Names", "represents" -> "represent", "developer" ->
> "developers" or "developer(s)" if you prefer.
>
> I could really go a language that doesn't distinguish between singular
> and plural, but looks like we're stuck with English for the duration.
>
> And "truely" -> "truly".

Thanks, got it.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Dimitri Fontaine wrote:
-- Start of PGP signed section.
> Hi,
>
> Le lundi 10 d?cembre 2007, Bruce Momjian a ?crit?:
> > Based on this discussion I think it is clear the release notes chapter
> > needs an introductory section. This would not be for any specific
> > release but the release notes in general.
>
> Excellent idea, IMHO.
>
> > I need help with the CVS section. Do we publish full CVS logs for a
> > release? I like the SVN display because it groups commits but can
> > improvements I listed above be made?
>
> I tend to like the git web interface found here:
> http://repo.or.cz/w/PostgreSQL.git
> http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=master
> http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=REL8_2_STABLE

I like the branch option but I don't like the title being duplicated as
the first line in "log" mode. The Next button is returning " 403
Forbidden - Project needed".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
> Bruce Momjian wrote:
> > Joshua D. Drake wrote:
>
> >> I assumed the white paper would have proper attribution.
> >
> > Right, but is the white paper going to be thorough to mention _all_
> > changes?
> >
>
> Hmmm good question which gets back to where we started :). My very first
> thought on all of this was that we would list all notable changes but
> that we wouldn't mention anyone's name.

Isn't that listing what is already in the release notes?

> Then, we would have a "Who contributed to this release" section that
> just listed names without attribution to the specific feature. IMO, that
> is the only "fair" way.
>
> I realize that notable is subjective.
>
> Here is the deal :). I think as long as a single person is making the
> decision as to what goes and stays, there will always be friction.
> Perhaps it is time for a "release team"? Odd numbers only, +1/-1 voting
> etc... I don't know maybe that is too much.

Not really. We can have anyone suggest changes to the release notes up
until release, and there already have been tons of user-suggested
changes. The issue is if I disagree with a suggested change, and no one
else backs up the suggested change, it gets rejected, just like the
community patch process works.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Pavan Deolasee wrote:
>
>
> On Dec 8, 2007 3:42 AM, Andrew Dunstan <andrew(at)dunslane(dot)net
> <mailto:andrew(at)dunslane(dot)net>> wrote:
>
>
>
> Bruce Momjian wrote:
> > Andrew Dunstan wrote:
> >
> >>>> I still think this needs to be qualified either way. As it
> stands it's
> >>>> quite misleading. Many update scenarios will not benefit one
> whit from
> >>>> HOT updates.
> >>>>
> >>>>
> >>> Doesn't the detail description qualify it enought? The
> heading isn't
> >>> suppose to have all the information or it would be unreadable.
> >>>
> >>>
> >>>
> >> If you don't want to be more specific I'd say "certain updates"
> or "some
> >> updates" or something similar, just some flag to say it's not
> all of them.
> >>
> >
> > Good idea. I added "most":
> >
> > Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse
> for most
> > <command>UPDATE</>s (Pavan Deolasee, with ideas from many
> others)
> >
>
> But that's not true either. For example, in my current $dayjob app not
> one significant update will benefit - we have an index rich
> environment.
> You have no basis for saying "most" that I can see. We really
> should not
> be in the hyp business in our release notes - that job belongs to the
> commercial promoters ;-)
>
>
>
>
> I don't agree completely. HOT updates is just one significant benefit of
> HOT and is constrained by the non-index column updates. But the other
> major benefit of truncating the tuples to their line pointers applies to
> HOT as well as COLD updates and DELETEs. This should also have
> a non trivial positive impact on the performance.
>
> There might be few scenarios where HOT may not show any improvement
> such as CPU-bound applications, but I am not sure if its worth mentioning.
>
>
> <http://www.enterprisedb.com>

Um, I don't understand. I freely admit that I haven't kept up with all
the nuances of the HOT discussions, but this bit has totally eluded me,
so please elucidate.

cheers

andrew

Le lundi 10 décembre 2007, Bruce Momjian a écrit :
> > http://repo.or.cz/w/PostgreSQL.git
> > http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=master
> > http://repo.or.cz/w/PostgreSQL.git?a=shortlog;h=REL8_2_STABLE
>
> I like the branch option but I don't like the title being duplicated as
> the first line in "log" mode.

I'm not that acquainted to git and gitweb to provide authoritative answer, but
my guess here is that git (as darcs) is used to have a first line to title
patch, then a optional description. It seems the automatic cvs-to-git tool
(or maybe cvsps) is not smart enough to extract a good title from cvs log
entries (which may or may not provide one).
http://www.cobite.com/cvsps/

> The Next button is returning " 403
> Forbidden - Project needed".

Ouch. ISTR it used to work well... and replacing the ARRAY(0x...) garbage with
PostgreSQL.git makes it render the wanted web page...

Should I add I'm not trying to push towards git/gitweb nor trying to have
decentralized revision system on the spot; but rather am trying to help
finding the right tool for the job at hand.

Regards,
--
dim

Bruce Momjian wrote:
> Joshua D. Drake wrote:
>> Bruce Momjian wrote:
>>> Joshua D. Drake wrote:
>>>> I assumed the white paper would have proper attribution.
>>> Right, but is the white paper going to be thorough to mention _all_
>>> changes?
>>>
>> Hmmm good question which gets back to where we started :). My very first
>> thought on all of this was that we would list all notable changes but
>> that we wouldn't mention anyone's name.
>
> Isn't that listing what is already in the release notes?
>

No :). What is listed already in the release notes is what "you" think
is notable, which is why I mention the subjective below. What is cool to
you may not be cool to others etc... I am not saying you are doing a bad
job just that it is subjective.

Case in point I think the work that Stefan did for this release is
notable. I believe it is notable for several reasons.

1. Usability

2. Recognition (yes I am aware of the thoughts on that)

3. It was grunt work that should have been done with the original patch
that didn't get done. Stefan picked up the ball and ran with it and
produced something that will make our product more usable for the end user.

Tom and you disagreed. I understand the reasoning and I don't actually
disagree with the thought process but I think the thought process is
flawed. I do not believe people only look at the release notes for "wow
cool". I believe they look at them to see who deserves kudos in this
release.

Joshua D. Drake

On Mon, 10 Dec 2007, Joshua D. Drake wrote:

> 3. It was grunt work that should have been done with the original patch
> that didn't get done. Stefan picked up the ball and ran with it and
> produced something that will make our product more usable for the end
> user.

Then why not list Stefan as a contributor to the original feature rather
than calling it out as a separate item?

Kris Jurka

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 10 Dec 2007 12:14:58 -0500 (EST)
Kris Jurka <books(at)ejurka(dot)com> wrote:

>
>
> On Mon, 10 Dec 2007, Joshua D. Drake wrote:
>
> > 3. It was grunt work that should have been done with the original
> > patch that didn't get done. Stefan picked up the ball and ran with
> > it and produced something that will make our product more usable
> > for the end user.
>
> Then why not list Stefan as a contributor to the original feature
> rather than calling it out as a separate item?

*shrug* that is semantic to me. My point was the attribution :)

Joshua D. Drake

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD4DBQFHXXmUATb/zqfZUUQRAlfmAKCYhSEKuarEchkxlJJMf3Z8SVtT9gCY2Hex
ufDRdcgXQqEcnnNYh3JiCg==
=Qr0T
-----END PGP SIGNATURE-----

Kris Jurka wrote:
>
>
> On Mon, 10 Dec 2007, Joshua D. Drake wrote:
>
> > 3. It was grunt work that should have been done with the original patch
> > that didn't get done. Stefan picked up the ball and ran with it and
> > produced something that will make our product more usable for the end
> > user.
>
> Then why not list Stefan as a contributor to the original feature rather
> than calling it out as a separate item?

That is an excellent suggestion, done:

Full text search is integrated into the core database
system (Teodor, Oleg, Stefan Kaltenbrunner)

Can people thing of other cases?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Joshua D. Drake wrote:
> Bruce Momjian wrote:
> > Joshua D. Drake wrote:
> >> Bruce Momjian wrote:
> >>> Joshua D. Drake wrote:
> >>>> I assumed the white paper would have proper attribution.
> >>> Right, but is the white paper going to be thorough to mention _all_
> >>> changes?
> >>>
> >> Hmmm good question which gets back to where we started :). My very first
> >> thought on all of this was that we would list all notable changes but
> >> that we wouldn't mention anyone's name.
> >
> > Isn't that listing what is already in the release notes?
> >
>
> No :). What is listed already in the release notes is what "you" think
> is notable, which is why I mention the subjective below. What is cool to
> you may not be cool to others etc... I am not saying you are doing a bad
> job just that it is subjective.
>
> Case in point I think the work that Stefan did for this release is
> notable. I believe it is notable for several reasons.
>
> 1. Usability
>
> 2. Recognition (yes I am aware of the thoughts on that)
>
> 3. It was grunt work that should have been done with the original patch
> that didn't get done. Stefan picked up the ball and ran with it and
> produced something that will make our product more usable for the end user.
>
> Tom and you disagreed. I understand the reasoning and I don't actually
> disagree with the thought process but I think the thought process is
> flawed. I do not believe people only look at the release notes for "wow
> cool". I believe they look at them to see who deserves kudos in this
> release.

OK, that was clear to me. You are saying based on the criteria we have
used in the past our system of feedback works (good), but the criteria
used isn't 100% agreed.

My point is that it isn't that I am too subjective and closed to
feedback on adding/removing items, but rather the policy used isn't open
for feedback; but it really is. Should we open discussion of changing
the policy?

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Mon, 10 Dec 2007, Bruce Momjian wrote:

> Kris Jurka wrote:
>>
>>
>> On Mon, 10 Dec 2007, Joshua D. Drake wrote:
>>
>>> 3. It was grunt work that should have been done with the original patch
>>> that didn't get done. Stefan picked up the ball and ran with it and
>>> produced something that will make our product more usable for the end
>>> user.
>>
>> Then why not list Stefan as a contributor to the original feature rather
>> than calling it out as a separate item?
>
> That is an excellent suggestion, done:
>
> Full text search is integrated into the core database
> system (Teodor, Oleg, Stefan Kaltenbrunner)
>
> Can people thing of other cases?

Wait, I think we need more words about original authors ! For example,

Full text search (Teodor,Oleg) is integrated into the core database
system (Teodor, Oleg, Stefan Kaltenbrunner).

Honestly, there were more developers who participated in integration.

Regards,
Oleg
_____________________________________________________________
Oleg Bartunov, Research Scientist, Head of AstroNet (www.astronet.ru),
Sternberg Astronomical Institute, Moscow University, Russia
Internet: oleg(at)sai(dot)msu(dot)su, http://www.sai.msu.su/~megera/
phone: +007(495)939-16-83, +007(495)939-23-83

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 10 Dec 2007 17:16:12 -0500 (EST)
Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> Kris Jurka wrote:
> >
> >
> > On Mon, 10 Dec 2007, Joshua D. Drake wrote:
> >
> > > 3. It was grunt work that should have been done with the original
> > > patch that didn't get done. Stefan picked up the ball and ran
> > > with it and produced something that will make our product more
> > > usable for the end user.
> >
> > Then why not list Stefan as a contributor to the original feature
> > rather than calling it out as a separate item?
>
> That is an excellent suggestion, done:
>
> Full text search is integrated into the core database
> system (Teodor, Oleg, Stefan Kaltenbrunner)

I do still think we should add last names. AndrewD mentioned that and
it is far better than Magnus and his pgp fingerprint idea ;)

Joshua D. Drake

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXb9DATb/zqfZUUQRAslwAJ9jxbYCaBUyv6Q1PpJvKbctMmwgFwCfQp5u
gGNnlzxi0AbySfKy6M3qMxE=
=uzgF
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 10 Dec 2007 17:26:11 -0500 (EST)
Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> > Tom and you disagreed. I understand the reasoning and I don't
> > actually disagree with the thought process but I think the thought
> > process is flawed. I do not believe people only look at the release
> > notes for "wow cool". I believe they look at them to see who
> > deserves kudos in this release.
>
> OK, that was clear to me. You are saying based on the criteria we
> have used in the past our system of feedback works (good), but the
> criteria used isn't 100% agreed.
>
> My point is that it isn't that I am too subjective and closed to
> feedback on adding/removing items, but rather the policy used isn't
> open for feedback; but it really is. Should we open discussion of
> changing the policy?
>

That sounds reasonable.

Sincerely,

Joshua D. Drake

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXb93ATb/zqfZUUQRAvdXAJkBuL3h1ZRA3j40JBJ9V1pWglF1uACdFBYr
e+234rZRsjT+2AqFoyoQ2Nc=
=33Td
-----END PGP SIGNATURE-----

Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> On Mon, 10 Dec 2007, Bruce Momjian wrote:
>> That is an excellent suggestion, done:
>>
>> Full text search is integrated into the core database
>> system (Teodor, Oleg, Stefan Kaltenbrunner)

> Wait, I think we need more words about original authors !

I agree, that is an *utterly* horrid change, as it makes it look like
Stefan's contribution was of the same order of magnitude as Teodor's
and Oleg's. I don't mind giving Stefan an appropriate amount of credit,
but this is not it --- in fact, I read this as a direct insult to the
two of them. They have spent years on tsearch; I doubt he spent more
than a day.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 10 Dec 2007 17:56:01 -0500
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> > On Mon, 10 Dec 2007, Bruce Momjian wrote:
> >> That is an excellent suggestion, done:
> >>
> >> Full text search is integrated into the core database
> >> system (Teodor, Oleg, Stefan Kaltenbrunner)
>
> > Wait, I think we need more words about original authors !
>
> I agree, that is an *utterly* horrid change, as it makes it look like
> Stefan's contribution was of the same order of magnitude as Teodor's
> and Oleg's. I don't mind giving Stefan an appropriate amount of
> credit, but this is not it --- in fact, I read this as a direct
> insult to the two of them. They have spent years on tsearch; I doubt
> he spent more than a day.

That's fair. There is a psql section in the release notes. Why not just
add it there.

Various psql improvements including \d commands for tsearch, \prompt
capability, and better \timing functionality. (Andrew Dunstan,
Stefan Kaltenbrunner, Tom Lane, Chad Wagner)

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXcWMATb/zqfZUUQRAkjlAKCotR8AE8SWmvbUPLH8+s0E9LmIzACfSKW1
Zwnth6lhpbah7PJ/NAvY8RE=
=d2gZ
-----END PGP SIGNATURE-----

Tom Lane wrote:
> Oleg Bartunov <oleg(at)sai(dot)msu(dot)su> writes:
> > On Mon, 10 Dec 2007, Bruce Momjian wrote:
> >> That is an excellent suggestion, done:
> >>
> >> Full text search is integrated into the core database
> >> system (Teodor, Oleg, Stefan Kaltenbrunner)
>
> > Wait, I think we need more words about original authors !
>
> I agree, that is an *utterly* horrid change, as it makes it look like
> Stefan's contribution was of the same order of magnitude as Teodor's
> and Oleg's. I don't mind giving Stefan an appropriate amount of credit,
> but this is not it --- in fact, I read this as a direct insult to the
> two of them. They have spent years on tsearch; I doubt he spent more
> than a day.

By trying to make one developer happy I have made two unhappy.
I have remved Stefan Kaltenbrunner's name from that item.

Basically I should have expected this because once we divert from a
clear policy we are going to cause an imbalance. Unless we change the
policy I am not going to make any adjustments "just to give credit".

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Mon, 10 Dec 2007 18:40:52 -0500 (EST)
Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> By trying to make one developer happy I have made two unhappy.
> I have remved Stefan Kaltenbrunner's name from that item.
>
> Basically I should have expected this because once we divert from a
> clear policy we are going to cause an imbalance. Unless we change the
> policy I am not going to make any adjustments "just to give credit".
>

See my other email in response to Tom. However, I would note that there
is no clear policy. There is clear policy to you and Tom but I have
been here a lot of years and didn't know about it.

If that is the policy, fine :) but it needs to get documented somewhere
that is no buried in a list archive that gets thousands of messages a
month.

Sincerely,

Joshua D. Drake

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHXdFOATb/zqfZUUQRAhkrAJ45fgaJ0EIw2xZ4XMZMYGeVWHxvLgCeLAcG
A9h4i3ULbmZK0+1UZnazeew=
=KU7V
-----END PGP SIGNATURE-----

Joshua D. Drake wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Mon, 10 Dec 2007 18:40:52 -0500 (EST)
> Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
> > By trying to make one developer happy I have made two unhappy.
> > I have removed Stefan Kaltenbrunner's name from that item.
> >
> > Basically I should have expected this because once we divert from a
> > clear policy we are going to cause an imbalance. Unless we change the
> > policy I am not going to make any adjustments "just to give credit".
> >
>
> See my other email in response to Tom. However, I would note that there
> is no clear policy. There is clear policy to you and Tom but I have
> been here a lot of years and didn't know about it.
>
> If that is the policy, fine :) but it needs to get documented somewhere
> that is no buried in a list archive that gets thousands of messages a
> month.

Ah, glad you asked. It is now in the release note introduction that I
added as part of this discussion:

http://momjian.us/main/writings/pgsql/sgml/release-introduction.html

I realize it wasn't documented but the issue was always open for
discussion, as you have seen. I think this is the first time we really
talked about it because in the past no one had an issue with the policy.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:

> Ah, glad you asked. It is now in the release note introduction that I
> added as part of this discussion:
>
> http://momjian.us/main/writings/pgsql/sgml/release-introduction.html
>
> I realize it wasn't documented but the issue was always open for
> discussion, as you have seen. I think this is the first time we really
> talked about it because in the past no one had an issue with the policy.
>

Sure :). It is good to rehash things occasionally. It keeps us on our
toes and up with the times.

I do still think we need to figure out a way to resolve the underlying
issue though. Did you see my email to Tom on the matter? About grouping?

Sincerely,

Joshua D. Drake

On Dec 10, 2007 6:43 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:

>
>
> Pavan Deolasee wrote:
> >
> >
> >
> >
> >
> > I don't agree completely. HOT updates is just one significant benefit of
> > HOT and is constrained by the non-index column updates. But the other
> > major benefit of truncating the tuples to their line pointers applies to
> > HOT as well as COLD updates and DELETEs. This should also have
> > a non trivial positive impact on the performance.
> >
> > There might be few scenarios where HOT may not show any improvement
> > such as CPU-bound applications, but I am not sure if its worth
> mentioning.
> >
> >
>
>
> Um, I don't understand. I freely admit that I haven't kept up with all
> the nuances of the HOT discussions, but this bit has totally eluded me,
> so please elucidate.
>
>

One of the improvements of HOT is to truncate a DEAD tuple to its
line pointer. A DEAD tuple could be an old version of an updated
tuple or a deleted tuple. When a tuple is truncated, the space used
by the line pointer can not be reused (until the index entries are removed).
But the space used by the actual tuple can be reused for a later update,
after
the page is defragmented. Note that this defragmentation can happen
outside of a VACUUM.

This gives us an ability to run VACUUM less frequently on a table. We
still need to run VACUUM to remove the line pointer bloat, but may be less
frequently for the given percentage of bloat. IMHO this should have a
positive
effect on performance atleast in an IO bound scenario.

Thanks,
Pavan

--
Pavan Deolasee
EnterpriseDB http://www.enterprisedb.com

Pavan Deolasee wrote:
> One of the improvements of HOT is to truncate a DEAD tuple to its
> line pointer. A DEAD tuple could be an old version of an updated
> tuple or a deleted tuple. When a tuple is truncated, the space used
> by the line pointer can not be reused (until the index entries are removed).
> But the space used by the actual tuple can be reused for a later update,
> after
> the page is defragmented. Note that this defragmentation can happen
> outside of a VACUUM.
>
> This gives us an ability to run VACUUM less frequently on a table. We
> still need to run VACUUM to remove the line pointer bloat, but may be less
> frequently for the given percentage of bloat. IMHO this should have a
> positive
> effect on performance atleast in an IO bound scenario.

I have updated the release note item title to mention DELETE as well:

Heap-Only Tuples (<acronym>HOT</>) accelerate space reuse for
most <command>UPDATE</>s and <command>DELETE</>s

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +