Re: Per tuple overhead, cmin, cmax

Hi,

having been chased away from pgsql-novice by Rasmus Mohr, I come here
to try my luck :-) I'm still new to this; so please be patient, if I
ask silly questions.

There has been a discussion recently about saving some bytes per tuple
header. Well, I have the suspicion, we can eliminate 4 bytes more by
using one single field for t_cmin and t_cmax. Here are my thoughts
supporting this feeling:

(1) The purpose of the command ids is to prevent a command from
working on the same tuple more than once. 2002-04-20 Tom Lane wrote
in "On-disk Tuple Size":
> You don't want an individual command's effects to become visible
> until you do CommandCounterIncrement.
and
> The command IDs aren't interesting anymore once the originating
> transaction is over, but I don't see a realistic way to recycle the
> space ...

(2) The command ids are of any meaning (2a) only during the active
transaction (with the exception of the (ab)use of t_cmin by vacuum);
and (2b) we are only interested in whether t_cxxx is less than the
current command id, if it is, we don't care about the exact value.

(3) From a command's view a tuple can be in one of these states:

(3a) Neither t_xmin nor t_xmax is the current transaction. The tuple
has been neither inserted nor deleted (and thus not updated) by this
transaction, and the command ids are irrelevant.

(3b) t_xmin is the current transaction, t_xmax is
InvalidTransactionId; i.e. the tuple has been inserted by the current
transaction and it has not been deleted (or replaced). In this case
t_cmin identifies the command having inserted the tuple, and t_cmax is
irrelevant.

(3c) t_xmin is some other transaction, t_xmax is the current
transaction; i.e. the current transaction has deleted the tuple.
Then t_cmax identifies the command having deleted the tuple, t_cmin is
irrelevant.

(3d) t_xmin == t_xmax == current transaction. The tuple has been
inserted and then deleted by the current transaction. Then I claim
(but I'm not absolutely sure), that insert and delete cannot have
happened in the same command,
so t_cmin < t_cmax,
so t_cmin < CurrentCommandId,
so the exact value of t_cmin is irrelevant.

So at any moment at most one of the two fields t_cmin and t_cmax is
needed.

(4) If (3) is true, we can have a single field t_cnum in
HeapTupleHeaderData, the meaning of which is t_cmax, if t_xmax is the
current transaction, otherwise t_cmin.

t_cmin is used in:
. backend/access/common/heaptuple.c
. backend/access/heap/heapam.c
. backend/access/transam/xlogutils.c
. backend/commands/vacuum.c
. backend/utils/time/tqual.c

t_cmax is used in:
. backend/access/common/heaptuple.c
. backend/access/heap/heapam.c
. backend/utils/time/tqual.c

As far as I have checked these sources (including the abuse of c_tmin
by vacuum) my suggested change should be possible, but as I told you
I'm new here and so I have the following questions:

(Q1) Is assumption (3d) true? Do you have any counter examples?

(Q2) Is there any possibiltity of t_cmax being set and t_cmin still
being needed? (Preferred answer: no :-)

(Q3) Are my thoughts WAL compatible?

(Q4) Is it really easy to change the size of HeapTupleHeaderData? Are
the data of this struct only accessed by field names or are there
dirty tricks using memcpy() and pointer arithmetic?

(Q5) Are these thoughts obsolete as soon as nested transactions are
considered?

Thank you for reading this long message.

Servus
Manfred

Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> (3d) t_xmin == t_xmax == current transaction. The tuple has been
> inserted and then deleted by the current transaction. Then I claim
> (but I'm not absolutely sure), that insert and delete cannot have
> happened in the same command,
> so t_cmin < t_cmax,
> so t_cmin < CurrentCommandId,
> so the exact value of t_cmin is irrelevant.

The hole in this logic is that there can be multiple active scans with
different values of CurrentCommandId (eg, within a function
CurrentCommandId may be different than it is outside). If you overwrite
cmin with cmax then you are destroying the information needed by a scan
with smaller CurrentCommandId than yours.

> (Q4) Is it really easy to change the size of HeapTupleHeaderData? Are
> the data of this struct only accessed by field names or are there
> dirty tricks using memcpy() and pointer arithmetic?

AFAIK there are no dirty tricks there. I am hesitant to change the
header layout without darn good reason, because it breaks any chance
of having a working pg_upgrade process. But that's strictly a
production-system concern, and need not discourage you from
experimenting.

> (Q5) Are these thoughts obsolete as soon as nested transactions are
> considered?

Possibly. We haven't worked out exactly how nested transactions would
work, but to the extent that they are handled as different CommandIds
we'd have the same issue already mentioned: we should not assume that
execution of different CommandIds can't overlap in time.

regards, tom lane

Tom,
thanks for answering.

On Thu, 02 May 2002 17:16:38 -0400, you wrote:
>The hole in this logic is that there can be multiple active scans with
>different values of CurrentCommandId (eg, within a function
>CurrentCommandId may be different than it is outside). If you overwrite
>cmin with cmax then you are destroying the information needed by a scan
>with smaller CurrentCommandId than yours.

Oh, I see :-(
Let me throw in one of my infamous wild ideas in an attempt to rescue
my proposal: We have 4 32-bit-numbers: xmin, cmin, xmax, and cmax.
The only case, where we need cmin *and* cmax, is, when xmin == xmax.
So if we find a single bit to flag this case, we only need 3
32-bit-numbers to store this information on disk.

To keep the code readable we probably would need some accessor
functions or macros to access these fields.

As of 7.2 there are three unused bits in t_infomask.

>
>> (Q4) Is it really easy to change the size of HeapTupleHeaderData? Are
>> the data of this struct only accessed by field names or are there
>> dirty tricks using memcpy() and pointer arithmetic?
>
>AFAIK there are no dirty tricks there. I am hesitant to change the
>header layout without darn good reason, because it breaks any chance
>of having a working pg_upgrade process. But that's strictly a
>production-system concern, and need not discourage you from
>experimenting.

Is saving 4 bytes per tuple a "darn good reason"? Is a change
acceptable for 7.3? Do you think it's worth the effort?

> We haven't worked out exactly how nested transactions would
>work, but to the extent that they are handled as different CommandIds
>we'd have the same issue already mentioned: we should not assume that
>execution of different CommandIds can't overlap in time.

Assuming that a subtransaction is completely contained in the outer
transaction and there is no activity by the outer transaction while
the subtransaction is active, I believe, this problem can be solved
...
It's late now, I'll try to think clearer tomorrow.

Good night
Manfred

Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> Let me throw in one of my infamous wild ideas in an attempt to rescue
> my proposal: We have 4 32-bit-numbers: xmin, cmin, xmax, and cmax.
> The only case, where we need cmin *and* cmax, is, when xmin == xmax.
> So if we find a single bit to flag this case, we only need 3
> 32-bit-numbers to store this information on disk.

Hmm ... that might work. Actually, we are trying to stuff *five*
numbers into these fields: xmin, xmax, cmin, cmax, and a VACUUM FULL
transaction id (let's call it xvac just to have a name). The code
currently assumes that cmin is not interesting simultaneously with xvac.
I think it might be true that cmax is not interesting simultaneously
with xvac either, in which case this could be made to work. (Vadim,
your thoughts?)

> To keep the code readable we probably would need some accessor
> functions or macros to access these fields.

Amen. But that would be cleaner than now, at least for VACUUM;
it's just using cmin where it means xvac.

> Is saving 4 bytes per tuple a "darn good reason"? Is a change
> acceptable for 7.3? Do you think it's worth the effort?

I'm on the fence about it. My thoughts are probably colored by the
fact that I prefer platforms that have MAXALIGN=8, so half the time
(including all null-free rows) there'd be no savings at all. Now if
we could get rid of 8 bytes in the header, I'd get excited ;-)

Any other opinions out there?

regards, tom lane

PS: I did like your point about BITMAPLEN; I think that might be
a free savings. I was waiting for you to bring it up on hackers
before commenting though...

On Thu, 02 May 2002 21:10:40 -0400, Tom Lane wrote:

>Hmm ... that might work. Actually, we are trying to stuff *five*
>numbers into these fields: xmin, xmax, cmin, cmax, and a VACUUM FULL
>transaction id (let's call it xvac just to have a name). The code
>currently assumes that cmin is not interesting simultaneously with xvac.
>I think it might be true that cmax is not interesting simultaneously
>with xvac either, in which case this could be made to work. (Vadim,
>your thoughts?)
Having read the sources recently I'm pretty sure you're right.

>I'm on the fence about it. My thoughts are probably colored by the
>fact that I prefer platforms that have MAXALIGN=8, so half the time
>(including all null-free rows) there'd be no savings at all.
But the other half of the time we'd save 8 bytes. So on average we
get savings of 4 bytes per tuple, don't we?

> Now if
>we could get rid of 8 bytes in the header, I'd get excited ;-)
I keep trying :-)

Servus
Manfred

On Thu, 02 May 2002 21:10:40 -0400, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:
>Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
>> Is saving 4 bytes per tuple a "darn good reason"?
>
>[...] Now if
>we could get rid of 8 bytes in the header, I'd get excited ;-)

Tom,

what about WITHOUT OIDS? I know dropping the OID from some tables and
keeping it for others is not trivial, because t_oid is the _first_
field of HeapTupleHeaderData. I'm vaguely considering a few possible
implementations and will invest more work in a detailed proposal, if
it's wanted.

Servus
Manfred

Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> what about WITHOUT OIDS? I know dropping the OID from some tables and
> keeping it for others is not trivial, because t_oid is the _first_
> field of HeapTupleHeaderData. I'm vaguely considering a few possible
> implementations and will invest more work in a detailed proposal, if
> it's wanted.

Yeah, I had been toying with the notion of treating OID like a user
field --- ie, it'd be present in the variable-length part of the record
if at all. It'd be a bit tricky to find all the places that would need
to change, but I think there are not all that many.

As usual, the major objection to any such change is losing the chance
of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
cycle either. If we put together several such changes and did them
all at once, the benefit might be enough to overcome that complaint.

regards, tom lane

On Tue, 21 May 2002 09:57:32 -0400, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:
>Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
>> what about WITHOUT OIDS? I know dropping the OID from some tables and
>> keeping it for others is not trivial, because t_oid is the _first_
>> field of HeapTupleHeaderData. I'm vaguely considering a few possible
>> implementations and will invest more work in a detailed proposal, if
>> it's wanted.
>
>Yeah, I had been toying with the notion of treating OID like a user
>field --- ie, it'd be present in the variable-length part of the record
>if at all. It'd be a bit tricky to find all the places that would need
>to change, but I think there are not all that many.
That was one of the possible solutions I thought of, unfortunately the
one I'm most afraid of. Not because I think it's not the cleanest
way, but I don't (yet) feel comfortable enough with the code to rip
out oids from system tables. However, if you tell me it's feasible
and if you give me some hints where to start, I'll give it a try...

Other possible implementations would leave the oid in the tuple
header:

. typedef two structs HeapTupleHeaderDataWithOid and
HeapTupleHeaderDataWithoutOid, wrap access to *all* HeapTupleHeader
fields in accessor macros/functions, give these accessors enough
information to know which variant to use.

. Decouple on-disk format from in-memory structures, use
HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
header data to/from disk buffers. Concurrency?

>As usual, the major objection to any such change is losing the chance
>of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
>cycle either.

I thought, it is quite common to need pg_dump/restore when upgrading
between releases. Or are you talking about those hackers, testers and
users(?), who are using a cvs version now?

Anyway, as long as our changes don't make heap tuples larger, it
should be possible to write a tool that converts version x data files
to version y data files. I've done that before (not for PG though)
and I know it's a lot of work, but wouldn't it be great for the PG
marketing department ;-)

>If we put together several such changes [...]

I can't guarantee that; my ideas come in drop by drop :-)
BTW, is there a 7.3 schedule?

Servus
Manfred

Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> That was one of the possible solutions I thought of, unfortunately the
> one I'm most afraid of. Not because I think it's not the cleanest
> way, but I don't (yet) feel comfortable enough with the code to rip
> out oids from system tables.

The system tables that have OIDs will certainly continue to have OIDs.

I suppose the messiest aspect of that solution would be changing all
the places that currently do "tuple->t_data->t_oid". If OID is not at
a fixed offset in the tuple then it'll be necessary to change *all*
those places. Ugh. While certainly we should have been using accessor
macros for that, I'm not sure I want to try to change it.

> Other possible implementations would leave the oid in the tuple
> header:

> . typedef two structs HeapTupleHeaderDataWithOid and
> HeapTupleHeaderDataWithoutOid, wrap access to *all* HeapTupleHeader
> fields in accessor macros/functions, give these accessors enough
> information to know which variant to use.

If OID is made to be the last fixed-offset field, instead of the first,
then this approach would be fairly workable. Actually I'd still use
just one struct definition, but do offsetof() calculations to decide
where the null-bitmap starts.

> Decouple on-disk format from in-memory structures, use
> HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
> header data to/from disk buffers. Concurrency?

Inefficient, and you'd have problems still with the changeable fields
(t_infomask etc).

>> As usual, the major objection to any such change is losing the chance
>> of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
>> cycle either.

> I thought, it is quite common to need pg_dump/restore when upgrading
> between releases.

Yes, and we get loud complaints every time we require it...

> Anyway, as long as our changes don't make heap tuples larger, it
> should be possible to write a tool that converts version x data files
> to version y data files. I've done that before (not for PG though)
> and I know it's a lot of work, but wouldn't it be great for the PG
> marketing department ;-)

I'd be afraid to use a conversion-in-place tool for this sort of thing.
If it crashes halfway through, what then?

regards, tom lane

On Tue, 21 May 2002 11:53:04 -0400, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:
>The system tables that have OIDs will certainly continue to have OIDs.

That's clear. I should have written: "... rip out oids from tuple
headers of system tables."

>Ugh. While certainly we should have been using accessor
>macros for that, I'm not sure I want to try to change it.

I already did this for xmin, xmax, cmin, cmax, and xvac (see my patch
posted 2002-05-12).

>If OID is made to be the last fixed-offset field, instead of the first,

That would introduce some padding.

>then this approach would be fairly workable. Actually I'd still use
>just one struct definition, but do offsetof() calculations to decide
>where the null-bitmap starts.

... and for calculating the tuple header size.

>> Decouple on-disk format from in-memory structures, use
>> HeapTupleHeaderPack() and HeapTupleHeaderUnpack() to store/extract
>> header data to/from disk buffers. Concurrency?
>
>Inefficient,

Just to be sure: You mean the CPU cycles wasted by Pack() and
Unpack()?

>I'd be afraid to use a conversion-in-place tool for this sort of thing.

Me too. No, not in place! I thought of a filter reading an old
format data file, one page at a time, and writing a new format data
file. This would work as long as the conversions don't cause page
overflow.

No comment on a planned 7.3 timeframe? :-(

Servus
Manfred

Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> No comment on a planned 7.3 timeframe? :-(

I think we are planning to go beta in late summer (end of August, say).
Probably in July we'll start pressing people to finish up any major
development items, or admit that they won't happen for 7.3. So we've
still got a couple months of full-tilt development mode before we
start to worry about tightening up for release.

regards, tom lane

Tom Lane wrote:
> Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> > No comment on a planned 7.3 timeframe? :-(
>
> I think we are planning to go beta in late summer (end of August, say).
> Probably in July we'll start pressing people to finish up any major
> development items, or admit that they won't happen for 7.3. So we've
> still got a couple months of full-tilt development mode before we
> start to worry about tightening up for release.

I am concerned about slowing down too early. We did that in previous
releases and didn't get the beta focus we needed, and it was too
paralyzing on people to know what is to be slowed and what to keep
going. I think a slowdown two weeks before beta would be fine.

Basically, I think the slowdown lengthened the time we were not doing
something productive.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Tom Lane wrote:
> Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> > what about WITHOUT OIDS? I know dropping the OID from some tables and
> > keeping it for others is not trivial, because t_oid is the _first_
> > field of HeapTupleHeaderData. I'm vaguely considering a few possible
> > implementations and will invest more work in a detailed proposal, if
> > it's wanted.
>
> Yeah, I had been toying with the notion of treating OID like a user
> field --- ie, it'd be present in the variable-length part of the record
> if at all. It'd be a bit tricky to find all the places that would need
> to change, but I think there are not all that many.
>
> As usual, the major objection to any such change is losing the chance
> of doing pg_upgrade. But we didn't have pg_upgrade during the 7.2
> cycle either. If we put together several such changes and did them
> all at once, the benefit might be enough to overcome that complaint.

I think it is inevitable that there be enough binary file changes the
pg_upgrade will not work for 7.3 --- it just seems it is only a matter
of time.

One idea is to allow alternate page layouts using the heap page version
number, but that will be difficult/confusing in the code.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

> Tom Lane wrote:
> > Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> > > No comment on a planned 7.3 timeframe? :-(
> >
> > I think we are planning to go beta in late summer (end of August, say).
> > Probably in July we'll start pressing people to finish up any major
> > development items, or admit that they won't happen for 7.3. So we've
> > still got a couple months of full-tilt development mode before we
> > start to worry about tightening up for release.
>
> I am concerned about slowing down too early. We did that in previous
> releases and didn't get the beta focus we needed, and it was too
> paralyzing on people to know what is to be slowed and what to keep
> going. I think a slowdown two weeks before beta would be fine.

Other then personal slow downs, there is no reason for anything to "slow
down" until beta itself starts ...

Marc G. Fournier wrote:
> On Fri, 7 Jun 2002, Bruce Momjian wrote:
>
> > Tom Lane wrote:
> > > Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> > > > No comment on a planned 7.3 timeframe? :-(
> > >
> > > I think we are planning to go beta in late summer (end of August, say).
> > > Probably in July we'll start pressing people to finish up any major
> > > development items, or admit that they won't happen for 7.3. So we've
> > > still got a couple months of full-tilt development mode before we
> > > start to worry about tightening up for release.
> >
> > I am concerned about slowing down too early. We did that in previous
> > releases and didn't get the beta focus we needed, and it was too
> > paralyzing on people to know what is to be slowed and what to keep
> > going. I think a slowdown two weeks before beta would be fine.
>
> Other then personal slow downs, there is no reason for anything to "slow
> down" until beta itself starts ...

I assume Tom doesn't want a huge patch applied the day before beta, and
I can understand that, but patch problems usually appear within two
weeks of application, so I think we only have to worry about those last
two weeks. Of course, another option is to continue in full development
until the end of August, then start beta in September as soon as the
code is stable.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

> Marc G. Fournier wrote:
> > On Fri, 7 Jun 2002, Bruce Momjian wrote:
> >
> > > Tom Lane wrote:
> > > > Manfred Koizar <mkoi-pg(at)aon(dot)at> writes:
> > > > > No comment on a planned 7.3 timeframe? :-(
> > > >
> > > > I think we are planning to go beta in late summer (end of August, say).
> > > > Probably in July we'll start pressing people to finish up any major
> > > > development items, or admit that they won't happen for 7.3. So we've
> > > > still got a couple months of full-tilt development mode before we
> > > > start to worry about tightening up for release.
> > >
> > > I am concerned about slowing down too early. We did that in previous
> > > releases and didn't get the beta focus we needed, and it was too
> > > paralyzing on people to know what is to be slowed and what to keep
> > > going. I think a slowdown two weeks before beta would be fine.
> >
> > Other then personal slow downs, there is no reason for anything to "slow
> > down" until beta itself starts ...
>
> I assume Tom doesn't want a huge patch applied the day before beta, and

No offence to Tom, but who cares whether Tom wants a huge patch or not?

> I can understand that, but patch problems usually appear within two
> weeks of application, so I think we only have to worry about those last
> two weeks. Of course, another option is to continue in full development
> until the end of August, then start beta in September as soon as the
> code is stable.

That kinda was the plan ... code will be frozen around the 1st of
September, with a release packaged then that will be label'd beta1 ...
regardless of how stable it is ... beta is "we're not taking any more
changes except fixes, so pound on this and let us know what needs to be
fixed" ... there shouldn't be any 'lead up' to beta, the lead up is the
development period itself ... *shrug*

Marc G. Fournier wrote:
> On Fri, 7 Jun 2002, Bruce Momjian wrote:
> > > > I am concerned about slowing down too early. We did that in previous
> > > > releases and didn't get the beta focus we needed, and it was too
> > > > paralyzing on people to know what is to be slowed and what to keep
> > > > going. I think a slowdown two weeks before beta would be fine.
> > >
> > > Other then personal slow downs, there is no reason for anything to "slow
> > > down" until beta itself starts ...
> >
> > I assume Tom doesn't want a huge patch applied the day before beta, and
>
> No offence to Tom, but who cares whether Tom wants a huge patch or not?

Well, he does clean up lots of breakage, so I was willing to settle on a
compromise of 2 weeks. However, if other feel we should just go
full-bore until beta starts, I am fine with that.

I will say that I was disapointed by previous release delays and will be
more vocal about moving things forward than I have in the past.

> > I can understand that, but patch problems usually appear within two
> > weeks of application, so I think we only have to worry about those last
> > two weeks. Of course, another option is to continue in full development
> > until the end of August, then start beta in September as soon as the
> > code is stable.
>
> That kinda was the plan ... code will be frozen around the 1st of
> September, with a release packaged then that will be label'd beta1 ...
> regardless of how stable it is ... beta is "we're not taking any more
> changes except fixes, so pound on this and let us know what needs to be
> fixed" ... there shouldn't be any 'lead up' to beta, the lead up is the
> development period itself ... *shrug*

Agreed.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002, Bruce Momjian wrote:

> I will say that I was disapointed by previous release delays and will be
> more vocal about moving things forward than I have in the past.

I don't know ... I kinda like being able to confidently say to clients
that "the latest release is always the most stable/reliable" and not being
burned by it :) Hell, last release, I started moving to v7.2 on some of
our production servers around beta4 or so, since a) I was confident and b)
I needed some of the features ...

Not many software packages you can say that about anymore, eh?

Marc G. Fournier wrote:
> On Fri, 7 Jun 2002, Bruce Momjian wrote:
>
> > I will say that I was disappointed by previous release delays and will be
> > more vocal about moving things forward than I have in the past.
>
> I don't know ... I kinda like being able to confidently say to clients
> that "the latest release is always the most stable/reliable" and not being
> burned by it :) Hell, last release, I started moving to v7.2 on some of
> our production servers around beta4 or so, since a) I was confident and b)
> I needed some of the features ...
>
> Not many software packages you can say that about anymore, eh?

Yes, true, I am not going to push the schedule. What I want to try to
do is keep people more informed of where we are in the release process,
and more informed about the open issues involved.

That squishy freeze is one of those items that is very hard to organize
because it is so arbitrary about what should be added. I think we need
to communicate CODE-CODE-CODE until September 1, then communicate
TEST-TEST-TEST after that. We will not release before it is ready, but
we will marshall our forces better.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Fri, 7 Jun 2002 02:01:40 -0400 (EDT), Bruce Momjian
<pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
>I think it is inevitable that there be enough binary file changes the
>pg_upgrade will not work for 7.3 --- it just seems it is only a matter
>of time.

As far as it concerns changes proposed by me, I'll (try to) provide a
conversion program, if that's necessary for my patches to be accepted.
Then move_objfiles() in pg_update would have to call pg_convert, or
whatever we call it, instead of mv. And yes, users would need twice
the disk space during pg_upgrade.

>One idea is to allow alternate page layouts using the heap page version
>number, but that will be difficult/confusing in the code.

This is a good idea, IMHO. By saying "*the* heap page version number"
do you mean, that there already is such a number by now? I could not
find one in bufpage.h. Should I have looked somewhere else?

While we're at it, does each file start with a magic number
identifying its type? AFAICS nbtree does; but I can't tell for heap
and have not yet checked for rtree, gist, ... This is the reason for
the "try to" in the first paragraph.

Servus
Manfred

Manfred Koizar wrote:
> On Fri, 7 Jun 2002 02:01:40 -0400 (EDT), Bruce Momjian
> <pgman(at)candle(dot)pha(dot)pa(dot)us> wrote:
> >I think it is inevitable that there be enough binary file changes the
> >pg_upgrade will not work for 7.3 --- it just seems it is only a matter
> >of time.
>
> As far as it concerns changes proposed by me, I'll (try to) provide a
> conversion program, if that's necessary for my patches to be accepted.
> Then move_objfiles() in pg_update would have to call pg_convert, or
> whatever we call it, instead of mv. And yes, users would need twice
> the disk space during pg_upgrade.

>
> >One idea is to allow alternate page layouts using the heap page version
> >number, but that will be difficult/confusing in the code.
>
> This is a good idea, IMHO. By saying "*the* heap page version number"
> do you mean, that there already is such a number by now? I could not
> find one in bufpage.h. Should I have looked somewhere else?

Oops, I see I added to TODO lately:

* Add version file format stamp to heap and other table types

Guess we would have to add it. Btree has it in nbtree.h:

uint32 btm_version;

I though heap should have it too. Of course, there are problems with
having the tuple read _know_ its version, and preventing mixing of
tuples of different versions in the same page.

>
> While we're at it, does each file start with a magic number
> identifying its type? AFAICS nbtree does; but I can't tell for heap
> and have not yet checked for rtree, gist, ... This is the reason for
> the "try to" in the first paragraph.

Yep, only btree. I guess I didn't add it because it would cause
problems for pg_upgrade. :-)

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Uh guys ... what I *said* was:

> I think we are planning to go beta in late summer (end of August, say).
> Probably in July we'll start pressing people to finish up any major
> development items, or admit that they won't happen for 7.3.

By which I meant that in July we should start hounding anyone who's got
major unfinished work. (Like, say, me, if the schema changes are still
incomplete then.) Not that we won't accept the work when it gets here,
just that that'll be the time to start demanding closure on big 7.3
changes.

And yes, I *would* be pretty upset with the idea of applying major
patches in the last weeks of August, if they are changes that pop up
out-of-the-blue at that time. If it's finishing up work that the
community has already approved, that's a different scenario. But big,
poorly-reviewed feature additions right before beta are exactly my idea
of how to mess up that reputation for stability that Marc was touting...

regards, tom lane

Tom Lane wrote:
> Uh guys ... what I *said* was:
>
> > I think we are planning to go beta in late summer (end of August, say).
> > Probably in July we'll start pressing people to finish up any major
> > development items, or admit that they won't happen for 7.3.
>
> By which I meant that in July we should start hounding anyone who's got
> major unfinished work. (Like, say, me, if the schema changes are still
> incomplete then.) Not that we won't accept the work when it gets here,
> just that that'll be the time to start demanding closure on big 7.3
> changes.

OK.

> And yes, I *would* be pretty upset with the idea of applying major
> patches in the last weeks of August, if they are changes that pop up
> out-of-the-blue at that time. If it's finishing up work that the
> community has already approved, that's a different scenario. But big,
> poorly-reviewed feature additions right before beta are exactly my idea
> of how to mess up that reputation for stability that Marc was touting...

Yes, but there is a downside to this. We have trouble enough figuring
out if a patch is a "feature" or "bug fix" during beta. How are people
going to decide if a feature is "big" or not to work on during August?
It has a paralyzing effect on our developers.

Now, I don't want to apply a partially-implemented feature in the last
week of August, but I don't want to slow things down during August,
because the last time we did this we were all looking at each other
waiting for beta, and nothing was getting done. This is the paralyzing
effect I want to avoid.

We have beta for testing. That's were our reliability comes from too.
And last beta, we did almost nothing because we had shut down
development so early, and it dragged because there _wasn't_ a clear line
between development time and beta.

So, I we should:

Warn people in July that beta is September 1 and all features
have to be complete by then, or they get ripped out.

Reject non-complete patches during August, meaning accepted
patches in August have to be fully functional features; no
partial patches and I will work on the rest later.

Vote on any patches where there is disagreement.

So, in summary, for me, August patches have to be 100% complete. That
takes the guess work out of the deadline. There isn't the question of
whether we will accept such a feature or not. The burden is on the
developer to provide a 100% complete patch.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> Now, I don't want to apply a partially-implemented feature in the last
> week of August, but I don't want to slow things down during August,
> because the last time we did this we were all looking at each other
> waiting for beta, and nothing was getting done. This is the paralyzing
> effect I want to avoid.

Well, my take on it is that the reason beta was delayed the last two
go-rounds was that we allowed major work to be committed in an
incomplete state, and then we were stuck waiting for those people to
finish. (The fact that the people in question were core members didn't
improve my opinion of the situation ;-)) I'd like to stop making that
mistake.

> So, I we should:
> Warn people in July that beta is September 1 and all features
> have to be complete by then, or they get ripped out.
> Reject non-complete patches during August, meaning accepted
> patches in August have to be fully functional features; no
> partial patches and I will work on the rest later.

I thought that was more or less the same thing I was proposing...

regards, tom lane

On Sat, 8 Jun 2002, Bruce Momjian wrote:

> Yes, but there is a downside to this. We have trouble enough figuring
> out if a patch is a "feature" or "bug fix" during beta. How are people
> going to decide if a feature is "big" or not to work on during August?
> It has a paralyzing effect on our developers.

How is this any different then our other releases? I think you've totally
lost me as to where the problem is ... reading your above, you are
suggesting that ppl don't work on big projects during the month of August,
since it might not get in for the release? We've never advocated that
before, nor do I believe we should at this point ... in fact, I think its
about time we start dealing with beta using the tools that we have
available ...

Beta starts, we branch out a -STABLE vs -DEVELOPMENT branch in CVS ... we
release a beta1 and deal with bug releases as they come in, followed by a
beta2 until we are ready for release ... I think everyone is old enough
now to be able to decide whatfixed have gone into -STABLE that should be
reflected in -DEVELOPMENT, no? Our mistake last release wasn't how long
beta lasted, but how long we stalled development ...

Marc G. Fournier wrote:
> On Sat, 8 Jun 2002, Bruce Momjian wrote:
>
> > Yes, but there is a downside to this. We have trouble enough figuring
> > out if a patch is a "feature" or "bug fix" during beta. How are people
> > going to decide if a feature is "big" or not to work on during August?
> > It has a paralyzing effect on our developers.
>
> How is this any different then our other releases? I think you've totally
> lost me as to where the problem is ... reading your above, you are
> suggesting that ppl don't work on big projects during the month of August,
> since it might not get in for the release? We've never advocated that
> before, nor do I believe we should at this point ... in fact, I think its
> about time we start dealing with beta using the tools that we have
> available ...

In previous releases, we had this "it is too close to beta to add
feature X" mentality, and I see Tom reiterating that in his email.

It is the idea were are supposed to go into beta with a bug-free release
that bother me. August is prime time for open-source development. Many
countries have holidays, and business is slow, so people have time to
work projects. Let's use that time to improve PostgreSQL, and leave
beta for fixing.

> Beta starts, we branch out a -STABLE vs -DEVELOPMENT branch in CVS ... we
> release a beta1 and deal with bug releases as they come in, followed by a
> beta2 until we are ready for release ... I think everyone is old enough
> now to be able to decide whatfixed have gone into -STABLE that should be
> reflected in -DEVELOPMENT, no? Our mistake last release wasn't how long
> beta lasted, but how long we stalled development ...

Agreed. Let's split sometime during beta as soon as we are ready to
work on 7.4. Sooner and we just double-patch for no purpose, later and
we stall development.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > Now, I don't want to apply a partially-implemented feature in the last
> > week of August, but I don't want to slow things down during August,
> > because the last time we did this we were all looking at each other
> > waiting for beta, and nothing was getting done. This is the paralyzing
> > effect I want to avoid.
>
> Well, my take on it is that the reason beta was delayed the last two
> go-rounds was that we allowed major work to be committed in an
> incomplete state, and then we were stuck waiting for those people to
> finish. (The fact that the people in question were core members didn't
> improve my opinion of the situation ;-)) I'd like to stop making that
> mistake.

I am going to recommend disabling features that people can't fix in a
timely manner during beta. Sounds harsh, but we can't have the whole
project waiting on one person to have a free weekend. If they can
generate a patch, we can re-enable the feature, but we need to get some
discipline for everyone's benefit. I don't think any of us wants to be
embarrassed by the beta duration again.

> > So, I we should:
> > Warn people in July that beta is September 1 and all features
> > have to be complete by then, or they get ripped out.
> > Reject non-complete patches during August, meaning accepted
> > patches in August have to be fully functional features; no
> > partial patches and I will work on the rest later.
>
> I thought that was more or less the same thing I was proposing...

This is the text I objected to:

Tom Lane wrote:
> And yes, I *would* be pretty upset with the idea of applying major
> patches in the last weeks of August, if they are changes that pop up
> out-of-the-blue at that time. If it's finishing up work that the
> community has already approved, that's a different scenario. But big,
> poorly-reviewed feature additions right before beta are exactly my idea
> of how to mess up that reputation for stability that Marc was touting...

It emphasizes August as primarily finish-up time. And there is that
"pre-approved" part I don't like. Feature has to be done by the end of
August, doesn't matter whether it is approved or not. If someone wants
to start and complete a feature during August, "go ahead" is my moto.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Sat, 8 Jun 2002, Bruce Momjian wrote:

> It is the idea were are supposed to go into beta with a bug-free release
> that bother me.

But its you that's always tried to advocate that ... no? If not, then I
am confused, cause I know *I've* never ... to me, switching to beta mode
has always been the switch from 'add features' to 'fix the bugss to
release' ...

> Agreed. Let's split sometime during beta as soon as we are ready to
> work on 7.4. Sooner and we just double-patch for no purpose, later and
> we stall development.

No, let's split *at* beta ... I imagine there are several ppl out there
that have alot of work they wish to do, and sitting around twiddling their
thumbs waiting for someone to *maybe* report a bug in their area/code is a
waste of everyone's time ... not to mention a delay in being ready to
release the next version ...

Hell, each 'beta period' we've done so far has caused that ... where we
get in patches and changes that aren't appropriate for beta and they get
sat on until after its been released ... then you risk the fun of merging
in conflicting changes, so the patch that was perfect when it was
submitted has to be redone because someone else's patch changed enough to
the code that it doesn't apply cleanly ...

Its not like it was years ago when there were a couple of us in the code
... there are enough developers out there now (and growing) that
'stalling' things isn't fair to them (or, in some cases, the companies
that are paying them to develop features) ...

We have the tools to do this, its time to start using them the way they
were meant to be used ...

On Sat, 8 Jun 2002, Bruce Momjian wrote:

> Tom Lane wrote:
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > Now, I don't want to apply a partially-implemented feature in the last
> > > week of August, but I don't want to slow things down during August,
> > > because the last time we did this we were all looking at each other
> > > waiting for beta, and nothing was getting done. This is the paralyzing
> > > effect I want to avoid.
> >
> > Well, my take on it is that the reason beta was delayed the last two
> > go-rounds was that we allowed major work to be committed in an
> > incomplete state, and then we were stuck waiting for those people to
> > finish. (The fact that the people in question were core members didn't
> > improve my opinion of the situation ;-)) I'd like to stop making that
> > mistake.
>
> I am going to recommend disabling features that people can't fix in a
> timely manner during beta. Sounds harsh, but we can't have the whole
> project waiting on one person to have a free weekend. If they can
> generate a patch, we can re-enable the feature, but we need to get some
> discipline for everyone's benefit. I don't think any of us wants to be
> embarrassed by the beta duration again.

I wasn't embarrassed by it ... when I talk to ppl asking about QA on
PostgreSQL, I quite proudly point out that we'd rather delay then release
something we aren't confident about *shrug*

> It emphasizes August as primarily finish-up time. And there is that
> "pre-approved" part I don't like. Feature has to be done by the end of
> August, doesn't matter whether it is approved or not. If someone wants
> to start and complete a feature during August, "go ahead" is my moto.

Personally ... I'm really curious as to why you are even trying to
'formalize' stuff that has been done for years now ... end of August rolls
around and someone submits a feature patch, we do as we've always done ...
we discuss its merits, and its risk factor ... if it presents too high a
risk, it gets put on the 'patch stack' for the next release ... or do you
think our judgement in such matters is such that we have to formalize/set
in stone this common sense stuff beforehand?

I *really* wish ppl would stop harping on the length of the last beta
cycle ... I will always rather delay a release due to an *known*
outstanding bug, especially one that just needs a little bit more time to
work out, then to release software "on time" ala Microsoft ...

Hell, you are trying to set in stone when beta starts (end of august) ...
but with some of the massive changes that we tend to see over the course
of a development project, for all we know, Tom will be 90% finished
something and only need another week to get it complete ... personally,
he's one of many whose code I wouldn't question, so giving another week to
get it done and in, IMHO, is perfectly acceptable ... but, for what you
are trying to get set in stone, it wasn't finished by Sept 1st, so we'll
throw it all out until the next release ...

Right now, Sept 1st is the "preferred date to go beta" ... when Sept 1st
rolls around, like we've always done in the past, we will review that and
if we need to delay a little, we will *shrug*

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> I *really* wish ppl would stop harping on the length of the last beta
> cycle ... I will always rather delay a release due to an *known*
> outstanding bug, especially one that just needs a little bit more time to
> work out, then to release software "on time" ala Microsoft ...

I don't think that's at issue here. No one was suggesting that we'd
force an *end* to beta cycle because of schedule issues. We ship when
we're satisfied and not before. I'm saying that I want to try to
*start* the beta test period on-time, rather than letting the
almost-beta state drag on for months --- which we did in each of the
last two cycles. Development time is productive, and beta-test time
is productive, but we're-trying-to-start-beta time is not very
productive ...

regards, tom lane

On Sun, 9 Jun 2002, Tom Lane wrote:

> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > I *really* wish ppl would stop harping on the length of the last beta
> > cycle ... I will always rather delay a release due to an *known*
> > outstanding bug, especially one that just needs a little bit more time to
> > work out, then to release software "on time" ala Microsoft ...
>
> I don't think that's at issue here. No one was suggesting that we'd
> force an *end* to beta cycle because of schedule issues. We ship when
> we're satisfied and not before. I'm saying that I want to try to
> *start* the beta test period on-time, rather than letting the
> almost-beta state drag on for months --- which we did in each of the
> last two cycles. Development time is productive, and beta-test time
> is productive, but we're-trying-to-start-beta time is not very
> productive ...

Agreed on all accounts ... which is why this time, I want to do a proper
branch when beta starts ... hell, from what I've seen suggested here so
far, we have no choice ... At least then we can 'rip out' something from
the beta tree without having to remove and re-add it to the development
one later, hoping that they're changes haven't been affected by someone
else's ...

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> Agreed on all accounts ... which is why this time, I want to do a proper
> branch when beta starts ... hell, from what I've seen suggested here so
> far, we have no choice ... At least then we can 'rip out' something from
> the beta tree without having to remove and re-add it to the development
> one later, hoping that they're changes haven't been affected by someone
> else's ...

Well, let's give that a try and see how it goes. I'm a bit worried
about the amount of double-patching we'll have to do, but other projects
seem to manage to cope with multiple active branches...

regards, tom lane

On Sun, Jun 09, 2002 at 02:32:22AM -0300, Marc G. Fournier wrote:

> Right now, Sept 1st is the "preferred date to go beta" ... when Sept 1st

I agree with Bruce, Sept 1st is the deadline and right time for all
discussion about shift of this date is Sept 2nd. Not now, else you
never will see end of the cycle :-)

Karel

--
Karel Zak <zakkr(at)zf(dot)jcu(dot)cz>
http://home.zf.jcu.cz/~zakkr/

C, PostgreSQL, PHP, WWW, http://docs.linux.cz, http://mape.jcu.cz

Tom Lane wrote:
> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > I *really* wish ppl would stop harping on the length of the last beta
> > cycle ... I will always rather delay a release due to an *known*
> > outstanding bug, especially one that just needs a little bit more time to
> > work out, then to release software "on time" ala Microsoft ...
>
> I don't think that's at issue here. No one was suggesting that we'd
> force an *end* to beta cycle because of schedule issues. We ship when
> we're satisfied and not before. I'm saying that I want to try to
> *start* the beta test period on-time, rather than letting the
> almost-beta state drag on for months --- which we did in each of the
> last two cycles. Development time is productive, and beta-test time
> is productive, but we're-trying-to-start-beta time is not very
> productive ...

Yes, this was exactly my point. By slowing down in August, we enter
that "almost beta" period where there is uncertainty over what should be
worked on. I know myself I am uncertain what is appropriate to work on,
so I usually end up doing nothing, which is a waste.

I think the only message should be "finish before the end of August".
People can understand that, and it is under the control of the
contributor. The message "no big patches in August" is too imprecise and
leads to uncertainty.

Of course, if we don't finish by the end of August, our new message may
be "finish before the end of September". This brings up another point.
We have delayed beta to wait for single patches in the past, usually a
week at a time. When that week drags to two, and then four, we have
lost development time. If we had just said "four weeks" from the start,
people could have continued development, knowing they had a month, but
our one-week-at-a-time strategy basically holds up the whole group
waiting for single developer to finish a patch. What I am suggesting is
that our small delays for beta are hurting us _if_ the delay drags
longer than anticipated, and we keep pushing back the deadline. In such
cases, we would be better just choosing a longer deadline from the
start. Perhaps we should have delays that are a month at a time.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Tom Lane wrote:
> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > Agreed on all accounts ... which is why this time, I want to do a proper
> > branch when beta starts ... hell, from what I've seen suggested here so
> > far, we have no choice ... At least then we can 'rip out' something from
> > the beta tree without having to remove and re-add it to the development
> > one later, hoping that they're changes haven't been affected by someone
> > else's ...
>
> Well, let's give that a try and see how it goes. I'm a bit worried
> about the amount of double-patching we'll have to do, but other projects
> seem to manage to cope with multiple active branches...

Yes, Marc has been advocating this, and perhaps it is time to give it a
try. There are some downsides:

o All committers need to know that they have to double-patch
o We might have developers working on new features rather than
focusing on beta testing/fixing.

One interesting idea would be to create a branch for 7.4, and apply
_only_ 7.4 patches to that branch. Then, when we release 7.3, we merge
that branch back into the main CVS tree. That would eliminate
double-patching _and_ give people a place to commit 7.4 changes. I
don't think the merge would be too difficult because 7.3 will not change
significantly during beta.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Mon, 10 Jun 2002, Bruce Momjian wrote:

> Tom Lane wrote:
> > "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > > Agreed on all accounts ... which is why this time, I want to do a proper
> > > branch when beta starts ... hell, from what I've seen suggested here so
> > > far, we have no choice ... At least then we can 'rip out' something from
> > > the beta tree without having to remove and re-add it to the development
> > > one later, hoping that they're changes haven't been affected by someone
> > > else's ...
> >
> > Well, let's give that a try and see how it goes. I'm a bit worried
> > about the amount of double-patching we'll have to do, but other projects
> > seem to manage to cope with multiple active branches...
>
> Yes, Marc has been advocating this, and perhaps it is time to give it a
> try. There are some downsides:
>
> o All committers need to know that they have to double-patch

*Wrong* .. only if its a fix for a problem with -STABLE .. otherwise it
*just* goes in the development tree ...

> o We might have developers working on new features rather than
> focusing on beta testing/fixing.

Its not the developers responsibility to beta test the software, its is
their responsibility to test patches as they are applied during the
'development cycle' ... and even after we've branched in the past, ppl
have "fixed reported bugs" and applied such fixes to the -STABLE branch
... why would that be any different now? All we're doing is letting
developers work on their projects instead of sitting on their hands
waiting for a bug report ...

*Plus* ... good chance that any bugs that are reports are in the -DEV
branch also, so it has to be fixed regardless ...

> One interesting idea would be to create a branch for 7.4, and apply
> _only_ 7.4 patches to that branch. Then, when we release 7.3, we merge
> that branch back into the main CVS tree. That would eliminate
> double-patching _and_ give people a place to commit 7.4 changes. I
> don't think the merge would be too difficult because 7.3 will not change
> significantly during beta.

Four words: when hell freezes over

Why must you overcomplicate a process most *large* projects seem to find
sooooo simple to deal with? God, what you are proposing above requires
ppl to predict what v7.3 is going to look like when its finished, so that
their work on v7.4 can follow?

Bruce, I think this whole thread has just about dried up now ... when v7.3
goes beta, we will branch just like other large projects do so that we
don't hold up any developers until we release the software, which, based
on past experiences and history, will end up being delayed ... hell, just
think, we branch on the 1st of Sept, release on the 15 of October (lets
say one month for beta plus a bit of delay), and are ready to go with the
next beta around the 1st of January since we did't lose that 1.5mo of
development time ... wow, imagine a *solid* 4 month development cycle
before beta? :)

Based on everything I've heard/seen in this thread, we seem to be looking
at:

1. Branch on Sept 1st, regardless of almost anything

2. Once Branch created, any *partially implemented* features will get
rip'd out of the -STABLE branch and only fixes to the existing, fully
implement features will go in

3. Beta1 released once developers comfortable with the state of the code

Now, *if*, the week before the Branch, someone submits a bit patch that in
*anyway* concerns someone to apply, we can hold it off for a week and put
it into the -DEV branch so that its not shelved for a couple of months,
and possibly going out of date ... but that would be a judgement call at
the time, nothing set in stone ...

The only thing we are really "setting in stone" here is when we are
branching/freezing the code for release ...

Marc G. Fournier wrote:
> On Mon, 10 Jun 2002, Bruce Momjian wrote:
>
> > Tom Lane wrote:
> > > "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> > > > Agreed on all accounts ... which is why this time, I want to do a proper
> > > > branch when beta starts ... hell, from what I've seen suggested here so
> > > > far, we have no choice ... At least then we can 'rip out' something from
> > > > the beta tree without having to remove and re-add it to the development
> > > > one later, hoping that they're changes haven't been affected by someone
> > > > else's ...
> > >
> > > Well, let's give that a try and see how it goes. I'm a bit worried
> > > about the amount of double-patching we'll have to do, but other projects
> > > seem to manage to cope with multiple active branches...
> >
> > Yes, Marc has been advocating this, and perhaps it is time to give it a
> > try. There are some downsides:
> >
> > o All committers need to know that they have to double-patch
>
> *Wrong* .. only if its a fix for a problem with -STABLE .. otherwise it
> *just* goes in the development tree ...

Not totally wrong. I predict >90% of patches during those first two
weeks will have to be double-applied. Do you disagree? Remember, we
used to branch earlier and double-apply, and it did get confusing when
people forgot to double-patch. I am not saying it is impossible, but I
don't want to minimize it either.

> > o We might have developers working on new features rather than
> > focusing on beta testing/fixing.
>
> Its not the developers responsibility to beta test the software, its is
> their responsibility to test patches as they are applied during the
> 'development cycle' ... and even after we've branched in the past, ppl
> have "fixed reported bugs" and applied such fixes to the -STABLE branch
> ... why would that be any different now? All we're doing is letting
> developers work on their projects instead of sitting on their hands
> waiting for a bug report ...

Yes, good point. Developers are not testing. However, when we do need
people to track down bugs and fixes, I hope they aren't too busy working
on new features to help us.

> *Plus* ... good chance that any bugs that are reports are in the -DEV
> branch also, so it has to be fixed regardless ...
>
> > One interesting idea would be to create a branch for 7.4, and apply
> > _only_ 7.4 patches to that branch. Then, when we release 7.3, we merge
> > that branch back into the main CVS tree. That would eliminate
> > double-patching _and_ give people a place to commit 7.4 changes. I
> > don't think the merge would be too difficult because 7.3 will not change
> > significantly during beta.
>
> Four words: when hell freezes over
>
> Why must you overcomplicate a process most *large* projects seem to find
> sooooo simple to deal with? God, what you are proposing above requires
> ppl to predict what v7.3 is going to look like when its finished, so that
> their work on v7.4 can follow?

Only bug fixes are going into 7.3 during beta, so how much is it going
to change?

And I have done the double-patching, so I remember the problems. Aside
from the hassle of doing everything twice, as development drifts from
beta, the patches do become harder to apply.

> Bruce, I think this whole thread has just about dried up now ... when v7.3
> goes beta, we will branch just like other large projects do so that we
> don't hold up any developers until we release the software, which, based
> on past experiences and history, will end up being delayed ... hell, just
> think, we branch on the 1st of Sept, release on the 15 of October (lets
> say one month for beta plus a bit of delay), and are ready to go with the
> next beta around the 1st of January since we did't lose that 1.5mo of
> development time ... wow, imagine a *solid* 4 month development cycle
> before beta? :)

Yes, it will be good.

> Based on everything I've heard/seen in this thread, we seem to be looking
> at:
>
> 1. Branch on Sept 1st, regardless of almost anything
>
> 2. Once Branch created, any *partially implemented* features will get
> rip'd out of the -STABLE branch and only fixes to the existing, fully
> implement features will go in

Now, that is an interesting idea.

> 3. Beta1 released once developers comfortable with the state of the code
>
> Now, *if*, the week before the Branch, someone submits a bit patch that in
> *anyway* concerns someone to apply, we can hold it off for a week and put
> it into the -DEV branch so that its not shelved for a couple of months,
> and possibly going out of date ... but that would be a judgement call at
> the time, nothing set in stone ...
>
> The only thing we are really "setting in stone" here is when we are
> branching/freezing the code for release ...

OK. I am making these points because the previous betas have been very
disorganized, with lots of wasted time. I don't want it to happen
again. We can't say we don't understand the issues. It has happened so
many times that we are destined to repeat those problems unless we do
something differently. Clearly, branch at beta, patch development tree,
disable partially implemented features, is a change. I still think not
double patching for the first few weeks will be a win, though.

Remember, if you fix something in current, you have to generate a patch
and apply it to the STABLE tree for _anything_ you fix in stable. And
almost any changes in STABLE have to be applied in current. Also, bug
reports will have to indentify stable or development tree.

I know FreeBSD does it this way, but I don't hold them up as a model of
good organization.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Monday 10 June 2002 02:46 pm, Marc G. Fournier wrote:
> Based on everything I've heard/seen in this thread, we seem to be looking
> at:

> 1. Branch on Sept 1st, regardless of almost anything

> 2. Once Branch created, any *partially implemented* features will get
> rip'd out of the -STABLE branch and only fixes to the existing, fully
> implement features will go in

> 3. Beta1 released once developers comfortable with the state of the code

> Now, *if*, the week before the Branch, someone submits a bit patch that in
> *anyway* concerns someone to apply, we can hold it off for a week and put
> it into the -DEV branch so that its not shelved for a couple of months,
> and possibly going out of date ... but that would be a judgement call at
> the time, nothing set in stone ...

> The only thing we are really "setting in stone" here is when we are
> branching/freezing the code for release ...

This seems to me to be reasonable. My only question would be 'why haven't we
always done it this way' but that isn't terribly productive. I actually know
the answer to my question, in fact, but that's not relevant to the future.

Many large projects do this, in some form or another. FreeBSD, Debian, even
the Linux kernel all follow this basic form.

Historically we've concentrated our development efforts during beta to 'fixing
beta problems only' -- but that model produces these extraordinarily long
cycles, IMHO. In the meantime people are literally chomping at the bit to do
a new feature -- to the point that one developer got rather upset that his
patch wasn't being looked at and 'stomped off' in a huff. All because we
were in beta-only mode.

However, I do think at that point we need to look at what the patch manager
(historically Bruce) can deal with realistically. Is it a job for two patch
managers, one for the STABLE and one for the DEV? Only Bruce can answer
whether he can realistically handle it (I personally have confidence he can).
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> Historically we've concentrated our development efforts during beta to
> 'fixing beta problems only' -- but that model produces these
> extraordinarily long cycles, IMHO. In the meantime people are
> literally chomping at the bit to do a new feature -- to the point that
> one developer got rather upset that his patch wasn't being looked at
> and 'stomped off' in a huff. All because we were in beta-only mode.

There is a downside to changing away from that approach. Bruce
mentioned it but didn't really give it the prominence I think it
deserves: beta mode encourages developers to work on testing, debugging,
and oh yes documenting. Without that forced "non development" time,
some folks will just never get around to the mop-up stages of their
projects; they'll be off in new-feature-land all the time. I won't name
names, but there are more than a couple around here ;-)

I think our develop mode/beta mode pattern has done a great deal to
contribute to the stability of our releases. If we go over to the same
approach that everyone else uses, you can bet your last dollar that our
releases will be no better than everyone else's. How many people here
run dot-zero releases of the Linux kernel, or gcc? Anyone find them
trustworthy? Anyone really eager to have to maintain old releases for
several years, because no sane DBA will touch the latest release?

I'm not trying to sound like Cassandra, but we've done very very well
with only limited resources over the past several years. We should not
be too eager to mess with a proven-successful approach.

regards, tom lane

On Monday 10 June 2002 04:11 pm, Tom Lane wrote:
> Lamar Owen <lamar(dot)owen(at)wgcr(dot)org> writes:
> > Historically we've concentrated our development efforts during beta to
> > 'fixing beta problems only'

> There is a downside to changing away from that approach.

There are downsides to every approach. The question is 'Which set of
downsides are we most comfortable with?'

> Bruce
> mentioned it but didn't really give it the prominence I think it
> deserves: beta mode encourages developers to work on testing, debugging,
> and oh yes documenting. Without that forced "non development" time,
> some folks will just never get around to the mop-up stages of their
> projects; they'll be off in new-feature-land all the time. I won't name
> names, but there are more than a couple around here ;-)

Well, this is the one downside the Marc's proposal. It boils down to
self-discipline, though. Unfortunately not everyone is as disciplined as you
seem to be in the area, Tom. I certainly cannot claim a great deal of
self-discipline. BTW, that is meant as a compliment to you, Tom.

> I think our develop mode/beta mode pattern has done a great deal to
> contribute to the stability of our releases. If we go over to the same
> approach that everyone else uses, you can bet your last dollar that our
> releases will be no better than everyone else's.

I'll have to agree here -- but I also must remind people that our 'dot zero'
releases are typically solid, but our 'dot one' releases have not been so
solid. So I wouldn't be too confident in our existing model.

And I'm not so sure the model is the producer of our sterling record
heretofore. I'm more of the mindset that the quality and discipline of the
developers is the real reason.

> How many people here
> run dot-zero releases of the Linux kernel, or gcc? Anyone find them
> trustworthy? Anyone really eager to have to maintain old releases for
> several years, because no sane DBA will touch the latest release?

We already have some of that problem due to the difficulty in upgrading.
People wait and see if the features warrant the downtime and pain of
upgrading. Meantime they live with security holes and bugs in our own
unmaintained older releases. And dump and restore upgrades are not painless.
I will admit that I've not used pg_upgrade in some time -- I understand
moving from 7.1 to 7.2 is much less painful using pg_upgrade. However,
pg_upgrade was released in contrib as being a 'handle with great care'
utility that no sane DBA is going to touch.... Catch 22.

So, I don't necessarily agree that we should hold up our development model as
the panacea, and I'm not thoroughly convinced that the quality of our
releases is related directly to the development model. I believe it is
directly related to the caliber of the developers.

That said, good developers can produce good quality regardless of the model
used if they will discipline themselves accordingly.

> I'm not trying to sound like Cassandra, but we've done very very well
> with only limited resources over the past several years. We should not
> be too eager to mess with a proven-successful approach.

Interesting reference....

Why not try this one cycle and see what happens? No one is going to force
anyone else to develop new features when they want to fix bugs.
--
Lamar Owen
WGCR Internet Radio
1 Peter 4:11

On Mon, 10 Jun 2002, Tom Lane wrote:

> There is a downside to changing away from that approach. Bruce
> mentioned it but didn't really give it the prominence I think it
> deserves: beta mode encourages developers to work on testing, debugging,
> and oh yes documenting. Without that forced "non development" time,
> some folks will just never get around to the mop-up stages of their
> projects; they'll be off in new-feature-land all the time. I won't name
> names, but there are more than a couple around here ;-)

Well, in alot of ways we have control over this ... we have a very limited
number of committers ... start requiring that any patches that come
through, instead of "just being applied and worry about documentation
later", require the documentation to be included at the same time ...
would definitely save alot of headaches down the road chasing down that
documentation ... I think we've actually done thta a few times in the
past, where we've held up a patch waiting for the documentation, but its
never been a real requirement, but I don't think its an unreasonable one
...

> I think our develop mode/beta mode pattern has done a great deal to
> contribute to the stability of our releases. If we go over to the same
> approach that everyone else uses, you can bet your last dollar that our
> releases will be no better than everyone else's. How many people here
> run dot-zero releases of the Linux kernel, or gcc? Anyone find them
> trustworthy? Anyone really eager to have to maintain old releases for
> several years, because no sane DBA will touch the latest release?

Again, we do have alot of control over this ... the only ppl that we
*really* have to worry about "not mopping up" their code are those with
committers access ... everyone else has to go through us, which means that
we can always "stale" a patch from a developer due to requirements for bug
fixes ...

... but, quite honestly, have we ever truly had a problem with this even
during development period? How many *large* OSS projects out there have?
My experience(s) with FreeBSD, for an example, are that most developers
take pride in their code ... if someone reports a bug, and its
recreateable, its generally fixed quite quickly ... its only the "hard to
recreate" bugs that take a long time to fix ... wasn't that just the case
with us with the sequences bug? You yourself, if I recall, admitted that
its always been there, but it obviously wasn't the easiest to
re-create/trigger, else we would have had more ppl yelling about it ...
once someone was able to narrow down the problem and how to re-create it
consistently, it was fixed ...

We've never really run "a tight ship" as far as code has gone ... Bruce
has been known to helter-skelter apply patches, even a couple that I
recall so obviously shouldn't have been that we beat him for it, but that
has never prevented us (or even slowed us down) from having *solid*
releases ... everyone that I've meet so far working on this project, IMHO,
have been *passionate* about what they do ... and, in some way or another,
*rely* on it being rock-solid ...

On Mon, 10 Jun 2002, Bruce Momjian wrote:

> > 2. Once Branch created, any *partially implemented* features will get
> > rip'd out of the -STABLE branch and only fixes to the existing, fully
> > implement features will go in
>
> Now, that is an interesting idea.

Ya, I thought it was when you -and- Tom proposed it :)

I quote from a message on June 8th:

=======================
Tom Lane wrote:
> Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > So, I we should:
> > Warn people in July that beta is September 1 and all features
> > have to be complete by then, or they get ripped out.
>
> I thought that was more or less the same thing I was proposing...
========================

Marc G. Fournier wrote:
> On Mon, 10 Jun 2002, Bruce Momjian wrote:
>
> > > 2. Once Branch created, any *partially implemented* features will get
> > > rip'd out of the -STABLE branch and only fixes to the existing, fully
> > > implement features will go in
> >
> > Now, that is an interesting idea.
>
> Ya, I thought it was when you -and- Tom proposed it :)
>
> I quote from a message on June 8th:
>
> =======================
> Tom Lane wrote:
> > Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> > > So, I we should:
> > > Warn people in July that beta is September 1 and all features
> > > have to be complete by then, or they get ripped out.
> >
> > I thought that was more or less the same thing I was proposing...
> ========================

What I thought was interesting was having the CURRENT branch keep the
feature so the guy could continue development, even if we disable in
STABLE.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

Lamar Owen wrote:
> On Monday 10 June 2002 04:11 pm, Tom Lane wrote:
> > I think our develop mode/beta mode pattern has done a great deal to
> > contribute to the stability of our releases. If we go over to the same
> > approach that everyone else uses, you can bet your last dollar that our
> > releases will be no better than everyone else's.
>
> I'll have to agree here -- but I also must remind people that our 'dot zero'
> releases are typically solid, but our 'dot one' releases have not been so
> solid. So I wouldn't be too confident in our existing model.

If that's a pattern, then we should discourage people from
using odd dot-releases.

My opinion? With each release we ship improvements and new
functionality people have long waited for. Think about
vacuum, toast, referential integrity. People need those
things and have great confidence in our releases. The
willingness to upgrade their production systems to dot zero
releases is the biggest compliment users can make.

Everything that endangers that quality is bad(tm). Our
develop/beta mode pattern keeps people from diving into the
next bigger thing, distracting them from the current beta or
release candidate. I don't think that would do us a really
good job.

Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #