Alpha releases: How to tag

As we are approaching the first alpha release, let's think about how to tag
and label it and how to schedule those two operations with respect to one
another.

The typical process for, say, a beta release is

- apply version number change to source tree
- commit
- tag
(repeat for next beta)

The problem, arguably, with this is that the source tree would then read
8.5alpha1 until alpha2, instead of the 8.5devel that we are used to. The fix
could be:

- apply version number change to source tree
- commit
- tag
- revert version number change in source tree
- commit
(repeat for next alpha)

or more sophisticatedly

- branch
- apply version number change to source tree
- commit
- tag

or alternatively no tag at all, just apply version number and build tarball.

Comments?

* Peter Eisentraut (peter_e(at)gmx(dot)net) wrote:
> - branch
> - apply version number change to source tree
> - commit
> - tag

If this wasn't CVS, this would certainly be the way to go.

> or alternatively no tag at all, just apply version number and build tarball.

As this *is* CVS, I'm thinking we should probably go with this approach.

Just my 2c.

Thanks,

Stephen

Peter Eisentraut wrote:
> As we are approaching the first alpha release, let's think about how to tag
> and label it and how to schedule those two operations with respect to one
> another.
>
> The typical process for, say, a beta release is
>
> - apply version number change to source tree
> - commit
> - tag
> (repeat for next beta)
>
> The problem, arguably, with this is that the source tree would then read
> 8.5alpha1 until alpha2, instead of the 8.5devel that we are used to. The fix
> could be:
>
> - apply version number change to source tree
> - commit
> - tag
> - revert version number change in source tree
> - commit
> (repeat for next alpha)
>
> or more sophisticatedly
>
> - branch
> - apply version number change to source tree
> - commit
> - tag
>
> or alternatively no tag at all, just apply version number and build tarball.
>
> Comments?
>
>

Does it need a version number change? Maybe just a tag (no branch) is
all that is required.

cheers

andrew

On Aug 3, 2009, at 7:57 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:

> * Peter Eisentraut (peter_e(at)gmx(dot)net) wrote:
>> - branch
>> - apply version number change to source tree
>> - commit
>> - tag
>
> If this wasn't CVS, this would certainly be the way to go.
>
>> or alternatively no tag at all, just apply version number and build
>> tarball.
>
> As this *is* CVS, I'm thinking we should probably go with this
> approach.

Sounds right to me.

...Robert

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Does it need a version number change? Maybe just a tag (no branch) is
> all that is required.

I think that we do want the alpha releases to identify themselves as
such. And we want a marker in CVS as to what state the alpha release
corresponds to. Peter's label-and-undo approach seems like a kluge;
and it doesn't scale to consider the possibility that we might
want to re-release an alpha after fixing some particularly evil bug.
A tag without a branch won't handle that either.

I feel that making a branch is the way to go. If we try to get away
with a shortcut, we'll probably regret it.

regards, tom lane

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>> Does it need a version number change? Maybe just a tag (no branch) is
>> all that is required.
>>
>
> I think that we do want the alpha releases to identify themselves as
> such. And we want a marker in CVS as to what state the alpha release
> corresponds to. Peter's label-and-undo approach seems like a kluge;
> and it doesn't scale to consider the possibility that we might
> want to re-release an alpha after fixing some particularly evil bug.
> A tag without a branch won't handle that either.
>
> I feel that making a branch is the way to go. If we try to get away
> with a shortcut, we'll probably regret it.
>
>
>

Yes, if that's what we want then definitely branch. I guess the branch
will (almost) only ever have exactly one commit on it.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Tom Lane wrote:
>> ... it doesn't scale to consider the possibility that we might
>> want to re-release an alpha after fixing some particularly evil bug.

> Yes, if that's what we want then definitely branch. I guess the branch
> will (almost) only ever have exactly one commit on it.

Yeah, I'd expect so; but it seems like a good idea to leave room for the
possibility of more than one commit. I was first thinking that we'd
just release a new alpha from CVS HEAD if the evil-bug situation comes
up. However, if we've already done a catversion bump or some other
incompatible change in HEAD, that wouldn't be very practical.

regards, tom lane

On Monday 03 August 2009 17:44:32 Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > Does it need a version number change? Maybe just a tag (no branch) is
> > all that is required.
>
> I think that we do want the alpha releases to identify themselves as
> such. And we want a marker in CVS as to what state the alpha release
> corresponds to. Peter's label-and-undo approach seems like a kluge;
> and it doesn't scale to consider the possibility that we might
> want to re-release an alpha after fixing some particularly evil bug.
> A tag without a branch won't handle that either.
>
> I feel that making a branch is the way to go. If we try to get away
> with a shortcut, we'll probably regret it.

Another more lightweight alternative is to tag and then change the version
number and build the tarball, without another commit. This assumes that the
version number stamping is automated and reproducible, so you don't have to
record it in history.

On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > Does it need a version number change? Maybe just a tag (no branch)
> > is all that is required.
>
> I think that we do want the alpha releases to identify themselves as
> such. And we want a marker in CVS as to what state the alpha
> release corresponds to. Peter's label-and-undo approach seems like
> a kluge;

Right.

> and it doesn't scale to consider the possibility that we might want
> to re-release an alpha after fixing some particularly evil bug. A
> tag without a branch won't handle that either.

Is this a use case? I truly hope nobody will try using a beta, let
alone an alpha, in production. Do we need to provide for such a
possibility? I don't recall that we've ever back-patched a beta, or
even a release candidate.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david(at)fetter(dot)org> writes:
> On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote:
>> and it doesn't scale to consider the possibility that we might want
>> to re-release an alpha after fixing some particularly evil bug. A
>> tag without a branch won't handle that either.

> Is this a use case? I truly hope nobody will try using a beta, let
> alone an alpha, in production. Do we need to provide for such a
> possibility? I don't recall that we've ever back-patched a beta, or
> even a release candidate.

I don't really know if it's a use-case or not; I just have a feeling
that if we use a release procedure that guarantees we can't do it,
we'll live to regret that.

The bug-fixing situation for betas and RCs is a bit different because
it's expected that there will be a compatible update available shortly.
So you can usually assume that updating to the next beta/RC/release will
fix whatever problems got found. Alphas are going to be out there on
their own with absolutely no expectation that the next alpha is
catversion-compatible. And I doubt we'd bother generating pg_migrator
builds that work for pairs of alpha releases.

I agree with you that it'd be insane to run anything mission-critical on
an alpha build; but I could see someone having loaded up quite a lot of
test data on one. (If we aren't hoping for some pretty serious testing
of these releases, I am not sure what is the point of doing them at all.)
So it seems to me that having the ability to fix show-stopper bugs
without forcing a migration to a later alpha would be a good thing.
Maybe we'll never need it, or maybe we will.

regards, tom lane

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On Monday 03 August 2009 17:44:32 Tom Lane wrote:
>> I feel that making a branch is the way to go. If we try to get away
>> with a shortcut, we'll probably regret it.

> Another more lightweight alternative is to tag and then change the version
> number and build the tarball, without another commit. This assumes that the
> version number stamping is automated and reproducible, so you don't have to
> record it in history.

The stamping is reproducible enough, I think, but don't we have the
tarball build process set up so that it pulls the tarball directly
from a particular CVS tag?

Anyway, the reason I'm for a branch is not that. It's so we have the
option to issue an 8.5alpha1.1 update if we need to.

regards, tom lane

On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
> David Fetter <david(at)fetter(dot)org> writes:
> > On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote:
> >> and it doesn't scale to consider the possibility that we might want
> >> to re-release an alpha after fixing some particularly evil bug. A
> >> tag without a branch won't handle that either.
>
> > Is this a use case? I truly hope nobody will try using a beta, let
> > alone an alpha, in production. Do we need to provide for such a
> > possibility? I don't recall that we've ever back-patched a beta, or
> > even a release candidate.
>
> I don't really know if it's a use-case or not; I just have a feeling
> that if we use a release procedure that guarantees we can't do it,
> we'll live to regret that.

I can see being cautious. I'm just questioning the value of this
precaution in particular.

> The bug-fixing situation for betas and RCs is a bit different
> because it's expected that there will be a compatible update
> available shortly. So you can usually assume that updating to the
> next beta/RC/release will fix whatever problems got found. Alphas
> are going to be out there on their own with absolutely no
> expectation that the next alpha is catversion-compatible.

We've bumped catversion in beta and even RC, if I recall right.

> And I doubt we'd bother generating pg_migrator builds that work for
> pairs of alpha releases.

That's an interesting idea. Shouldn't pg_migrator be mandated to work
for *any* catversion bump?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david(at)fetter(dot)org> writes:
> On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
>> And I doubt we'd bother generating pg_migrator builds that work for
>> pairs of alpha releases.

> That's an interesting idea. Shouldn't pg_migrator be mandated to work
> for *any* catversion bump?

Oh, are you volunteering to make that happen? That code is going to be
ugly enough just dealing with pairs of major releases ...

regards, tom lane

On Mon, Aug 3, 2009 at 17:32, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> David Fetter <david(at)fetter(dot)org> writes:
>> On Mon, Aug 03, 2009 at 10:44:32AM -0400, Tom Lane wrote:
>>> and it doesn't scale to consider the possibility that we might want
>>> to re-release an alpha after fixing some particularly evil bug.  A
>>> tag without a branch won't handle that either.
>
>> Is this a use case?  I truly hope nobody will try using a beta, let
>> alone an alpha, in production.  Do we need to provide for such a
>> possibility?  I don't recall that we've ever back-patched a beta, or
>> even a release candidate.
>
> I don't really know if it's a use-case or not; I just have a feeling
> that if we use a release procedure that guarantees we can't do it,
> we'll live to regret that.

Agreed.

> The bug-fixing situation for betas and RCs is a bit different because
> it's expected that there will be a compatible update available shortly.
> So you can usually assume that updating to the next beta/RC/release will
> fix whatever problems got found.  Alphas are going to be out there on
> their own with absolutely no expectation that the next alpha is
> catversion-compatible.  And I doubt we'd bother generating pg_migrator
> builds that work for pairs of alpha releases.
>
> I agree with you that it'd be insane to run anything mission-critical on
> an alpha build; but I could see someone having loaded up quite a lot of
> test data on one.  (If we aren't hoping for some pretty serious testing
> of these releases, I am not sure what is the point of doing them at all.)
> So it seems to me that having the ability to fix show-stopper bugs
> without forcing a migration to a later alpha would be a good thing.
> Maybe we'll never need it, or maybe we will.

I think the alpha->alpha migration would actually be very useful to
these testers. That'll get them onto the new alpha. I think that's
actually a lot more important than alpha->release or release->next
alpha.

I haven't actually looked into pg_migrator enough to know how likely
it is that it'll "just work" going alpha->alpha when there have only
been "normal" changes? How invasive are the changes that actually
require pg_migrator to be touched at all?

--
Magnus Hagander
Self: http://www.hagander.net/
Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> I haven't actually looked into pg_migrator enough to know how likely
> it is that it'll "just work" going alpha->alpha when there have only
> been "normal" changes? How invasive are the changes that actually
> require pg_migrator to be touched at all?

To my mind there are three categories of stuff that pg_migrator has to
do:

* catalog changes (handled by pg_dump/reload)
* on-disk data layout changes (not handled yet anyway)
* random weird unclassifiable stuff

It's the third category that is the problem. An example from the 8.3 to
8.4 migration is the need to specially handle sequences because they're
not compatible on-disk. Any pair of releases you might pick is going to
have its own little quirks like that. Our intention (at least as I see
it) is to minimize pg_migrator's complexity by decreeing that any given
release of pg_migrator deals with exactly one pair of PG releases.
I don't think that works if alpha->alpha has to be supported --- in
particular, 8.5alpha1 to 8.5alpha2 might be quite different from what
will be needed later for 8.4 to 8.5, so you would end up with multiple
live branches of pg_migrator, or else a lot of spaghetti code trying to
track which actions to take for a given case.

The other little problem is who's going to do this work and when.
The alpha-release idea was sold to us on the basis of being a small
amount of incremental work: just tag an alpha release right after
CommitFest and kick it out there. If we're waiting around for someone
to produce and test a compatible pg_migrator, that's not going to be
how it works.

regards, tom lane

On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote:
> David Fetter <david(at)fetter(dot)org> writes:
> > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
> >> And I doubt we'd bother generating pg_migrator builds that work
> >> for pairs of alpha releases.
>
> > That's an interesting idea. Shouldn't pg_migrator be mandated to
> > work for *any* catversion bump?
>
> Oh, are you volunteering to make that happen? That code is going to
> be ugly enough just dealing with pairs of major releases ...

With all due respect, if pg_migrator does not function in such a way
as to have data-driven composable changes, it's going to bang us up in
the long haul much worse than not branching alphas ever could.

We require that people supply docs with their changes, and it is
totally unreasonable to let them send in catalog changes which do not
include need migration changes. That's how it works in every other
RDBMS outfit that has changes on disk, and we do not need to be the
exception.

If pg_migrator doesn't have this ability, we need to refactor it until
it does, or go with something that can.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Monday 03 August 2009 21:07:00 David Fetter wrote:
> We require that people supply docs with their changes, and it is
> totally unreasonable to let them send in catalog changes which do not
> include need migration changes. That's how it works in every other
> RDBMS outfit that has changes on disk, and we do not need to be the
> exception.

Well, blocker number one for that is that pg_migrator is not even in the
PostgreSQL CVS repository, but is more like an endorsed third-party product.

On Mon, Aug 3, 2009 at 2:07 PM, David Fetter<david(at)fetter(dot)org> wrote:
> On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote:
>> David Fetter <david(at)fetter(dot)org> writes:
>> > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
>> >> And I doubt we'd bother generating pg_migrator builds that work
>> >> for pairs of alpha releases.
>>
>> > That's an interesting idea.  Shouldn't pg_migrator be mandated to
>> > work for *any* catversion bump?
>>
>> Oh, are you volunteering to make that happen?  That code is going to
>> be ugly enough just dealing with pairs of major releases ...
>
> With all due respect, if pg_migrator does not function in such a way
> as to have data-driven composable changes, it's going to bang us up in
> the long haul much worse than not branching alphas ever could.
>
> We require that people supply docs with their changes, and it is
> totally unreasonable to let them send in catalog changes which do not
> include need migration changes.  That's how it works in every other
> RDBMS outfit that has changes on disk, and we do not need to be the
> exception.
>
> If pg_migrator doesn't have this ability, we need to refactor it until
> it does, or go with something that can.

I don't disagree with this, but we don't have anyone to do the work
atm, even if it were exactly clear what to do, which it isn't. What
we could do, though, is try to figure out whether it will work between
8.4.0 and 8.5alpha1.

...Robert

On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote:
> On Monday 03 August 2009 21:07:00 David Fetter wrote:
> > We require that people supply docs with their changes, and it is
> > totally unreasonable to let them send in catalog changes which do
> > not include need migration changes. That's how it works in every
> > other RDBMS outfit that has changes on disk, and we do not need to
> > be the exception.
>
> Well, blocker number one for that is that pg_migrator is not even in
> the PostgreSQL CVS repository, but is more like an endorsed
> third-party product.

I'm not entirely sure that pg_migrator should be tied to releases of
PostgreSQL, given what it does. Or did you mean that it's not been
given the same scrutiny that the PostgreSQL code base has?

Thoughts?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Monday 03 August 2009 22:52:55 David Fetter wrote:
> On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote:
> > On Monday 03 August 2009 21:07:00 David Fetter wrote:
> > > We require that people supply docs with their changes, and it is
> > > totally unreasonable to let them send in catalog changes which do
> > > not include need migration changes. That's how it works in every
> > > other RDBMS outfit that has changes on disk, and we do not need to
> > > be the exception.
> >
> > Well, blocker number one for that is that pg_migrator is not even in
> > the PostgreSQL CVS repository, but is more like an endorsed
> > third-party product.
>
> I'm not entirely sure that pg_migrator should be tied to releases of
> PostgreSQL, given what it does. Or did you mean that it's not been
> given the same scrutiny that the PostgreSQL code base has?

Well, how to you expect to mandate that all patches come with upgrade support
if the PostgreSQL software proper does not contain any upgrade support in the
first place, because pg_migrator is maintained elsewhere.

On Tue, Aug 04, 2009 at 05:19:16PM +0300, Peter Eisentraut wrote:
> On Monday 03 August 2009 22:52:55 David Fetter wrote:
> > On Mon, Aug 03, 2009 at 09:22:52PM +0300, Peter Eisentraut wrote:
> > > On Monday 03 August 2009 21:07:00 David Fetter wrote:
> > > > We require that people supply docs with their changes, and it
> > > > is totally unreasonable to let them send in catalog changes
> > > > which do not include need migration changes. That's how it
> > > > works in every other RDBMS outfit that has changes on disk,
> > > > and we do not need to be the exception.
> > >
> > > Well, blocker number one for that is that pg_migrator is not
> > > even in the PostgreSQL CVS repository, but is more like an
> > > endorsed third-party product.
> >
> > I'm not entirely sure that pg_migrator should be tied to releases
> > of PostgreSQL, given what it does. Or did you mean that it's not
> > been given the same scrutiny that the PostgreSQL code base has?
>
> Well, how to you expect to mandate that all patches come with
> upgrade support if the PostgreSQL software proper does not contain
> any upgrade support in the first place, because pg_migrator is
> maintained elsewhere.

If I'm understanding you correctly, you're saying that pg_migrator (or
whatever actually does this) needs to be an official PostgreSQL
project in order for us to be able to require that people use it. For
what it's worth, I agree.

Is it strictly necessary that its release cycles match exactly those
of the database engine, or would it be OK for it to release as needed,
not triggering a major PostgreSQL release?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:

> Is it strictly necessary that its release cycles match exactly those
> of the database engine, or would it be OK for it to release as needed,
> not triggering a major PostgreSQL release?

Well, pg_migrator already released an 8.4.1 ...

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Wednesday 05 August 2009 06:00:19 David Fetter wrote:
> If I'm understanding you correctly, you're saying that pg_migrator (or
> whatever actually does this) needs to be an official PostgreSQL
> project in order for us to be able to require that people use it. For
> what it's worth, I agree.
>
> Is it strictly necessary that its release cycles match exactly those
> of the database engine, or would it be OK for it to release as needed,
> not triggering a major PostgreSQL release?

Right now, anything is possible. It mainly needs people to make something
happen. The current maintainers of pg_migrator are still focused on making
the 8.3 -> 8.4 path working robustly. And after that, they will likely take a
long rest. If people want pg_migrator to stay current with 8.5devel, they
need to take it upon themselves to create repeatable tests and code up the
necessary changes. And then later if that turns out to be a viable
undertaking, we can consider whether we merge pg_migrator and make updating it
a requirement for any patch.

Peter Eisentraut wrote:
> On Monday 03 August 2009 21:07:00 David Fetter wrote:
> > We require that people supply docs with their changes, and it is
> > totally unreasonable to let them send in catalog changes which do not
> > include need migration changes. That's how it works in every other
> > RDBMS outfit that has changes on disk, and we do not need to be the
> > exception.
>
> Well, blocker number one for that is that pg_migrator is not even in the
> PostgreSQL CVS repository, but is more like an endorsed third-party product.

I wouldn't say pg_migrator is "endorsed". It is on pgfoundry and was
mentioned in the press release, but it isn't mentioned in our
documentation about upgrading, it wasn't mentioned in the release notes,
and it isn't mentioned on our web site, except as a news item.

I believe this is because of concerns about pg_migrator's "experimental"
nature.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, Aug 7, 2009 at 11:35 AM, Bruce Momjian<bruce(at)momjian(dot)us> wrote:
> Peter Eisentraut wrote:
>> On Monday 03 August 2009 21:07:00 David Fetter wrote:
>> > We require that people supply docs with their changes, and it is
>> > totally unreasonable to let them send in catalog changes which do not
>> > include need migration changes.  That's how it works in every other
>> > RDBMS outfit that has changes on disk, and we do not need to be the
>> > exception.
>>
>> Well, blocker number one for that is that pg_migrator is not even in the
>> PostgreSQL CVS repository, but is more like an endorsed third-party product.
>
> I wouldn't say pg_migrator is "endorsed".  It is on pgfoundry and was
> mentioned in the press release, but it isn't mentioned in our
> documentation about upgrading, it wasn't mentioned in the release notes,
> and it isn't mentioned on our web site, except as a news item.
>
> I believe this is because of concerns about pg_migrator's "experimental"
> nature.

I think so. And also because it has a fair number of documented
restrictions. Hopefully we'll be able to remove some of those in a
future release.

...Robert

Robert Haas wrote:
> On Mon, Aug 3, 2009 at 2:07 PM, David Fetter<david(at)fetter(dot)org> wrote:
> > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote:
> >> David Fetter <david(at)fetter(dot)org> writes:
> >> > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
> >> >> And I doubt we'd bother generating pg_migrator builds that work
> >> >> for pairs of alpha releases.
> >>
> >> > That's an interesting idea. ?Shouldn't pg_migrator be mandated to
> >> > work for *any* catversion bump?
> >>
> >> Oh, are you volunteering to make that happen? ?That code is going to
> >> be ugly enough just dealing with pairs of major releases ...
> >
> > With all due respect, if pg_migrator does not function in such a way
> > as to have data-driven composable changes, it's going to bang us up in
> > the long haul much worse than not branching alphas ever could.
> >
> > We require that people supply docs with their changes, and it is
> > totally unreasonable to let them send in catalog changes which do not
> > include need migration changes. ?That's how it works in every other
> > RDBMS outfit that has changes on disk, and we do not need to be the
> > exception.
> >
> > If pg_migrator doesn't have this ability, we need to refactor it until
> > it does, or go with something that can.
>
> I don't disagree with this, but we don't have anyone to do the work
> atm, even if it were exactly clear what to do, which it isn't. What
> we could do, though, is try to figure out whether it will work between
> 8.4.0 and 8.5alpha1.

I just tested this and it works fine. In fact it even does 8.3 to
8.5devel, with no pg_migrator changes needed.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Alvaro Herrera wrote:
> David Fetter wrote:
>
> > Is it strictly necessary that its release cycles match exactly those
> > of the database engine, or would it be OK for it to release as needed,
> > not triggering a major PostgreSQL release?
>
> Well, pg_migrator already released an 8.4.1 ...

Well, actually at 8.4.4 right now, but who's counting. It is clear that
pg_migrator has to be separate so it can rapidly address deficiencies
without being tied to database server releases. We found a problem with
arrays/composites/enums two days ago and had it fixed/detected in 24
hours.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Magnus Hagander wrote:
> > The bug-fixing situation for betas and RCs is a bit different because
> > it's expected that there will be a compatible update available shortly.
> > So you can usually assume that updating to the next beta/RC/release will
> > fix whatever problems got found. ?Alphas are going to be out there on
> > their own with absolutely no expectation that the next alpha is
> > catversion-compatible. ?And I doubt we'd bother generating pg_migrator
> > builds that work for pairs of alpha releases.
> >
> > I agree with you that it'd be insane to run anything mission-critical on
> > an alpha build; but I could see someone having loaded up quite a lot of
> > test data on one. ?(If we aren't hoping for some pretty serious testing
> > of these releases, I am not sure what is the point of doing them at all.)
> > So it seems to me that having the ability to fix show-stopper bugs
> > without forcing a migration to a later alpha would be a good thing.
> > Maybe we'll never need it, or maybe we will.
>
> I think the alpha->alpha migration would actually be very useful to
> these testers. That'll get them onto the new alpha. I think that's
> actually a lot more important than alpha->release or release->next
> alpha.
>
> I haven't actually looked into pg_migrator enough to know how likely
> it is that it'll "just work" going alpha->alpha when there have only
> been "normal" changes? How invasive are the changes that actually
> require pg_migrator to be touched at all?

pg_migrator handles 8.5devel just fine now with no changes. What will
happen is that people will try pg_migrator in 8.5devel and if it fails
we will hear about it and and adjust pg_migrator (or the backend) to get
it working. This will give us early warning about backend changes and
hopefully make pg_migrator more useful when 8.5 final is released.

What I have found is that pg_migrator has a useful toolkit for detecting
and correcting database incompatibilities. It wasn't easy to get
working but now that it is complete adding stuff is actually easy ---
see the enum/array/composite fix that was done yesterday within 24 hours
of the report.

Now, I am not guaranteeing that pg_migrator is always going to work for
8.5devel, but while it does let's use it, and when it fails, we can just
update the README to say it doesn't work for 8.5devel, or fix things.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > I haven't actually looked into pg_migrator enough to know how likely
> > it is that it'll "just work" going alpha->alpha when there have only
> > been "normal" changes? How invasive are the changes that actually
> > require pg_migrator to be touched at all?
>
> To my mind there are three categories of stuff that pg_migrator has to
> do:
>
> * catalog changes (handled by pg_dump/reload)
> * on-disk data layout changes (not handled yet anyway)
> * random weird unclassifiable stuff
>
> It's the third category that is the problem. An example from the 8.3 to
> 8.4 migration is the need to specially handle sequences because they're
> not compatible on-disk. Any pair of releases you might pick is going to
> have its own little quirks like that. Our intention (at least as I see
> it) is to minimize pg_migrator's complexity by decreeing that any given
> release of pg_migrator deals with exactly one pair of PG releases.
> I don't think that works if alpha->alpha has to be supported --- in
> particular, 8.5alpha1 to 8.5alpha2 might be quite different from what
> will be needed later for 8.4 to 8.5, so you would end up with multiple
> live branches of pg_migrator, or else a lot of spaghetti code trying to
> track which actions to take for a given case.
>
> The other little problem is who's going to do this work and when.
> The alpha-release idea was sold to us on the basis of being a small
> amount of incremental work: just tag an alpha release right after
> CommitFest and kick it out there. If we're waiting around for someone
> to produce and test a compatible pg_migrator, that's not going to be
> how it works.

Because the 8.5 is alpha anyway, and because pg_migrator works with
current CVS, let's just say it works and wait for someone to report a
problem.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

David Fetter wrote:
> On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote:
> > David Fetter <david(at)fetter(dot)org> writes:
> > > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
> > >> And I doubt we'd bother generating pg_migrator builds that work
> > >> for pairs of alpha releases.
> >
> > > That's an interesting idea. Shouldn't pg_migrator be mandated to
> > > work for *any* catversion bump?
> >
> > Oh, are you volunteering to make that happen? That code is going to
> > be ugly enough just dealing with pairs of major releases ...
>
> With all due respect, if pg_migrator does not function in such a way
> as to have data-driven composable changes, it's going to bang us up in
> the long haul much worse than not branching alphas ever could.
>
> We require that people supply docs with their changes, and it is
> totally unreasonable to let them send in catalog changes which do not
> include need migration changes. That's how it works in every other
> RDBMS outfit that has changes on disk, and we do not need to be the
> exception.
>
> If pg_migrator doesn't have this ability, we need to refactor it until
> it does, or go with something that can.

Odds are that the patch submitters will not understand enough to know
how to modify pg_migrator, but just knowing something broke is usually
enough for the hackers group to find a fix.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, Aug 07, 2009 at 04:07:08PM -0400, Bruce Momjian wrote:
> David Fetter wrote:
> > On Mon, Aug 03, 2009 at 12:19:40PM -0400, Tom Lane wrote:
> > > David Fetter <david(at)fetter(dot)org> writes:
> > > > On Mon, Aug 03, 2009 at 11:32:48AM -0400, Tom Lane wrote:
> > > >> And I doubt we'd bother generating pg_migrator builds that
> > > >> work for pairs of alpha releases.
> > >
> > > > That's an interesting idea. Shouldn't pg_migrator be mandated
> > > > to work for *any* catversion bump?
> > >
> > > Oh, are you volunteering to make that happen? That code is
> > > going to be ugly enough just dealing with pairs of major
> > > releases ...
> >
> > With all due respect, if pg_migrator does not function in such a
> > way as to have data-driven composable changes, it's going to bang
> > us up in the long haul much worse than not branching alphas ever
> > could.
> >
> > We require that people supply docs with their changes, and it is
> > totally unreasonable to let them send in catalog changes which do
> > not include need migration changes. That's how it works in every
> > other RDBMS outfit that has changes on disk, and we do not need to
> > be the exception.
> >
> > If pg_migrator doesn't have this ability, we need to refactor it
> > until it does, or go with something that can.
>
> Odds are that the patch submitters will not understand enough to
> know how to modify pg_migrator, but just knowing something broke is
> usually enough for the hackers group to find a fix.

This is a pretty serious drawback. If we're going to require that
people send migration scripts when they change the on-disk format,
this needs to be easy. The program, whatever it turns out to be,
needs to take composable changes which people would submit along with
their patch, just as they do docs.

How much refactoring of pg_migrator would such a change take? Would
it be worthwhile to do that, or just start a different project?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:

> This is a pretty serious drawback. If we're going to require that
> people send migration scripts when they change the on-disk format,
> this needs to be easy.

But, are we?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera wrote:
> David Fetter wrote:
>
>
>> This is a pretty serious drawback. If we're going to require that
>> people send migration scripts when they change the on-disk format,
>> this needs to be easy.
>>
>
> But, are we?
>
>

I think it's a lot more nebulous than that. At the same time I think the
days when we can blithely change the on-disk format with hardly a
thought for migration are over. IOW, there's agreement things have to
change, but the exact shape of the change is not yet clear (at least to
me) ;-)

cheers

andrew

On Fri, Aug 07, 2009 at 05:32:13PM -0400, Alvaro Herrera wrote:
> David Fetter wrote:
>
> > This is a pretty serious drawback. If we're going to require that
> > people send migration scripts when they change the on-disk format,
> > this needs to be easy.
>
> But, are we?

This is an area where we have been sorely lacking and sloppy for
years. Every other RDBMS development team requires it.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> I think it's a lot more nebulous than that. At the same time I think the
> days when we can blithely change the on-disk format with hardly a
> thought for migration are over. IOW, there's agreement things have to
> change, but the exact shape of the change is not yet clear (at least to
> me) ;-)

Yeah. I think we're going to start paying more than zero attention to
this, but we don't yet have a handle on what the real parameters are.
In particular, it's hard to argue that pg_migrator has yet achieved
more than experimental status, so accepting or rejecting patches on
the grounds of whether they would or would not break pg_migrator might
be a bit premature. And at the other end of the spectrum, nobody except
Zdenek wants to deal with changes as invasive as the ones he's proposed.
So we're still feeling our way here. We do *not* have a framework in
which someone could submit a patch that includes an on-disk migration
aspect, so David's position that we should immediately institute a
hard requirement for such seems a bit ivory-tower. We might as well
just say that the on-disk format is frozen, because that's what the
effect would be.

But having said all that, I'm okay with a go-slow position on on-disk
changes for the next little while. That would give us some breathing
room to see if something like pg_migrator is really workable at all.
We need to find out just how far that approach goes before we can make
many decisions in this area.

regards, tom lane

On Fri, Aug 07, 2009 at 06:02:32PM -0400, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > I think it's a lot more nebulous than that. At the same time I think the
> > days when we can blithely change the on-disk format with hardly a
> > thought for migration are over. IOW, there's agreement things have to
> > change, but the exact shape of the change is not yet clear (at least to
> > me) ;-)
>
> Yeah. I think we're going to start paying more than zero attention to
> this, but we don't yet have a handle on what the real parameters are.
> In particular, it's hard to argue that pg_migrator has yet achieved
> more than experimental status, so accepting or rejecting patches on
> the grounds of whether they would or would not break pg_migrator might
> be a bit premature. And at the other end of the spectrum, nobody except
> Zdenek wants to deal with changes as invasive as the ones he's proposed.
> So we're still feeling our way here. We do *not* have a framework in
> which someone could submit a patch that includes an on-disk migration
> aspect, so David's position that we should immediately institute a
> hard requirement for such seems a bit ivory-tower.

I am not suggesting that this change be immediate, and it's not ivory
tower. It's just how everybody else does it.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter <david(at)fetter(dot)org> writes:
> I am not suggesting that this change be immediate, and it's not ivory
> tower. It's just how everybody else does it.

You keep saying that, and it's completely meaningless. What do you know
about the development practices of Oracle, or DB2, or even Mysql?

regards, tom lane

Tom Lane wrote:
> David Fetter <david(at)fetter(dot)org> writes:
>
>> I am not suggesting that this change be immediate, and it's not ivory
>> tower. It's just how everybody else does it.
>>
>
> You keep saying that, and it's completely meaningless. What do you know
> about the development practices of Oracle, or DB2, or even Mysql?
>
>
>

The other point is that we don't change the on-disk format for fun, but
for good reason. If it's going to get harder we'll pay a price. That
doesn't mean we should go on as we have been, but just saying "we've
been sloppy" is telling less than half the story. If one believes the
rumors, Oracle at least has in the past paid dearly for supporting old
formats.

cheers

andrew

David Fetter wrote:
> > Odds are that the patch submitters will not understand enough to
> > know how to modify pg_migrator, but just knowing something broke is
> > usually enough for the hackers group to find a fix.
>
> This is a pretty serious drawback. If we're going to require that
> people send migration scripts when they change the on-disk format,
> this needs to be easy. The program, whatever it turns out to be,
> needs to take composable changes which people would submit along with
> their patch, just as they do docs.
>
> How much refactoring of pg_migrator would such a change take? Would
> it be worthwhile to do that, or just start a different project?

I have no idea what you are thinking. pg_migrator is in C just like the
backend code. Do you want some plugin module to do migrations? I have
neither the time or desire to implement that.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Aug 7, 2009, at 11:50 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:

> David Fetter wrote:
>>> Odds are that the patch submitters will not understand enough to
>>> know how to modify pg_migrator, but just knowing something broke is
>>> usually enough for the hackers group to find a fix.
>>
>> This is a pretty serious drawback. If we're going to require that
>> people send migration scripts when they change the on-disk format,
>> this needs to be easy. The program, whatever it turns out to be,
>> needs to take composable changes which people would submit along with
>> their patch, just as they do docs.
>>
>> How much refactoring of pg_migrator would such a change take? Would
>> it be worthwhile to do that, or just start a different project?
>
> I have no idea what you are thinking. pg_migrator is in C just like
> the
> backend code. Do you want some plugin module to do migrations? I
> have
> neither the time or desire to implement that.

It seems that it would also negate one of the major benefits of
pg_migrator, which is that it doesn't require you to rewrite all of
your data.

...Robert

Robert Haas wrote:
> On Aug 7, 2009, at 11:50 PM, Bruce Momjian <bruce(at)momjian(dot)us> wrote:
>
> > David Fetter wrote:
> >>> Odds are that the patch submitters will not understand enough to
> >>> know how to modify pg_migrator, but just knowing something broke is
> >>> usually enough for the hackers group to find a fix.
> >>
> >> This is a pretty serious drawback. If we're going to require that
> >> people send migration scripts when they change the on-disk format,
> >> this needs to be easy. The program, whatever it turns out to be,
> >> needs to take composable changes which people would submit along with
> >> their patch, just as they do docs.
> >>
> >> How much refactoring of pg_migrator would such a change take? Would
> >> it be worthwhile to do that, or just start a different project?
> >
> > I have no idea what you are thinking. pg_migrator is in C just like
> > the
> > backend code. Do you want some plugin module to do migrations? I
> > have
> > neither the time or desire to implement that.
>
> It seems that it would also negate one of the major benefits of
> pg_migrator, which is that it doesn't require you to rewrite all of
> your data.

I don't understand how this relates to rewriting data. David was asking
if we should require patch submitters to update pg_migrator. If they
can do that, great, but if not, we can figure out an approach for them.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Fri, Aug 07, 2009 at 06:28:34PM -0400, Tom Lane wrote:
> David Fetter <david(at)fetter(dot)org> writes:
> > I am not suggesting that this change be immediate, and it's not ivory
> > tower. It's just how everybody else does it.
>
> You keep saying that, and it's completely meaningless. What do you know
> about the development practices of Oracle, or DB2, or even Mysql?

When I was at Sybase, changes to the on disk structure were required to
provide code to do the migration. Nonetheless, at release time, the
migrate process was almost always discovered to be broken, sometimes even
before it was shipped to customers.

Of course, Sybase implemented its own complete filesystem layer on top of
raw partitions, so there was more scope to go wrong, especially since it
was possible to corrupt the on disk structure in subtle ways that would
not be discovered in normal operation but that would cause migration to
corrupt it still further.

In fairness, this is a very difficult problem to solve well and I expect
to rely on dump/load migrations for quite sometime.

-dg

--
David Gould daveg(at)sonic(dot)net 510 536 1443 510 282 0869
If simplicity worked, the world would be overrun with insects.

On Saturday 08 August 2009 01:28:34 Tom Lane wrote:
> David Fetter <david(at)fetter(dot)org> writes:
> > I am not suggesting that this change be immediate, and it's not ivory
> > tower. It's just how everybody else does it.
>
> You keep saying that, and it's completely meaningless. What do you know
> about the development practices of Oracle, or DB2, or even Mysql?

Whenever MySQL requires a disk format change, they write a new storage engine.

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > I think it's a lot more nebulous than that. At the same time I think the
> > days when we can blithely change the on-disk format with hardly a
> > thought for migration are over. IOW, there's agreement things have to
> > change, but the exact shape of the change is not yet clear (at least to
> > me) ;-)
>
> Yeah. I think we're going to start paying more than zero attention to
> this, but we don't yet have a handle on what the real parameters are.
> In particular, it's hard to argue that pg_migrator has yet achieved
> more than experimental status, so accepting or rejecting patches on
> the grounds of whether they would or would not break pg_migrator might
> be a bit premature. And at the other end of the spectrum, nobody except
> Zdenek wants to deal with changes as invasive as the ones he's proposed.
> So we're still feeling our way here. We do *not* have a framework in
> which someone could submit a patch that includes an on-disk migration
> aspect, so David's position that we should immediately institute a
> hard requirement for such seems a bit ivory-tower. We might as well
> just say that the on-disk format is frozen, because that's what the
> effect would be.
>
> But having said all that, I'm okay with a go-slow position on on-disk
> changes for the next little while. That would give us some breathing
> room to see if something like pg_migrator is really workable at all.
> We need to find out just how far that approach goes before we can make
> many decisions in this area.

I would be happy if we can just allow pg_migrator to _detect_
incompatible changes, and that nothing _major_ changes between releases
that pg_migrator can't fix (we fixed the sequence changes in 8.4).

/contrib changes are part of the first group, so I am happy to know
about them and detect them; changing the tuple format, on the other
hand, could make pg_migrator useless.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Monday 03 August 2009 17:44:32 Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > Does it need a version number change? Maybe just a tag (no branch) is
> > all that is required.
>
> I think that we do want the alpha releases to identify themselves as
> such. And we want a marker in CVS as to what state the alpha release
> corresponds to. Peter's label-and-undo approach seems like a kluge;
> and it doesn't scale to consider the possibility that we might
> want to re-release an alpha after fixing some particularly evil bug.
> A tag without a branch won't handle that either.
>
> I feel that making a branch is the way to go. If we try to get away
> with a shortcut, we'll probably regret it.

Well then, naming:

Branch: ???
Tag: REL8_5_ALPHA1

daveg <daveg(at)sonic(dot)net> wrote:

> When I was at Sybase, changes to the on disk structure were required
> to provide code to do the migration. Nonetheless, at release time,
> the migrate process was almost always discovered to be broken,
> sometimes even before it was shipped to customers.

As a long-time user of Sybase SQL Server and it's later incarnation as
Adaptive Server Enterprise, I can confirm that the result of this
approach was often unsatisfactory from a user perspective. That is,
the discovery was not always made before shipping to customers. :-(

I have high hopes for pg_migrator, but the pg_dump | psql approach has
always worked well for us -- no surprises and no pain beyond the
resource management issues.

-Kevin