Re: State of Beta 2

On Thu, 18 Sep 2003 12:11:18 -0400, Lamar Owen <lowen(at)pari(dot)edu> wrote:
>Marc G. Fournier wrote:
>> [...] upgrading is a key feature [...]
> a migration tool
>that could read the old format _without_a_running_old_backend_ [...]
> the new backend is powerless to recover the old data.
> OS upgrades [...], FreeBSD ports upgrades, and RPM
>upgrades are absolutely horrid at this point. [...]
>[censored] has a better system than we
>[...] the pain of upgrading [...]
>*I* should complain about a ramble? :-)

Lamar, I *STRONGLY* agree with almost everything you say here and in
other posts, except perhaps ...

You et al. seem to think that system catalog changes wouldn't be a
problem if only we could avoid page format changes. This is not
necessarily so. Page format changes can be handled without much
effort, if

. the changes are local to each page (the introduction of a level
indicator in btree pages is a counter-example),

. we can tell page type and version for every page,

. the new format does not need more space than the old one.

You wrote earlier:
| the developers who changed the on-disk format ...

Oh, that's me, I think. I am to blame for the heap tuple header
changes between 7.2 and 7.3; Tom did some cleanup work behind me but
cannot be held responsible for the on-disk-format incompatibilities.
I'm not aware of any other changes falling into this category for 7.3.
So you might as well have used the singular form ;-)

| ... felt it wasn't important to make it continue working.

This is simply not true. Seamless upgrade is *very* important, IMHO.
See http://archives.postgresql.org/pgsql-hackers/2002-06/msg00136.php
for example, and please keep in mind that I was still very "fresh" at
that time. Nobody demanded that I keep my promise and I got the
impression that a page format conversion tool was not needed because
there wouldn't be a pg_upgrade anyway.

Later, in your "Upgrading rant" thread, I even posted some code
(http://archives.postgresql.org/pgsql-hackers/2003-01/msg00294.php).
Unfortunately this went absolutely unnoticed, probably because it
looked so long because I fat-fingered the mail and included the code
twice. :-(

>It's all
>in the archives that nobdy seems willing to read over again. Why do we
>even have archives if they're not going to be used?

Sic!

While I'm at it, here are some comments not directly addressed to
Lamar:

Elsewhere in this current thread it has been suggested that the
on-disk format will stabilize at some time in the future and should
then be frozen to ensure painless upgrades. IMHO, at the moment when
data structures are declared stable and immutable the project is dead.

And I don't believe the myth that commercial database vendors have
reached a stable on-disk representation. Whoever said this, is kindly
asked to reveal his source of insight.

A working pg_upgrade is *not* the first thing we need. What we need
first is willingness to not break backwards compatibility. When
Postgres adopts a strategy of not letting in any change unless it is
fully compatible with the previous format or accompanied by an upgrade
script/program/whatever, that would be a huge step forward. First
breaking things for six month or more and then, when the release date
comes nearer, trying to build an upgrade tool is not the right
approach.

A - hopefully not too unrealistic - vision: _At_any_time_ during a
development cycle for release n+1 it is possible to take a cvs
snapshot, build it, take any release n database cluster, run a
conversion script over it (or not), and start the new postmaster with
-D myOldDataDir ...

Granted, this slows down development, primarily while developers are
not yet used to it. But once the infrastructure is in place, things
should get easier. While a developer is working on a new feature he
knows the old data structures as well as the new ones; this is the
best moment to design and implement an upgrade path, which is almost
hopeless if tried several months later by someone else.

And who says that keeping compatibility in mind while developing new
features cannot be fun? I assure you, it is!

Servus
Manfred