Toward pg_upgrade

Folks,

I'm sure I'm not the first to bring up this way of doing pg_upgrade,
but perhaps I can help seed a fruitful discussion on the matter.

As background, I'd like to go over our policy of, "The code patch must
be accompanied by any doc patches that it implies." I believe that
this policy is good and wise, as it has helped produce both extremely
high code quality and product usability over the years, and that it
will continue to do the same as time goes on.

So here's my Modest Proposal™.

Where the rule now reads,

The code patch must be accompanied by any doc patches that it implies.

It should read,

The code patch must be accompanied by any doc patches *and any needed
upgrade transformations* that it implies.

By "upgrade transformations," I mean:

* Things that change the storage format from the pre-patched
version to what the post-patched version, and

* Things that transform configuration files from the pre-patched
version to the post-patched version.

* The ever-popular Stuff Dave Hasn't Thought Of That People Bark Their
Shins On.

Ideally, these transformations would be both idempotent and
reversible, although I understand that they may, by their nature, be
neither.

Advantages of making this policy change:
* Pg_upgrade actually happens as a matter of routine
* It's testable one change at a time

Disadvantages:
* Increased work on the front-end for new changes
* Higher barriers to entry

Cheers,
D
--
David Fetter david(at)fetter(dot)org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

David Fetter wrote:
> As background, I'd like to go over our policy of, "The code patch must
> be accompanied by any doc patches that it implies."

Although it is worth noting this policy is not religiously followed
anyway (e.g. the recent roles patch). I think we basically assume that
the person contributing a code patch is on the hook to write the docs at
some point before the next release, unless they can convince someone
else to do it for them.

> Where the rule now reads,
>
> The code patch must be accompanied by any doc patches that it implies.
>
> It should read,
>
> The code patch must be accompanied by any doc patches *and any needed
> upgrade transformations* that it implies.

I think this misses the point. The hurdle that needs to be cleared for
pg_upgrade is to write the infrastructure needed to migrate the system
catalogs and data directories from one release to another in a reliable
way. Once that is done, then yes, subsequent system catalog
modifications would need to include the necessary changes to the upgrade
infrastructure to make pg_upgrade continue to work. But until we have
pg_upgrade in the first place, the requirement you state above could be
simplified to "no changes that would require an initdb", which is
obviously a non-starter.

-Neil

On Thu, Jul 14, 2005 at 02:41:13PM +1000, Neil Conway wrote:
> David Fetter wrote:
> >As background, I'd like to go over our policy of, "The code patch
> >must be accompanied by any doc patches that it implies."
>
> Although it is worth noting this policy is not religiously followed
> anyway (e.g. the recent roles patch). I think we basically assume
> that the person contributing a code patch is on the hook to write
> the docs at some point before the next release, unless they can
> convince someone else to do it for them.

A similar, but not *that* similar thing could work for pg_upgrade
components. That's because the skill set for patching C code is more
similar to the skill set for making pg_upgrade components than it is
to the skillset for writing doc patches. I'm guessing, but I'm pretty
sure I'm right here.

For example, I've documented (and had accepted) a fair number of
things whose implementation details I still don't understand, but I
suspect that I'd be out of the pool as far as making pg_upgrade
components.

> >Where the rule now reads,
> >
> > The code patch must be accompanied by any doc patches that it implies.
> >
> >It should read,
> >
> > The code patch must be accompanied by any doc patches *and any needed
> > upgrade transformations* that it implies.
>
> I think this misses the point. The hurdle that needs to be cleared
> for pg_upgrade is to write the infrastructure needed to migrate the
> system catalogs and data directories from one release to another in
> a reliable way.

What work has been done to this end?

Cheers,
D
--
David Fetter david(at)fetter(dot)org http://fetter.org/
phone: +1 510 893 6100 mobile: +1 415 235 3778

Remember to vote!

On Wed, 2005-07-13 at 18:35 -0700, David Fetter wrote:
> I'm sure I'm not the first to bring up this way of doing pg_upgrade,
> but perhaps I can help seed a fruitful discussion on the matter.

> Ideally, these transformations would be both idempotent and
> reversible, although I understand that they may, by their nature, be
> neither.

I'm not sure it is easily possible to do all that you say as a general
rule. Each release will be different, so I think we need a person, not a
procedure. If we have a procedure, but no person, who will enforce the
procedure?

There is clearly an opening for an individual to track committed code
and to analyse what the upgrade actions would need to be for each. That
way, we might be able to implement things in a more upgrade friendly
manner (when we have a choice). We specialise in most other areas....

Changes to data format mostly will need an external program, since no
version of the server understands both formats.

Best Regards, Simon Riggs