Re: pg_upgrade project status

The current project is not in good shape. Feature freeze is coming and
nothing is committed. Currently there are three patches in the game:

1) Space reservation
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php

This patch is mandatory for page online conversion and MUST TO be part
of postgreSQL 8.4. if not ... then we will be at the beginning next
year.

I sent updated version today.

2) pg_upgrade.sh
http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php

Pg_upgrade.sh is shell script for catalog conversion. It works for
8.3->8.4 upgrade. It will be useful while we will not have better
solution. Disadvantage is that it is korn shell script. The idea is to
rewrite it in PERL which is more portable, but I'm not PERL expert and
currently there is no workable solution.

And what is deadline for it? We can delivery it later with 8.4.1, but
best time for testing is during betas and RC period.

3) preupgrade script
http://archives.postgresql.org/pgsql-hackers/2008-12/msg01273.php

I sent WIP version of this script, which shows how reservation space
feature will be used. This part is not important now. It will be
important for 8.4->8.5 upgrade and cannot be finished until 8.5beta.

Thats all

Zdenek

Zdenek Kotala wrote:
> The current project is not in good shape. Feature freeze is coming and
> nothing is committed. Currently there are three patches in the game:
>
>
[...]

> 2) pg_upgrade.sh
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>
> Pg_upgrade.sh is shell script for catalog conversion. It works for
> 8.3->8.4 upgrade. It will be useful while we will not have better
> solution. Disadvantage is that it is korn shell script. The idea is to
> rewrite it in PERL which is more portable, but I'm not PERL expert and
> currently there is no workable solution.
>
> And what is deadline for it? We can delivery it later with 8.4.1, but
> best time for testing is during betas and RC period.
>
>
>

I have had a very brief look at this. Translation to perl doesn't look
difficult. I'll see what I can do during the next week or so.

cheers

andrew

Andrew Dunstan wrote:
> Zdenek Kotala wrote:
>> 2) pg_upgrade.sh
>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>>
>> Pg_upgrade.sh is shell script for catalog conversion. It works for
>> 8.3->8.4 upgrade. It will be useful while we will not have better
>> solution. Disadvantage is that it is korn shell script. The idea is to
>> rewrite it in PERL which is more portable, but I'm not PERL expert and
>> currently there is no workable solution.
>
> I have had a very brief look at this. Translation to perl doesn't look
> difficult. I'll see what I can do during the next week or so.

We don't require perl for any other feature, do we? Seems like a pretty
onerous requireemnt for Windows in particular. We do use perl in the
build scripts, but that's only required if you want to compile from source.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas wrote:
> Andrew Dunstan wrote:
>> Zdenek Kotala wrote:
>>> 2) pg_upgrade.sh
>>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>>>
>>> Pg_upgrade.sh is shell script for catalog conversion. It works for
>>> 8.3->8.4 upgrade. It will be useful while we will not have better
>>> solution. Disadvantage is that it is korn shell script. The idea is to
>>> rewrite it in PERL which is more portable, but I'm not PERL expert and
>>> currently there is no workable solution.
>>
>> I have had a very brief look at this. Translation to perl doesn't look
>> difficult. I'll see what I can do during the next week or so.
>
> We don't require perl for any other feature, do we? Seems like a pretty
> onerous requireemnt for Windows in particular. We do use perl in the
> build scripts, but that's only required if you want to compile from
> source.

I think it's fairly easy to install Perl on Windows actually. It
doesn't sound too onerous a requirement if you want in-place upgrade;
actually it looks a very reasonable one.

Much more reasonable than Korn shell in any case (or any shell for that
matter; I think anything is going to be more of a potentially painful
platform dependency than Perl).

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:

> I think it's fairly easy to install Perl on Windows actually. It
> doesn't sound too onerous a requirement if you want in-place upgrade;
> actually it looks a very reasonable one.

There are installers for it, but given that we made a point of porting
everything to C to avoid using any scripting languages on end-user
machines when we ported to Windows, it seems strange to relax that
'policy' now for convenience.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Tue, Jan 27, 2009 at 11:39:50AM -0300, Alvaro Herrera wrote:
> Heikki Linnakangas wrote:
> > Andrew Dunstan wrote:
> >> Zdenek Kotala wrote:
> >>> 2) pg_upgrade.sh
> >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
> >>>
> >>> Pg_upgrade.sh is shell script for catalog conversion. It works for
> >>> 8.3->8.4 upgrade. It will be useful while we will not have better
> >>> solution. Disadvantage is that it is korn shell script. The idea is to
> >>> rewrite it in PERL which is more portable, but I'm not PERL expert and
> >>> currently there is no workable solution.
> >>
> >> I have had a very brief look at this. Translation to perl doesn't look
> >> difficult. I'll see what I can do during the next week or so.
> >
> > We don't require perl for any other feature, do we? Seems like a pretty
> > onerous requireemnt for Windows in particular. We do use perl in the
> > build scripts, but that's only required if you want to compile from
> > source.
>
> I think it's fairly easy to install Perl on Windows actually. It
> doesn't sound too onerous a requirement if you want in-place upgrade;
> actually it looks a very reasonable one.
>
> Much more reasonable than Korn shell in any case (or any shell for that
> matter; I think anything is going to be more of a potentially painful
> platform dependency than Perl).
>
> --
> Alvaro Herrera http://www.CommandPrompt.com/
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
>
+1

I agree with Alvaro. Perl is a breeze to install on Windows with
Activestate and that using shell code to perform this task adds a
huge platform dependency to the code. Perl is a known and well defined
quantity for scripting.

Cheers,
Ken

On Tue, Jan 27, 2009 at 8:44 AM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> The current project is not in good shape. Feature freeze is coming and
> nothing is committed. Currently there are three patches in the game:

Correction: feature freeze is long past.

> 1) Space reservation
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php
>
> This patch is mandatory for page online conversion and MUST TO be part
> of postgreSQL 8.4. if not ... then we will be at the beginning next
> year.
>
> I sent updated version today.

I thought we pretty much had agreement that space reservation was not
a good solution to anything, although I admit I'm not quite clear on
what alternative was being proposed.

> 2) pg_upgrade.sh
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>
> Pg_upgrade.sh is shell script for catalog conversion. It works for
> 8.3->8.4 upgrade. It will be useful while we will not have better
> solution. Disadvantage is that it is korn shell script. The idea is to
> rewrite it in PERL which is more portable, but I'm not PERL expert and
> currently there is no workable solution.
>
> And what is deadline for it? We can delivery it later with 8.4.1, but
> best time for testing is during betas and RC period.

Why is the deadline different than anything else? Surely the deadline
is 11/1/2008, and you missed it.

Admittedly, there is some window for reworking existing patches after
the start of the commitfest, but this patch wasn't even added to the
CommitFest wiki until December 5th, after being sent to the list the
previous day. That's not close, and there's been little discussion of
it on the mailing list since then, and given that it's written in ksh,
it's clearly going to require a complete rewrite to be committable.

Three months after the CommitFest started is not the right time to
start a complete rewrite of a feature that wasn't even on time in the
first place. There is nothing whatever to prevent you from releasing
this on pgfoundry, but the idea that we should spend time on this
rather than the half a dozen (or more) patches that have had far more
work and are probably far closer to being committable strikes me as
100% wrong.

> 3) preupgrade script
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg01273.php
>
> I sent WIP version of this script, which shows how reservation space
> feature will be used. This part is not important now. It will be
> important for 8.4->8.5 upgrade and cannot be finished until 8.5beta.

...Robert

Zdenek Kotala wrote:

> 2) pg_upgrade.sh
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>
> Pg_upgrade.sh is shell script for catalog conversion. It works for
> 8.3->8.4 upgrade.

What's the reason this script uses a postmaster? It seems it would be
easier to control if you used a standalone backend (--single) for each
time you are piping stuff to psql. That would reduce the need to
configure authentication, hostnames, etc etc.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 1/27/09, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> This patch is mandatory for page online conversion and MUST TO be part
> of postgreSQL 8.4. if not ... then we will be at the beginning next
> year.

Just to clarify, does that mean that your patch has to be in for there
to be any chance of in-place upgrade 8.4->8.5?

merlin

Heikki Linnakangas wrote:
> Andrew Dunstan wrote:
>> Zdenek Kotala wrote:
>>> 2) pg_upgrade.sh
>>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>>>
>>> Pg_upgrade.sh is shell script for catalog conversion. It works for
>>> 8.3->8.4 upgrade. It will be useful while we will not have better
>>> solution. Disadvantage is that it is korn shell script. The idea is to
>>> rewrite it in PERL which is more portable, but I'm not PERL expert and
>>> currently there is no workable solution.
>>
>> I have had a very brief look at this. Translation to perl doesn't
>> look difficult. I'll see what I can do during the next week or so.
>
> We don't require perl for any other feature, do we? Seems like a
> pretty onerous requireemnt for Windows in particular. We do use perl
> in the build scripts, but that's only required if you want to compile
> from source.

Well, from that POV the only portable thing is to translate it into C.
That's just a whole lot more work (remember initdb?). The perl port for
Windows is easily installable, widely used and well regarded. It doesn't
strike me as too high a price to pay for the ability to do upgrades, but
I'll defer to more Windows-centric commenters.

cheers

andrew

Dave Page wrote:
> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
>
> > I think it's fairly easy to install Perl on Windows actually. It
> > doesn't sound too onerous a requirement if you want in-place upgrade;
> > actually it looks a very reasonable one.
>
> There are installers for it, but given that we made a point of porting
> everything to C to avoid using any scripting languages on end-user
> machines when we ported to Windows, it seems strange to relax that
> 'policy' now for convenience.

If you prefer to not have pg_upgrade at all for 8.4, feel free to
request it to be written in C ... But I'm sure that's not what you
meant.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Dave Page <dpage(at)pgadmin(dot)org> writes:
> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
>> I think it's fairly easy to install Perl on Windows actually. It
>> doesn't sound too onerous a requirement if you want in-place upgrade;
>> actually it looks a very reasonable one.

> There are installers for it, but given that we made a point of porting
> everything to C to avoid using any scripting languages on end-user
> machines when we ported to Windows, it seems strange to relax that
> 'policy' now for convenience.

Indeed. We might put up with a perl script for awhile for the sake of
development expediency, but the long-term expectation would have to be
that someone would rewrite it in C. Given that, I wonder whether
there's much point in a rewrite into Perl if we already have a working
shell script. I suppose someone will say "but you'll get no testing
from Windows users then..."

regards, tom lane

Andrew Dunstan wrote:
>
>
> Heikki Linnakangas wrote:
>> Andrew Dunstan wrote:
>>> Zdenek Kotala wrote:
>>>> 2) pg_upgrade.sh
>>>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>>>>
>>>> Pg_upgrade.sh is shell script for catalog conversion. It works for
>>>> 8.3->8.4 upgrade. It will be useful while we will not have better
>>>> solution. Disadvantage is that it is korn shell script. The idea is to
>>>> rewrite it in PERL which is more portable, but I'm not PERL expert and
>>>> currently there is no workable solution.
>>>
>>> I have had a very brief look at this. Translation to perl doesn't
>>> look difficult. I'll see what I can do during the next week or so.
>>
>> We don't require perl for any other feature, do we? Seems like a
>> pretty onerous requireemnt for Windows in particular. We do use perl
>> in the build scripts, but that's only required if you want to compile
>> from source.
>
> Well, from that POV the only portable thing is to translate it into C.
> That's just a whole lot more work (remember initdb?). The perl port for
> Windows is easily installable, widely used and well regarded. It doesn't
> strike me as too high a price to pay for the ability to do upgrades, but
> I'll defer to more Windows-centric commenters.

Either way, there's no point to discuss that in detail until there
actually is a working implementation out there... perl will do fine
until then. Once we have that, we can discuss if doing it in C will be
worthwhile, or if we're just going to require perl for that one feature.

I have a hard time thinking that we'll have wasted a lot of time on
first doing a perl implementation if we have to rewrite it in C later.
The other way around would be a waste though. The amount of time spent
on the perl implementation I expect to be a *lot* less than the
combination of thinking up the *way* to do it in general and the C
implementation time.

//Magnus

On Tue, Jan 27, 2009 at 2:49 PM, Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:
> Dave Page wrote:
>> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
>> <alvherre(at)commandprompt(dot)com> wrote:
>>
>> > I think it's fairly easy to install Perl on Windows actually. It
>> > doesn't sound too onerous a requirement if you want in-place upgrade;
>> > actually it looks a very reasonable one.
>>
>> There are installers for it, but given that we made a point of porting
>> everything to C to avoid using any scripting languages on end-user
>> machines when we ported to Windows, it seems strange to relax that
>> 'policy' now for convenience.
>
> If you prefer to not have pg_upgrade at all for 8.4, feel free to
> request it to be written in C ... But I'm sure that's not what you
> meant.

I'd rather it was written in an appropriate language before feature
freeze, not in a language that makes it easier for the author but a
shade harder for thousands of users three months into feature freeze.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

* Magnus Hagander (magnus(at)hagander(dot)net) wrote:
> Either way, there's no point to discuss that in detail until there
> actually is a working implementation out there... perl will do fine
> until then. Once we have that, we can discuss if doing it in C will be
> worthwhile, or if we're just going to require perl for that one feature.

+1

Stephen

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
[pg_upgrade...]
> Why is the deadline different than anything else?

err, isn't it because it'd be kind of difficult to do an upgrade script
with large catalog-changing patches outstanding..? I thought some
leeway was given for pg_upgrade specifically due to that, tho perhaps
I'm wrong.

Thanks,

Stephen

On Tue, Jan 27, 2009 at 10:08 AM, Stephen Frost <sfrost(at)snowman(dot)net> wrote:
> * Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> [pg_upgrade...]
>> Why is the deadline different than anything else?
>
> err, isn't it because it'd be kind of difficult to do an upgrade script
> with large catalog-changing patches outstanding..? I thought some
> leeway was given for pg_upgrade specifically due to that, tho perhaps
> I'm wrong.

Sure... if this script had been 100% commitable on 11/1 and now needed
to be adjusted, I can't imagine anyone objecting. But the patch
wasn't submitted until 12/4 and still needs a complete rewrite in a
different programming language as of 1/27. Do you think we would be
arguing about whether to accept Hot Standby now if it were written in
ksh? And that was at least submitted on time.

...Robert

> I think it's fairly easy to install Perl on Windows actually. It
> doesn't sound too onerous a requirement if you want in-place upgrade;
> actually it looks a very reasonable one.
>
> Much more reasonable than Korn shell in any case (or any shell for that
> matter; I think anything is going to be more of a potentially painful
> platform dependency than Perl).
>

May I humbly recommend to rewrite in Python? That should be as
difficult / easy as PERL, AND there is a very robust py2exe
implementation, which allows to create a single .exe file which
contains everything.

Python is present on all Linux, Windows users are totally comfortable
with .exe files.

Harald

--
GHUM Harald Massa
persuadere et programmare
Harald Armin Massa
Spielberger Straße 49
70435 Stuttgart
0173/9409607
no fx, no carrier pigeon
-
EuroPython 2009 will take place in Birmingham - Stay tuned!

Hi

> I have had a very brief look at this. Translation to perl doesn't look
> difficult. I'll see what I can do during the next week or so.

Perhaps I can lend you a hand if you need help with this.

--

Med venlig hilsen
Kaare Rasmussen, Jasonic

Jasonic Telefon: +45 3816 2582
Nordre Fasanvej 12
2000 Frederiksberg Email: kaare(at)jasonic(dot)dk

Harald Armin Massa wrote:
>> I think it's fairly easy to install Perl on Windows actually. It
>> doesn't sound too onerous a requirement if you want in-place upgrade;
>> actually it looks a very reasonable one.
>>
>> Much more reasonable than Korn shell in any case (or any shell for that
>> matter; I think anything is going to be more of a potentially painful
>> platform dependency than Perl).
>>
>>
>
> May I humbly recommend to rewrite in Python? That should be as
> difficult / easy as PERL, AND there is a very robust py2exe
> implementation, which allows to create a single .exe file which
> contains everything.
>
> Python is present on all Linux, Windows users are totally comfortable
> with .exe files.
>
>
>

No, I don't think so ;-) Without getting into language wars, Perl is
already our de facto cross-platform scripting tool. We don't need to be
adding extra knowledge requirements to the project, nor extra build
requirements (right now, perl is already required for building from
source, or when building with MSVC, or when running a buildfarm animal)

cheers

andrew

Alvaro Herrera píše v út 27. 01. 2009 v 11:47 -0300:
> Zdenek Kotala wrote:
>
> > 2) pg_upgrade.sh
> > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
> >
> > Pg_upgrade.sh is shell script for catalog conversion. It works for
> > 8.3->8.4 upgrade.
>
> What's the reason this script uses a postmaster? It seems it would be
> easier to control if you used a standalone backend (--single) for each
> time you are piping stuff to psql. That would reduce the need to
> configure authentication, hostnames, etc etc.

It is 8.1->8.2 upgrade relict. IIRC 8.1 does not have --single. I think
single mode can be used.

thanks Zdenek

On Tue, 2009-01-27 at 09:48 -0500, Andrew Dunstan wrote:

> > We don't require perl for any other feature, do we? Seems like a
> > pretty onerous requireemnt for Windows in particular. We do use perl
> > in the build scripts, but that's only required if you want to compile
> > from source.
>
> Well, from that POV the only portable thing is to translate it into C.
> That's just a whole lot more work (remember initdb?). The perl port for
> Windows is easily installable, widely used and well regarded. It doesn't
> strike me as too high a price to pay for the ability to do upgrades, but
> I'll defer to more Windows-centric commenters.

Actually as much as perl is ubiquitous it isn't. What version of perl
shall we require? Will we require other modules? Does that version work
on all our supported platforms (HPUX, NETBSD?)

Joshua D. Drake

--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Joshua D. Drake wrote:
> On Tue, 2009-01-27 at 09:48 -0500, Andrew Dunstan wrote:
>
>
>>> We don't require perl for any other feature, do we? Seems like a
>>> pretty onerous requireemnt for Windows in particular. We do use perl
>>> in the build scripts, but that's only required if you want to compile
>>> from source.
>>>
>> Well, from that POV the only portable thing is to translate it into C.
>> That's just a whole lot more work (remember initdb?). The perl port for
>> Windows is easily installable, widely used and well regarded. It doesn't
>> strike me as too high a price to pay for the ability to do upgrades, but
>> I'll defer to more Windows-centric commenters.
>>
>
> Actually as much as perl is ubiquitous it isn't. What version of perl
> shall we require? Will we require other modules? Does that version work
> on all our supported platforms (HPUX, NETBSD?)
>
>
>

That's what my brief examination of the script was about - looking to
see if it could be translated portably. I think it very probably can. I
suspect it won't need any modules at all. I suspect any

cheers

andrew

On Tue, 2009-01-27 at 10:56 -0500, Andrew Dunstan wrote:
>

>
> That's what my brief examination of the script was about - looking to
> see if it could be translated portably. I think it very probably can. I
> suspect it won't need any modules at all. I suspect any

Accidental ctrl-enter?

O.k. well you certainly know more about perl than I do. If you are
comfortable with the portability, I have no complaints (except that I am
not willing to wait for it to be done for 8.4).

Joshua D. Drake

>
> cheers
>
> andrew
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

On Tue, Jan 27, 2009 at 04:23:18PM +0100, Harald Armin Massa wrote:
> > I think it's fairly easy to install Perl on Windows actually. It
> > doesn't sound too onerous a requirement if you want in-place upgrade;
> > actually it looks a very reasonable one.
> >
> > Much more reasonable than Korn shell in any case (or any shell for that
> > matter; I think anything is going to be more of a potentially painful
> > platform dependency than Perl).
> >
>
> May I humbly recommend to rewrite in Python? That should be as
> difficult / easy as PERL, AND there is a very robust py2exe
> implementation, which allows to create a single .exe file which
> contains everything.
>
> Python is present on all Linux, Windows users are totally comfortable
> with .exe files.
>
> Harald
>
>
Great idea, perl2exe is available as well and will allow the continued use
of perl for our internal scripting language.

Cheers,
Ken

Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500:
> On 1/27/09, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> > This patch is mandatory for page online conversion and MUST TO be part
> > of postgreSQL 8.4. if not ... then we will be at the beginning next
> > year.
>
> Just to clarify, does that mean that your patch has to be in for there
> to be any chance of in-place upgrade 8.4->8.5?

Ok, There two patch in the queue for 8.5 which will bump page layout
version. Then we will need it.

Zdenek

Robert Haas píše v út 27. 01. 2009 v 09:45 -0500:

>
> > 1) Space reservation
> > http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
> > http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php
> >
> > This patch is mandatory for page online conversion and MUST TO be part
> > of postgreSQL 8.4. if not ... then we will be at the beginning next
> > year.
> >
> > I sent updated version today.
>
> I thought we pretty much had agreement that space reservation was not
> a good solution to anything, although I admit I'm not quite clear on
> what alternative was being proposed.

Maybe I miss something, but space reservation was selected as a best
way. Please, Could you point me related mailing thread?

Zdenek

Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100:

>
> I have a hard time thinking that we'll have wasted a lot of time on
> first doing a perl implementation if we have to rewrite it in C later.
> The other way around would be a waste though. The amount of time spent
> on the perl implementation I expect to be a *lot* less than the
> combination of thinking up the *way* to do it in general and the C
> implementation time.

Agree with Magnus. I'm happy with korn shell for now :-). And we need
better solution anyway for future releases. Current approach does not
support e.g. dropped column.

Zdenek

Dave Page píše v út 27. 01. 2009 v 14:56 +0000:
> On Tue, Jan 27, 2009 at 2:49 PM, Alvaro Herrera
> <alvherre(at)commandprompt(dot)com> wrote:
> > Dave Page wrote:
> >> On Tue, Jan 27, 2009 at 2:39 PM, Alvaro Herrera
> >> <alvherre(at)commandprompt(dot)com> wrote:
> >>
> >> > I think it's fairly easy to install Perl on Windows actually. It
> >> > doesn't sound too onerous a requirement if you want in-place upgrade;
> >> > actually it looks a very reasonable one.
> >>
> >> There are installers for it, but given that we made a point of porting
> >> everything to C to avoid using any scripting languages on end-user
> >> machines when we ported to Windows, it seems strange to relax that
> >> 'policy' now for convenience.
> >
> > If you prefer to not have pg_upgrade at all for 8.4, feel free to
> > request it to be written in C ... But I'm sure that's not what you
> > meant.
>
> I'd rather it was written in an appropriate language before feature
> freeze, not in a language that makes it easier for the author but a
> shade harder for thousands of users three months into feature freeze.

The script is one year old. It is integrated in the OpenSolaris and
Solaris 10U6. I presented the script on PGCon2008. Everybody could
complain about it long time ago. And from my point of view it is first
step then better solution will be developed in C - (see my presentation
- "initdb upgrade" command).

Zdenek

Zdenek Kotala wrote:
> Robert Haas píše v út 27. 01. 2009 v 09:45 -0500:
>
>>> 1) Space reservation
>>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
>>> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php
>>>
>>> This patch is mandatory for page online conversion and MUST TO be part
>>> of postgreSQL 8.4. if not ... then we will be at the beginning next
>>> year.
>>>
>>> I sent updated version today.
>> I thought we pretty much had agreement that space reservation was not
>> a good solution to anything, although I admit I'm not quite clear on
>> what alternative was being proposed.
>
> Maybe I miss something, but space reservation was selected as a best
> way. Please, Could you point me related mailing thread?

Space reservation is the way to go, but I remain of the opinion that
trying to predict the future is futile. When we know what we need (=
around the beginning of 8.5 beta), we can backpatch a patch to reserve
the needed amount of space on pages. It has to be dead simple to
consider applying it to a stable branch, but something like "reserve X
bytes for heap pages and Y bytes for b-tree pages if pre_upgrade_mode
GUC is set" should be OK. And we don't want to do anything more
complicated than that anyway before we know what we need.

(http://archives.postgresql.org/message-id/49425D07.6030701@enterprisedb.com)

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Zdenek Kotala wrote:
> Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100:
>
>> I have a hard time thinking that we'll have wasted a lot of time on
>> first doing a perl implementation if we have to rewrite it in C later.
>> The other way around would be a waste though. The amount of time spent
>> on the perl implementation I expect to be a *lot* less than the
>> combination of thinking up the *way* to do it in general and the C
>> implementation time.
>
> Agree with Magnus. I'm happy with korn shell for now :-). And we need
> better solution anyway for future releases. Current approach does not
> support e.g. dropped column.

FWIW, pg-migrator is written in C, and last I looked it's roughly on par
feature-wise. Perhaps Zdenek can list what exactly the differences are,
so that we can figure out how much effort it would be to just fill the
gaps in pg-migrator.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas píše v út 27. 01. 2009 v 21:07 +0200:
> Zdenek Kotala wrote:
> > Robert Haas píše v út 27. 01. 2009 v 09:45 -0500:
> >
> >>> 1) Space reservation
> >>> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
> >>> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php
> >>>
> >>> This patch is mandatory for page online conversion and MUST TO be part
> >>> of postgreSQL 8.4. if not ... then we will be at the beginning next
> >>> year.
> >>>
> >>> I sent updated version today.
> >> I thought we pretty much had agreement that space reservation was not
> >> a good solution to anything, although I admit I'm not quite clear on
> >> what alternative was being proposed.
> >
> > Maybe I miss something, but space reservation was selected as a best
> > way. Please, Could you point me related mailing thread?
>
> Space reservation is the way to go, but I remain of the opinion that
> trying to predict the future is futile. When we know what we need (=
> around the beginning of 8.5 beta), we can backpatch a patch to reserve
> the needed amount of space on pages. It has to be dead simple to
> consider applying it to a stable branch, but something like "reserve X
> bytes for heap pages and Y bytes for b-tree pages if pre_upgrade_mode
> GUC is set" should be OK. And we don't want to do anything more
> complicated than that anyway before we know what we need.

Set general value for heap or btree is possible but not optimal. For
example If you have 5TB table and page layout changes do not affected
this table but some small table needs perform a cleanup then 5TB will be
processed in general approach, but in optimal variant it will be mark as
prepared on upgrade automatically.

The patch which I send do this. It is optimal. Only what we don't know
is correct calculation, but when 8.5 will be release we will able to
create script/procedure which calculate exact space for each relation.

Nothing should be backpatched, because script calls something like

ALTER TABLE test SET (rs_perpage=4);

or

UPDATE pg_class SET rs_ perpage=4 where relname='test';

This is a whole idea.

thanks Zdenek

Zdenek Kotala <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:
> Merlin Moncure pe v t 27. 01. 2009 v 09:47 -0500:
>> Just to clarify, does that mean that your patch has to be in for there
>> to be any chance of in-place upgrade 8.4->8.5?

> Ok, There two patch in the queue for 8.5 which will bump page layout
> version. Then we will need it.

I see nothing on the 2009-First list that requires any such thing.

In any case we've been over this ground before: we have agreed in the
past that we'd be willing to reject/postpone patches that change on-disk
data layout if in-place upgrade would otherwise be available. I think
that space reservation is extremely far down the list of "must haves"
for getting 8.4->8.5 upgrade into place. You should first work on an
update process that supports catalog changes, and get that committed.
Once that's in place you'll have enough political capital to prevent
changes in user data layout, and then we'd start to think about schemes
like space reservation that could relax that ground rule.

regards, tom lane

On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
>
> Dave Page píše v út 27. 01. 2009 v 14:56 +0000:
>> I'd rather it was written in an appropriate language before feature
>> freeze, not in a language that makes it easier for the author but a
>> shade harder for thousands of users three months into feature freeze.
>
> The script is one year old. It is integrated in the OpenSolaris and
> Solaris 10U6. I presented the script on PGCon2008. Everybody could
> complain about it long time ago.

Neither of which are the correct place to submit work for inclusion in
PostgreSQL. We're currently arguing about bouncing features that are
in late stages of development and were submitted long before feature
freeze. Your upgrade script is in the early stages of development
(although it may be mature as a korn script), and needs to be entirely
rewritten in a different language before it can even be considered for
inclusion. If it's been around for a year, why hasn't it been
submitted long ago so we could have rewritten and reviewed etc. in
plenty of time?

I'd love to see UIP, but even I, as someone that generally (and
sometimes loudly) supports good work that is submitted close to
feature freeze am having a hard time seeing why this work should be
accepted so late, even if its in /contrib.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Zdenek Kotala wrote:
> Set general value for heap or btree is possible but not optimal. For
> example If you have 5TB table and page layout changes do not affected
> this table but some small table needs perform a cleanup then 5TB will be
> processed in general approach, but in optimal variant it will be mark as
> prepared on upgrade automatically.

Only the space reservation of *new* inserts/updates need to be
integrated in the backend. In addition to that, we'll need a pre-upgrade
script that decides which tables need processing, and process them. This
is no different whether we accept some generic space reservation patch
now, or backpatch a more precise one later.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas píše v út 27. 01. 2009 v 21:13 +0200:
> Zdenek Kotala wrote:
> > Magnus Hagander píše v út 27. 01. 2009 v 15:54 +0100:
> >
> >> I have a hard time thinking that we'll have wasted a lot of time on
> >> first doing a perl implementation if we have to rewrite it in C later.
> >> The other way around would be a waste though. The amount of time spent
> >> on the perl implementation I expect to be a *lot* less than the
> >> combination of thinking up the *way* to do it in general and the C
> >> implementation time.
> >
> > Agree with Magnus. I'm happy with korn shell for now :-). And we need
> > better solution anyway for future releases. Current approach does not
> > support e.g. dropped column.
>
> FWIW, pg-migrator is written in C, and last I looked it's roughly on par
> feature-wise. Perhaps Zdenek can list what exactly the differences are,
> so that we can figure out how much effort it would be to just fill the
> gaps in pg-migrator.

The reason why I wrote own script was that I found lot of small issues
which needed to solve. OK finally maybe fixing them could take short
time, but I did not thing this one year ago :(.

OK there is what I remember:

The main difference is interaction with postgreSQL - stoping/starting
and so on. IIRC pg migrator does not do that. pg_upgrade script sets
trust permisions on database to perform upgrade and so on.

The migration process is mostly identical. if IIRC pg_migrator ignores
locales - it fails when is not run under English locales and also initdb
is not called with correct parameters.

Check also freezing database. I think it is not correctly performed.

Remove all ANSI color staff. You see nothing when you run it on black
background :(.

I guess, Tablespaces are handled differently. I try to keep data on same
mounpoint.

And last I think license is not BSD, I don't know how big problem it is.

Zdenek

Tom Lane píše v út 27. 01. 2009 v 14:31 -0500:
> Zdenek Kotala <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:
> > Merlin Moncure píše v út 27. 01. 2009 v 09:47 -0500:
> >> Just to clarify, does that mean that your patch has to be in for there
> >> to be any chance of in-place upgrade 8.4->8.5?
>
> > Ok, There two patch in the queue for 8.5 which will bump page layout
> > version. Then we will need it.
>
> I see nothing on the 2009-First list that requires any such thing.

There are page CRC and item alignment optimization. And If IIRC that
somebody what to modify compression?

> In any case we've been over this ground before: we have agreed in the
> past that we'd be willing to reject/postpone patches that change on-disk
> data layout if in-place upgrade would otherwise be available. I think
> that space reservation is extremely far down the list of "must haves"
> for getting 8.4->8.5 upgrade into place.

If it means that we will not change Page Layout Version in 8.5 then I'm
happy.

> You should first work on an
> update process that supports catalog changes, and get that committed.
> Once that's in place you'll have enough political capital to prevent
> changes in user data layout, and then we'd start to think about schemes
> like space reservation that could relax that ground rule.

OK. There is pg_upgrade.sh for 8.3->8.4 while I or someone develop
something better. But new solution will be at first for 8.4->8.5.

thanks Zdenek

On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote:

> And last I think license is not BSD, I don't know how big problem it is.

Uhh.. that kills it. We only accept BSD.

Joshua D. Drake

>
> Zdenek
>
>
>
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Heikki Linnakangas píše v út 27. 01. 2009 v 21:37 +0200:
> Zdenek Kotala wrote:
> > Set general value for heap or btree is possible but not optimal. For
> > example If you have 5TB table and page layout changes do not affected
> > this table but some small table needs perform a cleanup then 5TB will be
> > processed in general approach, but in optimal variant it will be mark as
> > prepared on upgrade automatically.
>
> Only the space reservation of *new* inserts/updates need to be
> integrated in the backend. In addition to that, we'll need a pre-upgrade
> script that decides which tables need processing, and process them. This
> is no different whether we accept some generic space reservation patch
> now, or backpatch a more precise one later.

If it is possible then OK, but I little bit afraid about it.

Zdenek

Joshua D. Drake wrote:
> On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote:
>
>> And last I think license is not BSD, I don't know how big problem it is.
>
> Uhh.. that kills it. We only accept BSD.

Actually, that was just fixed. Once somebody raised the issue, EDB
changed the license (there have been on other contributors). I'm
waiting for some documentation from Denis on that, but I expect to have
it in a few days.

--Josh

Joshua D. Drake wrote:
> On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote:
>
>> And last I think license is not BSD, I don't know how big problem it is.
>
> Uhh.. that kills it. We only accept BSD.

Actually, it has been relicensed to BSD license, after a short
discussion started by Bruce within EnterpriseDB (all the code is written
by EDB employees). This happened yesterday, so I can't blame you for not
noticing ;-).

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Tue, 2009-01-27 at 22:12 +0200, Heikki Linnakangas wrote:
> Joshua D. Drake wrote:
> > On Tue, 2009-01-27 at 20:46 +0100, Zdenek Kotala wrote:
> >
> >> And last I think license is not BSD, I don't know how big problem it is.
> >
> > Uhh.. that kills it. We only accept BSD.
>
> Actually, it has been relicensed to BSD license, after a short
> discussion started by Bruce within EnterpriseDB (all the code is written
> by EDB employees). This happened yesterday, so I can't blame you for not
> noticing ;-).

Phew! :)

Joshua D. Drake

>
> --
> Heikki Linnakangas
> EnterpriseDB http://www.enterprisedb.com
>
--
PostgreSQL - XMPP: jdrake(at)jabber(dot)postgresql(dot)org
Consulting, Development, Support, Training
503-667-4564 - http://www.commandprompt.com/
The PostgreSQL Company, serving since 1997

Dave Page píše v út 27. 01. 2009 v 19:36 +0000:
> On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> >
> > Dave Page píše v út 27. 01. 2009 v 14:56 +0000:
> >> I'd rather it was written in an appropriate language before feature
> >> freeze, not in a language that makes it easier for the author but a
> >> shade harder for thousands of users three months into feature freeze.
> >
> > The script is one year old. It is integrated in the OpenSolaris and
> > Solaris 10U6. I presented the script on PGCon2008. Everybody could
> > complain about it long time ago.
>
> Neither of which are the correct place to submit work for inclusion in
> PostgreSQL.

http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php

I sent it in April.

Zdenek

On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
>
> Dave Page píše v út 27. 01. 2009 v 19:36 +0000:
>> On Tue, Jan 27, 2009 at 7:04 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
>> >
>> > Dave Page píše v út 27. 01. 2009 v 14:56 +0000:
>> >> I'd rather it was written in an appropriate language before feature
>> >> freeze, not in a language that makes it easier for the author but a
>> >> shade harder for thousands of users three months into feature freeze.
>> >
>> > The script is one year old. It is integrated in the OpenSolaris and
>> > Solaris 10U6. I presented the script on PGCon2008. Everybody could
>> > complain about it long time ago.
>>
>> Neither of which are the correct place to submit work for inclusion in
>> PostgreSQL.
>
> http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php
>
> I sent it in April.

Thats a patch to the server, not a korn script. Wrong URL perhaps?

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

Dave Page píše v út 27. 01. 2009 v 20:36 +0000:
> On Tue, Jan 27, 2009 at 8:31 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> >>
> >> Neither of which are the correct place to submit work for inclusion in
> >> PostgreSQL.
> >
> > http://archives.postgresql.org/pgsql-hackers/2008-04/msg00990.php
> >
> > I sent it in April.
>
> Thats a patch to the server, not a korn script. Wrong URL perhaps?

Sorry, I copied wrong email URL from thread

http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php

But you can read there that it is korn shell script and there is link to
it.

Thats all
Zdenek

On Tuesday 27 January 2009 21:36:01 Dave Page wrote:
> If it's been around for a year, why hasn't it been
> submitted long ago so we could have rewritten and reviewed etc. in
> plenty of time?

It's in the commit fest listing, and the discussion about which programming
language to port it to happened weeks ago.

On Tue, Jan 27, 2009 at 8:52 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
>
> http://archives.postgresql.org/pgsql-hackers/2008-04/msg01051.php
>
> But you can read there that it is korn shell script and there is link to
> it.

OK, yes that does mention it. It's not exactly clear that it's
something you wanted to include though.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Tuesday 27 January 2009 16:52:15 Tom Lane wrote:
> Indeed. We might put up with a perl script for awhile for the sake of
> development expediency, but the long-term expectation would have to be
> that someone would rewrite it in C. Given that, I wonder whether
> there's much point in a rewrite into Perl if we already have a working
> shell script. I suppose someone will say "but you'll get no testing
> from Windows users then..."

The existing ksh script needs about two weeks of work to make it work outside
of Solaris and to make it more robust. Then you might as well rewrite it in
a more portable and robust language.

But yes, you have to think weeks here in terms of getting this polished.

On Tuesday 27 January 2009 16:54:02 Magnus Hagander wrote:
> Either way, there's no point to discuss that in detail until there
> actually is a working implementation out there... perl will do fine
> until then

There is a working implementation out there. It is -- you guessed it --
sitting in the commit fest queue waiting for review.

On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> On Tuesday 27 January 2009 21:36:01 Dave Page wrote:
>> If it's been around for a year, why hasn't it been
>> submitted long ago so we could have rewritten and reviewed etc. in
>> plenty of time?
>
> It's in the commit fest listing, and the discussion about which programming
> language to port it to happened weeks ago.

It was added to the commitfest page on December 5th, and links to an
email from the day before. The only discussion on the language I can
see in the linked thread is David Fetter offering to rewrite the code
in perl.

--
Dave Page
EnterpriseDB UK: http://www.enterprisedb.com

On Tuesday 27 January 2009 17:27:32 Kaare Rasmussen wrote:
> Hi
>
> > I have had a very brief look at this. Translation to perl doesn't look
> > difficult. I'll see what I can do during the next week or so.
>
> Perhaps I can lend you a hand if you need help with this.

Not to stop you are anything, but in the lifetime of this script, there have
been about half a dozen people offering to port or already porting it to
Perl. What it needs is a design and functional review to see if it is worth
porting or doing anything with at all.

On Tue, 27 Jan 2009, Peter Eisentraut wrote:

> On Tuesday 27 January 2009 21:36:01 Dave Page wrote:
>> If it's been around for a year, why hasn't it been
>> submitted long ago so we could have rewritten and reviewed etc. in
>> plenty of time?
>
> It's in the commit fest listing, and the discussion about which programming
> language to port it to happened weeks ago.

But as Robert Haas pointed out upthread that didn't show up there until
after the start of the main commitfest, and I will admit being the culprit
who snuck it onto there. The script was brought up repeatedly going back
to April, but Zdenek didn't explicitly submit it as part of his final set
for the commitfest. It was obvious to me you needed it all
along--Zdenek's other CF submissions implicitly depended on it if your
goal was to upgrade from 8.3 to 8.4 in-place--which is why I nudged him to
do that and then added the result to the page. I'm not sure what arguing
over the exact sequence here accomplishes anyway. It's come up a bunch of
times before, nobody ran with it, and Zdenek was more concerned about
space reservation.

Anyway, I see multiple people who seem excited by playing with a Perl
port. I found a shell->Perl conversion tool recently:
http://search.cpan.org/~clive/App-sh2p-0.04/

If you comment out all the calls to create_mapping which it chokes on for
some reason, it produces a non-functional but helpful conversion into
Perl. But since the primary dirty job here is to review whether the
original script works as expected on the database problems it tries to
solve rather than play with programming languages, I didn't think
reporting progress just on that front was particularly exciting.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Tuesday 27 January 2009 23:02:30 Dave Page wrote:
> On Tue, Jan 27, 2009 at 8:54 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:
> > On Tuesday 27 January 2009 21:36:01 Dave Page wrote:
> >> If it's been around for a year, why hasn't it been
> >> submitted long ago so we could have rewritten and reviewed etc. in
> >> plenty of time?
> >
> > It's in the commit fest listing, and the discussion about which
> > programming language to port it to happened weeks ago.
>
> It was added to the commitfest page on December 5th, and links to an
> email from the day before. The only discussion on the language I can
> see in the linked thread is David Fetter offering to rewrite the code
> in perl.

Well, the is ultimately about the same discussion we are having now, except
that people are a lot more excited when you threaten to cancel the
project. ;-)

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On Tuesday 27 January 2009 16:52:15 Tom Lane wrote:
>> Indeed. We might put up with a perl script for awhile for the sake of
>> development expediency, but the long-term expectation would have to be
>> that someone would rewrite it in C. Given that, I wonder whether
>> there's much point in a rewrite into Perl if we already have a working
>> shell script. I suppose someone will say "but you'll get no testing
>> from Windows users then..."

> The existing ksh script needs about two weeks of work to make it work outside
> of Solaris and to make it more robust. Then you might as well rewrite it in
> a more portable and robust language.

Agreed, if it has to be gone over in that much detail, conversion to perl
might not be a bad idea. I still say it'd have to be C eventually, but
it'd be good to use something more concise until all the design issues
are shaken out.

regards, tom lane

>> I thought we pretty much had agreement that space reservation was not
>> a good solution to anything, although I admit I'm not quite clear on
>> what alternative was being proposed.
>
> Maybe I miss something, but space reservation was selected as a best
> way. Please, Could you point me related mailing thread?

Hmm, maybe I misrepresented it slightly. But I did find these:

http://archives.postgresql.org/message-id/49425D07.6030701@enterprisedb.com
http://archives.postgresql.org/message-id/4948D567.4060908@enterprisedb.com

...and just today:
http://archives.postgresql.org/message-id/497F5B7D.90904@enterprisedb.com

...which I think are at least an argument against this patch, if not a
total refutation of the concept of space reservation in general.

Personally, I think you're both barking up the wrong tree. If the
amount of space that we need to reserve is more complicated than X
bytes per page or per tuple, then your approach will fail. But it
doesn't have to be a lot more complicated than that before Heikki's
approach of back-patching the space reservation code will fail too. I
foresee vehement arguments about whether the space reservation code
for feature Y is too complex to back-patch, with the argument going
something like this:

UIP Dude: If we don't back-patch this code, we won't be able to
upgrade-in-place for this release. That's a huge problem, man!
Conservative Guy: Yeah, but if we do, and we fry somebody's database,
that's a lot worse than not being able to upgrade-in-place.
UIP Dude: I don't think that's going to happen. This code is pretty solid.
Conservative Guy: You can't prove it doesn't have bugs.

Conservative Guy has a point, but the real issue is that UIP Dude is
making extra work for himself by splitting the work of upgrading the
system between release N and release N+1. Sure, with enough work, it
will be possible to test the back branch enough to convince yourself
that there are no horrible bugs, and it will also be possible to test
that CVS HEAD can complete the upgrade, but now you have to
exhaustively test TWO releases both independently and for
compatibility with each other. It's not my project, but I'd rather
walk through fire ants. I just can't imagine that it will be easier
to change the behavior of the old release to be compatible with what
the new release needs than visca versa. Even if the new release has
to reshuffle data between pages or stand on its head and wrestle an
epiletic porcupine, I still think it will not be as bad as putting the
space reservation code in the old release and the new page format in
the new release.

...Robert

> for getting 8.4->8.5 upgrade into place. You should first work on an
> update process that supports catalog changes, and get that committed.
> Once that's in place you'll have enough political capital to prevent
> changes in user data layout, and then we'd start to think about schemes
> like space reservation that could relax that ground rule.

I agree that catalog changes are the best way to start. And I also
think it's important that this logic get written in C, primarily so
that it gets deeply integrated into core. When a patch submitter adds
a function like array_length and puts in the markup for the BKI, they
should also be responsible for adding the function to the list of
changes that the UIP code needs to make when upgrading from the
catversion of the previous release to the current catversion. If
upgrade in place consists of Zdenek running around during every beta
cycle trying to figure out everything that happened to get done during
that dev cycle, it's not going to work (especially if Zdenek gets hit
by a bus).

That implies a fairly robust and configurable system for adding to and
rewriting system catalogs, which today we haven't got. Once we do
have it, we can think about how to handle things like page layout
bumps and toast chunk changes. Frankly, with things as they are
today, there's no way I'm using pg_upgrade on my production system.
Even if the whole thing gets rewritten in perl, it's an ex post facto
attempt to catch up with a gazillion changes that got made without any
regard to whether they'd work with pg_upgrade. It seems very
difficult to be confident that this will be 100% correct...

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> That implies a fairly robust and configurable system for adding to and
> rewriting system catalogs, which today we haven't got.

And we won't ever have, because it's unnecessary and would be impossibly
complex. We know how to do the catalog update: basically, dump, initdb,
reload, then move the data in. There are some corner case issues like
how to preserve toast table OIDs, but the idea that we are going to
invent a special process for each catalog change is just not reasonable.

regards, tom lane

On Tue, 27 Jan 2009, Robert Haas wrote:

> That implies a fairly robust and configurable system for adding to and
> rewriting system catalogs, which today we haven't got.

Right, and designing such a system requires a fairly deep analysis of the
catalog changes that have historically shown up, to make sure you've got
something that will handle them. The first step here that bears some
immediate fruit would be to sit down and analyze everything that changed
in the catalog from 8.3 to 8.4. Then you can do two things. You can
confirm whether the path used by pg_upgrade really can be expected to
work. You could also then talk intelligently about what a design for a
catalog rewriting system would look like, using "something that could have
handled all of these" as your spec. Then it's onto worrying about page
format stuff, which you can already get a handle on by looking at 8.2->8.3
under the same lens.

Tom even generated a helpful starting list of things to chase down a few
months ago:
http://archives.postgresql.org/pgsql-hackers/2008-09/msg00451.php

As someone who has been investigating this particular path, the part that
troubles me is that you'd have to pushing a new and completely
non-glamorous overhead burden ("create an in-place implementation for your
catalog change") on people who don't necessarily care about that
particular goal, which would create considerable new friction for patch
submission that required a catversion bump. Not the sort of boring work
volunteers on a project are traditionally good at, and you can expect any
developers who aren't managing databases too large to dump/reload to
scream about how it will slow the advance of the project.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> That implies a fairly robust and configurable system for adding to and
>> rewriting system catalogs, which today we haven't got.
>
> And we won't ever have, because it's unnecessary and would be impossibly
> complex. We know how to do the catalog update: basically, dump, initdb,
> reload, then move the data in. There are some corner case issues like
> how to preserve toast table OIDs, but the idea that we are going to
> invent a special process for each catalog change is just not reasonable.

Right, the dump+initdb+reload approach works quite well in both
pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is
supporting dropped columns, and maybe there's something else, but it's
fairly robust and works across any versions.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Heikki Linnakangas píše v st 28. 01. 2009 v 09:24 +0200:
> Tom Lane wrote:
> > Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> >> That implies a fairly robust and configurable system for adding to and
> >> rewriting system catalogs, which today we haven't got.
> >
> > And we won't ever have, because it's unnecessary and would be impossibly
> > complex. We know how to do the catalog update: basically, dump, initdb,
> > reload, then move the data in. There are some corner case issues like
> > how to preserve toast table OIDs, but the idea that we are going to
> > invent a special process for each catalog change is just not reasonable.
>
> Right, the dump+initdb+reload approach works quite well in both
> pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is
> supporting dropped columns, and maybe there's something else, but it's
> fairly robust and works across any versions.

It works but it is not ideal. Supporting dropped column requires lot of
magic which probably will decrease robustness. When you have a
tablespace then there is another shuffling magic which does not seems
like something robust.

And very important thing is that you need old version of postgreSQL
installed, which is something what packagers does not want. Look on
Oracle how does it.

I have idea how to do it without old PostgreSQL version and with
bootstrap process extension which should not be invasive and easily
maintainable. I will send idea latter ... stay tuned ;-)

Zdenek

* Zdenek Kotala (Zdenek(dot)Kotala(at)Sun(dot)COM) wrote:
> And very important thing is that you need old version of postgreSQL
> installed, which is something what packagers does not want. Look on
> Oracle how does it.

Just as a counter-point, Debian handles multiple concurrently installed
versions of PostgreSQL just fine, in large part to specifically deal
with the smooth migration challenge (though also because we realize
people may want to continue using the old version while others may want
to install the new version).

Not sure if that's something the community wants to encourage other
packagers to do or if we should look at making it easier to do, but it's
at least possible and has been done for a pretty large distribution.

Thanks,

Stephen

Zdenek Kotala <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:
> Heikki Linnakangas pe v st 28. 01. 2009 v 09:24 +0200:
>> Right, the dump+initdb+reload approach works quite well in both
>> pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is
>> supporting dropped columns, and maybe there's something else, but it's
>> fairly robust and works across any versions.

> It works but it is not ideal.

Sure, but the other way is just a complete non-starter. It's enormous
amounts of work even for simple changes, and it's not hard to think of
potentially-desirable catalog rearrangements for which it would be
completely unworkable.

> I have idea how to do it without old PostgreSQL version and with
> bootstrap process extension which should not be invasive and easily
> maintainable. I will send idea latter ... stay tuned ;-)

New ideas welcome of course.

regards, tom lane

Stephen Frost <sfrost(at)snowman(dot)net> writes:
> * Zdenek Kotala (Zdenek(dot)Kotala(at)Sun(dot)COM) wrote:
>> And very important thing is that you need old version of postgreSQL
>> installed, which is something what packagers does not want. Look on
>> Oracle how does it.

> Just as a counter-point, Debian handles multiple concurrently installed
> versions of PostgreSQL just fine, in large part to specifically deal
> with the smooth migration challenge (though also because we realize
> people may want to continue using the old version while others may want
> to install the new version).

> Not sure if that's something the community wants to encourage other
> packagers to do or if we should look at making it easier to do, but it's
> at least possible and has been done for a pretty large distribution.

The Red Hat/Fedora brigade has also been thinking seriously about that,
though we've not gotten further than thinking yet. Of course, if
pg_upgrade becomes a reality we'd likely stop thinking about it.

IMHO, it would not by any means be a disaster for pg_upgrade to require
a copy of the older-version postmaster. The way I'd foresee packaging
it is to build a separate postgresql-upgrade RPM containing pg_upgrade
itself and a version-N-minus-1 postmaster that gets installed in a
nonstandard location (that pg_upgrade knows about). After you've
finished the upgrade you can remove that RPM and get the extra disk
space back. Most of the possible alternatives mean a *permanent*
disk space hit, because the postmaster will have to contain one-time-use
upgrade code that can't be dropped afterwards.

regards, tom lane

On Wed, Jan 28, 2009 at 10:52 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Zdenek Kotala <Zdenek(dot)Kotala(at)Sun(dot)COM> writes:
>> Heikki Linnakangas píše v st 28. 01. 2009 v 09:24 +0200:
>>> Right, the dump+initdb+reload approach works quite well in both
>>> pg_upgrade and pg-migrator. I believe the biggest issue with that ATM is
>>> supporting dropped columns, and maybe there's something else, but it's
>>> fairly robust and works across any versions.
>
>> It works but it is not ideal.
>
> Sure, but the other way is just a complete non-starter. It's enormous
> amounts of work even for simple changes, and it's not hard to think of
> potentially-desirable catalog rearrangements for which it would be
> completely unworkable.

Well, the goal of coding is to make such things easier. Already the
solution you're advocating has one huge wart: the need to represent
dropped columns in pg_dump output. It seems optimistic to assume that
there won't be any more, and it seems possible that some change might
happen that requires the wart without the necessary wart actually
getting inserted. We'll end up testing after the fact to see whether
in-place upgrade has gotten broken, and given that no one was under
any pressure to give it some forethought in advance, the answer will
probably be "yes", at least sometimes...

I'm not dead set against doing it this way, I'm just not sold on it.

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Wed, Jan 28, 2009 at 10:52 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Sure, but the other way is just a complete non-starter.

> Well, the goal of coding is to make such things easier. Already the
> solution you're advocating has one huge wart: the need to represent
> dropped columns in pg_dump output.

Dropped columns are certainly an issue, and TOAST pointers are another,
but they seem to me to be soluble with relatively limited impact
(I don't think pg_dump necessarily needs to be involved; rather I'd
imagine pg_upgrade itself adjusting the new catalog entries after it's
used pg_dump to do most of the work). Trying to do catalog upgrade
in-place is going to be a complete mess. I'd be interested to know,
for example, how you imagine rearranging the contents of pg_class would
work. You don't get to modify pg_class if you can't even find it, which
you can't because you can't read it. And in the time it takes you to
think of an approach for that, I will be able to think of a dozen more
that are all equally nasty. There are probably some thousands of places
in the backend where we expect the system catalogs to have layout
matching what the code expects.

regards, tom lane

Tom Lane píše v st 28. 01. 2009 v 14:06 -0500:

> Trying to do catalog upgrade
> in-place is going to be a complete mess. I'd be interested to know,
> for example, how you imagine rearranging the contents of pg_class would
> work. You don't get to modify pg_class if you can't even find it, which
> you can't because you can't read it.

It is relatively easy. There are very high design:

1) upgrade move old catalog table into save location. We can create for
example special tablespace for it

2) bootstrap creates catalog + create records for old catalog structure

3) data will be copied like

INSERT INTO pg_class select from pg_upgrade.pg_class where oid>16...

In reality it will be more complicated command but pure SQL.

Zdenek

On Wed, Jan 28, 2009 at 5:00 PM, Zdenek Kotala <Zdenek(dot)Kotala(at)sun(dot)com> wrote:
> Tom Lane píše v st 28. 01. 2009 v 14:06 -0500:
>> Trying to do catalog upgrade
>> in-place is going to be a complete mess. I'd be interested to know,
>> for example, how you imagine rearranging the contents of pg_class would
>> work. You don't get to modify pg_class if you can't even find it, which
>> you can't because you can't read it.
>
> It is relatively easy. There are very high design:
>
> 1) upgrade move old catalog table into save location. We can create for
> example special tablespace for it
>
> 2) bootstrap creates catalog + create records for old catalog structure
>
> 3) data will be copied like
>
> INSERT INTO pg_class select from pg_upgrade.pg_class where oid>16...
>
> In reality it will be more complicated command but pure SQL.

I really like this idea, assuming I understand it. Basically, I think
you're proposing that we move the old system catalogs out of the way,
bootstrap a new catalog, and then using SQL (running inside a
standalone backend?) to migrate data from the old catalog to the new
one. That sounds really good to me. Even a relatively complicated
catalog reorganization should be able to be handled this way without
too much work or a lot of special-purpose code.

...Robert

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> I really like this idea, assuming I understand it. Basically, I think
> you're proposing that we move the old system catalogs out of the way,
> bootstrap a new catalog, and then using SQL (running inside a
> standalone backend?) to migrate data from the old catalog to the new
> one. That sounds really good to me. Even a relatively complicated
> catalog reorganization should be able to be handled this way without
> too much work or a lot of special-purpose code.

Well, what it really means is that all the special-purpose conversion
code is in SQL instead of C. Which is sort of good as long as whatever
transformation you have in mind can be done easily in SQL. (Good luck
if you need to control the OIDs of the inserted rows, for instance.
And I *really* want to see Zdenek handle conversion of stored-rule query
trees in SQL...) But far more importantly, it doesn't fix the problem
that you have to write conversion code in the first place.

The appeal of the pg_dump approach is that it will automatically handle
everything that there exists a plain-SQL representation for, which is to
say darn near everything. We will need special purpose code to deal
with the dropped-column and TOAST-oid issues, but that can probably be
written in SQL if it makes anyone feel better to do so ;-). The more
important point is that once we're done with those two issues, we're
done, and will stay done for the foreseeable future (at least with
respect to catalog upgrades).

I am not sure why everyone is so hot to create a conversion path that
guarantees extra maintenance pain for every future catalog
reorganization, when it would be no more complex to create one without
such a burden.

regards, tom lane

On Wed, Jan 28, 2009 at 6:05 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Well, what it really means is that all the special-purpose conversion
> code is in SQL instead of C. Which is sort of good as long as whatever
> transformation you have in mind can be done easily in SQL. (Good luck
> if you need to control the OIDs of the inserted rows, for instance.
> And I *really* want to see Zdenek handle conversion of stored-rule query
> trees in SQL...) But far more importantly, it doesn't fix the problem
> that you have to write conversion code in the first place.
>
> The appeal of the pg_dump approach is that it will automatically handle
> everything that there exists a plain-SQL representation for, which is to
> say darn near everything. We will need special purpose code to deal
> with the dropped-column and TOAST-oid issues, but that can probably be
> written in SQL if it makes anyone feel better to do so ;-). The more
> important point is that once we're done with those two issues, we're
> done, and will stay done for the foreseeable future (at least with
> respect to catalog upgrades).
>
> I am not sure why everyone is so hot to create a conversion path that
> guarantees extra maintenance pain for every future catalog
> reorganization, when it would be no more complex to create one without
> such a burden.

Well, I don't personally believe that there will be no extra
maintenance pain associated with the pg_dump approach. In fact, the
extra maintenance pain will be exactly proportional to the difference
between the "darn near everything" that it handles and "everything".
Basically, every time you invent a feature that can do things to a
system catalog that aren't visible at the SQL-level, you're going to
experience the searing pain of having to invent SQL-ish syntax that
can be dumped-and-restored without losing that mysterious system
catalog magic.

The first problem with that is that it is really ugly. Full stop.

The second problem with that is that you are relying on your ability
to translate tuples in a database table into text format (and not,
mind you, the same text format that we normally use to back up and
restore databases, but some modified text format with special hacks
that are only used when we need to represent things like dropped
columns) and then to translate that text format back into a set of new
tables that are semantically identical to the original ones in every
particular. To my way of thinking, this is a Rube Goldberg machine.
Transforming one set of tuples into a slightly different set of tuples
using SQL seems way less prone to errors and omissions.

I also kind of think that it might open the door to using the system
catalogs to indicate things like "the earliest page version that
appears in relation X". There's no joy in inventing some kind of
pg_dump syntax for that sort of thing just so that you can set it
properly when someone does an in-place upgrade. It's useless for
normal operation since any NON-uip relations will always have whatever
the current version is in that field, and it feels wrong for users to
have the ability to fiddle with that value via SQL anyway.

With respect to the specific problems you mention, OIDs are definitely
an issue but do you think that's an insurmountable obstacle? Seems
like we should be able to find a hammer large enough to solve that
problem. As for rules, just because the core of the engine is written
in SQL doesn't mean that it can't make outcalls to C functions; we
already have an interface for that. It is better than writing the
whole thing in C, to be sure...

I don't know, I'm not completely sure how hard this will be, or which
approach is better. But it sounds to me like this has potential, if
done right.

...Robert

Tom Lane wrote:
> The appeal of the pg_dump approach is that it will automatically handle
> everything that there exists a plain-SQL representation for, which is to
> say darn near everything. We will need special purpose code to deal
> with the dropped-column and TOAST-oid issues, but that can probably be
> written in SQL if it makes anyone feel better to do so ;-). The more
> important point is that once we're done with those two issues, we're
> done, and will stay done for the foreseeable future (at least with
> respect to catalog upgrades).
>
> I am not sure why everyone is so hot to create a conversion path that
> guarantees extra maintenance pain for every future catalog
> reorganization, when it would be no more complex to create one without
> such a burden.

I am stumped as well. In the 12 years I have been involved, there are
perhaps five issues that the original pg_upgrade written in 1998 didn't
handle, and mostly handles now. Considering the number of catalog
changes since 1998, the ratio is enormous.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Thursday 29 January 2009 01:05:07 Tom Lane wrote:
> The appeal of the pg_dump approach is that it will automatically handle
> everything that there exists a plain-SQL representation for, which is to
> say darn near everything.  We will need special purpose code to deal
> with the dropped-column and TOAST-oid issues, but that can probably be
> written in SQL if it makes anyone feel better to do so ;-).

Dropped columns are certainly solvable. You just include the dropped column
in the dumped CREATE TABLE statement and then issue a DROP COLUMN statement
afterwards. You might have to do some extra work if there is a name conflict
between a dropped and a later-added column, but that shouldn't be so hard.
All you need is the space, not the column names, after all.

Hm the previous proposal was to add syntax to create table to create
placeholder columns of specified width.

On the one hand the special syntax is less kludgy but on the other
hand keeping all the compatibility code in pg_dump is attractive. Net
I think prefer your solution.

I don't think name conflicts are a problem either since the original
name is long gone anyways. All you need to do is find a type with a
matching width and make a an arbitrary dummy name.

--
Greg

On 29 Jan 2009, at 09:33, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:

> On Thursday 29 January 2009 01:05:07 Tom Lane wrote:
>> The appeal of the pg_dump approach is that it will automatically
>> handle
>> everything that there exists a plain-SQL representation for, which
>> is to
>> say darn near everything. We will need special purpose code to deal
>> with the dropped-column and TOAST-oid issues, but that can probably
>> be
>> written in SQL if it makes anyone feel better to do so ;-).
>
> Dropped columns are certainly solvable. You just include the
> dropped column
> in the dumped CREATE TABLE statement and then issue a DROP COLUMN
> statement
> afterwards. You might have to do some extra work if there is a name
> conflict
> between a dropped and a later-added column, but that shouldn't be so
> hard.
> All you need is the space, not the column names, after all.
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers

Bruce Momjian píše v st 28. 01. 2009 v 23:19 -0500:

> I am stumped as well. In the 12 years I have been involved, there are
> perhaps five issues that the original pg_upgrade written in 1998 didn't
> handle, and mostly handles now. Considering the number of catalog
> changes since 1998, the ratio is enormous.

You don't need convert all of catalogs and only few of them are usually
modified in each version (like pg_class, pg_type...).

Zdenek

Tom Lane píše v st 28. 01. 2009 v 18:05 -0500:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> > I really like this idea, assuming I understand it. Basically, I think
> > you're proposing that we move the old system catalogs out of the way,
> > bootstrap a new catalog, and then using SQL (running inside a
> > standalone backend?) to migrate data from the old catalog to the new
> > one. That sounds really good to me. Even a relatively complicated
> > catalog reorganization should be able to be handled this way without
> > too much work or a lot of special-purpose code.
>
> Well, what it really means is that all the special-purpose conversion
> code is in SQL instead of C. Which is sort of good as long as whatever
> transformation you have in mind can be done easily in SQL. (Good luck
> if you need to control the OIDs of the inserted rows, for instance.
> And I *really* want to see Zdenek handle conversion of stored-rule query
> trees in SQL...) But far more importantly, it doesn't fix the problem
> that you have to write conversion code in the first place.

OK, it was too optimistic to take only SQL. But all problem are should
be fixable with C procedure and storing plain ascii for rewrite rules
and reparse it again is not so big problem I guess.

> The appeal of the pg_dump approach is that it will automatically handle
> everything that there exists a plain-SQL representation for, which is to
> say darn near everything. We will need special purpose code to deal
> with the dropped-column and TOAST-oid issues, but that can probably be
> written in SQL if it makes anyone feel better to do so ;-). The more
> important point is that once we're done with those two issues, we're
> done, and will stay done for the foreseeable future (at least with
> respect to catalog upgrades).

One one side I was convinced to select on read conversion, because it is
simple and self contained. And now when I try to do catalog upgrade to
same way, the another solution is preferred which needs many
unsystematic hacks into SQL syntax, pg_dump and so on.

> I am not sure why everyone is so hot to create a conversion path that
> guarantees extra maintenance pain for every future catalog
> reorganization, when it would be no more complex to create one without
> such a burden.

It is not only about catalog. Currently you have FSM which works fine if
it is empty, but when somebody invent any new fork which will needs have
complete data, then you need to handle it separately anyway.

By main opinion to extend parser and add another complexity into already
complex part instead of writing store procedure for each affected
catalog which convert its data.

Both version will work and I think that maintenance pain will not be so
big problem. I think the amount of hot modified catalogs is not big.

Advantage of direct conversion are:
1) no pg_dump
2) no old database server
3) no shuffling with files/tablespaces
4) it should be relatively robust and easy to use

Thats what I can say now It needs more analysis.

Zdenek

Zdenek Kotala wrote:
> The current project is not in good shape. Feature freeze is coming and
> nothing is committed. Currently there are three patches in the game:
>
> 1) Space reservation
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00886.php
> http://archives.postgresql.org/pgsql-hackers/2009-01/msg02031.php
>
> This patch is mandatory for page online conversion and MUST TO be part
> of postgreSQL 8.4. if not ... then we will be at the beginning next
> year.
>
> I sent updated version today.

I think we have decided that any system we come up with now for space
reservation will be inadequate, so we will just backpatch any needed
space reservation code. Fortunately space expansion is a rare event.

And we decided we don't need a pg_class column because a custom table
will do just as well in reporting the tables that have been prepared
with reserved space.

> 2) pg_upgrade.sh
> http://archives.postgresql.org/pgsql-hackers/2008-12/msg00248.php
>
> Pg_upgrade.sh is shell script for catalog conversion. It works for
> 8.3->8.4 upgrade. It will be useful while we will not have better
> solution. Disadvantage is that it is korn shell script. The idea is to
> rewrite it in PERL which is more portable, but I'm not PERL expert and
> currently there is no workable solution.
>
> And what is deadline for it? We can delivery it later with 8.4.1, but
> best time for testing is during betas and RC period.

Now that pg_migrator is BSD licensed, and already in C, I am going to
spend my time trying to improve pg_migrator for 8.4:

http://pgfoundry.org/projects/pg-migrator/

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Bruce Momjian wrote:
> Now that pg_migrator is BSD licensed, and already in C, I am going to
> spend my time trying to improve pg_migrator for 8.4:
>
> http://pgfoundry.org/projects/pg-migrator/

What is the plan now? Get pg_upgrade working, get pg_migrator working,
ship pg_migrator in core or separately? Is there any essential
functionality that we need to get into the server code before release?
Should we try to get dropped columns working? It's quite late to be
wondering about this, so unless we get a clear and definite plan this
week, I say we stop kidding ourselves and drop it.

Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > Now that pg_migrator is BSD licensed, and already in C, I am going to
> > spend my time trying to improve pg_migrator for 8.4:
> >
> > http://pgfoundry.org/projects/pg-migrator/
>
> What is the plan now? Get pg_upgrade working, get pg_migrator working,
> ship pg_migrator in core or separately? Is there any essential
> functionality that we need to get into the server code before release?
> Should we try to get dropped columns working? It's quite late to be
> wondering about this, so unless we get a clear and definite plan this
> week, I say we stop kidding ourselves and drop it.

Oh, a plan? ;-)

Basically I am trying to add functionality to the code and clean it up
so it is easier to maintain. You can grab the CVS to see my current
version:

http://pgfoundry.org/scm/?group_id=1000235

Everyone seemed to prefer a migration utility in C, and pg_migrator is
in C so I am working on that. It has a BSD license now so we could
include it if we wanted to.

I am working on pg_migrator TODO items. My current list is:

o Makefiles are not yet complete.
o need to check crc when we reading the pg_control file
o compare the pg_controls of old and new servers
o fix loaded tables with dropped columns; the dropped column location
is not
part of pg_dump; pg_attribute.attisdropped must be checked and a
replacement
table created and the column dropped
o must call vacuum freeze on system tables before clog is copied
o restore pg_database.datfrozenxid to their original values
o restore pg_class.relfrozenxid to their original values

I am not sure about the first two items, but I want to try to address
the other ones, though I am not sure how to handle the drop column case.

Once I am complete the existing TODOs I will start on testing and
getting more feedback on missing features.

If folks want to help out, please let me know.

I am not aware of any server changes needed for 8.3-8.4 migration.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

On Thursday 12 February 2009 05:15:02 Bruce Momjian wrote:
> I am working on pg_migrator TODO items. My current list is:
[long]

Can we clarify now that this is not going to ship with the PostgreSQL 8.4
tarball?

> I am not aware of any server changes needed for 8.3-8.4 migration.

OK, Zdenek, any concerns, or can we consider this chapter closed?

Peter Eisentraut wrote:
> On Thursday 12 February 2009 05:15:02 Bruce Momjian wrote:
> > I am working on pg_migrator TODO items. My current list is:
> [long]
>
> Can we clarify now that this is not going to ship with the PostgreSQL 8.4
> tarball?

That is not really my decision, but I am not going to advocate its
inclusion unless there is some unusual ground-swell of demand and others
are going to advocate its inclusion, not me.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +

Peter Eisentraut píše v čt 12. 02. 2009 v 15:30 +0200:

>
> > I am not aware of any server changes needed for 8.3-8.4 migration.
>
> OK, Zdenek, any concerns, or can we consider this chapter closed?

I think, that it is closed now. Space reservation will be backported if
we need it for 8.4->8.5. And I don't know about any "must" changes in
server for migration from 8.3->8.4.

Zdenek