Git migration timeline

Hi!

According to the decision at the developer meeting, the migration to
git should happen 17-20 Aug. Here's my proposed timeline. This will
obviously affect development work some, and since the original
timeline called for us having already released 9.0 buy then ;)

1. Tuesday evening, around 19:00 central european time, which is 17:00
GMT or 12:00 EST, I will freeze the current cvs repository. I will do
this by disabling committer login on that box, so please note that
this will also make it impossible for committers to do a "cvs update"
from the authenticated repository. The anonymous repository will of
course still be available.

2. Tuesday evening and wednesday morning (european time again, please
extrapolate as necessary yourself :P), I will convert the latest cvs
version including running all my comparison scripts. (this takes quite
a long time). once the conversion is done, I'll snapshot whatever we
have by then so we can back out to that.

3. I will then as soon as this is done run the script to remove
keywords from active branches.

4. At this point, I will push a copy of this to the
postgresql-migration.git project on git.postgresql.org, asking
everybody to test. Those of you around, please do so as quickly as you
can. It will also be made available to all committers at this time as
a read-only repository.

5. Once enough is verified, we'll open the repository up for commits.
I expect this to happen sometime on thursday probably, assuming there
were no issues.

6. Once things have been running for a few days, we'll replace the
postgresql.git project on git.postgresql.org with an automatic push of
the repository, renaming the old one to something like
postgresql-old.git for a while.

7. As things are done, I'll get the anoncvs mirror moving as soon as
possible. But I want to make sure that the git stuff is fully working
before we do that.

This means the following:
* If anybody is working on some specific patch they want to get in
soon, please try to do so before tuesday evening european time.

* If you are working off the old git tree, you are recommended to do a
pull sometime between that and thuersday. That'll make sure you have
the latest copy of the *old* git repository. You can then generate a
patch off that and submit it or re-apply to your old repository. Note
that the new repository will *NOT* be compatible with the old one, so
you'll need to do a manual move over. All commits will have new
SHA1'sl.

* Buildfarm coverage will obviously be weak until BF members have been
converted to run off git. Those running directly against anoncvs
should have a fair chance to still work on the new one once that's up,
but the rsync-the-cvs-repository method will no longer work.

Um, I'm sure there's more, but...
.
Are there any objections to this timeline, or will this work out
reasonably well for everybody?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> According to the decision at the developer meeting, the migration to
> git should happen 17-20 Aug. Here's my proposed timeline. This will
> obviously affect development work some, and since the original
> timeline called for us having already released 9.0 buy then ;)

Core is currently talking about exactly when we want to push out
9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has
to wait on that decision. I don't think we're going to be ready for it
to begin on Tuesday. Ignoring the exact timing, this plan sounds good
except for this bit:

> 1. Tuesday evening, around 19:00 central european time, which is 17:00
> GMT or 12:00 EST, I will freeze the current cvs repository. I will do
> this by disabling committer login on that box, so please note that
> this will also make it impossible for committers to do a "cvs update"
> from the authenticated repository.

That's not really going to do, especially since it will also lock out
"cvs log". I certainly want to do a final update and cvs2cl after the
last commits have happened, and I imagine other people will want that
too. If there were a way to make the repository read-only that would be
nice, but I think what we're going to need to settle for is relying on
the committers to not do any more commits after the specified time.

I can see the point of wanting to be dead certain the repository isn't
changing under you during the data migration. Perhaps we need an agreed
window between last call for commits and the actual lock-out.

regards, tom lane

* Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> [100816 09:58]:

> That's not really going to do, especially since it will also lock out
> "cvs log". I certainly want to do a final update and cvs2cl after the
> last commits have happened, and I imagine other people will want that
> too. If there were a way to make the repository read-only that would be
> nice, but I think what we're going to need to settle for is relying on
> the committers to not do any more commits after the specified time.
>
> I can see the point of wanting to be dead certain the repository isn't
> changing under you during the data migration. Perhaps we need an agreed
> window between last call for commits and the actual lock-out.

What about 2 minutes long for the repository to be "synced" to wherever
he wants to do the conversion from. There isn't any reason for the
repository to be locked while he does the conversion/checks, but he does
need a guaranteed consistent copy of the repository to work from.

I'ld suggest having the repository "locked" for long enough to:
1) Get a consistent rsync/snapshot/tarball of $CVSROOT
2) Send an email to committers saying CVS repository has been "taken",
don't commit/tag/branch on it anymore "please"

And then unlock it with the expectation that committers will continue in
their good stewardship of the CVS repository (in this case meaning not
make any changes to it anymore) as they are already doing.

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

On Mon, Aug 16, 2010 at 15:58, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> According to the decision at the developer meeting, the migration to
>> git should happen 17-20 Aug. Here's my proposed timeline. This will
>> obviously affect development work some, and since the original
>> timeline called for us having already released 9.0 buy then ;)
>
> Core is currently talking about exactly when we want to push out
> 9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has
> to wait on that decision.  I don't think we're going to be ready for it
> to begin on Tuesday.  Ignoring the exact timing, this plan sounds good
> except for this bit:

Ok. The sooner you can let me (and others) know, the better.

Note that if it gets pushed past the end of this week (for the whole
process), it's quite likely that the whole process will be stretched
out further. I'll have other committments I need to take care of
meanwhile then - this week i've set of a fair amount of dedicated time
to work on this. Next week, for example, almost all the work would
have to be done durning the evenings (european time) outside of work
hours, which obviously makes for a lot less available time.

>> 1. Tuesday evening, around 19:00 central european time, which is 17:00
>> GMT or 12:00 EST, I will freeze the current cvs repository. I will do
>> this by disabling committer login on that box, so please note that
>> this will also make it impossible for committers to do a "cvs update"
>> from the authenticated repository.
>
> That's not really going to do, especially since it will also lock out
> "cvs log".  I certainly want to do a final update and cvs2cl after the
> last commits have happened, and I imagine other people will want that
> too.  If there were a way to make the repository read-only that would be
> nice, but I think what we're going to need to settle for is relying on
> the committers to not do any more commits after the specified time.

You can still rsync the copy off the anoncvs server (or use the
anoncvs one). That one is an exact rsync of the main one, so I don't
see why something like "cvs2cl" wouldn't work off that one
automatically. I was just assuming every developer was already using
an rsynced copy for that kind of work :-)

I would like to make the repo readonly if I could, but I don't know a
reliable way of doing that with cvs. If someone can enlighten me on
one, that's obviously preferred.

> I can see the point of wanting to be dead certain the repository isn't
> changing under you during the data migration.  Perhaps we need an agreed
> window between last call for commits and the actual lock-out.

To prevent that, I just need to shut it down for 2 minutes to rsync
the latest changes off it and onto the new git box. Maybe that's how
we should do it.

This is a defense against committers forgetting about things and
accidentally committing to the repository *after* it's been
snapshotted. I guess we can just declare those as ignorant and throw
away those commits.. :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Mon, Aug 16, 2010 at 15:58, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> I can see the point of wanting to be dead certain the repository isn't
>> changing under you during the data migration. Perhaps we need an agreed
>> window between last call for commits and the actual lock-out.

> To prevent that, I just need to shut it down for 2 minutes to rsync
> the latest changes off it and onto the new git box. Maybe that's how
> we should do it.

That sounds like a reasonable scheme to me, especially since it leaves
the cvs repository functional. Dunno about you, but I want a fallback
plan in case this turns into a disaster ...

regards, tom lane

On Mon, Aug 16, 2010 at 9:58 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> 1. Tuesday evening, around 19:00 central european time, which is 17:00
>> GMT or 12:00 EST, I will freeze the current cvs repository. I will do
>> this by disabling committer login on that box, so please note that
>> this will also make it impossible for committers to do a "cvs update"
>> from the authenticated repository.
>
> That's not really going to do, especially since it will also lock out
> "cvs log".  I certainly want to do a final update and cvs2cl after the
> last commits have happened, and I imagine other people will want that
> too.  If there were a way to make the repository read-only that would be
> nice, but I think what we're going to need to settle for is relying on
> the committers to not do any more commits after the specified time.
>
> I can see the point of wanting to be dead certain the repository isn't
> changing under you during the data migration.  Perhaps we need an agreed
> window between last call for commits and the actual lock-out.

I would feel a lot more comfortable if the CVS repository was locked
down before we start running the migration. Any accidental commits
would be a nuisance. However, giving a window between last call for
commits and actual lock-out seems reasonable.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Mon, Aug 16, 2010 at 10:15 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Mon, Aug 16, 2010 at 15:58, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> I can see the point of wanting to be dead certain the repository isn't
>>> changing under you during the data migration.  Perhaps we need an agreed
>>> window between last call for commits and the actual lock-out.
>
>> To prevent that, I just need to shut it down for 2 minutes to rsync
>> the latest changes off it and onto the new git box. Maybe that's how
>> we should do it.
>
> That sounds like a reasonable scheme to me, especially since it leaves
> the cvs repository functional.  Dunno about you, but I want a fallback
> plan in case this turns into a disaster ...

I don't think anybody proposed permanently deleting the CVS repository.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Mon, Aug 16, 2010 at 16:15, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Mon, Aug 16, 2010 at 15:58, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> I can see the point of wanting to be dead certain the repository isn't
>>> changing under you during the data migration.  Perhaps we need an agreed
>>> window between last call for commits and the actual lock-out.
>
>> To prevent that, I just need to shut it down for 2 minutes to rsync
>> the latest changes off it and onto the new git box. Maybe that's how
>> we should do it.
>
> That sounds like a reasonable scheme to me, especially since it leaves
> the cvs repository functional.  Dunno about you, but I want a fallback
> plan in case this turns into a disaster ...

Oh, I had no intention whatsoever of *removing* it. Just disabling login to it.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Mon, 2010-08-16 at 16:19 +0200, Magnus Hagander wrote:
> On Mon, Aug 16, 2010 at 16:15, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > Magnus Hagander <magnus(at)hagander(dot)net> writes:
> >> On Mon, Aug 16, 2010 at 15:58, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >>> I can see the point of wanting to be dead certain the repository isn't
> >>> changing under you during the data migration. Perhaps we need an agreed
> >>> window between last call for commits and the actual lock-out.
> >
> >> To prevent that, I just need to shut it down for 2 minutes to rsync
> >> the latest changes off it and onto the new git box. Maybe that's how
> >> we should do it.
> >
> > That sounds like a reasonable scheme to me, especially since it leaves
> > the cvs repository functional. Dunno about you, but I want a fallback
> > plan in case this turns into a disaster ...
>
> Oh, I had no intention whatsoever of *removing* it. Just disabling login to it.

+1

JD

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

I wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> According to the decision at the developer meeting, the migration to
>> git should happen 17-20 Aug. Here's my proposed timeline. This will
>> obviously affect development work some, and since the original
>> timeline called for us having already released 9.0 buy then ;)

> Core is currently talking about exactly when we want to push out
> 9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has
> to wait on that decision. I don't think we're going to be ready for it
> to begin on Tuesday.

The current feeling among core seems to be that we should allow the git
conversion to proceed according to Magnus' proposed schedule. This
would mean that we will *not* be able to wrap either a 9.0 update or
9.1alpha1 releases this week. What we'd probably try to do instead is
wrap 9.1alpha1 early next week, as the first attempt to generate
tarballs from the git repository, and then wrap 9.0beta5 (or rc1) later
in the week. This slips the 9.0 schedule an extra week, but considering
that nobody seems to be doing anything about the open 9.0 issues, that
was likely to happen anyway.

If anybody's really unhappy with this timeline, speak up now.

regards, tom lane

On Mon, Aug 16, 2010 at 1:47 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> According to the decision at the developer meeting, the migration to
>>> git should happen 17-20 Aug. Here's my proposed timeline. This will
>>> obviously affect development work some, and since the original
>>> timeline called for us having already released 9.0 buy then ;)
>
>> Core is currently talking about exactly when we want to push out
>> 9.0beta5 and/or 9.1alpha1, and the scheduling of the git transition has
>> to wait on that decision.  I don't think we're going to be ready for it
>> to begin on Tuesday.
>
> The current feeling among core seems to be that we should allow the git
> conversion to proceed according to Magnus' proposed schedule.  This
> would mean that we will *not* be able to wrap either a 9.0 update or
> 9.1alpha1 releases this week.  What we'd probably try to do instead is
> wrap 9.1alpha1 early next week, as the first attempt to generate
> tarballs from the git repository, and then wrap 9.0beta5 (or rc1) later
> in the week.  This slips the 9.0 schedule an extra week, but considering
> that nobody seems to be doing anything about the open 9.0 issues, that
> was likely to happen anyway.
>
> If anybody's really unhappy with this timeline, speak up now.

That sounds good, except for the part about nobody doing anything
about the 9.0 open issues. Those issues are:

* Backup procedure is wrong? - Nobody's been able to clearly
articulate what the problem is, and according to Fujii Masao it's been
this way since 8.2.

* Walreceiver crashes in AIX - The person who originally reported this
problem has been slow to get back to us with the information needed to
figure out what is going on. It would be helpful if someone else
could test to see if this a general problem with AIX, or something
specific to the reporter's installation. Or if someone wants to
provide me with a login...

* BUG #5595: Documentation is not installs from VPATH build. - I'm not
sure this is really a release-blocker, but either way, I have been
assuming it's Peter's issue to fix.

* trace_recovery_messages inconsistency - This one seems like a simple
case of deciding what makes most sense, and doing it. We should
definitely fix this before the wrap.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> wrote:

> That sounds good, except for the part about nobody doing anything
> about the 9.0 open issues. Those issues are:
>
> [four issues listed]

Nobody responded when I asked about this recently, but shouldn't
that list include "BUG #5607: memmory leak in ecpg"? We have a
patch from Zoltán Böszörményi from before this bug report which
seems to address the issue and which Michael Meskes said "Feel free
to apply".

We don't want to ship 9.0 with known memory leaks, do we?

-Kevin

"Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> writes:
> Nobody responded when I asked about this recently, but shouldn't
> that list include "BUG #5607: memmory leak in ecpg"? We have a
> patch from Zoltn Bszrmnyi from before this bug report which
> seems to address the issue and which Michael Meskes said "Feel free
> to apply".

> We don't want to ship 9.0 with known memory leaks, do we?

Better a memory leak than broken ecpg ;-). Nobody except Michael
is terribly comfortable with that code, so we'd all rather wait for
him to review and apply the patch.

More generally, pre-existing bugs have never been considered release
stoppers. At this point what we would block the release for is *new*
bugs in 9.0. (An exception to that general rule is pre-existing bugs
that would require an initdb to fix; but this one isn't that either.)

regards, tom lane

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> That sounds good, except for the part about nobody doing anything
> about the 9.0 open issues. Those issues are:

> * Backup procedure is wrong? - Nobody's been able to clearly
> articulate what the problem is, and according to Fujii Masao it's been
> this way since 8.2.

Just because it's old doesn't mean it's not a bug ;-). Normally I would
say that a pre-existing documentation problem isn't a release blocker,
but in this case that procedure is likely to become of interest to a
whole lot more people than it was before, thanks to SR/HS. So we need
to understand whether there's a problem and fix it.

> * Walreceiver crashes in AIX - The person who originally reported this
> problem has been slow to get back to us with the information needed to
> figure out what is going on.

Yes. If he doesn't provide adequate data, and we can't reproduce it
elsewhere, we should not consider this a release blocker.

> * BUG #5595: Documentation is not installs from VPATH build. - I'm not
> sure this is really a release-blocker, but either way, I have been
> assuming it's Peter's issue to fix.

I believe this is a regression from prior branches, probably caused by
the switch away from distributing the prebuilt docs as sub-tarballs.
As such, it's at least a candidate release-blocker.

> * trace_recovery_messages inconsistency - This one seems like a simple
> case of deciding what makes most sense, and doing it. We should
> definitely fix this before the wrap.

Agreed, it's a matter of getting some consensus.

regards, tom lane

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> According to the decision at the developer meeting, the migration to
> git should happen 17-20 Aug. Here's my proposed timeline. This will
> obviously affect development work some, and since the original
> timeline called for us having already released 9.0 buy then ;)

> 1. Tuesday evening, around 19:00 central european time, which is 17:00
> GMT or 12:00 EST, I will freeze the current cvs repository. I will do
> this by disabling committer login on that box, so please note that
> this will also make it impossible for committers to do a "cvs update"
> from the authenticated repository.

So, per discussion, I'd like to suggest that we have a "quiet time" for
say three hours before the repository is closed off, to give interested
committers a chance to capture final snapshots of the current
repository.

IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17.

regards, tom lane

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > According to the decision at the developer meeting, the migration to
> > git should happen 17-20 Aug. Here's my proposed timeline. This will
> > obviously affect development work some, and since the original
> > timeline called for us having already released 9.0 buy then ;)
>
> > 1. Tuesday evening, around 19:00 central european time, which is 17:00
> > GMT or 12:00 EST, I will freeze the current cvs repository. I will do
> > this by disabling committer login on that box, so please note that
> > this will also make it impossible for committers to do a "cvs update"
> > from the authenticated repository.
>
> So, per discussion, I'd like to suggest that we have a "quiet time" for
> say three hours before the repository is closed off, to give interested
> committers a chance to capture final snapshots of the current
> repository.
>
> IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17.

OK, would someone please send an email to hackers at that time as a
reminder? I might not be available then.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://enterprisedb.com

+ It's impossible for everything to be true. +

On Tue, Aug 17, 2010 at 1:59 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> According to the decision at the developer meeting, the migration to
>> git should happen 17-20 Aug. Here's my proposed timeline. This will
>> obviously affect development work some, and since the original
>> timeline called for us having already released 9.0 buy then ;)
>
>> 1. Tuesday evening, around 19:00 central european time, which is 17:00
>> GMT or 12:00 EST, I will freeze the current cvs repository. I will do
>> this by disabling committer login on that box, so please note that
>> this will also make it impossible for committers to do a "cvs update"
>> from the authenticated repository.
>
> So, per discussion, I'd like to suggest that we have a "quiet time" for
> say three hours before the repository is closed off, to give interested
> committers a chance to capture final snapshots of the current
> repository.
>
> IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17.

Works for me. I'll send out status reports as I change things, so be
sure to check the latest on -hackers if something you thought would
work doesn't ;)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Mon, Aug 16, 2010 at 02:50:35PM -0400, Tom Lane wrote:
> Better a memory leak than broken ecpg ;-). Nobody except Michael
> is terribly comfortable with that code, so we'd all rather wait for
> him to review and apply the patch.

This patch was small enough that I felt good about it without having a chance
to test it. Anyway, I applied Zoltan's memleak patch to HEAD and 8.4.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
ICQ 179140304, AIM/Yahoo/Skype michaelmeskes, Jabber meskes(at)jabber(dot)org
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

On Tue, Aug 17, 2010 at 11:46 AM, Michael Meskes <meskes(at)postgresql(dot)org> wrote:
> On Mon, Aug 16, 2010 at 02:50:35PM -0400, Tom Lane wrote:
>> Better a memory leak than broken ecpg ;-).  Nobody except Michael
>> is terribly comfortable with that code, so we'd all rather wait for
>> him to review and apply the patch.
>
> This patch was small enough that I felt good about it without having a
> chance
> to test it. Anyway, I applied Zoltan's memleak patch to HEAD and 8.4.

What about 9.0?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 17, 2010 at 11:50:20AM +0200, Magnus Hagander wrote:
> What about 9.0?

Will come in a few minutes. I didn't have a checked out version of the 9.0
branch handy in my development environment. Regression test is currently
running.

Michael
--
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
ICQ 179140304, AIM/Yahoo/Skype michaelmeskes, Jabber meskes(at)jabber(dot)org
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

Bruce Momjian <bruce(at)momjian(dot)us> writes:
> Tom Lane wrote:
>> IOW, please no more CVS commits after 14:00 GMT tomorrow, Tuesday 8/17.

> OK, would someone please send an email to hackers at that time as a
> reminder? I might not be available then.

Here you go.

Per yesterday's discussion, access to the master PG CVS repository will
be shut off at approximately 17:00 GMT, three hours from now, to begin
the conversion to git. So that all interested people can capture final
snapshots of the repository for archival purposes, please do not make
any further commits into the CVS repository. Development will resume
once the git repository is up and validated.

regards, tom lane