Re: git: uh-oh

It appears that the git conversion of the CVS repository is seriously
screwed up. For example, if you look at this:

http://git.postgresql.org/gitweb?p=postgresql-migration.git;a=shortlog;h=refs/tags/REL8_3_10

The first few revs look OK, but the you get to this:

2010-02-28
PostgreSQL...
This commit was manufactured by cvs2svn to create branch REL8_3_STABLE

Prior to that commit, this history is nonsense - it appears to be the
history of our 9.0 development prior to that date. I would say we're
going back to good old CVS.

It's too bad that nobody noticed this sooner, but I'm glad I noticed
today rather than tomorrow.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> It appears that the git conversion of the CVS repository is seriously
> screwed up. For example, if you look at this:

Um ... Magnus has not given any report that he's finished running
the conversion. What exactly are you looking at?

> It's too bad that nobody noticed this sooner, but I'm glad I noticed
> today rather than tomorrow.

We're not going to start using the git repository until everyone is
satisfied it's OK, both as to current contents and history.

regards, tom lane

On Tue, Aug 17, 2010 at 21:16, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> It appears that the git conversion of the CVS repository is seriously
>> screwed up.  For example, if you look at this:
>
> Um ... Magnus has not given any report that he's finished running
> the conversion.  What exactly are you looking at?

That's the previous conversion. The one that we used to verify that
things looked ok. Seems nobody caught this :S

The new migration looks similarly weird.

Does anybody with some more git-fu have any clue how this can be?

The tip of every branch, and every single tag, all have the correct
data in them, but some revisions in between seem majorly confused.

>> It's too bad that nobody noticed this sooner, but I'm glad I noticed
>> today rather than tomorrow.
>
> We're not going to start using the git repository until everyone is
> satisfied it's OK, both as to current contents and history.

Yeah..

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> The tip of every branch, and every single tag, all have the correct
> data in them, but some revisions in between seem majorly confused.

It seems to me that what we'll need to do here is write a script to
look through the CVS history of each file and make sure that the
versions of that file which appear on each branch match the revs in
CVS in content, order, and the commit message associated with any
changes. However, that's not going to do get done today.

>>> It's too bad that nobody noticed this sooner, but I'm glad I noticed
>>> today rather than tomorrow.
>>
>> We're not going to start using the git repository until everyone is
>> satisfied it's OK, both as to current contents and history.

Duh. But obviously no one's checked that carefully enough up until now.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Tue, Aug 17, 2010 at 21:16, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Um ... Magnus has not given any report that he's finished running
>> the conversion. What exactly are you looking at?

> That's the previous conversion. The one that we used to verify that
> things looked ok. Seems nobody caught this :S

> The new migration looks similarly weird.

> Does anybody with some more git-fu have any clue how this can be?

I lack git-fu pretty completely, but I do have the CVS logs ;-).
It looks like some of these commits that are being ascribed to the
REL8_3_STABLE branch were actually only committed on HEAD. For
instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
and that's also shown here, but the HEAD commit shouldn't be.

I wonder whether the repository is completely OK and the problem
is that this webpage isn't filtering the commits correctly.

regards, tom lane

On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> The tip of every branch, and every single tag, all have the correct
>> data in them, but some revisions in between seem majorly confused.
>
> It seems to me that what we'll need to do here is write a script to
> look through the CVS history of each file and make sure that the
> versions of that file which appear on each branch match the revs in
> CVS in content, order, and the commit message associated with any
> changes.  However, that's not going to do get done today.

Yeah. Unless someone comes up with a good way to fix this, or even
better an explanation why it's actually ont broken and we're looking
at things wrong :D, I think we have no choice but aborting the
conversion for now and come back to it later.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Aug 17, 2010, at 2:29 PM, Magnus Hagander wrote:

> On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>> The tip of every branch, and every single tag, all have the correct
>>> data in them, but some revisions in between seem majorly confused.
>>
>> It seems to me that what we'll need to do here is write a script to
>> look through the CVS history of each file and make sure that the
>> versions of that file which appear on each branch match the revs in
>> CVS in content, order, and the commit message associated with any
>> changes. However, that's not going to do get done today.
>
> Yeah. Unless someone comes up with a good way to fix this, or even
> better an explanation why it's actually ont broken and we're looking
> at things wrong :D, I think we have no choice but aborting the
> conversion for now and come back to it later.

Can you post the cvs2svn command line used for conversion?

Regards,

David
--
David Christensen
End Point Corporation
david(at)endpoint(dot)com

On Tue, Aug 17, 2010 at 3:29 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I lack git-fu pretty completely, but I do have the CVS logs ;-).
> It looks like some of these commits that are being ascribed to the
> REL8_3_STABLE branch were actually only committed on HEAD.  For
> instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
> only in HEAD.  It was back-patched a few hours later (1 Mar 3:41),
> and that's also shown here, but the HEAD commit shouldn't be.

It looks to me like the commit I referenced in my original email is a
manufactured merge commit that completely rewrites the tree while
asserting that it doesn't do any such thing.

> I wonder whether the repository is completely OK and the problem
> is that this webpage isn't filtering the commits correctly.

No. The repository itself has the same problem, or at least my clone
of it does. I have to say I am totally underwhelmed by the quality of
the CVS-to-git conversion tools I've seen so far. It's fine for Linus
to say that CVS will eat your data, but these tools were evidently
written with grossly inadequate error and sanity checks. Whatever
we've been using for the incremental conversions doesn't seem to think
it's a problem if the new commit it's pushing doesn't make the head of
the tree match CVS HEAD, which seeems like a pretty darn obvious thing
to check for, and this tool evidently can't follow branch history
properly.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

* Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> [100817 15:30]:

> I lack git-fu pretty completely, but I do have the CVS logs ;-).
> It looks like some of these commits that are being ascribed to the
> REL8_3_STABLE branch were actually only committed on HEAD. For
> instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
> only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
> and that's also shown here, but the HEAD commit shouldn't be.
>
> I wonder whether the repository is completely OK and the problem
> is that this webpage isn't filtering the commits correctly.

No, that git branch is definately strange. The commit Robert pointed
out is a merge commit.

But looking at your explanation of when similar commits with the same
message were committed, I'm guessng the "timestamp fudge factor" along
with the "look for same commit message" behaviour of Magnus's cvs2git
conversion is trying "too hard" to make "atomic" commits of non-atomic
commits.

If you use a git viewer that shows the fork/merge points, you can see
that there are lots of these little "common" commits that have been
"unified" onto multiple brances.

Magnus, can you check if you can reduce the time fudge?

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

BTW, those two "manufactured" commits seem to directly follow commits
into HEAD that added files that were later also added on the branch.
I dunno exactly how git represents that type of event, but maybe an
extra commit is needed? It'd be interesting to look into the cvs2git
source code to see exactly what causes it to add a commit message
like that.

regards, tom lane

On Tue, Aug 17, 2010 at 3:40 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>  It'd be interesting to look into the cvs2git
> source code to see exactly what causes it to add a commit message
> like that.

I vigorously agree.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Tue, Aug 17, 2010 at 21:35, David Christensen <david(at)endpoint(dot)com> wrote:
>
> On Aug 17, 2010, at 2:29 PM, Magnus Hagander wrote:
>
>> On Tue, Aug 17, 2010 at 21:28, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> On Tue, Aug 17, 2010 at 3:23 PM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>>>> The tip of every branch, and every single tag, all have the correct
>>>> data in them, but some revisions in between seem majorly confused.
>>>
>>> It seems to me that what we'll need to do here is write a script to
>>> look through the CVS history of each file and make sure that the
>>> versions of that file which appear on each branch match the revs in
>>> CVS in content, order, and the commit message associated with any
>>> changes.  However, that's not going to do get done today.
>>
>> Yeah. Unless someone comes up with a good way to fix this, or even
>> better an explanation why it's actually ont broken and we're looking
>> at things wrong :D, I think we have no choice but aborting the
>> conversion for now and come back to it later.
>
>
> Can you post the cvs2svn command line used for conversion?

Sure:
cvs2git --options=/root/cvs2git.options

Attached is the options file.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 17, 2010 at 13:43, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Tue, Aug 17, 2010 at 3:40 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>  It'd be interesting to look into the cvs2git
>> source code to see exactly what causes it to add a commit message
>> like that.
>
> I vigorously agree.

How sure are we that its not the cvs2svn step that is screwing it up?
I know way back when I tried to convert a cvs tree to svn it fell over
horribly. Course the same was true when we went from svn to git...
(due to how we organized things in svn mainly)

On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex(at)gmail(dot)com> wrote:
> How sure are we that its not the cvs2svn step that is screwing it up?

urp, I jumped to a conclusion while skimming the cvs2git.options file
Magnus posted. With all the references to svn and things like
"GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
like it converts to svn first and then to git... im not sure what it
does.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> It looks to me like the commit I referenced in my original email is a
> manufactured merge commit that completely rewrites the tree while
> asserting that it doesn't do any such thing.

AFAICS, the commits in the 8.3 history *after* that point are sane;
at least there's one for each actual 8.3 commit in the CVS logs.
Before that point, though, the history shown for 8.3 seems to include
all HEAD commits as well. The merge commit is probably cleaning up
those incorrectly included HEAD commits.

I concur that we gotta abort the git conversion. This looks like it
might be a pretty simple bug in the converter, but we can't block
Postgres development while we look for it.

regards, tom lane

On Tue, Aug 17, 2010 at 01:57:02PM -0600 I heard the voice of
Alex Hunsaker, and lo! it spake thus:
> On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex(at)gmail(dot)com> wrote:
> > How sure are we that its not the cvs2svn step that is screwing it up?
>
> urp, I jumped to a conclusion while skimming the cvs2git.options
> file Magnus posted. With all the references to svn and things like
> "GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
> like it converts to svn first and then to git... im not sure what
> it does.

It's not that it converts to svn, but that it's built on (/part of)
cvs2svn, so presumably a lot of the "figure out changesets and branch
membership" logic and the "get things in the shape svn wants" logic
are intertwined.

--
Matthew Fuller (MF4839) | fullermd(at)over-yonder(dot)net
Systems/Network Administrator | http://www.over-yonder.net/~fullermd/
On the Internet, nobody can hear you scream.

Alex Hunsaker wrote:
> On Tue, Aug 17, 2010 at 13:52, Alex Hunsaker <badalex(at)gmail(dot)com> wrote:
>> How sure are we that its not the cvs2svn step that is screwing it up?
>
> urp, I jumped to a conclusion while skimming the cvs2git.options file
> Magnus posted. With all the references to svn and things like
> "GitRevisionRecorder('cvs2svn-tmp/git-blob.dat')". It sure sounded
> like it converts to svn first and then to git... im not sure what it
> does.

cvs2git converts directly from CVS to git. There is no intermediate SVN
step. However, given that cvs2git is built on top of cvs2svn, it is
true that some subversion terminology appears in the configuration file
and even in some of the manufactured commit messages.

Michael
the cvs2svn/cvs2git maintainer

Tom Lane wrote:
> I lack git-fu pretty completely, but I do have the CVS logs ;-).
> It looks like some of these commits that are being ascribed to the
> REL8_3_STABLE branch were actually only committed on HEAD. For
> instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
> only in HEAD. It was back-patched a few hours later (1 Mar 3:41),
> and that's also shown here, but the HEAD commit shouldn't be.
>
> I wonder whether the repository is completely OK and the problem
> is that this webpage isn't filtering the commits correctly.

Please don't panic :-)

The problem is that it is *impossible* to faithfully represent a CVS or
Subversion history with its ancestry information in a git repository (or
AFAIK any of the DVCS repositories). The reason is that CVS
fundamentally records the history of single files, and each file can
have a branching history that is incompatible with those of other files.
For example, in CVS, a file can be added to a branch after the branch
already exists, different files can be added to a branch from multiple
parent branches, and even more perverse things are allowed. The CVS
history can record this mish-mash (albeit with much ambiguity).

Git, on the other hand, fundamentally only records a single history that
is considered to apply to the entire source tree. If a commit is
created with more than one parent, git treats it as a merge and
implicitly assumes that all of the contents of all of the ancestor
commits of all of the parents have been merged into the new version of
the source tree.

See [1] for more discussion of the impedance mismatch between the
branching model of CVS/Subversion vs. that of the DVCSs.

So let's take the simplest example: a branch BRANCH1 is created from
trunk commit T1, then some time later another FILE1 from trunk commit T3
is added to BRANCH1 in commit B4. How should this series of events be
represented in a git repository?

The "inclusive" possibility is to say that some content was merged from
trunk to BRANCH1, and therefore to treat B4 as a merge commit:

T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
\ \
B1 -- B2 -- B3 -- B4 BRANCH1

This is wrong because there might be other changes in T2 and T3 (besides
the addition of FILE1) that were *not* merged to BRANCH1.

The "exclusive" possibility is to ignore the fact that some of the
content of B4 came from trunk and to pretend that FILE1 just appeared
out of nowhere in commit B4 independent of the FILE1 in TRUNK:

T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
\
B1 -- B2 -- B3 -- B4 BRANCH1

This is also wrong, because it doesn't reflect the true lineage of FILE1.

Given the choice between two wrong histories, cvs2git uses the
"inclusive" style. The result is that the ancestors of B4 include not
only T0, T1, B1, B2, and B3 (as might be expected), but also T2 and T3.
The display in the website that was quoted [2] seems to mash all of the
ancestors together without showing the topology of the history, making
the result quite confusing. The true history looks more like this:

$ git log --oneline --graph REL8_3_10 master
[...]
| * 2a91f07 tag 8.3.10
| * eb1b49f Preliminary release notes for releases 8.4.3, 8.3
| * dcf9673 Use SvROK(sv) rather than directly checking SvTYP
| * 1194fb9 Update time zone data files to tzdata release 201
| * fdfd1ec Return proper exit code (3) from psql when ON_ERR
| * 77524a1 Backport fix from HEAD that makes ecpglib give th
| * 55391af Add missing space in example.
| * 982aa23 Require hostname to be set when using GSSAPI auth
| * cb58615 Update time zone data files to tzdata release 201
| * ebe1e29 When reading pg_hba.conf and similar files, do no
| * 5a401e6 Fix a couple of places that would loop forever if
| * 5537492 Make contrib/xml2 use core xml.c's error handler,
| * c720f38 Export xml.c's libxml-error-handling support so t
| * 42ac390 Make iconv work like other optional libraries for
| * b03d523 pgindent run on xml.c in 8.3 branch, per request
| * 7efcdaa Add missing library and include dir for XSLT in M
| * 6ab1407 Do not run regression tests for contrib/xml2 on M
| * fff18e6 Backpatch MSVC build fix for XSLT
| * 7ae09ef Fix numericlocale psql option when used with a nu
| * de92a3d Fix contrib/xml2 so regression test still works w
| * 80f81c3 This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | a08b04f Fix contrib/xml2 so regression test still works w
* | 0d69e0f It's clearly now pointless to do backwards compat
* | 4ad348c Buildfarm still unhappy, so I'll bet it's EACCES
* | 6e96e1b Remove xmlCleanupParser calls from contrib/xml2.
* | 5b65b67 add EPERM to the list of return codes to expect f
| * a4067b3 Remove xmlCleanupParser calls from contrib/xml2.
| * 91b76a4 Back-patch today's memory management fixups in co
| * 5e74f21 Back-patch changes of 2009-05-13 in xml.c's memor
| * 043041e This commit was manufactured by cvs2svn to crea
| |\
| |/
|/|
* | 98cc16f Fix up memory management problems in contrib/xml2
* | 17e1420 Second try at fsyncing directories in CREATE DATA
* | a350f70 Assorted code cleanup for contrib/xml2. No chang
* | 3524149 Update complex locale example in the documentatio
[...]

The left branch is master, the right branch is the one leading to
REL8_3_10. You can see that there are multiple merges from master to
the branch, presumably when new files from trunk were ported to the
branch. This is even easier to see using a graphical history browser
like gitk.

There are good arguments for both the "inclusive" and the "exclusive"
representation of history. The ideal would require a lot more
intelligence and better heuristics (and slow down the conversion
dramatically). But even the smartest conversion would still be wrong,
because git is simply incapable of representing an arbitrary CVS
history. The main practical result of the impedance mismatch is that it
will be more difficult to merge between branches that originated in CVS
(but that is no surprise!)

Michael
the cvs2svn/cvs2git maintainer

[1]
http://softwareswirl.blogspot.com/2009/08/git-mercurial-and-bazaarsimplicity.html

On Wed, Aug 18, 2010 at 08:25:45AM +0200, Michael Haggerty wrote:
> So let's take the simplest example: a branch BRANCH1 is created from
> trunk commit T1, then some time later another FILE1 from trunk commit T3
> is added to BRANCH1 in commit B4. How should this series of events be
> represented in a git repository?

<snip>

> The "exclusive" possibility is to ignore the fact that some of the
> content of B4 came from trunk and to pretend that FILE1 just appeared
> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>
> T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
> \
> B1 -- B2 -- B3 -- B4 BRANCH1
>
> This is also wrong, because it doesn't reflect the true lineage of FILE1.

But the "true lineage" is not stored anywhere in CVS so I don't see why
you need to fabricate it for git. Sure, it would be really nice if you
could, but if you can't do it reliably, you may as well not do it at
all. What's the loss?

Have a nice day,
--
Martijn van Oosterhout <kleptog(at)svana(dot)org> http://svana.org/kleptog/
> Patriotism is when love of your own people comes first; nationalism,
> when hate for people other than your own comes first.
> - Charles de Gaulle

On Wed, Aug 18, 2010 at 08:25, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
> Tom Lane wrote:
>> I lack git-fu pretty completely, but I do have the CVS logs ;-).
>> It looks like some of these commits that are being ascribed to the
>> REL8_3_STABLE branch were actually only committed on HEAD.  For
>> instance my commit in contrib/xml2 on 28 Feb 2010 21:31:57 was
>> only in HEAD.  It was back-patched a few hours later (1 Mar 3:41),
>> and that's also shown here, but the HEAD commit shouldn't be.
>>
>> I wonder whether the repository is completely OK and the problem
>> is that this webpage isn't filtering the commits correctly.
>
> Please don't panic :-)

We're not panic'ing just yet :-)

> The problem is that it is *impossible* to faithfully represent a CVS or
> Subversion history with its ancestry information in a git repository (or
> AFAIK any of the DVCS repositories).  The reason is that CVS
> fundamentally records the history of single files, and each file can
> have a branching history that is incompatible with those of other files.
>  For example, in CVS, a file can be added to a branch after the branch
> already exists, different files can be added to a branch from multiple
> parent branches, and even more perverse things are allowed.  The CVS
> history can record this mish-mash (albeit with much ambiguity).

It can. IIRC we have cleaned a couple of such things out.

<snip some good descriptions of how git works>

> Given the choice between two wrong histories, cvs2git uses the
> "inclusive" style.  The result is that the ancestors of B4 include not
> only T0, T1, B1, B2, and B3 (as might be expected), but also T2 and T3.
>  The display in the website that was quoted [2] seems to mash all of the
> ancestors together without showing the topology of the history, making
> the result quite confusing.  The true history looks more like this:
>
> $ git log --oneline --graph REL8_3_10 master
> [...]
> | * 2a91f07 tag 8.3.10
> | * eb1b49f Preliminary release notes for releases 8.4.3, 8.3
> | * dcf9673 Use SvROK(sv) rather than directly checking SvTYP
> | * 1194fb9 Update time zone data files to tzdata release 201
> | * fdfd1ec Return proper exit code (3) from psql when ON_ERR
> | * 77524a1 Backport fix from HEAD that makes ecpglib give th
> | * 55391af Add missing space in example.
> | * 982aa23 Require hostname to be set when using GSSAPI auth
> | * cb58615 Update time zone data files to tzdata release 201
> | * ebe1e29 When reading pg_hba.conf and similar files, do no
> | * 5a401e6 Fix a couple of places that would loop forever if
> | * 5537492 Make contrib/xml2 use core xml.c's error handler,
> | * c720f38 Export xml.c's libxml-error-handling support so t
> | * 42ac390 Make iconv work like other optional libraries for
> | * b03d523 pgindent run on xml.c in 8.3 branch, per request
> | * 7efcdaa Add missing library and include dir for XSLT in M
> | * 6ab1407 Do not run regression tests for contrib/xml2 on M
> | * fff18e6 Backpatch MSVC build fix for XSLT
> | * 7ae09ef Fix numericlocale psql option when used with a nu
> | * de92a3d Fix contrib/xml2 so regression test still works w
> | *   80f81c3 This commit was manufactured by cvs2svn to crea
> | |\
> | |/
> |/|
> * | a08b04f Fix contrib/xml2 so regression test still works w
> * | 0d69e0f It's clearly now pointless to do backwards compat
> * | 4ad348c Buildfarm still unhappy, so I'll bet it's EACCES
> * | 6e96e1b Remove xmlCleanupParser calls from contrib/xml2.
> * | 5b65b67 add EPERM to the list of return codes to expect f
> | * a4067b3 Remove xmlCleanupParser calls from contrib/xml2.
> | * 91b76a4 Back-patch today's memory management fixups in co
> | * 5e74f21 Back-patch changes of 2009-05-13 in xml.c's memor
> | *   043041e This commit was manufactured by cvs2svn to crea
> | |\
> | |/
> |/|
> * | 98cc16f Fix up memory management problems in contrib/xml2
> * | 17e1420 Second try at fsyncing directories in CREATE DATA
> * | a350f70 Assorted code cleanup for contrib/xml2.  No chang
> * | 3524149 Update complex locale example in the documentatio
> [...]
>
> The left branch is master, the right branch is the one leading to
> REL8_3_10.  You can see that there are multiple merges from master to
> the branch, presumably when new files from trunk were ported to the
> branch.  This is even easier to see using a graphical history browser
> like gitk.

Yeah, this is clearly the problem.

> There are good arguments for both the "inclusive" and the "exclusive"
> representation of history.  The ideal would require a lot more
> intelligence and better heuristics (and slow down the conversion
> dramatically).  But even the smartest conversion would still be wrong,
> because git is simply incapable of representing an arbitrary CVS
> history.  The main practical result of the impedance mismatch is that it
> will be more difficult to merge between branches that originated in CVS
> (but that is no surprise!)

Our requirements are simple: our cvs history is linear, the git
history should be linear. It is *not* the same commit that's on head
and the branch. They are two different commits, that happen to have
the same commit message and mostly the same content.

Bottom line is, we want zero merge commits in the git repository. We
may start using that sometime in the future (but for now, we've
decided we don't want that even in the future), but we most
*definitely* don't want it in the past. We don't care about
"representing the proper heritage of FILE1" in git, because we never
did in cvs.

Is there some way to make cvs2git work this way, and just not bother
even trying to create merge commits, or is that fundamentally
impossible and we need to look at another tool?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Martijn van Oosterhout wrote:
> On Wed, Aug 18, 2010 at 08:25:45AM +0200, Michael Haggerty wrote:
>> So let's take the simplest example: a branch BRANCH1 is created from
>> trunk commit T1, then some time later another FILE1 from trunk commit T3
>> is added to BRANCH1 in commit B4. How should this series of events be
>> represented in a git repository?
>
> <snip>
>
>> The "exclusive" possibility is to ignore the fact that some of the
>> content of B4 came from trunk and to pretend that FILE1 just appeared
>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>>
>> T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
>> \
>> B1 -- B2 -- B3 -- B4 BRANCH1
>>
>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>
> But the "true lineage" is not stored anywhere in CVS so I don't see why
> you need to fabricate it for git. Sure, it would be really nice if you
> could, but if you can't do it reliably, you may as well not do it at
> all. What's the loss?

CVS does record (albeit somewhat ambiguously) the branch from which a
new branch sprouted. The history above might result from commands like

cvs update -A
cvs tag -b BRANCH1
<hack hack> cvs update -r BRANCH1
cvs commit -m T2 <hack hack>
touch FILE1 cvs commit -m B1
cvs add FILE1 <hack hack>
cvs commit -m T3 cvs commit -m B2
<hack hack>
cvs commit -m B3
cvs tag -b BRANCH1 FILE1

or the last step might have been an explicit merge into BRANCH1:

cvs update -j T1 -j T3
cvs commit -m B4

Either way, the CVS history relatively clearly indicates that content
was ported from TRUNK to BRANCH1. There is no way to distinguish
whether it was a cherry-pick (not recordable in git's history) vs. a
full merge without more information or more intelligence.

Magnus Hagander wrote:
> Our requirements are simple: our cvs history is linear, the git
> history should be linear. It is *not* the same commit that's on head
> and the branch. They are two different commits, that happen to have
> the same commit message and mostly the same content.

I don't think this is at all an issue of cvs2svn merging commits that
happen to have the same commit message and/or commit time. The merge
commits are all manufactured by cvs2svn to do two things:

1. Add content that needs to be on the branch, because a file was added
to the branch after the branch's creation. This *needs* to be done to
ensure that the branch has the correct content.

2. Indicate the origin of the new branch content. This goal is debatable.

> Bottom line is, we want zero merge commits in the git repository. We
> may start using that sometime in the future (but for now, we've
> decided we don't want that even in the future), but we most
> *definitely* don't want it in the past. We don't care about
> "representing the proper heritage of FILE1" in git, because we never
> did in cvs.
>
> Is there some way to make cvs2git work this way, and just not bother
> even trying to create merge commits, or is that fundamentally
> impossible and we need to look at another tool?

A merge is just a special case of content being taken from one branch
and added to another. Logically, the same thing happens when a branch
is created, and some of the same problems can occur in that situation.
A branch can be created using content from multiple source branches,
which cvs2git currently also represents as a merge.

Assuming that you don't want to discard all record of where a branch
sprouted from, it is therefore necessary to choose a single parent
branch for each branch creation. To be sure, this choice can be
incorrect the same way as the merge commits discussed above are
incorrect. But one reasonable "mostly-exclusive" approach would be to
choose the most likely parent as the source of the branch and ignore all
others.

cvs2git doesn't currently have this option. I'm not sure how much work
it would be to implement; probably a few days'. Alternatively, you
could write a tool that would rewrite the ancestry information in the
repository *after* the cvs2git conversion using .git/info/grafts (see
git-filter-branch(1)). Such rewriting would have to occur before the
repository is published, because the rewriting will change the hashes of
most commits.

Michael

On Wed, Aug 18, 2010 at 11:01, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
> Martijn van Oosterhout wrote:
>> On Wed, Aug 18, 2010 at 08:25:45AM +0200, Michael Haggerty wrote:
>>> So let's take the simplest example: a branch BRANCH1 is created from
>>> trunk commit T1, then some time later another FILE1 from trunk commit T3
>>> is added to BRANCH1 in commit B4.  How should this series of events be
>>> represented in a git repository?
>>
>> <snip>
>>
>>> The "exclusive" possibility is to ignore the fact that some of the
>>> content of B4 came from trunk and to pretend that FILE1 just appeared
>>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>>>
>>> T0 -- T1 -- T2 -------- T3 -- T4        TRUNK
>>>        \
>>>         B1 -- B2 -- B3 -- B4            BRANCH1
>>>
>>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>>
>> But the "true lineage" is not stored anywhere in CVS so I don't see why
>> you need to fabricate it for git. Sure, it would be really nice if you
>> could, but if you can't do it reliably, you may as well not do it at
>> all. What's the loss?
>
> CVS does record (albeit somewhat ambiguously) the branch from which a
> new branch sprouted.  The history above might result from commands like
>
> cvs update -A
> cvs tag -b BRANCH1
> <hack hack>                   cvs update -r BRANCH1
> cvs commit -m T2              <hack hack>
> touch FILE1                   cvs commit -m B1
> cvs add FILE1                 <hack hack>
> cvs commit -m T3              cvs commit -m B2
>                              <hack hack>
>                              cvs commit -m B3
> cvs tag -b BRANCH1 FILE1
>
> or the last step might have been an explicit merge into BRANCH1:
>
>                              cvs update -j T1 -j T3
>                              cvs commit -m B4
>
> Either way, the CVS history relatively clearly indicates that content
> was ported from TRUNK to BRANCH1.  There is no way to distinguish
> whether it was a cherry-pick (not recordable in git's history) vs. a
> full merge without more information or more intelligence.

Well, in *our* case we know that it was a "cherry-pick". Because we've
done no full merges ;) So if there's a way for us to short-wire the
tool, that'd be great.

> Magnus Hagander wrote:
>> Our requirements are simple: our cvs history is linear, the git
>> history should be linear. It is *not* the same commit that's on head
>> and the branch. They are two different commits, that happen to have
>> the same commit message and mostly the same content.
>
> I don't think this is at all an issue of cvs2svn merging commits that
> happen to have the same commit message and/or commit time.  The merge
> commits are all manufactured by cvs2svn to do two things:
>
> 1. Add content that needs to be on the branch, because a file was added
> to the branch after the branch's creation.  This *needs* to be done to
> ensure that the branch has the correct content.

Ok.

> 2. Indicate the origin of the new branch content.  This goal is debatable.

I agree this is debatable. We've kind of debated it already (though
not in exactly this context) and decided we'd rather have it appear as
brand new content on this branch and not as a merge.

>> Bottom line is, we want zero merge commits in the git repository. We
>> may start using that sometime in the future (but for now, we've
>> decided we don't want that even in the future), but we most
>> *definitely* don't want it in the past. We don't care about
>> "representing the proper heritage of FILE1" in git, because we never
>> did in cvs.
>>
>> Is there some way to make cvs2git work this way, and just not bother
>> even trying to create merge commits, or is that fundamentally
>> impossible and we need to look at another tool?
>
> A merge is just a special case of content being taken from one branch
> and added to another.  Logically, the same thing happens when a branch
> is created, and some of the same problems can occur in that situation.
> A branch can be created using content from multiple source branches,
> which cvs2git currently also represents as a merge.

Can be, yes. AFAIK, we don't ever do that (though I can't swear to
that, since there have been some funky things in our cvs repository
earlier)

> Assuming that you don't want to discard all record of where a branch
> sprouted from, it is therefore necessary to choose a single parent
> branch for each branch creation.  To be sure, this choice can be
> incorrect the same way as the merge commits discussed above are
> incorrect.  But one reasonable "mostly-exclusive" approach would be to
> choose the most likely parent as the source of the branch and ignore all
> others.

Yes, I believe that is what we'd prefer, as it's what most closely
matches how *we*'ve been using CVS.

> cvs2git doesn't currently have this option.  I'm not sure how much work
> it would be to implement; probably a few days'.  Alternatively, you

Would this be something you'd consider doing, since it might be of
interest to others? I'm sure if it's a few days work for you, it'd be
weeks for one of us, given no knowledge of the codebase :-)

Obviously not saying it needs to be done in two days or anything, now
that we've postponed the migration this time, we're not on as tight a
schedule anymore.

> could write a tool that would rewrite the ancestry information in the
> repository *after* the cvs2git conversion using .git/info/grafts (see
> git-filter-branch(1)).  Such rewriting would have to occur before the
> repository is published, because the rewriting will change the hashes of
> most commits.

That could definitely be done.

Um, I don't see a "info/grafts" though (our repo is a bare one, could
that be why?)

Do you have any more specifics, or a reference, as to how you'd
suggest we look at that?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
> So let's take the simplest example: a branch BRANCH1 is created from
> trunk commit T1, then some time later another FILE1 from trunk commit T3
> is added to BRANCH1 in commit B4. How should this series of events be
> represented in a git repository?
> ...
> The "exclusive" possibility is to ignore the fact that some of the
> content of B4 came from trunk and to pretend that FILE1 just appeared
> out of nowhere in commit B4 independent of the FILE1 in TRUNK:

> T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
> \
> B1 -- B2 -- B3 -- B4 BRANCH1

> This is also wrong, because it doesn't reflect the true lineage of FILE1.

Maybe not, but that *is* how things appeared in the CVS history, and
we'd rather have a git history that looks like the CVS history than
one that claims that boatloads of utterly unrelated commits are part
of a branch's history.

The "inclusive" possibility might be tolerable if it restricted itself
to mentioning commits that actually touched FILE1 in between its
addition to TRUNK and its addition to BRANCH1. So far as I can see,
though, cvs2git is mentioning *every* commit on TRUNK between T1 and B4
... not even between T3 and B4, but back to the branch point. How can
you possibly justify that as either sane or useful?

regards, tom lane

Excerpts from Michael Haggerty's message of mié ago 18 05:01:29 -0400 2010:

> cvs2git doesn't currently have this option. I'm not sure how much work
> it would be to implement; probably a few days'. Alternatively, you
> could write a tool that would rewrite the ancestry information in the
> repository *after* the cvs2git conversion using .git/info/grafts (see
> git-filter-branch(1)). Such rewriting would have to occur before the
> repository is published, because the rewriting will change the hashes of
> most commits.

AFAICT, graft points are not checked in[1], thus they don't propagate; are
you saying that we should run the migration, then manually inject the
graft points, then run some conversion tool that writes a different
repository with those graft points welded into the history? This sounds
like it needs some manual work (namely find out the appropriate graft
points for each branch), that can be prepared beforehand. Otherwise it
seems easier than reworking the cvs2git code for the "mostly-exclusive"
option.

I am sort of assuming that this "conversion tool" already exists, but
maybe this is not the case?

[1] http://stackoverflow.com/questions/1488753/how-to-merge-two-branches-without-a-common-ancestor

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

I previously proposed off-list an alternate solution to generate the git
repository which was turned down due to it not being able to handle
incremental updates. However, since we are now looking at a one-time
conversion, this method might come in handy.

---
Caveat: cvs2git apparently requires CVSROOT somewhere in the path for
it to work. I did a symbolic link of the current directory $PWD with
CVSROOT to bypass the quirk cvs2git requires.

mkdir work
cd work
wget http://ftp.netbsd.se/pkgsrc/distfiles/cvsclone-0.00/cvsclone.l
flex cvsclone.l && gcc -Wall -O2 lex.yy.c -o cvsclone
cvsclone -d :pserver:anoncvs(at)anoncvs(dot)postgresql(dot)org:/projects/cvsroot pgsql
ln -s $PWD CVSROOT
cvs2git --blobfile=blobfile --dumpfile=dumpfile --username pgdude
--encoding=UTF8 --fallback-encoding=UTF8 CVSROOT/pgsql > cvs2git.log
mkdir git && cd git && git init .
cat ../blobfile ../dumpfile | git fast-import
git reset --hard
cd ..
---

Regards,
Khee Chin.

On Wed, Aug 18, 2010 at 11:14 PM, Alvaro Herrera <alvherre(at)commandprompt(dot)com
> wrote:

> Excerpts from Michael Haggerty's message of mié ago 18 05:01:29 -0400 2010:
>
> > cvs2git doesn't currently have this option. I'm not sure how much work
> > it would be to implement; probably a few days'. Alternatively, you
> > could write a tool that would rewrite the ancestry information in the
> > repository *after* the cvs2git conversion using .git/info/grafts (see
> > git-filter-branch(1)). Such rewriting would have to occur before the
> > repository is published, because the rewriting will change the hashes of
> > most commits.
>
> AFAICT, graft points are not checked in[1], thus they don't propagate; are
> you saying that we should run the migration, then manually inject the
> graft points, then run some conversion tool that writes a different
> repository with those graft points welded into the history? This sounds
> like it needs some manual work (namely find out the appropriate graft
> points for each branch), that can be prepared beforehand. Otherwise it
> seems easier than reworking the cvs2git code for the "mostly-exclusive"
> option.
>
> I am sort of assuming that this "conversion tool" already exists, but
> maybe this is not the case?
>
> [1]
> http://stackoverflow.com/questions/1488753/how-to-merge-two-branches-without-a-common-ancestor
>
> --
> Álvaro Herrera <alvherre(at)commandprompt(dot)com>
> The PostgreSQL Company - Command Prompt, Inc.
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>

Tom Lane wrote:
> Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
>> So let's take the simplest example: a branch BRANCH1 is created from
>> trunk commit T1, then some time later another FILE1 from trunk commit T3
>> is added to BRANCH1 in commit B4. How should this series of events be
>> represented in a git repository?
>> ...
>> The "exclusive" possibility is to ignore the fact that some of the
>> content of B4 came from trunk and to pretend that FILE1 just appeared
>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>
>> T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
>> \
>> B1 -- B2 -- B3 -- B4 BRANCH1
>
>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>
> Maybe not, but that *is* how things appeared in the CVS history, and
> we'd rather have a git history that looks like the CVS history than
> one that claims that boatloads of utterly unrelated commits are part
> of a branch's history.
>
> The "inclusive" possibility might be tolerable if it restricted itself
> to mentioning commits that actually touched FILE1 in between its
> addition to TRUNK and its addition to BRANCH1. So far as I can see,
> though, cvs2git is mentioning *every* commit on TRUNK between T1 and B4
> ... not even between T3 and B4, but back to the branch point. How can
> you possibly justify that as either sane or useful?

There is no way, in git, to claim that (say) T3 was incorporated into B4
but that T2 was not. If T3 is listed as a parent of B4, then it is
implied that all ancestors of T3 are also incorporated into B4. This is
a crucial simplification that helps DVCSs merge reliably. So an
"exclusive" option is definitely the way to go for the postgresql project.

[By the way, it *is* possible to list the commits that touched FILE1:

git log BRANCH1 -- FILE1

The user would first have to find out that FILE1 is the file that is the
subject of merge B4, which could be done using "git diff B3..B4". But I
am not arguing that this is the preferred solution, given your project's
practice to do cherry-picks and never full merges.]

Michael

On Wed, Aug 18, 2010 at 17:33, Khee Chin <kheechin(at)gmail(dot)com> wrote:
> I previously proposed off-list an alternate solution to generate the git
> repository which was turned down due to it not being able to handle
> incremental updates. However, since we are now looking at a one-time
> conversion, this method might come in handy.

cvs2git *is* the tool we've been using now that it's a one-off
conversion. It's the one that's causing the current problems.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Wed, Aug 18, 2010 at 11:03 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
>> So let's take the simplest example: a branch BRANCH1 is created from
>> trunk commit T1, then some time later another FILE1 from trunk commit T3
>> is added to BRANCH1 in commit B4.  How should this series of events be
>> represented in a git repository?
>> ...
>> The "exclusive" possibility is to ignore the fact that some of the
>> content of B4 came from trunk and to pretend that FILE1 just appeared
>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>
>> T0 -- T1 -- T2 -------- T3 -- T4        TRUNK
>>        \
>>         B1 -- B2 -- B3 -- B4            BRANCH1
>
>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>
> Maybe not, but that *is* how things appeared in the CVS history, and
> we'd rather have a git history that looks like the CVS history than
> one that claims that boatloads of utterly unrelated commits are part
> of a branch's history.

Exactly. IMHO, the way this should work is by starting at the
beginning of time and working forward. At each step, we examine the
earliest revision of each file for which no git commit has yet been
written. From among those, we select the one with the earliest
timestamp. We then also select all other files whose most recent
unprocessed revision is nearly contemporaneous and shares the same
author and log message. From the results, we generate a commit. Then
we repeat. When we arrive at a branch point, the branch gets
processed separately from the trunk. If there is no trunk rev which
has every file at the rev where it starts on the branch, then we use
some sane algorithm to pick the best one (perhaps, the one that has
the right revs of the most files) and then insert a fixup commit on
the branch to remove the deltas and carry on as before.

> The "inclusive" possibility might be tolerable if it restricted itself
> to mentioning commits that actually touched FILE1 in between its
> addition to TRUNK and its addition to BRANCH1.  So far as I can see,
> though, cvs2git is mentioning *every* commit on TRUNK between T1 and B4
> ... not even between T3 and B4, but back to the branch point.  How can
> you possibly justify that as either sane or useful?

git can't do that. It's finding those commits by following parent
pointers from the merge commits.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Alvaro Herrera wrote:
> Excerpts from Michael Haggerty's message of mié ago 18 05:01:29 -0400 2010:
>
>> [...] Alternatively, you
>> could write a tool that would rewrite the ancestry information in the
>> repository *after* the cvs2git conversion using .git/info/grafts (see
>> git-filter-branch(1)). Such rewriting would have to occur before the
>> repository is published, because the rewriting will change the hashes of
>> most commits.
>
> AFAICT, graft points are not checked in[1], thus they don't propagate; are
> you saying that we should run the migration, then manually inject the
> graft points, then run some conversion tool that writes a different
> repository with those graft points welded into the history? This sounds
> like it needs some manual work (namely find out the appropriate graft
> points for each branch), that can be prepared beforehand. Otherwise it
> seems easier than reworking the cvs2git code for the "mostly-exclusive"
> option.

It is true that grafts are not propagated, but they can be baked into a
repository (at the cost of rewriting the SHA1 hashes) using "git
filter-branch". The procedure would be as follows:

1. Convert using cvs2git

2. Create a file .git/info/grafts containing the changes that you want
to make to the project's ancestry. The file has the format

commit parent0 parent1 ...

where each of the entries is a SHA1 hash from the existing repository.
Only commits whose parentage should be changed need to be mentioned.
This is the tricky step because it requires some logic to decide what
needs changing. And it can only be done after the cvs2git conversion,
because it requires the SHA1s resulting from the conversion.

3. Run

git filter-branch

This rewrites the commits using any parentage changes from the grafts
file. This changes most commits' SHA1 hashes. After this you can
discard the .git/info/grafts file. You would then want to remove the
original references, which were moved to "refs/original".

4. Publish the repository.

As long as the repository is only published after the grafts have been
baked in, there is no reason that anybody else would need the grafts file.

Michael

Robert Haas wrote:
> Exactly. IMHO, the way this should work is by starting at the
> beginning of time and working forward. [...]

What you are describing is more or less the algorithm that was used by
cvs2svn version 1.x. It mostly works, but has nasty edge cases that are
impossible to fix.

cvs2svn version 2.x uses a better algorithm [1]. It can be changed to
add an "exclusive" mode, it's a simple matter of programming. I will
try to find some time to work on it.

Michael

[1]
http://cvs2svn.tigris.org/source/browse/cvs2svn/trunk/doc/design-notes.txt?view=markup

Tom Lane wrote:
> Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
>> The "exclusive" possibility is to ignore the fact that some of the
>> content of B4 came from trunk and to pretend that FILE1 just appeared
>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>
>> T0 -- T1 -- T2 -------- T3 -- T4 TRUNK
>> \
>> B1 -- B2 -- B3 -- B4 BRANCH1
>
>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>
> Maybe not, but that *is* how things appeared in the CVS history, [...]

I forgot to point out that "the CVS history" looks nothing like this,
because the CVS history is only defined file by file. So the CVS
history of FILE0 might look like this:

1.0 - 1.1 ------ 1.2 ----------------- 1.3 ----- 1.4 TRUNK
\
1.1.2.1 -- 1.1.2.2 -- 1.1.2.3 -- 1.1.2.4 BRANCH1

whereas the history of FILE1 probably looks more like this:

1.1 ----------------- 1.2 ----- 1.3 TRUNK
\
1.2.2.1 -- 1.2.2.2 BRANCH1

(here I've tried to put corresponding commits in the same relative
location) and there might be a FILE2 that looks like this:

1.0 ------------ 1.1 --------------------------- 1.2 TRUNK
\
*no commit here* BRANCH1

Perhaps this makes it clearer why creating a single git history requires
some compromises.

Michael

Excerpts from Magnus Hagander's message of mié ago 18 11:52:58 -0400 2010:
> On Wed, Aug 18, 2010 at 17:33, Khee Chin <kheechin(at)gmail(dot)com> wrote:
> > I previously proposed off-list an alternate solution to generate the git
> > repository which was turned down due to it not being able to handle
> > incremental updates. However, since we are now looking at a one-time
> > conversion, this method might come in handy.
>
> cvs2git *is* the tool we've been using now that it's a one-off
> conversion. It's the one that's causing the current problems.

I think the point is to run the repo through cvsclone, which apparently
changes the repo in some (not documented) ways, removing "corruption".
Not sure how this is an essential part of Khee Chin's proposal.

The cited URL is no longer valid however. The code can be found here
http://samba.org/ftp/tridge/rtc/cvsclone.l

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Excerpts from Michael Haggerty's message of mié ago 18 12:00:44 -0400 2010:

> 3. Run
>
> git filter-branch
>
> This rewrites the commits using any parentage changes from the grafts
> file. This changes most commits' SHA1 hashes. After this you can
> discard the .git/info/grafts file. You would then want to remove the
> original references, which were moved to "refs/original".

Hmm. If I need to do two changes in the same branch, do I need to
mention the new SHA1 for the second one (after filter-branch changes its
SHA1), or the original one? If the former, then this is going to be a
very painful process.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Wed, 2010-08-18 at 12:26 -0400, Alvaro Herrera wrote:
> Excerpts from Magnus Hagander's message of mié ago 18 11:52:58 -0400 2010:
> > On Wed, Aug 18, 2010 at 17:33, Khee Chin <kheechin(at)gmail(dot)com> wrote:
> > > I previously proposed off-list an alternate solution to generate the git
> > > repository which was turned down due to it not being able to handle
> > > incremental updates. However, since we are now looking at a one-time
> > > conversion, this method might come in handy.
> >
> > cvs2git *is* the tool we've been using now that it's a one-off
> > conversion. It's the one that's causing the current problems.

We had a lot of luck with cvs to svn conversion in the past. And
supposedly the git-svn stuff is top notch. It may be worth a shot.

JD

> --
> Álvaro Herrera <alvherre(at)commandprompt(dot)com>
> The PostgreSQL Company - Command Prompt, Inc.
> PostgreSQL Replication, Consulting, Custom Development, 24x7 support
>

On Wed, Aug 18, 2010 at 12:18 PM, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
> Tom Lane wrote:
>> Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
>>> The "exclusive" possibility is to ignore the fact that some of the
>>> content of B4 came from trunk and to pretend that FILE1 just appeared
>>> out of nowhere in commit B4 independent of the FILE1 in TRUNK:
>>
>>> T0 -- T1 -- T2 -------- T3 -- T4        TRUNK
>>>        \
>>>         B1 -- B2 -- B3 -- B4            BRANCH1
>>
>>> This is also wrong, because it doesn't reflect the true lineage of FILE1.
>>
>> Maybe not, but that *is* how things appeared in the CVS history, [...]
>
> I forgot to point out that "the CVS history" looks nothing like this,
> because the CVS history is only defined file by file.  So the CVS
> history of FILE0 might look like this:
>
>  1.0 - 1.1 ------ 1.2 ----------------- 1.3 ----- 1.4        TRUNK
>        \
>         1.1.2.1 -- 1.1.2.2 -- 1.1.2.3 -- 1.1.2.4            BRANCH1
>
> whereas the history of FILE1 probably looks more like this:
>
>                  1.1 ----------------- 1.2 ----- 1.3        TRUNK
>                                         \
>                                          1.2.2.1 -- 1.2.2.2 BRANCH1
>
> (here I've tried to put corresponding commits in the same relative
> location) and there might be a FILE2 that looks like this:
>
>  1.0 ------------ 1.1 --------------------------- 1.2        TRUNK
>                   \
>                    *no commit here*                         BRANCH1
>
> Perhaps this makes it clearer why creating a single git history requires
> some compromises.

I think we all understand that the conversion process may create some
artifacts. Also, since I think this has not yet been mentioned, I
really appreciate you being willing to jump into this discussion and
possibly try to write some code to help us get what we want.

I think what is frustrating is that we have a mental image of what the
history looks like in CVS based on what we actually do, and it doesn't
look anything like the history that cvs2git created. You can to all
kinds of crazy things in CVS, like tag the whole tree and then move
the tags on half a dozen individual files forward or backward in time,
or delete the tags off them altogether. But we believe (perhaps
naively) that we haven't done those things, so we're expecting to get
a simple linear history without merges, and definitely without commits
from one branch jumping into the midst of other branches. What was
really alarming to me about what I found yesterday is that - even
after reading your explanation - I can't understand why it did that.
I think it's human nature to like it when good things happen to us and
to dislike it when bad things happen to us, but we tend to hate the
bad things a lot more when we feel like we didn't deserve it. If
you're going 90 MPH and get a speeding ticket, you may be steamed, but
at some level you know you deserved it. If you were going 50 MPH on a
road where the speed limit is 55 MPH and the cop tickets you for 60
MPH, even the most mild-mannered driver may feel an urge to say
something less polite than "thank you, officer". Hence our
consternation. Perhaps there is some way to tilt your head so that
these merge commits are the Right Thing To Do, but to me at least it
feels extremely weird and inexplicable. If at some point, we had
taken the majority of the deltas between 9.0 and 8.3 and put them into
8.3 and the converter said "oh, that's a merge", well, we might want
an option to turn that behavior off, but at least it would be clear
why it happened. But the merge commit that got fabricated here almost
by definition has to be ignoring the vast bulk of the activity on one
side, which just doesn't feel right.

To what degree does your proposed solution (an "exclusive" option)
resemble "don't ever create merge commits"?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Excerpts from Robert Haas's message of mié ago 18 13:10:19 -0400 2010:

> I think what is frustrating is that we have a mental image of what the
> history looks like in CVS based on what we actually do, and it doesn't
> look anything like the history that cvs2git created. You can to all
> kinds of crazy things in CVS, like tag the whole tree and then move
> the tags on half a dozen individual files forward or backward in time,
> or delete the tags off them altogether. But we believe (perhaps
> naively) that we haven't done those things, so we're expecting to get
> a simple linear history without merges, and definitely without commits
> from one branch jumping into the midst of other branches.

In fact, we went some lengths to remove some of the more problematic
artifacts in our original CVS repository, so that a Git conversion
wouldn't have a problem with them. It's disappointing that it ends up
punting in this manner.

I do welcome the offer of Michael's development time to solve our
problems.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Alvaro Herrera wrote:
> Excerpts from Michael Haggerty's message of mié ago 18 12:00:44 -0400 2010:
>
>> 3. Run
>>
>> git filter-branch
>>
>> This rewrites the commits using any parentage changes from the grafts
>> file. This changes most commits' SHA1 hashes. After this you can
>> discard the .git/info/grafts file. You would then want to remove the
>> original references, which were moved to "refs/original".
>
> Hmm. If I need to do two changes in the same branch, do I need to
> mention the new SHA1 for the second one (after filter-branch changes its
> SHA1), or the original one? If the former, then this is going to be a
> very painful process.

No, all SHA1s refer to the values for the *old* versions of the commits.

Michael

Magnus Hagander wrote:
> Is there some way to make cvs2git work this way, and just not bother
> even trying to create merge commits, or is that fundamentally
> impossible and we need to look at another tool?

The good news: (I just reminded myself/realized that) Max Bowsher has
already implemented pretty much exactly what you want in the cvs2svn
trunk version, including noting in the commit messages any cherry-picks
that are not reflected in the repo ancestry.

The bad news: It is broken [1]. But I don't think it should be too much
work to fix it.

Michael

[1]
http://cvs2svn.tigris.org/ds/viewMessage.do?dsForumId=1670&dsMessageId=2624153

On Thu, Aug 19, 2010 at 07:00, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
> Magnus Hagander wrote:
>> Is there some way to make cvs2git work this way, and just not bother
>> even trying to create merge commits, or is that fundamentally
>> impossible and we need to look at another tool?
>
> The good news: (I just reminded myself/realized that) Max Bowsher has
> already implemented pretty much exactly what you want in the cvs2svn
> trunk version, including noting in the commit messages any cherry-picks
> that are not reflected in the repo ancestry.

Ah, that's great.

> The bad news: It is broken [1].  But I don't think it should be too much
> work to fix it.

That's less great of course, but it gives hope!

Thanks for your continued efforts!

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 19/08/10 10:35, Magnus Hagander wrote:
> On Thu, Aug 19, 2010 at 07:00, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
>> Magnus Hagander wrote:
>>> Is there some way to make cvs2git work this way, and just not bother
>>> even trying to create merge commits, or is that fundamentally
>>> impossible and we need to look at another tool?
>>
>> The good news: (I just reminded myself/realized that) Max Bowsher has
>> already implemented pretty much exactly what you want in the cvs2svn
>> trunk version, including noting in the commit messages any cherry-picks
>> that are not reflected in the repo ancestry.
>
> Ah, that's great.

I should mention that the way it notes this is to reference commits by
their timestamp, author and initial line of log message - it does this
because cvs2git doesn't know the commit sha ever - that doesn't appear
until the stream is fed through git fast-import. I did briefly raise the
idea of augmenting the fast-import process to support substituting
fast-import marks to shas in log messages, but didn't get time to take
it beyond an idea.

>> The bad news: It is broken [1]. But I don't think it should be too much
>> work to fix it.
>
> That's less great of course, but it gives hope!
>
> Thanks for your continued efforts!

I've just made a commit to cvs2svn trunk. I hope this should now be fixed.

Max.

On Fri, Aug 20, 2010 at 09:49, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 19/08/10 10:35, Magnus Hagander wrote:
>> On Thu, Aug 19, 2010 at 07:00, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
>>> Magnus Hagander wrote:
>>>> Is there some way to make cvs2git work this way, and just not bother
>>>> even trying to create merge commits, or is that fundamentally
>>>> impossible and we need to look at another tool?
>>>
>>> The good news: (I just reminded myself/realized that) Max Bowsher has
>>> already implemented pretty much exactly what you want in the cvs2svn
>>> trunk version, including noting in the commit messages any cherry-picks
>>> that are not reflected in the repo ancestry.
>>
>> Ah, that's great.
>
> I should mention that the way it notes this is to reference commits by
> their timestamp, author and initial line of log message - it does this
> because cvs2git doesn't know the commit sha ever - that doesn't appear
> until the stream is fed through git fast-import. I did briefly raise the
> idea of augmenting the fast-import process to support substituting
> fast-import marks to shas in log messages, but didn't get time to take
> it beyond an idea.
>
>>> The bad news: It is broken [1].  But I don't think it should be too much
>>> work to fix it.
>>
>> That's less great of course, but it gives hope!
>>
>> Thanks for your continued efforts!
>
> I've just made a commit to cvs2svn trunk. I hope this should now be fixed.

Great. I will download and test the trunk version soon. I'm currently
running a test using cvs2svn and then git-svn clone from that - but
it's insanely slow (been going for 30+ hours now, and probably has
8-10 hours more to go)...

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 20/08/10 12:02, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 09:49, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> On 19/08/10 10:35, Magnus Hagander wrote:
>>> On Thu, Aug 19, 2010 at 07:00, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
>>>> Magnus Hagander wrote:
>>>>> Is there some way to make cvs2git work this way, and just not bother
>>>>> even trying to create merge commits, or is that fundamentally
>>>>> impossible and we need to look at another tool?
>>>>
>>>> The good news: (I just reminded myself/realized that) Max Bowsher has
>>>> already implemented pretty much exactly what you want in the cvs2svn
>>>> trunk version, including noting in the commit messages any cherry-picks
>>>> that are not reflected in the repo ancestry.
>>>
>>> Ah, that's great.
>>
>> I should mention that the way it notes this is to reference commits by
>> their timestamp, author and initial line of log message - it does this
>> because cvs2git doesn't know the commit sha ever - that doesn't appear
>> until the stream is fed through git fast-import. I did briefly raise the
>> idea of augmenting the fast-import process to support substituting
>> fast-import marks to shas in log messages, but didn't get time to take
>> it beyond an idea.
>>
>>>> The bad news: It is broken [1]. But I don't think it should be too much
>>>> work to fix it.
>>>
>>> That's less great of course, but it gives hope!
>>>
>>> Thanks for your continued efforts!
>>
>> I've just made a commit to cvs2svn trunk. I hope this should now be fixed.
>
>
> Great. I will download and test the trunk version soon. I'm currently
> running a test using cvs2svn and then git-svn clone from that - but
> it's insanely slow (been going for 30+ hours now, and probably has
> 8-10 hours more to go)...

Uh, you are? Why do it that way?

The thing I fixed pertains to the direct use of cvs2git, and will have
no effect on executions of cvs2svn.

I have run cvs2git on the pgsql module of your CVS locally (is that the
right thing to convert?) if you'd like to compare notes on specific
parts of the conversion.

Max.

On Fri, Aug 20, 2010 at 13:50, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 20/08/10 12:02, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 09:49, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>> On 19/08/10 10:35, Magnus Hagander wrote:
>>>> On Thu, Aug 19, 2010 at 07:00, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
>>>>> Magnus Hagander wrote:
>>>>>> Is there some way to make cvs2git work this way, and just not bother
>>>>>> even trying to create merge commits, or is that fundamentally
>>>>>> impossible and we need to look at another tool?
>>>>>
>>>>> The good news: (I just reminded myself/realized that) Max Bowsher has
>>>>> already implemented pretty much exactly what you want in the cvs2svn
>>>>> trunk version, including noting in the commit messages any cherry-picks
>>>>> that are not reflected in the repo ancestry.
>>>>
>>>> Ah, that's great.
>>>
>>> I should mention that the way it notes this is to reference commits by
>>> their timestamp, author and initial line of log message - it does this
>>> because cvs2git doesn't know the commit sha ever - that doesn't appear
>>> until the stream is fed through git fast-import. I did briefly raise the
>>> idea of augmenting the fast-import process to support substituting
>>> fast-import marks to shas in log messages, but didn't get time to take
>>> it beyond an idea.
>>>
>>>>> The bad news: It is broken [1].  But I don't think it should be too much
>>>>> work to fix it.
>>>>
>>>> That's less great of course, but it gives hope!
>>>>
>>>> Thanks for your continued efforts!
>>>
>>> I've just made a commit to cvs2svn trunk. I hope this should now be fixed.
>>
>>
>> Great. I will download and test the trunk version soon. I'm currently
>> running a test using cvs2svn and then git-svn clone from that - but
>> it's insanely slow (been going for 30+ hours now, and probably has
>> 8-10 hours more to go)...
>
> Uh, you are? Why do it that way?

Trying other possible options, in case this one doesn't work out :-) I
figured I might try something while you guys were working on a fix -
didn't expect the fix to show up quite so quickly :)

> The thing I fixed pertains to the direct use of cvs2git, and will have
> no effect on executions of cvs2svn.

Right. I started this one yesterday...

> I have run cvs2git on the pgsql module of your CVS locally (is that the
> right thing to convert?) if you'd like to compare notes on specific
> parts of the conversion.

Correct, that's the one. Can you put your repo up somewhere so we can
look at it? Then I don't have to wait for my process to finish :D

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 20/08/10 12:55, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 13:50, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> I have run cvs2git on the pgsql module of your CVS locally (is that the
>> right thing to convert?) if you'd like to compare notes on specific
>> parts of the conversion.
>
> Correct, that's the one. Can you put your repo up somewhere so we can
> look at it? Then I don't have to wait for my process to finish :D

Placed at http://red-bean.com/~maxb/pgsql-test.git - about 230MB -
sorry, dumb transport only, but hopefully that's not an issue for this
use case.

Max.

On Fri, Aug 20, 2010 at 14:11, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 20/08/10 12:55, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 13:50, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>> I have run cvs2git on the pgsql module of your CVS locally (is that the
>>> right thing to convert?) if you'd like to compare notes on specific
>>> parts of the conversion.
>>
>> Correct, that's the one. Can you put your repo up somewhere so we can
>> look at it? Then I don't have to wait for my process to finish :D
>
> Placed at http://red-bean.com/~maxb/pgsql-test.git - about 230MB -
> sorry, dumb transport only, but hopefully that's not an issue for this
> use case.

It does. I've pushed up a mirror to
http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary -
that one is a lot faster to work with for me at least.

I'm also going to run my branch-verification script on it to see that
it deosn't mess any of that up - that one takes a few hours to run
(mainly the fault of the cvs we compare to :D) - I'll get back to you
when it's done.

For other who test this - it's obviously missing the author name
mapping, but that's a minor thing.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, Aug 20, 2010 at 15:04, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> On Fri, Aug 20, 2010 at 14:11, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> On 20/08/10 12:55, Magnus Hagander wrote:
>>> On Fri, Aug 20, 2010 at 13:50, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>>> I have run cvs2git on the pgsql module of your CVS locally (is that the
>>>> right thing to convert?) if you'd like to compare notes on specific
>>>> parts of the conversion.
>>>
>>> Correct, that's the one. Can you put your repo up somewhere so we can
>>> look at it? Then I don't have to wait for my process to finish :D
>>
>> Placed at http://red-bean.com/~maxb/pgsql-test.git - about 230MB -
>> sorry, dumb transport only, but hopefully that's not an issue for this
>> use case.
>
> It does. I've pushed up a mirror to
> http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary -
> that one is a lot faster to work with for me at least.
>
> I'm also going to run my branch-verification script on it to see that
> it deosn't mess any of that up - that one takes a few hours to run
> (mainly the fault of the cvs we compare to :D) - I'll get back to you
> when it's done.

That turned out to be a non-starter, since that clone doesn't have
expanded keywords. I'll run a new conversion with the same options
file used last time, and we can work off that.

I believe Robert had some comments/questions as well :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Fri, Aug 20, 2010 at 9:28 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> I believe Robert had some comments/questions as well :-)

What Magnus means is that I'm a grumpy old developer who complains
about everything.

Anyway, what I noticed was that we're getting stuff like this:

http://git.postgresql.org/gitweb?p=git-migration-test.git;a=commit;h=586b324c255a4316d72a5757566ebe6e630df47e

commit 586b324c255a4316d72a5757566ebe6e630df47e
Author: cvs2git <>
Date: Thu May 13 16:39:49 2010 +0000

This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'.

Cherrypick from master 2010-05-13 16:39:43 UTC adunstan 'Abandon the use of
src/pl/plperl/plperl_opmask.pl

We're not getting that on EVERY back-patch, just on some of them. I
really just want to turn this code to detect merges and cherry-picks
OFF altogether, so that we get the original committer and commit
message instead off the above. It's much easier to read if you're
browsing the back-branch history, and it's probably easier to match up
commits across branches, too.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Fri, Aug 20, 2010 at 2:36 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Fri, Aug 20, 2010 at 9:28 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> I believe Robert had some comments/questions as well :-)
>
> What Magnus means is that I'm a grumpy old developer who complains
> about everything.

Don't put yourself down. You're not that old :-p

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise Postgres Company

On 20/08/10 14:36, Robert Haas wrote:
> On Fri, Aug 20, 2010 at 9:28 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
>> I believe Robert had some comments/questions as well :-)
>
> What Magnus means is that I'm a grumpy old developer who complains
> about everything.
>
> Anyway, what I noticed was that we're getting stuff like this:
>
> http://git.postgresql.org/gitweb?p=git-migration-test.git;a=commit;h=586b324c255a4316d72a5757566ebe6e630df47e
>
> commit 586b324c255a4316d72a5757566ebe6e630df47e
> Author: cvs2git <>
> Date: Thu May 13 16:39:49 2010 +0000
>
> This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'.
>
> Cherrypick from master 2010-05-13 16:39:43 UTC adunstan 'Abandon the use of
> src/pl/plperl/plperl_opmask.pl
>
> We're not getting that on EVERY back-patch, just on some of them. I
> really just want to turn this code to detect merges and cherry-picks
> OFF altogether, so that we get the original committer and commit
> message instead off the above. It's much easier to read if you're
> browsing the back-branch history, and it's probably easier to match up
> commits across branches, too.

The history that cvs2svn is aiming to represent here is this:

1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
did *not* exist.

2) Later, it was added to trunk.

3) Then, someone retroactively added the branch tag to the file, marking
it as included in the REL8_4_STABLE branch. [This corresponds to the git
changeset that Robert is questioning]

4) Then, adunstan committed a change to it on the branch.

cvs2svn/git/etc seeks to faithfully represent what the result would have
been of doing a CVS checkout of the REL8_4_STABLE branch, at various
points in time, which is why this changeset is introduced.

I should also say that the autogenerated commit message is rather poor -
it should say 'update' not 'create' in this case. I'm actually looking
at fixing that.

Max.

On Fri, 2010-08-20 at 09:36 -0400, Robert Haas wrote:
> On Fri, Aug 20, 2010 at 9:28 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> > I believe Robert had some comments/questions as well :-)
>
> What Magnus means is that I'm a grumpy old developer who complains
> about everything.

+1

JD

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

On Fri, 2010-08-20 at 14:47 +0100, Dave Page wrote:
> On Fri, Aug 20, 2010 at 2:36 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> > On Fri, Aug 20, 2010 at 9:28 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> >> I believe Robert had some comments/questions as well :-)
> >
> > What Magnus means is that I'm a grumpy old developer who complains
> > about everything.
>
> Don't put yourself down. You're not that old :-p

Does that mean he is only going to get worse?

;)

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

Max Bowsher <maxb(at)f2s(dot)com> writes:
> The history that cvs2svn is aiming to represent here is this:

> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
> did *not* exist.

> 2) Later, it was added to trunk.

> 3) Then, someone retroactively added the branch tag to the file, marking
> it as included in the REL8_4_STABLE branch. [This corresponds to the git
> changeset that Robert is questioning]

Uh, no. We have never "retroactively added" anything to any branch.
I don't know enough about the innards of CVS to know what its internal
representation of this sort of thing is, but all that actually happened
here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
branch occurred. We would like the git history to look like that too.

regards, tom lane

On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>> The history that cvs2svn is aiming to represent here is this:
>
>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>> did *not* exist.
>
>> 2) Later, it was added to trunk.
>
>> 3) Then, someone retroactively added the branch tag to the file, marking
>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>> changeset that Robert is questioning]
>
> Uh, no.  We have never "retroactively added" anything to any branch.
> I don't know enough about the innards of CVS to know what its internal
> representation of this sort of thing is, but all that actually happened
> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
> branch occurred.  We would like the git history to look like that too.

Yeah.

In fact, is the only thing that's wrong here the commit message?
Because it's probably trivial to just patch that away.. Hmm, but i
guess we'd like to hav ethe actual commit message and not just another
fixed one..

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 20/08/10 18:28, Tom Lane wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>> The history that cvs2svn is aiming to represent here is this:
>
>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>> did *not* exist.
>
>> 2) Later, it was added to trunk.
>
>> 3) Then, someone retroactively added the branch tag to the file, marking
>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>> changeset that Robert is questioning]
>
> Uh, no. We have never "retroactively added" anything to any branch.
> I don't know enough about the innards of CVS to know what its internal
> representation of this sort of thing is, but all that actually happened
> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
> branch occurred. We would like the git history to look like that too.

When I try reproducing these circumstances locally, that is executing a
"cvs add" and a "cvs commit" of a file on a branch where that file
already exists on trunk, CVS writes an internal representation different
to what I see in your repository for this file.

I'm at a loss to explain how your repository came to be this way, but I
can tell you that cvs2git is faithfully rendering what your repository
says into git.

Max.

On 20/08/10 18:30, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>> The history that cvs2svn is aiming to represent here is this:
>>
>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>> did *not* exist.
>>
>>> 2) Later, it was added to trunk.
>>
>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>> changeset that Robert is questioning]
>>
>> Uh, no. We have never "retroactively added" anything to any branch.
>> I don't know enough about the innards of CVS to know what its internal
>> representation of this sort of thing is, but all that actually happened
>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>> branch occurred. We would like the git history to look like that too.
>
> Yeah.
>
> In fact, is the only thing that's wrong here the commit message?
> Because it's probably trivial to just patch that away.. Hmm, but i
> guess we'd like to hav ethe actual commit message and not just another
> fixed one..

There is no "actual commit message" - the entire changeset is
synthesized by cvs2git to represent the addition of a branch tag to the
file - i.e. the logical equivalent of a "cvs tag -b", which has no
commit message.

Max.

On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 20/08/10 18:30, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>> The history that cvs2svn is aiming to represent here is this:
>>>
>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>> did *not* exist.
>>>
>>>> 2) Later, it was added to trunk.
>>>
>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>> changeset that Robert is questioning]
>>>
>>> Uh, no.  We have never "retroactively added" anything to any branch.
>>> I don't know enough about the innards of CVS to know what its internal
>>> representation of this sort of thing is, but all that actually happened
>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>> branch occurred.  We would like the git history to look like that too.
>>
>> Yeah.
>>
>> In fact, is the only thing that's wrong here the commit message?
>> Because it's probably trivial to just patch that away.. Hmm, but i
>> guess we'd like to hav ethe actual commit message and not just another
>> fixed one..
>
> There is no "actual commit message" - the entire changeset is
> synthesized by cvs2git to represent the addition of a branch tag to the
> file - i.e. the logical equivalent of a "cvs tag -b", which has no
> commit message.

There is a commit message on the trunk when the file was added there.
Is there any chance of being able to lift that message off trunk and
just copy it into the branch, instead of saying "this is a cherry-pick
of this commit over here"?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 20/08/10 18:43, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> On 20/08/10 18:30, Magnus Hagander wrote:
>>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>>> The history that cvs2svn is aiming to represent here is this:
>>>>
>>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>>> did *not* exist.
>>>>
>>>>> 2) Later, it was added to trunk.
>>>>
>>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>>> changeset that Robert is questioning]
>>>>
>>>> Uh, no. We have never "retroactively added" anything to any branch.
>>>> I don't know enough about the innards of CVS to know what its internal
>>>> representation of this sort of thing is, but all that actually happened
>>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>>> branch occurred. We would like the git history to look like that too.
>>>
>>> Yeah.
>>>
>>> In fact, is the only thing that's wrong here the commit message?
>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>> guess we'd like to hav ethe actual commit message and not just another
>>> fixed one..
>>
>> There is no "actual commit message" - the entire changeset is
>> synthesized by cvs2git to represent the addition of a branch tag to the
>> file - i.e. the logical equivalent of a "cvs tag -b", which has no
>> commit message.
>
> There is a commit message on the trunk when the file was added there.
> Is there any chance of being able to lift that message off trunk and
> just copy it into the branch, instead of saying "this is a cherry-pick
> of this commit over here"?

It wouldn't be accurate to do so, because the synthetic commit is not
copying the entire change, only registering the addition of (in this
case) one file which happens to be part of the changeset. Please note
that there is a changeset on the branch immediately following the
synthetic one under discussion which contains the 'real' commit message
used when committing to the branch.

My guess at this point is that there may be a (very old?) version of cvs
which, when adding a file to a branch, actually misrecorded the file as
having existed on the branch from the moment it was first added to trunk
- this would explain this anomaly.

Max.

On Fri, Aug 20, 2010 at 19:56, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 20/08/10 18:43, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>> On 20/08/10 18:30, Magnus Hagander wrote:
>>>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>>>> The history that cvs2svn is aiming to represent here is this:
>>>>>
>>>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>>>> did *not* exist.
>>>>>
>>>>>> 2) Later, it was added to trunk.
>>>>>
>>>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>>>> changeset that Robert is questioning]
>>>>>
>>>>> Uh, no.  We have never "retroactively added" anything to any branch.
>>>>> I don't know enough about the innards of CVS to know what its internal
>>>>> representation of this sort of thing is, but all that actually happened
>>>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>>>> branch occurred.  We would like the git history to look like that too.
>>>>
>>>> Yeah.
>>>>
>>>> In fact, is the only thing that's wrong here the commit message?
>>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>>> guess we'd like to hav ethe actual commit message and not just another
>>>> fixed one..
>>>
>>> There is no "actual commit message" - the entire changeset is
>>> synthesized by cvs2git to represent the addition of a branch tag to the
>>> file - i.e. the logical equivalent of a "cvs tag -b", which has no
>>> commit message.
>>
>> There is a commit message on the trunk when the file was added there.
>> Is there any chance of being able to lift that message off trunk and
>> just copy it into the branch, instead of saying "this is a cherry-pick
>> of this commit over here"?
>
> It wouldn't be accurate to do so, because the synthetic commit is not
> copying the entire change, only registering the addition of (in this
> case) one file which happens to be part of the changeset. Please note
> that there is a changeset on the branch immediately following the
> synthetic one under discussion which contains the 'real' commit message
> used when committing to the branch.

Hmm. Good point.

I wonder if we should try to locate these places and fix them with git
filter-branch or rebase -i after the fact, with history rewriting.

There seem to be 57 of them.

Searching for those, I also found a bunch with the comment "Sprouted
from <branch>". What do those mean?

> My guess at this point is that there may be a (very old?) version of cvs
> which, when adding a file to a branch, actually misrecorded the file as
> having existed on the branch from the moment it was first added to trunk
> - this would explain this anomaly.

Well, the one Robert pointed to is a very recent commit. Not sure if
it uses the client version or the server version - the version on
cvs.postgresql.org is:

[mha(at)cvs ~]$ cvs --version

Concurrent Versions System (CVS) 1.11.17-FreeBSD (client/server)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Max Bowsher <maxb(at)f2s(dot)com> writes:
> My guess at this point is that there may be a (very old?) version of cvs
> which, when adding a file to a branch, actually misrecorded the file as
> having existed on the branch from the moment it was first added to trunk
> - this would explain this anomaly.

I have no idea what version of CVS is running on our master server.
I have noticed that it sometimes generates its own synthetic commit
messages for cases related to this, for example these events on HEAD:

2010-05-13 12:40 adunstan

* src/pl/plperl/sql/plperlu_plperl.sql: file plperlu_plperl.sql was
initially added on branch REL8_4_STABLE.

2010-05-13 12:40 adunstan

* src/pl/plperl/expected/plperlu_plperl.out: file
plperlu_plperl.out was initially added on branch REL8_4_STABLE.

I don't see one of these for plperl_opmask.pl in particular, so there
may be more than one anomaly involved.

However, the bottom line here is that we don't want the history that
cvs2git is preparing for these events, because it doesn't correspond to
what we did. Whether this is the "most faithful" representation of the
CVS history is academic; it simply is not reality. What we would like
is for the history to look like the file got added to the branch as of
the first commit that touched it on that branch. That is reality, as
it appears from our neck of the woods anyway.

regards, tom lane

On 20/08/10 19:07, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 19:56, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> On 20/08/10 18:43, Magnus Hagander wrote:
>>> On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>>> On 20/08/10 18:30, Magnus Hagander wrote:
>>>>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>>>>> The history that cvs2svn is aiming to represent here is this:
>>>>>>
>>>>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>>>>> did *not* exist.
>>>>>>
>>>>>>> 2) Later, it was added to trunk.
>>>>>>
>>>>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>>>>> changeset that Robert is questioning]
>>>>>>
>>>>>> Uh, no. We have never "retroactively added" anything to any branch.
>>>>>> I don't know enough about the innards of CVS to know what its internal
>>>>>> representation of this sort of thing is, but all that actually happened
>>>>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>>>>> branch occurred. We would like the git history to look like that too.
>>>>>
>>>>> Yeah.
>>>>>
>>>>> In fact, is the only thing that's wrong here the commit message?
>>>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>>>> guess we'd like to hav ethe actual commit message and not just another
>>>>> fixed one..
>>>>
>>>> There is no "actual commit message" - the entire changeset is
>>>> synthesized by cvs2git to represent the addition of a branch tag to the
>>>> file - i.e. the logical equivalent of a "cvs tag -b", which has no
>>>> commit message.
>>>
>>> There is a commit message on the trunk when the file was added there.
>>> Is there any chance of being able to lift that message off trunk and
>>> just copy it into the branch, instead of saying "this is a cherry-pick
>>> of this commit over here"?
>>
>> It wouldn't be accurate to do so, because the synthetic commit is not
>> copying the entire change, only registering the addition of (in this
>> case) one file which happens to be part of the changeset. Please note
>> that there is a changeset on the branch immediately following the
>> synthetic one under discussion which contains the 'real' commit message
>> used when committing to the branch.
>
> Hmm. Good point.
>
> I wonder if we should try to locate these places and fix them with git
> filter-branch or rebase -i after the fact, with history rewriting.
>
> There seem to be 57 of them.

It sounds cumbersome.

Michael Haggerty might be better placed than me to assess whether
eliding them within cvs2git is practically achievable.

> Searching for those, I also found a bunch with the comment "Sprouted
> from <branch>". What do those mean?

It appears as part of the description of what a synthetic branch
creation commit did, existing only to put into context the operations
that follow - i.e. the creation of the REL7_4_STABLE branch involved
sprouting from trunk, then deleting 4 files which were not included on
the branch.

The revision described in the "Sprout ..." line isn't particularly
interesting, since it's always the same as the parent of the commit -
it's just listed for symmetry with "Cherrypick ..." lines which may follow.

The presence/absence of a "Sprout ..." line indicates whether the
particular commit is the initial creation of a branch, versus the
grafting in of additional files to the branch. (The latter occurs when a
file is tagged as if it was part of the branch from the creation of the
branch, but only initially came into being *after* there were already
commits to the branch.)

>> My guess at this point is that there may be a (very old?) version of cvs
>> which, when adding a file to a branch, actually misrecorded the file as
>> having existed on the branch from the moment it was first added to trunk
>> - this would explain this anomaly.
>
> Well, the one Robert pointed to is a very recent commit. Not sure if
> it uses the client version or the server version - the version on
> cvs.postgresql.org is:
>
> [mha(at)cvs ~]$ cvs --version
>
> Concurrent Versions System (CVS) 1.11.17-FreeBSD (client/server)

Unsure, I'm afraid. Though I might try hunting through CVS's CVS.

Max.

On 20/08/10 19:30, Tom Lane wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>> My guess at this point is that there may be a (very old?) version of cvs
>> which, when adding a file to a branch, actually misrecorded the file as
>> having existed on the branch from the moment it was first added to trunk
>> - this would explain this anomaly.
>
> I have no idea what version of CVS is running on our master server.
> I have noticed that it sometimes generates its own synthetic commit
> messages for cases related to this, for example these events on HEAD:
>
> 2010-05-13 12:40 adunstan
>
> * src/pl/plperl/sql/plperlu_plperl.sql: file plperlu_plperl.sql was
> initially added on branch REL8_4_STABLE.
>
> 2010-05-13 12:40 adunstan
>
> * src/pl/plperl/expected/plperlu_plperl.out: file
> plperlu_plperl.out was initially added on branch REL8_4_STABLE.

This is actually what's supposed to occur, and cvs2git will elide these
synthetic entries, which exist to represent the concept of adding a file
to a branch after the initial creation of the branch, within the fairly
arcane constraints of the RCS file format.

> I don't see one of these for plperl_opmask.pl in particular, so there
> may be more than one anomaly involved.

Just the one anomaly - the absence of one of those for plperl_opmask.pl
is the original anomaly.

> However, the bottom line here is that we don't want the history that
> cvs2git is preparing for these events, because it doesn't correspond to
> what we did. Whether this is the "most faithful" representation of the
> CVS history is academic; it simply is not reality. What we would like
> is for the history to look like the file got added to the branch as of
> the first commit that touched it on that branch. That is reality, as
> it appears from our neck of the woods anyway.

Michael, what's your take on this? I have a feeling that such a thing is
*not* going to be a quick hack in cvs2svn.

Max.

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> In fact, is the only thing that's wrong here the commit message?
> Because it's probably trivial to just patch that away.. Hmm, but i
> guess we'd like to hav ethe actual commit message and not just another
> fixed one..

If I understand Max's statements correctly, there is an observable
problem in the actual git history, not just the commit log entries:
it will believe that a file added on a branch had been there since
the branch forked off, not just as of the time it got added.

Now, I would think that your tests of file contents as of the various
release tags should have caught extra files, so maybe I'm
misunderstanding.

regards, tom lane

On Fri, Aug 20, 2010 at 20:52, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> In fact, is the only thing that's wrong here the commit message?
>> Because it's probably trivial to just patch that away.. Hmm, but i
>> guess we'd like to hav ethe actual commit message and not just another
>> fixed one..
>
> If I understand Max's statements correctly, there is an observable
> problem in the actual git history, not just the commit log entries:
> it will believe that a file added on a branch had been there since
> the branch forked off, not just as of the time it got added.
>
> Now, I would think that your tests of file contents as of the various
> release tags should have caught extra files, so maybe I'm
> misunderstanding.

I haven't been able to complete that test on the repo converted by the
new version yet, because the repo Max prepared for us had the keyword
problem. The other process is still running.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 20/08/10 19:54, Magnus Hagander wrote:
> On Fri, Aug 20, 2010 at 20:52, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> In fact, is the only thing that's wrong here the commit message?
>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>> guess we'd like to hav ethe actual commit message and not just another
>>> fixed one..
>>
>> If I understand Max's statements correctly, there is an observable
>> problem in the actual git history, not just the commit log entries:
>> it will believe that a file added on a branch had been there since
>> the branch forked off, not just as of the time it got added.

Not since the branch forked off, but rather it will believe the file
added to the branch from the moment it was added to trunk - the issue is
actually in the cvs repository too - were you to ask CVS for the state
of the branch at the relevant time, you'd see the extra file there too.

In the specific case we've been looking at so far, the file is only
appearing less than a minute prematurely.

>> Now, I would think that your tests of file contents as of the various
>> release tags should have caught extra files, so maybe I'm
>> misunderstanding.
>
> I haven't been able to complete that test on the repo converted by the
> new version yet, because the repo Max prepared for us had the keyword
> problem. The other process is still running.

Would it help at all for you to send me the options file and related
file so I can produce a repository converted as you are expecting?

Max.

On Fri, Aug 20, 2010 at 21:22, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 20/08/10 19:54, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 20:52, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>> In fact, is the only thing that's wrong here the commit message?
>>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>>> guess we'd like to hav ethe actual commit message and not just another
>>>> fixed one..
>>>
>>> If I understand Max's statements correctly, there is an observable
>>> problem in the actual git history, not just the commit log entries:
>>> it will believe that a file added on a branch had been there since
>>> the branch forked off, not just as of the time it got added.
>
> Not since the branch forked off, but rather it will believe the file
> added to the branch from the moment it was added to trunk - the issue is
> actually in the cvs repository too - were you to ask CVS for the state
> of the branch at the relevant time, you'd see the extra file there too.
>
> In the specific case we've been looking at so far, the file is only
> appearing less than a minute prematurely.

Yeah, that's because in our "backpatching" we generally do them at the
same time, so cvs2cl will pick it up. E.g. you modify all the branches
and have a script commit to them all with the same commit message.

>>> Now, I would think that your tests of file contents as of the various
>>> release tags should have caught extra files, so maybe I'm
>>> misunderstanding.
>>
>> I haven't been able to complete that test on the repo converted by the
>> new version yet, because the repo Max prepared for us had the keyword
>> problem. The other process is still running.
>
> Would it help at all for you to send me the options file and related
> file so I can produce a repository converted as you are expecting?

In fact, the conversion *just* finished. I'm running the comparison
script now. It's at least looking reasonably right - no changes in
REL6_4. It'll take a while for it to finish on the rest... This, in
fact, means that it's doing better than version 2.3.0 with regards to
the small issues with had with vendor branches as well, which is good
news (see other threads in the archives).

That said, the options file is certainly not secret. I've sent the one
used for 2.3.0 before, here's the one I used for trunk (trunk of
cvs2git).

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Max Bowsher <maxb(at)f2s(dot)com> writes:
>> On Fri, Aug 20, 2010 at 20:52, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> If I understand Max's statements correctly, there is an observable
>>> problem in the actual git history, not just the commit log entries:
>>> it will believe that a file added on a branch had been there since
>>> the branch forked off, not just as of the time it got added.

> Not since the branch forked off, but rather it will believe the file
> added to the branch from the moment it was added to trunk - the issue is
> actually in the cvs repository too - were you to ask CVS for the state
> of the branch at the relevant time, you'd see the extra file there too.

Ah. So Magnus' tests didn't catch that because he only looked at
release tag times, and none of these event pairs occurred across a
release.

> In the specific case we've been looking at so far, the file is only
> appearing less than a minute prematurely.

Hmm. I wonder whether the "anomaly" is dependent on the order in which
the cvs add's and cvs commit's are done in the two different branches.

I'm still confused as to why this results in such massive weirdness in
the generated git history, though. If it simply caused an extra commit
that adds the new file slightly earlier than the commit we think of as
adding the file, I wouldn't be complaining. It's the fact that there
are all those unrelated HEAD commits showing up in the log for a branch
that bugs me.

regards, tom lane

On 20/08/10 21:08, Tom Lane wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>> On Fri, Aug 20, 2010 at 20:52, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> If I understand Max's statements correctly, there is an observable
>>>> problem in the actual git history, not just the commit log entries:
>>>> it will believe that a file added on a branch had been there since
>>>> the branch forked off, not just as of the time it got added.
>
>> Not since the branch forked off, but rather it will believe the file
>> added to the branch from the moment it was added to trunk - the issue is
>> actually in the cvs repository too - were you to ask CVS for the state
>> of the branch at the relevant time, you'd see the extra file there too.
>
> Ah. So Magnus' tests didn't catch that because he only looked at
> release tag times, and none of these event pairs occurred across a
> release.
>
>> In the specific case we've been looking at so far, the file is only
>> appearing less than a minute prematurely.
>
> Hmm. I wonder whether the "anomaly" is dependent on the order in which
> the cvs add's and cvs commit's are done in the two different branches.
>
> I'm still confused as to why this results in such massive weirdness in
> the generated git history, though. If it simply caused an extra commit
> that adds the new file slightly earlier than the commit we think of as
> adding the file, I wouldn't be complaining.

Isn't this what's happening?

> It's the fact that there
> are all those unrelated HEAD commits showing up in the log for a branch
> that bugs me.

You mean in the synthetic log message? Well, they're not exactly
unrelated - the overall effect is that the file was added on trunk,
'merged' into the branch, and then modified appropriately for that branch.

Max.

Max Bowsher <maxb(at)f2s(dot)com> writes:
> On 20/08/10 21:08, Tom Lane wrote:
>> I'm still confused as to why this results in such massive weirdness in
>> the generated git history, though. If it simply caused an extra commit
>> that adds the new file slightly earlier than the commit we think of as
>> adding the file, I wouldn't be complaining.

> Isn't this what's happening?

Uh, no, the excitement is about this:
http://git.postgresql.org/gitweb?p=postgresql-migration.git;a=shortlog;h=refs/tags/REL8_3_10

There are a whole lot of commits listed there that have nothing to do
with anything that ever happened on the 8.3 branch.

regards, tom lane

On Fri, Aug 20, 2010 at 4:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>> On 20/08/10 21:08, Tom Lane wrote:
>>> I'm still confused as to why this results in such massive weirdness in
>>> the generated git history, though.  If it simply caused an extra commit
>>> that adds the new file slightly earlier than the commit we think of as
>>> adding the file, I wouldn't be complaining.
>
>> Isn't this what's happening?
>
> Uh, no, the excitement is about this:
> http://git.postgresql.org/gitweb?p=postgresql-migration.git;a=shortlog;h=refs/tags/REL8_3_10
>
> There are a whole lot of commits listed there that have nothing to do
> with anything that ever happened on the 8.3 branch.

Tom,

The problem you are looking at here has been fixed. We are looking at
a different problem now. See:

http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

* Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> [100820 16:28]:

> Uh, no, the excitement is about this:
> http://git.postgresql.org/gitweb?p=postgresql-migration.git;a=shortlog;h=refs/tags/REL8_3_10
>
> There are a whole lot of commits listed there that have nothing to do
> with anything that ever happened on the 8.3 branch.

Sure, but "gitweb" isn't doing a good job of showing you the branches &
merges. If you use a tool that can show you the brances and merges,
what you'll see is that the the "merges" are all cases where commits
happend to all branches, and cvs2svn/cvs2git is trying to gropu them
together.

So, the real "git repository" history looks something like:

master --A___B___C__C___C__F__G__H__I___
8.4 \___a_____________\__f______

But gitweb is showing you (for 8.4) something like
f
F
E
D
C
B
a
A

If you view it in something like gitk:
gitk --date-order origin/REL8_3_STABLE origin/REL8_4_STABLE origin/master
you can see much more clearly that it's just synthisizing merge commtis to combine changes
from the 1st commited change (usually in HEAD/master) to the branches.

Take a look:

Author: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:39:43
Committer: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:39:43
Parent: 63c9dfe37db08b03ebbb91a96814c685a80ed257 (Assorted fixes to make pg_upgrade build on MSVC.)
Child: 4a34da914360d7d3edaa26e9a0242be74a7d90ea (Prevent PL/Tcl from loading the "unknown" module from pltcl_modules unless)
Child: d8597e4d2896841437e9cfe12f403c83021d6f29 (This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'.)
Child: a18a19c0b0d62e8a707755ff4b0125b67eb8f7ea (This commit was manufactured by cvs2svn to create branch 'REL8_3_STABLE'.)
Branches: master, remotes/origin/REL7_4_STABLE, remotes/origin/REL8_0_STABLE, remotes/origin/REL8_1_STABLE, remotes/origin/REL8_2_STABLE, remotes/origin/REL8_3_STABLE, remotes/origin/REL8_4_STABLE, remotes/origin/REL9_0_STABLE, remotes/origin/foobranch, remotes/origin/master
Follows: REL9_0_BETA1
Precedes: REL7_4_29, REL8_0_25, REL8_1_21, REL8_2_17, REL8_3_11, REL8_4_4, REL9_0_BETA2

Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is
fundamentally insecure. Instead apply an opmask to the whole interpreter that
imposes restrictions on unsafe operations. These restrictions are much harder
to subvert than is Safe.pm, since there is no container to be broken out of.
Backported to release 7.4.

In releases 7.4, 8.0 and 8.1 this also includes the necessary backporting of
the two interpreters model for plperl and plperlu adopted in release 8.2.

In versions 8.0 and up, the use of Perl's POSIX module to undo its locale
mangling on Windows has become insecure with these changes, so it is
replaced by our own routine, which is also faster.

Nice side effects of the changes include that it is now possible to use perl's
"strict" pragma in a natural way in plperl, and that perl's $a and
$b variables now work as expected in sort routines, and that function
compilation is significantly faster.

Tim Bunce and Andrew Dunstan, with reviews from Alex Hunsaker and
Alexey Klyukin.

Security: CVE-2010-1169

So, this was where Andrew committed this fix to cvs HEAD, and then immediately
afterwards to all the other branches. But in the other branches, cvs2git
"merges" this commit it, masaghing the tree to be able todo that ;-(

In 8.4, we see:
Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org> 2010-05-13 12:39:49
Committer: PostgreSQL Daemon <webmaster(at)postgresql(dot)org> 2010-05-13 12:39:49
Parent: b62e4ea4823443ac6cbd01e14ccbc4010159490b (Fix some spelling errors.)
Parent: 97dfa4bccb5d7a2d5951b60b3f4e122b633126d5 (Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is)
Child: 851d3e0a9b9214337f96b65c39c43feae272daad (Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is)
Branch: remotes/origin/REL8_4_STABLE
Follows: REL8_4_2, REL9_0_BETA1
Precedes: REL8_4_4

This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'.

Author: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:40:36
Committer: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:40:36
Parent: d8597e4d2896841437e9cfe12f403c83021d6f29 (This commit was manufactured by cvs2svn to create branch 'REL8_4_STABLE'.)
Child: 3b0a36679590e6db1d93cc2c4631f910d789c286 (Prevent PL/Tcl from loading the "unknown" module from pltcl_modules unless)
Branch: remotes/origin/REL8_4_STABLE
Follows: REL8_4_2, REL9_0_BETA1
Precedes: REL8_4_4

Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is
fundamentally insecure. Instead apply an opmask to the whole interpreter that
imposes restrictions on unsafe operations. These restrictions are much harder
to subvert than is Safe.pm, since there is no container to be broken out of.
Backported to release 7.4.

In releases 7.4, 8.0 and 8.1 this also includes the necessary backporting of
the two interpreters model for plperl and plperlu adopted in release 8.2.

In versions 8.0 and up, the use of Perl's POSIX module to undo its locale
mangling on Windows has become insecure with these changes, so it is
replaced by our own routine, which is also faster.

Nice side effects of the changes include that it is now possible to use perl's
"strict" pragma in a natural way in plperl, and that perl's $a and
$b variables now work as expected in sort routines, and that function
compilation is significantly faster.

Tim Bunce and Andrew Dunstan, with reviews from Alex Hunsaker and
Alexey Klyukin.

In 8.3, we see:
Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org> 2010-05-13 12:39:48
Committer: PostgreSQL Daemon <webmaster(at)postgresql(dot)org> 2010-05-13 12:39:48
Parent: 210e6b90636a734ac43de2ff0b882497527ce489 (Fix some spelling errors.)
Parent: 97dfa4bccb5d7a2d5951b60b3f4e122b633126d5 (Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is)
Child: 0dc3ceaf04e89c759b46570a885eb04ca1980eef (Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is)
Branch: remotes/origin/REL8_3_STABLE
Follows: REL8_3_10, REL9_0_BETA1
Precedes: REL8_3_11

This commit was manufactured by cvs2svn to create branch 'REL8_3_STABLE'.

Author: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:42:51
Committer: Andrew Dunstan <andrew(at)dunslane(dot)net> 2010-05-13 12:42:51
Parent: a18a19c0b0d62e8a707755ff4b0125b67eb8f7ea (This commit was manufactured by cvs2svn to create branch 'REL8_3_STABLE'.)
Child: 20b4301ee7bda4ff5d1d46deddd318c964ca1cc8 (Prevent PL/Tcl from loading the "unknown" module from pltcl_modules unless)
Branch: remotes/origin/REL8_3_STABLE
Follows: REL8_3_10, REL9_0_BETA1
Precedes: REL8_3_11

Abandon the use of Perl's Safe.pm to enforce restrictions in plperl, as it is
fundamentally insecure. Instead apply an opmask to the whole interpreter that
imposes restrictions on unsafe operations. These restrictions are much harder
to subvert than is Safe.pm, since there is no container to be broken out of.
Backported to release 7.4.

In releases 7.4, 8.0 and 8.1 this also includes the necessary backporting of
the two interpreters model for plperl and plperlu adopted in release 8.2.

In versions 8.0 and up, the use of Perl's POSIX module to undo its locale
mangling on Windows has become insecure with these changes, so it is
replaced by our own routine, which is also faster.

Nice side effects of the changes include that it is now possible to use perl's
"strict" pragma in a natural way in plperl, and that perl's $a and
$b variables now work as expected in sort routines, and that function
compilation is significantly faster.

Tim Bunce and Andrew Dunstan, with reviews from Alex Hunsaker and
Alexey Klyukin.

Security: CVE-2010-1169

*but*, since it's "merged in", if you look at gitweb's history of 8.3, or 8.4,
you see the complete history of HEAD/master in as well (or, at least up to the latest
merge into the branch). That's because gitweb is showing you a "graph" of
histroy as a single long list.

So, I'm agreeing, it's not the history that PostgreSQL wants in it's git repo,
but it is self-consistent with what cvs2svn/cvs2git is trying to do - group
non-atomic changes done with the same "commit message" and close enough
timestamps to a non-atomic CVS repository together as atomic changesets in a
atomic git repository. And it's having to synthisize these commits to be able
to "link" the commits together.

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, Aug 20, 2010 at 4:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> There are a whole lot of commits listed there that have nothing to do
>> with anything that ever happened on the 8.3 branch.

> The problem you are looking at here has been fixed. We are looking at
> a different problem now. See:
> http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary

Ah, my apologies for the noise then. I confess to not having been
paying close attention to the git thread, but in a quick read-through
I didn't see any statement that the problem had changed.

regards, tom lane

On Fri, Aug 20, 2010 at 23:39, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Fri, Aug 20, 2010 at 4:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> There are a whole lot of commits listed there that have nothing to do
>>> with anything that ever happened on the 8.3 branch.
>
>> The problem you are looking at here has been fixed.  We are looking at
>> a different problem now.  See:
>> http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary
>
> Ah, my apologies for the noise then.  I confess to not having been
> paying close attention to the git thread, but in a quick read-through
> I didn't see any statement that the problem had changed.

I have now pushed a complete copy of the latest migrated repository to
http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary.

This one has tkey keyword expansion on, which we decided we want. My
script that compares branch tips and tags to cvs now shows *zero*
differences. Which in itself is an improvement over the old version of
cvs2git :-)

I have not checked the state of the "newly added files issue" that
Robert found, nor in general verified anything other than branch tips
and tags so far. Anybody who has time to do that, please go right
ahead :-)

If you pulled from this repository before, it' sbeen completely wiped
and replaced, so you'll need to do that to your clone as well.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> I have now pushed a complete copy of the latest migrated repository to
> http://git.postgresql.org/gitweb?p=git-migration-test.git;a=summary.

> This one has tkey keyword expansion on, which we decided we want. My
> script that compares branch tips and tags to cvs now shows *zero*
> differences. Which in itself is an improvement over the old version of
> cvs2git :-)

Cool --- this alone is probably worth the delay in converting.

> I have not checked the state of the "newly added files issue" that
> Robert found, nor in general verified anything other than branch tips
> and tags so far. Anybody who has time to do that, please go right
> ahead :-)

I just spent some time comparing the REL8_3_STABLE history to what
I have from cvs2cl. It is *much* better --- no more unrelated commits.
I do see the "manufactured commits" that Robert is on about, but what
is now apparent is that those correspond to artifact commits on the CVS
side too. For example, here's what cvs2cl claims happened in the 8.3
branch on Feb 28 (all times EST = GMT-5):

2010-02-28 22:41 tgl

* contrib/xml2/: Makefile, xpath.c, xslt_proc.c, expected/xml2.out,
sql/xml2.sql (REL8_3_STABLE): Back-patch today's memory management
fixups in contrib/xml2.

Prior to 8.3, these changes are not critical for compatibility with
core Postgres, since core had no libxml2 calls then. However there
is still a risk if contrib/xml2 is used along with libxml2
functionality in Perl or other loadable modules. So back-patch to
all versions.

Also back-patch addition of regression tests. I'm not sure how
many of the cases are interesting without the interaction with core
xml code, but a silly regression test is still better than none at
all.

2010-02-28 21:21 tgl

* src/: backend/access/transam/xact.c, backend/utils/adt/xml.c,
include/utils/xml.h (REL8_3_STABLE): Back-patch changes of
2009-05-13 in xml.c's memory management.

I was afraid to do this when these changes were first made, but now
that 8.4 has seen some field use it should be all right to
back-patch. These changes are really quite necessary in order to
give xml.c any hope of co-existing with loadable modules that also
wish to use libxml2.

2010-02-28 16:31 tgl

* contrib/xml2/: expected/xml2.out, sql/xml2.sql: Fix up memory
management problems in contrib/xml2.

Get rid of the code that attempted to funnel libxml2's memory
allocations into palloc. We already knew from experience with the
core xml datatype that trying to do this is simply not reliable.
Unlike the core code, I did not bother adding a lot of
PG_TRY/PG_CATCH logic to try to ensure that everything is cleaned
up on error exit. Hence, we might leak some memory if one of these
functions fails partway through. Given the deprecated status of
this contrib module and the fact that errors partway through the
functions shouldn't be too common, it doesn't seem worth worrying
about.

Also fix a separate bug in xpath_table, that it did the wrong
things if given a result tuple descriptor with less than 2 columns.
While such a case isn't very useful in practice, we shouldn't fail
or stomp memory when it occurs.

Add some simple regression tests based on all the reported crash
cases that I have on hand.

This should be back-patched, but let's see if the buildfarm likes
it first.

Notice that that last entry doesn't say (REL8_3_STABLE). Which is
correct, because *there was no such commit against 8.3*. This entry is
quoting the commit message, and the commit time, of the HEAD commit that
added xml2.sql and xml2.out --- and notice it only lists those two
files, not the other ones touched by that HEAD commit.

In the git conversion, there is a "manufactured commit" corresponding to
this one, although the timestamp seems slightly different, and it
appears to inject the HEAD versions of these test files into the branch.
Then in the later git commit corresponding to the first cvs2cl entry
above, there is a diff that makes the test files match the way they
really look in 8.3.

I suspect that this happened every time we did a back-branch file
addition, and that in most cases the oddity got masked because the HEAD
and back-branch commits happened at approximately the same time and with
identical commit messages. So they got folded into one report by
cvs2cl. In this example, with the messages being different and a few
hours elapsed between the commits, we can see that some weirdness did
happen in the CVS history too. This also becomes apparent when you
look at cvsweb:
http://anoncvs.postgresql.org/cvsweb.cgi/pgsql/contrib/xml2/sql/xml2.sql
The back-branch versions of the file are shown as being updates from the
HEAD version 1.1, which is surely not the way things happened in
reality, but ...

So at this point I'm willing to buy Max and Michael's assertion that
this is a faithful conversion of the CVS history. The fact that the
commit messages are tagged as manufactured seems like it might be a good
thing not a bad thing --- they're manufactured on the CVS side too.

We need to do more testing of this conversion, but right at the moment
I'm thinking it might be OK as-is.

regards, tom lane

Max Bowsher wrote:
> On 20/08/10 19:07, Magnus Hagander wrote:
>> On Fri, Aug 20, 2010 at 19:56, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>> On 20/08/10 18:43, Magnus Hagander wrote:
>>>> On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>>>> On 20/08/10 18:30, Magnus Hagander wrote:
>>>>>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>>>>>> The history that cvs2svn is aiming to represent here is this:
>>>>>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>>>>>> did *not* exist.
>>>>>>>> 2) Later, it was added to trunk.
>>>>>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>>>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>>>>>> changeset that Robert is questioning]
>>>>>>> Uh, no. We have never "retroactively added" anything to any branch.
>>>>>>> I don't know enough about the innards of CVS to know what its internal
>>>>>>> representation of this sort of thing is, but all that actually happened
>>>>>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>>>>>> branch occurred. We would like the git history to look like that too.
>>>>>> Yeah.
>>>>>>
>>>>>> In fact, is the only thing that's wrong here the commit message?
>>>>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>>>>> guess we'd like to hav ethe actual commit message and not just another
>>>>>> fixed one..
>>>>> There is no "actual commit message" - the entire changeset is
>>>>> synthesized by cvs2git to represent the addition of a branch tag to the
>>>>> file - i.e. the logical equivalent of a "cvs tag -b", which has no
>>>>> commit message.
>>>> There is a commit message on the trunk when the file was added there.
>>>> Is there any chance of being able to lift that message off trunk and
>>>> just copy it into the branch, instead of saying "this is a cherry-pick
>>>> of this commit over here"?
>>> It wouldn't be accurate to do so, because the synthetic commit is not
>>> copying the entire change, only registering the addition of (in this
>>> case) one file which happens to be part of the changeset. Please note
>>> that there is a changeset on the branch immediately following the
>>> synthetic one under discussion which contains the 'real' commit message
>>> used when committing to the branch.
>> Hmm. Good point.
>>
>> I wonder if we should try to locate these places and fix them with git
>> filter-branch or rebase -i after the fact, with history rewriting.
>>
>> There seem to be 57 of them.
>
> It sounds cumbersome.
>
> Michael Haggerty might be better placed than me to assess whether
> eliding them within cvs2git is practically achievable.

I think this would be nontrivial.

It is (relatively) easy to tweak a file's history during
FilterSymbolsPass, which is the last time during the conversion when the
file's whole history is in memory at once. But you don't want to omit
all connections between file-on-branch and parent branch; you only want
to omit the information if the branching of the particular file cannot
be included with the first commit that creates the branch.
Unfortunately, determination of commits requires *global* information
and is done *after* FilterSymbolsPass.

The elision of the file branching event could conceivably be done at the
point when it would otherwise be output to the dumpfile, but its elision
would affect how the first change to the file on the branch had to be
treated, so information would have to be kept around.

Moreover, this is a pretty specialized request that would be useless to
people who are not so disciplined about their repository as you seem to be.

It seems like you already have a way to find these events in the git
repository after conversion, so I think it would be more practical to
use git-filter-branch to remove the unwanted commits *after* the conversion.

Michael

On Sat, Aug 21, 2010 at 08:15, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
> Max Bowsher wrote:
>> On 20/08/10 19:07, Magnus Hagander wrote:
>>> On Fri, Aug 20, 2010 at 19:56, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>>> On 20/08/10 18:43, Magnus Hagander wrote:
>>>>> On Fri, Aug 20, 2010 at 19:41, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>>>>> On 20/08/10 18:30, Magnus Hagander wrote:
>>>>>>> On Fri, Aug 20, 2010 at 19:28, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>>>>>> Max Bowsher <maxb(at)f2s(dot)com> writes:
>>>>>>>>> The history that cvs2svn is aiming to represent here is this:
>>>>>>>>> 1) At the time of creation of the REL8_4_STABLE branch, plperl_opmask.pl
>>>>>>>>> did *not* exist.
>>>>>>>>> 2) Later, it was added to trunk.
>>>>>>>>> 3) Then, someone retroactively added the branch tag to the file, marking
>>>>>>>>> it as included in the REL8_4_STABLE branch. [This corresponds to the git
>>>>>>>>> changeset that Robert is questioning]
>>>>>>>> Uh, no.  We have never "retroactively added" anything to any branch.
>>>>>>>> I don't know enough about the innards of CVS to know what its internal
>>>>>>>> representation of this sort of thing is, but all that actually happened
>>>>>>>> here was a "cvs add" and a "cvs commit" in REL8_4_STABLE long after the
>>>>>>>> branch occurred.  We would like the git history to look like that too.
>>>>>>> Yeah.
>>>>>>>
>>>>>>> In fact, is the only thing that's wrong here the commit message?
>>>>>>> Because it's probably trivial to just patch that away.. Hmm, but i
>>>>>>> guess we'd like to hav ethe actual commit message and not just another
>>>>>>> fixed one..
>>>>>> There is no "actual commit message" - the entire changeset is
>>>>>> synthesized by cvs2git to represent the addition of a branch tag to the
>>>>>> file - i.e. the logical equivalent of a "cvs tag -b", which has no
>>>>>> commit message.
>>>>> There is a commit message on the trunk when the file was added there.
>>>>> Is there any chance of being able to lift that message off trunk and
>>>>> just copy it into the branch, instead of saying "this is a cherry-pick
>>>>> of this commit over here"?
>>>> It wouldn't be accurate to do so, because the synthetic commit is not
>>>> copying the entire change, only registering the addition of (in this
>>>> case) one file which happens to be part of the changeset. Please note
>>>> that there is a changeset on the branch immediately following the
>>>> synthetic one under discussion which contains the 'real' commit message
>>>> used when committing to the branch.
>>> Hmm. Good point.
>>>
>>> I wonder if we should try to locate these places and fix them with git
>>> filter-branch or rebase -i after the fact, with history rewriting.
>>>
>>> There seem to be 57 of them.
>>
>> It sounds cumbersome.
>>
>> Michael Haggerty might be better placed than me to assess whether
>> eliding them within cvs2git is practically achievable.
>
> I think this would be nontrivial.

<snip>

> Moreover, this is a pretty specialized request that would be useless to
> people who are not so disciplined about their repository as you seem to be.

Yeah, I think we're unusually disciplined with our repository - that's
one reason the change won't be that drastic wrt how things are done
:-)

> It seems like you already have a way to find these events in the git
> repository after conversion, so I think it would be more practical to
> use git-filter-branch to remove the unwanted commits *after* the conversion.

Not sure that we do have an automated way, but I agree that this is
probably going to be easier to do with git-filter-branch.

If we need to do it at all. Tom's latest lookover indicates that he
thinks it may be good the way it is, and we need some more detailed
checks. I know Robert has said he wants to dedicate some time to doing
such checks this week, and I'll see if I can find some time for that
as well. If anybody else would like to help us dig through mainly the
backbranches - with focus on branchpoints and taggings - to look for
any kind of "weird stuff" (meaning anything that's not a straight
commit), then please do so and let us know your results!

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Mon, Aug 23, 2010 at 4:50 AM, Magnus Hagander <magnus(at)hagander(dot)net> wrote:
> If we need to do it at all. Tom's latest lookover indicates that he
> thinks it may be good the way it is, and we need some more detailed
> checks. I know Robert has said he wants to dedicate some time to doing
> such checks this week, and I'll see if I can find some time for that
> as well. If anybody else would like to help us dig through mainly the
> backbranches - with focus on branchpoints and taggings - to look for
> any kind of "weird stuff" (meaning anything that's not a straight
> commit), then please do so and let us know your results!

So far I've found a couple of minor issues by comparing 'git log
master' on the current, incremental conversion with the
git-migration-test repo (incidentally, what happened to discipline in
naming these repos?).

1. The new conversion seems to have stolen the apostrophe from "D'Arcy
J.M. Cain <darcy(at)druid(dot)net>", rendering him "DArcy J.M. Cain
<darcy(at)druid(dot)net>".

2. Any non-ASCII characters in, for example, contributor's names show
up differently in the two repos. Generally, the original repo is OK
and the new repo is garbled; although I found one very old example
that went the other way.

There are also a number of commits that differ in order between the
two repos, and an even larger number where commits are duplicated or
merged in one repository relative to the other. So far, all the
examples I've checked have appeared to be saner in the new repository
than in the old one, but I have not done a full audit.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Fri, Aug 20, 2010 at 1:56 PM, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> My guess at this point is that there may be a (very old?) version of cvs
> which, when adding a file to a branch, actually misrecorded the file as
> having existed on the branch from the moment it was first added to trunk
> - this would explain this anomaly.

I think this is what is happening, except I'm unable to account for it
by the age of the CVS version we're runnning. The machine the CVS
repo is running on is running 1.11.17-FreeBSD (client/server). I
don't know how long it's been that way, but there are examples of this
in the relatively recent past - like July 2nd of this year. I am 100%
positive that what I did was 'cvs add' one new file, 'cvs delete' one
old file, modify a few other things, and commit the whole deal. But
in the git conversion there are two commits, one of which adds a copy
of the file as it exists in HEAD and the other of which contains the
balance of the changes. Every recent manufactured commit is of this
same form: it immediately precedes the commit of which (in my view) it
should be considered a part.

Looking back a bit further in history, there is some stranger stuff.

commit ec0274633871c43da670fa90d0ac4fd7090639f2
Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org>
Date: Mon Jun 6 16:30:43 2005 +0000

This commit was manufactured by cvs2svn to create branch 'REL8_0_STABLE'.

Cherrypick from master 2005-06-06 16:30:42 UTC Bruce Momjian <bruce(at)momjian(dot)
doc/src/FAQ/FAQ_hungarian.html

And then, much later, the following completely empty commit:

commit 446b749c2eaeff3c0611d33bc12b3df28e2cf8fa
Author: Bruce Momjian <bruce(at)momjian(dot)us>
Date: Tue Oct 4 14:17:44 2005 +0000

Add FAQ_hungarian.html to 8.0.X branch.

What really happened is:

http://archives.postgresql.org/pgsql-committers/2005-10/msg00044.php

So that's pretty much the same thing, except the time lag between the
two commits that should be married is much larger.

The odder cases are the ones involving deletion. There are a couple
of branches/tags that, or so I'm guessing, are only present for a
subset of the files in the repository: ecpg_big_bison, creation,
Release-1-6-0, MANUAL_1_0, REL2_0B, and SUPPORT. I'm wondering if we
shouldn't just nuke those, or at least nuke them from the copy of the
repository upon which we are running the conversion.

This series of commits also seems pretty messed up:

http://archives.postgresql.org/pgsql-committers/2007-04/msg00222.php
http://archives.postgresql.org/pgsql-committers/2007-04/msg00223.php

The commit messages make it clear that CVS did something funky,
although it's not exactly clear retrospectively what it was. At any
rate, it's evidently still not right, because in the converted
repository we get a whole slough of commits like this:

commit c50da22b6050e0bdd5e2ef97541d91aa1d2e63fb
Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org>
Date: Sat Dec 2 08:36:42 2006 +0000

This commit was manufactured by cvs2svn to create branch 'REL8_2_STABLE'.

Sprout from master 2006-12-02 08:36:41 UTC PostgreSQL Daemon <webmaster(at)post
Delete:
src/backend/parser/gram.c
src/interfaces/ecpg/preproc/pgc.c
src/interfaces/ecpg/preproc/preproc.c

There are similar (but separate) commits for tag REL8_2_RC1,
REL8_2_BETA3, REL8_2_BETA2, REL8_2_BETA1, REL8_1_STABLE, REL8_1_0_RC1,
REL8_1_0BETA4, REL8_1_0BETA3, REL8_1_0BETA2, REL8_1_0BETA1, REL8_0_0,
REL8_0_0RC5, REL8_0_0RC4, REL8_0_0RC3, REL8_0_0RC2, REL8_0_0RC1,
REL8_0_0BETA5, REL8_0_0BETA4, REL8_0_0BETA3, REL8_0_0BETA2,
REL8_0_0BETA1, REL7_4_STABLE, REL7_4_BETA5, REL7_4_BETA4,
REL7_4_BETA3, REL7_4_BETA2, REL7_4_BETA1, REL7_2_STABLE, REL7_2,
REL7_2_RC2, REL7_2_RC1, REL7_2_BETA5, REL7_2_BETA4, REL7_2_BETA3,
REL7_2_BETA2, REL7_2_BETA1, REL7_1_STABLE, REL7_1_BETA3, REL7_1_BETA2,
REL7_0_PATCHES, REL7_0, REL6_5_PATCHES, and release-6-3. That's
pretty crazy. I think we should try to do something to clean this up,
perhaps by doctoring the file on the CVS side.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> On Fri, Aug 20, 2010 at 1:56 PM, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> My guess at this point is that there may be a (very old?) version of cvs
>> which, when adding a file to a branch, actually misrecorded the file as
>> having existed on the branch from the moment it was first added to trunk
>> - this would explain this anomaly.

> I think this is what is happening, except I'm unable to account for it
> by the age of the CVS version we're runnning. The machine the CVS
> repo is running on is running 1.11.17-FreeBSD (client/server).

Um, how old do you think that is? A look at the cvs sources says 2004...

It looks to me like the bogus commits for back-branch additions are
indeed part of our CVS history. While perhaps it would be nice if the
git conversion cleaned them up, I'm not sure that we want to put off
doing the conversion for however long it might take to make that happen.

> The odder cases are the ones involving deletion. There are a couple
> of branches/tags that, or so I'm guessing, are only present for a
> subset of the files in the repository: ecpg_big_bison, creation,
> Release-1-6-0, MANUAL_1_0, REL2_0B, and SUPPORT. I'm wondering if we
> shouldn't just nuke those, or at least nuke them from the copy of the
> repository upon which we are running the conversion.

Yeah, I noticed some of those in my copy of the test repository too,
but I see a slightly different set:

remotes/origin/REL2_0B
remotes/origin/REL6_4
remotes/origin/Release_1_0_3
remotes/origin/WIN32_DEV
remotes/origin/ecpg_big_bison

I doubt they're of any more than archaeological interest, but do we want
to be deleting history? What seemed more likely to be artifacts were
these:

remotes/origin/unlabeled-1.44.2
remotes/origin/unlabeled-1.51.2
remotes/origin/unlabeled-1.59.2
remotes/origin/unlabeled-1.87.2
remotes/origin/unlabeled-1.90.2

Any idea where those came from?

> This series of commits also seems pretty messed up:
> http://archives.postgresql.org/pgsql-committers/2007-04/msg00222.php
> http://archives.postgresql.org/pgsql-committers/2007-04/msg00223.php

You can find out about the reasons for that in this *other* discussion
of conversion to git:
http://archives.postgresql.org/pgsql-hackers/2007-04/msg00670.php
particularly here:
http://archives.postgresql.org/pgsql-hackers/2007-04/msg00685.php

> ... pretty crazy. I think we should try to do something to clean this up,
> perhaps by doctoring the file on the CVS side.

On the whole I feel that you're moving the goalposts. AFAIR the agreed
criteria for an acceptable SCM conversion were that it reproduce the
historical states of our tree at least at all the release tags, and that
it provide a close approximation of the CVS commit logs. I think that
manufactured commits that correspond to CVS's artifacts might be a bit
ugly, but trying to get rid of them sounds way too much like putting
lipstick on a pig. And if it means removing real, if ugly, history,
I'm not sure I'm in favor of it at all.

regards, tom lane

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> 1. The new conversion seems to have stolen the apostrophe from "D'Arcy
> J.M. Cain <darcy(at)druid(dot)net>", rendering him "DArcy J.M. Cain
> <darcy(at)druid(dot)net>".

Yeah, I see that too. It's probably bad input rather than the
converter's fault ;-)

> 2. Any non-ASCII characters in, for example, contributor's names show
> up differently in the two repos. Generally, the original repo is OK
> and the new repo is garbled; although I found one very old example
> that went the other way.

What it looks like to me is that a Latin1->UTF8 conversion has been
applied to the log text. Which might be a good idea if it all *was*
Latin1, but a fair-sized percentage isn't. Applying this conversion to
UTF8 entries results in garbage, of course. Even if this could be done
reliably, I think this counts as editorializing on the historical
record, and should be switched off if possible.

> There are also a number of commits that differ in order between the
> two repos, and an even larger number where commits are duplicated or
> merged in one repository relative to the other.

I suspect that this is an artifact of the converter trying to merge
nearby commits into one commit, which it more or less *has* to do for
sanity since CVS commits aren't atomic. I don't have a problem with
the concept, but I notice cases where the converted commit has a
timestamp some minutes later than what the cvs2cl output claims.
I suspect this is what the converter was using as a cutoff time.
Would it be possible to make sure that the converted commit is always
timestamped with the latest individual file update timestamp from the
included CVS commits?

regards, tom lane

On Wed, Aug 25, 2010 at 07:11, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> 1. The new conversion seems to have stolen the apostrophe from "D'Arcy
>> J.M. Cain <darcy(at)druid(dot)net>", rendering him "DArcy J.M. Cain
>> <darcy(at)druid(dot)net>".
>
> Yeah, I see that too.  It's probably bad input rather than the
> converter's fault ;-)

indeed. Wrong type of escaping. For some reason I used '' when I
should've used \'. I wonder where I got that idea :D

>> 2. Any non-ASCII characters in, for example, contributor's names show
>> up differently in the two repos.  Generally, the original repo is OK
>> and the new repo is garbled; although I found one very old example
>> that went the other way.
>
> What it looks like to me is that a Latin1->UTF8 conversion has been
> applied to the log text.  Which might be a good idea if it all *was*
> Latin1, but a fair-sized percentage isn't.  Applying this conversion to
> UTF8 entries results in garbage, of course.  Even if this could be done
> reliably, I think this counts as editorializing on the historical
> record, and should be switched off if possible.

I think the problem is that we have a mix of them :( git requires it to be utf8.

cvs2git is configured to try, in order, latin1, utf8 and ascii, and
use whichever first returns correct result. In this case it seems it
does return saying things are right, because the result is valid utf8
- just not the utf8 we expected.

I can give it a try the other way around - trying utf8 *before*
latin1, to see if that makes it better - utf8 tends to be more strict.

>> There are also a number of commits that differ in order between the
>> two repos, and an even larger number where commits are duplicated or
>> merged in one repository relative to the other.
>
> I suspect that this is an artifact of the converter trying to merge
> nearby commits into one commit, which it more or less *has* to do for
> sanity since CVS commits aren't atomic.  I don't have a problem with
> the concept, but I notice cases where the converted commit has a
> timestamp some minutes later than what the cvs2cl output claims.
> I suspect this is what the converter was using as a cutoff time.
> Would it be possible to make sure that the converted commit is always
> timestamped with the latest individual file update timestamp from the
> included CVS commits?

I can't comment o nthis part - Michael or Max?

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 25/08/10 09:18, Magnus Hagander wrote:
> On Wed, Aug 25, 2010 at 07:11, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:

>>> 2. Any non-ASCII characters in, for example, contributor's names show
>>> up differently in the two repos. Generally, the original repo is OK
>>> and the new repo is garbled; although I found one very old example
>>> that went the other way.
>>
>> What it looks like to me is that a Latin1->UTF8 conversion has been
>> applied to the log text. Which might be a good idea if it all *was*
>> Latin1, but a fair-sized percentage isn't. Applying this conversion to
>> UTF8 entries results in garbage, of course. Even if this could be done
>> reliably, I think this counts as editorializing on the historical
>> record, and should be switched off if possible.
>
> I think the problem is that we have a mix of them :( git requires it to be utf8.
>
> cvs2git is configured to try, in order, latin1, utf8 and ascii, and
> use whichever first returns correct result. In this case it seems it
> does return saying things are right, because the result is valid utf8
> - just not the utf8 we expected.
>
> I can give it a try the other way around - trying utf8 *before*
> latin1, to see if that makes it better - utf8 tends to be more strict.

*Every* byte sequence is valid latin1, therefore if you try latin1,
utf8, ascii in that order, latin1 will always be used.

You most likely want utf8, latin1 (no point also including ascii since
it's a strict subset of latin1).

>>> There are also a number of commits that differ in order between the
>>> two repos, and an even larger number where commits are duplicated or
>>> merged in one repository relative to the other.
>>
>> I suspect that this is an artifact of the converter trying to merge
>> nearby commits into one commit, which it more or less *has* to do for
>> sanity since CVS commits aren't atomic. I don't have a problem with
>> the concept, but I notice cases where the converted commit has a
>> timestamp some minutes later than what the cvs2cl output claims.
>> I suspect this is what the converter was using as a cutoff time.
>> Would it be possible to make sure that the converted commit is always
>> timestamped with the latest individual file update timestamp from the
>> included CVS commits?
>
> I can't comment o nthis part - Michael or Max?

cvs2git will try to use the timestamps from the commits, but sometimes
the ordering of how revisions and tags relate to each other will
actually disagree with the timestamps. In such a case, cvs2git nudges
commit timestamps forward in time, to force the defined temporal
ordering into consistency with the topological ordering of events.

In other words, no, you can't make cvs2git *always* use the timestamp
from a cvs commit, but it should have a good reason for doing so when it
deviates from that.

Max.

On Wed, Aug 25, 2010 at 13:03, Max Bowsher <maxb(at)f2s(dot)com> wrote:
> On 25/08/10 09:18, Magnus Hagander wrote:
>> On Wed, Aug 25, 2010 at 07:11, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>
>>>> 2. Any non-ASCII characters in, for example, contributor's names show
>>>> up differently in the two repos.  Generally, the original repo is OK
>>>> and the new repo is garbled; although I found one very old example
>>>> that went the other way.
>>>
>>> What it looks like to me is that a Latin1->UTF8 conversion has been
>>> applied to the log text.  Which might be a good idea if it all *was*
>>> Latin1, but a fair-sized percentage isn't.  Applying this conversion to
>>> UTF8 entries results in garbage, of course.  Even if this could be done
>>> reliably, I think this counts as editorializing on the historical
>>> record, and should be switched off if possible.
>>
>> I think the problem is that we have a mix of them :( git requires it to be utf8.
>>
>> cvs2git is configured to try, in order, latin1, utf8 and ascii, and
>> use whichever first returns correct result. In this case it seems it
>> does return saying things are right, because the result is valid utf8
>> - just not the utf8 we expected.
>>
>> I can give it a try the other way around - trying utf8 *before*
>> latin1, to see if that makes it better - utf8 tends to be more strict.
>
> *Every* byte sequence is valid latin1, therefore if you try latin1,
> utf8, ascii in that order, latin1 will always be used.
>
> You most likely want utf8, latin1 (no point also including ascii since
> it's a strict subset of latin1).

Yup. I re-ran it with utf8, latin1, ascii and that commit looks better now.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On Tue, Aug 24, 2010 at 11:21 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> On Fri, Aug 20, 2010 at 1:56 PM, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>>> My guess at this point is that there may be a (very old?) version of cvs
>>> which, when adding a file to a branch, actually misrecorded the file as
>>> having existed on the branch from the moment it was first added to trunk
>>> - this would explain this anomaly.
>
>> I think this is what is happening, except I'm unable to account for it
>> by the age of the CVS version we're runnning.  The machine the CVS
>> repo is running on is running 1.11.17-FreeBSD (client/server).
>
> Um, how old do you think that is?  A look at the cvs sources says 2004...

Oh, really? I didn't look that carefully; I just checked the date on
the download directory, which was 2008. But I guess the actual code
is older.

>> The odder cases are the ones involving deletion.  There are a couple
>> of branches/tags that, or so I'm guessing, are only present for a
>> subset of the files in the repository: ecpg_big_bison, creation,
>> Release-1-6-0, MANUAL_1_0, REL2_0B, and SUPPORT.  I'm wondering if we
>> shouldn't just nuke those, or at least nuke them from the copy of the
>> repository upon which we are running the conversion.
>
> Yeah, I noticed some of those in my copy of the test repository too,
> but I see a slightly different set:
>
>  remotes/origin/REL2_0B
>  remotes/origin/REL6_4
>  remotes/origin/Release_1_0_3
>  remotes/origin/WIN32_DEV
>  remotes/origin/ecpg_big_bison
>
> I doubt they're of any more than archaeological interest, but do we want
> to be deleting history?

Well, I think what those represent are partial tags. git has no
equivalent, so anything that pops out this way is going to be totally
wacko. We're not really deleting history; we're just declining to
convert things that git can't represent accurately. It is sort of an
interesting question why REL6_4 would fall into this category, but I
can't imagine we care about any of the other ones. And if we do,
well, we're not deleting the CVS tree.

> What seemed more likely to be artifacts were
> these:
>
>  remotes/origin/unlabeled-1.44.2
>  remotes/origin/unlabeled-1.51.2
>  remotes/origin/unlabeled-1.59.2
>  remotes/origin/unlabeled-1.87.2
>  remotes/origin/unlabeled-1.90.2
>
> Any idea where those came from?

No; I don't see anything like that. What command did you run?

>> This series of commits also seems pretty messed up:
>> http://archives.postgresql.org/pgsql-committers/2007-04/msg00222.php
>> http://archives.postgresql.org/pgsql-committers/2007-04/msg00223.php
>
> You can find out about the reasons for that in this *other* discussion
> of conversion to git:
> http://archives.postgresql.org/pgsql-hackers/2007-04/msg00670.php
> particularly here:
> http://archives.postgresql.org/pgsql-hackers/2007-04/msg00685.php
>
>> ... pretty crazy.  I think we should try to do something to clean this up,
>> perhaps by doctoring the file on the CVS side.
>
> On the whole I feel that you're moving the goalposts.  AFAIR the agreed
> criteria for an acceptable SCM conversion were that it reproduce the
> historical states of our tree at least at all the release tags, and that
> it provide a close approximation of the CVS commit logs.  I think that
> manufactured commits that correspond to CVS's artifacts might be a bit
> ugly, but trying to get rid of them sounds way too much like putting
> lipstick on a pig.  And if it means removing real, if ugly, history,
> I'm not sure I'm in favor of it at all.

Well, when did it become a goal to get this git conversion done as
soon as humanly possible? We *cannot* retroactively fix these issues
after the conversion is done; or at least not without rewriting the
entire repository history, which is something we do not want to do
lightly - it is a major inconvenience for anyone who has already
cloned, and particularly for, ahem, any companies that might be
merging off of the repo. I don't think we should decide that we're
unwilling to fix these issues without even discussing whether that's
feasible or what would be involved. I don't think we're talking about
removing history; I think we're talking about cleaning up corruption
in CVS that will be irretrievably baked-in by the conversion.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On 25/08/10 01:15, Robert Haas wrote:
> On Fri, Aug 20, 2010 at 1:56 PM, Max Bowsher <maxb(at)f2s(dot)com> wrote:
>> My guess at this point is that there may be a (very old?) version of cvs
>> which, when adding a file to a branch, actually misrecorded the file as
>> having existed on the branch from the moment it was first added to trunk
>> - this would explain this anomaly.
>
> I think this is what is happening, except I'm unable to account for it
> by the age of the CVS version we're runnning. The machine the CVS
> repo is running on is running 1.11.17-FreeBSD (client/server). I
> don't know how long it's been that way, but there are examples of this
> in the relatively recent past - like July 2nd of this year. I am 100%
> positive that what I did was 'cvs add' one new file, 'cvs delete' one
> old file, modify a few other things, and commit the whole deal. But
> in the git conversion there are two commits, one of which adds a copy
> of the file as it exists in HEAD and the other of which contains the
> balance of the changes. Every recent manufactured commit is of this
> same form: it immediately precedes the commit of which (in my view) it
> should be considered a part.
>
> Looking back a bit further in history, there is some stranger stuff.
>
> commit ec0274633871c43da670fa90d0ac4fd7090639f2
> Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org>
> Date: Mon Jun 6 16:30:43 2005 +0000
>
> This commit was manufactured by cvs2svn to create branch 'REL8_0_STABLE'.
>
> Cherrypick from master 2005-06-06 16:30:42 UTC Bruce Momjian <bruce(at)momjian(dot)
> doc/src/FAQ/FAQ_hungarian.html
>
> And then, much later, the following completely empty commit:
>
> commit 446b749c2eaeff3c0611d33bc12b3df28e2cf8fa
> Author: Bruce Momjian <bruce(at)momjian(dot)us>
> Date: Tue Oct 4 14:17:44 2005 +0000
>
> Add FAQ_hungarian.html to 8.0.X branch.
>
> What really happened is:
>
> http://archives.postgresql.org/pgsql-committers/2005-10/msg00044.php
>
> So that's pretty much the same thing, except the time lag between the
> two commits that should be married is much larger.

Yup, exact same problem, the file was added to the branch, and CVS
erroneously recorded that it *had existed on the branch* from the moment
it was created on trunk.

> The odder cases are the ones involving deletion. There are a couple
> of branches/tags that, or so I'm guessing, are only present for a
> subset of the files in the repository: ecpg_big_bison, creation,
> Release-1-6-0, MANUAL_1_0, REL2_0B, and SUPPORT. I'm wondering if we
> shouldn't just nuke those, or at least nuke them from the copy of the
> repository upon which we are running the conversion.

Well, I'd caution against being too revisionist with your history, but
if you're convinced you want to drop certain tags/branches, you can
configure cvs2git to ignore them (see the symbol strategy rules part of
the options file).

Max.

On 25/08/10 14:03, Max Bowsher wrote:
> On 25/08/10 09:18, Magnus Hagander wrote:
>> On Wed, Aug 25, 2010 at 07:11, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Robert Haas<robertmhaas(at)gmail(dot)com> writes:
>>>> There are also a number of commits that differ in order between the
>>>> two repos, and an even larger number where commits are duplicated or
>>>> merged in one repository relative to the other.
>>>
>>> I suspect that this is an artifact of the converter trying to merge
>>> nearby commits into one commit, which it more or less *has* to do for
>>> sanity since CVS commits aren't atomic. I don't have a problem with
>>> the concept, but I notice cases where the converted commit has a
>>> timestamp some minutes later than what the cvs2cl output claims.
>>> I suspect this is what the converter was using as a cutoff time.
>>> Would it be possible to make sure that the converted commit is always
>>> timestamped with the latest individual file update timestamp from the
>>> included CVS commits?
>>
>> I can't comment o nthis part - Michael or Max?
>
> cvs2git will try to use the timestamps from the commits, but sometimes
> the ordering of how revisions and tags relate to each other will
> actually disagree with the timestamps. In such a case, cvs2git nudges
> commit timestamps forward in time, to force the defined temporal
> ordering into consistency with the topological ordering of events.

Hmm, why does it force that consistency? AFAIK git is happy with a
commit with an older timestamp following a commit with a newer timestamp.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

On Wed, Aug 25, 2010 at 13:19, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> What seemed more likely to be artifacts were
>> these:
>>
>>  remotes/origin/unlabeled-1.44.2
>>  remotes/origin/unlabeled-1.51.2
>>  remotes/origin/unlabeled-1.59.2
>>  remotes/origin/unlabeled-1.87.2
>>  remotes/origin/unlabeled-1.90.2
>>
>> Any idea where those came from?
>
> No; I don't see anything like that.  What command did you run?

They were the originally. I later removed them from the repo - I bet
Tom just managed to clone before I did.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

On 25/08/10 04:21, Tom Lane wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:

> What seemed more likely to be artifacts were
> these:
>
> remotes/origin/unlabeled-1.44.2
> remotes/origin/unlabeled-1.51.2
> remotes/origin/unlabeled-1.59.2
> remotes/origin/unlabeled-1.87.2
> remotes/origin/unlabeled-1.90.2
>
> Any idea where those came from?

These occur when there are numbered revisions in one or more RCS files,
which lack a branch tag to identify their name. The most likely cause is
deleting a branch after having committed to it.

Indeed, all of these five correspond to a commit with the message:

Make the world at least somewhat safe for zero-column tables, and
remove the special case in ALTER DROP COLUMN to prohibit dropping a
table's last column.

I have an idea you can fix this by running the following on your live
CVS repository:

cvs rtag -D "2002-09-23 20:43:41 UTC" zero-column-tables pgsql
cvs rtag -F -B -r 1.44.2 zero-column-tables \
pgsql/src/backend/commands/tablecmds.c
cvs rtag -F -B -r 1.90.2 zero-column-tables \
pgsql/src/backend/parser/parse_target.c
cvs rtag -F -B -r 1.90.2 zero-column-tables \
pgsql/src/backend/access/common/tupdesc.c
cvs rtag -F -B -r 1.59.2 zero-column-tables \
pgsql/src/backend/executor/execTuples.c
cvs rtag -F -B -r 1.87.2 zero-column-tables \
pgsql/src/backend/executor/nodeAgg.c,v
cvs rtag -F -B -r 1.51.2 zero-column-tables \
pgsql/src/test/regress/expected/alter_table.out

(Untested as yet, I have a test conversion running.)

>> This series of commits also seems pretty messed up:
>> http://archives.postgresql.org/pgsql-committers/2007-04/msg00222.php
>> http://archives.postgresql.org/pgsql-committers/2007-04/msg00223.php
>
> You can find out about the reasons for that in this *other* discussion
> of conversion to git:
> http://archives.postgresql.org/pgsql-hackers/2007-04/msg00670.php
> particularly here:
> http://archives.postgresql.org/pgsql-hackers/2007-04/msg00685.php
>
>> ... pretty crazy. I think we should try to do something to clean this up,
>> perhaps by doctoring the file on the CVS side.
>
> On the whole I feel that you're moving the goalposts. AFAIR the agreed
> criteria for an acceptable SCM conversion were that it reproduce the
> historical states of our tree at least at all the release tags, and that
> it provide a close approximation of the CVS commit logs. I think that
> manufactured commits that correspond to CVS's artifacts might be a bit
> ugly, but trying to get rid of them sounds way too much like putting
> lipstick on a pig. And if it means removing real, if ugly, history,
> I'm not sure I'm in favor of it at all.

I'm mostly with Tom on this one. Basically you are now discovering what
a mess CVS has made. The mess has always existed, but only now do you
have the tools to notice this.

Your options are:

1) Accept that.

2) Retroactively modify history to say that those generated files NEVER
existed in the repository.

3) Retroactively modify history to say that those generated files are
actually included in all those release tags.

Max.

On 25/08/10 12:36, Heikki Linnakangas wrote:
> On 25/08/10 14:03, Max Bowsher wrote:
>> On 25/08/10 09:18, Magnus Hagander wrote:
>>> On Wed, Aug 25, 2010 at 07:11, Tom Lane<tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>>> Robert Haas<robertmhaas(at)gmail(dot)com> writes:
>>>>> There are also a number of commits that differ in order between the
>>>>> two repos, and an even larger number where commits are duplicated or
>>>>> merged in one repository relative to the other.
>>>>
>>>> I suspect that this is an artifact of the converter trying to merge
>>>> nearby commits into one commit, which it more or less *has* to do for
>>>> sanity since CVS commits aren't atomic. I don't have a problem with
>>>> the concept, but I notice cases where the converted commit has a
>>>> timestamp some minutes later than what the cvs2cl output claims.
>>>> I suspect this is what the converter was using as a cutoff time.
>>>> Would it be possible to make sure that the converted commit is always
>>>> timestamped with the latest individual file update timestamp from the
>>>> included CVS commits?
>>>
>>> I can't comment o nthis part - Michael or Max?
>>
>> cvs2git will try to use the timestamps from the commits, but sometimes
>> the ordering of how revisions and tags relate to each other will
>> actually disagree with the timestamps. In such a case, cvs2git nudges
>> commit timestamps forward in time, to force the defined temporal
>> ordering into consistency with the topological ordering of events.
>
> Hmm, why does it force that consistency? AFAIK git is happy with a
> commit with an older timestamp following a commit with a newer timestamp.

Um. Good point. Why do enforce that?

Michael, do you think anything would break if we just removed the
"ensure monotonicity" code?

Max.

Max Bowsher <maxb(at)f2s(dot)com> writes:
> On 25/08/10 12:36, Heikki Linnakangas wrote:
>> On 25/08/10 14:03, Max Bowsher wrote:
>>> cvs2git will try to use the timestamps from the commits, but sometimes
>>> the ordering of how revisions and tags relate to each other will
>>> actually disagree with the timestamps. In such a case, cvs2git nudges
>>> commit timestamps forward in time, to force the defined temporal
>>> ordering into consistency with the topological ordering of events.
>>
>> Hmm, why does it force that consistency? AFAIK git is happy with a
>> commit with an older timestamp following a commit with a newer timestamp.

> Um. Good point. Why do enforce that?

> Michael, do you think anything would break if we just removed the
> "ensure monotonicity" code?

Yes, the cases that I noticed all had to do with some curious condition,
like a time-extended CVS commit overlapping with another one on a
disjoint set of files. (The sets of files had to be disjoint or CVS
would have failed one commit at some point.) AFAICS there is no reason
the git conversion can't arbitrarily choose one order or the other, and
I would like it to choose an order based on real file commit timestamps
rather than made-up ones.

Some other cases that I noticed involved these manufactured commits that
we've been whining about --- the "real" commit that straightens things
out tends to be displaced by a minute or so, to no purpose whatsoever
since in most cases there are no nearby commits.

regards, tom lane

Max Bowsher wrote:
> On 25/08/10 12:36, Heikki Linnakangas wrote:
>> On 25/08/10 14:03, Max Bowsher wrote:
>>> cvs2git will try to use the timestamps from the commits, but sometimes
>>> the ordering of how revisions and tags relate to each other will
>>> actually disagree with the timestamps. In such a case, cvs2git nudges
>>> commit timestamps forward in time, to force the defined temporal
>>> ordering into consistency with the topological ordering of events.
>> Hmm, why does it force that consistency? AFAIK git is happy with a
>> commit with an older timestamp following a commit with a newer timestamp.
>
> Um. Good point. Why do enforce that?

Shallow answers:

* It was adopted from cvs2svn, where timestamp monotonicity is not quite
required but definitely advantageous.

* Non-monotonic timestamps give one a spooky feeling of time travel.

Deeper answers:

* Even though git is tolerant of timestamps that are out of order, that
doesn't mean that they are desirable. The most common reason for
out-of-order CVS timestamps is that the CVS clients and/or server had
clocks that were incorrect. (Several projects have reported that their
server had a dead CMOS battery, causing the clock to be reset to 1970
for a while.) So often the enforced-monotonic timestamps produced by
cvs2svn are an improvement on the CVS timestamps.

* CVS (when functioning correctly) cannot generate events that are out
of chronological order. Therefore, non-chronological events ipso facto
represent repository corruption and it would be silly to try to preserve
them. (The exception is that cvs2svn might combine commits too
aggressively within its 5-minute timestamp window.)

So I think that it makes sense to keep at least part of the "ensure
monotonicity" behavior.

However, there is one big difference between Subversion and git:
Subversion requires a total ordering of commits, whereas git only
requires a topological ordering. Currently, the "ensure monotonicity"
code is applied after the commits have been totally ordered. Therefore,
any mistakes made in choosing a total order among those consistent with
the topological ordering constraints can lead to monotonicity fixups
that are not justified by the topology. It might make sense, in the
case of DVCSs, to fix up timestamps at an earlier step in the conversion.

> Michael, do you think anything would break if we just removed the
> "ensure monotonicity" code?

No. It might be interesting to turn it off and see where the
differences appear.

Michael

Max Bowsher <maxb(at)f2s(dot)com> writes:
> On 25/08/10 04:21, Tom Lane wrote:
>> What seemed more likely to be artifacts were these:
>>
>> remotes/origin/unlabeled-1.44.2
>> remotes/origin/unlabeled-1.51.2
>> remotes/origin/unlabeled-1.59.2
>> remotes/origin/unlabeled-1.87.2
>> remotes/origin/unlabeled-1.90.2
>>
>> Any idea where those came from?

> These occur when there are numbered revisions in one or more RCS files,
> which lack a branch tag to identify their name. The most likely cause is
> deleting a branch after having committed to it.

> Indeed, all of these five correspond to a commit with the message:

> Make the world at least somewhat safe for zero-column tables, and
> remove the special case in ALTER DROP COLUMN to prohibit dropping a
> table's last column.

It seems likely to me that this has something to do with the aborted
early branch for 7.4 development:
http://archives.postgresql.org/pgsql-hackers/2002-09/msg01733.php

If you read that thread you'll find an agreement that we'd continue
development on HEAD and then do a mega back-patch into REL7_3_STABLE,
but there is no mega back-patch later in the CVS logs. What actually
happened is explained here:
http://archives.postgresql.org/pgsql-hackers/2002-11/msg00113.php

The first actual commit into REL7_3_STABLE that cvs2cl finds is
a mass delete pursuant to my comment there. I am not sure exactly
what Marc did to "move the REL7_3_STABLE tag up to today", but I'll
bet that the funny state of the 2002-09-28 commit has something to
do with that, as it was the first commit into HEAD after Marc
originally established the REL7_3_STABLE branch.

Max's proposed fix seems to involve recognizing those extra versions
as a legitimate branch, which I think we don't really want. It'd be
better if we deleted them.

regards, tom lane

On 25/08/10 16:43, Tom Lane wrote:
> Max Bowsher <maxb(at)f2s(dot)com> writes:
>> On 25/08/10 04:21, Tom Lane wrote:
>>> What seemed more likely to be artifacts were these:
>>>
>>> remotes/origin/unlabeled-1.44.2
>>> remotes/origin/unlabeled-1.51.2
>>> remotes/origin/unlabeled-1.59.2
>>> remotes/origin/unlabeled-1.87.2
>>> remotes/origin/unlabeled-1.90.2
>>>
>>> Any idea where those came from?
>
>> These occur when there are numbered revisions in one or more RCS files,
>> which lack a branch tag to identify their name. The most likely cause is
>> deleting a branch after having committed to it.
>
>> Indeed, all of these five correspond to a commit with the message:
>
>> Make the world at least somewhat safe for zero-column tables, and
>> remove the special case in ALTER DROP COLUMN to prohibit dropping a
>> table's last column.
>
> It seems likely to me that this has something to do with the aborted
> early branch for 7.4 development:
> http://archives.postgresql.org/pgsql-hackers/2002-09/msg01733.php
>
> If you read that thread you'll find an agreement that we'd continue
> development on HEAD and then do a mega back-patch into REL7_3_STABLE,
> but there is no mega back-patch later in the CVS logs. What actually
> happened is explained here:
> http://archives.postgresql.org/pgsql-hackers/2002-11/msg00113.php
>
> The first actual commit into REL7_3_STABLE that cvs2cl finds is
> a mass delete pursuant to my comment there. I am not sure exactly
> what Marc did to "move the REL7_3_STABLE tag up to today", but I'll
> bet that the funny state of the 2002-09-28 commit has something to
> do with that, as it was the first commit into HEAD after Marc
> originally established the REL7_3_STABLE branch.
>
> Max's proposed fix seems to involve recognizing those extra versions
> as a legitimate branch, which I think we don't really want. It'd be
> better if we deleted them.

In that case, either employ an ExcludeRegexpStrategyRule('unlabeled-.*')
in the cvs2git options file, or drop those refs after converting to git.

Max.

Robert Haas wrote:
> This series of commits also seems pretty messed up:
>
> http://archives.postgresql.org/pgsql-committers/2007-04/msg00222.php
> http://archives.postgresql.org/pgsql-committers/2007-04/msg00223.php
>
> The commit messages make it clear that CVS did something funky,
> although it's not exactly clear retrospectively what it was. At any
> rate, it's evidently still not right, because in the converted
> repository we get a whole slough of commits like this:
>
> commit c50da22b6050e0bdd5e2ef97541d91aa1d2e63fb
> Author: PostgreSQL Daemon <webmaster(at)postgresql(dot)org>
> Date: Sat Dec 2 08:36:42 2006 +0000
>
> This commit was manufactured by cvs2svn to create branch 'REL8_2_STABLE'.
>
> Sprout from master 2006-12-02 08:36:41 UTC PostgreSQL Daemon <webmaster(at)post
> Delete:
> src/backend/parser/gram.c
> src/interfaces/ecpg/preproc/pgc.c
> src/interfaces/ecpg/preproc/preproc.c
>
> There are similar (but separate) commits for tag REL8_2_RC1,
> REL8_2_BETA3, REL8_2_BETA2, REL8_2_BETA1, REL8_1_STABLE, REL8_1_0_RC1,
> REL8_1_0BETA4, REL8_1_0BETA3, REL8_1_0BETA2, REL8_1_0BETA1, REL8_0_0,
> REL8_0_0RC5, REL8_0_0RC4, REL8_0_0RC3, REL8_0_0RC2, REL8_0_0RC1,
> REL8_0_0BETA5, REL8_0_0BETA4, REL8_0_0BETA3, REL8_0_0BETA2,
> REL8_0_0BETA1, REL7_4_STABLE, REL7_4_BETA5, REL7_4_BETA4,
> REL7_4_BETA3, REL7_4_BETA2, REL7_4_BETA1, REL7_2_STABLE, REL7_2,
> REL7_2_RC2, REL7_2_RC1, REL7_2_BETA5, REL7_2_BETA4, REL7_2_BETA3,
> REL7_2_BETA2, REL7_2_BETA1, REL7_1_STABLE, REL7_1_BETA3, REL7_1_BETA2,
> REL7_0_PATCHES, REL7_0, REL6_5_PATCHES, and release-6-3. That's
> pretty crazy. I think we should try to do something to clean this up,
> perhaps by doctoring the file on the CVS side.

This is probably caused by cvs2svn's failure to consider file deletions
when choosing the best revision from which to branch [1]. It would be
better to branch all of these symbols from the commit *after* the files
were deleted, which would make them all exact copies of the original
(rather than requiring a fixup branch).

I don't think that this can be fixed by doctoring the CVS repository (at
least, not short of removing the three files from the entire project
history). It could be fixed post-conversion by using grafts, or by
shifting the tags and rebasing the branches.

I must say, it is refreshing to have users who actually care about their
conversion, as opposed to the usual rabble who think that git-cvsimport
is Just Fine :-) I guess if the postgresql project didn't care about
data integrity then we would all have to worry :-)

Michael

[1] http://cvs2svn.tigris.org/issues/show_bug.cgi?id=55

On Wed, Aug 25, 2010 at 12:02 PM, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:
>> I think we should try to do something to clean this up,
>> perhaps by doctoring the file on the CVS side.
>
> This is probably caused by cvs2svn's failure to consider file deletions
> when choosing the best revision from which to branch [1].  It would be
> better to branch all of these symbols from the commit *after* the files
> were deleted, which would make them all exact copies of the original
> (rather than requiring a fixup branch).
>
> I don't think that this can be fixed by doctoring the CVS repository (at
> least, not short of removing the three files from the entire project
> history).  It could be fixed post-conversion by using grafts, or by
> shifting the tags and rebasing the branches.

Well, the history here is pretty weird. In relevant part, here's the
result of cvs log on src/backend/parser/gram.c:

revision 2.92
date: 2007/04/17 01:06:27; author: tgl; state: dead; lines: +0 -0
And remove 'em again ...
----------------------------
revision 2.91
date: 2007/04/17 01:05:07; author: tgl; state: Exp; lines: +0 -12088
Temporarily re-add derived files, in hopes of straightening out their
CVS status.
----------------------------
revision 2.90
date: 1999/05/07 01:22:54; author: vadim; state: Exp; lines: +6001 -5942
branches: 2.90.2;
Fix LMGR for MVCC.
Get rid of Extend lock mode.
----------------------------
revision 2.89
date: 1999/03/28 20:32:04; author: vadim; state: Exp; lines: +3292 -3225
1. Vacuum is updated for MVCC.
2. Much faster btree tuples deletion in the case when first on page
index tuple is deleted (no movement to the left page(s)).
3. Remember blkno of new root page in BTPageOpaque of
left/right siblings when root page is splitted.
----------------------------
revision 2.88
date: 1999/03/20 18:43:49; author: tgl; state: dead; lines: +1 -1
Remove yacc/lex output files from CVS repository.

The fact that the file was "modified" twice after being removed at rev
2.88 seems really wacko. Are you sure that's not contributing to what
we're seeing here?

> I must say, it is refreshing to have users who actually care about their
> conversion, as opposed to the usual rabble who think that git-cvsimport
> is Just Fine :-)  I guess if the postgresql project didn't care about
> data integrity then we would all have to worry :-)

I laughed when I read this - yeah, we're kind of paranoid about that.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Wed, Aug 25, 2010 at 5:35 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Well, the history here is pretty weird.  In relevant part, here's the
> result of cvs log on src/backend/parser/gram.c:
>

Interestingly this weirdness first surfaced due to a previous
discussion of using git about 3 and a half years ago:

http://thread.gmane.org/gmane.comp.db.postgresql.devel.general/80800/focus=80809

--
greg

Robert Haas <robertmhaas(at)gmail(dot)com> writes:
> The fact that the file was "modified" twice after being removed at rev
> 2.88 seems really wacko. Are you sure that's not contributing to what
> we're seeing here?

Yeah, that was discussed in the earlier git-conversion thread that I
pointed to. We never did figure out how that happened, though I
speculated it might have been due to weirdness in Vadim's local
checkout.

Is it possible to just delete those two revisions from the CVS
repository, and if so would it help? We certainly don't need 'em.

regards, tom lane

Robert Haas wrote:
> Well, the history here is pretty weird. In relevant part, here's the
> result of cvs log on src/backend/parser/gram.c:
>
> revision 2.92
> date: 2007/04/17 01:06:27; author: tgl; state: dead; lines: +0 -0
> And remove 'em again ...
> ----------------------------
> revision 2.91
> date: 2007/04/17 01:05:07; author: tgl; state: Exp; lines: +0 -12088
> Temporarily re-add derived files, in hopes of straightening out their
> CVS status.
> ----------------------------
> revision 2.90
> date: 1999/05/07 01:22:54; author: vadim; state: Exp; lines: +6001 -5942
> branches: 2.90.2;
> Fix LMGR for MVCC.
> Get rid of Extend lock mode.
> ----------------------------
> revision 2.89
> date: 1999/03/28 20:32:04; author: vadim; state: Exp; lines: +3292 -3225
> 1. Vacuum is updated for MVCC.
> 2. Much faster btree tuples deletion in the case when first on page
> index tuple is deleted (no movement to the left page(s)).
> 3. Remember blkno of new root page in BTPageOpaque of
> left/right siblings when root page is splitted.
> ----------------------------
> revision 2.88
> date: 1999/03/20 18:43:49; author: tgl; state: dead; lines: +1 -1
> Remove yacc/lex output files from CVS repository.
>
> The fact that the file was "modified" twice after being removed at rev
> 2.88 seems really wacko. Are you sure that's not contributing to what
> we're seeing here?

I think this is the normal behavior when a file is deleted then
re-added. In version 2.89 the file was re-added, and its delta is
against the pre-deleted version (presumably 2.87).

(Actually, even deleted versions can have deltas, so technically the
delta in 2.89 is against the "hidden content" of version 2.88.

Michael

Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> writes:
> Robert Haas wrote:
>> The fact that the file was "modified" twice after being removed at rev
>> 2.88 seems really wacko. Are you sure that's not contributing to what
>> we're seeing here?

> I think this is the normal behavior when a file is deleted then
> re-added. In version 2.89 the file was re-added, and its delta is
> against the pre-deleted version (presumably 2.87).

The thing that was confusing was that Vadim apparently saw the file as
being still live, while none of the rest of us did. I don't think he
did an explicit "cvs add" to make it live again, because if he had,
that should have propagated to the repository and the rest of us would
have seen it.

It took some considerable fooling around (though unfortunately I don't
recall the exact details) to persuade my checkout that gram.c wasn't
deleted so that I could delete it again.

regards, tom lane

On Wed, Aug 25, 2010 at 1:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>> The fact that the file was "modified" twice after being removed at rev
>> 2.88 seems really wacko.  Are you sure that's not contributing to what
>> we're seeing here?
>
> Yeah, that was discussed in the earlier git-conversion thread that I
> pointed to.  We never did figure out how that happened, though I
> speculated it might have been due to weirdness in Vadim's local
> checkout.
>
> Is it possible to just delete those two revisions from the CVS
> repository, and if so would it help?  We certainly don't need 'em.

cvs admin -o ?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Wed, Aug 25, 2010 at 12:35:53PM -0400, Robert Haas wrote:
> On Wed, Aug 25, 2010 at 12:02 PM, Michael Haggerty <mhagger(at)alum(dot)mit(dot)edu> wrote:

[...]

> > I must say, it is refreshing to have users who actually care about their
> > conversion, as opposed to the usual rabble who think that git-cvsimport
> > is Just Fine :-)  I guess if the postgresql project didn't care about
> > data integrity then we would all have to worry :-)
>
> I laughed when I read this - yeah, we're kind of paranoid about that.

Going a bit off-topic -- although I'm extremely strapped on time I have
been following this thread with attention. The above reminds me why I
appreciate git; it's not just the technical parts, but the culture
behind it what convince me.

Thanks for a very interesting thread
- -- tomás
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFMeLSNBcgs9XrR2kYRAoTUAJ9AnpqkVID3YO1l3RhwU1rRMMtIIwCfR9yW
T4NsNX5Ju5BZQhbxIEmDIpg=
=+u0H
-----END PGP SIGNATURE-----

On Wed, Aug 25, 2010 at 2:39 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Aug 25, 2010 at 1:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>> The fact that the file was "modified" twice after being removed at rev
>>> 2.88 seems really wacko.  Are you sure that's not contributing to what
>>> we're seeing here?
>>
>> Yeah, that was discussed in the earlier git-conversion thread that I
>> pointed to.  We never did figure out how that happened, though I
>> speculated it might have been due to weirdness in Vadim's local
>> checkout.
>>
>> Is it possible to just delete those two revisions from the CVS
>> repository, and if so would it help?  We certainly don't need 'em.
>
> cvs admin -o ?

Magnus, is this something that you can try? Prune those could of
wonky revisions after the delete and before the re-add prior to
running the conversion, and see how that comes out?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise Postgres Company

On Mon, Aug 30, 2010 at 05:03, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Aug 25, 2010 at 2:39 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> On Wed, Aug 25, 2010 at 1:27 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> Robert Haas <robertmhaas(at)gmail(dot)com> writes:
>>>> The fact that the file was "modified" twice after being removed at rev
>>>> 2.88 seems really wacko.  Are you sure that's not contributing to what
>>>> we're seeing here?
>>>
>>> Yeah, that was discussed in the earlier git-conversion thread that I
>>> pointed to.  We never did figure out how that happened, though I
>>> speculated it might have been due to weirdness in Vadim's local
>>> checkout.
>>>
>>> Is it possible to just delete those two revisions from the CVS
>>> repository, and if so would it help?  We certainly don't need 'em.
>>
>> cvs admin -o ?
>
> Magnus, is this something that you can try?  Prune those could of
> wonky revisions after the delete and before the re-add prior to
> running the conversion, and see how that comes out?

Yes, definitely.

Do we have list of exactly which revisions it is, or a good way to
find it? Other than random browsing of the history? :-)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> On Mon, Aug 30, 2010 at 05:03, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> cvs admin -o ?
>>
>> Magnus, is this something that you can try? Prune those could of
>> wonky revisions after the delete and before the re-add prior to
>> running the conversion, and see how that comes out?

> Yes, definitely.

> Do we have list of exactly which revisions it is, or a good way to
> find it? Other than random browsing of the history? :-)

I think the files in question are these:

2007-04-16 21:05 tgl

* src/: backend/parser/gram.c, interfaces/ecpg/preproc/pgc.c,
interfaces/ecpg/preproc/preproc.c: Temporarily re-add derived
files, in hopes of straightening out their CVS status.

2007-04-16 21:06 tgl

* src/: backend/parser/gram.c, interfaces/ecpg/preproc/pgc.c,
interfaces/ecpg/preproc/preproc.c: And remove 'em again ...

Look at the histories of these in cvsweb, and try to zap the versions
that are later than the first FILE REMOVED event.

regards, tom lane

On Tue, Aug 31, 2010 at 17:12, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> On Mon, Aug 30, 2010 at 05:03, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>>> cvs admin -o ?
>>>
>>> Magnus, is this something that you can try?  Prune those could of
>>> wonky revisions after the delete and before the re-add prior to
>>> running the conversion, and see how that comes out?
>
>> Yes, definitely.
>
>> Do we have list of exactly which revisions it is, or a good way to
>> find it? Other than random browsing of the history? :-)
>
> I think the files in question are these:
>
> 2007-04-16 21:05  tgl
>
>        * src/: backend/parser/gram.c, interfaces/ecpg/preproc/pgc.c,
>        interfaces/ecpg/preproc/preproc.c: Temporarily re-add derived
>        files, in hopes of straightening out their CVS status.
>
> 2007-04-16 21:06  tgl
>
>        * src/: backend/parser/gram.c, interfaces/ecpg/preproc/pgc.c,
>        interfaces/ecpg/preproc/preproc.c: And remove 'em again ...
>
> Look at the histories of these in cvsweb, and try to zap the versions
> that are later than the first FILE REMOVED event.

Ok. I've got a new migration runinng. Here's the revisions removed:
RCS file: /usr/local/cvsroot/pgsql/src/backend/parser/Attic/gram.c,v
deleting revision 2.90.2.1
deleting revision 2.90.2.2
done
RCS file: /usr/local/cvsroot/pgsql/src/backend/parser/Attic/gram.c,v
deleting revision 2.92
deleting revision 2.91
deleting revision 2.90
deleting revision 2.89
deleting revision 2.88
done
RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/pgc.c,v
deleting revision 1.5.2.1
deleting revision 1.5.2.2
done
RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/pgc.c,v
deleting revision 1.7
deleting revision 1.6
deleting revision 1.5
deleting revision 1.4
deleting revision 1.3
deleting revision 1.2
done
RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/preproc.c,v
deleting revision 1.11.2.1
deleting revision 1.11.2.2
done
RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/preproc.c,v
deleting revision 1.13
deleting revision 1.12
deleting revision 1.11
deleting revision 1.10
deleting revision 1.9
deleting revision 1.8
deleting revision 1.7
deleting revision 1.6
done

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Ok. I've got a new migration runinng. Here's the revisions removed:
> RCS file: /usr/local/cvsroot/pgsql/src/backend/parser/Attic/gram.c,v
> deleting revision 2.88
> RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/pgc.c,v
> deleting revision 1.2
> RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/preproc.c,v
> deleting revision 1.6

Hmm, it looks like you deleted the file deletion events (the versions
cited above). Not sure this is the right thing. Check to see if the
files are still there according to the converted git history ...

regards, tom lane

On Tue, Aug 31, 2010 at 19:44, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Ok. I've got a new migration runinng. Here's the revisions removed:
>> RCS file: /usr/local/cvsroot/pgsql/src/backend/parser/Attic/gram.c,v
>> deleting revision 2.88
>> RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/pgc.c,v
>> deleting revision 1.2
>> RCS file: /usr/local/cvsroot/pgsql/src/interfaces/ecpg/preproc/Attic/preproc.c,v
>> deleting revision 1.6
>
> Hmm, it looks like you deleted the file deletion events (the versions
> cited above).  Not sure this is the right thing.  Check to see if the
> files are still there according to the converted git history ...

Oh, drat. That's right. It shouldn't have been inclusive :S

I'll abort the conversion and run it again :)

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/