Re: Several tags around PostgreSQL 7.1 broken

I have now managed to investigate why some conversions of the PostgreSQL CVS
repository to other formats are having trouble or are failing. Here, I am
looking at git-cvsimport in particular.

The problem appears to be that several tags around the time of PostgreSQL 7.1
are broken or inconsistent. For instance, here is a piece of output of
cvsps:

WARNING: Invalid PatchSet 9441, Tag REL7_1:
src/pl/plpgsql/src/mklang.sql.in:1.6=after,
src/pl/plpgsql/src/INSTALL:1.2=before. Treated as 'before'

It turns out that src/pl/plpgsql/src/mklang.sql.in:1.6 and
src/pl/plpgsql/src/INSTALL:1.2 are from the same commit ("PatchSet"), as
determined by a common log message and timestamp, but the REL7_1 tag is on
src/pl/plpgsql/src/mklang.sql.in:1.5 and src/pl/plpgsql/src/INSTALL:1.2. So
a part of the commit is before the tag and part of it is after the tag.

(The commit in question was to remove mklang.sql.in and adjust the INSTALL
contents accordingly.)

In fact, if you check out the REL7_1 tag, you get the new INSTALL file but
still the mklang.sql.in. The released postgresql-7.1.tar.gz tarball is
correct.

I guess the cause of this is that the tag was done on a partially updated
checkout.

There are a few dozen inconsistencies like this in the tags REL7_1_BETA,
REL7_1_BETA2, REL7_1_BETA3, REL7_1. As a consequence of this, checkouts of
the tags don't match the respective released tarballs.

Here are more examples:

WARNING: Invalid PatchSet 9297, Tag REL7_1:
src/backend/port/hpux/port-protos.h:1.10=after, COPYRIGHT:1.6=before.
Treated as 'before'
WARNING: Invalid PatchSet 8906, Tag REL7_1:
doc/src/sgml/populate.sgml:2.4=after,
doc/src/sgml/filelist.sgml:1.3=before. Treated as 'before'
WARNING: Invalid PatchSet 9371, Tag REL7_1:
doc/TODO.detail/subquery:1.3=after, doc/TODO:1.366=before. Treated
as 'before'
WARNING: Invalid PatchSet 8815, Tag REL7_1_BETA2:
src/include/c.h:1.2=after, contrib/pgcrypto/md5.c:1.2=before. Treated
as 'before'
etc.

I could get the conversion to run successfully if I remove the mentioned tags.
When I find more time, I'm going to try if I can move/fix the tags instead so
they correspond to the actual releases and the patch sets become consistent.

In the meantime, does anyone have more information about how this came about?

.

* Peter Eisentraut <peter_e(at)gmx(dot)net> [080401 12:01]:
> I have now managed to investigate why some conversions of the PostgreSQL CVS
> repository to other formats are having trouble or are failing. Here, I am
> looking at git-cvsimport in particular.
>
> The problem appears to be that several tags around the time of PostgreSQL 7.1
> are broken or inconsistent. For instance, here is a piece of output of
> cvsps:
>
> WARNING: Invalid PatchSet 9441, Tag REL7_1:
> src/pl/plpgsql/src/mklang.sql.in:1.6=after,
> src/pl/plpgsql/src/INSTALL:1.2=before. Treated as 'before'

I talked about this here:
http://mid.gmane.org/20080220225300(dot)GE16099(at)yugib(dot)highrise(dot)ca

Here's a "quick hack" to cvsps that I use to ignore the problematic
tags. Of course, I've stuck with fromcvs for my PostgreSQL git mirror
simply because even though I *can* use cvsps on PostgreSQL with that
hack, it's still loads slower than fromcvs. Since cvsps/cvsimport is so
slow, I didn't pursue making this patch usable, and moving forward with
using cvsps/cvsimport.

aidan(at)db1-dapper:~/build/cvsps.git$ git diff
diff --git a/cvsps.c b/cvsps.c
index 981cd78..d436591 100644
--- a/cvsps.c
+++ b/cvsps.c
@@ -2184,15 +2184,28 @@ static void parse_sym(CvsFile * file, char * sym)
}
}

+const char* skip_symbols[] =
+{
+ "REL7_1_BETA",
+ "REL7_1_BETA2",
+ "REL7_1_BETA3",
+ NULL
+};
+
void cvs_file_add_symbol(CvsFile * file, const char * rev_str, const char * p_tag_str)
{
CvsFileRevision * rev;
GlobalSymbol * sym;
Tag * tag;
+ int i;

/* get a permanent storage string */
char * tag_str = get_string(p_tag_str);

+ for (i = 0; skip_symbols[i] != NULL; i++)
+ if (strcmp(tag_str, skip_symbols[i]) == 0)
+ return;
+
debug(DEBUG_STATUS, "adding symbol to file: %s %s->%s", file->filename, tag_str, rev_str);
rev = cvs_file_add_revision(file, rev_str);
put_hash_object_ex(file->symbols, tag_str, rev, HT_NO_KEYCOPY, NULL, NULL);

a.

--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> In the meantime, does anyone have more information about how this came about?

Marc's always done both the tagging and the tarball-making, so you'd
have to ask him about that. I believe he's made it more scripted over
the years, so this might reflect a manual foulup that (hopefully) is no
longer possible.

+1 for adjusting the tags in CVS to match what we actually shipped.

regards, tom lane

* Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> [080401 14:15]:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > In the meantime, does anyone have more information about how this came about?
>
> Marc's always done both the tagging and the tarball-making, so you'd
> have to ask him about that. I believe he's made it more scripted over
> the years, so this might reflect a manual foulup that (hopefully) is no
> longer possible.
>
> +1 for adjusting the tags in CVS to match what we actually shipped.
>
> regards, tom lane

If somebody's going to be fudging around in $CVSROOT, is it possible to
give us all a big warning, and hopefully "pause" the anoncvs/rsync sync
stuff for the duration of the "history change", so we (those of us out
here working off anoncvs/rsync) get an atomic swap of ancient history?

a.
--
Aidan Van Dyk Create like a god,
aidan(at)highrise(dot)ca command like a king,
http://www.highrise.ca/ work like a slave.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, April 01, 2008 14:06:09 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> In the meantime, does anyone have more information about how this came about?
>
> Marc's always done both the tagging and the tarball-making, so you'd
> have to ask him about that. I believe he's made it more scripted over
> the years, so this might reflect a manual foulup that (hopefully) is no
> longer possible.

Ya, I'll go with that (considering 7.1 was back in 2001 ... ) ... but, from the
way Peter describes it (taging partially checked out code), I'm not 100% how
its possible to 'foul up' ... a tag operation is:

cvs -q update -APd .
cvs -q tag REL7_1 .

unless its a sub-tagging, which would have:

cvs -q update -rREL7_1_STABLE -Pd .
cvs -q tag REL7_1_1 .

And since I don't do the update until things are "quiet" (generally when Tom
has finished his last commit before release), I'm not sure how I could have
gotten a 'partial checkout' ...

> +1 for adjusting the tags in CVS to match what we actually shipped.

Agreed ... but, stupid question here ... if our tags are wrong in CVS, are the
7.1.x releases themselves wrong too? When I do a release tarball, I run:

cvs -q export -rREL7_1_1 pgsql

so, if the tags are wrong, then all of those releases are wrong too, since they
are based on the tag ...

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFH8tX24QvfyHIvDvMRAndoAJ9KA86BZl21zLb3rie9ynlmDL7BHQCfdtjB
VrYLsml4H+ppnXvC26ywKTU=
=RWHE
-----END PGP SIGNATURE-----

Marc G. Fournier wrote:
> - --On Tuesday, April 01, 2008 14:06:09 -0400 Tom Lane
> <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
> > Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> >> In the meantime, does anyone have more information about how this
> >> came about?
> >
> > Marc's always done both the tagging and the tarball-making, so you'd
> > have to ask him about that. I believe he's made it more scripted
> > over the years, so this might reflect a manual foulup that
> > (hopefully) is no longer possible.
>
> Ya, I'll go with that (considering 7.1 was back in 2001 ... ) ...
> but, from the way Peter describes it (taging partially checked out
> code), I'm not 100% how its possible to 'foul up' ... a tag operation
> is:
>
> cvs -q update -APd .
> cvs -q tag REL7_1 .
>
> unless its a sub-tagging, which would have:
>
> cvs -q update -rREL7_1_STABLE -Pd .
> cvs -q tag REL7_1_1 .
>
> And since I don't do the update until things are "quiet" (generally
> when Tom has finished his last commit before release), I'm not sure
> how I could have gotten a 'partial checkout' ...

Could it be that a commit was done while the tag operation was running?
Given that neither is an atomic operation in cvs, and it used to be
that large repo operations could take quite a long time?

//Magnus

Am Mittwoch, 2. April 2008 schrieb Marc G. Fournier:
> Agreed ... but, stupid question here ... if our tags are wrong in CVS, are
> the 7.1.x releases themselves wrong too?  When I do a release tarball, I
> run:
>
> cvs -q export -rREL7_1_1 pgsql
>
> so, if the tags are wrong, then all of those releases are wrong too, since
> they are based on the tag ...

I believe we moved to using cvs export many years after 7.1. Before that, the
releases were made straight out of a cvs checkout. With cvs export it is of
course nearly impossible to create such a mess.

I sporadically tested the correspondence of tarball and tag for some other
older releases, and they seem OK. At least to the extent that corresponding
tags exist at all ... ;-)

Peter Eisentraut wrote:
> Am Mittwoch, 2. April 2008 schrieb Marc G. Fournier:
> > Agreed ... but, stupid question here ... if our tags are wrong in CVS, are
> > the 7.1.x releases themselves wrong too?  When I do a release tarball, I
> > run:
> >
> > cvs -q export -rREL7_1_1 pgsql
>
> I believe we moved to using cvs export many years after 7.1. Before that, the
> releases were made straight out of a cvs checkout. With cvs export it is of
> course nearly impossible to create such a mess.

Hmm, if we use a CVS export, why do we have, on "make distdir", the
business to remove CVS files?

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

Alvaro Herrera <alvherre(at)commandprompt(dot)com> writes:
> Peter Eisentraut wrote:
>> I believe we moved to using cvs export many years after 7.1. Before
>> that, the releases were made straight out of a cvs checkout. With
>> cvs export it is of course nearly impossible to create such a mess.

> Hmm, if we use a CVS export, why do we have, on "make distdir", the
> business to remove CVS files?

Leftover from the old way, no doubt.

I have a vague recollection that the rest of core pressed Marc to start
using "cvs export" precisely because we were worried about, or actually
had seen, inconsistencies between the tagging and the tarballs. It
might be that what Peter has noticed was exactly what prompted that,
and we never attempted to clean it up. But it was a long time ago
and I'm not sure about it.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, April 02, 2008 10:10:59 +0200 Magnus Hagander
<magnus(at)hagander(dot)net> wrote:

> Could it be that a commit was done while the tag operation was running?
> Given that neither is an atomic operation in cvs, and it used to be
> that large repo operations could take quite a long time?

I doubt it, back in 7.1 days, we had something like 4 committers, I think, and
all core members ... very little chance of overlap ... the weird thing is Peter
is reporting its *all* on the 7.1 branch ... none of hte others affected ...

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFH8/It4QvfyHIvDvMRAqnKAJ4h/+4elPLe3XEfW9lRU6IfAAiSWgCcCRCd
pWs76n1JzLgiOExj6kGytBA=
=XVkS
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, April 02, 2008 12:33:24 -0400 Alvaro Herrera
<alvherre(at)commandprompt(dot)com> wrote:

> Peter Eisentraut wrote:
>> Am Mittwoch, 2. April 2008 schrieb Marc G. Fournier:
>> > Agreed ... but, stupid question here ... if our tags are wrong in CVS, are
>> > the 7.1.x releases themselves wrong too?  When I do a release tarball, I
>> > run:
>> >
>> > cvs -q export -rREL7_1_1 pgsql
>>
>> I believe we moved to using cvs export many years after 7.1. Before that,
>> the releases were made straight out of a cvs checkout. With cvs export it
>> is of course nearly impossible to create such a mess.
>
> Hmm, if we use a CVS export, why do we have, on "make distdir", the
> business to remove CVS files?

Using export was a relatively recent change ... wasn't a command I knew about
at the start, someone (possibly Peter) pointed it out to me and we changed the
script ... never thought to change distdir though ...

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFH8/vv4QvfyHIvDvMRAu2NAKDjiCnjJfJ3pXtUX9PFH8vZfSnr5ACfcWdg
EiNTuC6bd2PFxlZ1tt8oA68=
=VbGW
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, April 02, 2008 18:08:05 +0200 Peter Eisentraut
<peter_e(at)gmx(dot)net> wrote:

> I believe we moved to using cvs export many years after 7.1. Before that,
> the releases were made straight out of a cvs checkout. With cvs export it
> is of course nearly impossible to create such a mess.

'k, you need to explain this one to me ... how does export affect what files
are tag'd? export isn't used as part of the tagging process, only to create
the tar balls themselves ...

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFH8/w44QvfyHIvDvMRAmXeAKDc7V43uQmP8SdxuLBaIyvjntKlzgCfYF5N
Tsn/rXOk2AH9RgjlOO9duKQ=
=29dq
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> - --On Wednesday, April 02, 2008 18:08:05 +0200 Peter Eisentraut
> <peter_e(at)gmx(dot)net> wrote:
>> I believe we moved to using cvs export many years after 7.1. Before that,
>> the releases were made straight out of a cvs checkout. With cvs export it
>> is of course nearly impossible to create such a mess.

> 'k, you need to explain this one to me ... how does export affect what files
> are tag'd?

It doesn't, of course. What it does do is guarantee that the tarball
matches the tag that has already been laid down in CVS.

But there must have been more to it than that. Peter is reporting
that the tag is on mutually inconsistent versions of some files;
which means that the problem was with the tagging operation more than
with the tarball-making.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, April 02, 2008 17:49:49 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> It doesn't, of course. What it does do is guarantee that the tarball
> matches the tag that has already been laid down in CVS.

'k, that was my thought, so using export vs update to create the tarbal is
irrelevant to this discussion ...

> But there must have been more to it than that. Peter is reporting
> that the tag is on mutually inconsistent versions of some files;
> which means that the problem was with the tagging operation more than
> with the tarball-making.

Has anyone actually checked the tarballs themselves? If the tag's are wrong,
then doesn't it follow that the tarballs themselves are all wrong too?

- --
Marc G. Fournier Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFH9BQr4QvfyHIvDvMRAjC9AJ9qBRom7aU7LWmZGnhfOFtbwv7zRQCgxPqx
qv5B4ffClv4RRXc2FVg6LpI=
=zjTy
-----END PGP SIGNATURE-----