Re: CVS corruption/mistagging?

Looking into recent buildfarm failures on the 7.4 branch:

http://www.pgbuildfarm.org/cgi-bin/show_status.pl

It looks like parts of the CVS repository have been mistagged as belonging
to REL7_4_STABLE or have been corrupted somehow:

http://developer.postgresql.org/cvsweb.cgi/pgsql/contrib/btree_gist/btree_bit.c?sortby=date;only_with_tag=REL7_4_STABLE

I'm not sure what's going on here, but something has gone bad.

Kris Jurka

Kris Jurka <books(at)ejurka(dot)com> writes:
> It looks like parts of the CVS repository have been mistagged as belonging
> to REL7_4_STABLE or have been corrupted somehow:
> http://developer.postgresql.org/cvsweb.cgi/pgsql/contrib/btree_gist/btree_bit.c?sortby=date;only_with_tag=REL7_4_STABLE

Hmm ... btree_bit.c shouldn't be in 7.4 at all.

I can't help thinking this has something to do with the recent CVS
server move. Magnus, any thoughts?

regards, tom lane

I wrote:
> Kris Jurka <books(at)ejurka(dot)com> writes:
>> It looks like parts of the CVS repository have been mistagged as belonging
>> to REL7_4_STABLE or have been corrupted somehow:
>> http://developer.postgresql.org/cvsweb.cgi/pgsql/contrib/btree_gist/btree_bit.c?sortby=date;only_with_tag=REL7_4_STABLE

> Hmm ... btree_bit.c shouldn't be in 7.4 at all.

I did a fresh checkout of the 7.4 branch and diff'd against my local
copy, and it seems clear that every file that was not in 7.4 at all has
had its HEAD version tagged as REL7_4_STABLE. The files that did exist
then are all right. That's throughout the whole tree, not just in
contrib/btree_gist. I have no idea how you make CVS do that, but I'm
sure there is some magic one-liner for it.

Checking the files in contrib/btree_gist on the CVS server gives a
pretty good fix on who changed it and when:

> ls -la
total 466
drwxrwxr-x 6 scrappy dev 1024 Aug 14 22:30 .
drwxrwxr-x 90 scrappy dev 2048 Aug 14 22:27 ..
drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
-r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
-r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
-r--r--r-- 1 meskes dev 12052 Aug 12 09:50 btree_bit.c,v
-r--r--r-- 1 meskes dev 9921 Aug 12 09:50 btree_bytea.c,v
-r--r--r-- 1 meskes dev 10246 Aug 12 09:50 btree_cash.c,v
-r--r--r-- 1 meskes dev 11433 Aug 12 09:50 btree_date.c,v
-r--r--r-- 1 meskes dev 10230 Aug 12 09:50 btree_float4.c,v
-r--r--r-- 1 meskes dev 10093 Aug 12 09:50 btree_float8.c,v
-r--r--r-- 1 meskes dev 33935 Aug 12 09:50 btree_gist.c,v
-r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
-r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
-r--r--r-- 1 meskes dev 14736 Aug 12 09:50 btree_inet.c,v
-r--r--r-- 1 meskes dev 10253 Aug 12 09:50 btree_int2.c,v
-r--r--r-- 1 meskes dev 10240 Aug 12 09:50 btree_int4.c,v
-r--r--r-- 1 meskes dev 10254 Aug 12 09:50 btree_int8.c,v
-r--r--r-- 1 meskes dev 18111 Aug 12 09:50 btree_interval.c,v
-r--r--r-- 1 meskes dev 12025 Aug 12 09:50 btree_macaddr.c,v
-r--r--r-- 1 meskes dev 14671 Aug 12 09:50 btree_numeric.c,v
-r--r--r-- 1 meskes dev 9796 Aug 12 09:50 btree_oid.c,v
-r--r--r-- 1 meskes dev 18247 Aug 12 09:50 btree_text.c,v
-r--r--r-- 1 meskes dev 26180 Aug 12 09:50 btree_time.c,v
-r--r--r-- 1 258 dev 29712 Apr 20 15:16 btree_ts.c,v
-r--r--r-- 1 meskes dev 17588 Aug 12 09:50 btree_utils_num.c,v
-r--r--r-- 1 meskes dev 8959 Aug 12 09:50 btree_utils_num.h,v
-r--r--r-- 1 meskes dev 48824 Aug 12 09:50 btree_utils_var.c,v
-r--r--r-- 1 meskes dev 7099 Aug 12 09:50 btree_utils_var.h,v
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 data
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 expected
drwxrwxr-x 3 scrappy dev 1024 Aug 14 22:27 sql
-r--r--r-- 1 meskes dev 10021 Aug 12 09:50 uninstall_btree_gist.sql,v

Michael, want to fess up to what you did?

In the meantime, though, it's not quite clear why this would lead to
a buildfarm failure --- it should just mean a lot of extraneous files
appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
like btree_gist has some files that used to be autogenerated and are
now in CVS, so the bogusly new versions from CVS are suppressing the
desired generation from the old btree_num.c file.)

regards, tom lane

I wrote:
> I did a fresh checkout of the 7.4 branch and diff'd against my local
> copy, and it seems clear that every file that was not in 7.4 at all has
> had its HEAD version tagged as REL7_4_STABLE. The files that did exist
> then are all right. That's throughout the whole tree, not just in
> contrib/btree_gist.

As for getting out of it: it looks like the misapplied tags are revision
tags not branch tags, eg

RCS file: /cvsroot/pgsql/contrib/btree_gist/btree_bit.c,v
Working file: btree_bit.c
head: 1.7
branch:
locks: strict
access list:
symbolic names:
REL7_4_STABLE: 1.7 <--- wrong
REL8_0_13: 1.2
REL8_1_9: 1.6
REL8_2_4: 1.6
REL8_0_12: 1.2
REL8_1_8: 1.6
REL8_2_3: 1.6
REL8_0_11: 1.2
REL8_1_7: 1.6
REL8_2_2: 1.6
REL8_0_10: 1.2
REL8_1_6: 1.6
REL8_2_1: 1.6
REL8_2_STABLE: 1.6.0.4

which I think means that there's a one-liner way out of it. The CVS
manual quoth:

To delete a tag, specify the `-d' option to either `cvs tag' or `cvs
rtag'. For example:

cvs rtag -d rel-0-4 tc

deletes the non-branch tag `rel-0-4' from the module `tc'. In the
event that branch tags are encountered within the repository with the
given name, a warning message will be issued and the branch tag will
not be deleted. If you are absolutely certain you know what you are
doing, the `-B' option may be specified to allow deletion of branch
tags. In that case, any non-branch tags encountered will trigger
warnings and will not be deleted.

So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
I'd like someone to double-check that though. Also maybe we should back
up the repository first?

regards, tom lane

Tom Lane wrote:
> In the meantime, though, it's not quite clear why this would lead to
> a buildfarm failure --- it should just mean a lot of extraneous files
> appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
> like btree_gist has some files that used to be autogenerated and are
> now in CVS, so the bogusly new versions from CVS are suppressing the
> desired generation from the old btree_num.c file.)
>
>
>

Looking at the Committers mail, it looks like there have only been two
very small commits since Michael's series of commits around 12 to 13.5
hours ago, and before that nothing since around 28 hours ago. Do we have
a backup snapshot of the repo taken during that time that we can roll
back to? That might be simpler than doing surgery on the repo.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Looking at the Committers mail, it looks like there have only been two
> very small commits since Michael's series of commits around 12 to 13.5
> hours ago, and before that nothing since around 28 hours ago. Do we have
> a backup snapshot of the repo taken during that time that we can roll
> back to? That might be simpler than doing surgery on the repo.

It looks to me like the mistake was unrelated to any commit, and was
made at Aug 12 09:50 UTC. (Aside from the file-timestamp evidence,
caracara's build failure at 2007-08-12 213845 shows that the problem
is more than 2 days old.) I count seven commits of my own and several
of Michael's since then. If I'm right that a simple "cvs rtag" will fix
it, I'd rather do that than try to reapply patches.

regards, tom lane

Tom Lane wrote:
>
> So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
> I'd like someone to double-check that though.

I will test on a copy of my mirror.

> Also maybe we should back
> up the repository first?
>

Amen.

cheers

andrew

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>>
>> So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
>> I'd like someone to double-check that though.
>
> I will test on a copy of my mirror.
>

I copied the mirror, did a checkout from it, ran the command above in
the checked out version, then remove my checkout and checked out
REL7_4_STABLE and did a diff against my copy from the postgres repo that
hasn't been updated for some time. There were no significant
differences, and no extra files.

So I'd say your strategy looks good - backup and remove the phony tag.
I'd also say we should probably be logging tag commands in taginfo.
Presumably we mere mortal committers should not be doing any tagging
whatsoever, and tags should only be applied in preparation for releases.
Anyway, if we actually logged it we'd catch anything untoward early
rather than possibly days later.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> So I'd say your strategy looks good - backup and remove the phony tag.
> I'd also say we should probably be logging tag commands in taginfo.
> Presumably we mere mortal committers should not be doing any tagging
> whatsoever, and tags should only be applied in preparation for releases.

+1 ... we should at least log such commands, and maybe disallow to anyone
except Marc's "pgsql" account. Particularly since they don't get
reported in pgsql-committers messages (or is that something we
can/should change?)

Meanwhile, is there anyone around who can either (1) tar up the
repository directory tree as root, or (2) confirm that a tarball
made by a non-root committer is sufficient?

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 22:20:16 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> +1 ... we should at least log such commands, and maybe disallow to anyone
> except Marc's "pgsql" account. Particularly since they don't get
> reported in pgsql-committers messages (or is that something we
> can/should change?)
>
> Meanwhile, is there anyone around who can either (1) tar up the
> repository directory tree as root, or (2) confirm that a tarball
> made by a non-root committer is sufficient?

I've just finished tar'ng up the cvs repository as root ... I believe one as a
non-root would be sufficient also, since it *should* retain file ownership /
permissions with the -p option ...

cvs# ls -lt cvsroot.aug14_2007.tar.bz2
- -rw-r--r-- 1 root wheel 62555651 Aug 15 02:52 cvsroot.aug14_2007.tar.bz2

cvs# du -sk cvsroot
414406 cvsroot

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwmq14QvfyHIvDvMRAsyFAJ9AND3JTDmOMR48Hl3Fdse2lVCjSwCglsu1
Pxj1LPPPx311rpY2XFj+Kvo=
=Gz6E
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> On Tuesday, August 14, 2007 22:20:16 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
> wrote:
>> Meanwhile, is there anyone around who can either (1) tar up the
>> repository directory tree as root, or (2) confirm that a tarball
>> made by a non-root committer is sufficient?

> I've just finished tar'ng up the cvs repository as root ...

Great --- launching cvs rtag command now.

> I believe one as a
> non-root would be sufficient also, since it *should* retain file ownership /
> permissions with the -p option ...

That's what I thought too, but a little extra paranoia doesn't hurt.

regards, tom lane

I wrote:
> Great --- launching cvs rtag command now.

Done, and I got a plausible-looking mix of messages like
cvs rtag: Not removing branch tag `REL7_4_STABLE' from `/cvsroot/pgsql/src/tutorial/funcs_new.c,v'.
and a fresh checkout of REL7_4_STABLE now matches what I had locally.
So I think we walked away from that one.

TODO items:

* restrict, or at least log, cvs tag/rtag commands. Maybe report them
to pgsql-committers.

* I notice that the repository contains some files owned by
no-longer-existent accounts, eg 258 in contrib/btree_gist:

> ls -la
total 460
drwxrwxr-x 6 scrappy dev 1024 Aug 15 03:00 .
drwxrwxr-x 90 scrappy dev 2048 Aug 15 03:00 ..
drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
-r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
-r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
-r--r--r-- 1 tgl dev 12033 Aug 15 03:00 btree_bit.c,v
-r--r--r-- 1 tgl dev 9902 Aug 15 03:00 btree_bytea.c,v
-r--r--r-- 1 tgl dev 10227 Aug 15 03:00 btree_cash.c,v
-r--r--r-- 1 tgl dev 11414 Aug 15 03:00 btree_date.c,v
-r--r--r-- 1 tgl dev 10211 Aug 15 03:00 btree_float4.c,v
-r--r--r-- 1 tgl dev 10074 Aug 15 03:00 btree_float8.c,v
-r--r--r-- 1 tgl dev 33915 Aug 15 03:00 btree_gist.c,v
-r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
-r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
-r--r--r-- 1 tgl dev 14717 Aug 15 03:00 btree_inet.c,v
-r--r--r-- 1 tgl dev 10234 Aug 15 03:00 btree_int2.c,v
-r--r--r-- 1 tgl dev 10221 Aug 15 03:00 btree_int4.c,v
-r--r--r-- 1 tgl dev 10235 Aug 15 03:00 btree_int8.c,v

I suppose this is due to Magnus' cleanup of unused committers accounts.
That was a good thing, but for security's sake these files ought to be
chown'd to some existing committer's account.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 23:26:03 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> I wrote:
>> Great --- launching cvs rtag command now.
>
> Done, and I got a plausible-looking mix of messages like
> cvs rtag: Not removing branch tag `REL7_4_STABLE' from
> `/cvsroot/pgsql/src/tutorial/funcs_new.c,v'. and a fresh checkout of
> REL7_4_STABLE now matches what I had locally. So I think we walked away from
> that one.
>
> TODO items:
>
> * restrict, or at least log, cvs tag/rtag commands. Maybe report them
> to pgsql-committers.

working on it ... have code for it, I've just spent so much time with CVSROOT =
/cvsroot, I have to relearn remote CVS :)

>> ls -la
> total 460
> drwxrwxr-x 6 scrappy dev 1024 Aug 15 03:00 .
> drwxrwxr-x 90 scrappy dev 2048 Aug 15 03:00 ..
> drwxrwxr-x 2 scrappy dev 512 Apr 20 15:16 Attic
> -r--r--r-- 1 tgl dev 9555 Jun 26 22:05 Makefile,v
> -r--r--r-- 1 258 dev 5870 Apr 20 16:19 README.btree_gist,v
> -r--r--r-- 1 tgl dev 12033 Aug 15 03:00 btree_bit.c,v
> -r--r--r-- 1 tgl dev 9902 Aug 15 03:00 btree_bytea.c,v
> -r--r--r-- 1 tgl dev 10227 Aug 15 03:00 btree_cash.c,v
> -r--r--r-- 1 tgl dev 11414 Aug 15 03:00 btree_date.c,v
> -r--r--r-- 1 tgl dev 10211 Aug 15 03:00 btree_float4.c,v
> -r--r--r-- 1 tgl dev 10074 Aug 15 03:00 btree_float8.c,v
> -r--r--r-- 1 tgl dev 33915 Aug 15 03:00 btree_gist.c,v
> -r--r--r-- 1 258 dev 4325 Apr 20 15:16 btree_gist.h,v
> -r--r--r-- 1 258 dev 58744 Apr 20 16:19 btree_gist.sql.in,v
> -r--r--r-- 1 tgl dev 14717 Aug 15 03:00 btree_inet.c,v
> -r--r--r-- 1 tgl dev 10234 Aug 15 03:00 btree_int2.c,v
> -r--r--r-- 1 tgl dev 10221 Aug 15 03:00 btree_int4.c,v
> -r--r--r-- 1 tgl dev 10235 Aug 15 03:00 btree_int8.c,v
>
> I suppose this is due to Magnus' cleanup of unused committers accounts.
> That was a good thing, but for security's sake these files ought to be
> chown'd to some existing committer's account.

I can do a quick chown -R scrappy on the whole repository ... ok?

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwnf34QvfyHIvDvMRAnGTAJ43gELVZ5Q5SpV/jppR9rODB/ILaACeKCEN
DE16fnY/PmKlrvdHFaJeIKI=
=VCFu
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Tuesday, August 14, 2007 23:26:03 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> * restrict, or at least log, cvs tag/rtag commands. Maybe report them
> to pgsql-committers.

It should be done ... if you try and create a tag, it should generate an error
message ... not sure of the best way to test that it actually works, since, if
it doesn't work, of course, you'll add on another tag, but do you want to try
and create a 'fake tag' and then delete it afterwards?

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwnoO4QvfyHIvDvMRArHSAJ4qYuexUGn6Ff3/PkHU/DjJBlCQnwCgxyy9
Qc852wMVRRMYdwoD3+p7X3Y=
=FCD9
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
>> That was a good thing, but for security's sake these files ought to be
>> chown'd to some existing committer's account.

> I can do a quick chown -R scrappy on the whole repository ... ok?

Seems close enough, but please keep that tarball around for awhile just
in case we need to look at the prior state.

regards, tom lane

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> It should be done ... if you try and create a tag, it should generate
> an error message ...

Uh, nope:

$ cvs tag fooey README
T README
$ cvs log README | more

RCS file: /cvsroot/pgsql/src/backend/optimizer/README,v
Working file: README
head: 1.39
branch:
locks: strict
access list:
symbolic names:
fooey: 1.39
REL7_3_19: 1.25
REL7_4_17: 1.29
REL8_0_13: 1.30
REL8_1_9: 1.33
REL8_2_4: 1.35.2.1
REL8_0_12: 1.30
REL8_1_8: 1.33
REL8_2_3: 1.35
REL7_3_18: 1.25
REL7_4_16: 1.29
REL8_0_11: 1.30
REL8_1_7: 1.33
REL8_2_2: 1.35
REL8_0_10: 1.30
cvs [log aborted]: received broken pipe signal
$ cvs tag -d fooey README
D README
$

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

'k, let me play with it ...

- --On Wednesday, August 15, 2007 00:13:11 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
>> It should be done ... if you try and create a tag, it should generate
>> an error message ...
>
> Uh, nope:
>
> $ cvs tag fooey README
> T README
> $ cvs log README | more
>
> RCS file: /cvsroot/pgsql/src/backend/optimizer/README,v
> Working file: README
> head: 1.39
> branch:
> locks: strict
> access list:
> symbolic names:
> fooey: 1.39
> REL7_3_19: 1.25
> REL7_4_17: 1.29
> REL8_0_13: 1.30
> REL8_1_9: 1.33
> REL8_2_4: 1.35.2.1
> REL8_0_12: 1.30
> REL8_1_8: 1.33
> REL8_2_3: 1.35
> REL7_3_18: 1.25
> REL7_4_16: 1.29
> REL8_0_11: 1.30
> REL8_1_7: 1.33
> REL8_2_2: 1.35
> REL8_0_10: 1.30
> cvs [log aborted]: received broken pipe signal
> $ cvs tag -d fooey README
> D README
> $
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 7: You can help support the PostgreSQL project by donating at
>
> http://www.postgresql.org/about/donate

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwn9q4QvfyHIvDvMRAuqOAJwPcIcsfQ74wBwdjBR478C72nr+bwCfd8PY
EeH2sh+biTfoFbTF2KxqYVU=
=iMwZ
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

One better (and done now):

find . -nouser -exec chown scrappy {} \;

so, all files that had valid users owning them stay with those ...

- --On Wednesday, August 15, 2007 00:08:07 -0400 Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
wrote:

> "Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
>>> That was a good thing, but for security's sake these files ought to be
>>> chown'd to some existing committer's account.
>
>> I can do a quick chown -R scrappy on the whole repository ... ok?
>
> Seems close enough, but please keep that tarball around for awhile just
> in case we need to look at the prior state.
>
> regards, tom lane

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwonj4QvfyHIvDvMRAhNBAJ9b6Y1o4eICLrtuWncRIqN8g07XggCgnnc8
ttJoOScGbFEY/y3qP3MZYWY=
=RUO5
-----END PGP SIGNATURE-----

"Marc G. Fournier" <scrappy(at)hub(dot)org> writes:
> One better (and done now):
> find . -nouser -exec chown scrappy {} \;

[ eyeballs repository ... ] Check, that looks great from here.

regards, tom lane

Tom Lane wrote:
> In the meantime, though, it's not quite clear why this would lead to
> a buildfarm failure --- it should just mean a lot of extraneous files
> appearing in a fresh checkout. (Looks a bit harder ... Oh, it looks
> like btree_gist has some files that used to be autogenerated and are
> now in CVS, so the bogusly new versions from CVS are suppressing the
> desired generation from the old btree_num.c file.)

Funny enoughy, that's exactly the problem I saw on win32 when I rsynced
from a windows box and tried a local CVS checkout :-) Once I rsynced to
a local linux instead and checked out from there, things worked again :)

//Magnus

Tom Lane wrote:
> So I think that "cvs rtag -d REL7_4_STABLE pgsql" will fix it.
> I'd like someone to double-check that though. Also maybe we should back
> up the repository first?

Just for your info: all VMs on tribble, which includes cvs, are backed
up at 02:30 every day, CEST (that's 00:30 UTC right now, but it follows
the local time on tribble so it'll be 23:30 UTC when austra switches
later this year. Perhaps we should switch the machine to GMT, but that's
a different discussion)

For something like this it's good to run a special backup before that of
course, but it's probably good to know when the backups run anyway...

//Magnus

Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> So I'd say your strategy looks good - backup and remove the phony tag.
>> I'd also say we should probably be logging tag commands in taginfo.
>> Presumably we mere mortal committers should not be doing any tagging
>> whatsoever, and tags should only be applied in preparation for releases.
>
> +1 ... we should at least log such commands, and maybe disallow to anyone
> except Marc's "pgsql" account. Particularly since they don't get
> reported in pgsql-committers messages (or is that something we
> can/should change?)
>

I think you can, by sticking a command in the taginfo command. It's the
same script that Marc is using to try to restrict who can do it - once
he has that working, it should be as simple as adding a mail command to
the end of that script.

And +1 on doing it.

//Magnus

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Tom Lane wrote:
>> I'd like someone to double-check that though. Also maybe we should back
>> up the repository first?

> Just for your info: all VMs on tribble, which includes cvs, are backed
> up at 02:30 every day, CEST

Good, but the salient followup questions to that are (1) backed up to
where exactly?, and (2) how many days' past backups could we get at,
if we had to?

regards, tom lane

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Tom Lane wrote:
>>> I'd like someone to double-check that though. Also maybe we should back
>>> up the repository first?
>
>> Just for your info: all VMs on tribble, which includes cvs, are backed
>> up at 02:30 every day, CEST
>
> Good, but the salient followup questions to that are (1) backed up to
> where exactly?, and (2) how many days' past backups could we get at,
> if we had to?

1) The entire VM, with "dump"
2) We have today, yesterday and one weekly (copies the daily on monday,
if I read it right)

They are dumped to a NFS share on this schedule. That NFS share is
dumped to tape by systems at Conova - I'll let Stefan fill in the
details about that.

//Magnus

Magnus Hagander <magnus(at)hagander(dot)net> writes:
> Tom Lane wrote:
>> Good, but the salient followup questions to that are (1) backed up to
>> where exactly?, and (2) how many days' past backups could we get at,
>> if we had to?

> They are dumped to a NFS share on this schedule. That NFS share is
> dumped to tape by systems at Conova - I'll let Stefan fill in the
> details about that.

That's good as far as it goes, but seeing that PG is a worldwide
organization now, I wonder whether our primary CVS shouldn't have
backups on several continents. Pardon my paranoia ... but our
collective arses have been saved by offsite backups at least once
already ...

regards, tom lane

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>> Tom Lane wrote:
>>> Good, but the salient followup questions to that are (1) backed up to
>>> where exactly?, and (2) how many days' past backups could we get at,
>>> if we had to?
>
>> They are dumped to a NFS share on this schedule. That NFS share is
>> dumped to tape by systems at Conova - I'll let Stefan fill in the
>> details about that.
>
> That's good as far as it goes, but seeing that PG is a worldwide
> organization now, I wonder whether our primary CVS shouldn't have
> backups on several continents. Pardon my paranoia ... but our
> collective arses have been saved by offsite backups at least once
> already ...

The CVS repository is already synced almost-live back to both svr1 in
panama and the anoncvs box in the US. It will be synced to the new VM
marc is setting up as well. I assume these machines also have some kind
of backup - Marc will have to fill in on that.

Oh, and the plan is to have the entire VM synced up between austria and
panama machine once / day again once Marc has fixed the issues with that
box.

//Magnus

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
> > Tom Lane wrote:
> >> Good, but the salient followup questions to that are (1) backed up to
> >> where exactly?, and (2) how many days' past backups could we get at,
> >> if we had to?
>
> > They are dumped to a NFS share on this schedule. That NFS share is
> > dumped to tape by systems at Conova - I'll let Stefan fill in the
> > details about that.
>
> That's good as far as it goes, but seeing that PG is a worldwide
> organization now, I wonder whether our primary CVS shouldn't have
> backups on several continents. Pardon my paranoia ... but our
> collective arses have been saved by offsite backups at least once
> already ...

We already have that -- there's plenty of people who have rsync'ed
copies of the repository. Now, I don't back it up, so if someone makes
a mistake my copy will faithfully follow it.

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On 8/15/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> I wrote:
> > Kris Jurka <books(at)ejurka(dot)com> writes:
> >> It looks like parts of the CVS repository have been mistagged as belonging
> >> to REL7_4_STABLE or have been corrupted somehow:

> I have no idea how you make CVS do that, but I'm
> sure there is some magic one-liner for it.

Something like "cvs tag" without -F on the HEAD?

--
marko

Magnus Hagander wrote:
> Tom Lane wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> Tom Lane wrote:
>>>> Good, but the salient followup questions to that are (1) backed up to
>>>> where exactly?, and (2) how many days' past backups could we get at,
>>>> if we had to?
>>> They are dumped to a NFS share on this schedule. That NFS share is
>>> dumped to tape by systems at Conova - I'll let Stefan fill in the
>>> details about that.
>> That's good as far as it goes, but seeing that PG is a worldwide
>> organization now, I wonder whether our primary CVS shouldn't have
>> backups on several continents. Pardon my paranoia ... but our
>> collective arses have been saved by offsite backups at least once
>> already ...
>
> The CVS repository is already synced almost-live back to both svr1 in
> panama and the anoncvs box in the US. It will be synced to the new VM
> marc is setting up as well. I assume these machines also have some kind
> of backup - Marc will have to fill in on that.
>
> Oh, and the plan is to have the entire VM synced up between austria and
> panama machine once / day again once Marc has fixed the issues with that
> box.

yeah that has been my understanding as well though I'm not sure if marc
has set up the sync of the full VM already.

Stefan

Magnus Hagander wrote:
> Tom Lane wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>> Tom Lane wrote:
>>>> I'd like someone to double-check that though. Also maybe we should back
>>>> up the repository first?
>>> Just for your info: all VMs on tribble, which includes cvs, are backed
>>> up at 02:30 every day, CEST
>> Good, but the salient followup questions to that are (1) backed up to
>> where exactly?, and (2) how many days' past backups could we get at,
>> if we had to?
>
> 1) The entire VM, with "dump"
> 2) We have today, yesterday and one weekly (copies the daily on monday,
> if I read it right)
>
> They are dumped to a NFS share on this schedule. That NFS share is
> dumped to tape by systems at Conova - I'll let Stefan fill in the
> details about that.

that's exactly what happens on the backup side here - but maybe I missed
an earlier mail - is something broken on tribble or on the new CVS VM
that requires a restore from backup ?

Stefan

Stefan Kaltenbrunner wrote:
> Magnus Hagander wrote:
>> Tom Lane wrote:
>>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>>> Tom Lane wrote:
>>>>> I'd like someone to double-check that though. Also maybe we should back
>>>>> up the repository first?
>>>> Just for your info: all VMs on tribble, which includes cvs, are backed
>>>> up at 02:30 every day, CEST
>>> Good, but the salient followup questions to that are (1) backed up to
>>> where exactly?, and (2) how many days' past backups could we get at,
>>> if we had to?
>> 1) The entire VM, with "dump"
>> 2) We have today, yesterday and one weekly (copies the daily on monday,
>> if I read it right)
>>
>> They are dumped to a NFS share on this schedule. That NFS share is
>> dumped to tape by systems at Conova - I'll let Stefan fill in the
>> details about that.
>
> that's exactly what happens on the backup side here - but maybe I missed
> an earlier mail - is something broken on tribble or on the new CVS VM
> that requires a restore from backup ?

erm -ENOTENOUGHCOFFEE

Stefan

Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
> we should at least log such commands, and maybe disallow to anyone
> except Marc's "pgsql" account.

I don't think we should disallow it. Or otherwise we might one day be stuck
if we need to release while some specific person is on vacation.

I never understood why tagging uses a special account anyway. It should be
done as the person doing the tagging.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Tom Lane wrote:
> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>
>> Tom Lane wrote:
>>
>>> Good, but the salient followup questions to that are (1) backed up to
>>> where exactly?, and (2) how many days' past backups could we get at,
>>> if we had to?
>>>
>
>
>> They are dumped to a NFS share on this schedule. That NFS share is
>> dumped to tape by systems at Conova - I'll let Stefan fill in the
>> details about that.
>>
>
> That's good as far as it goes, but seeing that PG is a worldwide
> organization now, I wonder whether our primary CVS shouldn't have
> backups on several continents. Pardon my paranoia ... but our
> collective arses have been saved by offsite backups at least once
> already ...
>
>
>

Yes, I think we could improve on that. Have we considered more
sophisticated solutions that provide incremental backup on a more
frequent basis? I'd be inclined to use Bacula or similar (and it uses
Postgres for its metadata store :-) ). Ideally I think we'd like to be
able fairly easily and quickly to roll the repo (or some portion of it)
back to a fairly arbitrary and fairly precise (say within an hour or
two) point in recent time.

Meanwhile, those of us who rsync the entire repo could easily make
rolling backup copies. Arguably this might be better done from a repo
made using --numeric-ids. Tarred and compressed it's a shade under 90
Mb, which isn't huge. If people do this at various times of the day we'd
get pretty good coverage :-)

cheers

andrew

Andrew Dunstan wrote:
>
>
> Tom Lane wrote:
>> Magnus Hagander <magnus(at)hagander(dot)net> writes:
>>
>>> Tom Lane wrote:
>>>
>>>> Good, but the salient followup questions to that are (1) backed up to
>>>> where exactly?, and (2) how many days' past backups could we get at,
>>>> if we had to?
>>>>
>>
>>
>>> They are dumped to a NFS share on this schedule. That NFS share is
>>> dumped to tape by systems at Conova - I'll let Stefan fill in the
>>> details about that.
>>>
>>
>> That's good as far as it goes, but seeing that PG is a worldwide
>> organization now, I wonder whether our primary CVS shouldn't have
>> backups on several continents. Pardon my paranoia ... but our
>> collective arses have been saved by offsite backups at least once
>> already ...
>>
>>
>>
>
> Yes, I think we could improve on that. Have we considered more
> sophisticated solutions that provide incremental backup on a more
> frequent basis? I'd be inclined to use Bacula or similar (and it uses
> Postgres for its metadata store :-) ). Ideally I think we'd like to be
> able fairly easily and quickly to roll the repo (or some portion of it)
> back to a fairly arbitrary and fairly precise (say within an hour or
> two) point in recent time.

well yeah - while I do think that something as complex like bacula is
probably overkill for our needs we can certainly improve over the
current state.
One thing to consider is that we actually have two major scenarios to
deal with:

1. simple repo corruption (accident,cvs software bug, admin error)
this one might require us to restore the repo from an older backup in
the case the corruption cannot be repaired easily.
For this we already have myriads of copies of the trees out in the wild
but i might be a good idea to keep a number snapshots of the main repo
on the CVS-VPS itself (either done every few hours or made as part of
the push to anoncvs and svr1).
This way we could do a very simple "inplace" recovery on the same
running VPS instance with fairly low inpact to all the commiters (and
depending parts of teh infrastructure)

2. total loss of the main CVS-VPS (extended power failure, hardware
error, OS bug, admin error, fire, some other catastropic event) - in
this case we will have to fail over to one of the other project
datacenters and for this we need to have full regular copies of the
whole VM on external hosts.

Stefan

Peter Eisentraut wrote:
> Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
>> we should at least log such commands, and maybe disallow to anyone
>> except Marc's "pgsql" account.
>
> I don't think we should disallow it. Or otherwise we might one day be stuck
> if we need to release while some specific person is on vacation.

Is this really a problem in practise ?
If such a need arises any commiter (or sysadmin) would probably be able
to remove that restriction in a few minutes.
I think the point here is to prevent such things done by accident(vs. on
purpose) which a script like that seems like a fairly simple yet
effective solution.

Stefan

Stefan Kaltenbrunner <stefan(at)kaltenbrunner(dot)cc> writes:
> Peter Eisentraut wrote:
>> Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
>>> we should at least log such commands, and maybe disallow to anyone
>>> except Marc's "pgsql" account.
>>
>> I don't think we should disallow it. Or otherwise we might one day be stuck
>> if we need to release while some specific person is on vacation.

> Is this really a problem in practise ?

I'd be happy if such commands got reported to the pgsql-committers
mailing list. The real problem with this mistake is not so much
that it was made as that we had to do detective work to find out.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut
<peter_e(at)gmx(dot)net> wrote:

> Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
>> we should at least log such commands, and maybe disallow to anyone
>> except Marc's "pgsql" account.
>
> I don't think we should disallow it. Or otherwise we might one day be stuck
> if we need to release while some specific person is on vacation.

This isn't a big issue ... note that the 'restriction' is easy to remove ...
you checkout CVSROOT, modify taginfo and comment out the ALL l ine and check
that in ...

What this will prevent is an 'accidental tagging' ... you would have to
consciously remove the restriction .. but its something anyone could do ...

> I never understood why tagging uses a special account anyway. It should be
> done as the person doing the tagging.

Agreed, I'm going to start doing it as myself from now on ... I'm not even 100%
certain *why* I started doing it as pgsql in the first place ...

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGwye54QvfyHIvDvMRAiHEAKCHGtXA+r5PM1SoBewMJDo3An7BKACg0//z
gM7eZNoTEU4sqwBIHprFK1k=
=/Qga
-----END PGP SIGNATURE-----

Marc G. Fournier wrote:
>
>
> --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut
> <peter_e(at)gmx(dot)net> wrote:
>
>> Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
>>> we should at least log such commands, and maybe disallow to anyone
>>> except Marc's "pgsql" account.
>> I don't think we should disallow it. Or otherwise we might one day be stuck
>> if we need to release while some specific person is on vacation.
>
> This isn't a big issue ... note that the 'restriction' is easy to remove ...
> you checkout CVSROOT, modify taginfo and comment out the ALL l ine and check
> that in ...
>
> What this will prevent is an 'accidental tagging' ... you would have to
> consciously remove the restriction .. but its something anyone could do ...
>
>> I never understood why tagging uses a special account anyway. It should be
>> done as the person doing the tagging.
>
> Agreed, I'm going to start doing it as myself from now on ... I'm not even 100%
> certain *why* I started doing it as pgsql in the first place ...

If you're doing that, we should probably just delete the pgsql userid
from the system? Or at least change it so it doesn't have 'dev'
permissions. That way you can't do it wrong in that direction ;-)
Seems reasonable?

//Magnus

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

- --On Wednesday, August 15, 2007 20:57:14 +0200 Magnus Hagander
<magnus(at)hagander(dot)net> wrote:

> Marc G. Fournier wrote:
>>
>>
>> --On Wednesday, August 15, 2007 12:11:35 +0200 Peter Eisentraut
>> <peter_e(at)gmx(dot)net> wrote:
>>
>>> Am Mittwoch, 15. August 2007 04:20 schrieb Tom Lane:
>>>> we should at least log such commands, and maybe disallow to anyone
>>>> except Marc's "pgsql" account.
>>> I don't think we should disallow it. Or otherwise we might one day be stuck
>>> if we need to release while some specific person is on vacation.
>>
>> This isn't a big issue ... note that the 'restriction' is easy to remove
>> ... you checkout CVSROOT, modify taginfo and comment out the ALL l ine and
>> check that in ...
>>
>> What this will prevent is an 'accidental tagging' ... you would have to
>> consciously remove the restriction .. but its something anyone could do ...
>>
>>> I never understood why tagging uses a special account anyway. It should be
>>> done as the person doing the tagging.
>>
>> Agreed, I'm going to start doing it as myself from now on ... I'm not even
>> 100% certain *why* I started doing it as pgsql in the first place ...
>
> If you're doing that, we should probably just delete the pgsql userid
> from the system? Or at least change it so it doesn't have 'dev'
> permissions. That way you can't do it wrong in that direction ;-)
> Seems reasonable?

there is no pgsql user *on* cvs.postgresql.org, at least there wasn't when I
logged on last night:

cvs# grep pgsql /etc/passwd
cvs#

- ----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email . scrappy(at)hub(dot)org MSN . scrappy(at)hub(dot)org
Yahoo . yscrappy Skype: hub.org ICQ . 7615664
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFGw1AR4QvfyHIvDvMRArsSAJ9WzXyMu7Io6dP8kDnR5HHex1f1gQCg7iDq
3p2RrAzww4dCDPVCyMozAnM=
=L+/M
-----END PGP SIGNATURE-----

Marc G. Fournier wrote:
>> If you're doing that, we should probably just delete the pgsql userid
>> from the system? Or at least change it so it doesn't have 'dev'
>> permissions. That way you can't do it wrong in that direction ;-)
>> Seems reasonable?
>
> there is no pgsql user *on* cvs.postgresql.org, at least there wasn't when I
> logged on last night:
>
> cvs# grep pgsql /etc/passwd
> cvs#

Ah, seems I was faster than I expected myself :-) Problem solved then.

//Magnus