Re: Buildfarm coverage (was Re: OK, ready for RC1 or Beta6)

OK, where are we in the release process? We still have a few open
items, but those can be moved to the TODO list. Do we do RC1 or Beta6?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:
> OK, where are we in the release process? We still have a few open
> items, but those can be moved to the TODO list. Do we do RC1 or
> Beta6?

Considering all the patching that has been going on recently and the
fact that we don't have any port reports, I think it's too early for a
"release candidate". I think we should release beta 6 immediately,
call for port reports based on it, and then release RC1 as soon as we
have an idea where the port reports are going, possibly in less than
one week.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Bruce Momjian wrote:
> > OK, where are we in the release process? We still have a few open
> > items, but those can be moved to the TODO list. Do we do RC1 or
> > Beta6?
>
> Considering all the patching that has been going on recently and the
> fact that we don't have any port reports, I think it's too early for a
> "release candidate". I think we should release beta 6 immediately,
> call for port reports based on it, and then release RC1 as soon as we
> have an idea where the port reports are going, possibly in less than
> one week.

What about the build farm? Can't we assume we are pretty close to RC?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:
> Peter Eisentraut wrote:
>
>>Bruce Momjian wrote:
>>
>>>OK, where are we in the release process? We still have a few open
>>>items, but those can be moved to the TODO list. Do we do RC1 or
>>>Beta6?
>>
>>Considering all the patching that has been going on recently and the
>>fact that we don't have any port reports, I think it's too early for a
>>"release candidate". I think we should release beta 6 immediately,
>>call for port reports based on it, and then release RC1 as soon as we
>>have an idea where the port reports are going, possibly in less than
>>one week.
>
>
> What about the build farm? Can't we assume we are pretty close to RC?

I would say no.

1. Buildfarm doesn't yet have that many platforms on it.
2. There are critical notices on buildfarm for some more popular
platforms such as Solaris 9 and Open BSD.

Sincerely,

Joshua D. Drake

>

--
Command Prompt, Inc., home of PostgreSQL Replication, and plPHP.
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

On Fri, 3 Dec 2004, Peter Eisentraut wrote:

> Bruce Momjian wrote:
>> OK, where are we in the release process? We still have a few open
>> items, but those can be moved to the TODO list. Do we do RC1 or
>> Beta6?
>
> Considering all the patching that has been going on recently and the
> fact that we don't have any port reports, I think it's too early for a
> "release candidate". I think we should release beta 6 immediately,
> call for port reports based on it, and then release RC1 as soon as we
> have an idea where the port reports are going, possibly in less than
> one week.

If we are only concerned with Port related bug reports, and nothing that
should require an initdb, I'd be okay with an RC1 ... it would indicate to
everyone that we've finally locked down the code, and are only dealing
with critical bug reports and documentation ...

Thing is, I think we'd get more serious testing once we do say the code is
locked in "Release Mode", then if we release yet another beta ..

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

On December 3, 2004 10:31 am, you wrote:
> Bruce Momjian wrote:
> > Peter Eisentraut wrote:
> >>Bruce Momjian wrote:
> >>>OK, where are we in the release process? We still have a few open
> >>>items, but those can be moved to the TODO list. Do we do RC1 or
> >>>Beta6?
> >>
> >>Considering all the patching that has been going on recently and the
> >>fact that we don't have any port reports, I think it's too early for a
> >>"release candidate". I think we should release beta 6 immediately,
> >>call for port reports based on it, and then release RC1 as soon as we
> >>have an idea where the port reports are going, possibly in less than
> >>one week.
> >
> > What about the build farm? Can't we assume we are pretty close to RC?
>
> I would say no.
>
> 1. Buildfarm doesn't yet have that many platforms on it.
> 2. There are critical notices on buildfarm for some more popular
> platforms such as Solaris 9 and Open BSD.

The OpenBSD error should be fixed by
http://developer.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/libpq/fe-secure.c?f=h;rev=1.60

and other than that I see noting on the build far that is questionable (other
than the win32 regres test notice)

>
>
> Sincerely,
>
> Joshua D. Drake

--
Darcy Buskermolen
Wavefire Technologies Corp.
ph: 250.717.0200
fx: 250.763.1759
http://www.wavefire.com

Darcy Buskermolen <darcy(at)wavefire(dot)com> writes:
> On December 3, 2004 10:31 am, you wrote:
>> 2. There are critical notices on buildfarm for some more popular
>> platforms such as Solaris 9 and Open BSD.

> The OpenBSD error should be fixed by
> http://developer.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/libpq/fe-secure.c?f=h;rev=1.60

Right, that was my silly oversight last night.

> and other than that I see noting on the build far that is questionable (other
> than the win32 regres test notice)

It's too bad the buildfarm reports don't show more details about what
CVS pull they're using exactly. I think that this case might be fixed
by the tweaking I did yesterday, but I can't tell whether that run
occurred before or after that commit. In any case it's not a real
failure, just an output-ordering difference.

regards, tom lane

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> 1. Buildfarm doesn't yet have that many platforms on it.

It's not as bad as all that. Our current list of supported platforms
(ie, things that got tested last time) is

AIX
Free/Open/NetBSD covered by buildfarm
HPUX used daily by moi
IRIX
Linux covered by buildfarm
OS X tested pretty often by moi
Solaris covered by buildfarm
Tru64
UnixWare
Windows/Cygwin covered by buildfarm

With respect to hardware it's

x86 covered by buildfarm
ia64
x86_64 covered by buildfarm
ARM
Alpha
MIPS covered by buildfarm
m68k
PA-RISC used daily by moi
PPC tested pretty often by moi
RS6000 isn't this same as PPC?
S/390
Sparc covered by buildfarm

Considering that we have both 32- and 64-bit, little- and big-endian
hardware in there, most of the basic hardware gotchas are covered;
the only thing I think is at much risk is the spinlock assembler code,
which we change seldom.

Where the buildfarm falls down a bit is on the cross-product coverage.
But I think you're not going to get the cross product without a call for
port reports; there aren't that many people who are going to offer
dedicated time on every random platform there is.

It would be nice to get an Alpha into the buildfarm, and PPC too.

regards, tom lane

On Fri, Dec 03, 2004 at 03:20:48PM -0500, Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > 1. Buildfarm doesn't yet have that many platforms on it.
>
> It's not as bad as all that. Our current list of supported platforms
> (ie, things that got tested last time) is
>
> AIX
> Free/Open/NetBSD covered by buildfarm
> HPUX used daily by moi
> IRIX
> Linux covered by buildfarm
> OS X tested pretty often by moi
> Solaris covered by buildfarm
> Tru64
> UnixWare
> Windows/Cygwin covered by buildfarm
>
> With respect to hardware it's
>
> x86 covered by buildfarm
> ia64
> x86_64 covered by buildfarm
> ARM
> Alpha
> MIPS covered by buildfarm
> m68k
> PA-RISC used daily by moi
> PPC tested pretty often by moi
> RS6000 isn't this same as PPC?
This is the IBM Power4 and now Power5 architecture which is
different from the PowerPC.

Ken

Tom Lane wrote:
> Where the buildfarm falls down a bit is on the cross-product
> coverage. But I think you're not going to get the cross product
> without a call for port reports; there aren't that many people who
> are going to offer dedicated time on every random platform there is.

Once RC1 is out and the build farm has picked it up, we should start
filling out our little table with the build farm results and then look
for ways to fill the holes. Does the build farm turn on all the
compiler options? It really should. I'm looking for

/configure --prefix=SOMEWHERE --enable-thread-safety --with-tcl \
--with-perl --with-python --with-krb5 --with-pam -with-openssl
make
make install
make check

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Does the build farm turn on all the
> compiler options? It really should. I'm looking for

> /configure --prefix=SOMEWHERE --enable-thread-safety --with-tcl \
> --with-perl --with-python --with-krb5 --with-pam -with-openssl

I was just thinking about what the buildfarm should do with configure
options. IMHO it would be most useful if we could have it testing a
variety of different combinations. For instance, that stupid mistake
I made last night would only have been detected by testing the
combination --with-openssl and *not* --enable-thread-safety. We
obviously are not going to have enough machines to test every possible
combination, let alone every combination on every platform, but maybe we
could make sure that interesting option combinations appear at least
once among the set of buildfarm machines.

I think it would be useful to cover all 16 permutations of
--enable-thread-safety --with-krb5 --with-pam -with-openssl
if possible, since those potentially interact in libpq. The
--with-tcl --with-perl --with-python switches are probably pretty
independent of these, but we should have one or two buildfarm machines
trying each language if possible. Other switches that should be getting
used by some but not all machines:

--enable-integer-datetimes
--enable-nls
--without-readline (just to make sure it builds)
--without-zlib (ditto)

Finally, while I think most of the platforms should use --enable-debug
and --enable-cassert to aid in tracking down problems, there should be
one machine building without these, just to catch silly mistakes.

regards, tom lane

Tom Lane wrote:

>
>
>It's too bad the buildfarm reports don't show more details about what
>CVS pull they're using exactly.
>

Snapshot is the UTC time at which the cvs pull was done. Clients report
what files have changed since the last run, and also, in the case of a
failure, what files have changed since the last successful run. See for
example
http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=dog&dt=2004-12-03%2000:06:02

The Windows and Cygwin clients are not currently doing this, as they are
running experimental code in which it has been temporarily disabled.

I guess I could actually get CVS revision info via cvs status for these
files and report it, if you think that would be useful. This at least is
one case where another SCR system than CVS would be nicer - in SVN for
example you would just report the tree id.

>I think that this case might be fixed
>by the tweaking I did yesterday, but I can't tell whether that run
>occurred before or after that commit. In any case it's not a real
>failure, just an output-ordering difference.
>
>

I am running it again to see. I agree that at worst it would require an
alternative output file, assuming we aren't bothered by these ordering
differences.

cheers

andrew

Tom/all,

I have setup the following running debian linux. MIPS, MIPSEL, ALPHA, PARISC, M68K, ARM, SPARC, I386. I have the
build farm running local and I have just started to get the systems registered. I am also willing to aquire other
hardware/ operating systems in an effort to give something back to the Postgresql community

Jim

---------- Original Message -----------
From: Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us>
To: "Joshua D. Drake" <jd(at)commandprompt(dot)com>
Cc: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Sent: Fri, 03 Dec 2004 15:20:48 -0500
Subject: Buildfarm coverage (was Re: [HACKERS] OK, ready for RC1 or Beta6)

> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > 1. Buildfarm doesn't yet have that many platforms on it.
>
> It's not as bad as all that. Our current list of supported platforms
> (ie, things that got tested last time) is
>
> AIX
> Free/Open/NetBSD covered by buildfarm
> HPUX used daily by moi
> IRIX
> Linux covered by buildfarm
> OS X tested pretty often by moi
> Solaris covered by buildfarm
> Tru64
> UnixWare
> Windows/Cygwin covered by buildfarm
>
> With respect to hardware it's
>
> x86 covered by buildfarm
> ia64
> x86_64 covered by buildfarm
> ARM
> Alpha
> MIPS covered by buildfarm
> m68k
> PA-RISC used daily by moi
> PPC tested pretty often by moi
> RS6000 isn't this same as PPC?
> S/390
> Sparc covered by buildfarm
>
> Considering that we have both 32- and 64-bit, little- and big-endian
> hardware in there, most of the basic hardware gotchas are covered;
> the only thing I think is at much risk is the spinlock assembler code,
> which we change seldom.
>
> Where the buildfarm falls down a bit is on the cross-product coverage.
> But I think you're not going to get the cross product without a call for
> port reports; there aren't that many people who are going to offer
> dedicated time on every random platform there is.
>
> It would be nice to get an Alpha into the buildfarm, and PPC too.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
------- End of Original Message -------

Peter Eisentraut wrote:

>Tom Lane wrote:
>
>
>>Where the buildfarm falls down a bit is on the cross-product
>>coverage. But I think you're not going to get the cross product
>>without a call for port reports; there aren't that many people who
>>are going to offer dedicated time on every random platform there is.
>>
>>
>
>Once RC1 is out and the build farm has picked it up, we should start
>filling out our little table with the build farm results and then look
>for ways to fill the holes. Does the build farm turn on all the
>compiler options? It really should. I'm looking for
>
>/configure --prefix=SOMEWHERE --enable-thread-safety --with-tcl \
>--with-perl --with-python --with-krb5 --with-pam -with-openssl
>make
>make install
>make check
>
>

I know you're busy, but please take a few moments to check out how it works.

The configuration is chosen in the config file for each member, rather
than being dictated centrally. Every report (success and failure) shows
the config settings. The only setting that buildfarm needs to set itself
is --prefix which is chosen relative to another config file setting.
Everything else is up to the farm member sysadmin to choose appropriately.

A single member can run more than one branch, and per-branch config can
be set up. A single machine can run more than one farm member (e.g. to
use different compilers).

In addition to the steps above, we do the following:

initdb
pg_ctl start
make installcheck
make contrib
make contrib::installcheck
pg_ctl stop

The worth of these extra steps has already been shown - a number of bugs
of unknown age and provenance have been fixed, especially in contrib.

It is possible to run the buildfarm yourself without even being
registered on www.pgbuildfarm.org - the script has a --nosend option
which means it doesn't try to upload its results to the server. I
encourage anyone interested in running buildfarm to try this option first.

cheers

andrew

On December 3, 2004 11:14 am, Tom Lane wrote:
> Darcy Buskermolen <darcy(at)wavefire(dot)com> writes:
> > On December 3, 2004 10:31 am, you wrote:
> >> 2. There are critical notices on buildfarm for some more popular
> >> platforms such as Solaris 9 and Open BSD.
> >
> > The OpenBSD error should be fixed by
> > http://developer.postgresql.org/cvsweb.cgi/pgsql/src/interfaces/libpq/fe-
> >secure.c?f=h;rev=1.60
>
> Right, that was my silly oversight last night.
>
> > and other than that I see noting on the build far that is questionable
> > (other than the win32 regres test notice)
>
> It's too bad the buildfarm reports don't show more details about what
> CVS pull they're using exactly. I think that this case might be fixed
> by the tweaking I did yesterday, but I can't tell whether that run
> occurred before or after that commit. In any case it's not a real
> failure, just an output-ordering difference.

http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=loris&dt=2004-12-03%2020:54:53

Lends me to think your tweek didn't push hard enough in the right spots.

>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

--
Darcy Buskermolen
Wavefire Technologies Corp.
ph: 250.717.0200
fx: 250.763.1759
http://www.wavefire.com

Andrew Dunstan wrote:

>
>
>> I think that this case might be fixed
>> by the tweaking I did yesterday, but I can't tell whether that run
>> occurred before or after that commit. In any case it's not a real
>> failure, just an output-ordering difference.
>>
>>
>
> I am running it again to see. I agree that at worst it would require
> an alternative output file, assuming we aren't bothered by these
> ordering differences.
>
>

No, it's still there :-(

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> The configuration is chosen in the config file for each member, rather
> than being dictated centrally.

This is good. Now what we need is a little cooperation among the
buildfarm team to make sure that the collective set of cases tested
covers all the interesting combinations of configure flags, as per
my followup ...

> A single member can run more than one branch, and per-branch config can
> be set up. A single machine can run more than one farm member (e.g. to
> use different compilers).

Yeah, on platforms where there's a non-gcc vendor compiler, testing with
the vendor compiler is very interesting.

regards, tom lane

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Tom Lane wrote:
>> It's too bad the buildfarm reports don't show more details about what
>> CVS pull they're using exactly.

> Snapshot is the UTC time at which the cvs pull was done.

That's good but it's of limited use to me, since the snaps are (I
presume) against the anonymous-CVS server which lags commits on the
master by I'm-not-sure-how-much.

> Clients report
> what files have changed since the last run, and also, in the case of a
> failure, what files have changed since the last successful run.

Cool, I had not seen it do that before. If we could get the CVS version
number of each mentioned file into that, it would go a long way towards
helping identify exactly what's being tested.

regards, tom lane

"Jim Buttafuoco" <jim(at)contactbda(dot)com> writes:
> I have setup the following running debian linux. MIPS, MIPSEL, ALPHA,
> PARISC, M68K, ARM, SPARC, I386. I have the build farm running local
> and I have just started to get the systems registered.

Excellent, that's very good news.

regards, tom lane

Darcy Buskermolen <darcy(at)wavefire(dot)com> writes:
> http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=loris&dt=2004-12-03%2020:54:53

> Lends me to think your tweek didn't push hard enough in the right spots.

Yup, you're right. I used a bigger hammer ;-)

regards, tom lane

On Dec 3, 2004, at 2:33 PM, Kenneth Marshall wrote:

> On Fri, Dec 03, 2004 at 03:20:48PM -0500, Tom Lane wrote:
>> PPC tested pretty often by moi
>> RS6000 isn't this same as PPC?
> This is the IBM Power4 and now Power5 architecture which is
> different from the PowerPC.

Yeah, it's confusing. I believe that Power3 (also known as PowerPC
630), Power4, and Power5 satisfy the requirements of being both Power
architecture and PowerPC architecture processors.

Not all PowerPC processors are Power processors. I believe that all
modern Power processors are PowerPC processors (the Power2 "P2SC" was
the last non-PowerPC Power processor, IIRC).

IBM's Power architecture has architectural features for Server systems
(with a capital S there) that PowerPC for workstations (Apple) and
embedded (Moto/IBM) shouldn't be required to have, and is also IBM's
own solely-owned branding. Hence the differentiation.

That's what I've pieced together anyway.

You'll probably find multi-OS-testing (various versions of AIX, Linux,
MacOS X on PPC and/or PowerPC) much more important than differentiating
particular pieces of hardware in the PPC or RS6000 category, assuming
both 32-bit and 64-bit is covered and also that SMP tests are made.

Does 'make check' test SMP?

-Travis

On Fri, 3 Dec 2004, Tom Lane wrote:

> That's good but it's of limited use to me, since the snaps are (I
> presume) against the anonymous-CVS server which lags commits on the
> master by I'm-not-sure-how-much.

19 * * * * /projects/update_anoncvs.sh

----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: scrappy(at)hub(dot)org Yahoo!: yscrappy ICQ: 7615664

Travis P <twp(at)castle(dot)fastmail(dot)fm> writes:
> You'll probably find multi-OS-testing (various versions of AIX, Linux,
> MacOS X on PPC and/or PowerPC) much more important than differentiating
> particular pieces of hardware in the PPC or RS6000 category, assuming
> both 32-bit and 64-bit is covered and also that SMP tests are made.

Agreed.

> Does 'make check' test SMP?

Yes; not very hard, but it will usually catch bad problems, especially
over the long haul of many retries.

regards, tom lane

Tom Lane wrote:

>Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>
>
>>Tom Lane wrote:
>>
>>
>>>It's too bad the buildfarm reports don't show more details about what
>>>CVS pull they're using exactly.
>>>
>>>
>
>
>
>>Snapshot is the UTC time at which the cvs pull was done.
>>
>>
>
>That's good but it's of limited use to me, since the snaps are (I
>presume) against the anonymous-CVS server which lags commits on the
>master by I'm-not-sure-how-much.
>
>

In my case it's against my local CVSup mirror, which is updated every 15
minutes. The buildfarm config lets you choose what repo to use.

Nevertheless I take your point.

>
>
>>Clients report
>>what files have changed since the last run, and also, in the case of a
>>failure, what files have changed since the last successful run.
>>
>>
>
>Cool, I had not seen it do that before. If we could get the CVS version
>number of each mentioned file into that, it would go a long way towards
>helping identify exactly what's being tested.
>
>

Feature request filed on pgfoundry - not sure when I'll get to it.

cheers

andrew

On Fri, Dec 03, 2004 at 09:37:42PM +0100, Peter Eisentraut wrote:
>
> Once RC1 is out and the build farm has picked it up, we should start
> filling out our little table with the build farm results and then look
> for ways to fill the holes. Does the build farm turn on all the
> compiler options? It really should. I'm looking for
>
> /configure --prefix=SOMEWHERE --enable-thread-safety --with-tcl \
> --with-perl --with-python --with-krb5 --with-pam -with-openssl

I actually run panda (x86_64/amd64) with those options together with
--enable-nls and --enable-integer-datetimes. I based my options
on the debian postgresql package.

Kurt