Re: Machine available for community use

Recently I've been involved in or overheard discussions about SMP
scalability at both the PA PgSQL get together and in some list
traffic.

myYearbook.com would ike to make one of our previous production
machines available to established PgSQL Hackers who don't have access
to this level of hardware for testing, benchmarking and development to
work at improving SMP scalability and related projects.

The machine is a HP 585 G1, 8 Core AMD, 32GB RAM with one 400GB 14
Spindle DAS Array dedicated to community use. I've attached a text
file with dmesg and /proc/cpuinfo output.

I'm working on how this will be setup and am open to suggestions on
how to structure access.

I'm currently in the process of having Gentoo linux reinstalled on the
box since that is what I am most comfortable administering from a
security perspective. If this will be a blocker for developers who
would actually work on it, please let me know.

If you're interested in access, my only requirement is that you're a
current PgSQL Hacker with a proven track-record of committing patches
to the community. This is a resource we could be using for something
else, and I'd like to see the community get direct benefit from it as
opposed to it being a play sandbox for people who want to tinker.

Please let me know thoughts, concerns or suggestions.

Gavin M. Roy
CTO
myYearbook.com
gmr(at)myyearbook(dot)com

"Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> I'm currently in the process of having Gentoo linux reinstalled on the
> box since that is what I am most comfortable administering from a
> security perspective. If this will be a blocker for developers who
> would actually work on it, please let me know.

Personally I'd prefer almost any of the other Linux distros.
Gentoo always leaves me wondering exactly what I'm running today,
and I think reproducibility is an important attribute for a benchmarking
machine.

regards, tom lane

If you're interested in using the box, name what you want installed.

On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > I'm currently in the process of having Gentoo linux reinstalled on the
> > box since that is what I am most comfortable administering from a
> > security perspective. If this will be a blocker for developers who
> > would actually work on it, please let me know.
>
> Personally I'd prefer almost any of the other Linux distros.
> Gentoo always leaves me wondering exactly what I'm running today,
> and I think reproducibility is an important attribute for a benchmarking
> machine.
>
> regards, tom lane
>

On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > I'm currently in the process of having Gentoo linux reinstalled on the
> > box since that is what I am most comfortable administering from a
> > security perspective. If this will be a blocker for developers who
> > would actually work on it, please let me know.
>
> Personally I'd prefer almost any of the other Linux distros.
> Gentoo always leaves me wondering exactly what I'm running today,
> and I think reproducibility is an important attribute for a benchmarking
> machine.

Tom, have any specific ideas in mind for using the system? While I'm
used to having more disks it could be useful nonetheless for the tests
I used to run if there are no other ideas.

Rats, I've always liked Gentoo. ;)

Regards,
Mark

Note it's a 28 disk system, and I can allocate more if needed, but I
was going to use one MSA for internal use.

On 7/25/07, Mark Wong <markwkm(at)gmail(dot)com> wrote:
> On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > > I'm currently in the process of having Gentoo linux reinstalled on the
> > > box since that is what I am most comfortable administering from a
> > > security perspective. If this will be a blocker for developers who
> > > would actually work on it, please let me know.
> >
> > Personally I'd prefer almost any of the other Linux distros.
> > Gentoo always leaves me wondering exactly what I'm running today,
> > and I think reproducibility is an important attribute for a benchmarking
> > machine.
>
> Tom, have any specific ideas in mind for using the system? While I'm
> used to having more disks it could be useful nonetheless for the tests
> I used to run if there are no other ideas.
>
> Rats, I've always liked Gentoo. ;)
>
> Regards,
> Mark
>

"Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> If you're interested in using the box, name what you want installed.

Personally I use Fedora, but that's because of where I work ;-).
I have no objection to some other distro so long as it's one where
other people can duplicate your environment easily (no locally
compiled stuff). A disadvantage of Fedora is its relatively short
support lifetime --- if you don't want to have to reinstall at least
once a year, something else would be better.

regards, tom lane

Ubuntu server? Slackware? Not a fan of Centos, RHEL or Fedora...
What about on the BSD side of things?

On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > If you're interested in using the box, name what you want installed.
>
> Personally I use Fedora, but that's because of where I work ;-).
> I have no objection to some other distro so long as it's one where
> other people can duplicate your environment easily (no locally
> compiled stuff). A disadvantage of Fedora is its relatively short
> support lifetime --- if you don't want to have to reinstall at least
> once a year, something else would be better.
>
> regards, tom lane
>

On Wed, 25 Jul 2007, Gavin M. Roy wrote:

> Ubuntu server? Slackware? Not a fan of Centos, RHEL or Fedora...

Unless you did a custom intall, using Ubuntu server would expose the
people using your server to the quirks of how the Debian packages for
PostgreSQL differ from other Linux distributions. I'm not sure whether
that would be a good (shine some light on that underdocumented area) or
bad (get in people's way) thing. The way they make it easier to manage
multiple clusters might actually be ideal for what you're trying to do,
let people have their own cluster and stay out of each other's data space
at least.

I think Slackware has all the non-mainstream issues of Gentoo, but without
the advantages Portage brings.

> What about on the BSD side of things?

Since your goal is improve scalability on Linux, I think you'd be best
focusing on that. There's just enough low-level differences between the
two that I'd hate to see you put resources into improving scaling, only to
discover it doesn't actually help what you put into production because the
platform is too different.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Wed, 2007-07-25 at 08:50 -0700, Mark Wong wrote:
> On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > > I'm currently in the process of having Gentoo linux reinstalled on the
> > > box since that is what I am most comfortable administering from a
> > > security perspective. If this will be a blocker for developers who
> > > would actually work on it, please let me know.

Gavin, I'd like access please. This sounds very cool. We'll be able to
show each other directly what's going on, even log on together to
inspect various aspects of runs.

Will you run a booking system?

Could you give us some details about myYearbook.com's application? I
feel we should prioritise work slightly so that the contributor can see
some benefit coming their way in the longer term.

> > Personally I'd prefer almost any of the other Linux distros.
> > Gentoo always leaves me wondering exactly what I'm running today,
> > and I think reproducibility is an important attribute for a benchmarking
> > machine.
>
> Tom, have any specific ideas in mind for using the system? While I'm
> used to having more disks it could be useful nonetheless for the tests
> I used to run if there are no other ideas.

Mark, If you're thinking TPC-E, so am I. Where are we with the TPC-E
toolkit you guys were working on?

Initially though, I'd like to do some tests on CVS HEAD with large
shared_buffers settings, so the 32GB RAM will come in handy for that and
no worries about disks.

> Rats, I've always liked Gentoo. ;)

I'd agree with Tom on that: we need a system that remains the same over
longer periods, not simply a very fast one. I'm OK with Fedora.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

On 7/25/07, Simon Riggs <simon(at)2ndquadrant(dot)com> wrote:
> On Wed, 2007-07-25 at 08:50 -0700, Mark Wong wrote:
> > On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > > "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > > > I'm currently in the process of having Gentoo linux reinstalled on the
> > > > box since that is what I am most comfortable administering from a
> > > > security perspective. If this will be a blocker for developers who
> > > > would actually work on it, please let me know.
>
> Gavin, I'd like access please. This sounds very cool. We'll be able to
> show each other directly what's going on, even log on together to
> inspect various aspects of runs.
>
> Will you run a booking system?
>
> Could you give us some details about myYearbook.com's application? I
> feel we should prioritise work slightly so that the contributor can see
> some benefit coming their way in the longer term.
>
> > > Personally I'd prefer almost any of the other Linux distros.
> > > Gentoo always leaves me wondering exactly what I'm running today,
> > > and I think reproducibility is an important attribute for a benchmarking
> > > machine.
> >
> > Tom, have any specific ideas in mind for using the system? While I'm
> > used to having more disks it could be useful nonetheless for the tests
> > I used to run if there are no other ideas.
>
> Mark, If you're thinking TPC-E, so am I. Where are we with the TPC-E
> toolkit you guys were working on?
>
> Initially though, I'd like to do some tests on CVS HEAD with large
> shared_buffers settings, so the 32GB RAM will come in handy for that and
> no worries about disks.

Yeah, the the C stored functions are half done but there is a finished
implementation for the pl/pgsql stored functions. It's in decent
shape otherwise, although it's mostly based on the 0.32 version.

> > Rats, I've always liked Gentoo. ;)
>
> I'd agree with Tom on that: we need a system that remains the same over
> longer periods, not simply a very fast one. I'm OK with Fedora.

True, I'll settle for whatever everyone agrees with.

Regards,
Mark

On Wed, 25 Jul 2007, Tom Lane wrote:

> Gentoo always leaves me wondering exactly what I'm running today,
> and I think reproducibility is an important attribute for a benchmarking
> machine.

At this point, there's enough performance variations even between
individual Linux kernel releases that I'm not sure how much
reproducibility you're ever going to get here. Are the differences
between Gentoo and RHEL any bigger than those, say, between RHEL and SuSE?

The idea of setting this up with a long-term stable distribution runs
counter to one of the things that I think is important to explore here,
which is testing how more recent Linux kernels have improved their
scalability. Do you really want to put a lot of time into identifying and
working around the source of a problem with the typically older kernels
that ship with the more stable releases if one answer is "that goes away
if you use 2.6.21 or later because they fixed the bug that caused it"?
I've watched that sort of thing happen with PG+Linux, and when involved in
one of the recent roving talks Gavin mentioned I recall him mentioning a
bit of that experience himself. You'd be hard pressed to find a better
platform for that kind of experimentation than Gentoo.

The best you can hope for, I think, is that you can walk away with some
general benchmark expectations and "on Gavin's machine, this worked
better"; then try to replicate that improvement elsewhere. If you want to
push bleeding edge performance, I'd expect it's impractical to do that and
target long-term results stability at the same time.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Simon Riggs wrote:
> On Wed, 2007-07-25 at 08:50 -0700, Mark Wong wrote:
>> On 7/25/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>>> "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
>>>> I'm currently in the process of having Gentoo linux reinstalled on the
>>>> box since that is what I am most comfortable administering from a
>>>> security perspective. If this will be a blocker for developers who
>>>> would actually work on it, please let me know.
>
> Gavin, I'd like access please. This sounds very cool. We'll be able to
> show each other directly what's going on, even log on together to
> inspect various aspects of runs.
>
> Will you run a booking system?
>
> Could you give us some details about myYearbook.com's application? I
> feel we should prioritise work slightly so that the contributor can see
> some benefit coming their way in the longer term.
>
>>> Personally I'd prefer almost any of the other Linux distros.
>>> Gentoo always leaves me wondering exactly what I'm running today,
>>> and I think reproducibility is an important attribute for a benchmarking
>>> machine.
>> Tom, have any specific ideas in mind for using the system? While I'm
>> used to having more disks it could be useful nonetheless for the tests
>> I used to run if there are no other ideas.
>
> Mark, If you're thinking TPC-E, so am I. Where are we with the TPC-E
> toolkit you guys were working on?
>
> Initially though, I'd like to do some tests on CVS HEAD with large
> shared_buffers settings, so the 32GB RAM will come in handy for that and
> no worries about disks.
>
>> Rats, I've always liked Gentoo. ;)
>
> I'd agree with Tom on that: we need a system that remains the same over
> longer periods, not simply a very fast one. I'm OK with Fedora.

fedora is probably not a prime example for "stays same over long period"
(which I think is important) since it has pretty short release cycles.
Maybe something like ubuntu LTS, Debian Etch or even CentOS would be
more appropriate (we have debian on a number of very similiar HP boxes
and HP is doing Debian Support now too).

Stefan

Greg Smith wrote:
> On Wed, 25 Jul 2007, Gavin M. Roy wrote:
>
>> Ubuntu server? Slackware? Not a fan of Centos, RHEL or Fedora...
>
> Unless you did a custom intall, using Ubuntu server would expose the
> people using your server to the quirks of how the Debian packages for
> PostgreSQL differ from other Linux distributions. I'm not sure whether
> that would be a good (shine some light on that underdocumented area) or
> bad (get in people's way) thing. The way they make it easier to manage
> multiple clusters might actually be ideal for what you're trying to do,
> let people have their own cluster and stay out of each other's data
> space at least.

for a server like this I don't think anybody cares at all for the
prepackaged postgresql. People are likely to use such a box for
development/testing of new patches and development stuff. so what they
need is a proper toolchain and solid packages. Debian derived
distributions are quite good at that usually (debian etch ships with gcc
3.3,gcc 3.4 and gcc 4.1 for example) and I expect people to simply get
their accounts and do all their work in their home-directories
anyway(which sounds like the normal way to develop on unix like OSes to me).

Stefan

Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> Unless you did a custom intall, using Ubuntu server would expose the
> people using your server to the quirks of how the Debian packages for
> PostgreSQL differ from other Linux distributions.

I doubt we'd be doing much work with the distro-installed version of
Postgres anyway, so this doesn't seem like a big concern. In fact,
to avoid confusion it might be best if the machine has no
distro-installed Postgres at all. That would help avoid "oops, that
test was run against the wrong server" syndrome.

I do essentially all my development work with installations that are
--prefix'd to user directories and started/stopped by hand; it's just
a lot easier to manage a pile of different versions that way. Plus
I never need to become root. Not sure how other developers work,
though.

regards, tom lane

Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> On Wed, 25 Jul 2007, Tom Lane wrote:
>> Gentoo always leaves me wondering exactly what I'm running today,
>> and I think reproducibility is an important attribute for a benchmarking
>> machine.

> At this point, there's enough performance variations even between
> individual Linux kernel releases that I'm not sure how much
> reproducibility you're ever going to get here. Are the differences
> between Gentoo and RHEL any bigger than those, say, between RHEL and SuSE?

The problem I've got with Gentoo is that it encourages homegrown builds
with randomly-chosen options and compiler switches. That may help
squeeze out a bit more speed but it does nothing for stability, nor
reproduceability of results on other platforms which is what we really
care about here.

Another fairly big issue is that we need to know whether measurements we
take in August are comparable to measurements we take in October, so a
fairly stable platform is important. As you say, a fast-changing kernel
would make it difficult to have any confidence about comparability over
time. That would tend to make me vote for RHEL/Centos, where long-term
stability is an explicit development goal. Debian stable might do too,
though I'm not as clear about their update criteria as I am about Red Hat's.

> The idea of setting this up with a long-term stable distribution runs
> counter to one of the things that I think is important to explore here,
> which is testing how more recent Linux kernels have improved their
> scalability.

Dunno if Gavin wants to manage multiple systems, but for most of what
I'd like to do a bleeding-edge kernel is exactly what I do not want.

regards, tom lane

Tom Lane wrote:
> Greg Smith <gsmith(at)gregsmith(dot)com> writes:
>> On Wed, 25 Jul 2007, Tom Lane wrote:
>>> Gentoo always leaves me wondering exactly what I'm running today,
>>> and I think reproducibility is an important attribute for a benchmarking
>>> machine.
>
>> At this point, there's enough performance variations even between
>> individual Linux kernel releases that I'm not sure how much
>> reproducibility you're ever going to get here. Are the differences
>> between Gentoo and RHEL any bigger than those, say, between RHEL and SuSE?
>
> The problem I've got with Gentoo is that it encourages homegrown builds
> with randomly-chosen options and compiler switches. That may help
> squeeze out a bit more speed but it does nothing for stability, nor
> reproduceability of results on other platforms which is what we really
> care about here.
>
> Another fairly big issue is that we need to know whether measurements we
> take in August are comparable to measurements we take in October, so a
> fairly stable platform is important. As you say, a fast-changing kernel
> would make it difficult to have any confidence about comparability over
> time. That would tend to make me vote for RHEL/Centos, where long-term
> stability is an explicit development goal. Debian stable might do too,
> though I'm not as clear about their update criteria as I am about Red Hat's.

Fully agreed (on the RH/CentOS and longterm stability stuff) debian is
even more stricter/conservatve than RH usually - they only have security
bugs and on very rare occation bugfixes for major issues(RH sometimes
adds new features and stuff in their point-releases).
Debian etch seems to be (very) slightly relaxing that - and in fact a
number of people were very surprised to see PostgreSQL updated from
8.1.8 (as shipped in etch) to 8.1.9 with the latest security release :-)
I would agree however that gentoo and also slackware are not "that"
attractive for this kind of work.

Stefan

On Wed, 2007-07-25 at 19:35 +0200, Stefan Kaltenbrunner wrote:
> >> Rats, I've always liked Gentoo. ;)
> >
> > I'd agree with Tom on that: we need a system that remains the same over
> > longer periods, not simply a very fast one. I'm OK with Fedora.
>
> fedora is probably not a prime example for "stays same over long period"
> (which I think is important) since it has pretty short release cycles.
> Maybe something like ubuntu LTS, Debian Etch or even CentOS would be
> more appropriate (we have debian on a number of very similiar HP boxes
> and HP is doing Debian Support now too).

OK... Gavin please arbitrate: tis your box. I'm a DB tech, dont really
care about OS.

--
Simon Riggs
EnterpriseDB http://www.enterprisedb.com

On Wed, 25 Jul 2007, Tom Lane wrote:

> The problem I've got with Gentoo is that it encourages homegrown builds
> with randomly-chosen options and compiler switches.

It encourages it, but it certainly doesn't require it. Knowing that this
is a NOC machine, I don't think there's going to be a lot of fiddling with
custom builds.

> That would tend to make me vote for RHEL/Centos, where long-term
> stability is an explicit development goal. Debian stable might do too,
> though I'm not as clear about their update criteria as I am about Red
> Hat's.

RHEL is certainly on the stable at the expense of slow to change side of
things, and Debian stable is even slower. However, at this very moment,
there have been very recent refreshes from just about everybody such that
the options available are very similar. Here's the current state of
things:

RHEL 5.0: March 2007, kernel 2.6.18, glibc 2.5
Debian Stable 4.0: April 2007, kernel 2.6.18, glibc 2.3.6
Ubuntu 7.0.4: April 2007, kernel 2.6.20, glibc 2.5
Gentoo 2007.0: May 2007, kernel 2.6.19, glibc 2.5

(http://distrowatch.com is the best site to drill through details like
this if anyone else wants to dig further/double-check me here)

I would hate to see this system installed with any kernel <2.6.18 or with
glibc<2.5 because that's clearly where the line of current generation
releases starts. I'd consider Debian Stable a poor choice accordingly.
I don't think you're going to see a lot of difference right now between
RHEL 5/Gentoo 2007.0/Ubuntu 7.0.4; all the major packages and kernels are
really similar. A year from now, there will be much more divergance were
a fresh install done with current versions of each at that point, but
there's nothing that says the system has to be upgraded then.

The think the main argument for either Gentoo or Ubuntu over RHEL/Centos
comes down to ease of installing additional packages to support building
the kinds of random software that you end up needing on a development
system. Not the core code, but the add-on packages needed to run the
various benchmark/monitoring packages people may want. To pick a random
example, the last time I was using an older SuSE system it was a pain to
get DBT2 running on it, and I ended up having to build the documentation
on another system altogether because it was easier than sorting out a
weird RPM issue I ran into.

Pulling packages from the Ubuntu universe with apt-get is usually trivial
and the available package base is very broad. Running emerge to get new
things into Gentoo is normally straightforward. RPM-based installs on
RHEL are still sometimes tricky, and my take on the breadth of the
official repositories is that they're not as wide.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

"Greg Smith" <gsmith(at)gregsmith(dot)com> writes:

> On Wed, 25 Jul 2007, Tom Lane wrote:
>
>> The problem I've got with Gentoo is that it encourages homegrown builds
>> with randomly-chosen options and compiler switches.
>
> It encourages it, but it certainly doesn't require it. Knowing that this is a
> NOC machine, I don't think there's going to be a lot of fiddling with custom
> builds.

Does gentoo these days have binary packages? source packages do implicitly
require custom builds because even if you don't fiddle with compiler switches
or other options you end up with a different build than someone who had a
different set of libraries installed when they installed it.

>> That would tend to make me vote for RHEL/Centos, where long-term stability is
>> an explicit development goal. Debian stable might do too, though I'm not as
>> clear about their update criteria as I am about Red Hat's.

Personally I'm a huge fan of Debian but even with that I think for this
situation I would actually agree that Redhat is a better fit in that it's
"canonical". You can tell someone else install Redhat vFoo and know they'll
have precisely the same set of packages with the same set of services running.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

Tom Lane wrote:
> I do essentially all my development work with installations that are
> --prefix'd to user directories and started/stopped by hand; it's just
> a lot easier to manage a pile of different versions that way. Plus
> I never need to become root. Not sure how other developers work,
> though.
>
>
>

That's exactly how I work - I have a set of source trees and a script
that invokes configure with port and prefix arguments to make sure they
don't collide.

Like you I do almost all my work on some edition of Fedora - not always
the latest by any means (e.g. currently it's FC6).

My vote would be for RHEL5/CentOS5 (they are basically the same thing -
CentOS is RHEL with the RH badging removed, for the most part, and you
don't need a RHN subscription). I think that would be a good combination
of stability and currency.

cheers

andrew

On Wed, 25 Jul 2007, Gregory Stark wrote:

> Does gentoo these days have binary packages? source packages do implicitly
> require custom builds...

You can install with binaries now so it doesn't take forever to get
started, but the minute you're adding/updating you're going to be
building. The main point I was trying to make is that if you don't do
anything special to customize the standard Gentoo compilation setup, the
amount of variation between Gentoo builds on different machines isn't
significantly greater than that which exists between the various Linux
distributions. One could make a case that the big glibc differences
between Debian Stable and everybody else right now provides a similar
scale of variation in results that would impact reproducibility.

> for this situation I would actually agree that Redhat is a better fit in
> that it's "canonical".

I threw out some criticism suggesting where RedHat is at a slight
disadvantage for completeness sake, and so Gavin wasn't completely alone
at expressing some distaste for the issues it introduces compared to
Gentoo (potentially harder package installation and less flexiblity for
running bleeding-edge kernels with RHEL). His preference for Gentoo is
completely defensible if you understand his priorities, and I'd hate to
see a knee-jerk reaction against that distribution based just on how
Gentoo can be abused and how it differs from other Linux variants.

But I run RHEL&Centos on several machines so I certainly wouldn't go so
far as to argue against it being appropriate here. The nice thing about
RedHat and its clones is that even when you run into a situation where
packages might be harder to install than you'd like them to be, the
userbase is so big and skilled that the problems are usually visible (odds
are good other people are running into the issue as well), reproducible on
other builds, and you can get plenty of help resolving them.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

"Greg Smith" <gsmith(at)gregsmith(dot)com> writes:

> On Wed, 25 Jul 2007, Gregory Stark wrote:
>
>> Does gentoo these days have binary packages? source packages do implicitly
>> require custom builds...
>
> You can install with binaries now so it doesn't take forever to get started,
> but the minute you're adding/updating you're going to be building. The main
> point I was trying to make is that if you don't do anything special to
> customize the standard Gentoo compilation setup, the amount of variation
> between Gentoo builds on different machines isn't significantly greater than
> that which exists between the various Linux distributions. One could make a
> case that the big glibc differences between Debian Stable and everybody else
> right now provides a similar scale of variation in results that would impact
> reproducibility.

Well even so another Debian system with the same set of packages (at the same
version) will be equivalent to mine.

Whereas gentoo system will depend on the order that the packages were
installed. If you installed kerberos while you had an older version of the
copiler or crypto libraries installed and then upgraded the crypto library or
compiler then your kerberos library will differ from mine which was compiled
by a different compiler or against a different set of crypto headers.

So for me to reproduce your environment you would have to send me the complete
history of what packages you installed. I would have to reproduce the entire
history including installing and building intermediate versions.

> I threw out some criticism suggesting where RedHat is at a slight disadvantage
> for completeness sake, and so Gavin wasn't completely alone at expressing some
> distaste for the issues it introduces compared to Gentoo (potentially harder
> package installation and less flexiblity for running bleeding-edge kernels with
> RHEL).

Sure, that's why I run Debian and get really annoyed whenever I use a Redhat
system. One Redhat I'm forever saying "where's this utility" or "why is this
program 6 months out of date?". But that's a personal desktop machine. This is
shared resource that shouldn't be constantly changing or having new versions
of stuff installed

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

On Thu, 26 Jul 2007, Gregory Stark wrote:

> So for me to reproduce your [Gentoo] environment you would have to send
> me the complete history of what packages you installed. I would have to
> reproduce the entire history including installing and building
> intermediate versions.

If one's goal is to be able to make several copies of a server run
completely identical builds of all software down to the build order level,
then Gentoo obviously makes that more difficult than other distributions.
It's easier if you build each replicant at the same time and then keep
them synchronized, but cloning a machine that's already out there and has
been through a series of updates that perfectly is as challenging as you
describe. If the primary goal here was reproducable benchmarks where you
needed SPEC-submission level version control, Gentoo would be a completely
inappropriate choice.

But this is pushing forward PostgreSQL development you're doing here. If
you've got a problem such that something works differently based on the
order in which you built the packages, which is going to be unique to
every Linux distribution already, that is itself noteworthy and deserves
engineering out. You might think of this high-end machine being a little
different as usefully adding diversity robustness in a similar way to how
the buildfarm helps improve the core right now.

I think I have to exit this discussion before I start sounding like a
Gentoo fanboi and make my Linux consulting clients nervous. Go RedHat!

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> But this is pushing forward PostgreSQL development you're doing here. If
> you've got a problem such that something works differently based on the
> order in which you built the packages, which is going to be unique to
> every Linux distribution already, that is itself noteworthy and deserves
> engineering out. You might think of this high-end machine being a little
> different as usefully adding diversity robustness in a similar way to how
> the buildfarm helps improve the core right now.

Actually, the thing that's concerning me is *exactly* lack of diversity.
If we have just one of these things then there's a significant risk of
unconsciously tuning PG towards that specific platform. I'd rather we
take that risk with a well-standardized, widely used platform than with
something no one else can reproduce.

Really there's a pretty good argument for having several different OS'es
available on the box --- I wonder whether Gavin is up to managing some
sort of VM or multiboot setup.

regards, tom lane

Let me look at what makes sense there, I am open to it.

On 7/26/07, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> > But this is pushing forward PostgreSQL development you're doing here. If
> > you've got a problem such that something works differently based on the
> > order in which you built the packages, which is going to be unique to
> > every Linux distribution already, that is itself noteworthy and deserves
> > engineering out. You might think of this high-end machine being a little
> > different as usefully adding diversity robustness in a similar way to how
> > the buildfarm helps improve the core right now.
>
> Actually, the thing that's concerning me is *exactly* lack of diversity.
> If we have just one of these things then there's a significant risk of
> unconsciously tuning PG towards that specific platform. I'd rather we
> take that risk with a well-standardized, widely used platform than with
> something no one else can reproduce.
>
> Really there's a pretty good argument for having several different OS'es
> available on the box --- I wonder whether Gavin is up to managing some
> sort of VM or multiboot setup.
>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>

Tom Lane wrote:
> Greg Smith <gsmith(at)gregsmith(dot)com> writes:

> Really there's a pretty good argument for having several different OS'es
> available on the box --- I wonder whether Gavin is up to managing some
> sort of VM or multiboot setup.

IMO, a multiboot is o.k. but a vm isn't worth it. This box is big enough
to actually starting looking at SMP and I/O issues for PostgreSQL that
we normally can't because we don't have access to the hardware in the
community.

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Tom Lane wrote:
>> Really there's a pretty good argument for having several different OS'es
>> available on the box --- I wonder whether Gavin is up to managing some
>> sort of VM or multiboot setup.

> IMO, a multiboot is o.k. but a vm isn't worth it.

Yeah, multiboot would be better --- otherwise you have to wonder if the
vm is affecting performance at all. But I suppose multiboot would be
harder to manage.

regards, tom lane

Tom Lane wrote:
> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
>> Tom Lane wrote:
>>> Really there's a pretty good argument for having several different OS'es
>>> available on the box --- I wonder whether Gavin is up to managing some
>>> sort of VM or multiboot setup.
>
>> IMO, a multiboot is o.k. but a vm isn't worth it.
>
> Yeah, multiboot would be better --- otherwise you have to wonder if the
> vm is affecting performance at all. But I suppose multiboot would be
> harder to manage.

Personally, I think CentOS 5 is probably the most reasonable choice. It
is what (or RHEL 5 which is the same) a good portion of our community is
going to be running. It is also easy to work with.

Another alternative would be Debian or Ubuntu Dapper but they are all
really the same thing :). The nice thing is any of these three are
fairly static installs that are going to be reasonably predictable.

Joshua D. Drake

>
> regards, tom lane
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive PostgreSQL solutions since 1997
http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

* Joshua D. Drake (jd(at)commandprompt(dot)com) wrote:
> Personally, I think CentOS 5 is probably the most reasonable choice. It is
> what (or RHEL 5 which is the same) a good portion of our community is going
> to be running. It is also easy to work with.
>
> Another alternative would be Debian or Ubuntu Dapper but they are all
> really the same thing :). The nice thing is any of these three are fairly
> static installs that are going to be reasonably predictable.

If we can generally agree on "Linux" then it might be reasonable to
consider using either VServers or just regular chroot's with different
OSes loaded (when/if we want to look at a particular OS). There'd be
little to no performance impact from such a solution while we'd still
have different OSes to play with.

Of course, the kernel would be the same for all of them, so if that's
what we're interested mostly in testing/stressing then it's no good. I
got the impression from some that various gcc builds, glibc versions,
etc, would be good to test though and a VServer or chroot setup could
work well for that.

As a Debian Developer, I have to also say that Debian would be my
choice. :) Though I've got a number of big toys to play w/ at work
already so it's unlikely I'd have need of this system (not to mention
that most of the stuff I work on in PG is usability rather than things
like large-scale performance, currently anyway).

Thanks,

Stephen

On Thu, 26 Jul 2007, Joshua D. Drake wrote:

> IMO, a multiboot is o.k. but a vm isn't worth it. This box is big enough to
> actually starting looking at SMP and I/O issues for PostgreSQL that we
> normally can't because we don't have access to the hardware in the community.

Certainly agree with that; VM overhead is much lower than it used to be,
but it's still going to fuzz exactly the kind of performance results that
this box would be most useful for exploring.

What I normally do in this situation is create a second primary partition
on the boot drive with around 10GB of space on it that doesn't get touched
by the initial OS install. Then it's straighforward to install a second
Linux into there; the only time that gets tricky is if you're doing two
RedHat style installs because of how they mount partitions by label. A
little bit of GRUB merging after the second install, and now you've got a
dual-boot system. Even in a NOC setup where you don't see the boot menu,
you'd just have to change the grub.conf default and reboot in order to
switch between the two.

As long as a bootable partition of reasonable size is set aside like this,
there's all kinds of flexibility for being able to confirm results apply
to multiple Linux distributions in the future. You might even put a BSD
or Solaris in that space one day.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Gavin,

I'm actually in the middle of assembling a general performance test lab for
the PostgreSQL hackers, using equipment donated by Sun, Hi5, and (hopefully)
Unisys and Intel. While your machine would obviously stay in Pennsylvania,
it would be cool if we could somehow arrange a unified authentication &
booking system.

I'm pretty sure I can even raise money to get one created.

How long will this system remain available to us?

--
Josh Berkus
PostgreSQL @ Sun
San Francisco

It's actually in Texas, and we have no intention to put a time limit
on its availability. I think the availability will be there as long as
there is use and we're in the Texas data center, which I don't see
ending any time soon.

On 7/31/07, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> Gavin,
>
> I'm actually in the middle of assembling a general performance test lab for
> the PostgreSQL hackers, using equipment donated by Sun, Hi5, and (hopefully)
> Unisys and Intel. While your machine would obviously stay in Pennsylvania,
> it would be cool if we could somehow arrange a unified authentication &
> booking system.
>
> I'm pretty sure I can even raise money to get one created.
>
> How long will this system remain available to us?
>
> --
> Josh Berkus
> PostgreSQL @ Sun
> San Francisco
>

Just a follow-up to note that Red Hat has graciously donated a 1 year
RHEL subscription and myYearbook is paying Command Prompt to setup the
RHEL box for community use.

We've not worked out a scheduling methodology, or how to best organize
the use of said hardware, but I know that Tom and others are
interested.

Does anyone have a scheduling solution for things like this to make
sure people aren't stepping on each others toes processor/ram/disk
wise?

Also, what should the policies be for making sure that people can use
the box for what they need to use the box for?

Should people clean up after themselves data usage wise after their
scheduled time?

Should people only be able to run PostgreSQL in the context of their
own user? Do we have experience with such setups in the past? What
has worked well and what hasn't?

Gavin

On 7/25/07, Gavin M. Roy <gavinmroy(at)gmail(dot)com> wrote:
> Recently I've been involved in or overheard discussions about SMP
> scalability at both the PA PgSQL get together and in some list
> traffic.
>
> myYearbook.com would ike to make one of our previous production
> machines available to established PgSQL Hackers who don't have access
> to this level of hardware for testing, benchmarking and development to
> work at improving SMP scalability and related projects.
>
> The machine is a HP 585 G1, 8 Core AMD, 32GB RAM with one 400GB 14
> Spindle DAS Array dedicated to community use. I've attached a text
> file with dmesg and /proc/cpuinfo output.
>
> I'm working on how this will be setup and am open to suggestions on
> how to structure access.
>
> I'm currently in the process of having Gentoo linux reinstalled on the
> box since that is what I am most comfortable administering from a
> security perspective. If this will be a blocker for developers who
> would actually work on it, please let me know.
>
> If you're interested in access, my only requirement is that you're a
> current PgSQL Hacker with a proven track-record of committing patches
> to the community. This is a resource we could be using for something
> else, and I'd like to see the community get direct benefit from it as
> opposed to it being a play sandbox for people who want to tinker.
>
> Please let me know thoughts, concerns or suggestions.
>
> Gavin M. Roy
> CTO
> myYearbook.com
> gmr(at)myyearbook(dot)com
>
>

"Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> Just a follow-up to note that Red Hat has graciously donated a 1 year
> RHEL subscription and myYearbook is paying Command Prompt to setup the
> RHEL box for community use.

Sorry that Red Hat was so slow about that :-(

> [ various interesting questions snipped ]

> Should people only be able to run PostgreSQL in the context of their
> own user? Do we have experience with such setups in the past? What
> has worked well and what hasn't?

Yeah, I'd vote for people just building private PG installations in
their own home directories. I am not aware of any performance-testing
reason why we'd want a shared installation, and given that people are
likely to be testing many different code variants, a shared installation
would be a management nightmare. Also, with personal installations,
nobody need have root privileges, which just seems like a real good idea.

I don't have any special insights about the other management issues
you mentioned, but I'm sure someone does ...

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 02 Nov 2007 15:37:17 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> "Gavin M. Roy" <gavinmroy(at)gmail(dot)com> writes:
> > Just a follow-up to note that Red Hat has graciously donated a 1
> > year RHEL subscription and myYearbook is paying Command Prompt to
> > setup the RHEL box for community use.
>
> Sorry that Red Hat was so slow about that :-(
>
> > [ various interesting questions snipped ]
>
> > Should people only be able to run PostgreSQL in the context of their
> > own user? Do we have experience with such setups in the past? What
> > has worked well and what hasn't?
>
> Yeah, I'd vote for people just building private PG installations in
> their own home directories. I am not aware of any performance-testing
> reason why we'd want a shared installation, and given that people are
> likely to be testing many different code variants, a shared

The only caveat here is that our thinking was that the actual arrays
would be able to be re-provisioned all the time. E.g; test with RAID 10
with x stripe size, Software RAID 6, what is the real difference
between 28 spindles with RAID 5 versus 10?

> installation would be a management nightmare. Also, with personal
> installations, nobody need have root privileges, which just seems
> like a real good idea.

No question.

Joshua D. Drake

>
> I don't have any special insights about the other management issues
> you mentioned, but I'm sure someone does ...
>
> regards, tom lane
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 4: Have you searched our
> list archives?
>
> http://archives.postgresql.org
>

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHK412ATb/zqfZUUQRAg4eAJ0YubwkLQ3mU0st5jPhUnC6dWrqeACeMjQe
TFxunw+efuh3XNtMv+whKBI=
=RzC/
-----END PGP SIGNATURE-----

"Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> Yeah, I'd vote for people just building private PG installations in
>> their own home directories. I am not aware of any performance-testing
>> reason why we'd want a shared installation, and given that people are
>> likely to be testing many different code variants, a shared

> The only caveat here is that our thinking was that the actual arrays
> would be able to be re-provisioned all the time. E.g; test with RAID 10
> with x stripe size, Software RAID 6, what is the real difference
> between 28 spindles with RAID 5 versus 10?

Well, we need some workspace that won't go away when that happens.
I'd suggest that the OS and people's home directories be mounted on
a "permanent" partition with plenty of space for source code, say a
few tens of GB, and then there be a farm of data workspace that's
understood to be transient and can be reconfigured as needed for tests
like that.

regards, tom lane

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Fri, 02 Nov 2007 17:11:30 -0400
Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> "Joshua D. Drake" <jd(at)commandprompt(dot)com> writes:
> > Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >> Yeah, I'd vote for people just building private PG installations in
> >> their own home directories. I am not aware of any
> >> performance-testing reason why we'd want a shared installation,
> >> and given that people are likely to be testing many different code
> >> variants, a shared
>
> > The only caveat here is that our thinking was that the actual arrays
> > would be able to be re-provisioned all the time. E.g; test with
> > RAID 10 with x stripe size, Software RAID 6, what is the real
> > difference between 28 spindles with RAID 5 versus 10?
>
> Well, we need some workspace that won't go away when that happens.

Right which is on the internal devices.

> I'd suggest that the OS and people's home directories be mounted on
> a "permanent" partition with plenty of space for source code, say a
> few tens of GB, and then there be a farm of data workspace that's
> understood to be transient and can be reconfigured as needed for tests
> like that.

Agreed.

Sincerely,

Joshua D. Drake

>
> regards, tom lane
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 1: if posting/reading
> through Usenet, please send an appropriate subscribe-nomail command
> to majordomo(at)postgresql(dot)org so that your message can get through to
> the mailing list cleanly
>

- --

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)

iD8DBQFHK5LwATb/zqfZUUQRApBQAJ9Gp+fpgOnA6ZONpdQl43giMcetZwCggv2Q
8A9FfkeP6VsQptWl1J8W4n8=
=nX1C
-----END PGP SIGNATURE-----