Re: Test lab

On Wed, 7 Nov 2007, Hannu Krosing wrote:

> To be really useful, we should always run general system monitoring
> alongside DB test runs, so we can see, and also later look up, where the
> bottleneck are.

The way the DBT-2 tests run involves spawning off the relevant monitoring
tools (iostat, vmstat, etc.) so that they write to a set of files. When
the test is over those process are killed and a Perl script sorts through
everything, drawing graphs and such using tools like gnuplot.

That particular model, where the benchmark drives the data collection,
makes it very easy to create graphs on a consistant time scale with
application-specific results (like transactions per second). But it also
requires that every application that wants to monitor in this area have
its own code.

There's certainly some value to something that instead monitors all the
time in the background, and then individual applications can just ask for
the period of time they're interested in rather than having their own
monitoring code. The main issue I've run into is that when you're
actually running a benchmark, the level of monitoring you want can be
smaller than what you may want to leave running all the time. For
example, I run iostat at 1 second intervals for some tests, because if you
average on a longer basis you miss how big the fsync spike is when
checkpoints happen. But it may not make sense to always have the system
monitoring at 1 second resolution.

> I guess we (Skype DB team) could help to set something up on test lab
> machines as we have been doing it on production machines for a few
> years.

I'd be curious to find out more about what you're doing. I've been
fighting this particular problem on my own mini-lab for a while now, and
it's pretty obvious to me that there's value to producing a more general
solution to how to handle this sort of monitoring.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD