Re: Performance Farm Release

[resending after noticing that "reply all" resulted in sending to
pgsql-hackers-owner rather than pgsql-hackers]

"Luxenberg, Scott I." <Scott(dot)Luxenberg(at)noblis(dot)org> wrote:

> This is just my email to notify you all that the project I've been
> working on with Stephen, the PostgreSQL Performance Farm, has been
> released. As of now, it only supports 9.0, due to the use of
> workers. More details can be found in the readme. The Git
> repository is located here:
> http://github.com/slux/Postgre-Performance-Farm

If this is covered in a file that I missed, or on a Wiki somewhere,
please point me in the right direction and accept my apologies. I
looked at the README and didn't find enough to really answer my
questions.

Would this project be useful for someone trying to assess the
performance impact of a proposed patch (at least on the developer's
machine)? What would someone do to use it in this way?

Thanks for any info.

-Kevin

Kevin,

* Kevin Grittner (Kevin(dot)Grittner(at)wicourts(dot)gov) wrote:
> Would this project be useful for someone trying to assess the
> performance impact of a proposed patch (at least on the developer's
> machine)? What would someone do to use it in this way?

The goal is to have this running in a similar manner as the build farm
to identify when a patch has an impact on performance (good or bad).
Hackers would then be able to view performance farm reports similar to
viewing build farm reports. Not sure if we'd have alerts or something,
but I'd think in alot of cases a given hacker would know that they're
commiting something performance-impacting (or saw someone else commit
something that could be) and they'd go check out the performance farm
reports a few days later to determine if there was a change.

Thanks,

Stephen

>Stephen Frost <sfrost(at)snowman(dot)net> wrote:

> The goal is to have this running in a similar manner as the build
> farm to identify when a patch has an impact on performance (good
> or bad). Hackers would then be able to view performance farm
> reports similar to viewing build farm reports. Not sure if we'd
> have alerts or something, but I'd think in alot of cases a given
> hacker would know that they're commiting something performance-
> impacting (or saw someone else commit something that could be) and
> they'd go check out the performance farm reports a few days later
> to determine if there was a change.

I actually understood that part, but was already wondering if it
could be bent to slightly different purposes. It seems as though
there would be value to using it to evaluate the performance impact
of a proposed patch, at least on a limited basis, *before* a commit.
If there's not an immediately obvious way to put it to that
alternative use, that's OK; I just thought I'd ask.

-Kevin

* Kevin Grittner (Kevin(dot)Grittner(at)wicourts(dot)gov) wrote:
> I actually understood that part, but was already wondering if it
> could be bent to slightly different purposes. It seems as though
> there would be value to using it to evaluate the performance impact
> of a proposed patch, at least on a limited basis, *before* a commit.
> If there's not an immediately obvious way to put it to that
> alternative use, that's OK; I just thought I'd ask.

You can certainly run it yourself locally w/o setting it up to report
back to the build or performance farm.. So, yes, you can, you'll just
have to look through the outputs yourself and it won't necessairly make
much sense unless you've been doing those runs for a period of time to
get a feel for how volatile the speed is on your system..

I guess one issue would be figuring out how to inject your patch into
the process.. If you have it committed to a git instance somewhere, you
could tell it to pull from that after running on the main PG w/o the
patch..

Stephen

Kevin Grittner wrote:
> I actually understood that part, but was already wondering if it
> could be bent to slightly different purposes. It seems as though
> there would be value to using it to evaluate the performance impact
> of a proposed patch, at least on a limited basis, *before* a commit

Eventual design here presumed that this would reach the point where one
of the sources you might want to pull from to test performance of was a
different git branch than the main one. That's what I was planning to
do with it.

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.us

Stephen Frost wrote:
> You can certainly run it yourself locally w/o setting it up to report
> back to the build or performance farm.. So, yes, you can, you'll just
> have to look through the outputs yourself and it won't necessairly make
> much sense unless you've been doing those runs for a period of time to
> get a feel for how volatile the speed is on your system..
>

Thanks again to Scott for working on this all summer, and to Stephen and
Noblis for helping fund it. There were a lot of small pieces that
needed to assemble just right to make this all fit together in a way I
hope will make it integrate into the core infrastructure soon.

It's probably not obvious to everyone else where this stands as far as
reporting results goes though, so let me expand on what Stephen said
here. When you run performance tests with this, those are stored
locally. Scott demonstrated that you can get basic reports out of that,
and I'm content the right initial tests are being run and the most
useful data to record is being saved. But none of the performance
numbers are sent anywhere yet--they're just stored as CSV files. From
the README: "WHENEVER YOU RUN A PERFORMANCE TEST, RUN WITH --test, so
not to spam the buildfarm server". That's the bad news that warning is
conveying: you can't upload results yet.

I expect the next steps here to look like this:

1) Nail down what subset of the information gathered locally should be
uploaded to the buildfarm master server. Probably just the same columns
of data already being saved for each test, but perhaps with some extra
metadata. The local animal will also have graphs and such, but
unrealistic to upload all of those for a number of reasons I think.
Only really useful to drill down when there is some sort of regression I
expect, and hopefully the animal is still alive when that happens.

2) Update the buildfarm server code to accept and store that data.

3) Update this perffarm client to talk to that.

4) Merge the perfarm fork changes into the mainline buildfarm code. I
expect continued bitrot of this code as changes are made to the regular
buildfarm client, so it might be worth considering that sooner rather
than later.

My understanding is that the code for the server side of the buildfarm
isn't public to everyone right now, just because of time limitations
getting it cleaned up for that. So a couple of parts here are being
funneled through how much spare time Andrew has, and there are more
important git and buildfarm related things for him to worry about right now.

Presuming nothing exciting on this happens before then, I'm hoping that
I can catch up with Andrew at PG West and map out how to get the rest of
this done, so it goes live somewhere during 9.1 development. Now that
the code has been released from the Noblis fortress, I can start
cleaning up some of the little details on it before then too (i.e. not
working for anything but 9.0 yet).

--
Greg Smith 2ndQuadrant US Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.us

On Tue, 2010-08-31 at 03:28 -0400, Greg Smith wrote:

> 4) Merge the perfarm fork changes into the mainline buildfarm code. I
> expect continued bitrot of this code as changes are made to the regular
> buildfarm client, so it might be worth considering that sooner rather
> than later.

As Andrew is currently on vacation, *now* is the time to do that.

JD

--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579
Consulting, Training, Support, Custom Development, Engineering
http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt

On 08/31/2010 03:28 AM, Greg Smith wrote:
>
> 1) Nail down what subset of the information gathered locally should be
> uploaded to the buildfarm master server. Probably just the same
> columns of data already being saved for each test, but perhaps with
> some extra metadata. The local animal will also have graphs and such,
> but unrealistic to upload all of those for a number of reasons I
> think. Only really useful to drill down when there is some sort of
> regression I expect, and hopefully the animal is still alive when that
> happens.
>
> 2) Update the buildfarm server code to accept and store that data.

The server code is in fact very generic. There is no hard coding of step
names or anything like that. The major piece I think we'd need to add in
here is somewhere to store performance test scores, as opposed to the
success/failure of normal buildfarm steps. That's not going to be
terrible difficult.

>
> 3) Update this perffarm client to talk to that.
>
> 4) Merge the perfarm fork changes into the mainline buildfarm code. I
> expect continued bitrot of this code as changes are made to the
> regular buildfarm client, so it might be worth considering that sooner
> rather than later.

OK, but the client code is hardly in a state of violent flux. In the
last year there have been eight commits, including the git changes.

>
> My understanding is that the code for the server side of the buildfarm
> isn't public to everyone right now, just because of time limitations
> getting it cleaned up for that. So a couple of parts here are being
> funneled through how much spare time Andrew has, and there are more
> important git and buildfarm related things for him to worry about
> right now.

The git buildfarm changes are pretty much bedded down now, I think.

I have broken the back of extensible enums a bit more quickly than I
expected, so I might have some flying time in the next week or so to
devote to cleaning stuff up in the server code, while taking a break
from COPY as a FROM target (which will undoubtedly turn my brain to mush
if I work on it for more than a couple of hours at a time).

>
> Presuming nothing exciting on this happens before then, I'm hoping
> that I can catch up with Andrew at PG West and map out how to get the
> rest of this done, so it goes live somewhere during 9.1 development.
> Now that the code has been released from the Noblis fortress, I can
> start cleaning up some of the little details on it before then too
> (i.e. not working for anything but 9.0 yet).
>

Sounds good.

cheers

andrew

I just dusted off this code and brought it back to current again.
Basically a lot of reformatting the new performance farm parts to
minimize their diff. Once that was done, all of the other buildfarm
client updates since then applied cleanly.

The result is now sitting as a fork of Andrew's client code repo at
https://github.com/greg2ndQuadrant/client-code , replacing the repo
Scott published at https://github.com/slux/PostgreSQL-Performance-Farm ;
much easier to avoid future bit-rot with this structure.

The main changes made here are now pretty easy to read on github:
https://github.com/greg2ndQuadrant/client-code/commit/d0339a59ceb4711a6b042d3f1d053c77f07720f4
and I've attached the code as a patch here. It also uses some scripts
from pgbench-tools that aren't the interesting part.

I got a nibble from Endpoint at PGEast about interest in hacking the
buildfarm server code to add support for a new test type, so this may
start moving forward again if that works out.

--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us

On Mar 27, 2011, at 3:20 PM, Greg Smith <greg(at)2ndQuadrant(dot)com> wrote:
> I just dusted off this code and brought it back to current again. Basically a lot of reformatting the new performance farm parts to minimize their diff. Once that was done, all of the other buildfarm client updates since then applied cleanly.
>
> The result is now sitting as a fork of Andrew's client code repo at https://github.com/greg2ndQuadrant/client-code , replacing the repo Scott published at https://github.com/slux/PostgreSQL-Performance-Farm ; much easier to avoid future bit-rot with this structure.
>
> The main changes made here are now pretty easy to read on github: https://github.com/greg2ndQuadrant/client-code/commit/d0339a59ceb4711a6b042d3f1d053c77f07720f4 and I've attached the code as a patch here. It also uses some scripts from pgbench-tools that aren't the interesting part.
>
> I got a nibble from Endpoint at PGEast about interest in hacking the buildfarm server code to add support for a new test type, so this may start moving forward again if that works out.

I don't have anything constructive to add to this conversation, but I think it's awesome that you're working on this!

...Robert