Re: Google SoC--Idea Request

Jonah H. Harris wrote:
> On 4/18/06, Jim C. Nasby <jnasby(at)pervasive(dot)com> wrote:
>> On Tue, Apr 18, 2006 at 11:27:40AM -0700, Mark Wong wrote:
>>> Jim C. Nasby wrote:
>>>> On Sat, Apr 15, 2006 at 03:05:20PM -0400, Jonah H. Harris wrote:
>>>>> All ideas welcome!
>>>> I know it's not directly PostgreSQL related, but I'd love to see the
>>>> dbt* code improved. Items on my wish-list:
>>>>
>>>> - make it easy to run the test framework and clients on a seperate
>>>> machine from the database server
>>>> - keep results in a database
>>>> - provide a front-end to allow users to schedule tests in a queue
>>>> - add support for windows, at least for the database (theoretically
>>>> possible to run that way now, but you have to do everything by hand)
>>>>
>>>> Another idea: afaik, spikesource is still offering a bounty for
>>>> improvements to OSS test suites, something that'd fit well with SoC.
>>> I second this. :) There are also the TPC-App (Java) fair-use
>>> implementation that I've started and the TPC-E (next gen OLTP) that I
>>> would like to start.
>> Maybe before starting on TPC-E it makes sense to try and get a common
>> framework for all the different tests built? AFAIK most of the
>> benchmarks all use a fairly standard client-server infrastructure, so we
>> should hopefully be able to share that between the different tests...
>
> I agree with Jim. A framework would really help out here. All of the
> tests are basically the same and would benefit from a framework.

This has crossed my mind before. I haven't been able to come up with
something that I've felt good about on my own though.

> However, Mark, do you think Java is a reliable benchmarking platform?
> At EnterpriseDB, we've tried several Java benchmarks and could never
> get as repeatable or reliable of a benchmark as DBT2 gives you.

I don't have much experience here yet. I've only got a portion of the
TPC-App implemented, although probably enough now to see how repeatable
it is thus far. Do you want to give my DBT4 kit a shot? :) I'm curious
to what platforms you've tried Java on as I've heard the Linux
implementations aren't as good as their Windows counterparts. I'm not
sure how true that is today though.

Mark