Re: Introducing the TPC-V benchmark, and its relationship to PostgreSQL

Thanks for reply, Craig. As far as publishing a draft, we are planning to do something along those lines.

For the schema and the queries, we are pretty much taking those wholesale from TPC-E, whose specification is public (http://www.tpc.org/tpce/spec/v1.12.0/TPCE-v1.12.0.pdf). The high-level differences with TPC-E are detailed in the 2010 and 2012 TPC TC papers I mentioned. We will stick closely to the TPC-E schema and queries. Anything new means a long specification writing process, which we are trying to avoid. We want to get this benchmark out there quickly.

I am not an expert in licensing. What I can tell you is that the kit will be available to anyone to download and use with a simple EULA based on existing TPC EULAs (although TPC hasn't had a complete end-to-end kit before, it has published partial code modules for its benchmarks). We broached the idea of open sourcing the kit, but it didn't pan out. The people on the subcommittee represent their companies, and different companies have different rules when their employees contribute to open source code. Satisfying the armies of lawyers would have been impossible. So the kit won't be open source, but readily available for use. It will probably be similar to the licensing for SPEC benchmarks if you are familiar with them.

I'll pick up Greg's book. We had been focusing on functionality, but our focus will shift to performance soon. To be blunt, the team is very experienced in benchmarks and in database performance, but most of us are new to PGSQL.

Thanks,
Reza

> -----Original Message-----
> From: Craig Ringer [mailto:ringerc(at)ringerc(dot)id(dot)au]
> Sent: Tuesday, July 03, 2012 10:19 PM
> To: pgsql-performance(at)postgresql(dot)org
> Cc: Reza Taheri; Andy Bond (abond(at)redhat(dot)com); Greg Kopczynski; Jignesh
> Shah; Greg Smith; Dave Page
> Subject: Re: [PERFORM] Introducing the TPC-V benchmark, and its
> relationship to PostgreSQL
>
> On 07/04/2012 07:08 AM, Reza Taheri wrote:
>
> > ... so the subcommittee moved forward with developing its own
> > reference kit. The reference kit has been developed to run on
> > PostgreSQL, and we are focusing our development efforts and testing on
> > PostgreSQL.
> That's a very positive step. The TPC seems to me to have a pretty poor
> reputation among open source database users and vendors. I think that's
> largely because the schema and tools are typically very closed and
> restrictively licensed, though the prohibition against publishing benchmarks
> by big commercial vendors doesn't help.
>
> This sounds like a promising change. The TPC benchmarks are really good for
> load-testing and regression testing, so having one that's directly PostgreSQL
> friendly will be a big plus, especially if it is appropriately licensed.
>
> The opportunity to audit the schema, queries, and test setup before the
> tool is finalized would certainly be appealing. What can you publish in draft
> form now?
>
> What license terms does the TPC plan to release the schema, queries, and
> data for TPC-V under?
>
> I've cc'd Greg Smith and Dave Page, both of whom I suspect will be
> interested in this development but could easily miss your message. If you
> haven't read Greg' book "PostgreSQL High Performance" it's probably a good
> idea to do so.
>
> --
> Craig Ringer