Re: MonetDB test says that PostgreSQL often has errors or missing results

See:
http://monetdb.cwi.nl/SQL/Benchmark/TPCH/
If the result is correct, then the problem queries should be added to
the regression test suite.
If the result is not correct, then perhaps they could get assistance on
proper configuration of PostgreSQL and rerun the tests.

Dann Corbit wrote:
> See:
> http://monetdb.cwi.nl/SQL/Benchmark/TPCH/
> If the result is correct, then the problem queries should be added to
> the regression test suite.
> If the result is not correct, then perhaps they could get assistance on
> proper configuration of PostgreSQL and rerun the tests.

this is a known issues and has been discussed in the past - the problem
here is that postgresql misparsed the interval syntax in older version
(not that they used 8.2 when 8.3 was long released in that test).
This has since been fixed in 8.4+ however.

Stefan

Dann Corbit wrote:
> See:
> http://monetdb.cwi.nl/SQL/Benchmark/TPCH/
> If the result is correct, then the problem queries should be added to
> the regression test suite.
> If the result is not correct, then perhaps they could get assistance on
> proper configuration of PostgreSQL and rerun the tests.
>

We've already gotten a couple of reports that talks about the specific
reasons why PostgreSQL 8.2 doesn't do well on this test, such as
http://www.mail-archive.com/pgsql-general(at)postgresql(dot)org/msg120882.html
(broken queries) and
http://archives.postgresql.org/pgsql-performance/2008-09/msg00157.php
(why the default setting are completely ludicrous for this test).

One of these days I'm going to re-run the whole suite with 8.4 and see
how we're doing. Seriously doubt any queries are still flat out broken,
given how many reports I've seen of successful (albeit slow sometimes)
runs of the whole thing. That's just MonetDB being negligent in
reporting things to favor their own agenda. It's completely permissible
to customize queries that won't execute at all in order to fix their
compatibility with a particular database, the fact that they didn't look
into that or even say what was wrong is just shoddy journalism. I'm
hoping to have some crow to serve to them about another benchmark result
they've published soon, they're back to really rubbing me the wrong way
again lately.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

All,

Actually, the report which MonetDB has published I believe is illegal.
If they're not running it through the TPC, they can't claim it's a
"TPCH" result.

--Josh Berkus

Josh Berkus wrote:
> Actually, the report which MonetDB has published I believe is illegal.
> If they're not running it through the TPC, they can't claim it's a
> "TPCH" result.
>

I just resisted getting into that but now you've set me off again.
Presumably they're using the public TPC-H data and query generator
distributed by the TPC, and there's certainly plenty of other unofficial
reports of results using that floating around. Where I think they
really crossed the line here is using that kit to produce unaudited
results, and then publishing results that included comparisons against a
competitor, which is clearly not what the TPC intends you to do here.

I'd rather refute their results than cry to the daddy TPC about it
though. Not that I'd be upset if someone else, perhaps someone who
already has contacts within the TPC, did so. (looking up into the air
and whistling)

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

On Tue, Jan 19, 2010 at 10:04 PM, Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> Josh Berkus wrote:
>>
>> Actually, the report which MonetDB has published I believe is illegal.
>> If they're not running it through the TPC, they can't claim it's a
>> "TPCH" result.
>>
>
> I just resisted getting into that but now you've set me off again.
>  Presumably they're using the public TPC-H data and query generator
> distributed by the TPC, and there's certainly plenty of other unofficial
> reports of results using that floating around.  Where I think they really
> crossed the line here is using that kit to produce unaudited results, and
> then publishing results that included comparisons against a competitor,
> which is clearly not what the TPC intends you to do here.

What the TPC provides isn't really a usable kit. It could be
entertaining to see how their kit works.

Regards,
Mark

Mark Wong wrote:
> What the TPC provides isn't really a usable kit. It could be
> entertaining to see how their kit works.
>

The one for TPC-H seems to work for a lot of people; the best of the
intros I found for how to make it go was
http://bhairav.serc.iisc.ernet.in/doc/Installation/tpch.htm , there are
others. I'd guess MonetDB followed a procedure just like that,
discovered some queries didn't work right, and just called it a day and
published rather than investigate.

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com