64-bit size pgbench

From: Greg Smith <greg(at)2ndquadrant(dot)com>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: 64-bit size pgbench
Date: 2010-01-28 23:20:03
Message-ID: 4B621BA3.7090306@2ndquadrant.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Attached is a patch that fixes a long standing bug in pgbench: it won't
handle scale factors above ~4000 (around 60GB) because it uses 32-bit
integers for its computations related to the number of accounts, and it
just crashes badly when you exceed that. This month I've run into two
systems where that was barely enough to exceed physical RAM, so I'd
expect this to be a significant limiting factor during 9.0's lifetime.
A few people have complained about it already in 8.4.

The index size on the big accounts table has to increase for this to
work, it's a bigint now instead of an int. That's going to mean a drop
in results for some tests, just because less index will fit in RAM.
I'll quantify that better before submitting something final here. I
still have some other testing left to do as well: making sure I didn't
break the new \setshell feature (am suspicious of strtol()), confirming
the random numbers are still as random as they should be (there was a
little bug in the debugging code related to that, too).

Was looking for general feedback on whether the way I've converted this
to use 64 bit integers for the account numbers seems appropriate, and to
see if there's any objection to fixing this in general given the
potential downsides.

Here's the patch in action on previously unreachable sizes (this is a
system with 8GB of RAM, so I'm basically just testing seek speed here):

$ ./pgbench -j 4 -c 8 -T 30 -S pgbench
starting vacuum...end.
transaction type: SELECT only
scaling factor: 5000
query mode: simple
number of clients: 8
number of threads: 4
duration: 30 s
number of transactions actually processed: 2466
tps = 82.010509 (including connections establishing)
tps = 82.042946 (excluding connections establishing)

$ psql -x -c "select relname,reltuples from pg_class where
relname='pgbench_accounts'" -d pgbench
relname | pgbench_accounts
reltuples | 5e+08

$ psql -x -c "select pg_size_pretty(pg_table_size('pgbench_accounts'))"
-d pgbench
pg_size_pretty | 63 GB

$ psql -x -c "select aid from pgbench_accounts order by aid limit 1" -d
pgbench
aid | 1

$ psql -x -c "select aid from pgbench_accounts order by aid desc limit
1" -d pgbench
aid | 500000000

--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg(at)2ndQuadrant(dot)com www.2ndQuadrant.com

Attachment Content-Type Size
pgbench-64-v1.patch text/x-patch 6.8 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Peter Eisentraut 2010-01-28 23:22:03 Re: Review: Typed Table
Previous Message Peter Eisentraut 2010-01-28 23:09:05 Re: make everything target