Dealing with big tables

From: "Mindaugas" <ml(at)kilimas(dot)com>
To: pgsql-performance(at)postgresql(dot)org
Subject: Dealing with big tables
Date: 2007-12-02 10:26:17
Message-ID: E1Iym1h-0004Tg-VQ@garm.runbox.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-performance

Hello,

Started to work with big tables (like 300GB) and performance problems started to appear. :(

To simplify things - table has an index on From an index on To columns. And it also have several other not indexed columns. There are 100000+ of different values for From and the same for To.

I execute simple query "select * from bigtable where From='something'". Query returns like 1000 rows and takes 5++ seconds to complete. As far as I understand the query is slow because:
- first it has to retrieve pointers to rows with data from index. That goes fast.
- retrieve all the rows one by one. There we have 100% random read because rows with the same From is distributed evenly through all the 300GB and most probably nothing is cached. So there we are limited by _one_ disk performance independently of how many disks we have in storage? And in case storage consists of 15k rpm Cheetahs with 3.5ms average read seek time we should expect not more than ~285 rows per second?

I feel that I'm overlooking something here. But I'm new into data warehousing. :)

Also this query should greatly benefit from parallel execution or async IO. Storage (seeks/second) scales almost linearly when it has a lot of disks. And query is completely IO bound so it should scale well on single server.

And I cannot use some index organized table or table partitioned by From :) because there are at least 2 similar indexes by which queries can be executed - From and To.

Ideas for improvement? Greenplum or EnterpriseDB? Or I forgot something from PostgreSQL features.

Thanks,

Mindaugas

Responses

Browse pgsql-performance by date

  From Date Subject
Next Message Sami Dalouche 2007-12-02 11:05:35 Re: Dealing with big tables
Previous Message Jean-David Beyer 2007-12-01 17:03:59 Re: Utilizing multiple cores for one query