Optimize kernel readahead using buffer access strategy

From: KONDO Mitsumasa <kondo(dot)mitsumasa(at)lab(dot)ntt(dot)co(dot)jp>
To: PostgreSQL-development <pgsql-hackers(at)postgresql(dot)org>
Subject: Optimize kernel readahead using buffer access strategy
Date: 2013-11-14 12:09:27
Message-ID: 5284BD77.20503@lab.ntt.co.jp
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers

Hi,

I create a patch that is improvement of disk-read and OS file caches. It can
optimize kernel readahead parameter using buffer access strategy and
posix_fadvice() in various disk-read situations.

In general OS, readahead parameter was dynamically decided by disk-read
situations. If long time disk-read was happened, readahead parameter becomes big.
However it is based on experienced or heuristic algorithm, it causes waste
disk-read and throws out useful OS file caches in some case. It is bad for
disk-read performance a lot.

My proposed method is controlling OS readahead parameter by using buffer access
strategy in PostgreSQL and posix_fadvice() system call which can control OS
readahead parameter. Though, it is a general method in database.

For your information of effect of this patch, I got results of pgbench which are
in-memory-size database and out-memory-size database, and postgresql.conf
settings are always used by us. It seems to improve performance to a better. And
I think that this feature is going to be necessary for business intelligence
which will be realized at PostgreSQL version 10. I seriously believe Simon's
presentation in PostgreSQL conference Europe 2013! It was very exciting!!!

PostgreSQL have a lot of kind of disk-read method that are selected by planner,
however. I think that we need to discuss more other situations except pgbench,
and other cache cold situations. I think that optimizing kernel readahead
parameter with considering planner in PostgreSQL seems to be quite difficult, so
I seriously recruit co-author in this patch:-)

Regards,
--
Mitsumasa KONDO
NTT Open Source Software Center

Attachment Content-Type Size
image/jpeg 27.4 KB
image/jpeg 27.6 KB
optimize_kernel-readahead_using_buffer-access-strategy_v1.patch text/x-diff 8.2 KB

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message KONDO Mitsumasa 2013-11-14 12:25:12 Re: Add min and max execute statement time in pg_stat_statement
Previous Message Haribabu kommi 2013-11-14 12:08:29 New option for pg_basebackup, to specify a different directory for pg_xlog