Re: Parallel Select query performance and shared buffers

From: Metin Doslu <metin(at)citusdata(dot)com>
To: postgres performance list <pgsql-performance(at)postgresql(dot)org>, pgsql-hackers <pgsql-hackers(at)postgresql(dot)org>
Cc: Amit Kapila <amit(dot)kapila16(at)gmail(dot)com>, Andres Freund <andres(at)2ndquadrant(dot)com>, Claudio Freire <klaussfreire(at)gmail(dot)com>
Subject: Re: Parallel Select query performance and shared buffers
Date: 2013-12-04 18:19:55
Message-ID: CAL1dPcdm1q0q0VtOpe2gfOyCwS7FkzURw=ZQ1RsLgfKgz5=gMg@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-hackers pgsql-performance

Here are some extra information:

- When we increased NUM_BUFFER_PARTITIONS to 1024, this problem is
disappeared for 8 core machines and come back with 16 core machines on
Amazon EC2. Would it be related with PostgreSQL locking mechanism?

- I tried this test with 4 core machines including my personel computer and
some other instances on Amazon EC2, I didn't see this problem with 4 core
machines. I started to see this problem in PostgreSQL when core count is 8
or more.

- Here are the results of "vmstat 1" while running 8 parallel select
count(*). Normally I would expect zero idle time.

procs -----------memory---------- ---swap-- -----io---- --system--
-----cpu-----
r b swpd free buff cache si so bi bo in cs us sy id
wa st
0 0 0 29838640 94000 38954740 0 0 0 0 22 21 0 0
100 0 0
7 2 0 29788416 94000 38954740 0 0 0 0 53922 108490 14
24 60 1 1
5 0 0 29747248 94000 38954740 0 0 0 0 68008 164571 22
48 27 2 1
8 0 0 29725796 94000 38954740 0 0 0 0 43587 150574 28
54 16 1 1
0 0 0 29838328 94000 38954740 0 0 0 0 15584 100459 26
55 18 1 0
0 0 0 29838328 94000 38954740 0 0 0 0 42 15 0 0
100 0 0

- When I run 8 parallel wc command or other scripts, they scale out as
expected and they utilize all cpu. This leads me to think that problem is
related with PostgreSQL instead of OS.

In response to

Responses

Browse pgsql-hackers by date

  From Date Subject
Next Message Andres Freund 2013-12-04 18:26:01 Re: [HACKERS] Parallel Select query performance and shared buffers
Previous Message Metin Doslu 2013-12-04 18:06:27 Re: Parallel Select query performance and shared buffers

Browse pgsql-performance by date

  From Date Subject
Next Message Andres Freund 2013-12-04 18:26:01 Re: [HACKERS] Parallel Select query performance and shared buffers
Previous Message Metin Doslu 2013-12-04 18:06:27 Re: Parallel Select query performance and shared buffers