Re: Scaling shared buffer eviction

On Thu, Oct 9, 2014 at 6:17 PM, Amit Kapila <amit(dot)kapila16(at)gmail(dot)com> wrote:
>
> On Fri, Sep 26, 2014 at 7:04 PM, Robert Haas <robertmhaas(at)gmail(dot)com>
wrote:
> >
> > On another point, I think it would be a good idea to rebase the
> > bgreclaimer patch over what I committed, so that we have a
> > clean patch against master to test with.
>
> Please find the rebased patch attached with this mail. I have taken
> some performance data as well and done some analysis based on
> the same.
>
>
>
> Below data is median of 3 runs.
>
> patch_ver/client_count 1 8 32 64 128 256
> HEAD 18884 118628 251093 216294 186625 177505
> PATCH 18743 122578 247243 205521 179712 175031
>
>
> Here we can see that the performance dips at higher client
> count(>=32) which was quite surprising for me, as I was expecting
> it to improve, because bgreclaimer reduces the contention by making
> buffers available on free list. So I tried to analyze the situation by
> using perf and found that in above configuration, there is a contention
> around freelist spinlock with HEAD and the same is removed by Patch,
> but still the performance goes down with Patch. On further analysis, I
> observed that actually after Patch there is an increase in contention
> around ProcArrayLock (shared LWlock) via GetSnapshotData which
> sounds bit odd, but that's what I can see in profiles. Based on analysis,
> few ideas which I would like to further investigate are:
> a. As there is an increase in spinlock contention, I would like to check
> with Andres's latest patch which reduces contention around shared
> lwlocks.

I have tried by applying Andres's Wait free LW_SHARED acquisition
patch posted by him at below link along with bgreclaimer patch:
http://www.postgresql.org/message-id/20141008133533.GA5053@alap3.anarazel.de

After that I observed that contention for LW_SHARED has reduced
for this load, but it didn't help much in terms of performance, so I again
rechecked the profile and this time most of the contention is moved
to spinlock used in dynahash for buf mapping tables, please refer
the profile (for 128 client count; Read only load) below:

bgreclaimer patch + wait free lw_shared acquisition patches -
------------------------------------------------------------------------------------------

9.24% swapper [unknown] [H] 0x00000000011d9c10
+ 7.19% postgres postgres [.] s_lock
+ 3.52% postgres postgres [.] GetSnapshotData
+ 3.02% postgres postgres [.] calc_bucket
+ 2.71% postgres postgres [.]
hash_search_with_hash_value
2.32% postgres [unknown] [H] 0x00000000011e0d7c
+ 2.17% postgres postgres [.]
pg_atomic_fetch_add_u32_impl
+ 1.84% postgres postgres [.] AllocSetAlloc
+ 1.57% postgres postgres [.] _bt_compare
+ 1.05% postgres postgres [.] AllocSetFreeIndex
+ 1.02% postgres [kernel.kallsyms] [k]
.__copy_tofrom_user_power7
+ 0.94% postgres postgres [.] tas
+ 0.85% swapper [kernel.kallsyms] [k] .int_sqrt
+ 0.80% postgres postgres [.] pg_encoding_mbcliplen
+ 0.78% pgbench [kernel.kallsyms] [k] .find_busiest_group
0.65% pgbench [unknown] [H] 0x00000000011d96e0
+ 0.59% postgres postgres [.] hash_any
+ 0.54% postgres postgres [.] LWLockRelease

Detailed Profile Data
-----------------------------------------
- 7.19% postgres postgres [.] s_lock

- s_lock

- 3.79% s_lock

- 1.69% get_hash_entry

hash_search_with_hash_value

BufTableInsert

BufferAlloc

0

- 1.63% hash_search_with_hash_value

- 1.63% BufTableDelete

BufferAlloc

ReadBuffer_common

ReadBufferExtended
- 1.45% hash_search_with_hash_value

- 1.45% hash_search_with_hash_value

BufTableDelete

BufferAlloc

ReadBuffer_common
- 1.43% get_hash_entry

- 1.43% get_hash_entry

hash_search_with_hash_value

BufTableInsert

BufferAlloc

ReadBuffer_common
- 3.52% postgres postgres [.] GetSnapshotData

- GetSnapshotData

- 3.50% GetSnapshotData

- 3.49% GetTransactionSnapshot

- 1.75% exec_bind_message

PostgresMain

0

- 1.74% PortalStart

exec_bind_message

PostgresMain

0

To reduce above contention, I tried to write a patch to replace spin lock
used in dynahash to manage free list by atomic operations. Still there
is work pending for this patch with respect to ensuring whether the
approach used in patch is completely sane, however I am posting the
patch so that others can have a look at it and give me feedback about
the approach.

With Regards,
Amit Kapila.
EnterpriseDB: http://www.enterprisedb.com