Re: High %SYS CPU usage

Hello there,

I have PostgreSQL 8.3.18 server running on Centos 6.2 (2.6.32-220.7.1) with
this specs:

2x CPU AMD Opteron 6282
128GB RAM
Raid 10 (12HD 15k rpm 1GB cache) with data
Raid 10 (4HD 15k rpm 1GB cache) with xlog
Raid 1 (15k rpm 1GB cache shared with xlog) with system

On this server I have only one database with 312GB of data. The database
had run fine during 4 months, but from two months ago, during high work
load periods, the server is collapsed by "%sys" type load.

For example "dstat -ar --socket --tcp" during %sys load problem:
http://pastebin.com/7zfDNvPh

Reboot the server mitigates the problem during few days, but always
reappear.
Server not is swapping, don't have excessive I/O, don't have %IRQ load.

I don't have any ideas...

Thank you very much for your help.

My sysctl and postgres.conf:

sysclt -a:
http://pastebin.com/EEVnNxsZ

My Postgres.conf:
max_connections = 500 # (change requires restart)
unix_socket_directory = '/var/run/postgres' # (change requires restart)
shared_buffers = 18GB # min 128kB or max_connections*16kB
work_mem = 30MB # min 64kB
maintenance_work_mem = 1GB # min 1MB
max_fsm_pages = 8553600 # min max_fsm_relations*16, 6 bytes each
max_fsm_relations = 409000 # min 100, ~70 bytes each
fsync = on # turns forced synchronization on or off
synchronous_commit = off # immediate fsync at commit
wal_buffers = 8MB # min 32kB
checkpoint_segments = 64 # in logfile segments, min 1, 16MB each
checkpoint_completion_target = 0.9 # checkpoint target duration, 0.0 - 1.0
archive_mode = on # allows archiving to be done
archive_command = 'exit 0'
effective_cache_size = 100GB
constraint_exclusion = on
default_text_search_config = 'pg_catalog.spanish'
max_locks_per_transaction = 100

--
César Martín Pérez
cmartinp(at)gmail(dot)com

Cesar,

> On this server I have only one database with 312GB of data. The database
> had run fine during 4 months, but from two months ago, during high work
> load periods, the server is collapsed by "%sys" type load.

Hmmm. Have you updated Linux any time recently? I'm wondering if this
is a PostgreSQL problem at all. It sounds like an OS issue.

Can you give us the results of mpstat -P ALL 3 ? The dstat output
doesn't tell me much.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Hi Josh,

Today is not the worse day for this issue, because Fridays DB have little
workload... But I have this output:

http://pastebin.com/bKg8tfKC

I don't have updated OS server from the install.
I also begin to think that the problem is OS misconfiguration (Kernel
parameters??) or Hardware problem...

Thanks for your help.

2012/10/30 Josh Berkus <josh(at)agliodbs(dot)com>

> Cesar,
>
> > On this server I have only one database with 312GB of data. The database
> > had run fine during 4 months, but from two months ago, during high work
> > load periods, the server is collapsed by "%sys" type load.
>
> Hmmm. Have you updated Linux any time recently? I'm wondering if this
> is a PostgreSQL problem at all. It sounds like an OS issue.
>
> Can you give us the results of mpstat -P ALL 3 ? The dstat output
> doesn't tell me much.
>
> --
> Josh Berkus
> PostgreSQL Experts Inc.
> http://pgexperts.com
>
>
> --
> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>

--
César Martín Pérez
cmartinp(at)gmail(dot)com

Finally I resolv the problem
setting /sys/kernel/mm/redhat_transparent_hugepage/enabled to "never"

2012/11/2 Cesar Martin <cmartinp(at)gmail(dot)com>

> Hi Josh,
>
> Today is not the worse day for this issue, because Fridays DB have little
> workload... But I have this output:
>
> http://pastebin.com/bKg8tfKC
>
> I don't have updated OS server from the install.
> I also begin to think that the problem is OS misconfiguration (Kernel
> parameters??) or Hardware problem...
>
> Thanks for your help.
>
>
> 2012/10/30 Josh Berkus <josh(at)agliodbs(dot)com>
>
>> Cesar,
>>
>> > On this server I have only one database with 312GB of data. The database
>> > had run fine during 4 months, but from two months ago, during high work
>> > load periods, the server is collapsed by "%sys" type load.
>>
>> Hmmm. Have you updated Linux any time recently? I'm wondering if this
>> is a PostgreSQL problem at all. It sounds like an OS issue.
>>
>> Can you give us the results of mpstat -P ALL 3 ? The dstat output
>> doesn't tell me much.
>>
>> --
>> Josh Berkus
>> PostgreSQL Experts Inc.
>> http://pgexperts.com
>>
>>
>> --
>> Sent via pgsql-performance mailing list (pgsql-performance(at)postgresql(dot)org
>> )
>> To make changes to your subscription:
>> http://www.postgresql.org/mailpref/pgsql-performance
>>
>
>
>
> --
> César Martín Pérez
> cmartinp(at)gmail(dot)com
>
>

--
César Martín Pérez
cmartinp(at)gmail(dot)com