Re: 100% cpu usage on some postmaster processes kill the complete database

From: Scott Marlowe <scott(dot)marlowe(at)gmail(dot)com>
To: Paul Dunkler <paul(dot)dunkler(at)xyrality(dot)com>
Cc: Richard Huxton <dev(at)archonet(dot)com>, pgsql-general(at)postgresql(dot)org
Subject: Re: 100% cpu usage on some postmaster processes kill the complete database
Date: 2012-03-01 21:14:56
Message-ID: CAOR=d=2x-U4CJsmaG4jCYk2N5YydurhVcMx87vKA_Y8gj03a0A@mail.gmail.com
Views: Raw Message | Whole Thread | Download mbox | Resend email
Thread:
Lists: pgsql-general

On Thu, Mar 1, 2012 at 1:52 PM, Paul Dunkler <paul(dot)dunkler(at)xyrality(dot)com> wrote:
>
> Hi,
>
> You're running on a box larger than I'm used to, so this is only speculation. I'm wondering whether you're hitting problems with lock contention or some such. It looks like you've got 48 cores there all at about 100% possibly none of them getting much chance to do any work.
>
>
> Yes. That is what i see too...
>
> Oddly, the totals you posted in your top output show 6.3% user cpu usage, which I can't make match with 50-odd processes all approaching 100% cpu.
>
>
> Sometimes the Cpu is only 7% used in this times but at other peak times, the cpu is used 100% (97% system load) as i posted before.
>
> Perhaps have a look at vmstat output too - see if context-switches spike unusually high during these periods (sorry - no idea what an unusually high number would be on a machine like yours).
>
>
> Thanks. i will have a look at it.
>
> Reducing the number of concurrent backends might help, but that rather depends on whether my guess is right.
>
>
> Yes... already thought about setting up a connection pool.
>
> If no-one more experienced than me comes along shortly, try reposting to the performance list. There are people there who are used to machines of this size.
>
>
> Thanks. I will wait a time and consider re-posting it to the perfornance list.
>

I'd look at vmstat and iostat output (vmstat 10, iostat -xd 10) for a
few minutes. In vmstat look for high (>100k) ints or cs numbers, in
iostat look at io utilization.

In response to

Browse pgsql-general by date

  From Date Subject
Next Message Tom Lane 2012-03-01 21:47:43 Re: 2 x duplicate key value violation: which exception comes first ?
Previous Message Paul Dunkler 2012-03-01 20:52:14 Re: 100% cpu usage on some postmaster processes kill the complete database