Re: Toooo many context switches (maybe SLES8?)

Hi,

we have a complex modperl database application using postgresql 7.4.1 on
a new Dual Xeon MP Machine with SLES8 which seems to generate too much
context switches (way more than 100.000) on higher load (meaning system
load > 2). System response times significantly slow down then. We have
tuned parameters for weeks now but could not come up with better
results. It seems that we have had better performance on an older Dual
XEON DP Machine running on RedHat 7.3.

Here is the config:

database machine on SuSE SLES 8:

F-S Primergy RX600
2x XEON MP 2.5GHz
8GB RAM
Hardware Raid 1+0 140GB
Kernel 2.4.21-169-smp

Postgresql 7.4.1 (self compiled) with
max_connections = 170
shared_buffers = 40000
effective_cache_size = 800000
sort_mem = 30000
vacuum_mem = 420000
max_fsm_relations = 2000
max_fsm_pages = 200000
random_page_cost = 4
checkpoint_segments = 24
wal_buffers = 32

modperl application machine on RH 7.3:

F-S Primergy RX200
2x XEON DP 2.4 GHz
4 GB RAM
Kernel 2.4.18-10smp, RedHat 7.3
Apache 1.3.27 setup:
MinSpareServers 15
MaxSpareServers 30
StartServers 15
MaxClients 80
MaxRequestsPerChild 100

vmstat 1 excerpt:

procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy
id wa
1 0 4868 242372 179488 6942316 0 0 12 8 18 9 6
2 92 0
2 1 4868 242204 179488 6942500 0 0 64 500 701 117921 35
18 48 0
0 1 4868 242032 179392 6941560 0 0 16 316 412 132295 28
25 47 0
1 0 4872 242396 179164 6933776 0 0 128 276 474 69708 21
24 56 0
3 0 4872 242536 179164 6933808 0 0 0 240 412 113643 27
27 46 0
2 0 4872 242872 179092 6931708 0 0 48 1132 521 127916 24
24 53 0
0 0 4876 242876 179092 6927512 0 0 48 532 504 117868 32
21 47 0
0 0 4876 242504 179096 6927560 0 0 0 188 412 127147 34
20 47 0
1 0 4876 242152 179096 6927856 0 0 96 276 529 117684 28
23 49 0
2 0 4876 242864 179096 6928384 0 0 88 560 507 135717 38
19 43 0
1 0 4876 242848 179096 6928520 0 0 64 232 433 151380 32
20 48 0
4 0 4876 242832 179144 6928916 0 0 16 10380 2913 112583 28
20 52 0
4 0 4876 242720 179144 6929240 0 0 196 0 329 154821 32
18 50 0
3 2 4876 243576 179144 6929408 0 0 0 460 451 160287 29
18 52 0
3 0 4876 243292 179180 6929468 0 0 16 436 614 51894 15
5 80 0
0 0 4876 243884 179180 6929580 0 0 0 236 619 154168 29
21 49 0
2 1 4876 243864 179180 6929860 0 0 128 380 493 155903 31
19 50 0
2 0 4876 244720 179180 6930276 0 0 16 1208 561 129336 27
16 56 0
2 0 4876 247204 179180 6930300 0 0 0 0 361 146268 33
20 47 0
3 0 4876 248620 179180 6930372 0 0 0 168 346 155915 32
12 56 0
2 0 4876 250476 179180 6930436 0 0 0 184 328 163842 35
20 46 0
0 0 4876 250496 179180 6930652 0 0 48 260 450 144930 31
15 53 0
1 0 4876 252236 179180 6930732 0 0 16 244 577 167259 35
15 50 0
0 0 4876 252236 179180 6930780 0 0 0 464 622 165488 31
15 54 0
1 0 4876 252268 179180 6930812 0 0 0 132 460 153381 34
15 52 0
2 0 4876 252268 179180 6930964 0 0 0 216 312 141009 31
19 50 0
1 0 4876 252264 179180 6930980 0 0 0 56 275 153143 33
20 47 0
2 0 4876 252212 179180 6931212 0 0 96 296 400 133982 32
18 50 0
1 0 4876 252264 179180 6931332 0 0 0 300 416 136034 32
18 50 0
1 1 4876 252264 179180 6931332 0 0 0 236 377 143300 34
22 44 0
4 0 4876 254876 179180 6931372 0 0 0 124 446 118117 34
20 45 0
1 0 4876 254876 179180 6931492 0 0 16 144 462 140499 38
16 46 0
2 0 4876 255860 179180 6931572 0 0 16 144 674 126250 33
20 47 0
1 0 4876 255860 179180 6931788 0 0 48 264 964 115679 36
13 51 0
3 0 4876 255864 179180 6931804 0 0 0 100 597 127619 36
19 46 0
5 1 4876 255864 179180 6931924 0 0 72 352 559 151620 34
18 48 0
2 0 4876 255860 179184 6932100 0 0 96 120 339 137821 34
20 47 0
0 0 4876 255860 179184 6932156 0 0 8 168 469 125281 36
21 43 0
2 0 4876 256092 179184 6932444 0 0 112 328 446 137939 34
19 48 0
2 0 4876 256092 179184 6932484 0 0 16 184 382 141800 35
16 49 0
3 0 4876 256464 179184 6932716 0 0 16 356 448 134238 30
18 51 0
5 0 4876 256464 179184 6932892 0 0 96 600 476 142838 34
20 46 0
1 0 4876 256464 179184 6933012 0 0 16 176 589 138546 35
22 43 0
2 0 4876 256436 179184 6933096 0 0 60 76 396 93110 42
17 41 0
1 0 4876 256464 179184 6933484 0 0 212 276 442 83060 45
11 44 0
5 0 4876 257612 179184 6933604 0 0 0 472 548 94158 39
17 45 0
0 0 4876 257560 179184 6933708 0 0 96 96 518 116764 38
19 43 0
1 0 4876 257612 179184 6933796 0 0 0 1768 729 139013 29
19 53 0
4 0 4876 257612 179184 6934188 0 0 296 108 332 134703 31
21 48 0
0 1 4876 258584 179184 6934380 0 0 0 492 405 141198 34
18 48 0
1 0 4876 258584 179184 6934492 0 0 0 176 575 134771 37
16 48 0
4 1 4876 257796 179184 6935724 0 0 1176 176 438 151240 33
20 48 0
1 0 4876 261448 179184 6935836 0 0 0 252 489 134348 29
19 51 0
2 0 4876 261448 179184 6935852 0 0 0 512 639 130875 34
16 49 0
2 1 4876 261724 179184 6935924 0 0 0 80 238 144970 33
20 47 0

Dirk Lutzebäck wrote:
> postgresql 7.4.1

> a new Dual Xeon MP

> too much context switches (way more than 100.000) on higher load (meaning system
> load > 2).

I believe this was fixed in 7.4.2, although I can't seem to find it in
the release notes.

Joe

Joe, do you know where I should look in the 7.4.2 code to find this out?

Dirk

Joe Conway wrote:

> Dirk Lutzebäck wrote:
>
>> postgresql 7.4.1
>
>> a new Dual Xeon MP
>
>> too much context switches (way more than 100.000) on higher load
>> (meaning system load > 2).
>
>
> I believe this was fixed in 7.4.2, although I can't seem to find it in
> the release notes.
>
> Joe
>
>

Dirk Lutzebäck wrote:
> Joe, do you know where I should look in the 7.4.2 code to find this out?

I think I was wrong. I just looked in CVS and found the commit I was
thinking about:

http://developer.postgresql.org/cvsweb.cgi/pgsql-server/src/backend/storage/lmgr/s_lock.c.diff?r1=1.22&r2=1.23
http://developer.postgresql.org/cvsweb.cgi/pgsql-server/src/include/storage/s_lock.h.diff?r1=1.123&r2=1.124

=========================
Revision 1.23 / (download) - [select for diffs] , Sat Dec 27 20:58:58
2003 UTC (3 months, 2 weeks ago) by tgl
Changes since 1.22: +5 -1 lines
Diff to previous 1.22

Improve spinlock code for recent x86 processors: insert a PAUSE
instruction in the s_lock() wait loop, and use test before test-and-set
in TAS() macro to avoid unnecessary bus traffic. Patch from Manfred
Spraul, reworked a bit by Tom.
=========================

I thought this had been committed to the 7.4 stable branch as well, but
it appears not.

Joe

Joe,

> I believe this was fixed in 7.4.2, although I can't seem to find it in
> the release notes.

Depends on the cause of the issue. If it's the same issue that I'm currently
struggling with, it's not fixed.

--
-Josh Berkus
Aglio Database Solutions
San Francisco

Joe Conway <mail(at)joeconway(dot)com> writes:
>> Improve spinlock code for recent x86 processors: insert a PAUSE
>> instruction in the s_lock() wait loop, and use test before test-and-set
>> in TAS() macro to avoid unnecessary bus traffic. Patch from Manfred
>> Spraul, reworked a bit by Tom.

> I thought this had been committed to the 7.4 stable branch as well, but
> it appears not.

I am currently chasing what seems to be the same issue: massive context
swapping on a dual Xeon system. I tried back-patching the above-mentioned
patch ... it helps a little but by no means solves the problem ...

regards, tom lane

Could this be related to the O(1) scheduler backpatches from 2.6 to 2.4
kernel on newer 2.4er distros (RedHat, SuSE)?

Tom Lane wrote:

>Joe Conway <mail(at)joeconway(dot)com> writes:
>
>
>>>Improve spinlock code for recent x86 processors: insert a PAUSE
>>>instruction in the s_lock() wait loop, and use test before test-and-set
>>>in TAS() macro to avoid unnecessary bus traffic. Patch from Manfred
>>>Spraul, reworked a bit by Tom.
>>>
>>>
>
>
>
>>I thought this had been committed to the 7.4 stable branch as well, but
>>it appears not.
>>
>>
>
>I am currently chasing what seems to be the same issue: massive context
>swapping on a dual Xeon system. I tried back-patching the above-mentioned
>patch ... it helps a little but by no means solves the problem ...
>
> regards, tom lane
>
>
>

Folks,

> I am currently chasing what seems to be the same issue: massive context
> swapping on a dual Xeon system. I tried back-patching the above-mentioned
> patch ... it helps a little but by no means solves the problem ...

BTW, I'm currently pursuing the possibility that this has something to do with
the ServerWorks chipset on those motherboards. If anyone knows a high-end
hardware+linux kernel geek I can corner, I'd appreciate it.

Maybe I should contact OSDL ...

--
Josh Berkus
Aglio Database Solutions
San Francisco

Isn't this a linux kernel issue ?

My understanding is that the scheduler doesn't know that 2 of the CPU's
are actually the same underlying hardware and sometimes two contexts end
up fighting for the same underlying chip?

--dc--

On Thu, 2004-04-15 at 16:37, Josh Berkus wrote:
> Folks,
>
> > I am currently chasing what seems to be the same issue: massive context
> > swapping on a dual Xeon system. I tried back-patching the above-mentioned
> > patch ... it helps a little but by no means solves the problem ...
>
> BTW, I'm currently pursuing the possibility that this has something to do with
> the ServerWorks chipset on those motherboards. If anyone knows a high-end
> hardware+linux kernel geek I can corner, I'd appreciate it.
>
> Maybe I should contact OSDL ...
--
Dave Cramer
519 939 0336
ICQ # 14675561

Don't think so, mine is a vanilla kernel from kernel.org

Dave
On Thu, 2004-04-15 at 16:03, Dirk Lutzebaeck wrote:
> Could this be related to the O(1) scheduler backpatches from 2.6 to 2.4
> kernel on newer 2.4er distros (RedHat, SuSE)?
>
>
> Tom Lane wrote:
>
> >Joe Conway <mail(at)joeconway(dot)com> writes:
> >
> >
> >>>Improve spinlock code for recent x86 processors: insert a PAUSE
> >>>instruction in the s_lock() wait loop, and use test before test-and-set
> >>>in TAS() macro to avoid unnecessary bus traffic. Patch from Manfred
> >>>Spraul, reworked a bit by Tom.
> >>>
> >>>
> >
> >
> >
> >>I thought this had been committed to the 7.4 stable branch as well, but
> >>it appears not.
> >>
> >>
> >
> >I am currently chasing what seems to be the same issue: massive context
> >swapping on a dual Xeon system. I tried back-patching the above-mentioned
> >patch ... it helps a little but by no means solves the problem ...
> >
> > regards, tom lane
> >
> >
> >
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>
>
>
> !DSPAM:408535ce93801252113544!
>
>
--
Dave Cramer
519 939 0336
ICQ # 14675561