Lists: | pgsql-performance |
---|
From: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
---|---|
To: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | serveRAID M5014 SAS |
Date: | 2011-05-25 07:33:52 |
Message-ID: | BANLkTi=fE2YCS0Lra42ei1hFrKqXdVk2jA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Does anyone here have any bad experiences with the RAID card in subject ?
This is in an IBM server, with 2.5" 10k drives.
But we seem to observe its poor performance in other configurations as
well (with different drives, different settings) in comparison with -
say, what dell provides.
Any experiences ?
--
GJ
From: | Greg Smith <greg(at)2ndquadrant(dot)com> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-25 14:00:49 |
Message-ID: | 4DDD0B91.9080300@2ndquadrant.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Grzegorz Jaśkiewicz wrote:
> Does anyone here have any bad experiences with the RAID card in subject ?
> This is in an IBM server, with 2.5" 10k drives.
>
> But we seem to observe its poor performance in other configurations as
> well (with different drives, different settings) in comparison with -
> say, what dell provides.
>
Older ServeRAID cards have never been reported as very fast. They were
an OK controller if you just want to mirror a pair of drives or
something simple like that. Their performance on larger RAID arrays is
terrible compared to the LSI products that Dell uses.
However, the M5014 *is* an LSI derived product, with a proper
battery-backed write cache and everything. I'm not sure if older views
are even useful now. Few things to check:
-Is the battery working, and the write cache set in write-back mode?
-Has read-ahead been set usefully?
-Did you try to use a more complicated RAID mode than this card can handle?
Those are the three easiest ways to trash performance here.
--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
From: | "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> |
---|---|
To: | "Greg Smith" <greg(at)2ndquadrant(dot)com>,<gryzman(at)gmail(dot)com> |
Cc: | "postgres performance list" <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-25 14:54:00 |
Message-ID: | 4DDCD1B8020000250003DC24@gw.wicourts.gov |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Greg Smith <greg(at)2ndquadrant(dot)com> wrote:
> -Is the battery working, and the write cache set in write-back
> mode?
My bet is on this point.
Our hardware tech says that the difference between an M5014 and an
M5015 is that the former takes a maximum of 256MB RAM while the
latter takes a maximum of 512MB RAM and that the M5014 ships with
*no* RAM by default. He says you have to order the RAM as an
extra-cost option on the M5014.
-Kevin
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-25 22:24:59 |
Message-ID: | 4DDD81BB.4080801@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 25/05/11 19:33, Grzegorz Jaśkiewicz wrote:
> Does anyone here have any bad experiences with the RAID card in subject ?
> This is in an IBM server, with 2.5" 10k drives.
>
> But we seem to observe its poor performance in other configurations as
> well (with different drives, different settings) in comparison with -
> say, what dell provides.
>
>
>
Interestingly enough, I've been benchmarking a M5015 SAS, with the
optional wee cable for enabling the battery backup for the 512MB of
cache. With a 6 disk raid 10 + 2 disk raid 1 - with the array settings
NORA+DIRECT, and writeback enabled we're seeing quite good pgbench
performance (12 cores + 48G ram, Ubuntu 10.04 with xfs):
scale 2500 db with 48 clients, 10 minute runs: 2300 tps
scale 500 db with 24 clients, 10 minute runs: 6050 tps
I did notice that the sequential performance was quite lackluster (using
bonnie) - but are not too concerned about that for the use case (could
probably fix using blockdev --setra).
I'm guessing that even tho your M5014 card comes with less ram (256M I
think), if you can enable the battery backup and cache writes it should
be quite good. Also I think the amount of ram on the card is upgradable
(4G is the max for the M5105 I *think* - can't find the right doc to
check this ATM sorry).
Cheers
Mark
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-25 22:42:15 |
Message-ID: | 4DDD85C7.6030109@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 26/05/11 10:24, Mark Kirkwood wrote:
>
>
> Also I think the amount of ram on the card is upgradable (4G is the
> max for the M5105 I *think* - can't find the right doc to check this
> ATM sorry).
>
>
Looking at the (very sparse) product docs, it looks I am mistaken above
- and that the cache sizes are 256M for M5014 and 512M for M5015 and are
not upgradable beyond that. Looking at Kevin's post, I recommend
checking if you ordered the cache and battery with your card.
Cheers
Mark
From: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
---|---|
To: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 08:11:18 |
Message-ID: | BANLkTi=PAymQvudx1eyuuOV9vAAoydan9g@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
The card is configured in 1+0 . with 128k stripe afaik (I'm a
developer, we don't have hardware guys here).
Are you's sure about the lack of cache by default on the card ? I
thought the difference is that 5104 has 256, and 5105 has 512 ram
already on it.
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 08:25:27 |
Message-ID: | 4DDE0E77.6070804@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 26/05/11 20:11, Grzegorz Jaśkiewicz wrote:
> The card is configured in 1+0 . with 128k stripe afaik (I'm a
> developer, we don't have hardware guys here).
> Are you's sure about the lack of cache by default on the card ? I
> thought the difference is that 5104 has 256, and 5105 has 512 ram
> already on it.
No, I'm not sure about what the default is for the M5014 - I'd recommend
checking this with your supplier (or looking at the invoice if you can
get it). My *feeling* is that you may have 256M cache but no battery kit
- as this is an optional part - so the the card will not got into
writeback mode if that is the case.
FWIW - we got our best (pgbench) results with 256K stripe, No (card)
readahead and hyperthreading off on the host.
You can interrogate the config of the card and the raid 10 array using
the megaraid cli package - you need to read the (frankly terrible)
manual to discover which switches to use to determine battery and cache
status etc. If you email me privately I'll get you a link to the
relevant docs!
Cheers
Mark
From: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
---|---|
To: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 08:31:05 |
Message-ID: | BANLkTin_R98uAP4gCutGB61uTNoQSgELbA@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Would HT have any impact to the I/O performance (postgresql, and fs in
general) ?.
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 22:43:25 |
Message-ID: | 4DDED78D.6060508@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 26/05/11 20:31, Grzegorz Jaśkiewicz wrote:
> Would HT have any impact to the I/O performance (postgresql, and fs in
> general) ?.
>
There have been previous discussions on this list about HT on vs off (I
can't recall what the consensus, if any about what the cause of any
performance difference was). In our case HT off gave us much better
results for what we think the typical number of clients will be - see
attached (server learn-db1 is setup with trivial hardware raid and then
software raided via md, learn-db2 has its raid all in hardware. We've
ended up going with the latter setup).
Note that the highest tps on the graph is about 2100 - we got this upto
just over 2300 by changing from ext4 to xfs in later tests, and managed
to push the tps for 100 clients up a little by setting no read ahead
(NORA) for the arrays.
Cheers
Mark
Attachment | Content-Type | Size |
---|---|---|
![]() |
image/png | 14.5 KB |
![]() |
image/png | 14.0 KB |
From: | Greg Smith <greg(at)2ndquadrant(dot)com> |
---|---|
To: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
Cc: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com>, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 23:19:27 |
Message-ID: | 4DDEDFFF.1060304@2ndquadrant.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Mark Kirkwood wrote:
> You can interrogate the config of the card and the raid 10 array using
> the megaraid cli package - you need to read the (frankly terrible)
> manual to discover which switches to use to determine battery and
> cache status etc. If you email me privately I'll get you a link to the
> relevant docs!
That's assuming the MegaCli utility will work against IBM's version of
the card. They use an LSI chipset for the RAID parts, but I don't know
if the card is so similar that it will talk using that utility or not.
The main useful site here is
http://tools.rapidsoft.de/perc/perc-cheat-sheet.html ; here's how to
dump all the main config info from an LSI card:
MegaCli64 -LDInfo -Lall -aALL
You want to see a line like this:
Current Cache Policy: WriteBack, ReadAheadNone, Direct, No Write Cache
if Bad BBU
For the arrays. And then check the battery like this:
MegaCli64 -AdpBbuCmd -aALL
--
Greg Smith 2ndQuadrant US greg(at)2ndQuadrant(dot)com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Greg Smith <greg(at)2ndquadrant(dot)com> |
Cc: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com>, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 23:22:55 |
Message-ID: | 4DDEE0CF.5040800@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 27/05/11 11:19, Greg Smith wrote:
> Mark Kirkwood wrote:
>> You can interrogate the config of the card and the raid 10 array
>> using the megaraid cli package - you need to read the (frankly
>> terrible) manual to discover which switches to use to determine
>> battery and cache status etc. If you email me privately I'll get you
>> a link to the relevant docs!
>
> That's assuming the MegaCli utility will work against IBM's version of
> the card. They use an LSI chipset for the RAID parts, but I don't
> know if the card is so similar that it will talk using that utility or
> not.
It does seem to.
Cheers
Mark
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Greg Smith <greg(at)2ndquadrant(dot)com> |
Cc: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com>, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-26 23:42:56 |
Message-ID: | 4DDEE580.3040305@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 27/05/11 11:22, Mark Kirkwood wrote:
> On 27/05/11 11:19, Greg Smith wrote:
>> Mark Kirkwood wrote:
>>> You can interrogate the config of the card and the raid 10 array
>>> using the megaraid cli package - you need to read the (frankly
>>> terrible) manual to discover which switches to use to determine
>>> battery and cache status etc. If you email me privately I'll get you
>>> a link to the relevant docs!
>>
>> That's assuming the MegaCli utility will work against IBM's version
>> of the card. They use an LSI chipset for the RAID parts, but I don't
>> know if the card is so similar that it will talk using that utility
>> or not.
>
> It does seem to.
>
> Cheers
>
> Mark
>
e.g checking battery status:
root(at)learn-db2:~# MegaCli64 -AdpBbuCmd -GetBbuStatus -a0
BBU status for Adapter: 0
BatteryType: iBBU
Voltage: 4040 mV
Current: 0 mA
Temperature: 28 C
BBU Firmware Status:
Charging Status : None
Voltage : OK
Temperature : OK
Learn Cycle Requested : No
Learn Cycle Active : No
Learn Cycle Status : OK
Learn Cycle Timeout : No
I2c Errors Detected : No
Battery Pack Missing : No
Battery Replacement required : No
Remaining Capacity Low : No
Periodic Learn Required : No
Transparent Learn : No
Battery state:
GasGuageStatus:
Fully Discharged : No
Fully Charged : Yes
Discharging : Yes
Initialized : Yes
Remaining Time Alarm : No
Remaining Capacity Alarm: No
Discharge Terminated : No
Over Temperature : No
Charging Terminated : No
Over Charged : No
Relative State of Charge: 99 %
Charger System State: 49168
Charger System Ctrl: 0
Charging current: 0 mA
Absolute state of charge: 99 %
Max Error: 2 %
Exit Code: 0x00
Reminds me of out from DB2 diag commands (years ago...am ok now).
From: | "Kevin Grittner" <Kevin(dot)Grittner(at)wicourts(dot)gov> |
---|---|
To: | "Mark Kirkwood" <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
Cc: | "Greg Smith" <greg(at)2ndquadrant(dot)com>,<gryzman(at)gmail(dot)com>, "postgres performance list" <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-27 14:38:24 |
Message-ID: | 4DDF7110020000250003DDD4@gw.wicourts.gov |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> wrote:
> Battery Pack Missing : No
> Fully Charged : Yes
> Initialized : Yes
I'm not familiar with that output (I leave that to the hardware
guys), but it sure looks like there's a battery there. The one
thing I didn't see is whether it's configured for write-through or
write-back. You want write-back for good performance.
-Kevin
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov> |
Cc: | Greg Smith <greg(at)2ndquadrant(dot)com>, gryzman(at)gmail(dot)com, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-27 23:24:43 |
Message-ID: | 4DE032BB.1070401@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 28/05/11 02:38, Kevin Grittner wrote:
> Mark Kirkwood<mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> wrote:
>
>> Battery Pack Missing : No
>> Fully Charged : Yes
>> Initialized : Yes
>
> I'm not familiar with that output (I leave that to the hardware
> guys), but it sure looks like there's a battery there. The one
> thing I didn't see is whether it's configured for write-through or
> write-back. You want write-back for good performance.
>
>
Sorry for the confusion Kevin - that's the output for *our* M5015 with a
battery - what we need to see is the output for
Grzegorz's M5014. Grzegorz - can you get that?
Cheers
Mark
From: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
---|---|
To: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
Cc: | Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, Greg Smith <greg(at)2ndquadrant(dot)com>, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-28 09:42:54 |
Message-ID: | BANLkTimiYqgbTBqhC9Ad0VgiE3Efo=UK7Q@mail.gmail.com |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
Yeah, I got it Mark.
Unfortunately my current management decided to suspend that
investigation for a while, so I can't get any tests done or anything
like that.
However we found out that another server we have shows similar issues.
The same card, slightly different motherboard and completely different
disks.
The basic issue is around concurrent reads and writes. The card is
ok-ish when one process hammers the disks, but as soon as it is
multiple ones - it just blows (and I use it as a technical term ;) ).
Still, with only one process reading/writing - the raid card sucks big
time performance wise.
From: | Mark Kirkwood <mark(dot)kirkwood(at)catalyst(dot)net(dot)nz> |
---|---|
To: | Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com> |
Cc: | Kevin Grittner <Kevin(dot)Grittner(at)wicourts(dot)gov>, Greg Smith <greg(at)2ndquadrant(dot)com>, postgres performance list <pgsql-performance(at)postgresql(dot)org> |
Subject: | Re: serveRAID M5014 SAS |
Date: | 2011-05-28 10:15:15 |
Message-ID: | 4DE0CB33.5060703@catalyst.net.nz |
Views: | Raw Message | Whole Thread | Download mbox | Resend email |
Lists: | pgsql-performance |
On 28/05/11 21:42, Grzegorz Jaśkiewicz wrote:
> Yeah, I got it Mark.
>
> Unfortunately my current management decided to suspend that
> investigation for a while, so I can't get any tests done or anything
> like that.
>
> However we found out that another server we have shows similar issues.
> The same card, slightly different motherboard and completely different
> disks.
> The basic issue is around concurrent reads and writes. The card is
> ok-ish when one process hammers the disks, but as soon as it is
> multiple ones - it just blows (and I use it as a technical term ;) ).
> Still, with only one process reading/writing - the raid card sucks big
> time performance wise.
>
Sorry, Grzegorz I didn't mean to suggest you were not listening, I meant
to ask if you could run the megaraid cli command to see if it showed up
a battery!
If/when your management decide to let you look at this again, I'll be
happy to help if I can!
Best wishes
Mark