Re: What's the best hardver for PostgreSQL 8.1?

Frank,

> You definitely DO NOT want to do RAID 5 on a database server. That
> is probably the worst setup you could have, I've seen it have lower
> performance than just a single hard disk.

I've seen that on RAID0 and RAID10 as well.

This is more about the quality and modernity of the RAID controller than
anything else at this point, although there are some theoretical
advantages of RAID10 from a random seek standpoint even if the adapter
CPU is infinitely fast at checksumming. We're using RAID5 in practice
for OLAP / Data Warehousing systems very successfully using the newest
RAID cards from 3Ware (9550SX).

Note that host-based SCSI raid cards from LSI, Adaptec, Intel, Dell, HP
and others have proven to have worse performance than a single disk
drive in many cases, whether for RAID0 or RAID5. In most circumstances
I've seen, people don't even notice until they write a message to a
mailing list about "my query runs slowly on xxx dbms". In many cases,
after they run a simple sequential transfer rate test using dd, they see
that their RAID controller is the culprit.

Recently, I helped a company named DeepData to improve their dbms
performance, which was a combination of moving them to software RAID50
on Linux and getting them onto Bizgres. The disk subsystem sped up on
the same hardware (minus the HW RAID card) by over a factor of 10. The
downside is that SW RAID is a pain in the neck for management - you have
to shut down the Linux host when a disk fails to replace it.

- Luke

On Sat, 24 Dec 2005, Luke Lonergan wrote:

> Recently, I helped a company named DeepData to improve their dbms
> performance, which was a combination of moving them to software RAID50
> on Linux and getting them onto Bizgres. The disk subsystem sped up on
> the same hardware (minus the HW RAID card) by over a factor of 10. The
> downside is that SW RAID is a pain in the neck for management - you have
> to shut down the Linux host when a disk fails to replace it.

Luke, you should not need to shut down the linux host when a disk fails.

you should be able to use mdadm to mark the drive as failed, then remove
it from the system and replace it, then use mdadm to add the drive to the
array.

I'm fighting through a double disk failure on my system at home and when I
hit a bad spot on a drive (failing it from the array) I can just re-add it
without having to restart everything (if it's the second drive I will have
to stop and restart the array, but that's becouse the entire array has
failed at that point)

now hot-swap may not be supported on all interface types, that may be what
you have run into, but with SCSI or SATA you should be able to hot-swap
with the right controller.

David Lang

David,

> now hot-swap may not be supported on all interface types, that may be what
> you have run into, but with SCSI or SATA you should be able to hot-swap
> with the right controller.

That's actually the problem - Linux hot swap is virtually non-functional for SCSI. You can write into the proper places in /proc, then remove and rescan to get a new drive up, but I've found that the resulting OS state is flaky. This is true of the latest 2.6 kernels and LSI and Adaptec SCSI controllers.

The problems I've seen are with Linux, not the controllers.

- Luke

Luke Lonergan wrote:
> Note that host-based SCSI raid cards from LSI, Adaptec, Intel, Dell, HP
> and others have proven to have worse performance than a single disk
> drive in many cases, whether for RAID0 or RAID5. In most circumstances

This is my own experience. Running a LSI MegaRAID in pure passthrough
mode + Linux software RAID10 is a ton faster than configuring the RAID
via the LSI card. One of the things I've noticed is that the card does
not seem to be able to parallel read on mirrors. While looking at iostat
under Linux, I can see software RAID1 reading all drives and the MD
number adding up to the sum of all drives.

The ARECA SATA controller I just got though doesn't seem to exhibit
these problems. Performance is a few % points above Linux software RAID
at lower CPU usage. In fact, I'm getting better single-threaded
bandwidth on a 4x7200RPM SATA config versus a 6x15K SCSI config on the
LSI. The drives are bigger for the SATA drive (300GB) versus 36GB for
the SCSI so that means the heads don't have to move any where as much
and can stay on the fast portion of the disk. Haven't had a chance to
test multi-user DB between the two setup though.

Luke Lonergan wrote:

>David,
>
>
>
>>now hot-swap may not be supported on all interface types, that may be what
>>you have run into, but with SCSI or SATA you should be able to hot-swap
>>with the right controller.
>>
>>
>
>That's actually the problem - Linux hot swap is virtually non-functional for SCSI. You can write into the proper places in /proc, then remove and rescan to get a new drive up, but I've found that the resulting OS state is flaky. This is true of the latest 2.6 kernels and LSI and Adaptec SCSI controllers.
>
>The problems I've seen are with Linux, not the controllers.
>
>
Interesting, I have had zero problems with Linux and SATA with LSI
controllers and hot plug. I wonder what the difference is. The LSI
controller even though SATA just uses the scsi driver.

Joshua D. Drake

>- Luke
>
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 2: Don't 'kill -9' the postmaster
>
>

--
The PostgreSQL Company - Command Prompt, Inc. 1.503.667.4564
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
Managed Services, Shared and Dedicated Hosting
Co-Authors: PLphp, PLperl - http://www.commandprompt.com/

On Sat, 24 Dec 2005, Luke Lonergan wrote:

> David,
>
>> now hot-swap may not be supported on all interface types, that may be what
>> you have run into, but with SCSI or SATA you should be able to hot-swap
>> with the right controller.
>
> That's actually the problem - Linux hot swap is virtually non-functional for SCSI. You can write into the proper places in /proc, then remove and rescan to get a new drive up, but I've found that the resulting OS state is flaky. This is true of the latest 2.6 kernels and LSI and Adaptec SCSI controllers.
>
> The problems I've seen are with Linux, not the controllers.

Thanks for the clarification, I knew that PATA didn't do hotswap, and I've
seen discussions on the linux-kernel list about SATA hotswap being worked
on, but I thought that scsi handled it. how recent a kernel have you had
problems with?

David Lang

On Sun, Dec 25, 2005 at 04:13:57AM -0800, David Lang wrote:
> Thanks for the clarification, I knew that PATA didn't do hotswap, and I've
> seen discussions on the linux-kernel list about SATA hotswap being worked
> on, but I thought that scsi handled it. how recent a kernel have you had
> problems with?

Is has largely worked for us, even though it's a bit hackish -- you _must_
disconnect the drive properly in the kernel before ejecting it physically,
though, or it will never reconnect. At least that's how it is with our
Adaptec 19160.

/* Steinar */
--
Homepage: http://www.sesse.net/

On Sat, Dec 24, 2005 at 22:13:43 -0500,
Luke Lonergan <LLonergan(at)greenplum(dot)com> wrote:
> David,
>
> > now hot-swap may not be supported on all interface types, that may be what
> > you have run into, but with SCSI or SATA you should be able to hot-swap
> > with the right controller.
>
> That's actually the problem - Linux hot swap is virtually non-functional for SCSI. You can write into the proper places in /proc, then remove and rescan to get a new drive up, but I've found that the resulting OS state is flaky. This is true of the latest 2.6 kernels and LSI and Adaptec SCSI controllers.
>
> The problems I've seen are with Linux, not the controllers.

The other option is to keep hot spares available so that you can have a failure
or two before you have to pull drives out. This might allow you to get to a
maintenance window to swap out the bad drives.

Have you done any benchmarking of the 9550SX against a software raid
configuration?

Luke Lonergan wrote:

>Frank,
>
>
>
>> You definitely DO NOT want to do RAID 5 on a database server. That
>> is probably the worst setup you could have, I've seen it have lower
>> performance than just a single hard disk.
>>
>>
>
>I've seen that on RAID0 and RAID10 as well.
>
>This is more about the quality and modernity of the RAID controller than
>anything else at this point, although there are some theoretical
>advantages of RAID10 from a random seek standpoint even if the adapter
>CPU is infinitely fast at checksumming. We're using RAID5 in practice
>for OLAP / Data Warehousing systems very successfully using the newest
>RAID cards from 3Ware (9550SX).
>
>Note that host-based SCSI raid cards from LSI, Adaptec, Intel, Dell, HP
>and others have proven to have worse performance than a single disk
>drive in many cases, whether for RAID0 or RAID5. In most circumstances
>I've seen, people don't even notice until they write a message to a
>mailing list about "my query runs slowly on xxx dbms". In many cases,
>after they run a simple sequential transfer rate test using dd, they see
>that their RAID controller is the culprit.
>
>Recently, I helped a company named DeepData to improve their dbms
>performance, which was a combination of moving them to software RAID50
>on Linux and getting them onto Bizgres. The disk subsystem sped up on
>the same hardware (minus the HW RAID card) by over a factor of 10. The
>downside is that SW RAID is a pain in the neck for management - you have
>to shut down the Linux host when a disk fails to replace it.
>
>- Luke
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 6: explain analyze is your friend
>
>

--
*Benjamin Arai*
barai(at)cs(dot)ucr(dot)edu <emailto:barai(at)cs(dot)ucr(dot)edu>
http://www.benjaminarai.com

Benjamin,

> Have you done any benchmarking of the 9550SX against a software raid configuration?

Interesting - no, not on SATA, mostly because I've had awful luck with Linux drivers and SATA. The popular manufacturers of SATA to PCI bridge chipsets are Silicon Image and Highpoint, and I've not seen Linux work with them at any reasonable performance yet. I've also had problems with Adaptec's cards - I think they manufacture their own SATA to PCI chipset as well. So far, I've only had good luck with the on-chipset Intel SATA implementation. I think the problems I've had could be entirely driver-related, but in the end it doesn't matter if you can't find drivers that work for Linux.

The other problem is getting enough SATA connections for the number of disks we want. I do have two new Areca SATA RAID cards and I'm going to benchmark those against the 3Ware 9550SX with 2 x 8 = 16 disks on one host.

I guess we could run the HW RAID controllers in JBOD mode to get a good driver / chipset configuration for software RAID, but frankly I prefer HW RAID if it performs well. So far the SATA host-based RAID is blowing the doors off of every other HW RAID solution I've tested.

- Luke

Have you have any experience rebuilding arrays in linux using the 3Ware
utilities? If so, did it work well?

Luke Lonergan wrote:

>Benjamin,
>
>
>
>>Have you done any benchmarking of the 9550SX against a software raid configuration?
>>
>>
>
>
>Interesting - no, not on SATA, mostly because I've had awful luck with Linux drivers and SATA. The popular manufacturers of SATA to PCI bridge chipsets are Silicon Image and Highpoint, and I've not seen Linux work with them at any reasonable performance yet. I've also had problems with Adaptec's cards - I think they manufacture their own SATA to PCI chipset as well. So far, I've only had good luck with the on-chipset Intel SATA implementation. I think the problems I've had could be entirely driver-related, but in the end it doesn't matter if you can't find drivers that work for Linux.
>
>The other problem is getting enough SATA connections for the number of disks we want. I do have two new Areca SATA RAID cards and I'm going to benchmark those against the 3Ware 9550SX with 2 x 8 = 16 disks on one host.
>
>I guess we could run the HW RAID controllers in JBOD mode to get a good driver / chipset configuration for software RAID, but frankly I prefer HW RAID if it performs well. So far the SATA host-based RAID is blowing the doors off of every other HW RAID solution I've tested.
>
>- Luke
>
>
>

--
*Benjamin Arai*
barai(at)cs(dot)ucr(dot)edu <emailto:barai(at)cs(dot)ucr(dot)edu>
http://www.benjaminarai.com

Benjamin,

On 12/26/05 10:21 AM, "Benjamin Arai" <barai(at)cs(dot)ucr(dot)edu> wrote:

> Have you have any experience rebuilding arrays in linux using the 3Ware
> utilities? If so, did it work well?

Sure we have - nowadays with disks failing as much as they do how could we
not? ;-)

3Ware has some *nice* tools - including a web browser utility for managing
the RAID. Rebuilds have been super easy - and the e-mail notification is
fine. They even have some decent migration options.

What they don't have are tools like snapshot backup, like EMC has, or SRDF
or any of the enterprise SAN features. We don't need them because Bizgres
MPP takes care of the need in software, but some people have become
accustomed to the features for other uses.

We're pretty happy with 3Ware, but their new 9550SX is, well, new. We
managed to find a good enough combination of driver and firmware to make it
work well on CentOs 4.1 and that's good enough for us, but there are
definitely some issues with some combinations now. Lastly, you do have to
set the block device readahead to 16MB to get performance.

- Luke

Yes - they work excellently. I have several medium and large servers
running 3ware 9500S series cards with great success. We have
rebuilding many failed RAID 10s over the course with no problems.

Alex

On 12/26/05, Benjamin Arai <barai(at)cs(dot)ucr(dot)edu> wrote:
> Have you have any experience rebuilding arrays in linux using the 3Ware
> utilities? If so, did it work well?
>
>
> Luke Lonergan wrote:
> Benjamin,
>
>
>
> Have you done any benchmarking of the 9550SX against a software raid
> configuration?
>
>
> Interesting - no, not on SATA, mostly because I've had awful luck with Linux
> drivers and SATA. The popular manufacturers of SATA to PCI bridge chipsets
> are Silicon Image and Highpoint, and I've not seen Linux work with them at
> any reasonable performance yet. I've also had problems with Adaptec's cards
> - I think they manufacture their own SATA to PCI chipset as well. So far,
> I've only had good luck with the on-chipset Intel SATA implementation. I
> think the problems I've had could be entirely driver-related, but in the end
> it doesn't matter if you can't find drivers that work for Linux.
>
> The other problem is getting enough SATA connections for the number of disks
> we want. I do have two new Areca SATA RAID cards and I'm going to benchmark
> those against the 3Ware 9550SX with 2 x 8 = 16 disks on one host.
>
> I guess we could run the HW RAID controllers in JBOD mode to get a good
> driver / chipset configuration for software RAID, but frankly I prefer HW
> RAID if it performs well. So far the SATA host-based RAID is blowing the
> doors off of every other HW RAID solution I've tested.
>
> - Luke
>
>
>
>
> --
> Benjamin Arai
> barai(at)cs(dot)ucr(dot)edu
> http://www.benjaminarai.com