Re: Performance with 2 AMD/Opteron 2.6Ghz and 8gig

Luke,

Yeah, I read those results, and I'm very disappointed with my results
from the MSA1500. I would however be interested in other people's
bonnie++ and benchmarksql results using a similar machine (2 cpu dual
core opteron) with other "off the shelf" storage systems
(EMC/Netapp/Xyratex/../). Could you run benchmarksql against that
machine with the 16 SATA disk and 3Ware 9550SX SATA RAID adapters? It
would be *very* interesting to see how the I/O performance correlates to
benchmarksql (postgres) transaction throughout.

/Mikael

-----Original Message-----
From: Luke Lonergan [mailto:LLonergan(at)greenplum(dot)com]
Sent: den 28 juli 2006 11:17
To: Mikael Carneholm; Kjell Tore Fossbakk;
pgsql-performance(at)postgresql(dot)org
Subject: RE: [PERFORM] Performance with 2 AMD/Opteron 2.6Ghz and 8gig

Mikael,

> -----Original Message-----
> From: Mikael Carneholm [mailto:Mikael(dot)Carneholm(at)WirelessCar(dot)com]
> Sent: Friday, July 28, 2006 2:05 AM
>
> My bonnie++ results are found in this message:
> http://archives.postgresql.org/pgsql-performance/2006-07/msg00164.php
>

Apologies if I've already said this, but those bonnie++ results are very
disappointing. The sequential transfer rates between 20MB/s and 57MB/s
are slower than a single SATA disk, and your SCSI disks might even do
80MB/s sequential transfer rate each.

Random access is also very poor, though perhaps equal to 5 disk drives
at 500/second.

By comparison, we routinely get 950MB/s sequential transfer rate using
16 SATA disks and 3Ware 9550SX SATA RAID adapters on Linux.

On Solaris ZFS on an X4500, we recently got this bonnie++ result on 36
SATA disk drives in RAID10 (single thread first):

Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr-
--Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
thumperdw-i-1 32G 120453 99 467814 98 290391 58 109371 99 993344
94 1801 4
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 +++++ +++ +++++ +++ +++++ +++ 30850 99 +++++ +++
+++++ +++

Bumping up the number of concurrent processes to 2, we get about 1.5x
speed reads of RAID10 with a concurrent workload (you have to add the
rates together):

Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
--Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
thumperdw-i-1 32G 111441 95 212536 54 171798 51 106184 98 719472
88 1233 2
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 26085 90 +++++ +++ 5700 98 21448 97 +++++ +++
4381 97

Version 1.03 ------Sequential Output------ --Sequential Input-
--Random-
-Per Chr- --Block-- -Rewrite- -Per Chr-
--Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
%CP /sec %CP
thumperdw-i-1 32G 116355 99 212509 54 171647 50 106112 98 715030
87 1274 3
------Sequential Create------ --------Random
Create--------
-Create-- --Read--- -Delete-- -Create-- --Read---
-Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
/sec %CP
16 26082 99 +++++ +++ 5588 98 21399 88 +++++ +++
4272 97

So that's 2500 seeks per second, 1440MB/s sequential block read, 212MB/s
per character sequential read.

- Luke

Hello.

Unfortunately, I'm leaving for my vacation now, gone 3 weeks. When I'm back
I'll run benchmarksql and bonnie++ and give the results here.

The spec I will be using:

Prolite DL585
2 x AMD/Opteron 64-bit 2,6GHZ
8G DDR PC3200
4 x 150G SCSI in SmartArray 5i
Running Gentoo 2006.0 AMD_64 Hardened kernel

Then I will remove the SmartArray 5i, and use a simple nonRAID SCSI
controller and implement Linux software RAID, and re-run the tests.

I'll give signal in 3 weeks

- Kjell Tore.

On 7/28/06, Mikael Carneholm <Mikael(dot)Carneholm(at)wirelesscar(dot)com> wrote:
>
> Luke,
>
> Yeah, I read those results, and I'm very disappointed with my results
> from the MSA1500. I would however be interested in other people's
> bonnie++ and benchmarksql results using a similar machine (2 cpu dual
> core opteron) with other "off the shelf" storage systems
> (EMC/Netapp/Xyratex/../). Could you run benchmarksql against that
> machine with the 16 SATA disk and 3Ware 9550SX SATA RAID adapters? It
> would be *very* interesting to see how the I/O performance correlates to
> benchmarksql (postgres) transaction throughout.
>
> /Mikael
>
> -----Original Message-----
> From: Luke Lonergan [mailto:LLonergan(at)greenplum(dot)com]
> Sent: den 28 juli 2006 11:17
> To: Mikael Carneholm; Kjell Tore Fossbakk;
> pgsql-performance(at)postgresql(dot)org
> Subject: RE: [PERFORM] Performance with 2 AMD/Opteron 2.6Ghz and 8gig
>
> Mikael,
>
> > -----Original Message-----
> > From: Mikael Carneholm [mailto:Mikael(dot)Carneholm(at)WirelessCar(dot)com]
> > Sent: Friday, July 28, 2006 2:05 AM
> >
> > My bonnie++ results are found in this message:
> > http://archives.postgresql.org/pgsql-performance/2006-07/msg00164.php
> >
>
> Apologies if I've already said this, but those bonnie++ results are very
> disappointing. The sequential transfer rates between 20MB/s and 57MB/s
> are slower than a single SATA disk, and your SCSI disks might even do
> 80MB/s sequential transfer rate each.
>
> Random access is also very poor, though perhaps equal to 5 disk drives
> at 500/second.
>
> By comparison, we routinely get 950MB/s sequential transfer rate using
> 16 SATA disks and 3Ware 9550SX SATA RAID adapters on Linux.
>
> On Solaris ZFS on an X4500, we recently got this bonnie++ result on 36
> SATA disk drives in RAID10 (single thread first):
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 120453 99 467814 98 290391 58 109371 99 993344
> 94 1801 4
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 +++++ +++ +++++ +++ +++++ +++ 30850 99 +++++ +++
> +++++ +++
>
> Bumping up the number of concurrent processes to 2, we get about 1.5x
> speed reads of RAID10 with a concurrent workload (you have to add the
> rates together):
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 111441 95 212536 54 171798 51 106184 98 719472
> 88 1233 2
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 26085 90 +++++ +++ 5700 98 21448 97 +++++ +++
> 4381 97
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 116355 99 212509 54 171647 50 106112 98 715030
> 87 1274 3
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 26082 99 +++++ +++ 5588 98 21399 88 +++++ +++
> 4272 97
>
> So that's 2500 seeks per second, 1440MB/s sequential block read, 212MB/s
> per character sequential read.
>
> - Luke
>
>
>

--
"Be nice to people on your way up because you meet them on your way down."

On Fri, 28 Jul 2006, Mikael Carneholm wrote:

> Luke,
>
> Yeah, I read those results, and I'm very disappointed with my results
> from the MSA1500. I would however be interested in other people's
> bonnie++ and benchmarksql results using a similar machine (2 cpu dual
> core opteron) with other "off the shelf" storage systems
> (EMC/Netapp/Xyratex/../). Could you run benchmarksql against that
> machine with the 16 SATA disk and 3Ware 9550SX SATA RAID adapters? It
> would be *very* interesting to see how the I/O performance correlates to
> benchmarksql (postgres) transaction throughout.

FWIW, once our vendor gets all the pieces (having some issues with
figuring out which multilane sata cables to get), I'll have a dual-core
opteron box with a 3Ware 9500SX-12MI and 8 drives. I need to benchmark to
compare this to our xeon/adaptec/scsi build we've been using.

I've also got a 1U with a 9500SX-4 and 4 drives. I like how the 3Ware
card scales there - started with 2 drives and got "drive speed" mirroring.
Added two more and most of the bonnie numbers doubled. This is not what
I'm used to with the Adaptec SCSI junk.

These SATA RAID controllers 3Ware is making seem to be leaps and bounds
beyond what the "old guard" is churning out (at much higher prices).

Charles

> /Mikael
>
> -----Original Message-----
> From: Luke Lonergan [mailto:LLonergan(at)greenplum(dot)com]
> Sent: den 28 juli 2006 11:17
> To: Mikael Carneholm; Kjell Tore Fossbakk;
> pgsql-performance(at)postgresql(dot)org
> Subject: RE: [PERFORM] Performance with 2 AMD/Opteron 2.6Ghz and 8gig
>
> Mikael,
>
>> -----Original Message-----
>> From: Mikael Carneholm [mailto:Mikael(dot)Carneholm(at)WirelessCar(dot)com]
>> Sent: Friday, July 28, 2006 2:05 AM
>>
>> My bonnie++ results are found in this message:
>> http://archives.postgresql.org/pgsql-performance/2006-07/msg00164.php
>>
>
> Apologies if I've already said this, but those bonnie++ results are very
> disappointing. The sequential transfer rates between 20MB/s and 57MB/s
> are slower than a single SATA disk, and your SCSI disks might even do
> 80MB/s sequential transfer rate each.
>
> Random access is also very poor, though perhaps equal to 5 disk drives
> at 500/second.
>
> By comparison, we routinely get 950MB/s sequential transfer rate using
> 16 SATA disks and 3Ware 9550SX SATA RAID adapters on Linux.
>
> On Solaris ZFS on an X4500, we recently got this bonnie++ result on 36
> SATA disk drives in RAID10 (single thread first):
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 120453 99 467814 98 290391 58 109371 99 993344
> 94 1801 4
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 +++++ +++ +++++ +++ +++++ +++ 30850 99 +++++ +++
> +++++ +++
>
> Bumping up the number of concurrent processes to 2, we get about 1.5x
> speed reads of RAID10 with a concurrent workload (you have to add the
> rates together):
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr- --Block--
> --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 111441 95 212536 54 171798 51 106184 98 719472
> 88 1233 2
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 26085 90 +++++ +++ 5700 98 21448 97 +++++ +++
> 4381 97
>
> Version 1.03 ------Sequential Output------ --Sequential Input-
> --Random-
> -Per Chr- --Block-- -Rewrite- -Per Chr-
> --Block-- --Seeks--
> Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec
> %CP /sec %CP
> thumperdw-i-1 32G 116355 99 212509 54 171647 50 106112 98 715030
> 87 1274 3
> ------Sequential Create------ --------Random
> Create--------
> -Create-- --Read--- -Delete-- -Create-- --Read---
> -Delete--
> files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
> /sec %CP
> 16 26082 99 +++++ +++ 5588 98 21399 88 +++++ +++
> 4272 97
>
> So that's 2500 seeks per second, 1440MB/s sequential block read, 212MB/s
> per character sequential read.
>
> - Luke
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

Hi, Charles,

Charles Sprickman wrote:

> I've also got a 1U with a 9500SX-4 and 4 drives. I like how the 3Ware
> card scales there - started with 2 drives and got "drive speed"
> mirroring. Added two more and most of the bonnie numbers doubled. This
> is not what I'm used to with the Adaptec SCSI junk.

Well, for sequential reading, you should be able to get double drive
speed on a 2-disk mirror with a good controller, as it can balance the
reads among the drives.

Markus
--
Markus Schaber | Logical Tracking&Tracing International AG
Dipl. Inf. | Software Development GIS

Fight against software patents in EU! www.ffii.org www.nosoftwarepatents.org

Although I for one have yet to see a controller that actualy does this (I
believe software RAID on linux doesn't either).

Alex.

On 8/7/06, Markus Schaber <schabi(at)logix-tt(dot)com> wrote:
>
> Hi, Charles,
>
> Charles Sprickman wrote:
>
> > I've also got a 1U with a 9500SX-4 and 4 drives. I like how the 3Ware
> > card scales there - started with 2 drives and got "drive speed"
> > mirroring. Added two more and most of the bonnie numbers doubled. This
> > is not what I'm used to with the Adaptec SCSI junk.
>
> Well, for sequential reading, you should be able to get double drive
> speed on a 2-disk mirror with a good controller, as it can balance the
> reads among the drives.
>
> Markus
> --
> Markus Schaber | Logical Tracking&Tracing International AG
> Dipl. Inf. | Software Development GIS
>
> Fight against software patents in EU! www.ffii.org
> www.nosoftwarepatents.org
>
> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
> choose an index scan if your joining column's datatypes do not
> match
>

On Mon, Aug 07, 2006 at 04:02:52PM -0400, Alex Turner wrote:
> Although I for one have yet to see a controller that actualy does this (I
> believe software RAID on linux doesn't either).

Linux' software RAID does. See earlier threads for demonstrations.

/* Steinar */
--
Homepage: http://www.sesse.net/

On Mon, Aug 07, 2006 at 10:20:02PM +0200, Steinar H. Gunderson wrote:
> On Mon, Aug 07, 2006 at 04:02:52PM -0400, Alex Turner wrote:
> > Although I for one have yet to see a controller that actualy does this (I
> > believe software RAID on linux doesn't either).
>
> Linux' software RAID does. See earlier threads for demonstrations.

The real question: will it balance within a single thread?

Cheaper raid setups will balance individual requests between devices,
but good ones should be able to service a single request from both
devices (assuming it's reading more than whatever the stripe size is).
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461