High Performance/High Reliability File system on SuSE64

Hello All

Just wanted to gather opinions on what file system has the best balance between performance and reliability when used on a quad processor machine running SuSE64. Thanks

DAve

Dave Thompson wrote:

> Hello All
>
> Just wanted to gather opinions on what file system has the best
> balance between performance and reliability when used on a quad
> processor machine running SuSE64. Thanks

XFS.. hands down.

>
> DAve

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL

On Fri, Jan 23, 2004 at 08:51:03AM -0800, Joshua D. Drake wrote:
>
>
> XFS.. hands down.

I thought it was you who recently said you thought there was some
sort of possible caching problem there?

A

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
The plural of anecdote is not data.
--Roger Brinner

>>XFS.. hands down.
>>
>>
>
>I thought it was you who recently said you thought there was some
>sort of possible caching problem there?
>
>
>
Not I. We have had issues with JFS and data corruption on a powerout but
XFS has been rock solid in all of our tests.

XFS also has the interesting ability (although I have yet to test it)
that will allow you
to take a snapshot of the filesystem. Thus you can have filesystem level
backups
of the PGDATA directory that are consistent even though the database is
running.
There is nothing else on Linux that comes close to that. Plus XFS has been
proven in a 64 bit environment (Irix).

Sincerely,

Joshua D. Drake

>A
>
>
>

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

Anyone have good/bad experience using OpenGFS or the Sistina GFS?
Any other cluster file systems out there?

Greg

--
Greg Spiegelberg
Sr. Product Development Engineer
Cranel, Incorporated.
Phone: 614.318.4314
Fax: 614.431.8388
Email: gspiegelberg(at)Cranel(dot)com
Cranel. Technology. Integrity. Focus.

**********************************************************************
This email and any files transmitted with it are confidential and
intended solely for the use of the individual or entity to whom they
are addressed. If you have received this email in error please notify
the system manager.

This footnote also confirms that this email message has been swept by
MIMEsweeper for the presence of computer viruses.

www.mimesweeper.com
**********************************************************************

On Fri, Jan 23, 2004 at 10:18:35AM -0800, Joshua D. Drake wrote:
> >
> Not I. We have had issues with JFS and data corruption on a powerout but
> XFS has been rock solid in all of our tests.

Sorry, it was Josh Berkus:

http://archives.postgresql.org/pgsql-performance/2004-01/msg00086.php

> There is nothing else on Linux that comes close to that. Plus XFS has been
> proven in a 64 bit environment (Irix).

I had lots of happy experiences with XFS when administering IRIX
boxes[1], but I don't know what differences the Linux port entailed.
Do you have details on that? We're certainly looking for an option
over JFS at the moment.

A

[1] I will note, however, that it was practically the only happy
experience I had with them. IRIX made the early Debian installer
look positively user-friendly, and SGI's desire to make everything
whiz-bang nifty by running practically every binary setuid root gave
me fits. But XFS was nice.

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
In the future this spectacle of the middle classes shocking the avant-
garde will probably become the textbook definition of Postmodernism.
--Brad Holland

>>There is nothing else on Linux that comes close to that. Plus XFS has been
>>proven in a 64 bit environment (Irix).
>>
>>
>
>I had lots of happy experiences with XFS when administering IRIX
>boxes[1], but I don't know what differences the Linux port entailed.
>Do you have details on that?
>

http://oss.sgi.com/projects/xfs/

>We're certainly looking for an option
>over JFS at the moment.
>
>A
>
>[1] I will note, however, that it was practically the only happy
>experience I had with them. IRIX made the early Debian installer
>look positively user-friendly, and SGI's desire to make everything
>whiz-bang nifty by running practically every binary setuid root gave
>me fits. But XFS was nice.
>
>
>

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

In article <40116DA1(dot)7030209(at)cranel(dot)com>, Greg Spiegelberg wrote:
> Anyone have good/bad experience using OpenGFS or the Sistina GFS?
> Any other cluster file systems out there?

Lustre and CFS spring to mind. CFS is the cluster filesystem layer
of openssi (http://www.openssi.org), which is layered on top of
ext3 and provides the necessary locking. It can work with shared
disk hardware, but I believe 1 node is always responsible for all
transactions. It can fail-over to another node in the cluster, if
that node has direct access to the disk.

You can also use it without shared disk hardware (you loose the HA
ofcourse). That way you can easily test openssi.

Postgresql should work without problems on an OpenSSI cluster.

I don't know much about Lustre (http://www.clusterfs.com)

On Fri, Jan 23, 2004 at 11:05:41AM -0800, Joshua D. Drake wrote:
> http://oss.sgi.com/projects/xfs/

Yes, I guess I shoulda thought of that, eh? Thanks. The docs do
suggest that there are some significant differences between the two
versions of the filesystem, so I'm not sure how sanguine I'd be about
the degree of "testing" the filesystem has received on Linux. On the
other hand, I wouldn't be surprised if it were no worse than the
other options.

A

--
Andrew Sullivan | ajs(at)crankycanuck(dot)ca
The fact that technology doesn't work is no bar to success in the marketplace.
--Philip Greenspun

>Yes, I guess I shoulda thought of that, eh? Thanks. The docs do
>suggest that there are some significant differences between the two
>versions of the filesystem, so I'm not sure how sanguine I'd be about
>the degree of "testing" the filesystem has received on Linux. On the
>
>
Well SuSE ships with XFS and SuSE tends to be really good about testing.
Better than RedHat IMHO. Just the fact that RedHat uses ext3 as the default
is a black eye.

XFS has been around a LONG time, and on Linux for a couple of years now.
Plus I believe it is the default FS for all of the really high end stuff
SGI is doing
with Linux.

I would (and do) trust XFS currently over ANY other journalled option on
Linux.

Sincerely,

Joshua D. Drake

>other hand, I wouldn't be surprised if it were no worse than the
>other options.
>
>A
>
>
>

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

jd(at)commandprompt(dot)com ("Joshua D. Drake") writes:
>>Yes, I guess I shoulda thought of that, eh? Thanks. The docs do
>>suggest that there are some significant differences between the two
>>versions of the filesystem, so I'm not sure how sanguine I'd be about
>>the degree of "testing" the filesystem has received on Linux. On the
>>
> Well SuSE ships with XFS and SuSE tends to be really good about
> testing. Better than RedHat IMHO. Just the fact that RedHat uses
> ext3 as the default is a black eye.

Well, I'd point to one major factor with RHAT; they employ Stephen
Tweedie, creator of ext3, and have been paying him to work on it for
some time now. If they _didn't_ promote use of ext3, they would be
very much vulnerable to the "won't eat their own dogfood" criticism.

> XFS has been around a LONG time, and on Linux for a couple of years
> now. Plus I believe it is the default FS for all of the really high
> end stuff SGI is doing with Linux.

Ah, but there is a bit of a 'problem' nonetheless; XFS is not
'officially supported' as part of the Linux kernel until version 2.6,
which is still pretty "bleeding edge." Until 2.6 solidifies a bit
more (aside: based on experiences with 2.6.0, "quite a lot more"), it
is a "patchy" add-on to the 'stable' 2.4 kernel series.

Do the patches work? As far as I have heard, quite well indeed. But
the fact of it not having been 'official' is a fair little bit of a
downside.

> I would (and do) trust XFS currently over ANY other journalled
> option on Linux.

I'm getting less and less inclined to trust ext3 or JFS, which "floats
upwards" any other boats that are lingering around...
--
let name="cbbrowne" and tld="libertyrms.info" in String.concat "@" [name;tld];;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

>Well, I'd point to one major factor with RHAT; they employ Stephen
>Tweedie, creator of ext3, and have been paying him to work on it for
>some time now. If they _didn't_ promote use of ext3, they would be
>very much vulnerable to the "won't eat their own dogfood" criticism.
>
>
>
True but frankly, they shouldn't. EXT3 has some serious issues. In fact
if you are running a stock RH kernel before 2.4.20 you can destroy your
PostgreSQL database with it.

Not to mention how slow it is ;)

>>XFS has been around a LONG time, and on Linux for a couple of years
>>now. Plus I believe it is the default FS for all of the really high
>>end stuff SGI is doing with Linux.
>>
>>
>
>Ah, but there is a bit of a 'problem' nonetheless; XFS is not
>'officially supported' as part of the Linux kernel until version 2.6,
>which is still pretty "bleeding edge."
>
That is not true see:

http://kerneltrap.org/node/view/1751

>Until 2.6 solidifies a bit
>more (aside: based on experiences with 2.6.0, "quite a lot more"), it
>is a "patchy" add-on to the 'stable' 2.4 kernel series.
>
>
>
Again see above :)

>Do the patches work? As far as I have heard, quite well indeed. But
>the fact of it not having been 'official' is a fair little bit of a
>downside.
>
>

What is official?

Sincerely,

Joshua D. Drake

>>I would (and do) trust XFS currently over ANY other journalled
>>option on Linux.
>>
>>
>
>I'm getting less and less inclined to trust ext3 or JFS, which "floats
>upwards" any other boats that are lingering around...
>
>

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

jd(at)commandprompt(dot)com ("Joshua D. Drake") writes:
>>Well, I'd point to one major factor with RHAT; they employ Stephen
>>Tweedie, creator of ext3, and have been paying him to work on it for
>>some time now. If they _didn't_ promote use of ext3, they would be
>>very much vulnerable to the "won't eat their own dogfood" criticism.
>
> True but frankly, they shouldn't. EXT3 has some serious issues. In fact
> if you are running a stock RH kernel before 2.4.20 you can destroy your
> PostgreSQL database with it.
>
> Not to mention how slow it is ;)

I'm not defending ext3's merits; just the clear reason why RHAT uses
it :-).

>>>XFS has been around a LONG time, and on Linux for a couple of years
>>>now. Plus I believe it is the default FS for all of the really high
>>>end stuff SGI is doing with Linux.
>>>
>>>
>>
>>Ah, but there is a bit of a 'problem' nonetheless; XFS is not
>>'officially supported' as part of the Linux kernel until version 2.6,
>> which is still pretty "bleeding edge."
>>
> That is not true see:
>
> http://kerneltrap.org/node/view/1751

Well, I just downloaded 2.4.24 this week, and I don't see XFS included
in it. I see ReiserFS, ext3, and JFS, but not XFS.

>>Until 2.6 solidifies a bit more (aside: based on experiences with
>>2.6.0, "quite a lot more"), it is a "patchy" add-on to the 'stable'
>>2.4 kernel series.
>>
> Again see above :)

>>Do the patches work? As far as I have heard, quite well indeed. But
>>the fact of it not having been 'official' is a fair little bit of a
>>downside.
>
> What is official?

"Is it included in the kernel sources hosted at ftp.kernel.org?"
--
let name="cbbrowne" and tld="libertyrms.info" in name ^ "@" ^ tld;;
<http://dev6.int.libertyrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

On 01/23/2004-10:18AM, Joshua D. Drake wrote:
>
> XFS also has the interesting ability (although I have yet to test it)
> that will allow you
> to take a snapshot of the filesystem. Thus you can have filesystem level
> backups
> of the PGDATA directory that are consistent even though the database is
> running.

You can do snapshots in FreeBSD 5.x with UFS2 as well but that (
nor XFS snapshots ) will let you backup with the database server
running. Just because you will get the file exactly as it was at
a particular instant does not mean that the postmaster did not
still have some some data that was not flushed to disk yet.

>You can do snapshots in FreeBSD 5.x with UFS2 as well but that (
>nor XFS snapshots ) will let you backup with the database server
>running. Just because you will get the file exactly as it was at
>a particular instant does not mean that the postmaster did not
>still have some some data that was not flushed to disk yet.
>
>
Ahh... isn't that what fsync is for?

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
Mammoth PostgreSQL Replicator. Integrated Replication for PostgreSQL

They seem pretty clean (have patched vanilla kernels + xfs for Mandrake
9.2/9.0).

And yes, I would recommend xfs - noticeably faster than ext3, and no
sign of any mysterious hangs under load.

best wishes

Mark

Christopher Browne wrote:

>Do the patches work? As far as I have heard, quite well indeed. But
>the fact of it not having been 'official' is a fair little bit of a
>downside.
>
>
>

Mark Kirkwood wrote:

> They seem pretty clean (have patched vanilla kernels + xfs for
> Mandrake 9.2/9.0).
>
> And yes, I would recommend xfs - noticeably faster than ext3, and no
> sign of any mysterious hangs under load.

The hangs you are having are due to several issues... one of them is the
way ext3 syncs. What kernel version are
you running?

Sincerely,

Joshua Drake

>
> best wishes
>
> Mark
>
> Christopher Browne wrote:
>
>> Do the patches work? As far as I have heard, quite well indeed. But
>> the fact of it not having been 'official' is a fair little bit of a
>> downside.
>>
>>
>>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org

--
Command Prompt, Inc., home of Mammoth PostgreSQL - S/ODBC and S/JDBC
Postgresql support, programming shared hosting and dedicated hosting.
+1-503-667-4564 - jd(at)commandprompt(dot)com - http://www.commandprompt.com
PostgreSQL Replicator -- production quality replication for PostgreSQL

jd(at)commandprompt(dot)com says...

> XFS.. hands down.

Off topic question here, but I'm a bit at a loss to understand exactly
what sgi are doing.

I thought that they were removing IRIX and going with Linux as the OS to
their high end graphical workstations, yet I see they still have IRIX on
their site.

What, exactly, is the story with this?

Paul...

--
plinehan y_a_h_o_o and d_o_t com
C++ Builder 5 SP1, Interbase 6.0.1.6 IBX 5.04 W2K Pro
Please do not top-post.

"XML avoids the fundamental question of what we should do,
by focusing entirely on how we should do it."

quote from http://www.metatorial.com

Paul Ganainm <paulsnewsgroups(at)hotmail(dot)com> writes:

> jd(at)commandprompt(dot)com says...
>
>
>> XFS.. hands down.
>
>
> Off topic question here, but I'm a bit at a loss to understand exactly
> what sgi are doing.
>
>
> I thought that they were removing IRIX and going with Linux as the OS to
> their high end graphical workstations, yet I see they still have IRIX on
> their site.
>
>
> What, exactly, is the story with this?

Well, they do have a very large installed base of Irix systems, and
a certain obligation to keep supporting them... As long as there are
people whi still want MIPS/Irix they will probably keep selling at
least a few models.

-Doug

> Mark Kirkwood wrote a little unclearly:
>
>>
>> And yes, I would recommend xfs - noticeably faster than ext3, and no
>> sign of any mysterious hangs under load.
>
>
I was thinking about the reported mini-hangs that folks are seeing with
jfs, except the all important keyword "jfs" didnt make it out of my head
and into the 2nd bit of the message :-( Sorry for the confusion.

So I meant "faster that ext3, and without strange mini hangs like jfs...".

Joshua wrote:

>
> The hangs you are having are due to several issues... one of them is
> the way ext3 syncs.

Never suffered this personally, it was an unexpected filesystem
corruption with ext3 that "encouraged" me to try out xfs instead.

best wishes

Mark

Christopher Browne <cbbrowne(at)libertyrms(dot)info> writes:

> Ah, but there is a bit of a 'problem' nonetheless; XFS is not
> 'officially supported' as part of the Linux kernel until version 2.6,
> which is still pretty "bleeding edge." Until 2.6 solidifies a bit
> more (aside: based on experiences with 2.6.0, "quite a lot more"), it
> is a "patchy" add-on to the 'stable' 2.4 kernel series.

Actually I think XFS is in 2.4.25.

--
greg

On 23 Jan, Dave Thompson wrote:
> Hello All
>
> Just wanted to gather opinions on what file system has the best balance between performance and reliability when used on a quad processor machine running SuSE64. Thanks
>
> DAve

Hi Dave,

I have some data for performance using our DBT-2 workload (OLTP type
transactions) with Linux-2.6 with various filesystems and i/o
schedulers. I know it doesn't address the reliability part of your
question, and it's in a completely different environment (32-bit as well
as a different distro), but if you think you'll find the results
interesting:

http://developer.osdl.org/markw/fs/project_results.html

Mark

Christopher Weimann <cweimann(at)k12hq(dot)com> writes:
> You can do snapshots in FreeBSD 5.x with UFS2 as well but that (
> nor XFS snapshots ) will let you backup with the database server
> running. Just because you will get the file exactly as it was at
> a particular instant does not mean that the postmaster did not
> still have some some data that was not flushed to disk yet.

It *will* work, if you have an instantaneous filesystem snapshot
covering the entire $PGDATA directory tree (both data files and WAL).
Restarting the postmaster on the backup will result in a WAL replay
sequence, and at the end the data files will be consistent. If this
were not so, we'd not be crash-proof. The instantaneous snapshot
is exactly equivalent to the on-disk state at the moment of a kernel
crash or power failure, no?

regards, tom lane

Christopher Weimann wrote:
> On 01/23/2004-10:18AM, Joshua D. Drake wrote:
> >
> > XFS also has the interesting ability (although I have yet to test it)
> > that will allow you
> > to take a snapshot of the filesystem. Thus you can have filesystem level
> > backups
> > of the PGDATA directory that are consistent even though the database is
> > running.
>
> You can do snapshots in FreeBSD 5.x with UFS2 as well but that (
> nor XFS snapshots ) will let you backup with the database server
> running. Just because you will get the file exactly as it was at
> a particular instant does not mean that the postmaster did not
> still have some some data that was not flushed to disk yet.

Uh, yea, it does. If the snapshot includes all of /data, including
WAL/xlog, you can then back up the snapshot and restore it on another
machine. It will restart just like a crash.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073