Re: Best OS for Postgres 8.2

Hi,

I am about to order a new server for my Postgres cluster. I will
probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
Which OS would you recommend to optimize Postgres behaviour (i/o
access, multithreading, etc) ?

I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
help with this ?

Regards

David Levy wrote:
> Hi,
>
> I am about to order a new server for my Postgres cluster. I will
> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
> Which OS would you recommend to optimize Postgres behaviour (i/o
> access, multithreading, etc) ?
>
> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> help with this ?

Well you just described three linux distributions, which is hardly a
question about which OS to use ;). I would stick with the long supported
versions of Linux, thus CentOS 5, Debian 4, Ubuntu Dapper.

Joshua D. Drake

>
>
> Regards
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org
>

In response to "Joshua D. Drake" <jd(at)commandprompt(dot)com>:

> David Levy wrote:
> > Hi,
> >
> > I am about to order a new server for my Postgres cluster. I will
> > probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
> > Which OS would you recommend to optimize Postgres behaviour (i/o
> > access, multithreading, etc) ?
> >
> > I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> > help with this ?
>
> Well you just described three linux distributions, which is hardly a
> question about which OS to use ;). I would stick with the long supported
> versions of Linux, thus CentOS 5, Debian 4, Ubuntu Dapper.

There used to be a prominent site that recommended FreeBSD for Postgres.
Don't know if that's still recommended or not -- but bringing it up is
likely to start a Holy War.

--
Bill Moran
Collaborative Fusion Inc.
http://people.collaborativefusion.com/~wmoran/

wmoran(at)collaborativefusion(dot)com
Phone: 412-422-3463x4023

Bill Moran wrote:
> In response to "Joshua D. Drake" <jd(at)commandprompt(dot)com>:
>
>> David Levy wrote:
>>> Hi,
>>>
>>> I am about to order a new server for my Postgres cluster. I will
>>> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>>> Which OS would you recommend to optimize Postgres behaviour (i/o
>>> access, multithreading, etc) ?
>>>
>>> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>>> help with this ?
>> Well you just described three linux distributions, which is hardly a
>> question about which OS to use ;). I would stick with the long supported
>> versions of Linux, thus CentOS 5, Debian 4, Ubuntu Dapper.
>
> There used to be a prominent site that recommended FreeBSD for Postgres.
> Don't know if that's still recommended or not -- but bringing it up is
> likely to start a Holy War.

Heh... I doubt it will start a war. FreeBSD is a good OS. However, I
specifically noted the Dual Xeon Quad Core, which means, 8 procs. It is
my understanding (and I certainly could be wrong) that FreeBSD doesn't
handle SMP nearly as well as Linux (and Linux not as well as Solaris).

Sincerely,

Joshua D. Drake

>

On May 7, 2007, at 2:55 PM, David Levy wrote:

> Hi,
>
> I am about to order a new server for my Postgres cluster. I will
> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
> Which OS would you recommend to optimize Postgres behaviour (i/o
> access, multithreading, etc) ?
>
> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> help with this ?

Well, all three you mention are much the same, just with a different
badge on the box, as far as performance is concerned. They're all
going to be a moderately recent Linux kernel, with your choice
of filesystems, so any choice between them is going to be driven
more by available staff and support or personal preference.

I'd probably go CentOS 5 over Fedora just because Fedora doesn't
get supported for very long - more of an issue with a dedicated
database box with a long lifespan than your typical desktop or
interchangeable webserver.

I might also look at Solaris 10, though. I've yet to play with it
much, but it
seems nice, and I suspect it might manage 8 cores better than current
Linux setups.

Cheers,
Steve

David Levy wrote:
> Hi,
>
> I am about to order a new server for my Postgres cluster. I will
> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
> Which OS would you recommend to optimize Postgres behaviour (i/o
> access, multithreading, etc) ?
>
> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> help with this ?

Use the one you're most comfortable with.

I don't think you'll notice *that* much difference between linux systems
for performance - but whether you're comfortable using any of them will
make a difference in managing it in general.

--
Postgresql & php tutorials
http://www.designmagick.com/

On Tue, 8 May 2007, Chris wrote:

> David Levy wrote:
>> Hi,
>>
>> I am about to order a new server for my Postgres cluster. I will
>> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>> Which OS would you recommend to optimize Postgres behaviour (i/o
>> access, multithreading, etc) ?
>>
>> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>> help with this ?
>
> Use the one you're most comfortable with.
>
> I don't think you'll notice *that* much difference between linux systems for
> performance - but whether you're comfortable using any of them will make a
> difference in managing it in general.

the tuneing that you do (both of the OS and of postgres) will make more
of a difference then anything else.

David Lang

david(at)lang(dot)hm wrote:
> On Tue, 8 May 2007, Chris wrote:
>
>> David Levy wrote:
>>> Hi,
>>>
>>> I am about to order a new server for my Postgres cluster. I will
>>> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>>> Which OS would you recommend to optimize Postgres behaviour (i/o
>>> access, multithreading, etc) ?
>>>
>>> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>>> help with this ?
>>
>> Use the one you're most comfortable with.
>>
>> I don't think you'll notice *that* much difference between linux
>> systems for performance - but whether you're comfortable using any of
>> them will make a difference in managing it in general.
>
> the tuneing that you do (both of the OS and of postgres) will make more
> of a difference then anything else.

Which is why it's best to know/understand the OS first ;)

--
Postgresql & php tutorials
http://www.designmagick.com/

I am using FC6 in production for our pg 8.2.4 DB server and am quite
happy with it.

The big advantage with FC6 for me was that the FC6 team seems to keep
more current with the latest stable revs of most OSSW (including
kernel revs!) better than any of the other major distros.

(Also, SE Linux is a =good= thing security-wise. If it's good enough
for the NSA...)

Downside is that initial install and config can be a bit complicated.

We're happy with it.

Cheers,
Ron Peacetree

At 05:55 PM 5/7/2007, David Levy wrote:
>Hi,
>
>I am about to order a new server for my Postgres cluster. I will
>probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>Which OS would you recommend to optimize Postgres behaviour (i/o
>access, multithreading, etc) ?
>
>I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>help with this ?
>
>
>Regards
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 4: Have you searched our list archives?
>
> http://archives.postgresql.org

On Mon, 7 May 2007, David Levy wrote:

> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> help with this ?

Debian packages PostgreSQL in a fashion unique to it; it's arguable
whether it's better or not (I don't like it), but going with that will
assure your installation is a bit non-standard compared with most Linux
installas. The main reasons you'd pick Debian are either that you like
that scheme (which tries to provide some structure to running multiple
clusters on one box), or that you plan to rely heavily on community
packages that don't come with the Redhat distributions and therefore would
appreciate how easy it is to use apt-get against the large Debian software
repository.

Given the buginess and unexpected changes from packages updates of every
Fedora Core release I've ever tried, I wouldn't trust any OS from that
line to run a database keeping track of where my socks are at. Core 6
seems better than most of the older ones. I find it hard to understand
what it offers that Centos doesn't such that you'd want Fedora instead.

Centos just released a new version 5 recently. It's running a fairly
modern kernel with several relevant performance improvements over the much
older V4; unless you have some odd piece of hardware where there is only a
driver available for Centos 4 (I ran into this with a disk controller),
the new version would better.

The main advantages of Centos over the other two are that so many people
are/will be running very similar configurations that you should able to
find help easily if you run into any issues. I revisited fresh installs
of each recently, and after trying both I found it more comfortable to run
the database server on Centos, but I did miss the gigantic and easy to
install Debian software repository.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> Debian packages PostgreSQL in a fashion unique to it; it's arguable
> whether it's better or not (I don't like it), but going with that will
> assure your installation is a bit non-standard compared with most Linux
> installas.

<dons red fedora>

What Debian has done is set up an arrangement that lets you run two (or
more) different PG versions in parallel. Since that's amazingly helpful
during a major-PG-version upgrade, most of the other packagers are
scheming how to do something similar. I'm not sure when this will
happen in the PGDG or Red Hat RPMs, but it probably will eventually.

> Given the buginess and unexpected changes from packages updates of every
> Fedora Core release I've ever tried, I wouldn't trust any OS from that
> line to run a database keeping track of where my socks are at. Core 6
> seems better than most of the older ones. I find it hard to understand
> what it offers that Centos doesn't such that you'd want Fedora instead.

Fedora is about cutting edge, RHEL is about stability, and Centos tracks
RHEL. No surprises there. (<plug> and if someday you want commercial
support for your OS, a Centos->RHEL update will get you there easily.
AFAIK Red Hat doesn't have a clean solution for someone running Fedora
who suddenly realizes he needs a 24x7-supportable OS right now.
Something to work on... </plug>)

</dons red fedora>

regards, tom lane

In #postgresql on freenode, somebody ever mentioned that ZFS from
Solaris helps a lot to the performance of pgsql, so dose anyone have
information about that?

Steve Atkins wrote:
>
> On May 7, 2007, at 2:55 PM, David Levy wrote:
>
>> Hi,
>>
>> I am about to order a new server for my Postgres cluster. I will
>> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>> Which OS would you recommend to optimize Postgres behaviour (i/o
>> access, multithreading, etc) ?
>>
>> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>> help with this ?
>
> Well, all three you mention are much the same, just with a different
> badge on the box, as far as performance is concerned. They're all
> going to be a moderately recent Linux kernel, with your choice
> of filesystems, so any choice between them is going to be driven
> more by available staff and support or personal preference.
>
> I'd probably go CentOS 5 over Fedora just because Fedora doesn't
> get supported for very long - more of an issue with a dedicated
> database box with a long lifespan than your typical desktop or
> interchangeable webserver.
>
> I might also look at Solaris 10, though. I've yet to play with it much,
> but it
> seems nice, and I suspect it might manage 8 cores better than current
> Linux setups.
>
> Cheers,
> Steve
>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 5: don't forget to increase your free space map settings
>

Regards

Ian

On Tue, 8 May 2007, ~]~N彦 Ian Li wrote:

> In #postgresql on freenode, somebody ever mentioned that ZFS from Solaris
> helps a lot to the performance of pgsql, so dose anyone have information
> about that?

the filesystem you use will affect the performance of postgres
significantly. I've heard a lot of claims for ZFS, unfortunantly many of
them from people who have prooven that they didn't know what they were
talking about by the end of their first or second e-mails.

much of the hype for ZFS is it's volume management capabilities and admin
tools. Linux has most (if not all) of the volume management capabilities,
it just seperates them from the filesystems so that any filesystem can use
them, and as a result you use one tool to setup your RAID, one to setup
snapshots, and a third to format your filesystems where ZFS does this in
one userspace tool.

once you seperate the volume management piece out, the actual performance
question is a lot harder to answer. there are a lot of people who say that
it's far faster then the alternate filesystems on Solaris, but I haven't
seen any good comparisons between it and Linux filesystems.

On Linux you have the choice of several filesystems, and the perfomance
will vary wildly depending on your workload. I personally tend to favor
ext2 (for small filesystems where the application is ensuring data
integrity) or XFS (for large filesystems)

I personally don't trust reiserfs, jfs seems to be a tools for
transitioning from AIX more then anything else, and ext3 seems to have all
the scaling issues of ext2 plus the overhead (and bottleneck) of
journaling.

one issue with journaling filesystems, if you journal the data as well as
the metadata you end up with a very reliable setup, however it means that
all your data needs to be written twice, oncce to the journal, and once to
the final location. the write to the journal can be slightly faster then a
normal write to the final location (the journal is a sequential write to
an existing file), however the need to write twice can effectivly cut your
disk I/O bandwidth in half when doing heavy writes. worse, when you end up
writing mor ethen will fit in the journal (128M is the max for ext3) the
entire system then needs to stall while the journal gets cleared to make
space for the additional writes.

if you don't journal your data then you avoid the problems above, but in a
crash you may find that you lost data, even though the filesystem is
'intact' according to fsck.

David Lang

> Steve Atkins wrote:
>>
>> On May 7, 2007, at 2:55 PM, David Levy wrote:
>>
>> > Hi,
>> >
>> > I am about to order a new server for my Postgres cluster. I will
>> > probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>> > Which OS would you recommend to optimize Postgres behaviour (i/o
>> > access, multithreading, etc) ?
>> >
>> > I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>> > help with this ?
>>
>> Well, all three you mention are much the same, just with a different
>> badge on the box, as far as performance is concerned. They're all
>> going to be a moderately recent Linux kernel, with your choice
>> of filesystems, so any choice between them is going to be driven
>> more by available staff and support or personal preference.
>>
>> I'd probably go CentOS 5 over Fedora just because Fedora doesn't
>> get supported for very long - more of an issue with a dedicated
>> database box with a long lifespan than your typical desktop or
>> interchangeable webserver.
>>
>> I might also look at Solaris 10, though. I've yet to play with it much,
>> but it
>> seems nice, and I suspect it might manage 8 cores better than current
>> Linux setups.
>>
>> Cheers,
>> Steve
>>
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: don't forget to increase your free space map settings
>>
>
> Regards
>
> Ian
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq
>
>From pgsql-performance-owner(at)postgresql(dot)org Tue May 8 05:09:47 2007
Received: from localhost (maia-2.hub.org [200.46.204.187])
by postgresql.org (Postfix) with ESMTP id B7F2B9FBB48
for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>; Tue, 8 May 2007 05:09:46 -0300 (ADT)
Received: from postgresql.org ([200.46.204.71])
by localhost (mx1.hub.org [200.46.204.187]) (amavisd-maia, port 10024)
with ESMTP id 21379-08 for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>;
Tue, 8 May 2007 05:09:40 -0300 (ADT)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.4
Received: from nz-out-0506.google.com (nz-out-0506.google.com [64.233.162.234])
by postgresql.org (Postfix) with ESMTP id C1AE49FBB21
for <pgsql-performance(at)postgresql(dot)org>; Tue, 8 May 2007 05:09:42 -0300 (ADT)
Received: by nz-out-0506.google.com with SMTP id s1so1988964nze
for <pgsql-performance(at)postgresql(dot)org>; Tue, 08 May 2007 01:09:41 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
d=gmail.com; s=beta;
h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
b=tMnqJ4jJFkANPEKsjO48H7tdlMkh1PtxD9ojia3Hs7kK4jLUtNUWSWo6GEPQw3nHKK/IFUU42X/xGazATgo19i5FXATTGs/NxHYcWcT4jYq/r84Fmsj2ndDu3lnjOLK8pGaJ+Di1Rw1oLfhcu+zCkBrfTX9KTtbO4BSU2riTwHc=
DomainKey-Signature: a=rsa-sha1; c=nofws;
d=gmail.com; s=beta;
h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
b=ibMOoo+Gg9CIe5nVV5w4I8kJBzehuai5dBAmssoI98av8cCWzpmlj29ellDiBihXGB3g1EbKZR/mldl4oZV3yA/kvjUtA5mAHvvxnkQMjMHzEOOokHx/41kpWk6p14IjD8PYoLxlcaggO4I9y9xyJR8/1+ikTKvqdLOk3eUa+tc=
Received: by 10.115.95.1 with SMTP id x1mr2497915wal.1178611781182;
Tue, 08 May 2007 01:09:41 -0700 (PDT)
Received: by 10.114.160.9 with HTTP; Tue, 8 May 2007 01:09:40 -0700 (PDT)
Message-ID: <b41c75520705080109g7655f5f2m5592595c9a578c16(at)mail(dot)gmail(dot)com>
Date: Tue, 8 May 2007 10:09:40 +0200
From: "Claus Guttesen" <kometen(at)gmail(dot)com>
To: "David Levy" <dvid(dot)levy(at)gmail(dot)com>
Subject: Re: Best OS for Postgres 8.2
Cc: pgsql-performance(at)postgresql(dot)org
In-Reply-To: <1178574927(dot)533849(dot)90730(at)e51g2000hsg(dot)googlegroups(dot)com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <1178574927(dot)533849(dot)90730(at)e51g2000hsg(dot)googlegroups(dot)com>
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Archive-Number: 200705/86
X-Sequence-Number: 24476

> I am about to order a new server for my Postgres cluster. I will
> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
> Which OS would you recommend to optimize Postgres behaviour (i/o
> access, multithreading, etc) ?
>
> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
> help with this ?

My only experience is with FreeBSD. My installation is running 6.2 and
pg 7.4 on a four-way woodcrest and besides being very stable it's also
performing very well. But then FreeBSD 6.x might not scale as well
beyond four cores atm. There you probably would need FreeBSD 7 which
is the development branch and should require extensive testing.

How big will the db be in size?

--
regards
Claus

david(at)lang(dot)hm wrote:
> if you don't journal your data then you avoid the problems above, but in
> a crash you may find that you lost data, even though the filesystem is
> 'intact' according to fsck.

PostgreSQL itself journals it's data to the WAL, so that shouldn't happen.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

> > In #postgresql on freenode, somebody ever mentioned that ZFS from Solaris
> > helps a lot to the performance of pgsql, so dose anyone have information
> > about that?
>
> the filesystem you use will affect the performance of postgres
> significantly. I've heard a lot of claims for ZFS, unfortunantly many of
> them from people who have prooven that they didn't know what they were
> talking about by the end of their first or second e-mails.
>
> much of the hype for ZFS is it's volume management capabilities and admin
> tools. Linux has most (if not all) of the volume management capabilities,
> it just seperates them from the filesystems so that any filesystem can use
> them, and as a result you use one tool to setup your RAID, one to setup
> snapshots, and a third to format your filesystems where ZFS does this in
> one userspace tool.

Even though those posters may have proven them selves wrong, zfs is
still a very handy fs and it should not be judged relative to these
statements.

> once you seperate the volume management piece out, the actual performance
> question is a lot harder to answer. there are a lot of people who say that
> it's far faster then the alternate filesystems on Solaris, but I haven't
> seen any good comparisons between it and Linux filesystems.

One could install pg on solaris 10 and format the data-area as ufs and
then as zfs and compare import- and query-times and other benchmarking
but comparing ufs/zfs to Linux-filesystems would also be a comparison
of those two os'es.

--
regards
Claus

On Tue, 8 May 2007, Claus Guttesen wrote:

>> > In #postgresql on freenode, somebody ever mentioned that ZFS from
>> > Solaris
>> > helps a lot to the performance of pgsql, so dose anyone have information
>> > about that?
>>
>> the filesystem you use will affect the performance of postgres
>> significantly. I've heard a lot of claims for ZFS, unfortunantly many of
>> them from people who have prooven that they didn't know what they were
>> talking about by the end of their first or second e-mails.
>>
>> much of the hype for ZFS is it's volume management capabilities and admin
>> tools. Linux has most (if not all) of the volume management capabilities,
>> it just seperates them from the filesystems so that any filesystem can use
>> them, and as a result you use one tool to setup your RAID, one to setup
>> snapshots, and a third to format your filesystems where ZFS does this in
>> one userspace tool.
>
> Even though those posters may have proven them selves wrong, zfs is
> still a very handy fs and it should not be judged relative to these
> statements.

I don't disagree with you, I'm just noteing that too many of the 'ZFS is
great' posts need to be discounted as a result (the same thing goes for
the 'reiserfs4 is great' posts)

>> once you seperate the volume management piece out, the actual performance
>> question is a lot harder to answer. there are a lot of people who say that
>> it's far faster then the alternate filesystems on Solaris, but I haven't
>> seen any good comparisons between it and Linux filesystems.
>
> One could install pg on solaris 10 and format the data-area as ufs and
> then as zfs and compare import- and query-times and other benchmarking
> but comparing ufs/zfs to Linux-filesystems would also be a comparison
> of those two os'es.

however, such a comparison is very legitimate, it doesn't really matter
which filesystem is better if the OS that it's tied to limits it so much
that the other one wins out with an inferior filesystem

currently ZFS is only available on Solaris, parts of it have been released
under GPLv2, but it doesn't look like enough of it to be ported to Linux
(enough was released for grub to be able to access it read-only, but not
the full filesystem). there are also patent concerns that are preventing
any porting to Linux.

on the other hand, it's integrated userspace tools are pushing people to
create similar tools for Linux (without needeing to combine the vairous
pieces in the kernel)

David Lang

david(at)lang(dot)hm wrote:
> On Tue, 8 May 2007, Claus Guttesen wrote:
>
>>> > In #postgresql on freenode, somebody ever mentioned that ZFS from
>>> > Solaris
>>> > helps a lot to the performance of pgsql, so dose anyone have
>>> information
>>> > about that?
>>>
>>> the filesystem you use will affect the performance of postgres
>>> significantly. I've heard a lot of claims for ZFS, unfortunantly
>>> many of
>>> them from people who have prooven that they didn't know what they were
>>> talking about by the end of their first or second e-mails.
>>>
>>> much of the hype for ZFS is it's volume management capabilities and
>>> admin
>>> tools. Linux has most (if not all) of the volume management
>>> capabilities,
>>> it just seperates them from the filesystems so that any filesystem
>>> can use
>>> them, and as a result you use one tool to setup your RAID, one to setup
>>> snapshots, and a third to format your filesystems where ZFS does
>>> this in
>>> one userspace tool.
>>
>> Even though those posters may have proven them selves wrong, zfs is
>> still a very handy fs and it should not be judged relative to these
>> statements.
>
> I don't disagree with you, I'm just noteing that too many of the 'ZFS is
> great' posts need to be discounted as a result (the same thing goes for
> the 'reiserfs4 is great' posts)
>
>>> once you seperate the volume management piece out, the actual
>>> performance
>>> question is a lot harder to answer. there are a lot of people who
>>> say that
>>> it's far faster then the alternate filesystems on Solaris, but I
>>> haven't
>>> seen any good comparisons between it and Linux filesystems.
>>
>> One could install pg on solaris 10 and format the data-area as ufs and
>> then as zfs and compare import- and query-times and other benchmarking
>> but comparing ufs/zfs to Linux-filesystems would also be a comparison
>> of those two os'es.
>
> however, such a comparison is very legitimate, it doesn't really matter
> which filesystem is better if the OS that it's tied to limits it so much
> that the other one wins out with an inferior filesystem
>
> currently ZFS is only available on Solaris, parts of it have been
> released under GPLv2, but it doesn't look like enough of it to be ported
> to Linux (enough was released for grub to be able to access it
> read-only, but not the full filesystem). there are also patent concerns
> that are preventing any porting to Linux.

This is not entirely correct. ZFS is only under the CDDL license and it
has been ported to FreeBSD.

http://mail.opensolaris.org/pipermail/zfs-discuss/2007-April/026922.html

--
Trygve

On Mon, May 07, 2007 at 11:56:14PM -0400, Greg Smith wrote:
> Debian packages PostgreSQL in a fashion unique to it; it's arguable
> whether it's better or not (I don't like it), but going with that will
> assure your installation is a bit non-standard compared with most Linux
> installas. The main reasons you'd pick Debian are either that you like
> that scheme (which tries to provide some structure to running multiple
> clusters on one box), or that you plan to rely heavily on community
> packages that don't come with the Redhat distributions and therefore would
> appreciate how easy it is to use apt-get against the large Debian software
> repository.

Just to add to this: As far as I understand it, this scheme was originally
mainly put in place to allow multiple _versions_ of Postgres to be installed
alongside each other, for smoother upgrades. (There's a command that does all
the details of running first pg_dumpall for the users and groups, then the
new pg_dump with -Fc to get all data and LOBs over, then some hand-fixing to
change explicit paths to $libdir, etc...)

Of course, you lose all that if you need a newer Postgres version than the OS
provides. (Martin Pitt, the Debian/Ubuntu maintainer of Postgres -- the
packaging in Debian and Ubuntu is the same, sans version differences -- makes
his own backported packages of the newest Postgres to Debian stable; it's up
to you if you'd trust that or not.)

/* Steinar */
--
Homepage: http://www.sesse.net/

On 5/8/07, david(at)lang(dot)hm <david(at)lang(dot)hm> wrote:
[snip]
> I personally don't trust reiserfs, jfs seems to be a tools for
> transitioning from AIX more then anything else [...]

What makes you say this? I have run JFS for years with complete
satisfaction, and I have never logged into an AIX box.

JFS has traditionally been seen as an underdog, but undeservedly so,
in my opinion; one cause might be the instability of the very early
releases, which seems to have tainted its reputation, or the alienness
of its AIX heritage. However, every benchmark I have come across puts
its on par with, and often surpassing, the more popular file systems
in performance. In particular, JFS seems to shine with respect to CPU
overhead.

Alexander.

On Mon, May 07, 2007 at 03:14:08PM -0700, Joshua D. Drake wrote:
> It is my understanding (and I certainly could be wrong) that FreeBSD
> doesn't handle SMP nearly as well as Linux (and Linux not as well as
> Solaris).

I'm not actually sure about the last part. There are installations as big as
1024 CPUs that run Linux -- most people won't need that, but it's probably an
indicator that eight cores should run OK :-)

/* Steinar */
--
Homepage: http://www.sesse.net/

On Tue, 8 May 2007, Trygve Laugstøl wrote:

>> currently ZFS is only available on Solaris, parts of it have been released
>> under GPLv2, but it doesn't look like enough of it to be ported to Linux
>> (enough was released for grub to be able to access it read-only, but not
>> the full filesystem). there are also patent concerns that are preventing
>> any porting to Linux.
>
> This is not entirely correct. ZFS is only under the CDDL license and it has
> been ported to FreeBSD.
>
> http://mail.opensolaris.org/pipermail/zfs-discuss/2007-April/026922.html
>
I wonder how they handled the license issues? I thought that if you
combined stuff that was BSD licensed with stuff with a more restrictive
license the result was under the more restrictive license. thanks for the
info.

here's a link about the GPLv2 stuff for zfs

http://blogs.sun.com/darren/entry/zfs_under_gplv2_already_exists
>From pgsql-performance-owner(at)postgresql(dot)org Tue May 8 07:22:31 2007
Received: from localhost (maia-4.hub.org [200.46.204.183])
by postgresql.org (Postfix) with ESMTP id 3ED559FB28A
for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>; Tue, 8 May 2007 07:22:29 -0300 (ADT)
Received: from postgresql.org ([200.46.204.71])
by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024)
with ESMTP id 58273-05 for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>;
Tue, 8 May 2007 07:22:26 -0300 (ADT)
X-Greylist: from auto-whitelisted by SQLgrey-1.7.5
Received: from oxford.xeocode.com (unknown [62.232.55.118])
by postgresql.org (Postfix) with ESMTP id 1ACC39FB1B7
for <pgsql-performance(at)postgresql(dot)org>; Tue, 8 May 2007 07:22:26 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=oxford.xeocode.com)
by oxford.xeocode.com with esmtp (Exim 4.67)
(envelope-from <stark(at)enterprisedb(dot)com>)
id 1HlMpr-00010j-V3; Tue, 08 May 2007 11:22:24 +0100
From: Gregory Stark <stark(at)enterprisedb(dot)com>
To: "Pomarede Nicolas" <npomarede(at)corp(dot)free(dot)fr>
Cc: <pgsql-performance(at)postgresql(dot)org>
Subject: Re: truncate a table instead of vaccum full when count(*) is 0
In-Reply-To: <Pine(dot)LNX(dot)4(dot)64(dot)0705081125140(dot)20675(at)localhost> (Pomarede Nicolas's
message of "Tue, 8 May 2007 11:43:14 +0200 (CEST)")
Organization: EnterpriseDB
References: <Pine(dot)LNX(dot)4(dot)64(dot)0705081125140(dot)20675(at)localhost>
X-Draft-From: ("nnimap+mail01.enterprisedb.com:INBOX.performance" 216)
Date: Tue, 08 May 2007 11:22:23 +0100
Message-ID: <87d51b1uow(dot)fsf(at)oxford(dot)xeocode(dot)com>
User-Agent: Gnus/5.110006 (No Gnus v0.6) Emacs/21.4 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Archive-Number: 200705/100
X-Sequence-Number: 24490

"Pomarede Nicolas" <npomarede(at)corp(dot)free(dot)fr> writes:

> But for the data (dead rows), even running a vacuum analyze every day is not
> enough, and doesn't truncate some empty pages at the end, so the data size
> remains in the order of 200-300 MB, when only a few effective rows are there.

Try running vacuum more frequently. Once per day isn't very frequent for
vacuum, every 60 or 30 minutes isn't uncommon. For your situation you might
even consider running it continuously in a loop.

> I see in the 8.3 list of coming changes that the FSM will try to re-use pages
> in a better way to help truncating empty pages. Is this correct ?

There are several people working on improvements to vacuum but it's not clear
right now exactly what we'll end up with. I think most of the directly vacuum
related changes wouldn't actually help you either.

The one that would help you is named "HOT". If you're interested in
experimenting with an experimental patch you could consider taking CVS and
applying HOT and seeing how it affects you. Or if you see an announcement that
it's been comitted taking a beta and experimenting with it before the 8.3
release could be interesting. Experiments with real-world databases can be
very helpful for developers since it's hard to construct truly realistic
benchmarks.

> So, I would like to truncate the table when the number of rows reaches 0 (just
> after the table was processed, and just before some new rows are added).
>
> Is there an easy way to do this under psql ? For example, lock the table, do a
> count(*), if result is 0 row then truncate the table, unlock the table (a kind
> of atomic 'truncate table if count(*) == 0').
>
> Would this work and what would be the steps ?

It would work but you may end up keeping the lock for longer than you're happy
for. Another option to consider would be to use CLUSTER instead of vacuum full
though the 8.2 CLUSTER wasn't entirely MVCC safe and I think in your situation
that might actually be a problem. It would cause transactions that started
before the cluster (but didn't access the table before the cluster) to not see
any records after the cluster.

--
Gregory Stark
EnterpriseDB http://www.enterprisedb.com

On Tue, 8 May 2007, Steinar H. Gunderson wrote:

> On Mon, May 07, 2007 at 03:14:08PM -0700, Joshua D. Drake wrote:
>> It is my understanding (and I certainly could be wrong) that FreeBSD
>> doesn't handle SMP nearly as well as Linux (and Linux not as well as
>> Solaris).
>
> I'm not actually sure about the last part. There are installations as big as
> 1024 CPUs that run Linux -- most people won't need that, but it's probably an
> indicator that eight cores should run OK :-)

over the weekend the question of scalability was raised on the linux
kernel mailing list and people are shipping 1024 cpu systems with linux,
and testing 4096 cpu systems. there are occasionally still bottlenecks
that limit scalability, butunless you run into a bad driver or filesystem
you should have no problems in the 8-16 core range.

any comparison between Linux and any other OS needs to include a date for
when the comparison was made, Linux is changing at a frightning pace (I
think I saw something within the last few weeks that said that the rate of
change for the kernel has averaged around 9000 lines of code per day over
the last couple of years) you need to re-check comparisons every year or
two or you end up working with obsolete data.

David Lang

david(at)lang(dot)hm wrote:
> On Tue, 8 May 2007, Trygve Laugstøl wrote:
>
>>> currently ZFS is only available on Solaris, parts of it have been
>>> released
>>> under GPLv2, but it doesn't look like enough of it to be ported to
>>> Linux
>>> (enough was released for grub to be able to access it read-only, but
>>> not
>>> the full filesystem). there are also patent concerns that are
>>> preventing
>>> any porting to Linux.
>>
>> This is not entirely correct. ZFS is only under the CDDL license and
>> it has been ported to FreeBSD.
>>
>> http://mail.opensolaris.org/pipermail/zfs-discuss/2007-April/026922.html
>>
> I wonder how they handled the license issues? I thought that if you
> combined stuff that was BSD licensed with stuff with a more restrictive
> license the result was under the more restrictive license. thanks for
> the info.

The CDDL is not a restrictive license like GPL, it is based on the MIT
license so it can be used with BSD stuff without problems. There are
lots of discussion going on (read: flamewars) on the opensolaris lists
about how it can/should it/will it be integrated into linux.

> here's a link about the GPLv2 stuff for zfs
>
> http://blogs.sun.com/darren/entry/zfs_under_gplv2_already_exists

That title is fairly misleading as it's only some read-only bits to be
able to boot off ZFS with grub.

--
Trygve

I'm really not a senior member around here and while all this licensing
stuff and underlying fs between OSs is very interesting can we please
think twice before continuing it.

Thanks for the minute,

./C

WRT ZFS on Linux, if someone were to port it, the license issue would get worked out IMO (with some discussion to back me up). From discussions with the developers, the biggest issue is a technical one: the Linux VFS layer makes the port difficult.

I don't hold any hope that the FUSE port will be a happy thing, the performance won't be there.

Any volunteers to port ZFS to Linux?

- Luke

> I'm really not a senior member around here and while all this licensing
> stuff and underlying fs between OSs is very interesting can we please
> think twice before continuing it.

Agree, there are other lists for this stuff; and back to what one of
the original posters said: it doesn't matter much.

[Also not a regular poster, but I always gain something from reading
this list.]

Most people who really go into OS selection / FS selection are looking
for a cheap/silver bullet for performance. No such thing exists. The
difference made by any modern OS/FS is almost immaterial. You need to
do the slow slogging work of site/application specific optimization and
tuning; that is where you will find significant performance
improvements.

-
Adam Tauno Williams, Network & Systems Administrator
Consultant - http://www.whitemiceconsulting.com
Developer - http://www.opengroupware.org

I've seen the FUSE port of ZFS, and it does run sslloowwllyy. It
appears that a native linux port is going to be required if we want
ZFS to be reasonably performant.

WRT which FS to use for pg; the biggest issue is what kind of DB you
will be building. The best pg FS for OLTP and OLAP are not the same
IME. Ditto a dependence on how large your records and the amount of
IO in your typical transactions are.

For lot's of big, more reads than writes transactions, SGI's XFS
seems to be best.
XFS is not the best for OLTP. Especially for OLTP involving lots of small IOs.

jfs seems to be best for that.

Caveat: I have not yet experimented with any version of reiserfs in
production.

Cheers,
Ron Peacetree

At 08:01 AM 5/8/2007, Luke Lonergan wrote:
>WRT ZFS on Linux, if someone were to port it, the license issue
>would get worked out IMO (with some discussion to back me up). From
>discussions with the developers, the biggest issue is a technical
>one: the Linux VFS layer makes the port difficult.
>
>I don't hold any hope that the FUSE port will be a happy thing, the
>performance won't be there.
>
>Any volunteers to port ZFS to Linux?
>
>- Luke
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 5: don't forget to increase your free space map settings

I am back with the chatlog and seem it's the Transparent compression
that helps a lot, very interesting...

here is the log of #postgresql on Apr. 21th around 13:20 GMT (snipped) :
<Solatis> why is that, when hard disk i/o is my bottleneck ?
<Solatis> well i have 10 disks in a raid1+0 config
<Solatis> it's sata2 yes
<Solatis> i run solaris express, whose kernel says SunOS
<Solatis> running 'SunOS solatis2 5.11 snv_61 i86pc i386 i86pc
<Solatis> well, the thing is, i'm using zfs
<Solatis> yeah, it was the reason for me to install solaris in
the first place
<Solatis> and a benchmark for my system comparing debian linux
with solaris express showed a +- 18% performance gain when switching
to solaris
<Solatis> so i'm happy
<Solatis> (note: the benchmarking was not scientifically
grounded at all, it was just around 50 million stored procedure
calls which do select/update/inserts on my database which would
simulate my specific case)
<Solatis> but the killer thing was to enable compression on zfs
<Solatis> that reduced the hard disk i/o with a factor 3, which
was the probable cause of the performance increase
<Solatis> oh, at the moment it's factor 2.23
<Solatis> still, it's funny to see that postgresql says that my
database is using around 41GB's, while only taking up 18GB on the
hard disk
=== end of log ===

david(at)lang(dot)hm wrote:
> On Tue, 8 May 2007, �~]~N彦 Ian Li wrote:
>
>> In #postgresql on freenode, somebody ever mentioned that ZFS from
>> Solaris helps a lot to the performance of pgsql, so dose anyone have
>> information about that?
>
> the filesystem you use will affect the performance of postgres
> significantly. I've heard a lot of claims for ZFS, unfortunantly many of
> them from people who have prooven that they didn't know what they were
> talking about by the end of their first or second e-mails.
>
> much of the hype for ZFS is it's volume management capabilities and
> admin tools. Linux has most (if not all) of the volume management
> capabilities, it just seperates them from the filesystems so that any
> filesystem can use them, and as a result you use one tool to setup your
> RAID, one to setup snapshots, and a third to format your filesystems
> where ZFS does this in one userspace tool.
>
> once you seperate the volume management piece out, the actual
> performance question is a lot harder to answer. there are a lot of
> people who say that it's far faster then the alternate filesystems on
> Solaris, but I haven't seen any good comparisons between it and Linux
> filesystems.
>
> On Linux you have the choice of several filesystems, and the perfomance
> will vary wildly depending on your workload. I personally tend to favor
> ext2 (for small filesystems where the application is ensuring data
> integrity) or XFS (for large filesystems)
>
> I personally don't trust reiserfs, jfs seems to be a tools for
> transitioning from AIX more then anything else, and ext3 seems to have
> all the scaling issues of ext2 plus the overhead (and bottleneck) of
> journaling.
>
> one issue with journaling filesystems, if you journal the data as well
> as the metadata you end up with a very reliable setup, however it means
> that all your data needs to be written twice, oncce to the journal, and
> once to the final location. the write to the journal can be slightly
> faster then a normal write to the final location (the journal is a
> sequential write to an existing file), however the need to write twice
> can effectivly cut your disk I/O bandwidth in half when doing heavy
> writes. worse, when you end up writing mor ethen will fit in the journal
> (128M is the max for ext3) the entire system then needs to stall while
> the journal gets cleared to make space for the additional writes.
>
> if you don't journal your data then you avoid the problems above, but in
> a crash you may find that you lost data, even though the filesystem is
> 'intact' according to fsck.
>
> David Lang
>
Regards
Ian

On Tue, 8 May 2007, david(at)lang(dot)hm wrote:

> one issue with journaling filesystems, if you journal the data as well as the
> metadata you end up with a very reliable setup, however it means that all
> your data needs to be written twice, oncce to the journal, and once to the
> final location. the write to the journal can be slightly faster then a normal
> write to the final location (the journal is a sequential write to an existing
> file), however the need to write twice can effectivly cut your disk I/O
> bandwidth in half when doing heavy writes. worse, when you end up writing mor
> ethen will fit in the journal (128M is the max for ext3) the entire system
> then needs to stall while the journal gets cleared to make space for the
> additional writes.
>
> if you don't journal your data then you avoid the problems above, but in a
> crash you may find that you lost data, even though the filesystem is 'intact'
> according to fsck.

That sounds like an ad for FreeBSD and UFS2+Softupdates. :)

Metadata is as safe as it is in a journaling filesystem, but none of the
overhead of journaling.

Charles

> David Lang
>
>> Steve Atkins wrote:
>>>
>>> On May 7, 2007, at 2:55 PM, David Levy wrote:
>>>
>>> > Hi,
>>> > > I am about to order a new server for my Postgres cluster. I will
>>> > probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>>> > Which OS would you recommend to optimize Postgres behaviour (i/o
>>> > access, multithreading, etc) ?
>>> > > I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>>> > help with this ?
>>>
>>> Well, all three you mention are much the same, just with a different
>>> badge on the box, as far as performance is concerned. They're all
>>> going to be a moderately recent Linux kernel, with your choice
>>> of filesystems, so any choice between them is going to be driven
>>> more by available staff and support or personal preference.
>>>
>>> I'd probably go CentOS 5 over Fedora just because Fedora doesn't
>>> get supported for very long - more of an issue with a dedicated
>>> database box with a long lifespan than your typical desktop or
>>> interchangeable webserver.
>>>
>>> I might also look at Solaris 10, though. I've yet to play with it much,
>>> but it
>>> seems nice, and I suspect it might manage 8 cores better than current
>>> Linux setups.
>>>
>>> Cheers,
>>> Steve
>>>
>>>
>>>
>>> ---------------------------(end of broadcast)---------------------------
>>> TIP 5: don't forget to increase your free space map settings
>>>
>>
>> Regards
>>
>> Ian
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 3: Have you checked our extensive FAQ?
>>
>> http://www.postgresql.org/docs/faq
>>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo(at)postgresql(dot)org so that your
> message can get through to the mailing list cleanly
>

On Tue, 8 May 2007, Tom Lane wrote:

> What Debian has done is set up an arrangement that lets you run two (or
> more) different PG versions in parallel. Since that's amazingly helpful
> during a major-PG-version upgrade, most of the other packagers are
> scheming how to do something similar.

I alluded to that but it is worth going into more detail on for those not
familiar with this whole topic. I normally maintain multiple different PG
versions in parallel already, mostly using environment variables to switch
between them with some shell code. Debian has taken an approach where
commands like pg_ctl are wrapped in multi-version/cluster aware scripts,
so you can do things like restarting multiple installations more easily
than that.

My issue wasn't with the idea, it was with the implementation. When I
have my newbie hat on, it adds a layer of complexity that isn't needed for
simple installs. And when I have my developer hat on, I found that need
to conform to the requirements of that system on top of Debian's already
unique install locations and packaging issues just made it painful to
build and work with with customized versions of Postgres, compared to
distributions that use a relatively simple packaging scheme (like the RPM
based RedHat or SuSE).

I hope anyone else working this problem is thinking about issues like
this. Debian's approach strikes me as being a good one for a seasoned
systems administrator or DBA, which is typical for them. I'd hate to see
a change in this area make it more difficult for new users though, as
that's already perceived as a PG weakness. I think you can build a layer
that adds the capability for the people who need it without complicating
things for people who don't.

> and if someday you want commercial support for your OS, a Centos->RHEL
> update will get you there easily.

For those that like to live dangerously, it's also worth mentioning that
it's possible to hack this conversion in either direction without actually
doing an OS re-install/upgrade just by playing with the packages that are
different between the two. So someone who installs CentOS now could swap
to RHEL very quickly in a pinch if they have enough cojones to do the
required package substitutions.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Tue, 8 May 2007, Luke Lonergan wrote:

> From discussions with the developers, the biggest issue is a technical
> one: the Linux VFS layer makes the [ZFS] port difficult.

Difficult on two levels. First you'd have to figure out how to make it
work at all; then you'd have to reshape it into a form that it would be
acceptable to the Linux kernel developers, who haven't seemed real keen on
the idea so far.

The standard article I'm you've already seen this week on this topic is
Jeff Bonwick's at
http://blogs.sun.com/bonwick/entry/rampant_layering_violation

What really bugged me was his earlier article linked to there where he
talks about how ZFS eliminates the need for hardware RAID controllers:
http://blogs.sun.com/bonwick/entry/raid_z

While there may be merit to that idea for some applications, like
situations where you have a pig of a RAID5 volume, that's just hype for
database writes. "We issue the SYNCHRONIZE CACHE command to the disks
after pushing all data in a transaction group"--see, that would be the
part the hardware controller is needed to accelerate. If you really care
about whether your data hit disk, there is no way to break the RPM barrier
without hardware support. The fact that he misunderstands such a
fundamental point makes me wonder what other gigantic mistakes might be
buried in his analysis.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Tue, 8 May 2007, Greg Smith wrote:

> On Tue, 8 May 2007, Luke Lonergan wrote:
>
>> From discussions with the developers, the biggest issue is a technical
>> one: the Linux VFS layer makes the [ZFS] port difficult.
>
> Difficult on two levels. First you'd have to figure out how to make it work
> at all; then you'd have to reshape it into a form that it would be acceptable
> to the Linux kernel developers, who haven't seemed real keen on the idea so
> far.

given that RAID, snapshots, etc are already in the linux kernel, I suspect
that what will need to happen is for the filesystem to be ported without
those features and then the userspace tools (that manipulate the volumes )
be ported to use the things already in the kernel.

> The standard article I'm you've already seen this week on this topic is Jeff
> Bonwick's at http://blogs.sun.com/bonwick/entry/rampant_layering_violation

yep, that sounds like what I've been hearing.

what the ZFS (and reiserfs4) folks haven't been wanting to hear from the
linux kernel devs is that they are interested in having all these neat
features available for use with all filesystems (and the linux kernel has
a _lot_ of filesystems available), with solaris you basicly have UFS and
ZFS so it's not as big a deal.

> What really bugged me was his earlier article linked to there where he talks
> about how ZFS eliminates the need for hardware RAID controllers:
> http://blogs.sun.com/bonwick/entry/raid_z
>
> While there may be merit to that idea for some applications, like situations
> where you have a pig of a RAID5 volume, that's just hype for database writes.
> "We issue the SYNCHRONIZE CACHE command to the disks after pushing all data
> in a transaction group"--see, that would be the part the hardware controller
> is needed to accelerate. If you really care about whether your data hit
> disk, there is no way to break the RPM barrier without hardware support. The
> fact that he misunderstands such a fundamental point makes me wonder what
> other gigantic mistakes might be buried in his analysis.

I've seen similar comments from some of the linux kernel devs, they've
used low-end raid controllers with small processors on them and think that
a second core/socket in the main system to run software raid on is better.

David Lang

On Wed, May 09, 2007 at 01:57:51AM -0700, david(at)lang(dot)hm wrote:
> given that RAID, snapshots, etc are already in the linux kernel, I suspect
> that what will need to happen is for the filesystem to be ported without
> those features and then the userspace tools (that manipulate the volumes )
> be ported to use the things already in the kernel.

Well, part of the idea behind ZFS is that these parts are _not_ separated in
"layers" -- for instance, the filesystem can push data down to the RAID level
to determine the stripe size used.

Whether this is a good idea is of course hotly debated, but I don't think you
can port just the filesystem part and call it a day.

/* Steinar */
--
Homepage: http://www.sesse.net/

On Wed, 9 May 2007, Steinar H. Gunderson wrote:

> On Wed, May 09, 2007 at 01:57:51AM -0700, david(at)lang(dot)hm wrote:
>> given that RAID, snapshots, etc are already in the linux kernel, I suspect
>> that what will need to happen is for the filesystem to be ported without
>> those features and then the userspace tools (that manipulate the volumes )
>> be ported to use the things already in the kernel.
>
> Well, part of the idea behind ZFS is that these parts are _not_ separated in
> "layers" -- for instance, the filesystem can push data down to the RAID level
> to determine the stripe size used.

there's nothing preventing this from happening if they are seperate layers
either.

there are some performance implications of the seperate layers, but until
someone has the ability to do head-to-head comparisons it's hard to say
which approach will win (in theory the lack of layers makes for faster
code, but in practice the fact that each layer is gone over by experts
looking for ways to optimize it may overwelm the layering overhead)

> Whether this is a good idea is of course hotly debated, but I don't think you
> can port just the filesystem part and call it a day.

Oh, I'm absolutly sure that doing so won't satidfy people (wnd would
generate howles of outrage from some parts), but having watched other
groups try and get things into the kernel that the kernel devs felt were
layering violations I think that it's wat will ultimatly happen.

David Lang

Hello Ian,

I have done some testing with postgresql and ZFS on Solaris 10 11/06.
While I work for Sun, I dont claim to be a ZFS expert (for that matter
not even Solaris or PostgreSQL).

Lets first look at the scenarios of how postgresql can be deployed on
Solaris
First the Solaris Options
1. UFS with default setup (which is buffered file system)
2. UFS with forcedirectio option (or unbuffered file system)
3. ZFS by default (128K recordsize with checksum but no compression)
4. ZFS with Compression (Default compression using LZ* algorithm .. now
even a gzip algorithm is supported)

(For simplicity I am not considering RAID levels here since that
increases the number of scenarios quite a bit and also skipping Solaris
Volume Manager - legacy volume management capabilities in Solaris)

Now for the postgresql.conf options
a. wal_sync_method set to default - maps to opendatasync
b. wal_sync_method set to fdatasync

(assuming checkpoint_segments and wal_buffers are high already)

(This are my tests results based on the way I used the workload and
your mileage will vary)
So with this type of configurations I found the following
1a. Default UFS with default wal_sync_method - Sucks for me mostly
using pgbench or EAStress type workloads
1b. Default UFS with fdatasync - works well specially increasing
segmapsize from default 12% to higher values
2a ForcedirectIO with default wal_sync_method - works well but then is
limited to hardware disk performances
(In a way good to have RAID controller with big Write cache for
it.. One advantage is lower system cpu utilization)
2b Didn't see huge difference from 2a in this case
3a It was better than 1a but still limited
3b It was better even than 3a and 1b but cpu utilization seemed higher
4a - Didn't test this out
4b - Hard to say since in my case since I wasnt disk bound (per se) but
CPU bound. The compression helps when number of IOs to the disk are high
and it helps to cut it down at the cost of CPU cycles

Overall ZFS seems to improve performance with PostgreSQL on Solaris 10
with a bit increased system times compared to UFS.
(So the final results depends on the metrics that you are measuring the
performance :-) ) (ZFS engineers are constantly improving the
performance and I have seen the improvements from Solaris 10 1/06
release to my current setup)

Of course I haven't compared against any other OS.. If someone has
already done that I would be interested in knowing the results.

Now comes the thing that I am still exploring
* Do we do checksum in WAL ? I guess we do .. Which means that we are
now doing double checksumming on the data. One in ZFS and one in
postgresql. ZFS does allow checksumming to be turned off (but on new
blocks allocated). But of course the philosophy is where should it be
done (ZFS or PostgreSQL). ZFS checksumming gives ability to correct the
data on the bad checksum if you use mirror devices. PostgreSQL doesnt
give that ability and in case of an error would fail. ( I dont know the
exact behavior of postgresql when it would encounter a failed checksum)

Hope this helps.

Regards,
Jignesh

李彦 Ian Li wrote:
> In #postgresql on freenode, somebody ever mentioned that ZFS from
> Solaris helps a lot to the performance of pgsql, so dose anyone have
> information about that?
>
> Steve Atkins wrote:
>>
>> On May 7, 2007, at 2:55 PM, David Levy wrote:
>>
>>> Hi,
>>>
>>> I am about to order a new server for my Postgres cluster. I will
>>> probably get a Dual Xeon Quad Core instead of my current Dual Xeon.
>>> Which OS would you recommend to optimize Postgres behaviour (i/o
>>> access, multithreading, etc) ?
>>>
>>> I am hesitating between Fedora Core 6, CentOS and Debian. Can anyone
>>> help with this ?
>>
>> Well, all three you mention are much the same, just with a different
>> badge on the box, as far as performance is concerned. They're all
>> going to be a moderately recent Linux kernel, with your choice
>> of filesystems, so any choice between them is going to be driven
>> more by available staff and support or personal preference.
>>
>> I'd probably go CentOS 5 over Fedora just because Fedora doesn't
>> get supported for very long - more of an issue with a dedicated
>> database box with a long lifespan than your typical desktop or
>> interchangeable webserver.
>>
>> I might also look at Solaris 10, though. I've yet to play with it
>> much, but it
>> seems nice, and I suspect it might manage 8 cores better than current
>> Linux setups.
>>
>> Cheers,
>> Steve
>>
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 5: don't forget to increase your free space map settings
>>
>
> Regards
>
> Ian
>
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
> http://www.postgresql.org/docs/faq

Jignesh Shah escribió:

> Now comes the thing that I am still exploring
> * Do we do checksum in WAL ? I guess we do .. Which means that we are
> now doing double checksumming on the data. One in ZFS and one in
> postgresql. ZFS does allow checksumming to be turned off (but on new
> blocks allocated). But of course the philosophy is where should it be
> done (ZFS or PostgreSQL).

Checksums on WAL are not optional in Postgres, because AFAIR they are
used to determine when it should stop recovering.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On May 8, 2007, at 2:59 AM, david(at)lang(dot)hm wrote:
> one issue with journaling filesystems, if you journal the data as
> well as the metadata you end up with a very reliable setup, however
> it means that all your data needs to be written twice, oncce to the
> journal, and once to the final location. the write to the journal
> can be slightly faster then a normal write to the final location
> (the journal is a sequential write to an existing file), however
> the need to write twice can effectivly cut your disk I/O bandwidth
> in half when doing heavy writes. worse, when you end up writing mor
> ethen will fit in the journal (128M is the max for ext3) the entire
> system then needs to stall while the journal gets cleared to make
> space for the additional writes.

That's why you want to mount ext3 partitions used with PostgreSQL
with data=writeback.

Some folks will also use a small filesystem for pg_xlog and mount
that as ext2.
--
Jim Nasby jim(at)nasby(dot)net
EnterpriseDB http://enterprisedb.com 512.569.9461 (cell)

But we still pay the penalty on WAL while writing them in the first
place I guess .. Is there an option to disable it.. I can test how much
is the impact I guess couple of %s but good to verify :-) )

Regards,
Jignesh

Alvaro Herrera wrote:
> Jignesh Shah escribió:
>
>
>> Now comes the thing that I am still exploring
>> * Do we do checksum in WAL ? I guess we do .. Which means that we are
>> now doing double checksumming on the data. One in ZFS and one in
>> postgresql. ZFS does allow checksumming to be turned off (but on new
>> blocks allocated). But of course the philosophy is where should it be
>> done (ZFS or PostgreSQL).
>>
>
> Checksums on WAL are not optional in Postgres, because AFAIR they are
> used to determine when it should stop recovering.
>
>

On Wed, 9 May 2007, Jignesh Shah wrote:

> But we still pay the penalty on WAL while writing them in the first place I
> guess .. Is there an option to disable it.. I can test how much is the impact
> I guess couple of %s but good to verify :-) )

on modern CPU's where the CPU is significantly faster then RAM,
calculating a checksum is free if the CPU has to touch the data anyway
(cycles where it would be waiting for a cache miss are spent doing the
calculations)

if you don't believe me, hack the source to remove the checksum and see if
you can measure any difference.

David Lang

>
> Regards,
> Jignesh
>
>
> Alvaro Herrera wrote:
>> Jignesh Shah escribió:
>>
>>
>> > Now comes the thing that I am still exploring
>> > * Do we do checksum in WAL ? I guess we do .. Which means that we are
>> > now doing double checksumming on the data. One in ZFS and one in
>> > postgresql. ZFS does allow checksumming to be turned off (but on new
>> > blocks allocated). But of course the philosophy is where should it be
>> > done (ZFS or PostgreSQL).
>> >
>>
>> Checksums on WAL are not optional in Postgres, because AFAIR they are
>> used to determine when it should stop recovering.
>>
>>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: Don't 'kill -9' the postmaster
>
>
>From pgsql-performance-owner(at)postgresql(dot)org Wed May 9 15:41:21 2007
Received: from localhost (maia-3.hub.org [200.46.204.184])
by postgresql.org (Postfix) with ESMTP id A90F09FB46E
for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>; Wed, 9 May 2007 15:41:20 -0300 (ADT)
Received: from postgresql.org ([200.46.204.71])
by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024)
with ESMTP id 85364-01 for <pgsql-performance-postgresql(dot)org(at)postgresql(dot)org>;
Wed, 9 May 2007 15:41:08 -0300 (ADT)
X-Greylist: domain auto-whitelisted by SQLgrey-1.7.4
Received: from wr-out-0506.google.com (wr-out-0506.google.com [64.233.184.235])
by postgresql.org (Postfix) with ESMTP id E15479FA217
for <pgsql-performance(at)postgresql(dot)org>; Wed, 9 May 2007 15:41:09 -0300 (ADT)
Received: by wr-out-0506.google.com with SMTP id 70so302762wra
for <pgsql-performance(at)postgresql(dot)org>; Wed, 09 May 2007 11:41:08 -0700 (PDT)
DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed;
d=gmail.com; s=beta;
h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references;
b=g1zIF4bAuqSx9xY+tHoUSfcG0PAUOwOTe/AxVWbVOTEMdg8dtdSN022EuYUR1Ow/OrtKdf7bzeTn1Lru6qy8mYM+ZCELf04aKExcZ1W7OE/+504RcvV1fEzZMMYPjl2cO1CppR+77BGvuUGv9MAW4YKTqiN8LXtTZi9C+FJehW4=
DomainKey-Signature: a=rsa-sha1; c=nofws;
d=gmail.com; s=beta;
h=received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references;
b=JvyBekBF3lPKnLONW3Ns3+UkNrN9jQpUvP8mqdN0sBpzp/hPdP01ie72wobmCY3FE3eomuoWgSk1t2sg/HR7w+tvpmK4kMN1PvTg9lNG0uJzdkMGGx70revTVRjuzk2Yb2Vbw2g1ZgrCyOqdm+LAhwi04iHei+QFI3YEzCmonEs=
Received: by 10.114.74.1 with SMTP id w1mr252771waa.1178736067911;
Wed, 09 May 2007 11:41:07 -0700 (PDT)
Received: by 10.115.19.4 with HTTP; Wed, 9 May 2007 11:41:07 -0700 (PDT)
Message-ID: <3ce9822f0705091141m227697d7h4b7f2b5b723cfbd7(at)mail(dot)gmail(dot)com>
Date: Wed, 9 May 2007 20:41:07 +0200
From: "Valentine Gogichashvili" <valgog(at)gmail(dot)com>
To: "Oleg Bartunov" <oleg(at)sai(dot)msu(dot)su>
Subject: Re: Cannot make GIN intarray index be used by the planner
Cc: pgsql-performance(at)postgresql(dot)org
In-Reply-To: <Pine(dot)LNX(dot)4(dot)64(dot)0705091744470(dot)12152(at)sn(dot)sai(dot)msu(dot)ru>
MIME-Version: 1.0
Content-Type: multipart/alternative;
boundary="----=_Part_110820_33458115.1178736067834"
References: <3ce9822f0705090612nd6198a2xd06da85f1accbae9(at)mail(dot)gmail(dot)com>
<Pine(dot)LNX(dot)4(dot)64(dot)0705091728560(dot)12152(at)sn(dot)sai(dot)msu(dot)ru>
<3ce9822f0705090636s46b71c48mafb6ba7d64781fbf(at)mail(dot)gmail(dot)com>
<Pine(dot)LNX(dot)4(dot)64(dot)0705091744470(dot)12152(at)sn(dot)sai(dot)msu(dot)ru>
X-Virus-Scanned: Maia Mailguard 1.0.1
X-Archive-Number: 200705/207
X-Sequence-Number: 24597

------=_Part_110820_33458115.1178736067834
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: base64
Content-Disposition: inline

SGkgYWdhaW4sCgp0aGUgdmVyc2lvbiBvZiB0aGUgc2VydmVyIEkgYW0gb24gaXMgUG9zdGdyZVNR
TCA4LjIuMyBvbiBpNjg2LXBjLWxpbnV4LWdudSwKY29tcGlsZWQgYnkgR0NDIGdjYyAoR0NDKSA0
LjAuMiAyMDA1MDkwMSAocHJlcmVsZWFzZSkgKFNVU0UgTGludXgpCgpoZXJlIGlzIHRoZSBEVAoK
Q1JFQVRFIFRBQkxFICJ2ZXJzaW9uQSIubXlpbnRhcnJheV90YWJsZV9ub251bGxzCigKICBpZCBp
bnRlZ2VyLAogIG15aW50YXJyYXlfaW50NCBpbnRlZ2VyW10KKQpXSVRIT1VUIE9JRFM7CgpDUkVB
VEUgSU5ERVggaWR4X25vbm51bGxzX215aW50YXJyYXlfaW50NF9naW4KICBPTiAidmVyc2lvbkEi
Lm15aW50YXJyYXlfdGFibGVfbm9udWxscwogIFVTSU5HIGdpbgogIChteWludGFycmF5X2ludDQp
OwoKdGhlcmUgYXJlIDc0NTk4OSByZWNvcmRzIGluIHRoZSB0YWJsZSB3aXRoIG5vIG51bGwgdmFs
dWVzIGZvciB0aGUKbXlpbnRhcnJheV9pbnQ0CmZpZWxkLgoKU28gaGVyZSBpcyB0aGUgZXhlY3V0
aW9uIHBsYW4KCm15dmlkZW9pbmRleD0jIGV4cGxhaW4gYW5hbHl6ZSBTRUxFQ1QgaWQsIGljb3Vu
dChteWludGFycmF5X2ludDQpCiAgRlJPTSAidmVyc2lvbkEiLm15aW50YXJyYXlfdGFibGVfbm9u
dWxscwogV0hFUkUgQVJSQVlbOF0gPEAgbXlpbnRhcnJheV9pbnQ0OwogICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIFFVRVJZIFBM
QU4KLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0KIFNlcSBTY2FuIG9uIG15aW50YXJyYXlfdGFi
bGVfbm9udWxscwooY29zdD0xMDAwMDAwMDAuMDAuLjEwMDAxNTI2Ny43M3Jvd3M9NzQ2IHdpZHRo
PTMyKSAoYWN0dWFsIHRpbWU9CjAuMDc5Li4xMTU2LjM5MyByb3dzPTI4MjA3IGxvb3BzPTEpCiAg
IEZpbHRlcjogKCd7OH0nOjppbnRlZ2VyW10gPEAgbXlpbnRhcnJheV9pbnQ0KQogVG90YWwgcnVu
dGltZTogMTI2Ni4zNDYgbXMKKDMgcm93cykKClRoZW4gSSBkcm9wIHRoZSBHSU4gYW5kIGNyZWF0
ZSBhIEdpU1QgaW5kZXgKCkRST1AgSU5ERVggInZlcnNpb25BIi5pZHhfbm9ubnVsbHNfbXlpbnRh
cnJheV9pbnQ0X2dpbjsKCkNSRUFURSBJTkRFWCBpZHhfbm9ubnVsbHNfbXlpbnRhcnJheV9pbnQ0
X2dpc3QKICBPTiAidmVyc2lvbkEiLm15aW50YXJyYXlfdGFibGVfbm9udWxscwogIFVTSU5HIGdp
c3QKICAobXlpbnRhcnJheV9pbnQ0KTsKCmFuZCBoZXJlIGFyZSB0aGUgcmVzdWx0cyBmb3IgdGhl
IGV4ZWN1dGlvbiBwbGFuCgpteXZpZGVvaW5kZXg9IyBleHBsYWluIGFuYWx5emUgU0VMRUNUIGlk
LCBpY291bnQobXlpbnRhcnJheV9pbnQ0KQpteXZpZGVvaW5kZXgtIyAgIEZST00gInZlcnNpb25B
Ii5teWludGFycmF5X3RhYmxlX25vbnVsbHMKbXl2aWRlb2luZGV4LSMgIFdIRVJFIEFSUkFZWzhd
IDxAIG15aW50YXJyYXlfaW50NDsKICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgUVVFUlkKUExBTgotLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLQogQml0bWFwIEhlYXAgU2NhbiBvbiBteWludGFycmF5X3Rh
YmxlX25vbnVsbHMgIChjb3N0PTQyLjM2Li4yMTM3LjYyIHJvd3M9NzQ2CndpZHRoPTMyKSAoYWN0
dWFsIHRpbWU9MTU0LjI3Ni4uMzAxLjYxNSByb3dzPTI4MjA3IGxvb3BzPTEpCiAgIFJlY2hlY2sg
Q29uZDogKCd7OH0nOjppbnRlZ2VyW10gPEAgbXlpbnRhcnJheV9pbnQ0KQogICAtPiAgQml0bWFw
IEluZGV4IFNjYW4gb24gaWR4X25vbm51bGxzX215aW50YXJyYXlfaW50NF9naXN0ICAoY29zdD0K
MC4wMC4uNDIuMTcgcm93cz03NDYgd2lkdGg9MCkgKGFjdHVhbCB0aW1lPTE1MC43MTMuLjE1MC43
MTMgcm93cz0yODIwNwpsb29wcz0xKQogICAgICAgICBJbmRleCBDb25kOiAoJ3s4fSc6OmludGVn
ZXJbXSA8QCBteWludGFycmF5X2ludDQpCiBUb3RhbCBydW50aW1lOiA0MTAuMzk0IG1zCig1IHJv
d3MpCgpBcyB5b3UgY2FuIHNlZSB0aGUgaW5kZXggaXMgaW4gdXNlLi4uCgpOb3cgSSBjcmVhdGUg
Y3JlYXRlIHRoZSBzYW1lIHRhYmxlIHdpdGggbXlpbnRhcnJheV9pbnQ0IGNvbnZlcnRlZCBpbnRv
IHRleHQKYXJyYXkgYW5kIGNyZWF0ZSBhIEdJTiBpbmRleCBvbiB0aGUgbmV3IHRleHQgYXJyYXkg
ZmllbGQKClNFTEVDVCBpZCwgbXlpbnRhcnJheV9pbnQ0Ojp0ZXh0W10gYXMgbXlpbnRhcnJheV9p
bnQ0X3RleHQgaW50bwpteWludGFycmF5X3RhYmxlX25vbnVsbHNfdGV4dCBmcm9tIG15aW50YXJy
YXlfdGFibGVfbm9udWxsczsKCkNSRUFURSBJTkRFWCBpZHhfbm9ubnVsbHNfbXlpbnRhcnJheV9p
bnQ0X3RleHRfZ2luCiAgT04gInZlcnNpb25BIi5teWludGFycmF5X3RhYmxlX25vbnVsbHNfdGV4
dAogIFVTSU5HIGdpbgogIChteWludGFycmF5X2ludDRfdGV4dCk7CgphbmQgaGF2ZSBhIHRhYmxl
IHdpdGggRFQ6CgpDUkVBVEUgVEFCTEUgInZlcnNpb25BIi5teWludGFycmF5X3RhYmxlX25vbnVs
bHNfdGV4dAooCiAgaWQgaW50ZWdlciwKICBteWludGFycmF5X2ludDRfdGV4dCB0ZXh0W10KKQpX
SVRIT1VUIE9JRFM7CgpOb3cgdGhlIHNhbWUgcmVxdWVzdCBoYXMgdGhlIGZvbGxvd2luZyBleGVj
dXRpb24gcGxhbjoKCm15dmlkZW9pbmRleD0jIGV4cGxhaW4gYW5hbHl6ZSBTRUxFQ1QgaWQsIGFy
cmF5X3VwcGVyKCBteWludGFycmF5X2ludDRfdGV4dCwKMSApCiAgRlJPTSAidmVyc2lvbkEiLm15
aW50YXJyYXlfdGFibGVfbm9udWxsc190ZXh0CiBXSEVSRSBBUlJBWVsnOCddIDxAIG15aW50YXJy
YXlfaW50NF90ZXh0OwogICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAg
ICAgICAgICAgICAgICAgICAgICAgICAgICAgUVVFUlkKUExBTgotLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0KIEJpdG1hcCBIZWFwIFNjYW4gb24gbXlpbnRhcnJheV90YWJsZV9u
b251bGxzX3RleHQKKGNvc3Q9MTAuMDYuLjIxMzYuOTdyb3dzPTc0NiB3aWR0aD0zNykgKGFjdHVh
bCB0aW1lPQoxNy40NjMuLjE5MS4wOTQgcm93cz0yODIwNyBsb29wcz0xKQogICBSZWNoZWNrIENv
bmQ6ICgnezh9Jzo6dGV4dFtdIDxAIG15aW50YXJyYXlfaW50NF90ZXh0KQogICAtPiAgQml0bWFw
IEluZGV4IFNjYW4gb24gaWR4X25vbm51bGxzX215aW50YXJyYXlfaW50NF90ZXh0X2dpbiAgKGNv
c3Q9CjAuMDAuLjkuODcgcm93cz03NDYgd2lkdGg9MCkgKGFjdHVhbCB0aW1lPTEzLjk4Mi4uMTMu
OTgyIHJvd3M9MjgyMDcgbG9vcHM9MSkKICAgICAgICAgSW5kZXggQ29uZDogKCd7OH0nOjp0ZXh0
W10gPEAgbXlpbnRhcnJheV9pbnQ0X3RleHQpCiBUb3RhbCBydW50aW1lOiAzMDMuMzQ4IG1zCig1
IHJvd3MpCgoKSSBob3BlIHRoaXMgaW5mb3JtYXRpb24gd2lsbCBtYWtlIHRoZSBxdWVzdGlvbiBt
b3JlIHVuZGVyc3RhbmRhYmxlLgoKV2l0aCBiZXN0IHJlZ2FyZHMsCgotLSBWYWxlbnRpbmUKCgoK
T24gNS85LzA3LCBPbGVnIEJhcnR1bm92IDxvbGVnQHNhaS5tc3Uuc3U+IHdyb3RlOgo+Cj4gT24g
V2VkLCA5IE1heSAyMDA3LCBWYWxlbnRpbmUgR29naWNoYXNodmlsaSB3cm90ZToKPgo+ID4gSSBo
YXZlIGV4cGVyaW1lbnRlZCBxdWl0ZSBhIGxvdC4gU28gZmlyc3QgSSBkaWQgd2hlbiBzdGFydGlu
ZyB0aGUKPiBhdHRlbXB0IHRvCj4gPiBtb3ZlIGZyb20gR2lTVCB0byBHSU4sIHdhcyB0byBkcm9w
IHRoZSBHaVNUIGluZGV4IGFuZCBjcmVhdGUgYSBicmFuZCBuZXcKPiBHSU4KPiA+IGluZGV4Li4u
IGFmdGVyIHRoYXQgZGlkIG5vdCBicmluZyB0aGUgcmVzdWx0cywgSSBzdGFydGVkIHRvIGNyZWF0
ZSBhbGwKPiB0aGlzCj4gPiB0YWJsZXMgd2l0aCBkaWZmZXJlbnQgc2V0cyBvZiBpbmRleGVzIGFu
ZCBzbyBvbi4uLgo+ID4KPiA+IFNvIHRoZSBhbnN3ZXIgdG8gdGhlIHF1ZXN0aW9uIGlzOiBubyB0
aGVyZSBpbiBvbmx5IEdJTiBpbmRleCBvbiB0aGUKPiB0YWJsZS4KPgo+IHRoZW4sIHlvdSBoYXZl
IHRvIHByb3ZpZGUgdXMgbW9yZSBpbmZvbWF0aW9uIC0KPiBwZyB2ZXJzaW9uLAo+IFxkdCBzb3Vy
Y2V0YWJsZXdpdGhfaW50NAo+IGV4cGxhaW4gYW5hbHl6ZQo+Cj4gYnR3LCBJIGRpZCB0ZXN0IG9m
IGRldmVsb3BtZW50IHZlcnNpb24gb2YgR2lOLCBzZWUKPiBodHRwOi8vd3d3LnNhaS5tc3Uuc3Uv
fm1lZ2VyYS93aWtpL0dpblRlc3QKPgo+ID4KPiA+IFRoYW5rIHlvdSBpbiBhZHZhbmNlLAo+ID4K
PiA+IFZhbGVudGluZQo+ID4KPiA+IE9uIDUvOS8wNywgT2xlZyBCYXJ0dW5vdiA8b2xlZ0BzYWku
bXN1LnN1PiB3cm90ZToKPiA+Pgo+ID4+IERvIHlvdSBoYXZlIGJvdGggaW5kZXhlcyAoR2lTVCwg
R0lOKSBvbiB0aGUgc2FtZSB0YWJsZSA/Cj4gPj4KPiA+PiBPbiBXZWQsIDkgTWF5IDIwMDcsIFZh
bGVudGluZSBHb2dpY2hhc2h2aWxpIHdyb3RlOgo+ID4+Cj4gPj4gPiBIZWxsbyBhbGwsCj4gPj4g
Pgo+ID4+ID4gSSBhbSB0cnlpbmcgdG8gbW92ZSBmcm9tIEdpU1QgaW50YXJyYXkgaW5kZXggdG8g
R0lOIGludGFycmF5IGluZGV4LAo+IGJ1dAo+ID4+IG15Cj4gPj4gPiBHSU4gaW5kZXggaXMgbm90
IGJlaW5nIHVzZWQgYnkgdGhlIHBsYW5uZXIuCj4gPj4gPgo+ID4+ID4gVGhlIG5vcm1hbCBxdWVy
eSBpcyBsaWtlIHRoYXQKPiA+PiA+Cj4gPj4gPiBzZWxlY3QgKgo+ID4+ID4gZnJvbSBzb3VyY2V0
YWJsZXdpdGhfaW50NAo+ID4+ID4gd2hlcmUgQVJSQVlbbXlpbnRdIDxAIG15aW50X2FycmF5Cj4g
Pj4gPiAgYW5kIHNvbWVfb3RoZXJfZmlsdGVycwo+ID4+ID4KPiA+PiA+ICh3aXRoIEdpU1QgaW5k
ZXggZXZlcnl0aGluZyB3b3JrcyBmaW5lLCBidXQgR0lOIGluZGV4IGlzIG5vdCBiZWluZwo+IHVz
ZWQpCj4gPj4gPgo+ID4+ID4gSWYgSSBjcmVhdGUgdGhlIHNhbWUgdGFibGUgcG9wdWxhdGluZyBp
dCB3aXRoIHRleHRbXSBkYXRhIGxpa2UKPiA+PiA+Cj4gPj4gPiBzZWxlY3QgbXlpbnRfYXJyYXk6
OnRleHRbXSBhcyBteWludF9hcnJheV9hc190ZXh0YXJyYXkKPiA+PiA+IGludG8gbmV3dGFibGV3
aXRoX3RleHQKPiA+PiA+IGZyb20gc291cmNldGFibGV3aXRoX2ludDQKPiA+PiA+Cj4gPj4gPiBh
bmQgdGhlbiBjcmVhdGUgYSBHSU4gaW5kZXggdXNpbmcgdGhpcyBuZXcgdGV4dFtdIGNvbHVtbgo+
ID4+ID4KPiA+PiA+IHRoZSBwbGFubmVyIHN0YXJ0cyB0byB1c2UgdGhlIGluZGV4IGFuZCBxdWVy
aWVzIHJ1biB3aXRoIGdyYXRlIHNwZWVkCj4gPj4gd2hlbgo+ID4+ID4gdGhlIHF1ZXJ5IGxvb2tz
IGxpa2UgdGhhdDoKPiA+PiA+Cj4gPj4gPiBzZWxlY3QgKgo+ID4+ID4gZnJvbSBuZXd0YWJsZXdp
dGhfdGV4dAo+ID4+ID4gd2hlcmUgQVJSQVlbJ215aW50J10gPEAgbXlpbnRfYXJyYXlfYXNfdGV4
dGFycmF5Cj4gPj4gPiAgYW5kIHNvbWVfb3RoZXJfZmlsdGVycwo+ID4+ID4KPiA+PiA+IFdoZXJl
IHRoZSBwcm9ibGVtIGNhbiBiZSB3aXRoIF9pbnQ0IEdJTiBpbmRleCBpbiB0aGlzIGNvbnN0ZWxs
YXRpb24/Cj4gPj4gPgo+ID4+ID4gYnkgbm93IHRoZSBlbmFibGVfc2Vxc2NhbiBpcyBzZXQgdG8g
b2ZmIGluIHRoZSBjb25maWd1cmF0aW9uLgo+ID4+ID4KPiA+PiA+IFdpdGggYmVzdCByZWdhcmRz
LAo+ID4+ID4KPiA+PiA+IC0tIFZhbGVudGluZSBHb2dpY2hhc2h2aWxpCj4gPj4gPgo+ID4+Cj4g
Pj4gICAgICAgICBSZWdhcmRzLAo+ID4+ICAgICAgICAgICAgICAgICBPbGVnCj4gPj4gX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+
ID4+IE9sZWcgQmFydHVub3YsIFJlc2VhcmNoIFNjaWVudGlzdCwgSGVhZCBvZiBBc3Ryb05ldCAo
d3d3LmFzdHJvbmV0LnJ1KSwKPiA+PiBTdGVybmJlcmcgQXN0cm9ub21pY2FsIEluc3RpdHV0ZSwg
TW9zY293IFVuaXZlcnNpdHksIFJ1c3NpYQo+ID4+IEludGVybmV0OiBvbGVnQHNhaS5tc3Uuc3Us
IGh0dHA6Ly93d3cuc2FpLm1zdS5zdS9+bWVnZXJhLwo+ID4+IHBob25lOiArMDA3KDQ5NSk5Mzkt
MTYtODMsICswMDcoNDk1KTkzOS0yMy04Mwo+ID4+Cj4gPgo+ID4KPiA+Cj4gPgo+Cj4gICAgICAg
ICBSZWdhcmRzLAo+ICAgICAgICAgICAgICAgICBPbGVnCj4gX19fX19fX19fX19fX19fX19fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fXwo+IE9sZWcgQmFydHVub3Ys
IFJlc2VhcmNoIFNjaWVudGlzdCwgSGVhZCBvZiBBc3Ryb05ldCAod3d3LmFzdHJvbmV0LnJ1KSwK
PiBTdGVybmJlcmcgQXN0cm9ub21pY2FsIEluc3RpdHV0ZSwgTW9zY293IFVuaXZlcnNpdHksIFJ1
c3NpYQo+IEludGVybmV0OiBvbGVnQHNhaS5tc3Uuc3UsIGh0dHA6Ly93d3cuc2FpLm1zdS5zdS9+
bWVnZXJhLwo+IHBob25lOiArMDA3KDQ5NSk5MzktMTYtODMsICswMDcoNDk1KTkzOS0yMy04Mwo+
CgoKCi0tIArhg5Xhg5Dhg5rhg5Thg5zhg6Lhg5jhg5wg4YOS4YOd4YOS4YOY4YOp4YOQ4YOo4YOV
4YOY4YOa4YOYClZhbGVudGluZSBHb2dpY2hhc2h2aWxpCg==
------=_Part_110820_33458115.1178736067834
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: base64
Content-Disposition: inline

SGkgYWdhaW4sIDxicj48YnI+dGhlIHZlcnNpb24gb2YgdGhlIHNlcnZlciBJIGFtIG9uIGlzIDxm
b250IHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyIgc2l6ZT0iMSI+
UG9zdGdyZVNRTCA4LjIuMyBvbiBpNjg2LXBjLWxpbnV4LWdudSwgY29tcGlsZWQgYnkgR0NDIGdj
YyAoR0NDKSA0LjAuMiAyMDA1MDkwMSAocHJlcmVsZWFzZSkgKFNVU0UgTGludXgpPC9mb250Pgo8
YnI+PGJyPmhlcmUgaXMgdGhlIERUPGJyPjxmb250IHNpemU9IjEiPjxiciBzdHlsZT0iZm9udC1m
YW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTog
Y291cmllciBuZXcsbW9ub3NwYWNlOyI+Q1JFQVRFIFRBQkxFICZxdW90O3ZlcnNpb25BJnF1b3Q7
Lm15aW50YXJyYXlfdGFibGVfbm9udWxsczwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBj
b3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVy
IG5ldyxtb25vc3BhY2U7Ij4oPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIg
bmV3LG1vbm9zcGFjZTsiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9u
b3NwYWNlOyI+Jm5ic3A7IGlkIGludGVnZXIsPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6
IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJp
ZXIgbmV3LG1vbm9zcGFjZTsiPiZuYnNwOyBteWludGFycmF5X2ludDQgaW50ZWdlcltdPC9zcGFu
PjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPjxzcGFuIHN0
eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+KSA8L3NwYW4+PGJyIHN0
eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJm
b250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+V0lUSE9VVCBPSURTOzwvc3Bhbj48
c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPjwvc3Bhbj48
YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48YnIgc3R5bGU9
ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQt
ZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48L3NwYW4+PHNwYW4gc3R5bGU9ImZvbnQt
ZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5DUkVBVEUgSU5ERVggaWR4X25vbm51bGxz
X215aW50YXJyYXlfaW50NF9naW48L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmll
ciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxt
b25vc3BhY2U7Ij4KJm5ic3A7IE9OICZxdW90O3ZlcnNpb25BJnF1b3Q7Lm15aW50YXJyYXlfdGFi
bGVfbm9udWxsczwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25v
c3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsi
PiZuYnNwOyBVU0lORyBnaW48L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBu
ZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9u
b3NwYWNlOyI+Jm5ic3A7IChteWludGFycmF5X2ludDQpOzwvc3Bhbj48YnIgc3R5bGU9ImZvbnQt
ZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48L2ZvbnQ+PGJyPnRoZXJlIGFyZSA3NDU5
ODkgcmVjb3JkcyBpbiB0aGUgdGFibGUgd2l0aCBubyBudWxsIHZhbHVlcyBmb3IgdGhlIDxmb250
IHNpemU9IjEiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNl
OyI+Cm15aW50YXJyYXlfaW50NCA8L3NwYW4+PC9mb250PjxzcGFuPmZpZWxkPC9zcGFuPjxmb250
IHNpemU9IjEiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNl
OyI+Ljwvc3Bhbj48L2ZvbnQ+PGJyPjxicj5TbyBoZXJlIGlzIHRoZSBleGVjdXRpb24gcGxhbjxi
cj48YnI+PGZvbnQgc2l6ZT0iMSI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5l
dyxtb25vc3BhY2U7Ij4KbXl2aWRlb2luZGV4PSMgZXhwbGFpbiBhbmFseXplIFNFTEVDVCBpZCwg
aWNvdW50KG15aW50YXJyYXlfaW50NCk8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291
cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5l
dyxtb25vc3BhY2U7Ij4mbmJzcDsgRlJPTSAmcXVvdDt2ZXJzaW9uQSZxdW90Oy5teWludGFycmF5
X3RhYmxlX25vbnVsbHMKPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3
LG1vbm9zcGFjZTsiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3Nw
YWNlOyI+Jm5ic3A7V0hFUkUgQVJSQVlbOF0gJmx0O0AgbXlpbnRhcnJheV9pbnQ0Ozwvc3Bhbj48
YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHls
ZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgombmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgUVVFUlkgUExBTjwvc3Bhbj48YnIgc3R5bGU9ImZv
bnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1p
bHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPi0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tCjwv
c3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3Bh
biBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPiZuYnNwO1NlcSBT
Y2FuIG9uIG15aW50YXJyYXlfdGFibGVfbm9udWxscyZuYnNwOyAoY29zdD0xMDAwMDAwMDAuMDAu
LjEwMDAxNTI2Ny43MyByb3dzPTc0NiB3aWR0aD0zMikgKGFjdHVhbCB0aW1lPTAuMDc5Li4xMTU2
LjM5Mwogcm93cz0yODIwNyBsb29wcz0xKTwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBj
b3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIg
bmV3LG1vbm9zcGFjZTsiPiZuYnNwOyZuYnNwOyBGaWx0ZXI6ICgmIzM5O3s4fSYjMzk7OjppbnRl
Z2VyW10gJmx0O0AgbXlpbnRhcnJheV9pbnQ0KTwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5
OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3Vy
aWVyIG5ldyxtb25vc3BhY2U7Ij4mbmJzcDtUb3RhbCBydW50aW1lOiAxMjY2LjM0NiBtczwvc3Bh
bj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBz
dHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPigzIHJvd3MpPC9zcGFu
PjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8L2ZvbnQ+
PGJyPlRoZW4gSSBkcm9wIHRoZSBHSU4gYW5kIGNyZWF0ZSBhIEdpU1QgaW5kZXg8YnI+PGJyPjxm
b250IHNpemU9IjEiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3Nw
YWNlOyI+RFJPUCBJTkRFWCAmcXVvdDt2ZXJzaW9uQSZxdW90Oy5pZHhfbm9ubnVsbHNfbXlpbnRh
cnJheV9pbnQ0X2dpbjs8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcs
bW9ub3NwYWNlOyI+CjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFj
ZTsiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+Q1JF
QVRFIElOREVYIGlkeF9ub25udWxsc19teWludGFycmF5X2ludDRfZ2lzdDwvc3Bhbj48YnIgc3R5
bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9u
dC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgombmJzcDsgT04gJnF1b3Q7dmVyc2lv
bkEmcXVvdDsubXlpbnRhcnJheV90YWJsZV9ub251bGxzPC9zcGFuPjxiciBzdHlsZT0iZm9udC1m
YW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTog
Y291cmllciBuZXcsbW9ub3NwYWNlOyI+Jm5ic3A7IFVTSU5HIGdpc3Q8L3NwYW4+PGJyIHN0eWxl
PSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJmb250
LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+Jm5ic3A7IChteWludGFycmF5X2ludDQp
Ozwvc3Bhbj48L2ZvbnQ+PGJyPjxicj5hbmQgaGVyZSBhcmUgdGhlIHJlc3VsdHMgZm9yIHRoZSBl
eGVjdXRpb24gcGxhbjxicj48YnI+PGZvbnQgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5l
dyxtb25vc3BhY2U7IiBzaXplPSIxIj5teXZpZGVvaW5kZXg9IyBleHBsYWluIGFuYWx5emUgU0VM
RUNUIGlkLCBpY291bnQobXlpbnRhcnJheV9pbnQ0KQo8YnI+bXl2aWRlb2luZGV4LSMmbmJzcDsm
bmJzcDsgRlJPTSAmcXVvdDt2ZXJzaW9uQSZxdW90Oy5teWludGFycmF5X3RhYmxlX25vbnVsbHM8
YnI+bXl2aWRlb2luZGV4LSMmbmJzcDsgV0hFUkUgQVJSQVlbOF0gJmx0O0AgbXlpbnRhcnJheV9p
bnQ0Ozxicj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsm
bmJzcDsmbmJzcDsmbmJzcDsgUVVFUlkgUExBTjxicj4tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLQo8YnI+Jm5ic3A7Qml0bWFwIEhlYXAgU2NhbiBvbiBteWludGFycmF5X3RhYmxl
X25vbnVsbHMmbmJzcDsgKGNvc3Q9NDIuMzYuLjIxMzcuNjIgcm93cz03NDYgd2lkdGg9MzIpIChh
Y3R1YWwgdGltZT0xNTQuMjc2Li4zMDEuNjE1IHJvd3M9MjgyMDcgbG9vcHM9MSk8YnI+Jm5ic3A7
Jm5ic3A7IFJlY2hlY2sgQ29uZDogKCYjMzk7ezh9JiMzOTs6OmludGVnZXJbXSAmbHQ7QCBteWlu
dGFycmF5X2ludDQpPGJyPiZuYnNwOyZuYnNwOyAtJmd0OyZuYnNwOyBCaXRtYXAgSW5kZXggU2Nh
biBvbiBpZHhfbm9ubnVsbHNfbXlpbnRhcnJheV9pbnQ0X2dpc3QmbmJzcDsgKGNvc3Q9CjAuMDAu
LjQyLjE3IHJvd3M9NzQ2IHdpZHRoPTApIChhY3R1YWwgdGltZT0xNTAuNzEzLi4xNTAuNzEzIHJv
d3M9MjgyMDcgbG9vcHM9MSk8YnI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7
Jm5ic3A7Jm5ic3A7IEluZGV4IENvbmQ6ICgmIzM5O3s4fSYjMzk7OjppbnRlZ2VyW10gJmx0O0Ag
bXlpbnRhcnJheV9pbnQ0KTxicj4mbmJzcDtUb3RhbCBydW50aW1lOiA0MTAuMzk0IG1zPGJyPig1
IHJvd3MpPC9mb250Pjxicj48YnI+QXMgeW91IGNhbiBzZWUgdGhlIGluZGV4IGlzIGluIHVzZS4u
LiAKPGJyPjxicj5Ob3cgSSBjcmVhdGUgY3JlYXRlIHRoZSBzYW1lIHRhYmxlIHdpdGggbXlpbnRh
cnJheV9pbnQ0IGNvbnZlcnRlZCBpbnRvIHRleHQgYXJyYXkgYW5kIGNyZWF0ZSBhIEdJTiBpbmRl
eCBvbiB0aGUgbmV3IHRleHQgYXJyYXkgZmllbGQ8YnI+PGJyPjxmb250IHNpemU9IjEiPjxzcGFu
IHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+U0VMRUNUIGlkLCBt
eWludGFycmF5X2ludDQ6OnRleHRbXSBhcyBteWludGFycmF5X2ludDRfdGV4dCBpbnRvIG15aW50
YXJyYXlfdGFibGVfbm9udWxsc190ZXh0IGZyb20gbXlpbnRhcnJheV90YWJsZV9ub251bGxzOwo8
L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PGJy
IHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9
ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5DUkVBVEUgSU5ERVggaWR4X25v
bm51bGxzX215aW50YXJyYXlfaW50NF90ZXh0X2dpbjwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFt
aWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBj
b3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4mbmJzcDsgT04gJnF1b3Q7dmVyc2lvbkEmcXVvdDsubXlp
bnRhcnJheV90YWJsZV9ub251bGxzX3RleHQ8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTog
Y291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVy
IG5ldyxtb25vc3BhY2U7Ij4mbmJzcDsgVVNJTkcgZ2luCjwvc3Bhbj48YnIgc3R5bGU9ImZvbnQt
ZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6
IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPiZuYnNwOyAobXlpbnRhcnJheV9pbnQ0X3RleHQpOzwv
c3Bhbj48L2ZvbnQ+PGJyPjxicj5hbmQgaGF2ZSBhIHRhYmxlIHdpdGggRFQ6PGJyPjxicj48Zm9u
dCBzaXplPSIxIj48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFj
ZTsiPgpDUkVBVEUgVEFCTEUgJnF1b3Q7dmVyc2lvbkEmcXVvdDsubXlpbnRhcnJheV90YWJsZV9u
b251bGxzX3RleHQ8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9u
b3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7
Ij4oPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsi
Pgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPiZuYnNw
OyBpZCBpbnRlZ2VyLDwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxt
b25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFj
ZTsiPiZuYnNwOyBteWludGFycmF5X2ludDRfdGV4dCB0ZXh0W108L3NwYW4+PGJyIHN0eWxlPSJm
b250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJmb250LWZh
bWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+KSA8L3NwYW4+PGJyIHN0eWxlPSJmb250LWZh
bWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBj
b3VyaWVyIG5ldyxtb25vc3BhY2U7Ij5XSVRIT1VUIE9JRFM7PC9zcGFuPjwvZm9udD48YnI+PGJy
Pk5vdyB0aGUgc2FtZSByZXF1ZXN0IGhhcyB0aGUgZm9sbG93aW5nIGV4ZWN1dGlvbiBwbGFuOgo8
YnI+PGJyPjxmb250IHNpemU9IjEiPjxzcGFuIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBu
ZXcsbW9ub3NwYWNlOyI+bXl2aWRlb2luZGV4PSMgZXhwbGFpbiBhbmFseXplIFNFTEVDVCBpZCwg
YXJyYXlfdXBwZXIoIG15aW50YXJyYXlfaW50NF90ZXh0LCAxICk8L3NwYW4+PGJyIHN0eWxlPSJm
b250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFt
aWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KJm5ic3A7IEZST00gJnF1b3Q7dmVyc2lvbkEm
cXVvdDsubXlpbnRhcnJheV90YWJsZV9ub251bGxzX3RleHQ8L3NwYW4+PGJyIHN0eWxlPSJmb250
LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5
OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4mbmJzcDtXSEVSRSBBUlJBWVsmIzM5OzgmIzM5O10g
Jmx0O0AgbXlpbnRhcnJheV9pbnQ0X3RleHQ7PC9zcGFuPgo8YnIgc3R5bGU9ImZvbnQtZmFtaWx5
OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJp
ZXIgbmV3LG1vbm9zcGFjZTsiPiZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBRVUVSWSBQTEFOPC9zcGFuPjxiciBz
dHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgo8c3BhbiBzdHlsZT0i
Zm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPi0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t
LS0tLS0tLS0tLS0tLS0tLTwvc3Bhbj48YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5l
dyxtb25vc3BhY2U7Ij4KPHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25v
c3BhY2U7Ij4mbmJzcDtCaXRtYXAgSGVhcCBTY2FuIG9uIG15aW50YXJyYXlfdGFibGVfbm9udWxs
c190ZXh0Jm5ic3A7IChjb3N0PTEwLjA2Li4yMTM2Ljk3IHJvd3M9NzQ2IHdpZHRoPTM3KSAoYWN0
dWFsIHRpbWU9MTcuNDYzLi4xOTEuMDk0IHJvd3M9MjgyMDcgbG9vcHM9MSk8L3NwYW4+PGJyIHN0
eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+CjxzcGFuIHN0eWxlPSJm
b250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+Jm5ic3A7Jm5ic3A7IFJlY2hlY2sg
Q29uZDogKCYjMzk7ezh9JiMzOTs6OnRleHRbXSAmbHQ7QCBteWludGFycmF5X2ludDRfdGV4dCk8
L3NwYW4+PGJyIHN0eWxlPSJmb250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNw
YW4gc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4KJm5ic3A7Jm5i
c3A7IC0mZ3Q7Jm5ic3A7IEJpdG1hcCBJbmRleCBTY2FuIG9uIGlkeF9ub25udWxsc19teWludGFy
cmF5X2ludDRfdGV4dF9naW4mbmJzcDsgKGNvc3Q9MC4wMC4uOS44NyByb3dzPTc0NiB3aWR0aD0w
KSAoYWN0dWFsIHRpbWU9MTMuOTgyLi4xMy45ODIgcm93cz0yODIwNyBsb29wcz0xKTwvc3Bhbj48
YnIgc3R5bGU9ImZvbnQtZmFtaWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij48c3BhbiBzdHls
ZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPgombmJzcDsmbmJzcDsmbmJz
cDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsgSW5kZXggQ29uZDogKCYjMzk7ezh9JiMz
OTs6OnRleHRbXSAmbHQ7QCBteWludGFycmF5X2ludDRfdGV4dCk8L3NwYW4+PGJyIHN0eWxlPSJm
b250LWZhbWlseTogY291cmllciBuZXcsbW9ub3NwYWNlOyI+PHNwYW4gc3R5bGU9ImZvbnQtZmFt
aWx5OiBjb3VyaWVyIG5ldyxtb25vc3BhY2U7Ij4mbmJzcDtUb3RhbCBydW50aW1lOiAzMDMuMzQ4
IG1zPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsi
Pgo8c3BhbiBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPig1IHJv
d3MpPC9zcGFuPjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsi
PjxiciBzdHlsZT0iZm9udC1mYW1pbHk6IGNvdXJpZXIgbmV3LG1vbm9zcGFjZTsiPjwvZm9udD48
YnI+SSBob3BlIHRoaXMgaW5mb3JtYXRpb24gd2lsbCBtYWtlIHRoZSBxdWVzdGlvbiBtb3JlIHVu
ZGVyc3RhbmRhYmxlLgo8YnI+PGJyPldpdGggYmVzdCByZWdhcmRzLCA8YnI+PGJyPi0tIFZhbGVu
dGluZTxicj48YnI+PGJyPjxicj48ZGl2PjxzcGFuIGNsYXNzPSJnbWFpbF9xdW90ZSI+T24gNS85
LzA3LCA8YiBjbGFzcz0iZ21haWxfc2VuZGVybmFtZSI+T2xlZyBCYXJ0dW5vdjwvYj4gJmx0Ozxh
IGhyZWY9Im1haWx0bzpvbGVnQHNhaS5tc3Uuc3UiPm9sZWdAc2FpLm1zdS5zdTwvYT4mZ3Q7IHdy
b3RlOjwvc3Bhbj4KPGJsb2NrcXVvdGUgY2xhc3M9ImdtYWlsX3F1b3RlIiBzdHlsZT0iYm9yZGVy
LWxlZnQ6IDFweCBzb2xpZCByZ2IoMjA0LCAyMDQsIDIwNCk7IG1hcmdpbjogMHB0IDBwdCAwcHQg
MC44ZXg7IHBhZGRpbmctbGVmdDogMWV4OyI+T24gV2VkLCA5IE1heSAyMDA3LCBWYWxlbnRpbmUg
R29naWNoYXNodmlsaSB3cm90ZTo8YnI+PGJyPiZndDsgSSBoYXZlIGV4cGVyaW1lbnRlZCBxdWl0
ZSBhIGxvdC4gU28gZmlyc3QgSSBkaWQgd2hlbiBzdGFydGluZyB0aGUgYXR0ZW1wdCB0bwo8YnI+
Jmd0OyBtb3ZlIGZyb20gR2lTVCB0byBHSU4sIHdhcyB0byBkcm9wIHRoZSBHaVNUIGluZGV4IGFu
ZCBjcmVhdGUgYSBicmFuZCBuZXcgR0lOPGJyPiZndDsgaW5kZXguLi4gYWZ0ZXIgdGhhdCBkaWQg
bm90IGJyaW5nIHRoZSByZXN1bHRzLCBJIHN0YXJ0ZWQgdG8gY3JlYXRlIGFsbCB0aGlzPGJyPiZn
dDsgdGFibGVzIHdpdGggZGlmZmVyZW50IHNldHMgb2YgaW5kZXhlcyBhbmQgc28gb24uLi4KPGJy
PiZndDs8YnI+Jmd0OyBTbyB0aGUgYW5zd2VyIHRvIHRoZSBxdWVzdGlvbiBpczogbm8gdGhlcmUg
aW4gb25seSBHSU4gaW5kZXggb24gdGhlIHRhYmxlLjxicj48YnI+dGhlbiwgeW91IGhhdmUgdG8g
cHJvdmlkZSB1cyBtb3JlIGluZm9tYXRpb24gLTxicj5wZyB2ZXJzaW9uLDxicj5cZHQgc291cmNl
dGFibGV3aXRoX2ludDQ8YnI+ZXhwbGFpbiBhbmFseXplPGJyPjxicj5idHcsIEkgZGlkIHRlc3Qg
b2YgZGV2ZWxvcG1lbnQgdmVyc2lvbiBvZiBHaU4sIHNlZQo8YnI+PGEgaHJlZj0iaHR0cDovL3d3
dy5zYWkubXN1LnN1L35tZWdlcmEvd2lraS9HaW5UZXN0Ij5odHRwOi8vd3d3LnNhaS5tc3Uuc3Uv
fm1lZ2VyYS93aWtpL0dpblRlc3Q8L2E+PGJyPjxicj4mZ3Q7PGJyPiZndDsgVGhhbmsgeW91IGlu
IGFkdmFuY2UsPGJyPiZndDs8YnI+Jmd0OyBWYWxlbnRpbmU8YnI+Jmd0Ozxicj4mZ3Q7IE9uIDUv
OS8wNywgT2xlZyBCYXJ0dW5vdiAmbHQ7PGEgaHJlZj0ibWFpbHRvOm9sZWdAc2FpLm1zdS5zdSI+
Cm9sZWdAc2FpLm1zdS5zdTwvYT4mZ3Q7IHdyb3RlOjxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0OyBE
byB5b3UgaGF2ZSBib3RoIGluZGV4ZXMgKEdpU1QsIEdJTikgb24gdGhlIHNhbWUgdGFibGUgPzxi
cj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0OyBPbiBXZWQsIDkgTWF5IDIwMDcsIFZhbGVudGluZSBHb2dp
Y2hhc2h2aWxpIHdyb3RlOjxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0OyAmZ3Q7IEhlbGxvIGFsbCwK
PGJyPiZndDsmZ3Q7ICZndDs8YnI+Jmd0OyZndDsgJmd0OyBJIGFtIHRyeWluZyB0byBtb3ZlIGZy
b20gR2lTVCBpbnRhcnJheSBpbmRleCB0byBHSU4gaW50YXJyYXkgaW5kZXgsIGJ1dDxicj4mZ3Q7
Jmd0OyBteTxicj4mZ3Q7Jmd0OyAmZ3Q7IEdJTiBpbmRleCBpcyBub3QgYmVpbmcgdXNlZCBieSB0
aGUgcGxhbm5lci48YnI+Jmd0OyZndDsgJmd0Ozxicj4mZ3Q7Jmd0OyAmZ3Q7IFRoZSBub3JtYWwg
cXVlcnkgaXMgbGlrZSB0aGF0Cjxicj4mZ3Q7Jmd0OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgc2Vs
ZWN0ICo8YnI+Jmd0OyZndDsgJmd0OyBmcm9tIHNvdXJjZXRhYmxld2l0aF9pbnQ0PGJyPiZndDsm
Z3Q7ICZndDsgd2hlcmUgQVJSQVlbbXlpbnRdICZsdDtAIG15aW50X2FycmF5PGJyPiZndDsmZ3Q7
ICZndDsmbmJzcDsmbmJzcDthbmQgc29tZV9vdGhlcl9maWx0ZXJzPGJyPiZndDsmZ3Q7ICZndDs8
YnI+Jmd0OyZndDsgJmd0OyAod2l0aCBHaVNUIGluZGV4IGV2ZXJ5dGhpbmcgd29ya3MgZmluZSwg
YnV0IEdJTiBpbmRleCBpcyBub3QgYmVpbmcgdXNlZCkKPGJyPiZndDsmZ3Q7ICZndDs8YnI+Jmd0
OyZndDsgJmd0OyBJZiBJIGNyZWF0ZSB0aGUgc2FtZSB0YWJsZSBwb3B1bGF0aW5nIGl0IHdpdGgg
dGV4dFtdIGRhdGEgbGlrZTxicj4mZ3Q7Jmd0OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgc2VsZWN0
IG15aW50X2FycmF5Ojp0ZXh0W10gYXMgbXlpbnRfYXJyYXlfYXNfdGV4dGFycmF5PGJyPiZndDsm
Z3Q7ICZndDsgaW50byBuZXd0YWJsZXdpdGhfdGV4dAo8YnI+Jmd0OyZndDsgJmd0OyBmcm9tIHNv
dXJjZXRhYmxld2l0aF9pbnQ0PGJyPiZndDsmZ3Q7ICZndDs8YnI+Jmd0OyZndDsgJmd0OyBhbmQg
dGhlbiBjcmVhdGUgYSBHSU4gaW5kZXggdXNpbmcgdGhpcyBuZXcgdGV4dFtdIGNvbHVtbjxicj4m
Z3Q7Jmd0OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgdGhlIHBsYW5uZXIgc3RhcnRzIHRvIHVzZSB0
aGUgaW5kZXggYW5kIHF1ZXJpZXMgcnVuIHdpdGggZ3JhdGUgc3BlZWQKPGJyPiZndDsmZ3Q7IHdo
ZW48YnI+Jmd0OyZndDsgJmd0OyB0aGUgcXVlcnkgbG9va3MgbGlrZSB0aGF0Ojxicj4mZ3Q7Jmd0
OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgc2VsZWN0ICo8YnI+Jmd0OyZndDsgJmd0OyBmcm9tIG5l
d3RhYmxld2l0aF90ZXh0PGJyPiZndDsmZ3Q7ICZndDsgd2hlcmUgQVJSQVlbJiMzOTtteWludCYj
Mzk7XSAmbHQ7QCBteWludF9hcnJheV9hc190ZXh0YXJyYXkKPGJyPiZndDsmZ3Q7ICZndDsmbmJz
cDsmbmJzcDthbmQgc29tZV9vdGhlcl9maWx0ZXJzPGJyPiZndDsmZ3Q7ICZndDs8YnI+Jmd0OyZn
dDsgJmd0OyBXaGVyZSB0aGUgcHJvYmxlbSBjYW4gYmUgd2l0aCBfaW50NCBHSU4gaW5kZXggaW4g
dGhpcyBjb25zdGVsbGF0aW9uPzxicj4mZ3Q7Jmd0OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgYnkg
bm93IHRoZSBlbmFibGVfc2Vxc2NhbiBpcyBzZXQgdG8gb2ZmIGluIHRoZSBjb25maWd1cmF0aW9u
Lgo8YnI+Jmd0OyZndDsgJmd0Ozxicj4mZ3Q7Jmd0OyAmZ3Q7IFdpdGggYmVzdCByZWdhcmRzLDxi
cj4mZ3Q7Jmd0OyAmZ3Q7PGJyPiZndDsmZ3Q7ICZndDsgLS0gVmFsZW50aW5lIEdvZ2ljaGFzaHZp
bGk8YnI+Jmd0OyZndDsgJmd0Ozxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7Jmd0OyZuYnNwOyZuYnNwOyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBSZWdhcmRzLDxicj4mZ3Q7Jmd0OyZu
YnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNw
OyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyZuYnNwOyBPbGVnPGJyPiZndDsmZ3Q7IF9f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
X18KPGJyPiZndDsmZ3Q7IE9sZWcgQmFydHVub3YsIFJlc2VhcmNoIFNjaWVudGlzdCwgSGVhZCBv
ZiBBc3Ryb05ldCAoPGEgaHJlZj0iaHR0cDovL3d3dy5hc3Ryb25ldC5ydSI+d3d3LmFzdHJvbmV0
LnJ1PC9hPiksPGJyPiZndDsmZ3Q7IFN0ZXJuYmVyZyBBc3Ryb25vbWljYWwgSW5zdGl0dXRlLCBN
b3Njb3cgVW5pdmVyc2l0eSwgUnVzc2lhPGJyPiZndDsmZ3Q7IEludGVybmV0OiA8YSBocmVmPSJt
YWlsdG86b2xlZ0BzYWkubXN1LnN1Ij4Kb2xlZ0BzYWkubXN1LnN1PC9hPiwgPGEgaHJlZj0iaHR0
cDovL3d3dy5zYWkubXN1LnN1L35tZWdlcmEvIj5odHRwOi8vd3d3LnNhaS5tc3Uuc3Uvfm1lZ2Vy
YS88L2E+PGJyPiZndDsmZ3Q7IHBob25lOiArMDA3KDQ5NSk5MzktMTYtODMsICswMDcoNDk1KTkz
OS0yMy04Mzxicj4mZ3Q7Jmd0Ozxicj4mZ3Q7PGJyPiZndDs8YnI+Jmd0Ozxicj4mZ3Q7PGJyPjxi
cj4mbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDsmbmJzcDtSZWdhcmRz
LAo8YnI+Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5i
c3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7Jm5ic3A7T2xlZzxicj5fX19f
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19f
PGJyPk9sZWcgQmFydHVub3YsIFJlc2VhcmNoIFNjaWVudGlzdCwgSGVhZCBvZiBBc3Ryb05ldCAo
PGEgaHJlZj0iaHR0cDovL3d3dy5hc3Ryb25ldC5ydSI+d3d3LmFzdHJvbmV0LnJ1PC9hPiksPGJy
PlN0ZXJuYmVyZyBBc3Ryb25vbWljYWwgSW5zdGl0dXRlLCBNb3Njb3cgVW5pdmVyc2l0eSwgUnVz
c2lhCjxicj5JbnRlcm5ldDogPGEgaHJlZj0ibWFpbHRvOm9sZWdAc2FpLm1zdS5zdSI+b2xlZ0Bz
YWkubXN1LnN1PC9hPiwgPGEgaHJlZj0iaHR0cDovL3d3dy5zYWkubXN1LnN1L35tZWdlcmEvIj5o
dHRwOi8vd3d3LnNhaS5tc3Uuc3Uvfm1lZ2VyYS88L2E+PGJyPnBob25lOiArMDA3KDQ5NSk5Mzkt
MTYtODMsICswMDcoNDk1KTkzOS0yMy04Mzxicj48L2Jsb2NrcXVvdGU+PC9kaXY+PGJyPjxiciBj
bGVhcj0iYWxsIj4KPGJyPi0tIDxicj7hg5Xhg5Dhg5rhg5Thg5zhg6Lhg5jhg5wg4YOS4YOd4YOS
4YOY4YOp4YOQ4YOo4YOV4YOY4YOa4YOYPGJyPlZhbGVudGluZSBHb2dpY2hhc2h2aWxpCg==
------=_Part_110820_33458115.1178736067834--

> currently ZFS is only available on Solaris, parts of it have been released
> under GPLv2, but it doesn't look like enough of it to be ported to Linux
> (enough was released for grub to be able to access it read-only, but not
> the full filesystem). there are also patent concerns that are preventing
> any porting to Linux.

I don't know if anyone mentioned this in the thread already, but it looks
like ZFS may be coming to MacOSX 10.5

http://news.worldofapple.com/archives/2006/12/17/zfs-file-system-makes-it-to
-mac-os-x-leopard/

On Tuesday 08 May 2007 23:31, Greg Smith wrote:
> On Tue, 8 May 2007, Tom Lane wrote:
> > What Debian has done is set up an arrangement that lets you run two (or
> > more) different PG versions in parallel. Since that's amazingly helpful
> > during a major-PG-version upgrade, most of the other packagers are
> > scheming how to do something similar.
>
> I alluded to that but it is worth going into more detail on for those not
> familiar with this whole topic. I normally maintain multiple different PG
> versions in parallel already, mostly using environment variables to switch
> between them with some shell code. Debian has taken an approach where
> commands like pg_ctl are wrapped in multi-version/cluster aware scripts,
> so you can do things like restarting multiple installations more easily
> than that.
>
> My issue wasn't with the idea, it was with the implementation. When I
> have my newbie hat on, it adds a layer of complexity that isn't needed for
> simple installs.

I think I would disagree with this. The confusion comes from the fact that it
is different, not that it is more complex. For new users what seems to be
most confusing is getting from install to initdb to logging in... if you tell
them to use pg_ctlcluster rather than pg_ctl, it isn't more confusing, there
just following directions at that point anyway. If the upstream project were
to switch to debian's system, I think you'd end most of the confusion, make
it easier to run concurrent servers and simplify the upgrade process for
source installs, and give other package maintiners a way to achive what
debian has. Maybe in PG 9...

--
Robert Treat
Build A Brighter LAMP :: Linux Apache {middleware} PostgreSQL

On Tue, 2007-05-08 at 23:31 -0400, Greg Smith wrote:
>
> My issue wasn't with the idea, it was with the implementation. When I
> have my newbie hat on, it adds a layer of complexity that isn't needed for
> simple installs.

I find it very hard to agree with that.

As a newbie I install postgresql and have a database server installed,
and operating. The fact that the DB files are installed somewhere
like /var/lib/postgresql/8.1/main is waaay beyond newbie.

At that point I can "createdb" or "createuser", but I _do_ _not_ need to
know anything about the cluster stuff until there is more than one DB on
the machine.

The Debian wrappers all default appropriately for the single-cluster
case, so having a single cluster has added _no_ perceivable complexity
for a newbie (as it should).

If you have a second cluster, whether it's the same Pg version or not,
things necessarily start to get complicated. OTOH I haven't had any
problem explaining to people that the --cluster option applies, and
there are sane ways to make that default to a reasonable thing as well.

All in all I think that the Debian scripts are excellent. I'm sure
there are improvements that could be made, but overall they don't get in
the way, they do the right thing in the minimal case, and they give the
advanced user a lot more choices about multiple DB instances on the same
machine.

Cheers,
Andrew McMillan.

-------------------------------------------------------------------------
Andrew @ Catalyst .Net .NZ Ltd, PO Box 11-053, Manners St, Wellington
WEB: http://catalyst.net.nz/ PHYS: Level 2, 150-154 Willis St
DDI: +64(4)803-2201 MOB: +64(272)DEBIAN OFFICE: +64(4)499-2267
Open Source: the difference between trust and antitrust
-------------------------------------------------------------------------