Re: Amazon EC2 | Any recent developments

Hello All,

I am investigating the possibility of hosting pgsql 8.3 on Amazon EC2
& implementing a simple HA solution. My search of postgresql & amazon
cloud has produced little result. Just wondering if there has been
any recent development with EBS etc. and anybody would care to share
their experiences.

Thank you very much.

aj

On Mon, Jun 15, 2009 at 11:12:32AM -0700, AJAY A wrote:
- Hello All,
-
- I am investigating the possibility of hosting pgsql 8.3 on Amazon EC2
- & implementing a simple HA solution. My search of postgresql & amazon
- cloud has produced little result. Just wondering if there has been
- any recent development with EBS etc. and anybody would care to share
- their experiences.

We've been researching Cloud/EC2 as well. The consensus so far from RightScale,
and IBM (the first 2 we've spoken with) is that if I/O performance is important to
you, then stay away from the cloud. (it's just VMs afterall)

Dave

Hi,

I have more than a few Postgres instances on EC2. For reliability I
use EBS, and take regular snapshots while also streaming the WAL files
to S3. So far, the few times that my machine died, I had no issue with
getting it back from EBS or the EBS volume. I also take tar backups
every day, and I keep a few days back of tar, snapshots and WAL log
files.

If you require high performance you might have to look into the large
or XL instances, as their networking is a lot faster, and EBS is
accessed through the network. They also have a lot more memory.

I actually think that Postgres has a big advantage on the cloud, and
that's the ability to easily recover from crashes. Because of the way
Postgres commits to disk and work with WALs, there is much higher
chance of recovering the DB than most other DB servers out there.

Bye,

Guy.

On Mon, Jun 15, 2009 at 11:12 AM, AJAY A<aagrawalla(at)gmail(dot)com> wrote:
> Hello All,
>
> I am investigating the possibility of hosting pgsql 8.3 on Amazon EC2
> & implementing a simple HA solution.  My search of postgresql & amazon
> cloud has produced little result.  Just wondering if there has been
> any recent development with EBS etc. and anybody would care to share
> their experiences.
>
> Thank you very much.
>
> aj
>
> --
> Sent via pgsql-general mailing list (pgsql-general(at)postgresql(dot)org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-general
>

--
Family management on rails: http://www.famundo.com
My development related blog: http://devblog.famundo.com

--
Family management on rails: http://www.famundo.com
My development related blog: http://devblog.famundo.com

On Mon, Jun 15, 2009 at 12:11:54PM -0700, Just Someone wrote:
- Hi,
-
- I have more than a few Postgres instances on EC2. For reliability I
- use EBS, and take regular snapshots while also streaming the WAL files
- to S3. So far, the few times that my machine died, I had no issue with
- getting it back from EBS or the EBS volume. I also take tar backups
- every day, and I keep a few days back of tar, snapshots and WAL log
- files.
Your machine died? Was it the cloud's fault or something else?

Dave

Hi,

I've seen both - some unknown reason for it to die (mostly related to
the underlying hardware having issues). We also see instance failure
from time to time with advanced notice. Just like a regular machine
dies from time to time, so do cloud instances. I'd say it's bit more
common on the cloud, but not by a big margin. I might see it more
because I have hundreds of instances running.

Bye,

Guy.

On Mon, Jun 15, 2009 at 3:46 PM, David Kerr<dmk(at)mr-paradox(dot)net> wrote:
> On Mon, Jun 15, 2009 at 12:11:54PM -0700, Just Someone wrote:
> - Hi,
> -
> - I have more than a few Postgres instances on EC2. For reliability I
> - use EBS, and take regular snapshots while also streaming the WAL files
> - to S3. So far, the few times that my machine died, I had no issue with
> - getting it back from EBS or the EBS volume. I also take tar backups
> - every day, and I keep a few days back of tar, snapshots and WAL log
> - files.
> Your machine died? Was it the cloud's fault or something else?
>
> Dave
>

--
Family management on rails: http://www.famundo.com
My development related blog: http://devblog.famundo.com

On Mon, Jun 15, 2009 at 06:53:00PM -0700, Just Someone wrote:
- Hi,
-
- I've seen both - some unknown reason for it to die (mostly related to
- the underlying hardware having issues). We also see instance failure
- from time to time with advanced notice. Just like a regular machine
- dies from time to time, so do cloud instances. I'd say it's bit more
- common on the cloud, but not by a big margin. I might see it more
- because I have hundreds of instances running.

Hi Guy,

Thanks!

So, when a cloud machine fails does it get de-allocated/wiped out? or
does it is it still out there in a bad state? how do you recover your
data?

For example, in a physical system if the RAM goes bad the machine
crashes but you replace the chip, boot, perform instance recovery and
continue on with your life...

Thanks

Dave

Hi,

> So, when a cloud machine fails does it get de-allocated/wiped out? or
> does it is it still out there in a bad state? how do you recover your
> data?

It depends. Sometimes it dies and you can't do anything with it. In
others you can restart it. As we store the data on EBS (which is a
network storage in AMazon's cloud), the data is not related to the
instance directly. And that's the beauty of it. If a machine dies I
can launch a replacement machine (it takes about 3-5 minutes for it to
boot and be ready), or in some cases I just have a replacement
instance waiting. I then mount the EBS volume holding the Postgres
data, let the server do the recovery if needed and I'm back online. No
need to replace a chip, go to the DC, etc...

In case the volume got corrupted (a very rare situation, as the EBS
volumes are very durable), there are snapshots I can recover from and
the WAL files I stream to another storage system (Amazon's S3). I have
some systems where I take daily tar backups of the PG directory and
ship those to a separate storage.

There is no doubt you can get a much stronger machine by building your
own hardware, but the ability to easily recover, and easily launch and
cluster are a huge advantage for the cloud. It does takes getting used
to, though. You need to think of machines as expendable, and plan for
easy failure preparation and replacement. It does make you really
prepare and test your recovery strategies. I know of too many
companies that just trust the DB to be ok. And it is most of the time,
but when a catastrophe happens, recovery is a long and risky process.

Bye,

Guy.

--
Family management on rails: http://www.famundo.com
My development related blog: http://devblog.famundo.com

On Tue, 16 Jun 2009, Just Someone wrote:

> In case the volume got corrupted (a very rare situation, as the EBS
> volumes are very durable), there are snapshots I can recover from and
> the WAL files I stream to another storage system (Amazon's S3).

I wouldn't go so far as to say "very durable", because the failure rate
they aim for isn't really very high relative to what people expect when
you use that term in a database context. The most definitive commentary
I've found on this is at
http://solutions.amazonwebservices.com/connect/thread.jspa;jsessionid=96A862FA1DC393FCDD94DAF0B43CF4E7?messageID=111953
where they say "we aim to provide an annual failure rate (AFR) of 0.1% -
0.5% for volumes"; frankly, that's garbage to most database people. But,
as you say, when combined with an alternative backup strategy when that
happens, the easy provisioning and such can give a reasonable system
design for some goals. You just have to recognize that the volumes are
statistically pretty fragile compared to a traditional RAID configuration
on dedicated hardware and plan accordingly.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Hi,

On Tue, Jun 16, 2009 at 11:05 AM, Greg Smith<gsmith(at)gregsmith(dot)com> wrote:

> You just have to recognize that the volumes are
> statistically pretty fragile compared to a traditional RAID configuration on
> dedicated hardware and plan accordingly.

I agree completely. I think the advantage is that it FORCES you to
plan for failure. Now, I know we all SHOULD plan for failure, but I
also know how many do not...

Bye,

Guy.

Family management on rails: http://www.famundo.com
My development related blog: http://devblog.famundo.com