Re: Recommended/Not Recommended Hosts?

All,

What hosts, both virtual hosts and colos, do you recommend for
PostgreSQL-based applications, and which ones have you had bad
experiences with? If we get a list together, we'll put it up somewhere
community.

For my part, I've had reasonably good experiences with:

-- Rackspace rental servers (provided you're OK with Dell hardware)
-- Layer42 for low-end colo, but watch your bandwith, they won't.
-- Joyent for online data warehousing (if you can deal with Solaris)

I've had bad experiences with:

-- GoGrid Cloud Servers ... I/O is very very bad, so your database
better fit in RAM, and they're limited to 8GB max. On the other hand,
no downtime whatsoever. (in GG's defense, they recommend colo for databases)

-- Amazon EC2: uptime and availability are great, but the servers are
sloooooooow and fulfillment of new instances is unreliable. Also,
CPU-stealing.

--Josh Berkus

On Thu, Dec 10, 2009 at 10:43 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> What hosts, both virtual hosts and colos, do you recommend for
> PostgreSQL-based applications, and which ones have you had bad
> experiences with?   If we get a list together, we'll put it up somewhere
> community.
>
> For my part, I've had reasonably good experiences with:
>
> -- Layer42 for low-end colo, but watch your bandwith, they won't.

At the risk of giving away one of my best kept secrets, Layer42 is
simply the most outstanding and professional colocation provider I
have ever had the pleasure to work with. Their network is FAST and
unbelievably reliable. Steve and his team are of the highest caliber.
After going through half a dozen different providers of virtual hosts
and colocations, I finally found a provider that I can trust with my
mission critical needs.

They are located in Santa Clara which is a bit of a drive, but
considering I only need to ever make the trip to add more capacity, I
don't mind it one bit. I highly recommend them.

>
>
> -- Amazon EC2: uptime and availability are great, but the servers are
> sloooooooow and fulfillment of new instances is unreliable. Also,
> CPU-stealing.
>
>
>
Pure curiosity on my part here ... I use EC2 a bit, tho not as much as the
serious users. A few large and small instances on all the time, and I boot
up new ones for shorter periods all the time. First - I've never had any
issue getting my instances fulfilled right away (I always use EAST-C, but
perhaps other datacenters are generally more full, or you are trying to boot
up many tens of servers at once?).

Also, when you say they are slow, do you mean in terms of $/cycle, or you
wish you had burst access to other users unused cycles like on some other
vps offerings? something else?

I like the bundle of offerings that AWS provides (EBS, especially), and I've
personally had great experience w/them (fwiw I've also had great experience
w/slicehost) -- but if I am getting missing out on how they're screwing me,
for example by stealing my CPU, I'd definitely love to learn more.

Josh,

> Pure curiosity on my part here ... I use EC2 a bit, tho not as much as
> the serious users. A few large and small instances on all the time, and
> I boot up new ones for shorter periods all the time. First - I've
> never had any issue getting my instances fulfilled right away (I always
> use EAST-C, but perhaps other datacenters are generally more full, or
> you are trying to boot up many tens of servers at once?).

Yeah, the two issues I've had are (a) requisitioning high-end instances
(like 32G/16core instances) and (b) allocating a lot at once. Sometimes
instances just "aren't available" and there's no way to find out when
they will be available.

> Also, when you say they are slow, do you mean in terms of $/cycle,
> or you wish you had burst access to other users unused cycles like on
> some other vps offerings? something else?

I mean that if you have an 8core/16GB instance, the actual processing
throughput you get is about 1/6 to 1/4 that of a new HP DL380 machine
with 8cores and 16GB. So you really need 4x as many EC2 instances to
match bare metal. Partly this is due to CPU-stealing, and partly to
erratic and lag-prone I/O, and partly to the fact that a lot of machines
in the EC2 pool are 4 years old.

> I like the bundle of offerings that AWS provides (EBS, especially), and
> I've personally had great experience w/them (fwiw I've also had great
> experience w/slicehost) -- but if I am getting missing out on how
> they're screwing me, for example by stealing my CPU, I'd definitely love
> to learn more.

On EC2, other VMs on the same hardware are permitted to "steal" portions
of the CPU which are allocated to you. So at any given time, you may
have as little as 50% of the CPUs you're being billed for. And, when
CPU availability is fluctuating up and down (as it does on EC2), real
throughput tends to be based on the slowest second rather than peak
availablity. Most Linux apps, especially databases, do quite poorly
with erratic resource availability.

--Josh

Cool, Thanks for the detailed response. I've certainly not done a ton of
research comparing different VPS solutions myself, but I was under the
(quite possible mis-impression) that at EC2 the underlying hardware was not
1:1 related to your performance (eg you get a set amount of cpu throughput,
and if they had older hardware underneath, then you'd just get more of it).
I also did not realize other users could steal cycles from you like happens
on most other VPS offerings. I haven't seen that much documentation to base
any of these assumptions on, of course, so it's good to hear your
perspective.

Some posts, such as http://journal.uggedal.com/vps-performance-comparison,
seem to imply different conclusions (not that he's suggesting EC2 is a good
deal, but for different reasons than cpu stealing), and I'd love to see
similar posts on the topic: I'd be happy to switch to something else if I
felt I was going to getting a much better deal (I really do like the
integrated EBS/S3/Cloudfront options tho for the type of projects I
generally work on).

Cheers,

-Josh

On Thu, Dec 10, 2009 at 11:20 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:

> Josh,
>
> > Pure curiosity on my part here ... I use EC2 a bit, tho not as much as
> > the serious users. A few large and small instances on all the time, and
> > I boot up new ones for shorter periods all the time. First - I've
> > never had any issue getting my instances fulfilled right away (I always
> > use EAST-C, but perhaps other datacenters are generally more full, or
> > you are trying to boot up many tens of servers at once?).
>
> Yeah, the two issues I've had are (a) requisitioning high-end instances
> (like 32G/16core instances) and (b) allocating a lot at once. Sometimes
> instances just "aren't available" and there's no way to find out when
> they will be available.
>
> > Also, when you say they are slow, do you mean in terms of $/cycle,
> > or you wish you had burst access to other users unused cycles like on
> > some other vps offerings? something else?
>
> I mean that if you have an 8core/16GB instance, the actual processing
> throughput you get is about 1/6 to 1/4 that of a new HP DL380 machine
> with 8cores and 16GB. So you really need 4x as many EC2 instances to
> match bare metal. Partly this is due to CPU-stealing, and partly to
> erratic and lag-prone I/O, and partly to the fact that a lot of machines
> in the EC2 pool are 4 years old.
>
> > I like the bundle of offerings that AWS provides (EBS, especially), and
> > I've personally had great experience w/them (fwiw I've also had great
> > experience w/slicehost) -- but if I am getting missing out on how
> > they're screwing me, for example by stealing my CPU, I'd definitely love
> > to learn more.
>
> On EC2, other VMs on the same hardware are permitted to "steal" portions
> of the CPU which are allocated to you. So at any given time, you may
> have as little as 50% of the CPUs you're being billed for. And, when
> CPU availability is fluctuating up and down (as it does on EC2), real
> throughput tends to be based on the slowest second rather than peak
> availablity. Most Linux apps, especially databases, do quite poorly
> with erratic resource availability.
>
> --Josh
>

Cool, Thanks for the detailed response. I've certainly not done a ton of
research comparing different VPS solutions myself, but I was under the
(quite possible mis-impression) that at EC2 the underlying hardware was not
1:1 related to your performance (eg you get a set amount of cpu throughput,
and if they had older hardware underneath, then you'd just get more of it).
I also did not realize other users could steal cycles from you like happens
on most other VPS offerings. I haven't seen that much documentation to base
any of these assumptions on, of course, so it's good to hear your
perspective.

Some posts, such as http://journal.uggedal.com/vps-performance-comparison,
seem to imply different conclusions (not that he's suggesting EC2 is a good
deal, but for different reasons than cpu stealing), and I'd love to see
similar posts on the topic. I'd be happy to switch to something else if I
felt I was going to getting a much better deal (I really do like the
integrated EBS/S3/Cloudfront options tho for the type of projects I
generally work on).

Interesting and useful
Cheers,

-Josh

On Thu, Dec 10, 2009 at 11:20 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:

> Josh,
>
> > Pure curiosity on my part here ... I use EC2 a bit, tho not as much as
> > the serious users. A few large and small instances on all the time, and
> > I boot up new ones for shorter periods all the time. First - I've
> > never had any issue getting my instances fulfilled right away (I always
> > use EAST-C, but perhaps other datacenters are generally more full, or
> > you are trying to boot up many tens of servers at once?).
>
> Yeah, the two issues I've had are (a) requisitioning high-end instances
> (like 32G/16core instances) and (b) allocating a lot at once. Sometimes
> instances just "aren't available" and there's no way to find out when
> they will be available.
>
> > Also, when you say they are slow, do you mean in terms of $/cycle,
> > or you wish you had burst access to other users unused cycles like on
> > some other vps offerings? something else?
>
> I mean that if you have an 8core/16GB instance, the actual processing
> throughput you get is about 1/6 to 1/4 that of a new HP DL380 machine
> with 8cores and 16GB. So you really need 4x as many EC2 instances to
> match bare metal. Partly this is due to CPU-stealing, and partly to
> erratic and lag-prone I/O, and partly to the fact that a lot of machines
> in the EC2 pool are 4 years old.
>
> > I like the bundle of offerings that AWS provides (EBS, especially), and
> > I've personally had great experience w/them (fwiw I've also had great
> > experience w/slicehost) -- but if I am getting missing out on how
> > they're screwing me, for example by stealing my CPU, I'd definitely love
> > to learn more.
>
> On EC2, other VMs on the same hardware are permitted to "steal" portions
> of the CPU which are allocated to you. So at any given time, you may
> have as little as 50% of the CPUs you're being billed for. And, when
> CPU availability is fluctuating up and down (as it does on EC2), real
> throughput tends to be based on the slowest second rather than peak
> availablity. Most Linux apps, especially databases, do quite poorly
> with erratic resource availability.
>
> --Josh
>

<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#ffffff" text="#003300">
<br>
I signed up for a year of hosting with host1plus listed as a vendor here:&nbsp; <a class="moz-txt-link-freetext" href="http://www.postgresql.org/support/professional_hosting_europe">
http://www.postgresql.org/support/professional_hosting_europe</a><br>
<br>
Based out of the UK, but they have servers in the US.&nbsp; The deal looked pretty good. But the hosting went down for days at a time with with no response and site off line.&nbsp;&nbsp; I just took my sight and moved it to another vendor.<br>
<br>
I was not using postgreSQL in my hosting solution, but I had chosen them so I could at some point in the future.&nbsp; I can give them a very strong not recommended.<br>
<br>
-Jeff<br>
<br>
Josh Berkus wrote:
<blockquote cite="mid:4B214163(dot)4000900(at)agliodbs(dot)com" type="cite">
<pre wrap="">All,

What hosts, both virtual hosts and colos, do you recommend for
PostgreSQL-based applications, and which ones have you had bad
experiences with? If we get a list together, we'll put it up somewhere
community.

For my part, I've had reasonably good experiences with:

-- Rackspace rental servers (provided you're OK with Dell hardware)
-- Layer42 for low-end colo, but watch your bandwith, they won't.
-- Joyent for online data warehousing (if you can deal with Solaris)

I've had bad experiences with:

-- GoGrid Cloud Servers ... I/O is very very bad, so your database
better fit in RAM, and they're limited to 8GB max. On the other hand,
no downtime whatsoever. (in GG's defense, they recommend colo for databases)

-- Amazon EC2: uptime and availability are great, but the servers are
sloooooooow and fulfillment of new instances is unreliable. Also,
CPU-stealing.

--Josh Berkus
</pre>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<title></title>
<meta name="GENERATOR" content="OpenOffice.org 2.4 (Linux)">
<meta name="CREATED" content="0;0">
<meta name="CHANGEDBY" content="Jeff Rule">
<meta name="CHANGED" content="20091202;20505400">
<style type="text/css">
<!--
@page { size: 8.5in 11in; margin-left: 1.25in; margin-right: 1.25in; margin-top: 1in; margin-bottom: 1in }
P { margin-bottom: 0.08in }
-->
</style>
<p style="margin-bottom: 0in;"><font color="#980000"><font face="Times New Roman"><font size="3"><i><b>Please note new address &amp; phone number below to be in effect as of Nov 25th.
</b></i></font></font></font></p>
<p style="margin-bottom: 0in;"><font color="#980000"><font face="Times New Roman"><font size="3"><i><b>Please update your records.</b></i></font></font></font></p>
<p style="margin-bottom: 0in;"><br>
</p>
<p style="margin-bottom: 0in;"><font color="#808080">
<meta name="CHANGEDBY" content="Jeff Rule">
<font face="Verdana, sans-serif"><font size="2"><b>Jeff Rule |</b><font color="#000080">
</font>Distinguished Engineer<br>
<font color="#800000"><b>DemandTec, Inc.</b></font> | <font face="Arial, sans-serif">
1 Franklin Parkway, Building 910, Suite 200 | San Mateo, CA 94403<br>
</font><font face="Arial, sans-serif"><b>t</b></font> <font face="Arial, sans-serif">
| 650.645.7110 </font><font face="Arial, sans-serif"><b>f</b></font> <font face="Arial, sans-serif">
| 650.645.7377 </font><font face="Arial, sans-serif"><b>m</b></font> <font face="Arial, sans-serif">
| 415.706.4928 </font></font></font></font></p>
<p style="margin-top: 0.07in; margin-bottom: 0.07in;"><a href="http://www.demandtec.com/"><font color="#000080"><font face="Verdana, sans-serif"><font size="2"><u>www.demandtec.com</u></font></font></font></a></p>
</div>
<br>
<hr>
<font face="Arial" color="Black" size="3">******************************************************************<br>
DemandTec Email Notice<br>
<br>
This email and any attachments may contain confidential and/or proprietary<br>
information and is intended solely for the use of the addressee. If you are not<br>
the intended recipient we request that you notify us via email or telephone and<br>
delete all copies of the message from your systems. Additionally, although<br>
DemandTec has taken reasonable precautions to ensure the security of this<br>
email and any attachments, we encourage you to take similar precautions and<br>
accept no liability for any loss or damage resulting from its use.<br>
<br>
DemandTec, 1 Franklin Parkway, Building 910, San Mateo, CA 94403-1906, 650-650-7100<br>
</font>
</body>
</html>

On Thu, Dec 10, 2009 at 10:43, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> All,
>
> What hosts, both virtual hosts and colos, do you recommend for
> PostgreSQL-based applications, and which ones have you had bad
> experiences with?   If we get a list together, we'll put it up somewhere
> community.
>

From the colo perspective:

If your app is bandwidth-hungry, I recommend Equinix or some other
carrier neutral facility where you can get good bandwidth prices.
Space will be more expensive, but if you're using a lot of bandwidth,
it should be cheaper and more reliable overall.

Now for carriers in such a facility (I generally run BGP with multiple
carriers):

*) Cogent - I've had good luck with them overall, and they're cheap.
*) Level(3) - No issues with them either. They also offer IPv6 now I
believe. NOC is kind of a pain though to get through to though.
*) Internap - They're fine, but I don't think they're worth their
price premium. This is all anecdotal, but I've had more reliability
issues with them than I've had with Cogent. So YMMV. If you're going
to only have *one* carrier though, this might be a good choice as they
have transit (not peering) agreements with everyone that should make
them immune to the various political peering disputes that have
happened over the years.

There are a ton of other carriers that I don't have recent experience
with, so ask around.

Usually, I'll setup 2-3 carriers, then give Cogent highest preference
for all traffic that doesn't terminate on one of the other carriers.
Otherwise the other carrier wins. I find that it's a good way to go
for bandwidth cost savings, but now I'm getting into more of a network
engineering discussion. Dedicated servers/cloud stuff can help you
avoid this infrastructure stuff, but at the cost of much higher MRCs
when you get above very small deployment sizes. But if you're at the
stage where you're wondering which host to use, that's probably the
size you're at.

>
> > Pure curiosity on my part here ... I use EC2 a bit, tho not as much as
> > the serious users. A few large and small instances on all the time, and
> > I boot up new ones for shorter periods all the time. First - I've
> > never had any issue getting my instances fulfilled right away (I always
> > use EAST-C, but perhaps other datacenters are generally more full, or
> > you are trying to boot up many tens of servers at once?).
>
> Yeah, the two issues I've had are (a) requisitioning high-end instances
> (like 32G/16core instances) and (b) allocating a lot at once. Sometimes
> instances just "aren't available" and there's no way to find out when
> they will be available.
>
> > Also, when you say they are slow, do you mean in terms of $/cycle,
> > or you wish you had burst access to other users unused cycles like on
> > some other vps offerings? something else?
>
> I mean that if you have an 8core/16GB instance, the actual processing
> throughput you get is about 1/6 to 1/4 that of a new HP DL380 machine
> with 8cores and 16GB. So you really need 4x as many EC2 instances to
> match bare metal. Partly this is due to CPU-stealing, and partly to
> erratic and lag-prone I/O, and partly to the fact that a lot of machines
> in the EC2 pool are 4 years old.
>
> > I like the bundle of offerings that AWS provides (EBS, especially), and
> > I've personally had great experience w/them (fwiw I've also had great
> > experience w/slicehost) -- but if I am getting missing out on how
> > they're screwing me, for example by stealing my CPU, I'd definitely love
> > to learn more.
>
> On EC2, other VMs on the same hardware are permitted to "steal" portions
> of the CPU which are allocated to you. So at any given time, you may
> have as little as 50% of the CPUs you're being billed for. And, when
> CPU availability is fluctuating up and down (as it does on EC2), real
> throughput tends to be based on the slowest second rather than peak
> availablity. Most Linux apps, especially databases, do quite poorly
> with erratic resource availability.
>
> --Josh
>

For a dev environment running a small database, we've had a great experience
with Rackspace cloud servers. The price point for server instances is
around $10 per 256 MB which is the low end for cloud services.

On Thu, Dec 10, 2009 at 11:20, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
> On EC2, other VMs on the same hardware are permitted to "steal" portions
> of the CPU which are allocated to you.  So at any given time, you may
> have as little as 50% of the CPUs you're being billed for.  And, when
> CPU availability is fluctuating up and down (as it does on EC2), real
> throughput tends to be based on the slowest second rather than peak
> availablity.  Most Linux apps, especially databases, do quite poorly
> with erratic resource availability.

Just to clarify, Joyent does this too, no? Or is it just that the
Solaris scheduler handles the prioritization of tasks/resources
better? I haven't yet had the chance to use Joyent, but I thought
this was one of their selling points.

Josh Berkus wrote:
> What hosts, both virtual hosts and colos, do you recommend for
> PostgreSQL-based applications...
> For my part, I've had reasonably good experiences with:
> -- Rackspace rental servers (provided you're OK with Dell hardware)
> -- Layer42 for low-end colo, but watch your bandwith, they won't.
> -- Joyent for online data warehousing (if you can deal with Solaris)

For the extremely low end of dedicated servers, I'm very happy with
Server4You ( http://www.server4you.com ) and have been running
postgres-backed hobby sites on a few of their machines for
years. $29/month for a tiny cheap dedicated box is great for
small cheap clients starting out, and it's easy to upgrade their
plans to their slightly larger systems. They have nothing high-end,
though.

For the high-end systems we run our own servers at 365 Main in
SF (where we have mixed experiences - including a painful unplanned
downtime but it was rather affordable) and a Sungard facility in
Texas (which was much better, but not at all cheap).

> I've had bad experiences with:
>
> -- Amazon EC2: uptime and availability are great, but the servers are
> sloooooooow and fulfillment of new instances is unreliable. Also,
> CPU-stealing.

Curious which instance type you had those experiences with.

Amazon provides (expensive) solutions to help address each of
those complaints, at least to some degree.

* Their "High CPU" Instances (and I think even moreso their poorly
named "High Memory" instances) have vastly improved performance
over their sloooooow normal instances. At least an order of
magnitude when I tried, and 26 times as fast if you believe
their docs[1]. Those things go for $2.40 instead of $0.085
per hour, though [2]. The I/O performance of their small
instances are horribly too; and again much improved in their
expensive ones. You can also make RAID 0 arrays of their EBS
volumes that have higher performance than one.

I haven't noticed CPU stealing on the high-CPU instance I ran.

* Fulfillment of new instances is supposed to be handled by
their "Reserved Instances" feature. You make a one-time
payment of between 350 and 2800 dollars they're supposed
to keep it available for you for 1-3 years.

I like Amazon for servers and clusters don't need to run 24x7.
For example ones that only need to be up during business hours
or peek traffic periods. Also for short-lived servers - for
example I'll be setting up an experimental hadoopdb cluster
there. For servers running 24x7 for 30 days or more, it
seems to me other solutions are more cost effective.

[1] http://aws.amazon.com/ec2/instance-types/
[2] http://aws.amazon.com/ec2/#pricing

> Just to clarify, Joyent does this too, no? Or is it just that the
> Solaris scheduler handles the prioritization of tasks/resources
> better? I haven't yet had the chance to use Joyent, but I thought
> this was one of their selling points.

It depends on the size of your instance. If you're using a whole CPU
(not core, CPU) Joyent can dedicate that CPU to you and nobody else can
touch it. One of the advantages of Solaris over Xen.

--Josh

Fred Moyer wrote:
> On Thu, Dec 10, 2009 at 10:43 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>> For my part, I've had reasonably good experiences with:
>>
>> -- Layer42 for low-end colo, but watch your bandwith, they won't.
>
> At the risk of giving away one of my best kept secrets, Layer42 is
> simply the most outstanding and professional colocation provider I
> have ever had the pleasure to work with. Their network is FAST and
> unbelievably reliable. Steve and his team are of the highest caliber.

I've been at Layer 42 for five years so I do believe they provide a good
service for the price. It's worth noting however they have suffered at
least two full power outages over the past couple of years. While they
have been brief (~seconds), they cause hardware to power cycle and we've
had hardware failures each time. The most recent outage was in September.

Brian

On Dec 10, 2009, at 10:43 AM, Josh Berkus wrote:

> All,
>
> What hosts, both virtual hosts and colos, do you recommend for
> PostgreSQL-based applications, and which ones have you had bad
> experiences with? If we get a list together, we'll put it up somewhere
> community.
>
> For my part, I've had reasonably good experiences with:
>
> -- Rackspace rental servers (provided you're OK with Dell hardware)
> -- Layer42 for low-end colo, but watch your bandwith, they won't.
> -- Joyent for online data warehousing (if you can deal with Solaris)

I've been with Integra Telecom (nee ELI) for quite a few years. If you're
big enough that you need somewhere between a full cabinet and
a small cage, but are happy with a default route then they're well worth
a look. Solid provider, and very helpful NOC / support (even when
dealing with someone dumb enough to have let his switch
autonegotiate...). They're in Santa Clara.

I've also had a couple of U of space at Hurricane Electric in Fremont
forever, mostly as a backup location, and it's been pretty solid. It's been
years since I've actually been there, so I don't even remember what
the physical facility is like. (World+Dog resells space there, including
http://www.asaservers.com/ who'll sell you a box and host it at HE
for you).

Cheers,
Steve

All,

I just found www.zill.net, which appears to be dedicated PostgreSQL
hosting. Anyone have experience with them?

--Josh Berkus

On Dec 11, 2009, at 2:49 PM, Josh Berkus wrote:

> All,
>
> I just found www.zill.net, which appears to be dedicated PostgreSQL
> hosting. Anyone have experience with them?

Not I. They look a bit zombie, though - their support page provides docs for Postgresql 7.2.1 and some OS9 scp client.

Cheers,
Steve

Has anyone used EBS on EC2 ?

I/O is supposed to be much better on it

Sailesh

On 12/10/09, Josh Livni <josh(at)umbrellaconsulting(dot)com> wrote:
> Cool, Thanks for the detailed response. I've certainly not done a ton of
> research comparing different VPS solutions myself, but I was under the
> (quite possible mis-impression) that at EC2 the underlying hardware was not
> 1:1 related to your performance (eg you get a set amount of cpu throughput,
> and if they had older hardware underneath, then you'd just get more of it).
> I also did not realize other users could steal cycles from you like happens
> on most other VPS offerings. I haven't seen that much documentation to base
> any of these assumptions on, of course, so it's good to hear your
> perspective.
>
> Some posts, such as http://journal.uggedal.com/vps-performance-comparison,
> seem to imply different conclusions (not that he's suggesting EC2 is a good
> deal, but for different reasons than cpu stealing), and I'd love to see
> similar posts on the topic: I'd be happy to switch to something else if I
> felt I was going to getting a much better deal (I really do like the
> integrated EBS/S3/Cloudfront options tho for the type of projects I
> generally work on).
>
> Cheers,
>
> -Josh
>
>
> On Thu, Dec 10, 2009 at 11:20 AM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>
>> Josh,
>>
>> > Pure curiosity on my part here ... I use EC2 a bit, tho not as much as
>> > the serious users. A few large and small instances on all the time, and
>> > I boot up new ones for shorter periods all the time. First - I've
>> > never had any issue getting my instances fulfilled right away (I always
>> > use EAST-C, but perhaps other datacenters are generally more full, or
>> > you are trying to boot up many tens of servers at once?).
>>
>> Yeah, the two issues I've had are (a) requisitioning high-end instances
>> (like 32G/16core instances) and (b) allocating a lot at once. Sometimes
>> instances just "aren't available" and there's no way to find out when
>> they will be available.
>>
>> > Also, when you say they are slow, do you mean in terms of $/cycle,
>> > or you wish you had burst access to other users unused cycles like on
>> > some other vps offerings? something else?
>>
>> I mean that if you have an 8core/16GB instance, the actual processing
>> throughput you get is about 1/6 to 1/4 that of a new HP DL380 machine
>> with 8cores and 16GB. So you really need 4x as many EC2 instances to
>> match bare metal. Partly this is due to CPU-stealing, and partly to
>> erratic and lag-prone I/O, and partly to the fact that a lot of machines
>> in the EC2 pool are 4 years old.
>>
>> > I like the bundle of offerings that AWS provides (EBS, especially), and
>> > I've personally had great experience w/them (fwiw I've also had great
>> > experience w/slicehost) -- but if I am getting missing out on how
>> > they're screwing me, for example by stealing my CPU, I'd definitely love
>> > to learn more.
>>
>> On EC2, other VMs on the same hardware are permitted to "steal" portions
>> of the CPU which are allocated to you. So at any given time, you may
>> have as little as 50% of the CPUs you're being billed for. And, when
>> CPU availability is fluctuating up and down (as it does on EC2), real
>> throughput tends to be based on the slowest second rather than peak
>> availablity. Most Linux apps, especially databases, do quite poorly
>> with erratic resource availability.
>>
>> --Josh
>>
>

--
Sent from my mobile device

Cheers
Sailesh

On Fri, Dec 11, 2009 at 15:48, Sailesh Krishnamurthy <sailesh(at)gmail(dot)com> wrote:
> Has anyone used EBS on EC2 ?
>
> I/O is supposed to be much better on it
>
> Sailesh
>

My testing has shown it to still be pretty awful. Although Amazon
does suggest doing software RAID on multiple EBS volumes. I haven't
tried that yet.

Jason DiCioccio wrote:
> On Fri, Dec 11, 2009 at 15:48, Sailesh Krishnamurthy <sailesh(at)gmail(dot)com> wrote:
>> Has anyone used EBS on EC2 ?
>> I/O is supposed to be much better on it
>
> My testing has shown it to still be pretty awful. Although Amazon
> does suggest doing software RAID on multiple EBS volumes. I haven't
> tried that yet.

The one thing I love about EC2 is that it's incredibly easy and
cheap to test such things. I just tried it for the cost of about $0.20.

(the script I used, and output from bonnie++ below)

Seek performance on the non-EBS default drive is poor -- but if I'm
reading the bonnie++ output right, the EBS volume otherwise wasn't
really faster - and perhaps even slower than the default drive for block
reads and writes. Also if I read the bonnie++ output right, strangely,
block reads seem much slower with a 7-EBS-volume RAID array than with a
single EBS block. RAID helped the writes significantly, though.

Does that make sense? Can anyone help me interpret that output?

All this was on a small instance (which end up with low I/O and CPU priority),
so I guess it's quite possible this test was dominated with people running more
expensive instances stealing the CPU and I/O resources.

Here's the script I just used after attaching 7 EBS volumes to one of my
instances:
================================================================================
sudo mdadm --create /dev/md0 --level=0 --raid-devices=7 /dev/sdb1 /dev/sdb2 /dev/sdb3 /dev/sdb4 /dev/sdb5 /dev/sdb6 /dev/sdb7
sudo mkfs.ext3 /dev/md0
sudo mkdir /mnt/md0
sudo tune2fs -o journal_data_writeback /dev/md0
sudo mount /dev/md0 /mnt/md0
sudo chmod a+w /mnt/md0
cd /mnt/md0
bonnie++ -d /mnt/md0
================================================================================

And here's the output:

================================================================================
= 1 local disk on EC2 small instance
===============================================================================
Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ars.forensicl 3408M 21139 35 65132 11 31224 2 24915 19 70138 1 189.1 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ars.forensiclogic.com,3408M,21139,35,65132,11,31224,2,24915,19,70138,1,189.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

================================================================================
= 1 EBS disk on EC2 small instance
===============================================================================

Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ars.forensicl 3408M 22239 34 40111 5 17178 0 23539 17 51379 0 570.1 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ars.forensiclogic.com,3408M,22239,34,40111,5,17178,0,23539,17,51379,0,570.1,0,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

================================================================================
= RAID0 of 7 EBS disks on EC2 small instance
===============================================================================

Version 1.03c ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
ars.forensicl 3408M 20884 34 115135 24 17586 1 15472 20 23198 0 5659 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++ +++++ +++
ars.forensiclogic.com,3408M,20884,34,115135,24,17586,1,15472,20,23198,0,5659.1,1,16,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

> Seek performance on the non-EBS default drive is poor -- but if I'm
> reading the bonnie++ output right, the EBS volume otherwise wasn't
> really faster - and perhaps even slower than the default drive for block
> reads and writes. Also if I read the bonnie++ output right, strangely,
> block reads seem much slower with a 7-EBS-volume RAID array than with a
> single EBS block. RAID helped the writes significantly, though.

Yeah, that's fairly odd. Is it possible that the CPU overhead of the SW
RAID is eating a lot of your available CPU?

It would also be great to run Bonnie 1.95 and look at lag times.

--Josh