Re: [HACKERS] [9.4 CF] Free VMs for Reviewers & Testers

Reviewer, Testers:

As part of an ongoing effort to encourage patch review for the
PostgreSQL project, we will be funding cloud servers for patch reviewers
and testers who need them for CommitFests. That is, if you want to help
with reviewing or testing a patch for a CommitFest, and don't have your
own test server to use, the PostgreSQL project (via our funds at
Software In the Public Interest) will pay for you to use a cloud server
for the duration of your review/testing.

So, if "I don't have anywhere to test it" was your excuse for not
reviewing in the past, time to stop making excuses and start reviewing!

Since these are cloud servers, they won't work well for performance
testing. However, they will be excellent for testing replication.

If you need one of these, please contact me to allocate a VM for you.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 07/09/2013 08:35 AM, Josh Berkus wrote:
> Since these are cloud servers, they won't work well for performance
> testing.

I did some work on that a while ago, and found that I was able to get
_astonishingly_ stable performance results out of EC2 EBS instances
using provisioned IOPS volumes. Running them as "EBS Optimized" didn't
seem to be required for the workloads I was testing on.

What isn't made very clear in the EBS provisioned I/O docs is that for
PIOPS volumes the provisioned I/O count is both a minimum and a limit;
it's a *target*, not just a guaranteed minimum. This is really handy for
testing.

For a 500 iops volume I consistently got 500 +- 5 in pg_test_fsync.

Real world performance was a bit more variable - my pgbench runs were
stable within 5-10% over 5 minute runs with a forced CHECKPOINT first.
They settled considerably over longer runs and bigger
checkpoint_segments, though, and I see the same kind of short-term
jitter in pgbench on real hardware.

So - don't write cloud hosts off for benchmarking. With proper QoS
they're actually really very good. While they're somewhat costly, being
able to spawn them for a day's runs and destroy them at the end is quite
handy, as is being able to easily automate their setup.

For testing of logical changeset streaming replication / BDR performance
I was using scripted sets of EC2 c1.medium instances with 500 piops
'io1' EBS volumes and found it to be an exceedingly useful way of
measuring relative performance.

Results were also pretty consistent across multiple launches of the same
VM, so multiple people should be able to compare results obtained with
different launches of the same VM type and configuration.

These VMs aren't well suited to vertical scaling performance tests and
pushing extremes, but they're really, really good for "what impact does
this patch have on regular real-world workloads".

Just remember to run pgbench from a different vm within the same
availability zone (with appropriate security group setup) if you try to
use EC2 for this sort of thing.

Use of instance store or regular EBS volumes will get you performance
that's absolutely all over the place, of course; only the provisioned
I/O feature seems to be any good for perf testing.

--
Craig Ringer http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Mon, Jul 8, 2013 at 7:25 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
> On 07/09/2013 08:35 AM, Josh Berkus wrote:
>> Since these are cloud servers, they won't work well for performance
>> testing.
>
> I did some work on that a while ago, and found that I was able to get
> _astonishingly_ stable performance results out of EC2 EBS instances
> using provisioned IOPS volumes. Running them as "EBS Optimized" didn't
> seem to be required for the workloads I was testing on.

My colleague, Greg Burek, has done similar measurements and has
assessed an overall similar conclusion: the EBS PIOPS product delivers
exactly what it says on the tin...even under random access. They can
be striped with software-RAID.

> These VMs aren't well suited to vertical scaling performance tests and
> pushing extremes, but they're really, really good for "what impact does
> this patch have on regular real-world workloads".

The really, really big ones are useful even for pushing limits, such
as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance
price (the heavily discounted "can die at any time" one) is $0.343/hr.
Otherwise, it's 3.500/hr.

Another instance offering that -- unlike the former -- I have yet to
experience in any way at all is the high-storage ones, also the
hs1.8xlarge, with 16CPU, 117GB RAM, and 24 instance-local rotational
media of 2TB apiece. This is enough to deliver sequential reads
measured in a couple of GB/s. I think this is a workhorse behind the
AWS Redshift data warehousing offering. I don't think this one has
spot pricing either: my guess is availability is low.

>
> The really, really big ones are useful even for pushing limits, such
> as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance
> price (the heavily discounted "can die at any time" one) is $0.343/hr.
> Otherwise, it's 3.500/hr.
>

Just to keep in mind cpus are similar throttled:

One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation.

Who knows what that does to memory bandwidth / context switches etc.

Jesper

On Tue, Jul 9, 2013 at 12:24 AM, Jesper Krogh <jesper(at)krogh(dot)cc> wrote:
>
> The really, really big ones are useful even for pushing limits, such
> as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance
> price (the heavily discounted "can die at any time" one) is $0.343/hr.
> Otherwise, it's 3.500/hr.
>
>
> Just to keep in mind cpus are similar throttled:
>
> One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz
> 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an
> early-2006 1.7 GHz Xeon processor referenced in our original documentation.

This is only a statement of measurement (notably, it also is a metric
that is SMP-processor-count-oblivious), for lack of a more sensible
metric (certainly not clock cycles nor bogomips) in common use.

> Who knows what that does to memory bandwidth / context switches etc.

Virtualization adds complexity, that is true, but so does a new
version of Linux or comparing across microprocessors, motherboards, or
operating systems. I don't see a good reason to be off-put in the
common cases, especially since I can't think of another way to produce
such large machines on a short-term obligation basis.

The advantages are probably diminished (except for testing
virtualization overhead on common platforms) for smaller machines that
can be located sans-virtualization more routinely.

On 07/08/2013 10:49 PM, Daniel Farina wrote:
> On Mon, Jul 8, 2013 at 7:25 PM, Craig Ringer <craig(at)2ndquadrant(dot)com> wrote:
>> I did some work on that a while ago, and found that I was able to get
>> _astonishingly_ stable performance results out of EC2 EBS instances
>> using provisioned IOPS volumes. Running them as "EBS Optimized" didn't
>> seem to be required for the workloads I was testing on.

> The really, really big ones are useful even for pushing limits, such
> as cr1.8xlarge, with 32 CPUs and 244GiB memory. Current spot instance
> price (the heavily discounted "can die at any time" one) is $0.343/hr.
> Otherwise, it's 3.500/hr.

Yes, and for that reason, they aren't realistic for offering to
reviewers for free, funded by the community.

For real performance testing, we already have servers hosted in Portland
which are real, non-shared servers.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com