Re: Concurrency testing

Folks,

As we move forward, we run into increasingly complex situations under
the general rubric of concurrency.

What test frameworks are already out there that we can use in our
regression test suite? If there aren't any, how might we build one?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Wed, Oct 07, 2009 at 09:17:52AM -0700, David Fetter wrote:
> Folks,
>
> As we move forward, we run into increasingly complex situations under
> the general rubric of concurrency.
>
> What test frameworks are already out there that we can use in our
> regression test suite? If there aren't any, how might we build one?

Apparently, this wasn't as crystal clear as I thought. Let's imagine
you want to test some odd combination of transactions:

T1: SET TRANSACTION ISOLATION SERIALIZABLE;
T1: SELECT...FOR UPDATE...;
T2: UPDATE...
T3: COPY...
T1: ...

At the moment, we have no way to test this, although with certain Perl
modules, it would be pretty trivial.

I seem to recall that there were some patches to get psql to help with
such things, but they didn't go in. Time to revive them?

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

David Fetter wrote:

> I seem to recall that there were some patches to get psql to help with
> such things, but they didn't go in. Time to revive them?

Yeah, the API they implemented wasn't ideal, so there was some
discussion that ended up with a specification everyone was happy with,
but then nobody got around to implementing it. Feel free to whack that
patch and resubmit ... See in the archives around here:
http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us

--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

On Oct 7, 2009, at 9:53 AM, David Fetter wrote:

> At the moment, we have no way to test this, although with certain Perl
> modules, it would be pretty trivial.

No non-core modules necessary. Just use Test::More and file handles
opened via pipes to two or more psql sessions.

Best,

David

On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote:
> On Oct 7, 2009, at 9:53 AM, David Fetter wrote:
>
>> At the moment, we have no way to test this, although with certain
>> Perl modules, it would be pretty trivial.
>
> No non-core modules necessary. Just use Test::More and file handles
> opened via pipes to two or more psql sessions.

When did Test::More become core? I believe we support back to Perl
5.6 :/

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Oct 7, 2009, at 10:38 AM, David Fetter wrote:

> When did Test::More become core? I believe we support back to Perl
> 5.6 :/

Module::CoreList says 5.006002, though I would have sworn it was added
much earlier than that. You could always use Test.pm, I suppose, which
has been in core since 5.00405.

Best,

David

On Wed, 7 Oct 2009, Alvaro Herrera wrote:

> Yeah, the API they implemented wasn't ideal, so there was some
> discussion that ended up with a specification everyone was happy with,
> but then nobody got around to implementing it.

I needed something like this and didn't implement those suggestions
because I thought the whole idea didn't scale up enough. That's close to
the right API to allow more complicated regression tests in psql itself,
but I doubted that would hit real complexity level needed to find the good
concurrent bugs.

The stuff I've been building lately takes "how many sessions at once?" as
an input, and the useful range on that (particularly as we wander further
toward multi-core land) is much higher than you'd want to manage one at a
time manually. I'd rather see a "session simulator" program that picks
among several types of behaviors and executes them, and then you can throw
as many of those as you want into the mix. You can do something like that
with pgbench right now if you pass it multiple files, the tricky part is
figuring out what regression output you should expect.

We had a quick session on more complicated PG testing at PGCon, and it
seems many of us who looked at this from multiple perspectives have
settled into the idea that the current regression tests should stay as
they are, and the more complicated ones are going to need to be in Perl to
keep the code complexity managable but still allow more complicated tests.
Another example of a complicted test is something that tests
pg_dump/pg_restore, which is also hard to shoehorn into pg_regress. I
personally would rather see progress made in that area than trying to hack
increasingly difficult tests into the existing regression framework,
particular given how changes there impact existing committer's daily
workflow. The complexity level of test you need to find the juicy
concurrent issues is not one you're going to want to run every time you do
"make check".

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

Greg Smith wrote:
> On Wed, 7 Oct 2009, Alvaro Herrera wrote:
>
> >Yeah, the API they implemented wasn't ideal, so there was some
> >discussion that ended up with a specification everyone was happy with,
> >but then nobody got around to implementing it.
>
> I needed something like this and didn't implement those suggestions
> because I thought the whole idea didn't scale up enough. That's
> close to the right API to allow more complicated regression tests in
> psql itself, but I doubted that would hit real complexity level
> needed to find the good concurrent bugs.

I don't find this a compelling argument against concurrent psql. Sure
there are things you can't do with it, but it doesn't mean it's not
useful. Are we going to need further tools to find "the good concurrent
bugs"? No doubt.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

David Fetter wrote:
> Folks,
>
> As we move forward, we run into increasingly complex situations under
> the general rubric of concurrency.
>
> What test frameworks are already out there that we can use in our
> regression test suite? If there aren't any, how might we build one?

Not entirely on-topic, but I thought I'd chime in:

After seeing the results from the hungarian blogger whose name I can't
find [1] and quite a bit of discussion in the german usergroup before
that I wanted to see how hard it would be to actually implement a
framework to automate performance test runs.

I mainly used it as a reason to have a look at Moose [2], but since it
was quite entertaining so far and I don't see any roadblocks I think
I'll follow through.

Design Targets:

*) OO Design so it's easily pluggable, extensible (& testable ?)
*) Standalone CLI mode as well as daemonized "benchmark client" (think
buildfarm)
*) Automate all aspects of benchmark operation, starting with a clean
slate and (optionally) returning to same
*) Extensible Benchmark system, currently thinking about:
*) Synthetic cpu/memory/disk benchmarks (for evaluating and verifying
the performance baseline of a given system, probably sysbench?)
*) pgbench
*) Whatever is freely available in the DBT/TPC environment
*) As discussed, custom benchmarks testing specific subsystems with
high concurrency/throughput, would need a proper tool I presume

Currently it doesn't do much more than building branches and creating
clusters, but I guess I'll have something rudimentarily usable by mid
november.

I haven't thought about result aggregation & rendering/UI part of the
whole thing so far, so if anyone has some ideas in that direction they'd
be very much appreciated when the time has come.

If I duplicated someones efforts already I'm sorry, I didn't want to
raise anybodys hopes before I was sure that I really want to do this ;).

best regards,
Michael

[1] http://suckit.blog.hu/2009/09/26/postgresql_history
[2] http://www.iinteractive.com/moose/

On Wed, 7 Oct 2009, Alvaro Herrera wrote:

> I don't find this a compelling argument against concurrent psql. Sure
> there are things you can't do with it, but it doesn't mean it's not
> useful. Are we going to need further tools to find "the good concurrent
> bugs"? No doubt.

Don't get me wrong, I wasn't arguing against concurrent psql being useful.
Certainly it is. I was just suggesting that the scale of issues it can be
useful for is still pretty limited, and that accordingly I found my time
better spent working on a higher-level solution that didn't need C-psql
anyway. Whether C-psql is sufficient for what David had in mind I can't
say.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Wed, Oct 07, 2009 at 08:06:50PM -0400, Greg Smith wrote:
> On Wed, 7 Oct 2009, Alvaro Herrera wrote:
>
>> I don't find this a compelling argument against concurrent psql.
>> Sure there are things you can't do with it, but it doesn't mean
>> it's not useful. Are we going to need further tools to find "the
>> good concurrent bugs"? No doubt.
>
> Don't get me wrong, I wasn't arguing against concurrent psql being
> useful. Certainly it is. I was just suggesting that the scale of
> issues it can be useful for is still pretty limited, and that
> accordingly I found my time better spent working on a higher-level
> solution that didn't need C-psql anyway. Whether C-psql is
> sufficient for what David had in mind I can't say.

The kind of stuff I have in mind is regression tests. As with
coverage and Dtrace, they might not be the ones that get run by
default, but right now, we have no real way in the regression tests to
test concurrency issues at all, as far as I know, so it would be good
to have some way to catch this stuff.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Wed, Oct 7, 2009 at 10:38 AM, David Fetter <david(at)fetter(dot)org> wrote:
> On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote:
>> On Oct 7, 2009, at 9:53 AM, David Fetter wrote:
>>
>>> At the moment, we have no way to test this, although with certain
>>> Perl modules, it would be pretty trivial.
>>
>> No non-core modules necessary. Jus use Test::More and file handles
>> opened via pipes to two or more psql sessions.

I'd much rather live without Test::More and use DBD::Pg, then have
Test::More but need to open pipes to psql to talk to the database,
rather than using DBI to do it. But I guess we would need to worry
about whether we can make DBD::Pg work with the installation being
tested, rather than finding some other install.

>
> When did Test::More become core?  I believe we support back to Perl
> 5.6 :/

Do we need to restrict ourselves to core? Developers already need
flex and bison, which aren't needed when installing from the tarball.
Couldn't we also have "make dev-check" that has higher requirements
than "make check" does, but does a more thorough job?

Jeff

On Wed, 7 Oct 2009, Michael Renner wrote:

> I haven't thought about result aggregation & rendering/UI part of the whole
> thing so far, so if anyone has some ideas in that direction they'd be very
> much appreciated when the time has come.

What I did in pgbench-tools (now available at http://git.postgresql.org/ )
is just dump all the individual run results, tagged by the name of test
and similar key information, into a database. That lets people aggregate
however they want using SQL, and cleanly separates out the data from the
UI. My current rendering approach just pulls data from there into
columnar text files I feed into gnuplot, but you could easily render with
pretty much anything else instead just by tweaking how you pull things
from the database.

All of the dbt-X tools (also in that git repo) instead use simple columnar
text files (and their memory resident equivalent) to hold everything,
feeding the data through the appropriate Perl libraries to compute stats
and render things. Since it drives me crazy to manipulate tables of data
that could go into a database in a scripting language, I never bought into
that approach. But it's completely viable too for a standalone tool.

I'm not aware of anyone working on the job management side of the
"performance farm" yet, so I don't think you duplicated anybody else's
work. Mark Wong (who manages the dbt stuff) and I have both been doing
work on the benchmark results collection and organization side of things,
but basically one test at a time. The specific piece I've been working on
lately is spawning off system monitoring daemons to collect information
during the test, I think I'm on my 3rd generation of trying to get a
solution I'm happy with to that problem.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:

> I'd much rather live without Test::More and use DBD::Pg, then have
> Test::More but need to open pipes to psql to talk to the database,
> rather than using DBI to do it. But I guess we would need to worry
> about whether we can make DBD::Pg work with the installation being
> tested, rather than finding some other install.

The test architecture depends on Perl, but not on the DBI. I don't
think that Andrew wants to add any dependencies. Therefore we'd need
to use file handles. That's not to say that we couldn't write a nice
little interface for it such that the implementation could later change.

> Do we need to restrict ourselves to core? Developers already need
> flex and bison, which aren't needed when installing from the tarball.
> Couldn't we also have "make dev-check" that has higher requirements
> than "make check" does, but does a more thorough job?

flex and bison are not Perl modules.

Best,

David

Greg Smith wrote:
> On Wed, 7 Oct 2009, Michael Renner wrote:
>
>> I haven't thought about result aggregation & rendering/UI part of the
>> whole thing so far, so if anyone has some ideas in that direction
>> they'd be very much appreciated when the time has come.
>
> What I did in pgbench-tools (now available at http://git.postgresql.org/

Thanks, I'll look into it! I think the nicest solution would be
something in the liking of Sun's Analytics [1] framework, especially
when you've got large amounts of historic data to correlate. But this
is probably a completely different beast to implement, though nice
Javascript rendering libraries become commonplace these days. We'll see...

> I'm not aware of anyone working on the job management side of the
> "performance farm" yet, so I don't think you duplicated anybody else's
> work.

Good to hear :)

> things, but basically one test at a time. The specific piece I've been
> working on lately is spawning off system monitoring daemons to collect
> information during the test, I think I'm on my 3rd generation of trying
> to get a solution I'm happy with to that problem.

Ahh, that'd be also nice to have to see how ressource consumption
differs between different runs & branches. This is probably highly
platform dependent though...

Michael

[1] http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/

On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
>> Do we need to restrict ourselves to core?  Developers already need
>> flex and bison, which aren't needed when installing from the tarball.
>> Couldn't we also have "make dev-check" that has higher requirements
>> than "make check" does, but does a more thorough job?
>
> flex and bison are not Perl modules.

True, but so what?

I don't really see what's wrong with using Perl modules that are
likely to be installed most places and easy to obtain where not, if it
makes writing a test framework much easier. But I also think that we
should not get bogged down on exactly which tools to use - it seems to
me the first thing is to find someone who is willing to do the work.
If someone makes an AWESOME test suite that uses a module which is a
little too adventurous, we can probably find a way of adjusting it
after the fact so as to remove the dependency (I fancy myself fairly
good at this sort of thing, where Perl is concerned). But if we argue
about tools now, we're just going to discourage anyone from taking a
stab at it.

Just MHO, of course.

...Robert

David E. Wheeler wrote:
> On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:
>
>> I'd much rather live without Test::More and use DBD::Pg, then have
>> Test::More but need to open pipes to psql to talk to the database,
>> rather than using DBI to do it. But I guess we would need to worry
>> about whether we can make DBD::Pg work with the installation being
>> tested, rather than finding some other install.
>
> The test architecture depends on Perl, but not on the DBI. I don't
> think that Andrew wants to add any dependencies. Therefore we'd need
> to use file handles. That's not to say that we couldn't write a nice
> little interface for it such that the implementation could later change.

Well, that's true of the buildfarm. And there are reasons I don't want
to use DBI for the buildfarm, mainly to do with its intended role in
simulating what a human would do by hand.

What we do for the core testing framework is a different question.
Nevertheless, a requirement for DBI and DBD::Pg would be a significant
escalation of testing prerequisites. Test::More is comparatively modest
requirement, and is fairly universal where Perl is installed. And since
we'd just be using it to drive psql, we wouldn't be having to decide if
a problem we saw was due to a problem in Postgres or a problem in DBD::Pg.

If we want something to drive a huge number of clients, I suspect
neither of these is a good way to go, and something more custom built
would be required. Last time I built something to drive a huge client
load (many thousands of simultaneous connections to a web app) I did it
in highly threaded Java using HttpUnit from a number of separate client
machines. You wouldn't believe what that managed to do to MySQL on the
backend ;-)

cheers

andrew

On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
>>> Do we need to restrict ourselves to core?  Developers already need
>>> flex and bison, which aren't needed when installing from the tarball.
>>> Couldn't we also have "make dev-check" that has higher requirements
>>> than "make check" does, but does a more thorough job?
>>
>> flex and bison are not Perl modules.
>
> True, but so what?

Right, my point was only that we already have different levels of
requirements for different things.

>
> I don't really see what's wrong with using Perl modules that are
> likely to be installed most places and easy to obtain where not, if it
> makes writing a test framework much easier.  But I also think that we
> should not get bogged down on exactly which tools to use - it seems to
> me the first thing is to find someone who is willing to do the work.
> If someone makes an AWESOME test suite that uses a module which is a
> little too adventurous, we can probably find a way of adjusting it
> after the fact so as to remove the dependency (I fancy myself fairly
> good at this sort of thing, where Perl is concerned).  But if we argue
> about tools now, we're just going to discourage anyone from taking a
> stab at it.

OK, but what would we be taking a stab at? Would it simply be
something like "make dev-check" or "make concurrency-check", which
the build farm would then just invoke, thereby killing two birds with
one stone? Or would it have to be something fancier than just that?
Or am I already too lost in the details?

Jeff

On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler <david(at)kineticode(dot)com> wrote:
> On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote:
>
>> I'd much rather live without Test::More and use DBD::Pg, then have
>> Test::More but need to open pipes to psql to talk to the database,
>> rather than using DBI to do it.  But I guess we would need to worry
>> about whether we can make DBD::Pg work with the installation being
>> tested, rather than finding some other install.
>
> The test architecture depends on Perl, but not on the DBI. I don't think
> that Andrew wants to add any dependencies. Therefore we'd need to use file
> handles.

If we are going to talk to multiple psql pipes at the same time, and
not self-deadlock in the process, we would probably need to use some
form of non-blocking IO. Unfortunately, I've often found it isn't as
easy as one might hope to do that in a completely portable way. Maybe
IPC::Run, but I think that that is non-core also.

Are we interested only in simple deterministic concurrency testing: T1
does A, T2 does B, T1 does C and must block until T2 does D, at which
point (and not before) T1 must respond with E. (Of course from some
recent testing I've done, we would also want "T1 does C and must block
either until T2 does D or until X+/-Y microseconds have passed")

Or would we want to beat the snot out the database in parallel in a
nondeterministic fashion, and checking for self-consistency and
non-deadlock?

Cheers,

Jeff

On Wed, Oct 7, 2009 at 10:41 PM, Jeff Janes <jeff(dot)janes(at)gmail(dot)com> wrote:
>> I don't really see what's wrong with using Perl modules that are
>> likely to be installed most places and easy to obtain where not, if it
>> makes writing a test framework much easier.  But I also think that we
>> should not get bogged down on exactly which tools to use - it seems to
>> me the first thing is to find someone who is willing to do the work.
>> If someone makes an AWESOME test suite that uses a module which is a
>> little too adventurous, we can probably find a way of adjusting it
>> after the fact so as to remove the dependency (I fancy myself fairly
>> good at this sort of thing, where Perl is concerned).  But if we argue
>> about tools now, we're just going to discourage anyone from taking a
>> stab at it.
>
> OK, but what would we be taking a stab at?  Would it simply be
> something like "make dev-check"  or "make concurrency-check", which
> the build farm would then just invoke, thereby killing two birds with
> one stone?  Or would it have to be something fancier than just that?
> Or am I already too lost in the details?

Well, I don't know. I think the first thing would be to figure out
what sort of tests we want to run. Taking a stab in the dark, I'm
thinking that the point of any sort of concurrency testing must be to
check for race conditions, so presumably it's the sort of thing you'd
want to have an easy way to run over and over again in a loop.
Possibly you would also want to test recovery by killing a backend in
the middle somewhere. It may be too much to put into the core tree at
all; it might be easier to maintain as a separate project. But I
don't have a strong opinion on this. The subject of the thread is
"concurrency testing" so it seems to me we ought to start by figuring
out exactly what that means and how we could test it.

...Robert

Greg Smith <gsmith(at)gregsmith(dot)com> writes: > The stuff I've been
building lately takes "how many sessions at once?" as an > input,
and the useful range on that (particularly as we wander further >
toward multi-core land) is much higher than you'd want to manage
one at a > time manually. I'd rather see a "session simulator"
program that picks > among several types of behaviors and executes
them, and then you can throw > as many of those as you want into
the mix. You can do something like that > with pgbench right now
if you pass it multiple files, the tricky part is > figuring out
what regression output you should expect. That tool exists and
do support PostgreSQL. It's called Tsung and is Open Source:

http://tsung.erlang-projects.org/
http://tsung.erlang-projects.org/user_manual.html
http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php

It can handle thousands of connections per node and as many nodes
as you
like, and already includes OS monitoring (adding to its own
session /
transaction monitoring, of course). It also support SNMP here. Oh,
and
thinktime too `randomize using a probability distribution
(currently exponential)'.

I even started a DBT2 implementation as tsung modules, back when
returning from pgcon 2006:

http://tapoueh.org/misc.html#sec7
darcs get http://pgsql.tapoueh.org/dbt2-tsung/

The thing that tsung isn't made for is tracking the results
themselves
to validate against regressions. This would have to be done by
other
means, maybe server side.

So Tsung ain't a direct answer to OP [waves to david], or only if
we
want to add it the checking capability. I'll talk about it with
tsung
main author. But it answers a lot of the content of this thread,
so I figured I'd better talk about it before it's being poorly
rewritten in perl.

Regards,
--
dim

PS: erlang is damn easy to learn.

PPS: sorry guys, perl is not made for concurrent programming, it's not
going to be easy reaching current tsung level (years of work with ad-hoc
tools) using a general hobbyist programming language.

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> Last time I built something to drive a huge client load (many
> thousands of simultaneous connections to a web app) I did it in highly
> threaded Java using HttpUnit from a number of separate client machines. You
> wouldn't believe what that managed to do to MySQL on the backend ;-)

Last time I've head about that kind of Java stuff, it was a JMeter
story. The webapp guys and the JVM admins were so proud that the load
test couldn't worry their setup that they were willing to retest using
tsung.

On a single client rather than the client farm, tsung pushed their setup
to an halt under heavy load... and that was with older erlang limits of
about 800 connections per node, now it's on the 50000 connections per
node ballpark IIRC.

Just saying :)
--
dim

On Thu, Oct 08, 2009 at 10:28:53AM +0200, Dimitri Fontaine wrote:
> http://tsung.erlang-projects.org/
> http://tsung.erlang-projects.org/user_manual.html
> http://archives.postgresql.org/pgsql-admin/2008-12/msg00032.php

It looks like we're talking about two orthogonal projects here:

1. Test situations which require more than one connection, possibly
to different clusters, for example in the cases of Hot Standby and
Streaming Replication. This further divides into event-based and
time-based tests. It's this situation I had in mind when I posted,
and it was inspired by bugs I've seen, the most recent being in
writeable CTEs.

2. Test situations which require *many* connections in order to find
issues caused, in essence, by load.

Tsung seems well-suited to the latter.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Thu, 8 Oct 2009, Dimitri Fontaine wrote:

> sorry guys, perl is not made for concurrent programming, it's not going
> to be easy reaching current tsung level (years of work with ad-hoc
> tools) using a general hobbyist programming language.

Perl is the best tool available that runs on the buildfarm, which makes
something written in it useful here, while Tsung is not. Unfortunately.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote:
> On Thu, 8 Oct 2009, Dimitri Fontaine wrote:
>
> > sorry guys, perl is not made for concurrent programming, it's not going
> > to be easy reaching current tsung level (years of work with ad-hoc
> > tools) using a general hobbyist programming language.

Woah... I am not much of a Perl fan but to call it a "general hobbyist
programming language" certainly identifies the ignorance of the poster.

Joshua D. Drake

>
> --
> * Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD
>
--
PostgreSQL.org Major Contributor
Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564
Consulting, Training, Support, Custom Development, Engineering
If the world pushes look it in the eye and GRR. Then push back harder. - Salamander

Le 8 oct. 2009 à 19:00, Greg Smith <gsmith(at)gregsmith(dot)com> a écrit :
> Perl is the best tool available that runs on the buildfarm, which
> makes something written in it useful here, while Tsung is not.
> Unfortunately.

I though we were talking about 2 separate issues here: david's one,
for concurrent regression testing, and yours, for load stressing.

It does not look like the buildfarm is a good match for the later, so
I imagined we could see about bypassing perl this time ( given a
serious motive to ).

Regards,
--
dim
>

On Thu, 2009-10-08 at 13:00 -0400, Greg Smith wrote:
> Perl is the best tool available that runs on the buildfarm, which makes
> something written in it useful here, while Tsung is not. Unfortunately.

But you're not going to do performance testing on the buildfarm anyway.

Hi,

David Fetter wrote:
> 1. Test situations which require more than one connection, possibly
> to different clusters, for example in the cases of Hot Standby and
> Streaming Replication. This further divides into event-based and
> time-based tests. It's this situation I had in mind when I posted,
> and it was inspired by bugs I've seen, the most recent being in
> writeable CTEs.

Hm.. not sure what you mean by time-base tests, but to me that
distinction sounds like regression vs. performance testing. Or do you
think of time-based regression tests? I certainly favor regression tests
that are event-based, as I've run into too many false errors due to
unexpected timings already.

> 2. Test situations which require *many* connections in order to find
> issues caused, in essence, by load.
>
> Tsung seems well-suited to the latter.

Agreed. I would also note here, that Tsung seems especially well suited
for performance testing of clustered solutions, as you very likely have
to cluster the testing agents as well to put a decent load on the SUT.

Regards

Markus Wanner

Hi,

Dimitri Fontaine wrote:
> I even started a DBT2 implementation as tsung modules, back when
> returning from pgcon 2006:
>
> http://tapoueh.org/misc.html#sec7
> darcs get http://pgsql.tapoueh.org/dbt2-tsung/

Now *that* is very cool! I definitely need to have a look at it.

Have you tried Sun Faban? It allows to manage test runs, compare test
results and configurations and features a web interface. Pretty nice as
well. But it's all Java, which I like even less than Perl.

However, as far as I know, neither of the two support downloading code
from a repository and building automatically, before testing. While the
buildfarm already does that (partly, testing single patches would be a
nice to have, too).

Regards

Markus Wanner

Markus Wanner wrote:
> Hi,
>
> Dimitri Fontaine wrote:
>> I even started a DBT2 implementation as tsung modules, back when
>> returning from pgcon 2006:
>>
>> http://tapoueh.org/misc.html#sec7
>> darcs get http://pgsql.tapoueh.org/dbt2-tsung/
>
> Now *that* is very cool! I definitely need to have a look at it.
>
> Have you tried Sun Faban? It allows to manage test runs, compare test
> results and configurations and features a web interface. Pretty nice
> as well. But it's all Java, which I like even less than Perl.
>
> However, as far as I know, neither of the two support downloading code
> from a repository and building automatically, before testing. While
> the buildfarm already does that (partly, testing single patches would
> be a nice to have, too).
>
>

It does, sort of. It has a switch called --from-source which lets you
specify a local source repository, which can be something you've applied
a patch to.

Use of this switch disables fetching the source code, checking it for
cleanliness, and uploading the results to the central server.

It was designed for testing patches.

Here's how I use it:

pushd srcroot/HEAD
cp -a pgsql pgsql.thistest
cd pgsql.thistest
cvs update
patch -p 0 < /path/to/patch_to_test
popd
./run_build.pl --verbose --from-source=`pwd`/srcroot/HEAD/pgsql.thistest

cheers

andrew

On Fri, Oct 09, 2009 at 08:34:55PM +0200, Markus Schiltknecht wrote:
> Hi,
>
> David Fetter wrote:
>> 1. Test situations which require more than one connection, possibly
>> to different clusters, for example in the cases of Hot Standby and
>> Streaming Replication. This further divides into event-based and
>> time-based tests. It's this situation I had in mind when I posted,
>> and it was inspired by bugs I've seen, the most recent being in
>> writeable CTEs.
>
> Hm.. not sure what you mean by time-base tests,

Kick off x.
Wait time t1
Kick off y.
Wait time t2.
etc., etc.

> but to me that distinction sounds like regression vs. performance
> testing. Or do you think of time-based regression tests? I
> certainly favor regression tests that are event-based, as I've run
> into too many false errors due to unexpected timings already.

Totally agree about regression tests. There might be other kinds of
test--stress- or performance tests, for example--that would make more
sense time-based.

>> 2. Test situations which require *many* connections in order to
>> find issues caused, in essence, by load.
>>
>> Tsung seems well-suited to the latter.
>
> Agreed. I would also note here, that Tsung seems especially well
> suited for performance testing of clustered solutions, as you very
> likely have to cluster the testing agents as well to put a decent
> load on the SUT.

I'd *love* to see PostgreSQL solutions that would outrun a single
decent Tsung box, but I suspect that's at least a year or two away.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Wed, 2009-10-07 at 13:07 -0400, Alvaro Herrera wrote:
> David Fetter wrote:
>
> > I seem to recall that there were some patches to get psql to help with
> > such things, but they didn't go in. Time to revive them?
>
> Yeah, the API they implemented wasn't ideal, so there was some
> discussion that ended up with a specification everyone was happy with,
> but then nobody got around to implementing it. Feel free to whack that
> patch and resubmit ... See in the archives around here:
> http://archives.postgresql.org/message-id/8204.1207689056@sss.pgh.pa.us

I would hope that Concurrent psql can be revived. There were some
issues, but not really major ones.

The main requirement is to be able to specify multiple sessions of
activity from a single script. I would prefer it if we could do that via
psql. If we start inventing new features in other tools we get situation
similar to pgbench, which has some cute features, but they aren't in
psql, which also has cute features, but different ones. Fragmentation
wastes effort.

I think Greg's comments are correct but I would say "also correct".
There is no reason to have just one test framework. We need as many as
we need.

--
Simon Riggs www.2ndQuadrant.com