Re: Two weeks to feature freeze

> -----Original Message-----
> From: Tom Lane [mailto:tgl(at)sss(dot)pgh(dot)pa(dot)us]
> Sent: Friday, June 20, 2003 8:58 PM
> To: Jason Earl
> Cc: Dann Corbit; PostgreSQL-development
> Subject: Re: [HACKERS] Two weeks to feature freeze
>
>
> Jason Earl <jason(dot)earl(at)simplot(dot)com> writes:
> > Hmm... I must have missed the huge corporation paying for in house
> > testing of PostgreSQL. In the Free Software world the
> "beta team" is
> > all of those people that need the new features so badly
> that they are
> > willing to risk their own data and hardware testing it.
>
> I don't have a lot of faith in huge automated test efforts.
> They're great at ensuring you don't make the same mistakes
> you made once before, but in my experience the nastiest bugs
> are the ones you haven't seen before and would never in a
> million years have dreamed to test for.

This is true if and only if the test design is poor.

> Thus, the best test
> team is a bunch of people doing unplanned things with the
> software, on a wide variety of platforms...

That is the worst possible test plan. It totally lacks organization and
there is no hint to define when the feature set has been covered. Ad
hoc testing is a useful addition, but it cannot replace all the standard
tests that have been used by the industry for decades.

If you run literally hundreds of tests designed to ensure that your
product conforms to ANSI/ISO standards then the bugs that are missed
will be few and far between. Unless you are bad at designing tests.

Designing tests is busywork. Desiging tests is boring. Nobody wants to
design tests, let alone interpret the results and define correct
baselines. But testing is very, very important.

Dann Corbit wrote:
> That is the worst possible test plan. It totally lacks organization and
> there is no hint to define when the feature set has been covered. Ad
> hoc testing is a useful addition, but it cannot replace all the standard
> tests that have been used by the industry for decades.
>
> If you run literally hundreds of tests designed to ensure that your
> product conforms to ANSI/ISO standards then the bugs that are missed
> will be few and far between. Unless you are bad at designing tests.
>
> Designing tests is busywork. Desiging tests is boring. Nobody wants to
> design tests, let alone interpret the results and define correct
> baselines. But testing is very, very important.

I remember when I was with Great Bridge they said, "Oh, we are going to
have a test setup and do all sorts of testing to improve PostgreSQL." I
told them I doubted their testing was going to shake out many more bugs
than our existing testing setup, and you know what, I was pretty much
right. Sure, they found a few, but it wasn't much.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Fri, 20 Jun 2003, Dann Corbit wrote:

> Designing tests is busywork. Desiging tests is boring. Nobody wants to
> design tests, let alone interpret the results and define correct
> baselines. But testing is very, very important.

But we do do testing ... we even design testing (in the form of the
regression tests) ... we just don't do testing that you personally approve
of ... and, from what I've seen so far, you aren't willing to actually put
*your* time where your mouth is ... design some tests and submit them to
us ... if they are valid, they will get used ...

If you feel that crash-me is a valid starting point, start there and see
where it takes you ...

The Hermit Hacker wrote:
> On Fri, 20 Jun 2003, Dann Corbit wrote:
>
>> Designing tests is busywork. Desiging tests is boring. Nobody wants to
>> design tests, let alone interpret the results and define correct
>> baselines. But testing is very, very important.
>
> But we do do testing ... we even design testing (in the form of the
> regression tests) ... we just don't do testing that you personally approve
> of ... and, from what I've seen so far, you aren't willing to actually put
> *your* time where your mouth is ... design some tests and submit them to
> us ... if they are valid, they will get used ...
>
> If you feel that crash-me is a valid starting point, start there and see
> where it takes you ...

Not that fast! I didn't take the time to check but it wouldn't surprise
me if MySQL's crash-me is GPL'd and copyright MySQL AB. That's not an
optimal point to start PostgreSQL test code from, is it?

Jan

--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck(at)Yahoo(dot)com #

On Sun, 22 Jun 2003, Jan Wieck wrote:

> The Hermit Hacker wrote:
> > On Fri, 20 Jun 2003, Dann Corbit wrote:
> >
> >> Designing tests is busywork. Desiging tests is boring. Nobody wants to
> >> design tests, let alone interpret the results and define correct
> >> baselines. But testing is very, very important.
> >
> > But we do do testing ... we even design testing (in the form of the
> > regression tests) ... we just don't do testing that you personally approve
> > of ... and, from what I've seen so far, you aren't willing to actually put
> > *your* time where your mouth is ... design some tests and submit them to
> > us ... if they are valid, they will get used ...
> >
> > If you feel that crash-me is a valid starting point, start there and see
> > where it takes you ...
>
> Not that fast! I didn't take the time to check but it wouldn't surprise
> me if MySQL's crash-me is GPL'd and copyright MySQL AB. That's not an
> optimal point to start PostgreSQL test code from, is it?

I didn't say to copy it, but if the format is what Dann feels is required
to be taken seriously, it does give a starting point to work from ...

the thing is, as I think it was Tom that pointed out, the crash-me is more
a feature tester then anything ... but Dann appears to be stuck on it as
the 'be all, end all of testing suites' ...