Re: Postgres alpha testing docs and general test packs

Hi,

Are there any test guides/plans generated for alpha releases, or are
such things only distributed to other developers?  I've seen postings
which mention what the new features are, and links to documentation
and other postings as to what it can do, but no single page outlining
the changes together.

And are there any test packs which can be run against each release to
ensure everything still functions as normal?  What I mean is it would
run through individual tests, like performing an update, checking
whether the update has applied, and returning pass if successful, and
fail otherwise.  Such tests should be inherently massive to match the
feature set of PostgreSQL, but could be built up over time if it
doesn't already exist.  Would there be any value in such a thing, or
is this generally not really a problem that needs solving?

Obviously real-world testing is needed to see how it works in a
realistic scenario, so I'm not suggesting that's any less important.

Thanks

Thom

have you seen that one:
http://it.toolbox.com/blogs/database-soup/alpha2-is-out-and-we-need-you-to-test-it-35032?rss=1
?

2009/10/28 Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com>:
> have you seen that one:
> http://it.toolbox.com/blogs/database-soup/alpha2-is-out-and-we-need-you-to-test-it-35032?rss=1
> ?

That's partly why I was asking. It mentions the areas where the
changes have occurred, but not necessarily the changes themselves. An
example of this is hstore. There are mentions of improvements and
issues being eliminated, but these haven't been specified, not even in
the documentation. I'm not sure how to test whatever change has gone
in. I could open 8.4 and 8.5 documentation for that same page and
flip between the two until I find a difference, but even if I do that
and find changes, I doubt that covers what the fixes are. I'd want
scenarios that were problematic in 8.4 that are not so in 8.5.

Entirely new features are easier to deal with though. I still would,
however, want something like a detailed version of Josh's post which
breaks down where the changes have occurred. It seems quite scattered
and unclear at the moment.

Thom

On Wednesday 28 October 2009 6:46:13 am Thom Brown wrote:
> 2009/10/28 Grzegorz Jaśkiewicz <gryzman(at)gmail(dot)com>:
> > have you seen that one:
> > http://it.toolbox.com/blogs/database-soup/alpha2-is-out-and-we-need-you-t
> >o-test-it-35032?rss=1 ?
>
> That's partly why I was asking. It mentions the areas where the
> changes have occurred, but not necessarily the changes themselves. An
> example of this is hstore. There are mentions of improvements and
> issues being eliminated, but these haven't been specified, not even in
> the documentation. I'm not sure how to test whatever change has gone
> in. I could open 8.4 and 8.5 documentation for that same page and
> flip between the two until I find a difference, but even if I do that
> and find changes, I doubt that covers what the fixes are. I'd want
> scenarios that were problematic in 8.4 that are not so in 8.5.
>
> Entirely new features are easier to deal with though. I still would,
> however, want something like a detailed version of Josh's post which
> breaks down where the changes have occurred. It seems quite scattered
> and unclear at the moment.
>
> Thom

http://developer.postgresql.org/pgdocs/postgres/release-8-5.html

--
Adrian Klaver
aklaver(at)comcast(dot)net

2009/10/28 Adrian Klaver <aklaver(at)comcast(dot)net>:
>> Entirely new features are easier to deal with though.  I still would,
>> however, want something like a detailed version of Josh's post which
>> breaks down where the changes have occurred.  It seems quite scattered
>> and unclear at the moment.
>>
>> Thom
>
> http://developer.postgresql.org/pgdocs/postgres/release-8-5.html

Thanks Adrian. I just wasn't looking hard enough obviously :) That
list still doesn't appear to be explicit enough though as we have
"Multiple improvements in contrib/hstore, including raising limits on
keys and values". What exactly is meant by limit, what was this limit
before and what has it been raised to?

Similarly: "Fix encoding handling in binary input function of xml
type." What was the problem before?

And: "Allow the collection of statistics on sequences". How would
your average end-user see whether these statistics are being colelcted
on sequences? And are these statistics actually used anywhere yet?

I'm not really asking for the answer to those questions. I'm pointing
out that it isn't clear (at least to me) how to determine what exactly
has been fixed in order to test it. This doesn't apply to everything
listed as some of it is quite clear, like "pg_dump/pg_restore --clean
now drops large objects."

Thanks

Thom

Le mercredi 28 octobre 2009 à 15:13:06, Thom Brown a écrit :
> 2009/10/28 Adrian Klaver <aklaver(at)comcast(dot)net>:
> >> Entirely new features are easier to deal with though. I still would,
> >> however, want something like a detailed version of Josh's post which
> >> breaks down where the changes have occurred. It seems quite scattered
> >> and unclear at the moment.
> >>
> >> Thom
> >
> > http://developer.postgresql.org/pgdocs/postgres/release-8-5.html
>
> Thanks Adrian. I just wasn't looking hard enough obviously :) That
> list still doesn't appear to be explicit enough though as we have
> "Multiple improvements in contrib/hstore, including raising limits on
> keys and values". What exactly is meant by limit, what was this limit
> before and what has it been raised to?
>
> Similarly: "Fix encoding handling in binary input function of xml
> type." What was the problem before?
>
> And: "Allow the collection of statistics on sequences". How would
> your average end-user see whether these statistics are being colelcted
> on sequences? And are these statistics actually used anywhere yet?
>
> I'm not really asking for the answer to those questions. I'm pointing
> out that it isn't clear (at least to me) how to determine what exactly
> has been fixed in order to test it. This doesn't apply to everything
> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
> now drops large objects."
>

You're completely right. But release notes never intended to be this. What you
need is more a "visual" tour, and I don't think anyone did write such a thing
for any PostgreSQL releases (but I may be proven wrong). I wrote something
like this in french for 8.2, 8.3, and 8.4. The last two were even published in
a french Linux magazine. I suppose other people from other countries do the
same. The advocacy group would do a good thing if it starts working on this
kind of document. I could probably work on this too.

--
Guillaume.
http://www.postgresqlfr.org
http://dalibo.com

On ons, 2009-10-28 at 14:13 +0000, Thom Brown wrote:
> > http://developer.postgresql.org/pgdocs/postgres/release-8-5.html
>
> Thanks Adrian. I just wasn't looking hard enough obviously :) That
> list still doesn't appear to be explicit enough though as we have
> "Multiple improvements in contrib/hstore, including raising limits on
> keys and values". What exactly is meant by limit, what was this limit
> before and what has it been raised to?
>
> Similarly: "Fix encoding handling in binary input function of xml
> type." What was the problem before?
>
> And: "Allow the collection of statistics on sequences". How would
> your average end-user see whether these statistics are being colelcted
> on sequences? And are these statistics actually used anywhere yet?

I agree some of the release note items could be written in a more useful
way. But ultimately, we can't elaborate on every code fix in detail.
Concentrate on the new features and try them out. They should be
documented. If not, or you can't find the documentation, please report
that.

> I'm not really asking for the answer to those questions. I'm pointing
> out that it isn't clear (at least to me) how to determine what exactly
> has been fixed in order to test it. This doesn't apply to everything
> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
> now drops large objects."

You can be reasonably assured that the particular fixes have been tested
and work, unless they are explicitly documented otherwise. We don't
necessarily need more eyeballs to, say, check that the binary input
function of the xml type has *really* been fixed.

One point of the alpha releases is to test whether nothing else has been
broken by the various fixes, new features, and refactorings. And you
can check that by running your application on top of the new database
server. It helps if you have a test suite for your application. For
example, if the fix of the binary input function of the xml type breaks
your application because it had relied on some undocumented corner case,
now would be good time to find that out.

2009/11/1 Peter Eisentraut <peter_e(at)gmx(dot)net>:
> On ons, 2009-10-28 at 14:13 +0000, Thom Brown wrote:
>> I'm not really asking for the answer to those questions.  I'm pointing
>> out that it isn't clear (at least to me) how to determine what exactly
>> has been fixed in order to test it.  This doesn't apply to everything
>> listed as some of it is quite clear, like "pg_dump/pg_restore --clean
>> now drops large objects."
>
> You can be reasonably assured that the particular fixes have been tested
> and work, unless they are explicitly documented otherwise.  We don't
> necessarily need more eyeballs to, say, check that the binary input
> function of the xml type has *really* been fixed.

Fair enough. :) It's my experience in testing that makes me want to
test particular things to death.

> One point of the alpha releases is to test whether nothing else has been
> broken by the various fixes, new features, and refactorings.  And you
> can check that by running your application on top of the new database
> server.  It helps if you have a test suite for your application.  For
> example, if the fix of the binary input function of the xml type breaks
> your application because it had relied on some undocumented corner case,
> now would be good time to find that out.

Yeah, I should realise this is just an alpha release really.
Unfortunately I'd imagine that most places using PostgreSQL in
production won't be participating in alpha testing at all as there's
no call for it.

No doubt I can find what I'm looking for, even if it's looking through
the commit log. But I guess that's your point, I probably don't need
to do that.

At the moment Gentoo doesn't even want for format my primary partition
(some DRDY error), so I still haven't had the chance to give this a
go. :(

Thom