Re: multi-platform, multi-locale regression tests

I'm looking for some ideas on how to deal with the regression tests for
the per-column collation feature. These are the issues:

* The feature only works on some platforms (tentatively: Linux,
Windows).

-> Possible solution: like xml test

* The locale names are platform dependent, so there would need to be
different test files per locale.

* The test files need to use some non-ASCII characters. So far, I have
encoded the test file in UTF-8 and run the tests with make check
MULTIBYTE=UTF8.

* Also, the allowed collations depend on the server encoding, so any
solution for the previous point that results in the server encoding of
the test database being variable will make the setup of the regression
test SQL file more interesting.

* Of course the actual sort orders could also be different on different
platforms, but that problem can likely be contained.

One possible way out is not to include these tests in the main test set
and instead require manual invocation.

Better ideas?

On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote:

> One possible way out is not to include these tests in the main test set
> and instead require manual invocation.
>
> Better ideas?

I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.

The nice thing about using a TAP-based framework is that you can skip tests that don't meet platform requirements, and compare values within the tests, right where you write them, rather than in a separate file. You can also dynamically change how you compare things depending on the environment, such as the locales that vary on different platforms.

Thoughts?

Best,

David

2010/11/9 David E. Wheeler <david(at)kineticode(dot)com>:
> On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote:
>
>> One possible way out is not to include these tests in the main test set
>> and instead require manual invocation.
>>
>> Better ideas?
>
> I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.
>
> The nice thing about using a TAP-based framework is that you can skip tests that don't meet platform requirements, and compare values within the tests, right where you write them, rather than in a separate file. You can also dynamically change how you compare things depending on the environment, such as the locales that vary on different platforms.
>
> Thoughts?

Are you thinking of a contrib module 'pgtap' that we can use to
accomplish the optionnal regression tests ?

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote:

> Are you thinking of a contrib module 'pgtap' that we can use to
> accomplish the optionnal regression tests ?

Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow.

David

2010/11/9 David E. Wheeler <david(at)kineticode(dot)com>:
> On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote:
>
>> Are you thinking of a contrib module 'pgtap' that we can use to
>> accomplish the optionnal regression tests ?
>
> Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow.

Adding a unit test layer shipped with postgresql sounds good to me.
And pgTAP can claim to be platform agnostic.

--
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

On tis, 2010-11-09 at 14:00 -0800, David E. Wheeler wrote:
> I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they wanted (and had the prerequisites), and the build farm animals would run them regularly.

I'd welcome something like that, but I'm not sure that that's the best
overall solution to this particular problem in the short run. But it
would be great to have anyway.