Re: libpq URI and regression testing

Hi,

When I committed Alex Shulgin's patch to add URI support to libpq, I
included the test harness as well. However, due to it being in a
separate subdirectory that did not previously had tests, it's not being
run by buildfarm.

It's not considered in "make installcheck-world" either.

What's the preferred way to make it automatically tested as much as
possible? I know the buildfarm does not run "installcheck-world", so if
we want it there, it'd need a bit more code on the client side. I think
it would be wise to have it also run on installcheck-world.

Hmm. Just had this thought: not all platform support the same socket
types. Maybe we should have separated the .in file in three parts:
IPv4, IPv6, unix-domain socket. That way each platform would only run
tests that pertain to it. Right now there's a single "regress.in" file
that lists all the tests.

Opinions?

--
Álvaro Herrera <alvherre(at)alvh(dot)no-ip(dot)org>

On tis, 2012-04-17 at 10:47 -0300, Alvaro Herrera wrote:
> What's the preferred way to make it automatically tested as much as
> possible? I know the buildfarm does not run "installcheck-world", so if
> we want it there, it'd need a bit more code on the client side. I think
> it would be wise to have it also run on installcheck-world.

It was agreed during the patch discussion that it shouldn't be run
automatically.

> Hmm. Just had this thought: not all platform support the same socket
> types. Maybe we should have separated the .in file in three parts:
> IPv4, IPv6, unix-domain socket. That way each platform would only run
> tests that pertain to it. Right now there's a single "regress.in" file
> that lists all the tests.

That's one reason for that, but there are probably others in the way of
making this fully portable and automatable.

Excerpts from Peter Eisentraut's message of mar abr 17 15:41:04 -0300 2012:
> On tis, 2012-04-17 at 10:47 -0300, Alvaro Herrera wrote:
> > What's the preferred way to make it automatically tested as much as
> > possible? I know the buildfarm does not run "installcheck-world", so if
> > we want it there, it'd need a bit more code on the client side. I think
> > it would be wise to have it also run on installcheck-world.
>
> It was agreed during the patch discussion that it shouldn't be run
> automatically.

Oh, okay. I didn't notice that. I guess we should issue a
call-for-testing, then, so that we ensure it works (FSVO works) in all
(FSVO all) platforms.

> > Hmm. Just had this thought: not all platform support the same socket
> > types. Maybe we should have separated the .in file in three parts:
> > IPv4, IPv6, unix-domain socket. That way each platform would only run
> > tests that pertain to it. Right now there's a single "regress.in" file
> > that lists all the tests.
>
> That's one reason for that, but there are probably others in the way of
> making this fully portable and automatable.

Hmm.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On 04/17/2012 02:47 PM, Alvaro Herrera wrote:
> Excerpts from Peter Eisentraut's message of mar abr 17 15:41:04 -0300 2012:
>> On tis, 2012-04-17 at 10:47 -0300, Alvaro Herrera wrote:
>>> What's the preferred way to make it automatically tested as much as
>>> possible? I know the buildfarm does not run "installcheck-world", so if
>>> we want it there, it'd need a bit more code on the client side. I think
>>> it would be wise to have it also run on installcheck-world.
>> It was agreed during the patch discussion that it shouldn't be run
>> automatically.
> Oh, okay. I didn't notice that. I guess we should issue a
> call-for-testing, then, so that we ensure it works (FSVO works) in all
> (FSVO all) platforms.
>
>>> Hmm. Just had this thought: not all platform support the same socket
>>> types. Maybe we should have separated the .in file in three parts:
>>> IPv4, IPv6, unix-domain socket. That way each platform would only run
>>> tests that pertain to it. Right now there's a single "regress.in" file
>>> that lists all the tests.
>> That's one reason for that, but there are probably others in the way of
>> making this fully portable and automatable.

This test setup also appears to labor under the illusion that we live in
a Unix-only world. And for no good reason that I can tell. The shell
script should be ripped out and replaced by a perl script which could
actually be used on any windows build host. The MSVC build system also
needs adjusting to make it build the test driver, at least.

cheers

andrew

Excerpts from Andrew Dunstan's message of mar abr 17 16:03:50 -0300 2012:

> >> That's one reason for that, but there are probably others in the way of
> >> making this fully portable and automatable.
>
> This test setup also appears to labor under the illusion that we live in
> a Unix-only world. And for no good reason that I can tell. The shell
> script should be ripped out and replaced by a perl script which could
> actually be used on any windows build host. The MSVC build system also
> needs adjusting to make it build the test driver, at least.

I'll see about it.

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:

> On tis, 2012-04-17 at 10:47 -0300, Alvaro Herrera wrote:
>> What's the preferred way to make it automatically tested as much as
>> possible? I know the buildfarm does not run "installcheck-world", so if
>> we want it there, it'd need a bit more code on the client side. I think
>> it would be wise to have it also run on installcheck-world.
>
> It was agreed during the patch discussion that it shouldn't be run
> automatically.

Hm... the testing program we've actually committed avoids any connection
attempts and only deals with testing the parser routine. Does it change
that?

--
Regards,
Alex

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:

>>> That's one reason for that, but there are probably others in the way of
>>> making this fully portable and automatable.
>
> This test setup also appears to labor under the illusion that we live
> in a Unix-only world. And for no good reason that I can tell. The
> shell script should be ripped out and replaced by a perl script which
> could actually be used on any windows build host. The MSVC build
> system also needs adjusting to make it build the test driver, at
> least.

Good catch. Attached is my first take at writing Perl: replaces the
shell script, adds $libpq_uri_regress project to Mkvcbuild.pm.

I don't have access to a win32 box unfortunately, so if anyone who does
could try this out that'd be great.

--
Regards,
Alex

On tor, 2012-04-19 at 00:13 +0300, Alex wrote:
> +#!/usr/bin/env perl

Don't do that. Call the script using $(PERL) from the makefile.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:

> On tor, 2012-04-19 at 00:13 +0300, Alex wrote:
>> +#!/usr/bin/env perl
>
> Don't do that. Call the script using $(PERL) from the makefile.

Thank you for the suggestion. Attached v2 does just this (while keeping
a more commonly found shebang line in the perl script for running it w/o
the makefile.)

I figure src/tools/msvc/vcregress.pl will need to be updated too, but
trying to model all this after ecpg regression tests I'm stuck at
replicating ecpg_regression.proj for libpq's uri-regress. I'd
appreciate any help from the Windows guys at this point.

--
Alex

Excerpts from Alex's message of jue abr 19 17:06:05 -0300 2012:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>
> > On tor, 2012-04-19 at 00:13 +0300, Alex wrote:
> >> +#!/usr/bin/env perl
> >
> > Don't do that. Call the script using $(PERL) from the makefile.
>
> Thank you for the suggestion. Attached v2 does just this (while keeping
> a more commonly found shebang line in the perl script for running it w/o
> the makefile.)

I've applied this to 9.3. Andrew, can we have the non-MSVC buildfarm
members running on the master branch also run "make installcheck" in
src/interfaces/libpq? If there are platform dependencies here, it would
be good to know what they are; if we don't have the tests run
automatically we will never know.

We'll need to figure out some way for MSVC animals to run the tests as
well. Any takers?

I think this is necessary, because otherwise it is quite clear that the
tests are going to be kept failing forever. They were already failing
because of message style changes that didn't update the expected output.

(I am unsure about pushing the sh to perl test harness conversion in
9.2. If anybody thinks it should be done, please discuss.)

--
Álvaro Herrera <alvherre(at)commandprompt(dot)com>
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support