Review: tests for client programs

Hello

This patch creates a infrastructure for client side testing environment

1. Surely we want this patch, this infrastructure - lot of important
functionality is not covered by tests - pg_dump, pg_basebackup, ... Without
infrastructure is terrible work to create some tests.

2. This patch does exactly what was proposed

3. New functionality has zero impact on our server side or client side
software (performance, quality, readability, ..)

4. I was able to use this patch cleanly and it works

5. I didn't test it on windows

6. I found only few issues:

a) Configure doesn't check a required IRC::Run module

b) Prepared tests fails when PostgreSQL server was up - should be checked
and should to raise a valuable error message

c) I am not sure if infrastructure is enough - for test pg_dump I miss a
comparation of produced file with some other expected file. This objection
is not blocker - someone can enhance it (when will write tests of pg_dump
for example).

Regards

Pavel Stehule

2014-01-29 Pavel Stehule <pavel(dot)stehule(at)gmail(dot)com>:

> Hello
>
> I am looking on this patch.
>
> It is great idea, and I am sure, so we want this patch - it was requested
> and proposed more time.
>
> Some tips:
>
> a) possibility to test only selected tests
> b) possibility to verify generated file against expected file
> c) detection some warnings (expected/unexpected)
>
> p.s. some tests fails when other Postgres is up. It should be checked on
> start.and raise warning or stop testing.
>
> Regards
>
> Pavel
>
>
> ok 4 - pg_ctl initdb
> waiting for server to start....LOG: could not bind IPv6 socket: Address
> already in use
> HINT: Is another postmaster already running on port 5432? If not, wait a
> few seconds and retry.
> LOG: could not bind IPv4 socket: Address already in use
> HINT: Is another postmaster already running on port 5432? If not, wait a
> few seconds and retry.
> WARNING: could not create listen socket for "localhost"
> FATAL: could not create any TCP/IP sockets
> .... stopped waiting
> pg_ctl: could not start server
> Examine the log output.
> not ok 5 - pg_ctl start -w
>
> # Failed test 'pg_ctl start -w'
> # at
> /home/pavel/src/postgresql/src/bin/pg_ctl/../../../src/test/perl/TestLib.pm
> line 89.
> waiting for server to start....LOG: could not bind IPv6 socket: Address
> already in use
> HINT: Is another postmaster already running on port 5432? If not, wait a
> few seconds and retry.
> LOG: could not bind IPv4 socket: Address already in use
> HINT: Is another postmaster already running on port 5432? If not, wait a
> few seconds and retry.
> WARNING: could not create listen socket for "localhost"
> FATAL: could not create any TCP/IP sockets
> .... stopped waiting
> pg_ctl: could not start server
> Examine the log output.
> not ok 6 - second pg_ctl start succeeds
>
> # Failed test 'second pg_ctl start succeeds'
> # at
> /home/pavel/src/postgresql/src/bin/pg_ctl/../../../src/test/perl/TestLib.pm
> line 89.
> pg_ctl: PID file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.pid"
> does not exist
> Is server running?
> not ok 7 - pg_ctl stop -w
>
> # Failed test 'pg_ctl stop -w'
> # at
> /home/pavel/src/postgresql/src/bin/pg_ctl/../../../src/test/perl/TestLib.pm
> line 89.
> ok 8 - second pg_ctl stop fails
> pg_ctl: PID file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.pid"
> does not exist
> Is server running?
> starting server anyway
> pg_ctl: could not read file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.opts"
> not ok 9 - pg_ctl restart with server not running
>
> # Failed test 'pg_ctl restart with server not running'
> # at
> /home/pavel/src/postgresql/src/bin/pg_ctl/../../../src/test/perl/TestLib.pm
> line 89.
> pg_ctl: PID file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.pid"
> does not exist
> Is server running?
> starting server anyway
> pg_ctl: could not read file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.opts"
> not ok 10 - pg_ctl restart with server running
>
> # Failed test 'pg_ctl restart with server running'
> # at
> /home/pavel/src/postgresql/src/bin/pg_ctl/../../../src/test/perl/TestLib.pm
> line 89.
> pg_ctl: PID file
> "/home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data/postmaster.pid"
> does not exist
> Is server running?
> Bailout called. Further testing stopped: system pg_ctl stop -D
> /home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data -m fast failed: 256
> Bail out! system pg_ctl stop -D
> /home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data -m fast failed: 256
> FAILED--Further testing stopped: system pg_ctl stop -D
> /home/pavel/src/postgresql/src/bin/pg_ctl/testf2KW/data -m fast failed: 256
> make[1]: *** [check] Error 255
> make[1]: Leaving directory `/home/pavel/src/postgresql/src/bin/pg_ctl'
> make: *** [check-pg_ctl-recurse] Error 2
> make: Leaving directory `/home/pavel/src/postgresql/src/bin'
>
>