Re: TAP test breakage on MacOS X

On 10/27/2014 05:41 PM, Robert Haas wrote:
> On Sun, Oct 26, 2014 at 12:29 PM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>> The larger issue though is that even with both the above things fixed,
>> the TAP tests would still be an expensive no-op on the majority of
>> buildfarm members. AFAICT, I do not own a single machine on which the
>> current TAP tests will consent to run, and in most cases that's after
>> going out of my way to fetch CPAN modules that aren't in the vendor Perl
>> installs. Peter's mostly been fixing the portability issues by disabling
>> tests, which I guess is better than no fix at all, but it leaves darn
>> little useful functionality.
>
> I agree, emphatically. Honestly, if we can't get these tests running
> everywhere with reasonable effort, we should just rip them out. We've
> gone to a lot of trouble in general to make sure that our source code
> can be ported even to systems that arguably nobody uses any more, and
> instrumental to that effort is keeping the system requirements to
> install and test PostgreSQL minimal. At this point, I wouldn't mind
> moving the goalposts from C89 to C89 + a bunch of C99 features that
> are available on all the platforms we have buildfarm coverage for, and
> I wouldn't mind require perl to compile and install, full stop. But
> this patch has gone much further than that: you need a new-enough
> version of perl, and a new-enough version of a bunch of modules that
> aren't installed by default, and maybe not even in the vendor install,
> and the documentation on how to make it work is an embarrassment:
>
> http://www.postgresql.org/docs/devel/static/regress-tap.html
>
> Beyond all that, I have serious doubts about whether, even if we
> eventually get these tests mostly working in most places, whether they
> will actually catch any bugs.

The existing tests are not very useful, but it certainly would be nice
to have some infrastructure like TAP to write useful tests. For example,
I recently wrote a test suite for SSL, using TAP
(http://www.postgresql.org/message-id/53DF9AFE.4090706@vmware.com), and
it would be nice to have some TAP tests for hot standby, replication,
PITR etc.

I'm not sure if the current infrastructure is very good for that, but we
need something. If the current infrastructure is beyond repair, then
let's discuss what we could replace it with.
- Heikki