pg9 beta1, make check fails

$make check
... snip lots of output...
./pg_regress --inputdir=. --dlpath=. --multibyte=SQL_ASCII
--temp-install=./tmp_check --top-builddir=../../..
--schedule=./parallel_schedule
============== removing existing temp installation ==============
============== creating temporary installation ==============
============== initializing database system ==============
============== starting postmaster ==============

pg_regress: postmaster did not respond within 60 seconds
Examine
/home/andy/gis/postgresql-9.0beta1/src/test/regress/log/postmaster.log
for the reason
make[2]: *** [check] Error 2
make[2]: Leaving directory
`/home/andy/gis/postgresql-9.0beta1/src/test/regress'
make[1]: *** [check] Error 2
make[1]: Leaving directory `/home/andy/gis/postgresql-9.0beta1/src/test'
make: *** [check] Error 2

$cat /home/andy/gis/postgresql-9.0beta1/src/test/regress/log/postmaster.log
LOG: database system was shut down at 2010-05-03 12:39:58 CDT
LOG: database system is ready to accept connections
LOG: autovacuum launcher started

Andy Colson <andy(at)squeakycode(dot)net> writes:
> $make check
> ... snip lots of output...
> ./pg_regress --inputdir=. --dlpath=. --multibyte=SQL_ASCII
> --temp-install=./tmp_check --top-builddir=../../..
> --schedule=./parallel_schedule
> ============== removing existing temp installation ==============
> ============== creating temporary installation ==============
> ============== initializing database system ==============
> ============== starting postmaster ==============

> pg_regress: postmaster did not respond within 60 seconds

Hmm, apparently some sort of communication problem, since the postmaster
seems to think everything's fine. Maybe a firewall blocking packets
to the port that got selected? Another possibility is that you have
PGPORT or related variables set in your shell environment, and that's
leading libpq to try to connect to the wrong place. (I thought we had
pg_regress try to clear those, but maybe it doesn't.) What platform
is this anyway?

regards, tom lane

On 5/3/2010 12:59 PM, Tom Lane wrote:
> Andy Colson<andy(at)squeakycode(dot)net> writes:
>> $make check
>> ... snip lots of output...
>> ./pg_regress --inputdir=. --dlpath=. --multibyte=SQL_ASCII
>> --temp-install=./tmp_check --top-builddir=../../..
>> --schedule=./parallel_schedule
>> ============== removing existing temp installation ==============
>> ============== creating temporary installation ==============
>> ============== initializing database system ==============
>> ============== starting postmaster ==============
>
>> pg_regress: postmaster did not respond within 60 seconds
>
> Hmm, apparently some sort of communication problem, since the postmaster
> seems to think everything's fine. Maybe a firewall blocking packets
> to the port that got selected? Another possibility is that you have
> PGPORT or related variables set in your shell environment, and that's
> leading libpq to try to connect to the wrong place. (I thought we had
> pg_regress try to clear those, but maybe it doesn't.) What platform
> is this anyway?
>
> regards, tom lane

Its an internal only box (no outside connections), so no iptables.

Box is running Slackware 64:

# uname -a
Linux mapper 2.6.32.7 #1 SMP Fri Jan 29 21:04:54 CST 2010 x86_64 AMD
Athlon(tm) II X2 245 Processor AuthenticAMD GNU/Linux

I dont have PGPORT env set, or any env starting with PG* (not sure what
other env var's to check for).

I do have a .pgpass in ~, but no entries for this box, only to remote box.

no other .pg* files.

I did notice when I tried to backup my 8.4 db using the 9.0 pg_dump I
got errors:

$ ~/gis/postgresql-9.0beta1/src/bin/pg_dump/pg_dump -Fc -f gis.bak gis
/home/andy/gis/postgresql-9.0beta1/src/bin/pg_dump/pg_dump: symbol
lookup error:
/home/andy/gis/postgresql-9.0beta1/src/bin/pg_dump/pg_dump: undefined
symbol: PQconnectdbParams

Was not sure if that was a problem (maybe 'make check' was finding 8.4
.so's or something...).

-Andy

On 5/3/2010 12:59 PM, Tom Lane wrote:
> Andy Colson<andy(at)squeakycode(dot)net> writes:
>> $make check
>> ... snip lots of output...
>> ./pg_regress --inputdir=. --dlpath=. --multibyte=SQL_ASCII
>> --temp-install=./tmp_check --top-builddir=../../..
>> --schedule=./parallel_schedule
>> ============== removing existing temp installation ==============
>> ============== creating temporary installation ==============
>> ============== initializing database system ==============
>> ============== starting postmaster ==============
>
>> pg_regress: postmaster did not respond within 60 seconds

Ok, not sure what order its supposed to be run in, but I had done:

make
make check

which fails.

But if I do:

make
make install
make check

it works and passes.

-Andy

Andy Colson <andy(at)squeakycode(dot)net> writes:
> Ok, not sure what order its supposed to be run in, but I had done:

> make
> make check

> which fails.

> But if I do:

> make
> make install
> make check

> it works and passes.

That looks like a shared-library problem --- did the "make install"
overwrite an older version of libpq.so?

I'd have thought that pg_regress would have a more obvious failure if it
was trying to use an old libpq.so version though --- it should have
looked similar to what you referenced for pg_dump, for instance.

regards, tom lane

On 5/3/2010 1:21 PM, Tom Lane wrote:
> Andy Colson<andy(at)squeakycode(dot)net> writes:
>> Ok, not sure what order its supposed to be run in, but I had done:
>
>> make
>> make check
>
>> which fails.
>
>> But if I do:
>
>> make
>> make install
>> make check
>
>> it works and passes.
>
> That looks like a shared-library problem --- did the "make install"
> overwrite an older version of libpq.so?
>
> I'd have thought that pg_regress would have a more obvious failure if it
> was trying to use an old libpq.so version though --- it should have
> looked similar to what you referenced for pg_dump, for instance.
>
> regards, tom lane

I'm guessing so.. I'd renamed my install dir from pgsql (which is in the
path) to pg84, then done a 'make install'.

As a test I renamed my pgsql (which contains 9) to pg9 (so its not in
the path) then did a make check and it worked ok.

Then I renamed pg84 to pgsql (so 8.4 is in the path) and 'make check'
fails again.

-Andy

I wrote:
> I'd have thought that pg_regress would have a more obvious failure if it
> was trying to use an old libpq.so version though --- it should have
> looked similar to what you referenced for pg_dump, for instance.

I think I see what's going on here. pg_regress itself doesn't link to
libpq at all. psql uses some of the new functions that were added to
libpq in 9.0, so psql is guaranteed to fail if the linker tries to link
it to an 8.4 version of libpq, which is probably what was happening
before you did "make install". The reason that this led to the observed
behavior is that pg_regress does this to see if the postmaster is
running yet:

if (system("psql ... 2>/dev/null") == 0)
// postmaster is ready
else
// keep waiting

So psql failed, spewed something to stderr that went right into the bit
bucket, and pg_regress just saw that as an expected failure and kept
waiting.

I'm not immediately seeing a simple way to improve this. It'd be nice
if we didn't hide "unexpected" errors like the link failure with libpq.
On the other hand we surely don't want to show the expected connection
failures until the postmaster is up. Maybe psql should have a "really
quiet" mode that doesn't print anything even on errors, and then we
wouldn't have to route its stderr to /dev/null? But given how seldom
this sort of thing comes up (I don't recall any similar previous reports,
actually) maybe it's not worth the trouble.

regards, tom lane

On 5/3/2010 9:44 PM, Tom Lane wrote:
> I wrote:
>> I'd have thought that pg_regress would have a more obvious failure if it
>> was trying to use an old libpq.so version though --- it should have
>> looked similar to what you referenced for pg_dump, for instance.
>
> I think I see what's going on here. pg_regress itself doesn't link to
> libpq at all. psql uses some of the new functions that were added to
> libpq in 9.0, so psql is guaranteed to fail if the linker tries to link
> it to an 8.4 version of libpq, which is probably what was happening
> before you did "make install". The reason that this led to the observed
> behavior is that pg_regress does this to see if the postmaster is
> running yet:
>
> if (system("psql ... 2>/dev/null") == 0)
> // postmaster is ready
> else
> // keep waiting
>
> So psql failed, spewed something to stderr that went right into the bit
> bucket, and pg_regress just saw that as an expected failure and kept
> waiting.
>
> I'm not immediately seeing a simple way to improve this. It'd be nice
> if we didn't hide "unexpected" errors like the link failure with libpq.
> On the other hand we surely don't want to show the expected connection
> failures until the postmaster is up. Maybe psql should have a "really
> quiet" mode that doesn't print anything even on errors, and then we
> wouldn't have to route its stderr to /dev/null? But given how seldom
> this sort of thing comes up (I don't recall any similar previous reports,
> actually) maybe it's not worth the trouble.
>
> regards, tom lane
>

How about building a statically linked psql in 'make check', just for
pg_regress to use?

-Andy

Andy Colson <andy(at)squeakycode(dot)net> writes:
> On 5/3/2010 9:44 PM, Tom Lane wrote:
>> I'm not immediately seeing a simple way to improve this.

> How about building a statically linked psql in 'make check', just for
> pg_regress to use?

[ shrug... ] That sort of defeats the purpose of testing the binaries
we are intending to install.

Actually I guess the real question here is why psql failed to link to
the newly-built shared libraries. pg_regress sets up LD_LIBRARY_PATH
but that seems not to have done the trick for you. What platform are
you on exactly?

regards, tom lane

On 5/4/2010 9:37 AM, Tom Lane wrote:
> Andy Colson<andy(at)squeakycode(dot)net> writes:
>> On 5/3/2010 9:44 PM, Tom Lane wrote:
>>> I'm not immediately seeing a simple way to improve this.
>
>> How about building a statically linked psql in 'make check', just for
>> pg_regress to use?
>
> [ shrug... ] That sort of defeats the purpose of testing the binaries
> we are intending to install.
>
> Actually I guess the real question here is why psql failed to link to
> the newly-built shared libraries. pg_regress sets up LD_LIBRARY_PATH
> but that seems not to have done the trick for you. What platform are
> you on exactly?
>
> regards, tom lane

Slackware 64:

# uname -a
Linux mapper 2.6.32.7 #1 SMP Fri Jan 29 21:04:54 CST 2010 x86_64 AMD
Athlon(tm) II X2 245 Processor AuthenticAMD GNU/Linux

I do have /usr/local/pgsql/lib in my /etc/ld.so.conf. Not sure how
ldconfig and LD_LIBRARY_PATH interact.

> [ shrug... ] That sort of defeats the purpose of testing the binaries
> we are intending to install.

True, yea, but, I wonder how many times, it has used the previous
versions .so's? In which case you are testing the new server with the
old client. And never really knew. I'v tried out several versions
since I started with 8.0, and I'll bet I do make; make check; more often
than not. How many of those times picked up the prior .so's? I'd bet
quite a few.

-Andy