Re: pg_upgrade difficulties

I am encountering multiple issues in my attempt to upgrade from
PostgreSQL 9.0.4 to 9.1.0 on CentOS 5.6 i386.

The latest working version is 9.0.4 installed from the
http://yum.pgrpms.org/ site. After installing the 9.1 repo RPM and
installing 9.1 via yum I set up a script to do pg_upgrade with the
--check option. It is failing at the step when it tries to start the new
version. The last 4 lines of the upgrade log are (trimming the full path
from the log file name):

"/usr/pgsql-9.1/bin/pg_ctl" -w -l "pg_upgrade.log" -D
"/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start >> "upgrade.log" 2>&1

waiting for server to start....../usr/pgsql-9.1/bin/pg_ctl: symbol
lookup error: /usr/pgsql-9.1/bin/pg_ctl: undefined symbol: PQping

There were problems executing "/usr/pgsql-9.1/bin/pg_ctl" -w -l
"upgrade.log" -D "/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start >>
"upgrade.log" 2>&1

pg_ctl failed to start the new server

Note, however, that the server does start. To check if things were
really running I tried to connect but psql returns an error of:
psql: invalid connection option "client_encoding"

I checked which version of psql is actually being called
readlink -f $(which psql)
/usr/pgsql-9.1/bin/psql

So the psql is, indeed, version 9.1. I have read that this error may be
related to psql loading an older library.

Note, there is also a minor bug in the package - the start-script error
message advising to run initdb includes the minor version:

[root(at)foo ~]# service postgresql-9.1 start

/var/lib/pgsql/9.1/data is missing. Use "service postgresql-9.1.0
initdb" to initialize the cluster first.
[FAILED]
[root(at)foo ~]# service postgresql-9.1.0 initdb
postgresql-9.1.0: unrecognized service
[root(at)foo ~]# service postgresql-9.1 initdb
Initializing database:

Any thoughts?

Cheers,
Steve

Steve Crawford <scrawford(at)pinpointresearch(dot)com> writes:
> waiting for server to start....../usr/pgsql-9.1/bin/pg_ctl: symbol
> lookup error: /usr/pgsql-9.1/bin/pg_ctl: undefined symbol: PQping

> There were problems executing "/usr/pgsql-9.1/bin/pg_ctl" -w -l
> "upgrade.log" -D "/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start >>
> "upgrade.log" 2>&1

This looks like your pg_ctl executable is mistakenly getting linked to a
pre-9.1 version of libpq.so. Don't know enough about the file layout in
the current PGDG RPMs to speculate as to why.

regards, tom lane

On 09/13/2011 05:31 PM, Tom Lane wrote:
> Steve Crawford<scrawford(at)pinpointresearch(dot)com> writes:
>> waiting for server to start....../usr/pgsql-9.1/bin/pg_ctl: symbol
>> lookup error: /usr/pgsql-9.1/bin/pg_ctl: undefined symbol: PQping
>> There were problems executing "/usr/pgsql-9.1/bin/pg_ctl" -w -l
>> "upgrade.log" -D "/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start>>
>> "upgrade.log" 2>&1
> This looks like your pg_ctl executable is mistakenly getting linked to a
> pre-9.1 version of libpq.so. Don't know enough about the file layout in
> the current PGDG RPMs to speculate as to why.
>
> regards, tom lane
>
Devrim, any thoughts?

Cheers,
Steve

On 09/14/2011 08:48 AM, Steve Crawford wrote:
> On 09/13/2011 05:31 PM, Tom Lane wrote:
>> Steve Crawford<scrawford(at)pinpointresearch(dot)com> writes:
>>> waiting for server to start....../usr/pgsql-9.1/bin/pg_ctl: symbol
>>> lookup error: /usr/pgsql-9.1/bin/pg_ctl: undefined symbol: PQping
>>> There were problems executing "/usr/pgsql-9.1/bin/pg_ctl" -w -l
>>> "upgrade.log" -D "/var/lib/pgsql/9.1/data" -o "-p 5432 -b" start>>
>>> "upgrade.log" 2>&1
>> This looks like your pg_ctl executable is mistakenly getting linked to a
>> pre-9.1 version of libpq.so. Don't know enough about the file layout in
>> the current PGDG RPMs to speculate as to why.
>>
>> regards, tom lane
>>
> Devrim, any thoughts?
>
> Cheers,
> Steve
>

OK, it's a packaging/library issue that seems likely to bite anyone
trying to do a parallel install of 9.0 and 9.1 from PGDG CentOS repo.
Don't know if if affects any other packages. Basically, everything seems
to load the old libraries, not the new ones.

Here are the installed packages:
postgresql90.i386 9.0.4-2PGDG.rhel5 installed
postgresql90-contrib.i386 9.0.4-2PGDG.rhel5 installed
postgresql90-devel.i386 9.0.4-2PGDG.rhel5 installed
postgresql90-libs.i386 9.0.4-2PGDG.rhel5 installed
postgresql90-server.i386 9.0.4-2PGDG.rhel5 installed
postgresql91.i386 9.1.0-2PGDG.rhel5 installed
postgresql91-contrib.i386 9.1.0-2PGDG.rhel5 installed
postgresql91-devel.i386 9.1.0-2PGDG.rhel5 installed
postgresql91-libs.i386 9.1.0-2PGDG.rhel5 installed
postgresql91-server.i386 9.1.0-2PGDG.rhel5 installed

When pg_upgrade and psql do not work correctly:

root:~ ldconfig -p | grep pq
libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so
libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so

Here's how the libraries get listed for ldconfig:

root:ld.so.conf.d cd /etc/ld.so.conf.d
root:ld.so.conf.d ls -l post*
-rwx------ 1 root root 20 Sep 8 02:06 postgresql-9.0-libs.conf
-rwx------ 1 root root 20 Sep 12 10:57 postgresql-9.1-libs.conf

But if I rename postgresql-9.0-libs.conf so it sorts later and rerun
ldconfig I see:

root:ld.so.conf.d mv postgresql-9.0-libs.conf postgresql-9.old-libs.conf

root:ld.so.conf.d ls -l post*
-rwx------ 1 root root 20 Sep 12 10:57 postgresql-9.1-libs.conf
-rwx------ 1 root root 20 Sep 8 02:06 postgresql-9.old-libs.conf

root:ld.so.conf.d ldconfig

root:ld.so.conf.d ldconfig -p | grep pq
libpqwalreceiver.so (libc6) => /usr/pgsql-9.1/lib/libpqwalreceiver.so
libpqwalreceiver.so (libc6) => /usr/pgsql-9.0/lib/libpqwalreceiver.so
libpq.so.5 (libc6) => /usr/pgsql-9.1/lib/libpq.so.5
libpq.so.5 (libc6) => /usr/pgsql-9.0/lib/libpq.so.5
libpq.so (libc6) => /usr/pgsql-9.1/lib/libpq.so
libpq.so (libc6) => /usr/pgsql-9.0/lib/libpq.so

Now pg_upgrade works fine, I can start the new cluster and psql works as
expected. I can rename the 9.0 conf file back to its original name and
use yum to remove 9.0 and everything seems to work. I suspect I could
run 9.0 and 9.1 in parallel after fixing the library issue but did not try.

Cheers,
Steve