Use of zlib

Problem:
PL/Java use a JVM. On some platforms and with some JVM's (Sun's in
particular) a libzip.so is bundled that contains a 1.1.3 version of
functions also provided in zlib (why they do this is beyond me, but they
do so I'll have to live with it). PostgreSQL is linked with zlib by
default. This resuls in a conflict which resuls in a JVM crash.

I can circumvein this crash by using LD_PRELOAD to force a load of the
JVM bundled libzip.so but I suspect that might result in a crash as soon
as PostgreSQL calls on zlib to do something. It's of course possible to
configure postgresql with --without-zlib also provided you have accesst
o the source.

Question:
From what I can understand from the documentation, the only utility in
PostgreSQL that actually uses zlib is pg_dump? If so, why is the
postgres process linked with -lz?

Regards,

Thomas Hallgren

Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
> From what I can understand from the documentation, the only utility in
> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
> postgres process linked with -lz?

Because we are too lazy to fine-tune the build system for cases like this.
The best solution would be to build zlib with symbol versioning.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:

> Because we are too lazy to fine-tune the build system for cases like
> this.
>
>The best solution would be to build zlib with symbol versioning.
>
>
I'm not so sure. I think zlib is a commodity on most systems. You don't
want to build it at all. Perhaps if I submit a patch that takes care of
the fine-tuning?

- thomas

Peter Eisentraut wrote:

>Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>
>
>> From what I can understand from the documentation, the only utility in
>>PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>>postgres process linked with -lz?
>>
>>
>
>Because we are too lazy to fine-tune the build system for cases like this.
>The best solution would be to build zlib with symbol versioning.
>
>
>

Or to be less lazy ;-) Is it such a huge task? There have been
complaints before about our maximal linking.

cheers

andrew

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>> From what I can understand from the documentation, the only utility in
>> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>> postgres process linked with -lz?

> Because we are too lazy to fine-tune the build system for cases like this.

I was just reading about a linker option (--as-needed or something like
that) that exists in more recent versions of the GNU linker. It means
"only link the libraries actually referenced". Applying this or local
equivalent where available would seem like a fairly painless way to
solve the problem, at least on some platforms.

I agree that trying to keep track of the set of libraries really needed
for each executable would be a hopeless task, but if we can make the
linker fix it for us ...

regards, tom lane

Tom Lane wrote:

>Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>
>
>>Am Donnerstag, 23. September 2004 13:02 schrieb Thomas Hallgren:
>>
>>
>>>From what I can understand from the documentation, the only utility in
>>>PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>>>postgres process linked with -lz?
>>>
>>>
>
>
>
>>Because we are too lazy to fine-tune the build system for cases like this.
>>
>>
>
>I was just reading about a linker option (--as-needed or something like
>that) that exists in more recent versions of the GNU linker. It means
>"only link the libraries actually referenced". Applying this or local
>equivalent where available would seem like a fairly painless way to
>solve the problem, at least on some platforms.
>
>I agree that trying to keep track of the set of libraries really needed
>for each executable would be a hopeless task, but if we can make the
>linker fix it for us ...
>
>

Only in version 2.15, which as you say is fairly new. And people who
don't use GNU ld won't get it anyway. The docs (
http://sources.redhat.com/binutils/docs-2.15/ld/Options.html#Options ) say:

|--as-needed|

|--no-as-needed|
This option affects ELF DT_NEEDED tags for dynamic libraries
mentioned on the command line after the |--as-needed| option.
Normally, the linker will add a DT_NEEDED tag for each dynamic
library mentioned on the command line, regardless of whether the
library is actually needed. |--as-needed| causes DT_NEEDED tags to
only be emitted for libraries that satisfy some reference from
regular objects. |--no-as-needed| restores the default behaviour.

I'm not sure I understand why keeping track of what we need for each
executable is such a difficult task, though. I count 23 executables and
a handful of libraries. Is this such a herculean task?

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> I'm not sure I understand why keeping track of what we need for each
> executable is such a difficult task, though. I count 23 executables and
> a handful of libraries. Is this such a herculean task?

"Handful"? I count 32 AC_CHECK_LIB and AC_SEARCH_LIBS calls in
configure.in.

But that's only part of the problem. Consider that many of these
libraries are interdependent to some extent. The raw results from
Autoconf tell you that libfoo is available and we could link with it,
but not whether libfoo requires the previously-checked libbar to link.
We'd need to redo the whole library detection process for each
executable in order to be perfectly accurate about this. Since the
Autoconf macros only cope with one LIBS list, that means throwing away
the Autoconf macros and writing our own.

Multiply that by N supported platforms, and you start to see why this is
a mess. It's certainly work far out of proportion to the actual value
of not linking some unneeded libraries.

Adding "--as-needed" on platforms that support it seems like a
reasonable amount of work to expend in comparison to the value received.
Trying to do it manually does not.

regards, tom lane

Thomas Hallgren wrote:

> Problem:
> PL/Java use a JVM. On some platforms and with some JVM's (Sun's in
> particular) a libzip.so is bundled that contains a 1.1.3 version of
> functions also provided in zlib (why they do this is beyond me, but
> they do so I'll have to live with it). PostgreSQL is linked with zlib
> by default. This resuls in a conflict which resuls in a JVM crash.
>
> I can circumvein this crash by using LD_PRELOAD to force a load of the
> JVM bundled libzip.so but I suspect that might result in a crash as
> soon as PostgreSQL calls on zlib to do something. It's of course
> possible to configure postgresql with --without-zlib also provided you
> have accesst o the source.
>
> Question:
> From what I can understand from the documentation, the only utility in
> PostgreSQL that actually uses zlib is pg_dump? If so, why is the
> postgres process linked with -lz?
>
>

I did a small experiment by installing binutils 2.15 and adding
-Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.

This seemed to work quite well and trimmed back the needed libs quite a
bit. However, when you configure --with-openssl, libz is again linked in.

Not sure where that leaves us.

cheers

andrew

Andrew Dunstan wrote:

>
>
> Thomas Hallgren wrote:
>
>> Problem:
>> PL/Java use a JVM. On some platforms and with some JVM's (Sun's in
>> particular) a libzip.so is bundled that contains a 1.1.3 version of
>> functions also provided in zlib (why they do this is beyond me, but
>> they do so I'll have to live with it). PostgreSQL is linked with zlib
>> by default. This resuls in a conflict which resuls in a JVM crash.
>>
>> I can circumvein this crash by using LD_PRELOAD to force a load of
>> the JVM bundled libzip.so but I suspect that might result in a crash
>> as soon as PostgreSQL calls on zlib to do something. It's of course
>> possible to configure postgresql with --without-zlib also provided
>> you have accesst o the source.
>>
>> Question:
>> From what I can understand from the documentation, the only utility
>> in PostgreSQL that actually uses zlib is pg_dump? If so, why is the
>> postgres process linked with -lz?
>>
>>
>
> I did a small experiment by installing binutils 2.15 and adding
> -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.
>
> This seemed to work quite well and trimmed back the needed libs quite
> a bit. However, when you configure --with-openssl, libz is again
> linked in.

From PL/Javas standpoint, I think it would be great if this change
could be incorporated into the 8.0 release. The openssl issue is
something I'll have to investigate. Perhaps it works using the libzip
from the JRE if I use LD_PRELOAD. I'm happy to see that Sun has an open
bug on the subject in their Bug Database (bug 4953050 if anyone is
interested).

Regards,

Thomas Hallgren

I am confused. I just checked my backend binary and I don't see any
unusual libs required:

$ ldd postgres
libz.so => /usr/lib/libz.so (0x2833f000)
libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)
libdl.so => /shlib/libdl.so (0x2838c000)
libm.so => /shlib/libm.so.0.0 (0x2838f000)
libgcc.so.1 => /shlib/libgcc.so.1 (0x283a0000)
libc.so.2 => /shlib/libc.so.2 (0x283ac000)

The libncurses is a little unusual but I don't see libreadline in there
even though it is on the link line.

Are some OS's adding libs that aren't required by the binary?

---------------------------------------------------------------------------

Andrew Dunstan wrote:
>
>
> Thomas Hallgren wrote:
>
> > Problem:
> > PL/Java use a JVM. On some platforms and with some JVM's (Sun's in
> > particular) a libzip.so is bundled that contains a 1.1.3 version of
> > functions also provided in zlib (why they do this is beyond me, but
> > they do so I'll have to live with it). PostgreSQL is linked with zlib
> > by default. This resuls in a conflict which resuls in a JVM crash.
> >
> > I can circumvein this crash by using LD_PRELOAD to force a load of the
> > JVM bundled libzip.so but I suspect that might result in a crash as
> > soon as PostgreSQL calls on zlib to do something. It's of course
> > possible to configure postgresql with --without-zlib also provided you
> > have accesst o the source.
> >
> > Question:
> > From what I can understand from the documentation, the only utility in
> > PostgreSQL that actually uses zlib is pg_dump? If so, why is the
> > postgres process linked with -lz?
> >
> >
>
> I did a small experiment by installing binutils 2.15 and adding
> -Wl,--as-needed to the LDFLAGS, as Tom had suggested might be useful.
>
> This seemed to work quite well and trimmed back the needed libs quite a
> bit. However, when you configure --with-openssl, libz is again linked in.
>
> Not sure where that leaves us.
>
> cheers
>
> andrew
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
> (send "unregister YourEmailAddressHere" to majordomo(at)postgresql(dot)org)
>

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

On Sat, 25 Sep 2004, Bruce Momjian wrote:

> I am confused. I just checked my backend binary and I don't see any
> unusual libs required:
>
> $ ldd postgres
> libz.so => /usr/lib/libz.so (0x2833f000)
> libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)
> libdl.so => /shlib/libdl.so (0x2838c000)
> libm.so => /shlib/libm.so.0.0 (0x2838f000)
> libgcc.so.1 => /shlib/libgcc.so.1 (0x283a0000)
> libc.so.2 => /shlib/libc.so.2 (0x283ac000)
>
> The libncurses is a little unusual but I don't see libreadline in there
> even though it is on the link line.
>
> Are some OS's adding libs that aren't required by the binary?

This is the output in Fedora Core 2, and it includes readline:

$ ldd /usr/bin/postgres
linux-gate.so.1 => (0x00240000)
libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
libdl.so.2 => /lib/libdl.so.2 (0x00908000)
libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
/lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)

--
/Dennis Björklund

Bruce Momjian <pgman(at)candle(dot)pha(dot)pa(dot)us> writes:
> I am confused. I just checked my backend binary and I don't see any
> unusual libs required:

> $ ldd postgres
> libz.so => /usr/lib/libz.so (0x2833f000)
> libncurses.so.5 => /shlib/libncurses.so.5 (0x2834e000)

I do not believe either of those are actually *used* by the backend.

regards, tom lane

Dennis Bjorklund wrote:

>This is the output in Fedora Core 2, and it includes readline:
>
>$ ldd /usr/bin/postgres
> linux-gate.so.1 => (0x00240000)
> libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
> libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
> libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
> libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
> libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
> libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
> libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
> libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
> libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
> libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
> libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
> libdl.so.2 => /lib/libdl.so.2 (0x00908000)
> libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
> libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
> libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
> libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
> /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)
>
>
>
>

Now, to see the things postgres really needs, as opposed to things other
libraries need or that we just throw in for good measure in case
somebody needs them, run this:

LDFLAGS=-Wl,--as-necessary ./configure

without any third-party options (like pam, openssl, kerberos ...),
Alternatively, you can set the flag in src/Makefile.global

This should work on FC2, which I believe has binutils 2.15.

Almost every line above will disappear.

cheers

andrew

of course, I meant --as-needed rather than --as-necessary

Andrew Dunstan wrote:

>
>
> Dennis Bjorklund wrote:
>
>> This is the output in Fedora Core 2, and it includes readline:
>>
>> $ ldd /usr/bin/postgres
>> linux-gate.so.1 => (0x00240000)
>> libpam.so.0 => /lib/libpam.so.0 (0x00cf1000)
>> libssl.so.4 => /lib/libssl.so.4 (0x0014e000)
>> libcrypto.so.4 => /lib/libcrypto.so.4 (0x00241000)
>> libkrb5.so.3 => /usr/lib/libkrb5.so.3 (0x0052a000)
>> libcom_err.so.2 => /lib/libcom_err.so.2 (0x00e56000)
>> libz.so.1 => /usr/lib/libz.so.1 (0x00aca000)
>> libreadline.so.4 => /usr/lib/libreadline.so.4 (0x00111000)
>> libtermcap.so.2 => /lib/libtermcap.so.2 (0x00f25000)
>> libcrypt.so.1 => /lib/libcrypt.so.1 (0x00406000)
>> libresolv.so.2 => /lib/libresolv.so.2 (0x00f8a000)
>> libnsl.so.1 => /lib/libnsl.so.1 (0x00215000)
>> libdl.so.2 => /lib/libdl.so.2 (0x00908000)
>> libm.so.6 => /lib/tls/libm.so.6 (0x008a5000)
>> libc.so.6 => /lib/tls/libc.so.6 (0x00592000)
>> libgssapi_krb5.so.2 => /usr/lib/libgssapi_krb5.so.2 (0x00a57000)
>> libk5crypto.so.3 => /usr/lib/libk5crypto.so.3 (0x00ccd000)
>> /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x004ca000)
>>
>>
>>
>>
>
> Now, to see the things postgres really needs, as opposed to things
> other libraries need or that we just throw in for good measure in case
> somebody needs them, run this:
>
> LDFLAGS=-Wl,--as-necessary ./configure
>
> without any third-party options (like pam, openssl, kerberos ...),
> Alternatively, you can set the flag in src/Makefile.global
>
> This should work on FC2, which I believe has binutils 2.15.
>
> Almost every line above will disappear.
>
> cheers
>
> andrew
>