install-strip causes dyld errors on OS X

Hi all,

Three years ago, Andrew MacRae logged the following bug:

http://archives.postgresql.org/pgsql-bugs/2004-02/msg00042.php

In brief, when installing on OS X with "make install-strip",
installation goes fine, but initdb dies here:

creating conversions... ERROR: could not load library
"/usr/local/pgsql/lib/ascii_and_mic.so": dyld:
/usr/local/pgsql/bin/postgres
Undefined symbols:
/usr/local/pgsql/lib/ascii_and_mic.so undefined reference to
_pg_ascii2mic expected to be defined in the executable

I encountered this problem today with OS X 10.4 and CVS HEAD, and came
up with a few possible solutions. The problem has to do with the
difference between shared libraries and dynamic libraries, and how strip
behaves with them on OS X.

Here's how it works: When called on an executable which uses dyld, strip
silently passes itself the -u (save all undefined symbols) and -r (save
all dynamically-referenced symbols) options. However, the
internationalization libraries (and a few others, like libplperl and
libplpython) are compiled as shared libraries (.so as opposed to
.dylib), so strip doesn't know that those libraries contain undefined
symbols and are expecting to acquire them from the executable.

I see three possible fixes:

1) Patch config/install-sh such that on OS X, install-strip calls 'strip
-x'. This removes local symbols only.
2) Determine which symbols are global and undefined in the shared
libraries that are built, make a list of them, and call 'strip -u -r -s
symbols_to_keep'. This saves undefined symbols, dynamically referenced
symbols, and symbols that the shared libraries need.
3) Change the OS X build such that the i18n libs and other shared libs
are instead built as dylibs.

I've tested (1) as far as getting through initdb, starting the
postmaster and a few quick things with psql. It would be about a
four-line patch to install-sh (check for the OS, set stripcmd
appropriately). It might not leave the resulting binary *as* small as it
could be, but it certainly works.

(2) sounds like a giant pain in the ass. It's scriptable and could be
added to the build, but the build takes pretty long as it is and I can't
imagine anyone wants to make it longer. Unless there's a good reason,
I'm pre-emptively -1'ing this one.

I haven't done any looking into (3) yet. Why *are* the i18n libs built
as shared on OS X, anyway?

Cheers,
--mlp
_____________________
Meredith L. Patterson
Founder and CTO
Osogato, Inc.

"Meredith L. Patterson" <mlp(at)osogato(dot)com> writes:
> In brief, when installing on OS X with "make install-strip",
> installation goes fine, but initdb dies here:
> ...
> I see three possible fixes:
> 1) Patch config/install-sh such that on OS X, install-strip calls 'strip
> -x'. This removes local symbols only.

+1 on this one --- seems about the right level of effort, considering
that install-strip isn't exactly an encouraged option anyway. (Does
anyone still care about the amount of disk space involved? And it makes
debugging problems infinitely harder.)

Just for the record, I'd like to put forward the alternative solution
of removing support for install-strip. But assuming there are
objections to that, please send in the mentioned 4-line patch ...

regards, tom lane

Tom Lane wrote:

> Just for the record, I'd like to put forward the alternative solution
> of removing support for install-strip.

Seconded.

It would also be good to make --enable-debug the default.

--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

On Tue, 30 Oct 2007, Alvaro Herrera wrote:

> It would also be good to make --enable-debug the default.

The last time I did a build with both --enable-debug and --enable-cassert,
I was seeing pgbench results that were significantly (around 40% I think)
slower than without those two. I never circled back to assigning blame to
the individual options, but I'd suggest some caution here before turning
one of these on by default. I'd want to see benchmarking on a couple of
platforms to prove that it doesn't slow things down before making such a
change.

--
* Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore, MD

On Tue, 30 Oct 2007 13:00:12 -0400 (EDT)
Greg Smith <gsmith(at)gregsmith(dot)com> wrote:

> On Tue, 30 Oct 2007, Alvaro Herrera wrote:
>
> > It would also be good to make --enable-debug the default.
>
> The last time I did a build with both --enable-debug and
> --enable-cassert, I was seeing pgbench results that were
> significantly (around 40% I think) slower than without those two. I
> never circled back to assigning blame to the individual options, but
> I'd suggest some caution here before turning one of these on by
> default.

Assert certainly has a pretty big penalty but debug isn't nearly as bad.

Joshua D. Drake

I'd want to see benchmarking on a couple of platforms to
> prove that it doesn't slow things down before making such a change.
>
> --
> * Greg Smith gsmith(at)gregsmith(dot)com http://www.gregsmith.com Baltimore,
> MD
>
> ---------------------------(end of
> broadcast)--------------------------- TIP 4: Have you searched our
> list archives?
>
> http://archives.postgresql.org
>

--

=== The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997 http://www.commandprompt.com/
UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

Greg Smith wrote:
> On Tue, 30 Oct 2007, Alvaro Herrera wrote:
>
>> It would also be good to make --enable-debug the default.
>
> The last time I did a build with both --enable-debug and
> --enable-cassert, I was seeing pgbench results that were significantly
> (around 40% I think) slower than without those two. I never circled
> back to assigning blame to the individual options, but I'd suggest some
> caution here before turning one of these on by default. I'd want to see
> benchmarking on a couple of platforms to prove that it doesn't slow
> things down before making such a change.

--enable-cassert is the culprit. You don't want to enable that if you're
doing performance testing. I haven't seen a slowdown from
--enable-debug, though I haven't specifically tested that.

--
Heikki Linnakangas
EnterpriseDB http://www.enterprisedb.com

Greg Smith <gsmith(at)gregsmith(dot)com> writes:
> On Tue, 30 Oct 2007, Alvaro Herrera wrote:
>> It would also be good to make --enable-debug the default.

> The last time I did a build with both --enable-debug and --enable-cassert,
> I was seeing pgbench results that were significantly (around 40% I think)
> slower than without those two.

Yeah, cassert is expensive (mostly it's the CLOBBER_FREED_MEMORY part,
which you can disable separately if you have a mind to). --enable-debug
is supposed to be free when using gcc, though I've never tried to verify
that. We can *not* make --enable-debug the default on non-gcc
compilers, though, and that makes me uncomfortable about turning it on
by default for gcc ... platform-specific defaults don't sound good.

regards, tom lane

Tom Lane wrote:
> "Meredith L. Patterson" <mlp(at)osogato(dot)com> writes:
> > In brief, when installing on OS X with "make install-strip",
> > installation goes fine, but initdb dies here:
> > ...
> > I see three possible fixes:
> > 1) Patch config/install-sh such that on OS X, install-strip calls 'strip
> > -x'. This removes local symbols only.
>
> +1 on this one --- seems about the right level of effort, considering
> that install-strip isn't exactly an encouraged option anyway. (Does
> anyone still care about the amount of disk space involved? And it makes
> debugging problems infinitely harder.)
>
> Just for the record, I'd like to put forward the alternative solution
> of removing support for install-strip. But assuming there are
> objections to that, please send in the mentioned 4-line patch ...

I have developed the attached patch to implement this.

--
Bruce Momjian <bruce(at)momjian(dot)us> http://momjian.us
EnterpriseDB http://postgres.enterprisedb.com

+ If your life is a hard drive, Christ can be your backup. +