Re: PL/perl should fail on configure, not make

The way it is now (9.2.2):

1. set up a new system
2. forget to install libperl-dev
3. ./configure --with-perl
4. get no failures or warnings
5. make
6. make errors out on plperl.so

The way it should work:

- configure should error out, as it does with python.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> The way it is now (9.2.2):
> 1. set up a new system
> 2. forget to install libperl-dev
> 3. ./configure --with-perl
> 4. get no failures or warnings
> 5. make
> 6. make errors out on plperl.so

> The way it should work:

> - configure should error out, as it does with python.

Hm. I don't see any configure-time test for availability of perl.h,
which is probably what we ought to add here.

regards, tom lane

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> The way it is now (9.2.2):
> 1. set up a new system
> 2. forget to install libperl-dev
> 3. ./configure --with-perl
> 4. get no failures or warnings
> 5. make
> 6. make errors out on plperl.so

> The way it should work:

> - configure should error out, as it does with python.

I poked at this a bit more, and AFAICS your real beef is with Debian's
whacked-out packaging decisions for Perl.

Item: it's impossible to reproduce this failure on Red Hat-based
distros. perl-devel is required by perl-ExtUtils-MakeMaker, without
which we'll definitely fail at configure time. I doubt it's possible
in a hand-rolled install from source, either, because all that stuff
is part of the core perl tarball.

Item: there is *not* any test for libpython.so, as such, in our
configure script. There is a test for Python.h, which is sufficient on
Red Hat systems as well as on Debian, because python-devel
(resp. python-dev) carries both the header files and the .so links.

Item: there is not a test for perl.h, as such, in configure. There
probably should be, just because we have comparable tests for tcl.h
and Python.h. However, adding one won't fix your problem on
Debian-based distros, because for some wacko reason they put the
headers and the shlib .so symlink in different packages, cf
http://packages.debian.org/squeeze/amd64/perl/filelist
http://packages.debian.org/squeeze/amd64/libperl-dev/filelist
I am unfamiliar with a good reason for doing that. (I can certainly
see segregating the .a static library, or even not shipping it at
all, but what's it save to leave out the .so symlink?)

We could try adding an AC_TRY_LINK test using perl_embed_ldflags,
but given the weird stuff happening to redefine that value on Windows
in plperl/GNUmakefile I think there's a serious risk of breaking Cygwin
builds. Since I lack access to either Cygwin or a platform on which
there's a problem today, I'm not going to be the one to mess with it.

regards, tom lane

On Tue, 2013-01-08 at 22:37 -0500, Tom Lane wrote:
> We could try adding an AC_TRY_LINK test using perl_embed_ldflags,
> but given the weird stuff happening to redefine that value on Windows
> in plperl/GNUmakefile I think there's a serious risk of breaking
> Cygwin builds. Since I lack access to either Cygwin or a platform on
> which there's a problem today, I'm not going to be the one to mess
> with it.

For these and similar reasons, I think it's best not to mess with this
too much. It should fail where it actually fails. We shouldn't be
duplicating that logic in a remote place to predict that something will
fail later. In some cases, this is rather safe and easy and useful as a
service to the user, but once it gets slightly complicated, it can only
go wrong.

Tom,

> I poked at this a bit more, and AFAICS your real beef is with Debian's
> whacked-out packaging decisions for Perl.

Ah, I do so much Ubuntu these days I didn't recognize the pattern.

I'll try to figure out a good place to document a warning about it.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 01/08/2013 10:37 PM, Tom Lane wrote:

> We could try adding an AC_TRY_LINK test using perl_embed_ldflags,
> but given the weird stuff happening to redefine that value on Windows
> in plperl/GNUmakefile I think there's a serious risk of breaking Cygwin
> builds. Since I lack access to either Cygwin or a platform on which
> there's a problem today, I'm not going to be the one to mess with it.
>
>

ITYM Mingw - the Makefile doesn't do anything for Cygwin.

If you want to build a configure test, you could make it conditional on
the PORTNAME not being win32, since we don't seem to have a problem
there anyway.

cheers

andrew

Re: Tom Lane 2013-01-09 <9802(dot)1357702675(at)sss(dot)pgh(dot)pa(dot)us>
> Item: there is not a test for perl.h, as such, in configure. There
> probably should be, just because we have comparable tests for tcl.h
> and Python.h. However, adding one won't fix your problem on
> Debian-based distros, because for some wacko reason they put the
> headers and the shlib .so symlink in different packages, cf
> http://packages.debian.org/squeeze/amd64/perl/filelist
> http://packages.debian.org/squeeze/amd64/libperl-dev/filelist
> I am unfamiliar with a good reason for doing that. (I can certainly
> see segregating the .a static library, or even not shipping it at
> all, but what's it save to leave out the .so symlink?)

Because the .so symlink is only needed at build time. At runtime, you
need the .so.5.14 file. Hence .so.* -> $pkg, .h .a .so -> $pkg-dev.

Christoph
--
cb(at)df7cb(dot)de | http://www.df7cb.de/

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 01/08/2013 10:37 PM, Tom Lane wrote:
>> We could try adding an AC_TRY_LINK test using perl_embed_ldflags,
>> but given the weird stuff happening to redefine that value on Windows
>> in plperl/GNUmakefile I think there's a serious risk of breaking Cygwin
>> builds. Since I lack access to either Cygwin or a platform on which
>> there's a problem today, I'm not going to be the one to mess with it.

> ITYM Mingw - the Makefile doesn't do anything for Cygwin.

OK, sorry.

> If you want to build a configure test, you could make it conditional on
> the PORTNAME not being win32, since we don't seem to have a problem
> there anyway.

Actually, if we were to try to clean this up, I'd suggest moving that
logic into the configure script --- it's not apparent to me why it's
a good idea to be changing configure-determined values in the Makefile.
But in any case this would have to be done by somebody who's in a
position to test on affected platforms.

regards, tom lane

* Christoph Berg wrote:

> Re: Tom Lane 2013-01-09 <9802(dot)1357702675(at)sss(dot)pgh(dot)pa(dot)us>

>> and Python.h. However, adding one won't fix your problem on
>> Debian-based distros, because for some wacko reason they put the
>> headers and the shlib .so symlink in different packages, cf
>> http://packages.debian.org/squeeze/amd64/perl/filelist
>> http://packages.debian.org/squeeze/amd64/libperl-dev/filelist
>> I am unfamiliar with a good reason for doing that. (I can certainly
>> see segregating the .a static library, or even not shipping it at
>> all, but what's it save to leave out the .so symlink?)
>
> Because the .so symlink is only needed at build time. At runtime, you
> need the .so.5.14 file. Hence .so.* -> $pkg, .h .a .so -> $pkg-dev.

That was Tom's point, I believe -- Debian does not do it that way.

- perl-base has /usr/lib/libperl.so.5.etc
- perl has the headers and a dependency on perl-base
- libperl-dev has the symlink /usr/lib/libperl.so > libperl.so.5.etc

So by installing "perl", you get enough to satisfy configure, but there
is still no usable -lperl.

--
Christian

Christian Ullrich <chris(at)chrullrich(dot)net> writes:
> * Christoph Berg wrote:
>> Re: Tom Lane 2013-01-09 <9802(dot)1357702675(at)sss(dot)pgh(dot)pa(dot)us>
>>> I am unfamiliar with a good reason for doing that. (I can certainly
>>> see segregating the .a static library, or even not shipping it at
>>> all, but what's it save to leave out the .so symlink?)

>> Because the .so symlink is only needed at build time. At runtime, you
>> need the .so.5.14 file. Hence .so.* -> $pkg, .h .a .so -> $pkg-dev.

> That was Tom's point, I believe -- Debian does not do it that way.

> - perl-base has /usr/lib/libperl.so.5.etc
> - perl has the headers and a dependency on perl-base
> - libperl-dev has the symlink /usr/lib/libperl.so > libperl.so.5.etc

> So by installing "perl", you get enough to satisfy configure, but there
> is still no usable -lperl.

Right. [ dons red fedora for packaging purposes... ] The beef that
I've got with this is that there is no sane reason to do it like that.
If you are building C code against a library, you probably need both
some headers and the .so symlink, neither of which will be needed at
runtime; which is why both of those typically go into a foo-dev or
foo-devel subpackage. There are some other situations involving dynamic
loading where you might need the .so symlink at runtime, but in that
case you typically end up deciding that the symlink had better be in the
base package (Debian seems to have chosen this path for Python for
instance).

I'm not terribly familiar with building of Perl extensions, but it seems
possible that there's some use for Perl's C headers in that process,
which might explain why the headers are in "perl" and not a "perl-dev"
subpackage. But since the symlink requires no space to speak of, the
sensible thing to do with it would have been to include it in "perl"
along with the headers.

The libperl-dev package, as constituted, doesn't make any sense: it's
got the symlink which people need, and a very large static (.a) library
that most people don't need. Even worse, you can't tell without close
inspection which of those files is actually used by a package that
requires libperl-dev, and that is something that's important to know.
Under Red Hat packaging rules, the .a library likely wouldn't be shipped
at all; or if there was any demand for it, it would be all by its lonesome
in a package named something like "libperl-static", so that it'd be easy
to tell which packages actually use it. (Think about what happens if a
security update needs to be made in that library ... how do you tell
what has to be rebuilt? And in any case, discouraging use of the static
library will reduce the number of packages to be rebuilt.)

So IMO we're looking at some pretty broken packaging decisions here.
I doubt we'll get Debian to change it, so I don't mind if someone
wants to add a separate configure probe to verify libperl.so is
available --- but as I said upthread, it's not going to be me, because
I'm not in a position to test on the platforms that would be affected.

regards, tom lane

On 1/9/13 10:30 AM, Christian Ullrich wrote:
> * Christoph Berg wrote:
>
>> Re: Tom Lane 2013-01-09 <9802(dot)1357702675(at)sss(dot)pgh(dot)pa(dot)us>
>
>>> and Python.h. However, adding one won't fix your problem on
>>> Debian-based distros, because for some wacko reason they put the
>>> headers and the shlib .so symlink in different packages, cf
>>> http://packages.debian.org/squeeze/amd64/perl/filelist
>>> http://packages.debian.org/squeeze/amd64/libperl-dev/filelist
>>> I am unfamiliar with a good reason for doing that. (I can certainly
>>> see segregating the .a static library, or even not shipping it at
>>> all, but what's it save to leave out the .so symlink?)
>>
>> Because the .so symlink is only needed at build time. At runtime, you
>> need the .so.5.14 file. Hence .so.* -> $pkg, .h .a .so -> $pkg-dev.
>
> That was Tom's point, I believe -- Debian does not do it that way.
>
> - perl-base has /usr/lib/libperl.so.5.etc
> - perl has the headers and a dependency on perl-base
> - libperl-dev has the symlink /usr/lib/libperl.so > libperl.so.5.etc
>
> So by installing "perl", you get enough to satisfy configure, but there
> is still no usable -lperl.

The reason it's like that is that the header files are usable for
building Perl extensions, but that doesn't need the library. And the
expectation is that if you want to build against libfoo, you install
libfoo-dev. The fact that some other package also gives you half of
what you need is a coincidence.

The blame, if you want to assign any, is with configure, because it
assumes that the header files and the library are an atomic unit and
checks for the former and assumes the latter must be there as well.

On 1/9/13 11:00 AM, Tom Lane wrote:
> The libperl-dev package, as constituted, doesn't make any sense: it's
> got the symlink which people need, and a very large static (.a) library
> that most people don't need. Even worse, you can't tell without close
> inspection which of those files is actually used by a package that
> requires libperl-dev, and that is something that's important to know.

The expectation is that if you want to link against libfoo, you install
libfoo-dev, and after that you can uninstall it. What's wrong with that?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On 1/9/13 11:00 AM, Tom Lane wrote:
>> The libperl-dev package, as constituted, doesn't make any sense: it's
>> got the symlink which people need, and a very large static (.a) library
>> that most people don't need. Even worse, you can't tell without close
>> inspection which of those files is actually used by a package that
>> requires libperl-dev, and that is something that's important to know.

> The expectation is that if you want to link against libfoo, you install
> libfoo-dev, and after that you can uninstall it. What's wrong with that?

What's wrong is that it's hard to tell whether the resulting package
will contain a reference to the shared library (libperl.so.whatever)
or an embedded copy of the static library. As I tried to explain, this
is something that a well-run distro will want to be able to control,
or at least determine automatically from the package's BuildRequires
list (RPM-ism, not sure what Debian's package management stuff calls the
equivalent concept). That makes it a bad idea independently of the
problem of whether two configure tests are needed rather than one.

regards, tom lane

On 01/09/2013 10:16 AM, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> On 01/08/2013 10:37 PM, Tom Lane wrote:
>>> We could try adding an AC_TRY_LINK test using perl_embed_ldflags,
>>> but given the weird stuff happening to redefine that value on Windows
>>> in plperl/GNUmakefile I think there's a serious risk of breaking Cygwin
>>> builds. Since I lack access to either Cygwin or a platform on which
>>> there's a problem today, I'm not going to be the one to mess with it.
>> ITYM Mingw - the Makefile doesn't do anything for Cygwin.
> OK, sorry.
>
>> If you want to build a configure test, you could make it conditional on
>> the PORTNAME not being win32, since we don't seem to have a problem
>> there anyway.
> Actually, if we were to try to clean this up, I'd suggest moving that
> logic into the configure script --- it's not apparent to me why it's
> a good idea to be changing configure-determined values in the Makefile.
> But in any case this would have to be done by somebody who's in a
> position to test on affected platforms.

Here's a patch which does that and produces configure traces like this
on Mingw:

checking for Perl archlibexp... C:/Perl/lib
checking for Perl privlibexp... C:/Perl/lib
checking for Perl useshrplib... true
checking for flags to link embedded Perl... -LC:/Perl/lib/CORE -lperl512

which seems to be what we want.

Given that, you should be able to write a reasonably portable configure
test for library presence.

Barring objection I'll apply this shortly.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 01/09/2013 10:16 AM, Tom Lane wrote:
>> Actually, if we were to try to clean this up, I'd suggest moving that
>> logic into the configure script --- it's not apparent to me why it's
>> a good idea to be changing configure-determined values in the Makefile.
>> But in any case this would have to be done by somebody who's in a
>> position to test on affected platforms.

> Here's a patch which does that and produces configure traces like this
> on Mingw:

> checking for Perl archlibexp... C:/Perl/lib
> checking for Perl privlibexp... C:/Perl/lib
> checking for Perl useshrplib... true
> checking for flags to link embedded Perl... -LC:/Perl/lib/CORE -lperl512

> which seems to be what we want.

> Given that, you should be able to write a reasonably portable configure
> test for library presence.

Looks good. If you're happy with that then I can undertake to add a
libperl.so probe based on AC_TRY_LINK with the unmodified value of
$perl_embed_ldflags.

regards, tom lane

On 01/09/2013 04:12 PM, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> On 01/09/2013 10:16 AM, Tom Lane wrote:
>>> Actually, if we were to try to clean this up, I'd suggest moving that
>>> logic into the configure script --- it's not apparent to me why it's
>>> a good idea to be changing configure-determined values in the Makefile.
>>> But in any case this would have to be done by somebody who's in a
>>> position to test on affected platforms.
>> Here's a patch which does that and produces configure traces like this
>> on Mingw:
>> checking for Perl archlibexp... C:/Perl/lib
>> checking for Perl privlibexp... C:/Perl/lib
>> checking for Perl useshrplib... true
>> checking for flags to link embedded Perl... -LC:/Perl/lib/CORE -lperl512
>> which seems to be what we want.
>> Given that, you should be able to write a reasonably portable configure
>> test for library presence.
> Looks good. If you're happy with that then I can undertake to add a
> libperl.so probe based on AC_TRY_LINK with the unmodified value of
> $perl_embed_ldflags.
>

Go for it.

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 01/09/2013 04:12 PM, Tom Lane wrote:
>> Looks good. If you're happy with that then I can undertake to add a
>> libperl.so probe based on AC_TRY_LINK with the unmodified value of
>> $perl_embed_ldflags.

> Go for it.

Done, we'll soon see how much the buildfarm likes it. (I did test on
Fedora, OS X, and a from-source build on HPUX, though.)

While attempting to test this, I found out that my previous statement
that this failure is impossible on Red Hat platforms actually needs no
qualification. I had supposed that it would be possible to test the
failure path with sufficiently aggressive damage to the local perl
installation, but it's not. Observe:

$ which perl
/usr/bin/perl
$ ldd /usr/bin/perl
linux-vdso.so.1 => (0x00007fffe03ff000)
libperl.so => /usr/lib64/perl5/CORE/libperl.so (0x0000003c0d400000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003831600000)
libnsl.so.1 => /lib64/libnsl.so.1 (0x0000003843600000)
libdl.so.2 => /lib64/libdl.so.2 (0x000000382fe00000)
libm.so.6 => /lib64/libm.so.6 (0x000000382fa00000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x0000003c0d800000)
libutil.so.1 => /lib64/libutil.so.1 (0x0000003842a00000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x000000382f600000)
libc.so.6 => /lib64/libc.so.6 (0x000000382f200000)
/lib64/ld-linux-x86-64.so.2 (0x000000382ee00000)
libfreebl3.so => /lib64/libfreebl3.so (0x0000003c0e000000)
$ ls -l /usr/lib64/perl5/CORE/libperl*
-rwxr-xr-x. 1 root root 1552816 Nov 26 10:25 /usr/lib64/perl5/CORE/libperl.so

That is, the standard perl executable depends on libperl.so --- not
libperl.so.M.N, which isn't even there. Take the .so away, and you
don't get past configure's initial perl-related checks, because
/usr/bin/perl is broken. This isn't a Red-Hat-ism, either, because my
direct-from-upstream-source build on my old HPUX box is the same way.

I conclude that Debian has positively striven to be randomly different
from the rest of the Perl world, and so my sympathy for their packaging
design is even lower than it was this morning.

regards, tom lane

I wrote:
> Done, we'll soon see how much the buildfarm likes it.

Well, okapi (Gentoo) doesn't like it:

configure:29191: checking for libperl
configure:29217: icc -o conftest -O3 -xSSSE3 -parallel -ip -mp1 -fno-strict-aliasing -g -I/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/et conftest.c -Wl,-O1 -Wl,--as-needed -L/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt -lutil -lc -lxslt -lxml2 -lssl -lcrypto -lkrb5 -lz -lreadline -lcrypt -lm >&5
ld: conftest: hidden symbol `pthread_atfork' in /usr/lib64/libpthread_nonshared.a(pthread_atfork.oS) is referenced by DSO
ld: final link failed: Bad value

The previous successful build on that box shows that plperl was linked
like this without any error:

icc -O3 -xSSSE3 -parallel -ip -mp1 -fno-strict-aliasing -g -fpic -shared -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -Wl,-rpath,'/usr/lib64/perl5/5.12.4/x86_64-linux/CORE',--enable-new-dtags -Wl,-O1 -Wl,--as-needed -L/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt -lutil -lc

The best guess I can come up with is that this box is only able to link
libperl.so into a shared library, ie the "-fpic -shared" part of the
recipe is critical. If so, this idea is a failure and we're gonna have
to remove the link test, because there is no way I'm going to try to put
enough smarts into the configure script to do a shared-library link
correctly.

However, I don't know Gentoo at all, and it may be there's some other
explanation that we could cope with more readily. Thoughts?

regards, tom lane

On 01/09/2013 10:48 PM, Tom Lane wrote:
> I wrote:
>> Done, we'll soon see how much the buildfarm likes it.
> Well, okapi (Gentoo) doesn't like it:
>
> configure:29191: checking for libperl
> configure:29217: icc -o conftest -O3 -xSSSE3 -parallel -ip -mp1 -fno-strict-aliasing -g -I/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -D_GNU_SOURCE -I/usr/include/libxml2 -I/usr/include/et conftest.c -Wl,-O1 -Wl,--as-needed -L/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt -lutil -lc -lxslt -lxml2 -lssl -lcrypto -lkrb5 -lz -lreadline -lcrypt -lm >&5
> ld: conftest: hidden symbol `pthread_atfork' in /usr/lib64/libpthread_nonshared.a(pthread_atfork.oS) is referenced by DSO
> ld: final link failed: Bad value
>
> The previous successful build on that box shows that plperl was linked
> like this without any error:
>
> icc -O3 -xSSSE3 -parallel -ip -mp1 -fno-strict-aliasing -g -fpic -shared -o plperl.so plperl.o SPI.o Util.o -L../../../src/port -Wl,-rpath,'/usr/lib64/perl5/5.12.4/x86_64-linux/CORE',--enable-new-dtags -Wl,-O1 -Wl,--as-needed -L/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt -lutil -lc
>
> The best guess I can come up with is that this box is only able to link
> libperl.so into a shared library, ie the "-fpic -shared" part of the
> recipe is critical. If so, this idea is a failure and we're gonna have
> to remove the link test, because there is no way I'm going to try to put
> enough smarts into the configure script to do a shared-library link
> correctly.

This seems fairly unlikely. You're supposed to be able to use it in a
standalone executable - see the perlembed manpage. But I guess with
Gentoo anything is possible.

>
> However, I don't know Gentoo at all, and it may be there's some other
> explanation that we could cope with more readily. Thoughts?

Maybe we need a test that mirrors our use of PERL_SYS_INIT3?, something
like:

#include <EXTERN.h> /* from the Perl distribution */
#include <perl.h> /* from the Perl distribution */

static PerlInterpreter *my_perl; /*** The Perl interpreter ***/

int main(int argc, char **argv, char **env)
{
#if defined(PERL_SYS_INIT3) && !defined(MYMALLOC)
PERL_SYS_INIT3(&argc,&argv,&env);
#endif
my_perl = perl_alloc();
}

Or maybe for a standalone executable we need to use the ccopts as
suggested by the perlembed manpage:

cc -o interp interp.c `perl -MExtUtils::Embed -e ccopts -e ldopts`

cheers

andrew

Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> On 01/09/2013 10:48 PM, Tom Lane wrote:
>> Well, okapi (Gentoo) doesn't like it:
>> ...
>> The best guess I can come up with is that this box is only able to link
>> libperl.so into a shared library, ie the "-fpic -shared" part of the
>> recipe is critical.

> This seems fairly unlikely. You're supposed to be able to use it in a
> standalone executable - see the perlembed manpage. But I guess with
> Gentoo anything is possible.

It seemed pretty silly to me too. After posting it occurred to me that
maybe the problem wasn't from libperl but one of the other LIBS that
configure dredges up, but my fix along that line hasn't helped. So
I don't know what to think, other than that Gentoo is fragile as can be.

> Maybe we need a test that mirrors our use of PERL_SYS_INIT3?

I don't know what PERL_SYS_INIT3 expands to, but it's hard to see how it
would expand to something that would fix this. The problem looks to be
that something is pulling in libpthread and getting the wrong version.

> Or maybe for a standalone executable we need to use the ccopts as
> suggested by the perlembed manpage:
> cc -o interp interp.c `perl -MExtUtils::Embed -e ccopts -e ldopts`

Hm. It would be interesting to see what ccopts expands to, but if you
compare the command configure issued with the previous successful link
of plperl, there's nothing missing that ccopts would be likely to
supply.

regards, tom lane

On 01/10/2013 11:32 AM, Tom Lane wrote:
> Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
>> On 01/09/2013 10:48 PM, Tom Lane wrote:
>>> Well, okapi (Gentoo) doesn't like it:
>>> ...
>>> The best guess I can come up with is that this box is only able to link
>>> libperl.so into a shared library, ie the "-fpic -shared" part of the
>>> recipe is critical.
>> This seems fairly unlikely. You're supposed to be able to use it in a
>> standalone executable - see the perlembed manpage. But I guess with
>> Gentoo anything is possible.
> It seemed pretty silly to me too. After posting it occurred to me that
> maybe the problem wasn't from libperl but one of the other LIBS that
> configure dredges up, but my fix along that line hasn't helped. So
> I don't know what to think, other than that Gentoo is fragile as can be.
>
>> Maybe we need a test that mirrors our use of PERL_SYS_INIT3?
> I don't know what PERL_SYS_INIT3 expands to, but it's hard to see how it
> would expand to something that would fix this. The problem looks to be
> that something is pulling in libpthread and getting the wrong version.
>
>> Or maybe for a standalone executable we need to use the ccopts as
>> suggested by the perlembed manpage:
>> cc -o interp interp.c `perl -MExtUtils::Embed -e ccopts -e ldopts`
> Hm. It would be interesting to see what ccopts expands to, but if you
> compare the command configure issued with the previous successful link
> of plperl, there's nothing missing that ccopts would be likely to
> supply.
>
>

Without access to the machine it's pretty hard to know, so I was just
speculating fairly wildly. If Jeremy can find out what the problem is
that would be good, or if he can give us access to the machine it
shouldn't be terribly hard to get to the bottom.

cheers

andrew

On 1/9/13 12:50 PM, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> On 1/9/13 11:00 AM, Tom Lane wrote:
>>> The libperl-dev package, as constituted, doesn't make any sense: it's
>>> got the symlink which people need, and a very large static (.a) library
>>> that most people don't need. Even worse, you can't tell without close
>>> inspection which of those files is actually used by a package that
>>> requires libperl-dev, and that is something that's important to know.
>
>> The expectation is that if you want to link against libfoo, you install
>> libfoo-dev, and after that you can uninstall it. What's wrong with that?
>
> What's wrong is that it's hard to tell whether the resulting package
> will contain a reference to the shared library (libperl.so.whatever)
> or an embedded copy of the static library. As I tried to explain, this
> is something that a well-run distro will want to be able to control,
> or at least determine automatically from the package's BuildRequires
> list (RPM-ism, not sure what Debian's package management stuff calls the
> equivalent concept). That makes it a bad idea independently of the
> problem of whether two configure tests are needed rather than one.

Yeah, this has nothing to do with how Perl is packaged. The issue of
linking against static vs shared libraries is well understood, but a
separate debate. There are other packaging tools that make sure this
works out. Even if there were no static libraries, libperl-dev might
still exist. The core issue here is that header files and link-time
library files are in different packages that can be installed
separately. Any other library were this choice was made would create
the same issue. The fault is that configure checks only one thing and
assumes two.

On 1/9/13 4:06 PM, Andrew Dunstan wrote:
> Here's a patch which does that and produces configure traces like this
> on Mingw:
>
> checking for Perl archlibexp... C:/Perl/lib
> checking for Perl privlibexp... C:/Perl/lib
> checking for Perl useshrplib... true
> checking for flags to link embedded Perl... -LC:/Perl/lib/CORE -lperl512
>
> which seems to be what we want.

It would also be good to fix the same issue in plpython.

On 1/9/13 7:56 PM, Tom Lane wrote:
> That is, the standard perl executable depends on libperl.so --- not
> libperl.so.M.N, which isn't even there. Take the .so away, and you
> don't get past configure's initial perl-related checks, because
> /usr/bin/perl is broken. This isn't a Red-Hat-ism, either, because my
> direct-from-upstream-source build on my old HPUX box is the same way.

How does this survive a distribution upgrade with a new libperl soname?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> ... The core issue here is that header files and link-time
> library files are in different packages that can be installed
> separately. Any other library were this choice was made would create
> the same issue. The fault is that configure checks only one thing and
> assumes two.

It's complete folly to imagine that configure checks all and exactly the
things that we require to build. In fact, I'd argue that a majority of
what it does is inference or approximation of some sort. For an example
we need look no further than the AC_CHECK_FUNC logic, which is not
actually testing that a function-like symbol is available to be called.
If autoconf were trying to be exact, its scripts would be many times
larger and slower, but not many times more useful. Similarly, there is
no very good reason for our configure code to spend cycles testing for
cases that we don't expect to see in the field.

In the case at hand, up to now we have been checking that you have
ExtUtils::MakeMaker and assuming that if you do, you have perl headers
and libperl.so. The first of these inferences seems valid enough to me,
as the presence of MakeMaker suggests that you have facilities for
building Perl extensions. As for the second, it's pretty common to
suppose that headers and .so symlinks are packaged together.

I continue to assert that Debian's choices here are less than sane.

But in any case, the link test for libperl.so is going to go away again
unless someone can explain how to make it work on Gentoo.

regards, tom lane

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On 1/9/13 7:56 PM, Tom Lane wrote:
>> That is, the standard perl executable depends on libperl.so --- not
>> libperl.so.M.N, which isn't even there. Take the .so away, and you
>> don't get past configure's initial perl-related checks, because
>> /usr/bin/perl is broken. This isn't a Red-Hat-ism, either, because my
>> direct-from-upstream-source build on my old HPUX box is the same way.

> How does this survive a distribution upgrade with a new libperl soname?

Well, I'm not the package maintainer for perl, so this is not an
authoritative answer ... but I don't believe that there's any
expectation that you could replace the installation with a different
major perl version and still have C-level dependencies work. Adding a
soname number wouldn't exactly be enough to fix that type of problem,
anyhow, considering the number of ABI-incompatible ways you can build
libperl.

There is some version dependency checking, all right, but it's done off
of special RPM provides/requires symbols not the library soname. For
instance I see this on a Fedora build of plperl:

$ rpm -qp postgresql-plperl-9.2.2-3.fc16.x86_64.rpm --requires
libc.so.6()(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit)
libc.so.6(GLIBC_2.3.4)(64bit)
libc.so.6(GLIBC_2.4)(64bit)
libperl.so()(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
perl(:MODULE_COMPAT_5.14.3) <-----------------
postgresql-server(x86-64) = 9.2.2-3.fc16
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rtld(GNU_HASH)

I suspect they chose that method so it would do something useful for
Perl extension modules as well as programs that require libperl.

regards, tom lane

On 1/10/13 4:48 PM, Tom Lane wrote:
> Well, I'm not the package maintainer for perl, so this is not an
> authoritative answer ... but I don't believe that there's any
> expectation that you could replace the installation with a different
> major perl version and still have C-level dependencies work.

Well, Debian does support that. It is necessary to be able to upgrade
to the next distribution release and have old packages linked against an
older libperl keep working. Apparently, Fedora doesn't support that.

On 1/10/13 4:38 PM, Tom Lane wrote:
> It's complete folly to imagine that configure checks all and exactly the
> things that we require to build. In fact, I'd argue that a majority of
> what it does is inference or approximation of some sort.

I'm not saying they have to be exact. But you have to be prepared to
make them more exact when your approximations don't work out in practice.

On 1/10/13 4:38 PM, Tom Lane wrote:
> The first of these inferences seems valid enough to me,
> as the presence of MakeMaker suggests that you have facilities for
> building Perl extensions.

MakeMaker is perfectly useful without any C headers or C compiler nearby.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On 1/10/13 4:48 PM, Tom Lane wrote:
>> Well, I'm not the package maintainer for perl, so this is not an
>> authoritative answer ... but I don't believe that there's any
>> expectation that you could replace the installation with a different
>> major perl version and still have C-level dependencies work.

> Well, Debian does support that. It is necessary to be able to upgrade
> to the next distribution release and have old packages linked against an
> older libperl keep working. Apparently, Fedora doesn't support that.

If they wanted to install two incompatible versions at once, they'd just
stick them in different directory trees (ie, not both /usr/lib64/perl5).
Given the amount of stuff in a Perl distribution besides libperl.so
itself, I should think some such thing would be necessary regardless.

(Or, if you prefer, the thing that is binding /usr/bin/perl to its
correct library is an RPATH setting, not a soname version number.)

regards, tom lane

All,

OK, now I'm sorry I brought this up.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

Josh Berkus <josh(at)agliodbs(dot)com> writes:
> OK, now I'm sorry I brought this up.

Sorry about going off on a theological tangent. The important question
at this point is, can anybody fix the test so it works on Gentoo?
If not we'll have to pull it out, whether or not it would be a good
thing to have on Debian.

regards, tom lane

On Thu, Jan 10, 2013 at 08:12:56PM -0500, Tom Lane wrote:
> Josh Berkus <josh(at)agliodbs(dot)com> writes:
> > OK, now I'm sorry I brought this up.
>
> Sorry about going off on a theological tangent. The important question
> at this point is, can anybody fix the test so it works on Gentoo?
> If not we'll have to pull it out, whether or not it would be a good
> thing to have on Debian.
>
> regards, tom lane

I can give it a go.

At any rate, I'll have to fix the problem before I
can release the package to the Portage tree, even if that means I remove
the test.

--
Mr. Aaron W. Swenson
Gentoo Linux Developer
Email : titanofold(at)gentoo(dot)org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

"Aaron W. Swenson" <titanofold(at)gentoo(dot)org> writes:
> On Thu, Jan 10, 2013 at 08:12:56PM -0500, Tom Lane wrote:
>> Sorry about going off on a theological tangent. The important question
>> at this point is, can anybody fix the test so it works on Gentoo?
>> If not we'll have to pull it out, whether or not it would be a good
>> thing to have on Debian.

> I can give it a go.

Great, thanks!

One thing we can see now that we have a full buildfarm cycle with that
patch is that two out of the three Gentoo machines in the farm are
perfectly happy with it --- only okapi doesn't like it. So that raises
a new set of questions. It seems somewhat likely now that this is a
local misconfiguration problem on okapi, but unless we can diagnose it
I still feel that we'll have to drop the configure probe.

Just a moment ago I pushed another change to make configure place the
-I$perl_archlibexp/CORE switch at the end of CPPFLAGS not the beginning,
because that's how plperl's makefile does it. I have not got much hope
that the problem is an include search order issue, but that's the last
thing I can think to try. I don't see any other plausibly-significant
differences between what configure is doing and what the makefile is
doing.

regards, tom lane

On Thu, Jan 10, 2013 at 10:25:25PM -0500, Tom Lane wrote:
> "Aaron W. Swenson" <titanofold(at)gentoo(dot)org> writes:
> > On Thu, Jan 10, 2013 at 08:12:56PM -0500, Tom Lane wrote:
> >> Sorry about going off on a theological tangent. The important question
> >> at this point is, can anybody fix the test so it works on Gentoo?
> >> If not we'll have to pull it out, whether or not it would be a good
> >> thing to have on Debian.
>
> > I can give it a go.
>
> Great, thanks!
>
> One thing we can see now that we have a full buildfarm cycle with that
> patch is that two out of the three Gentoo machines in the farm are
> perfectly happy with it --- only okapi doesn't like it. So that raises
> a new set of questions. It seems somewhat likely now that this is a
> local misconfiguration problem on okapi, but unless we can diagnose it
> I still feel that we'll have to drop the configure probe.
>
> Just a moment ago I pushed another change to make configure place the
> -I$perl_archlibexp/CORE switch at the end of CPPFLAGS not the beginning,
> because that's how plperl's makefile does it. I have not got much hope
> that the problem is an include search order issue, but that's the last
> thing I can think to try. I don't see any other plausibly-significant
> differences between what configure is doing and what the makefile is
> doing.
>
> regards, tom lane

Just checked out that change and it works for me. It finds libperl.

--
Mr. Aaron W. Swenson
Gentoo Linux Developer
Email : titanofold(at)gentoo(dot)org
GnuPG FP : 2C00 7719 4F85 FB07 A49C 0E31 5713 AA03 D1BB FDA0
GnuPG ID : D1BBFDA0

On Thu, 10 Jan 2013, Andrew Dunstan wrote:

>
> On 01/10/2013 11:32 AM, Tom Lane wrote:
> > Andrew Dunstan <andrew(at)dunslane(dot)net> writes:
> > > cc -o interp interp.c `perl -MExtUtils::Embed -e ccopts -e ldopts`
> > Hm. It would be interesting to see what ccopts expands to, but if you
> > compare the command configure issued with the previous successful link
> > of plperl, there's nothing missing that ccopts would be likely to
> > supply.

$ perl -MExtUtils::Embed -e ccopts -e ldopts
-Wl,-E -Wl,-O1 -Wl,--as-needed
-L/usr/lib64/perl5/5.12.4/x86_64-linux/CORE -lperl -lnsl -ldl -lm -lcrypt
-lutil -lc
-fno-strict-aliasing -pipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
-I/usr/lib64/perl5/5.12.4/x86_64-linux/CORE

>
> Without access to the machine it's pretty hard to know, so I was just
> speculating fairly wildly. If Jeremy can find out what the problem is that
> would be good, or if he can give us access to the machine it shouldn't be
> terribly hard to get to the bottom.

I'll see what I can do. For now, I can tell you that it works with GCC
and fails with ICC.

pgbuildfarm(at)jdrake(dot)com writes:
> On Thu, 10 Jan 2013, Andrew Dunstan wrote:
>> Without access to the machine it's pretty hard to know, so I was just
>> speculating fairly wildly. If Jeremy can find out what the problem is that
>> would be good, or if he can give us access to the machine it shouldn't be
>> terribly hard to get to the bottom.

> I'll see what I can do. For now, I can tell you that it works with GCC
> and fails with ICC.

Well, that's darn interesting in itself, because the error message looks
like it should be purely a linker issue. (And I note that your other
buildfarm animal mongoose uses icc but is working anyway, so that's
definitely not the whole story ...)

Please note Aaron Swenson's offer of help too -- he's probably a lot
better qualified than anybody else here to figure out what is going on
with this.

regards, tom lane

On Fri, 11 Jan 2013, Tom Lane wrote:

> pgbuildfarm(at)jdrake(dot)com writes:
> Well, that's darn interesting in itself, because the error message looks
> like it should be purely a linker issue. (And I note that your other
> buildfarm animal mongoose uses icc but is working anyway, so that's
> definitely not the whole story ...)

mongoose is 32-bit, and a really old version of icc. okapi is 64-bit, and
a merely moderately old icc. I should set up a dedicated buildfarm
VM with the latest version...

>
> Please note Aaron Swenson's offer of help too -- he's probably a lot
> better qualified than anybody else here to figure out what is going on
> with this.

I'm sorry, I didn't see this. It must not have been CC'd to me, I don't
subscribe to -hackers anymore, I just couldn't keep up with the traffic
after I got a new job that wasn't postgres-related.

pgbuildfarm(at)jdrake(dot)com writes:
> On Thu, 10 Jan 2013, Andrew Dunstan wrote:
>> Without access to the machine it's pretty hard to know, so I was just
>> speculating fairly wildly. If Jeremy can find out what the problem is that
>> would be good, or if he can give us access to the machine it shouldn't be
>> terribly hard to get to the bottom.

> I'll see what I can do. For now, I can tell you that it works with GCC
> and fails with ICC.

BTW, just to close out this thread on the public list --- Andrew found
out that the problem goes away if one removes "-parallel" from the list
of compiler options. That makes it a pretty blatant compiler bug, and
one that we've not seen elsewhere, so it's probably specific to the
(old) icc version that Jeremy is running on this buildfarm animal.

Given that no other buildfarm members are showing a problem, I don't
intend to revert the configure change. Jeremy should update his icc
or remove the -parallel switch on that animal.

regards, tom lane