State of PL/Python build

The PL/Python build is now up to speed to as much as an extend as is under
our control. I can also confirm that it works (as defined by the test
suite) with Python 1.5 and 2.1. I've also put a chapter with
documentation in the programmer's guide.

We already know about the libpython (not-)shared library issue. As it
turns out, not only is libpython generally not a shared library, there
isn't even a designed in way to make one. See this discussion:

http://www.python.org/cgi-bin/faqw.py?req=show&file=faq03.030.htp

The way it's currently set up is that it will use whatever linking with
-lpythonx.y gives you. Using the static library this way works on many
platforms, although it's not as efficient (each backend process image will
have its own copy of the library). On some platforms it won't work at
all, of course.

In the distance you can hear the package maintainers rejoicing...

The second issue is that of dynamic loading. The way it currently looks
on platforms with the dlopen() interface (which is what most platforms
have) is that when your Python code imports a module which is implemented
in a C shared library, the Python interpreter fires up its own dynamic
loading facilities. However, due to the rules of relocation scoping,
those shared libraries cannot use the symbols defined in the plpython
image and therefore the libpython symbols themselves are not available.
Without those, no code works.

The way this is currently resolved is that all the C shared libraries that
form part of the modules that you might want to use are linked into
plpython.so directly, which puts them into a different relocation group,
so they have access to the symbols of libpython. This is a stupid hack,
of course.

The real fix is to change the dynamic loader(s) to make use of the
RTLD_GLOBAL flag, which makes all dlopen'ed symbols available to everyone.
Personally, I don't see any harm in using this option. Does anyone else?
(In fact, if someone were to create an untrusted PL/Perl language he would
probably run into this problem as well.)

Sounds a bit complicated? See this documentation:

http://docs.sun.com:80/ab2/coll.40.5/REFMAN3/@Ab2PageView/325438?DwebQuery=dlopen&oqt=dlopen&Ab2Lang=C&Ab2Enc=iso-8859-1

Resolving this issue on non-dlopen platforms is left as an exercise.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://funkturm.homeip.net/~peter

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> We already know about the libpython (not-)shared library issue. As it
> turns out, not only is libpython generally not a shared library, there
> isn't even a designed in way to make one.

Ugh. Can we get the Python boys to raise their level of concern about
that?

> The way it's currently set up is that it will use whatever linking with
> -lpythonx.y gives you. Using the static library this way works on many
> platforms, although it's not as efficient (each backend process image will
> have its own copy of the library).

Huh? As I understand it, what will happen is that libpython.a will get
physically incorporated into plpython.so (.sl, whatever). So all
Postgres backends that are using plpython should share one copy of that
library. It won't be shared with non-Postgres python processes, but
it's not nearly as bad as "one copy per backend".

The real problem is that on systems where non-PIC code can't be used to
build a shared library, the whole thing will not work at all. As with
plperl, it'd be nice if we could detect this at configure time.

I wonder whether people would like an option to statically link
libperl.a and/or libpython.a into the Postgres backend proper? That
would allow plperl/plpython to be used on platforms where this is an
issue, without having to make a nonstandard build of perl/python.

> The real fix is to change the dynamic loader(s) to make use of the
> RTLD_GLOBAL flag, which makes all dlopen'ed symbols available to everyone.
> Personally, I don't see any harm in using this option. Does anyone else?

No, on the platforms where it solves the problem...

> Resolving this issue on non-dlopen platforms is left as an exercise.

As near as I can tell from the man page, HPUX's shl_load behaves this
way all the time. Don't know about other non-dlopen platforms.

regards, tom lane

On Saturday 12 May 2001 14:46, Tom Lane wrote:
>
> I wonder whether people would like an option to statically link
> libperl.a and/or libpython.a into the Postgres backend proper? That
> would allow plperl/plpython to be used on platforms where this is an
> issue, without having to make a nonstandard build of perl/python.

I would certainly like this. Currently, I have to maintain a special perl
install that exists for no reason but to give me a shared lib version I can
link against.

--
Mark Hollomon

> The real fix is to change the dynamic loader(s) to make use of the
> RTLD_GLOBAL flag, which makes all dlopen'ed symbols available to everyone.
> Personally, I don't see any harm in using this option. Does anyone else?
> (In fact, if someone were to create an untrusted PL/Perl language he would
> probably run into this problem as well.)

I have RTLD_GLOBAL on BSD/OS, and it calls it a Linux-compatibility
flag. I am pretty amazed you got it working without RTLD_GLOBAL.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 853-3000
+ If your life is a hard drive, | 830 Blythe Avenue
+ Christ can be your backup. | Drexel Hill, Pennsylvania 19026

On Sat, May 12, 2001 at 02:46:45PM -0400, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > We already know about the libpython (not-)shared library issue. As it
> > turns out, not only is libpython generally not a shared library, there
> > isn't even a designed in way to make one.
>
> Ugh. Can we get the Python boys to raise their level of concern about
> that?

I looked through Python's configure script, and it appears that the
default is to build as a shared library.

> The real problem is that on systems where non-PIC code can't be used to
> build a shared library, the whole thing will not work at all. As with
> plperl, it'd be nice if we could detect this at configure time.

Python versions greater than 1.5.2 ship with the distutils module, it
can be used to detect if python was built with a shared library. Is
there anyway to detect whether non-PIC code can be used in a shared
library?

Andrew

--

Tom Lane writes:

> I wonder whether people would like an option to statically link
> libperl.a and/or libpython.a into the Postgres backend proper? That
> would allow plperl/plpython to be used on platforms where this is an
> issue, without having to make a nonstandard build of perl/python.

Not unless you also link in plperl/plpython itself or mess with
-whole-archive type linker flags.

--
Peter Eisentraut peter_e(at)gmx(dot)net http://funkturm.homeip.net/~peter

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> Tom Lane writes:
>> I wonder whether people would like an option to statically link
>> libperl.a and/or libpython.a into the Postgres backend proper? That
>> would allow plperl/plpython to be used on platforms where this is an
>> issue, without having to make a nonstandard build of perl/python.

> Not unless you also link in plperl/plpython itself or mess with
> -whole-archive type linker flags.

The former is what I had in mind.

Yes, it's ugly and it bloats the binary, but people would presumably
only do this if they intended to use the language. So the bloat is
somewhat illusory. And it's less ugly than having to build a
nonstandard install of python or perl.

I could even see people doing this on platforms where they didn't have
to (because a non-PIC libpython or libperl could be included into a
shared library plpython or plperl). It should give a performance
advantage, which might be interesting to heavy users of those PLs.

regards, tom lane

On Tuesday 15 May 2001 13:09, Tom Lane wrote:
> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > Tom Lane writes:
> >> I wonder whether people would like an option to statically link
> >> libperl.a and/or libpython.a into the Postgres backend proper? That
> >> would allow plperl/plpython to be used on platforms where this is an
> >> issue, without having to make a nonstandard build of perl/python.
> >
> > Not unless you also link in plperl/plpython itself or mess with
> > -whole-archive type linker flags.
>
> The former is what I had in mind.
>
> Yes, it's ugly and it bloats the binary, but people would presumably
> only do this if they intended to use the language. So the bloat is
> somewhat illusory. And it's less ugly than having to build a
> nonstandard install of python or perl.
>

I'm not sure it is any uglier than including the (to me) useless geometry
datatypes.

I would be happy to help get this working with plperl.

Another interesting idea is to allow the postmaster to 'preload' some of the
libraries. This would help cut down the startup cost of a new backend that
needed a widely used language.

--
Mark Hollomon