Re: Supporting plpython 2+3 builds better

I wrote:
> After reading the recent thread about python 2 vs python 3 support,
> I thought I'd amuse myself by trying to get plpython3 supported in
> the Fedora packages. That turned out to be unreasonably painful
> (which is something we oughta fix eventually), but it worked,

To give you an idea of what "unreasonably painful" means, attached is
the specfile diff needed to make this happen. I will not comment on
the fragility of this beyond observing that the "touch -r" commands
are *necessary*, else you get incorrect results.

I think we really need to do something to make this type of build less
painful, because as far as I can see most distros are going to be
wanting to support both python 2 and 3 for awhile to come.

A sketch of a solution might go like this:

* Extend configure to have two switches, say "--with-python" and
"--with-python3", which you can select either or both of. If you
want to pick executables that are not named "python" and "python3",
perhaps write it like "--with-python3=/usr/bin/python3.2mu". (I'd
be inclined to remove the dependency on a PYTHON envvar altogether.)

* Make configure output two independent sets of variables into
Makefile.global, viz

python2_includespec = ...
python2_libdir = ...
python2_libspec = ...
python2_additional_libs = ...
python2_configdir = ...
python2_majorversion = ...
python2_version = ...

python3_includespec = ...
python3_libdir = ...
python3_libspec = ...
python3_additional_libs = ...
python3_configdir = ...
python3_majorversion = ...
python3_version = ...

* Make plpython's Makefile build, test, install plpython2 and/or
plpython3 depending on what's set in Makefile.global.

I'm not sure yet whether it's possible to do this without duplicating a
lot of logic in config/python.m4 and plpython's Makefile.

Another problem is that Makefile.shlib isn't designed to build more than
one shared library per directory, which means that we might end up
having to copy all the source files into a new plpython3 subdirectory
anyway. We're already doing some of that with the regression test
files, though.

Thoughts?

regards, tom lane

On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
> To give you an idea of what "unreasonably painful" means, attached is
> the specfile diff needed to make this happen. I will not comment on
> the fragility of this beyond observing that the "touch -r" commands
> are *necessary*, else you get incorrect results.

I think an easier way is to configure two vpath builds and install the
pieces you want from each one. yo

> Another problem is that Makefile.shlib isn't designed to build more
> than one shared library per directory,

That's the main problem, but fixing it would be very useful in other
places as well. I had it on my radar to do something about that.

> which means that we might end up having to copy all the source files
> into a new plpython3 subdirectory anyway. We're already doing some of
> that with the regression test files, though.

Doing it with the test files is already annoying enough. For instance,
we don't have a static list of all the files we need to copy (because of
expected file variants that are not tracked in the makefiles), so we
need to copy the files again on each run. If you extend this to all
source files, things would get totally bizarre.

A less invasive method might be to set up a second directory
pl/plpython3 which vpath-builds from pl/plpython.

But frankly, I'd like to work on the multiple-shlibs-in-one-directory
issue, because all of the above issues and more are projected in their
own strange ways on something like the transforms feature. For example,
how would you organize the source code for an extension module that
provides a new data type and adapters for plpython{2,3}u and plperl{,u},
with tests and all that?

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
>> To give you an idea of what "unreasonably painful" means, attached is
>> the specfile diff needed to make this happen. I will not comment on
>> the fragility of this beyond observing that the "touch -r" commands
>> are *necessary*, else you get incorrect results.

> I think an easier way is to configure two vpath builds and install the
> pieces you want from each one.

I thought about that, and didn't like it, because then you have no proof
that the plpython3 you built in the one tree will actually play with the
core code you built in the other tree. Messy as my solution is, the
regression test step does actually test the intended live combination of
executables.

>> Another problem is that Makefile.shlib isn't designed to build more
>> than one shared library per directory,

> That's the main problem, but fixing it would be very useful in other
> places as well. I had it on my radar to do something about that.

This would be a good thing. Got any ideas how to do it?

regards, tom lane

On Sun, 2012-09-09 at 03:35 -0400, Tom Lane wrote:
> >> Another problem is that Makefile.shlib isn't designed to build more
> >> than one shared library per directory,
>
> > That's the main problem, but fixing it would be very useful in other
> > places as well. I had it on my radar to do something about that.
>
> This would be a good thing. Got any ideas how to do it?

Here is a very rough patch. It obviously will need a lot of
fine-tuning, but it's the idea.

Excerpts from Peter Eisentraut's message of lun sep 10 09:50:42 -0300 2012:
> On Sun, 2012-09-09 at 03:35 -0400, Tom Lane wrote:
> > >> Another problem is that Makefile.shlib isn't designed to build more
> > >> than one shared library per directory,
> >
> > > That's the main problem, but fixing it would be very useful in other
> > > places as well. I had it on my radar to do something about that.
> >
> > This would be a good thing. Got any ideas how to do it?
>
> Here is a very rough patch. It obviously will need a lot of
> fine-tuning, but it's the idea.

I remember trying to do this for the mb/conversion_procs subdir years
ago, to make them build in parallel to save some time. It didn't go
anywhere but the basic idea seems similar in spirit. Maybe we can use
this there too to make it fast.

--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On 9/10/12 9:26 AM, Alvaro Herrera wrote:
> I remember trying to do this for the mb/conversion_procs subdir years
> ago, to make them build in parallel to save some time. It didn't go
> anywhere but the basic idea seems similar in spirit. Maybe we can use
> this there too to make it fast.

Parallel builds across subdirectories should work now, so that shouldn't
be necessary anymore. Although removing some directory depth might be
nice, but there is a general hesitation about renaming files.