PATCH: Compiling PostgreSQL using ActiveState Python 3.2

Hi,

I am trying to build PostgreSQL 9.1beta3 using the ActiveState Python 3.2.
It did not compile successfully.

When I tried to figure out the exact reason for the failure, I found that:
1. 'python_configdir' variable is hardcoded, instead it should use the
configuration 'LIBPL'.
2. 'plpython' is trying get linked using '-lpython${*python_version*}', but
it should be '-lpython${*python_ldversion*}'.

Please find the attached patch, which resolve the issue on my side.

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com>

*http://www.linkedin.com/in/asheshvashi*

On ons, 2011-08-17 at 18:28 +0530, Ashesh Vashi wrote:
> I am trying to build PostgreSQL 9.1beta3 using the ActiveState Python 3.2.
> It did not compile successfully.

Note that building against Python 3.2 works at least on Debian, so this
is not a universal problem. It appears to have to do with the stable
ABI thing they introduced in Python 3.2, so it will be mainly relevant
to platforms targeted by that.

> When I tried to figure out the exact reason for the failure, I found that:
> 1. 'python_configdir' variable is hardcoded, instead it should use the
> configuration 'LIBPL'.

That looks reasonable. My Debian installation works around this by a
symlink, but that's perhaps a hack they put in for this reason.

> 2. 'plpython' is trying get linked using '-lpython${*python_version*}', but
> it should be '-lpython${*python_ldversion*}'.

That, on the other hand, will be a problem.
get_config_vars('LDVERSION') isn't defined before Python 3.2, so this
will break all previous versions.

I find it a bit curious that this is necessary, because the previous
coding works for me:

$ python3.2 -c "import distutils.sysconfig,string; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LDLIBRARY'))))"
libpython3.2mu.so

$ python3.2 -c "import distutils.sysconfig,string; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LDVERSION'))))"
3.2mu

So it is not in fact true that we are linking against '-lpython
${*python_version*}'.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On ons, 2011-08-17 at 18:28 +0530, Ashesh Vashi wrote:
>> When I tried to figure out the exact reason for the failure, I found that:
>> 1. 'python_configdir' variable is hardcoded, instead it should use the
>> configuration 'LIBPL'.

> That looks reasonable. My Debian installation works around this by a
> symlink, but that's perhaps a hack they put in for this reason.

FWIW, all three python installations I have handy (2.7 on Fedora 14, 2.7
on OS X Lion, 2.5 on HPUX) produce the same result from either of

python -c "from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))"
python -c "import distutils.sysconfig,string; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LIBPL'))))"

It's not immediately apparent to me why we should think that
get_python_lib is less trustworthy than LIBPL; but if someone
can make that case, I don't have any objection to this part of
the patch.

>> 2. 'plpython' is trying get linked using '-lpython${*python_version*}', but
>> it should be '-lpython${*python_ldversion*}'.

> That, on the other hand, will be a problem.
> get_config_vars('LDVERSION') isn't defined before Python 3.2, so this
> will break all previous versions.

Yes. In particular, this appears to be doing the wrong thing on my Lion
installation: it changes
python_libspec = -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7
to just
python_libspec = -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython
and there is no libpython.dylib in that directory. The build
accidentally fails to fail because there is a libpython.dylib
in /usr/lib and it happens to be symlinked to the right version of
python, but I hardly think we want to trust that.

I'm also wondering why a patch that's supposed to enable building
against python 3.2 should need to touch the "old way" code path.
If 3.2 isn't using the "new way", what exactly does?

regards, tom lane

On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
> FWIW, all three python installations I have handy (2.7 on Fedora 14, 2.7
> on OS X Lion, 2.5 on HPUX) produce the same result from either of
>
> python -c "from distutils.sysconfig import get_python_lib as f; import os; print(os.path.join(f(plat_specific=1,standard_lib=1),'config'))"
> python -c "import distutils.sysconfig,string; print(' '.join(filter(None,distutils.sysconfig.get_config_vars('LIBPL'))))"
>
> It's not immediately apparent to me why we should think that
> get_python_lib is less trustworthy than LIBPL; but if someone
> can make that case, I don't have any objection to this part of
> the patch.

The issue, at least for me, is that the file isn't necessarily called
'config' anymore. I have

/usr/lib/python3.2/config-3.2mu

because of some shared object ABI tagging system they introduced.
(/usr/lib/python3.2/config is a symlink to that, as a transition
measure, I guess.)

LIBPL exists at least as far back as Python 2.2, so its use should be
safe.

> >> 2. 'plpython' is trying get linked using '-lpython${*python_version*}', but
> >> it should be '-lpython${*python_ldversion*}'.
>
> > That, on the other hand, will be a problem.
> > get_config_vars('LDVERSION') isn't defined before Python 3.2, so this
> > will break all previous versions.
>
> Yes. In particular, this appears to be doing the wrong thing on my Lion
> installation: it changes
> python_libspec = -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython2.7
> to just
> python_libspec = -L/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/config -lpython
> and there is no libpython.dylib in that directory. The build
> accidentally fails to fail because there is a libpython.dylib
> in /usr/lib and it happens to be symlinked to the right version of
> python, but I hardly think we want to trust that.

Yes, because get_config_vars('LDVERSION') doesn't exist in that version.
In theory, it would return '2.7', so everything would fit back together,
but LDVERSION doesn't exist before 3.2.

> I'm also wondering why a patch that's supposed to enable building
> against python 3.2 should need to touch the "old way" code path.
> If 3.2 isn't using the "new way", what exactly does?

Analogously to the point above, the result on my system should be

-L something -lpython3.2mu

And that's what I get.

The claim is that on the ActiveState installation, this doesn't work
out, but we need to see some details here, I guess.

Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
>> It's not immediately apparent to me why we should think that
>> get_python_lib is less trustworthy than LIBPL; but if someone
>> can make that case, I don't have any objection to this part of
>> the patch.

> The issue, at least for me, is that the file isn't necessarily called
> 'config' anymore. I have
> /usr/lib/python3.2/config-3.2mu

Ah, I see.

> LIBPL exists at least as far back as Python 2.2, so its use should be
> safe.

Yeah, that part of the patch seems sane then.

> Yes, because get_config_vars('LDVERSION') doesn't exist in that version.
> In theory, it would return '2.7', so everything would fit back together,
> but LDVERSION doesn't exist before 3.2.

Could we have the code use 'LDVERSION' if it gets a nonempty result,
and otherwise fall back to the current scheme? But I guess first we
need some details as to why the current scheme isn't sufficient.

regards, tom lane

On Thu, Aug 18, 2011 at 1:25 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:

> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
> >> It's not immediately apparent to me why we should think that
> >> get_python_lib is less trustworthy than LIBPL; but if someone
> >> can make that case, I don't have any objection to this part of
> >> the patch.
>
> > The issue, at least for me, is that the file isn't necessarily called
> > 'config' anymore. I have
> > /usr/lib/python3.2/config-3.2mu
>
One of the reason, I say that - we do have hard-coded values for the config
directory.
Hence, I used the LIBPL.

>
> Ah, I see.
>
> > LIBPL exists at least as far back as Python 2.2, so its use should be
> > safe.
>
> Yeah, that part of the patch seems sane then.
>
> > Yes, because get_config_vars('LDVERSION') doesn't exist in that version.
> > In theory, it would return '2.7', so everything would fit back together,
> > but LDVERSION doesn't exist before 3.2.
>
Oops - sorry...
I did not know about it..

>
> Could we have the code use 'LDVERSION' if it gets a nonempty result,
> and otherwise fall back to the current scheme? But I guess first we
> need some details as to why the current scheme isn't sufficient.
>
Please find the attached patch as you suggested.

Reason:
- As per my findings, ActiveState Python 3.2 does not provide shared
libraries along with it.
(Though - I am not sure about the earlier version of ActiveState Python)
We can confirm the same using the following command:
${PYTHON} -c "import distutils.sysconfig,string;
print(distutils.sysconfig.get_config_vars('Py_ENABLE_SHARED'))"
Which returns in this case '0'.

And, getting values for the 'python_ldlibrary' and 'python_so' are
'libpython3.2m.a' and '.cpython-32m.so' respectively.
So, the condition - *x"${python_ldlibrary}" != x"${ldlibrary}"* is always
failing, and switching it back to link the old way.

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com/>

*http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>

>
> regards, tom lane
>

Please ignore the previous patch.
Please find the updated patch.

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com>

*http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>

On Thu, Aug 18, 2011 at 12:57 PM, Ashesh Vashi <
ashesh(dot)vashi(at)enterprisedb(dot)com> wrote:

> On Thu, Aug 18, 2011 at 1:25 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
>
>> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
>> > On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
>> >> It's not immediately apparent to me why we should think that
>> >> get_python_lib is less trustworthy than LIBPL; but if someone
>> >> can make that case, I don't have any objection to this part of
>> >> the patch.
>>
>> > The issue, at least for me, is that the file isn't necessarily called
>> > 'config' anymore. I have
>> > /usr/lib/python3.2/config-3.2mu
>>
> One of the reason, I say that - we do have hard-coded values for the config
> directory.
> Hence, I used the LIBPL.
>
>>
>> Ah, I see.
>>
>> > LIBPL exists at least as far back as Python 2.2, so its use should be
>> > safe.
>>
>> Yeah, that part of the patch seems sane then.
>>
>> > Yes, because get_config_vars('LDVERSION') doesn't exist in that version.
>> > In theory, it would return '2.7', so everything would fit back together,
>> > but LDVERSION doesn't exist before 3.2.
>>
> Oops - sorry...
> I did not know about it..
>
>>
>> Could we have the code use 'LDVERSION' if it gets a nonempty result,
>> and otherwise fall back to the current scheme? But I guess first we
>> need some details as to why the current scheme isn't sufficient.
>>
> Please find the attached patch as you suggested.
>
> Reason:
> - As per my findings, ActiveState Python 3.2 does not provide shared
> libraries along with it.
> (Though - I am not sure about the earlier version of ActiveState Python)
> We can confirm the same using the following command:
> ${PYTHON} -c "import distutils.sysconfig,string;
> print(distutils.sysconfig.get_config_vars('Py_ENABLE_SHARED'))"
> Which returns in this case '0'.
>
> And, getting values for the 'python_ldlibrary' and 'python_so' are
> 'libpython3.2m.a' and '.cpython-32m.so' respectively.
> So, the condition - *x"${python_ldlibrary}" != x"${ldlibrary}"* is always
> failing, and switching it back to link the old way.
>
> --
>
> Thanks & Regards,
>
> Ashesh Vashi
> EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com/>
>
>
>
> *http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>
>
>
>>
>> regards, tom lane
>>
>
>

On tor, 2011-08-18 at 14:51 +0530, Ashesh Vashi wrote:
> Please ignore the previous patch.
> Please find the updated patch.

Committed more or less like that.

In passing I also fixed the build with Python 3 on Windows, which
apparently never worked before. But I suppose you have been referring
to the ActiveState Linux build all along.

>
> --
>
> Thanks & Regards,
>
> Ashesh Vashi
> EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com>
>
>
>
> *http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>
>
>
>
> On Thu, Aug 18, 2011 at 12:57 PM, Ashesh Vashi <
> ashesh(dot)vashi(at)enterprisedb(dot)com> wrote:
>
> > On Thu, Aug 18, 2011 at 1:25 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> >
> >> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> >> > On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
> >> >> It's not immediately apparent to me why we should think that
> >> >> get_python_lib is less trustworthy than LIBPL; but if someone
> >> >> can make that case, I don't have any objection to this part of
> >> >> the patch.
> >>
> >> > The issue, at least for me, is that the file isn't necessarily called
> >> > 'config' anymore. I have
> >> > /usr/lib/python3.2/config-3.2mu
> >>
> > One of the reason, I say that - we do have hard-coded values for the config
> > directory.
> > Hence, I used the LIBPL.
> >
> >>
> >> Ah, I see.
> >>
> >> > LIBPL exists at least as far back as Python 2.2, so its use should be
> >> > safe.
> >>
> >> Yeah, that part of the patch seems sane then.
> >>
> >> > Yes, because get_config_vars('LDVERSION') doesn't exist in that version.
> >> > In theory, it would return '2.7', so everything would fit back together,
> >> > but LDVERSION doesn't exist before 3.2.
> >>
> > Oops - sorry...
> > I did not know about it..
> >
> >>
> >> Could we have the code use 'LDVERSION' if it gets a nonempty result,
> >> and otherwise fall back to the current scheme? But I guess first we
> >> need some details as to why the current scheme isn't sufficient.
> >>
> > Please find the attached patch as you suggested.
> >
> > Reason:
> > - As per my findings, ActiveState Python 3.2 does not provide shared
> > libraries along with it.
> > (Though - I am not sure about the earlier version of ActiveState Python)
> > We can confirm the same using the following command:
> > ${PYTHON} -c "import distutils.sysconfig,string;
> > print(distutils.sysconfig.get_config_vars('Py_ENABLE_SHARED'))"
> > Which returns in this case '0'.
> >
> > And, getting values for the 'python_ldlibrary' and 'python_so' are
> > 'libpython3.2m.a' and '.cpython-32m.so' respectively.
> > So, the condition - *x"${python_ldlibrary}" != x"${ldlibrary}"* is always
> > failing, and switching it back to link the old way.
> >
> > --
> >
> > Thanks & Regards,
> >
> > Ashesh Vashi
> > EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com/>
> >
> >
> >
> > *http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>
> >
> >
> >>
> >> regards, tom lane
> >>
> >
> >

On Thu, Aug 18, 2011 at 5:25 PM, Peter Eisentraut <peter_e(at)gmx(dot)net> wrote:

> On tor, 2011-08-18 at 14:51 +0530, Ashesh Vashi wrote:
> > Please ignore the previous patch.
> > Please find the updated patch.
>
> Committed more or less like that.
>
Thanks

>
> In passing I also fixed the build with Python 3 on Windows, which
> apparently never worked before. But I suppose you have been referring
> to the ActiveState Linux build all along.
>
Yes.

--

Thanks & Regards,

Ashesh Vashi
EnterpriseDB INDIA: Enterprise PostgreSQL Company<http://www.enterprisedb.com/>

*http://www.linkedin.com/in/asheshvashi*<http://www.linkedin.com/in/asheshvashi>

>
> >
> > --
> >
> > Thanks & Regards,
> >
> > Ashesh Vashi
> > EnterpriseDB INDIA: Enterprise PostgreSQL Company<
> http://www.enterprisedb.com>
> >
> >
> >
> > *http://www.linkedin.com/in/asheshvashi*<
> http://www.linkedin.com/in/asheshvashi>
> >
> >
> >
> > On Thu, Aug 18, 2011 at 12:57 PM, Ashesh Vashi <
> > ashesh(dot)vashi(at)enterprisedb(dot)com> wrote:
> >
> > > On Thu, Aug 18, 2011 at 1:25 AM, Tom Lane <tgl(at)sss(dot)pgh(dot)pa(dot)us> wrote:
> > >
> > >> Peter Eisentraut <peter_e(at)gmx(dot)net> writes:
> > >> > On ons, 2011-08-17 at 13:20 -0400, Tom Lane wrote:
> > >> >> It's not immediately apparent to me why we should think that
> > >> >> get_python_lib is less trustworthy than LIBPL; but if someone
> > >> >> can make that case, I don't have any objection to this part of
> > >> >> the patch.
> > >>
> > >> > The issue, at least for me, is that the file isn't necessarily
> called
> > >> > 'config' anymore. I have
> > >> > /usr/lib/python3.2/config-3.2mu
> > >>
> > > One of the reason, I say that - we do have hard-coded values for the
> config
> > > directory.
> > > Hence, I used the LIBPL.
> > >
> > >>
> > >> Ah, I see.
> > >>
> > >> > LIBPL exists at least as far back as Python 2.2, so its use should
> be
> > >> > safe.
> > >>
> > >> Yeah, that part of the patch seems sane then.
> > >>
> > >> > Yes, because get_config_vars('LDVERSION') doesn't exist in that
> version.
> > >> > In theory, it would return '2.7', so everything would fit back
> together,
> > >> > but LDVERSION doesn't exist before 3.2.
> > >>
> > > Oops - sorry...
> > > I did not know about it..
> > >
> > >>
> > >> Could we have the code use 'LDVERSION' if it gets a nonempty result,
> > >> and otherwise fall back to the current scheme? But I guess first we
> > >> need some details as to why the current scheme isn't sufficient.
> > >>
> > > Please find the attached patch as you suggested.
> > >
> > > Reason:
> > > - As per my findings, ActiveState Python 3.2 does not provide shared
> > > libraries along with it.
> > > (Though - I am not sure about the earlier version of ActiveState
> Python)
> > > We can confirm the same using the following command:
> > > ${PYTHON} -c "import distutils.sysconfig,string;
> > > print(distutils.sysconfig.get_config_vars('Py_ENABLE_SHARED'))"
> > > Which returns in this case '0'.
> > >
> > > And, getting values for the 'python_ldlibrary' and 'python_so' are
> > > 'libpython3.2m.a' and '.cpython-32m.so' respectively.
> > > So, the condition - *x"${python_ldlibrary}" != x"${ldlibrary}"* is
> always
> > > failing, and switching it back to link the old way.
> > >
> > > --
> > >
> > > Thanks & Regards,
> > >
> > > Ashesh Vashi
> > > EnterpriseDB INDIA: Enterprise PostgreSQL Company<
> http://www.enterprisedb.com/>
> > >
> > >
> > >
> > > *http://www.linkedin.com/in/asheshvashi*<
> http://www.linkedin.com/in/asheshvashi>
> > >
> > >
> > >>
> > >> regards, tom lane
> > >>
> > >
> > >
>
>
>
>