Re: x86_64 configure problem

On an x86_64 machine I'm finding that I cannot configure --with-python
without the attached patch. Undoubtedly there is a better way to fix
this -- any suggestions?

Another configure issue: I find that --enable-depend breaks parallel builds:

make -j 2
[...]
make[3]: Entering directory
`/opt/src/pgsql-cvs/pgsql-8.0/src/backend/utils/mb/conversion_procs/euc_tw_and_big5'
gcc -O2 -fno-strict-aliasing -g -Wall -Wmissing-prototypes
-Wmissing-declarations -fpic -I../../../../../../src/include
-D_GNU_SOURCE -I/usr/include/et -c -o big5.o big5.c -MMD
gcc -O2 -fno-strict-aliasing -g -Wall -Wmissing-prototypes
-Wmissing-declarations -fpic -I../../../../../../src/include
-D_GNU_SOURCE -I/usr/include/et -c -o euc_tw_and_big5.o
euc_tw_and_big5.c -MMD
mkdir: cannot create directory `.deps': File exists
make[3]: *** [big5.o] Error 1
make[3]: *** Deleting file `big5.o'
make[3]: *** Waiting for unfinished jobs....
make[3]: *** Waiting for unfinished jobs....
make[3]: *** Waiting for unfinished jobs....
make[3]: Leaving directory
`/opt/src/pgsql-cvs/pgsql-8.0/src/backend/utils/mb/conversion_procs/euc_tw_and_big5'
make[2]: *** [all] Error 2
make[2]: Leaving directory
`/opt/src/pgsql-cvs/pgsql-8.0/src/backend/utils/mb/conversion_procs'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/opt/src/pgsql-cvs/pgsql-8.0/src'
make: *** [all] Error 2

Any ideas about how to fix this?

Thanks,

Joe

On Thu, 2004-09-09 at 11:17, Joe Conway wrote:
> On an x86_64 machine I'm finding that I cannot configure --with-python
> without the attached patch. Undoubtedly there is a better way to fix
> this -- any suggestions?

python_configdir=`${PYTHON} -c "from distutils.sysconfig import
get_python_lib as f; import os; print os.path.join(f(plat_specific=1,
standard_lib=1),'config')"`

That should work, regardless of the lib directory that Python is
installed to.
(at least it works on my system):

flaw(at)void:~ % py -c "from distutils.sysconfig import get_python_lib as
f; import os; print
os.path.join(f(plat_specific=1,standard_lib=1),'config')"
/usr/local/lib/python2.3/config

--
Regards,
James William Pye

On Thu, 2004-09-09 at 12:28, James William Pye wrote:
> That should work, regardless of the lib directory that Python is
> installed to.

Looking at get_python_lib(), I'm not so sure that I'm correct:

if os.name == "posix":
libpython = os.path.join(prefix,
"lib", "python" + get_python_version())

Although, I'm getting a bit confused. I just compiled and installed
Python(CVS) configured as:

./configure --prefix=/usr/dev --libdir=/usr/dev/pylib

But it didn't install anything in /usr/dev/pylib, as one would expect.
It just threw everything in /usr/dev/lib. I even gmake'd distclean to
make sure there wasn't some configure option cache of some sort. Same
thing.

ISTM that Python's libdir wasn't meant to be located anywhere other than
EPREFIX/lib.

--
Regards,
James William Pye

James William Pye wrote:
> Looking at get_python_lib(), I'm not so sure that I'm correct:
>

It does seem to work for me:

# python -c "from distutils.sysconfig import get_python_lib as f; import
os; print os.path.join(f(plat_specific=1,standard_lib=1),'config')"
/usr/lib64/python2.3/config

Any other proposals? If not, any objections to the attached patch?

Joe

I've just been asked to clarify what actually happens when a DELETE takes
place, and what happens to TOASTed data. The MVCC never-update-in-place
phrase caused some debate over what happens. I couldn't find a specific and
conclusive comment on this that I trust to be completely up to date. One may
exist however....?

My answer was this, though this was not thought accurate (on the DELETE
aspect):
For DELETEs, the xmax field on the tuple/row header is updated-in-place to
show the xid that deleted the row. Thus, DELETEs of long data rows are just
as efficient as DELETEs of shorter data rows, since both effect only a
single data page. This still allows concurrent access because only one
backend may hold the page lock at any time, so nobody is actively reading
the row at the time of the setting of xmax for the DELETE.

MVCC uses a never-update-in-place algorithm to allow concurrency during
UPDATEs. In general, the whole row is re-written, just as if the data had
been DELETEd and then re-INSERTed. If there are TOASTed fields, then the
TOASTed data is only re-written if it is has changed as part of the UPDATE.
Thus an UPDATE of a row with TOASTed data, yet that doesn't alter the
TOASTed data itself, is no more expensive than an UPDATE of a shorter row,
since it will change only one page (with an equal risk of requiring a new
block write as a result of the insertion of the new main row version).

DELETE does NOT take a full copy of the row and THEN mark the xmax field as
the xid of the deleting transaction, so is NOT similar to an UPDATE action
on the same row.

I've checked heapam.c and tuptoaster.c, and think this is correct.

Any differing views? If possible, please refer me to the code - I'm still
working my way around the heap access code, so feel free to show me the
light...

Best regards, Simon Riggs

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:

> I've just been asked to clarify what actually happens when a DELETE takes
> place, and what happens to TOASTed data.

What does this have to do with "x86_64 configure problem"?

--
greg

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> My answer was this, though this was not thought accurate (on the DELETE
> aspect):

This is correct as far as it goes, but given the question I imagine some
further detail is appropriate:

* When deleting a row that has out-of-line-toasted fields, we must also
find the TOAST row(s) for those fields and mark them deleted. So it is
not strictly true that deleting a wide row has the same cost as deleting
a narrow one: you will have to touch all the pages containing TOAST data
as well.

* UPDATE is indeed basically equivalent to a DELETE and INSERT, with the
sole optimization being that we carry over any unchanged toasted fields.
So in particular, any changed toasted fields will need to be marked
deleted in the TOAST table.

It would be nice to push the TOAST deletions off to become the
responsibility of VACUUM, but I'm not entirely sure how to do that
without giving up the UPDATE optimization of shared values.

regards, tom lane

"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
>> Tom Lane wrote
>> It would be nice to push the TOAST deletions off to become the
>> responsibility of VACUUM, but I'm not entirely sure how to do that
>> without giving up the UPDATE optimization of shared values.

> That could be optimised, but there are advantages to removing TOASTed values
> immediately since they are potentially big space hogs, so I'm actually glad
> to hear that it works that way.

Well, they certainly aren't going to be *removed* immediately; we
couldn't roll back the deletion if we did. The physical removal
will happen during VACUUM, same as for ordinary table rows. So there's
not really any functional reason why we couldn't have VACUUM do the
marking for removal too, except that it does not know when looking at
the deleted table row whether the toast entries are shared with other
versions of the row.

regards, tom lane

> Tom Lane wrote
> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> > My answer was this, though this was not thought accurate (on the DELETE
> > aspect):
>
> This is correct as far as it goes, but given the question I imagine some
> further detail is appropriate:
>
> * When deleting a row that has out-of-line-toasted fields, we must also
> find the TOAST row(s) for those fields and mark them deleted. So it is
> not strictly true that deleting a wide row has the same cost as deleting
> a narrow one: you will have to touch all the pages containing TOAST data
> as well.
>
> * UPDATE is indeed basically equivalent to a DELETE and INSERT, with the
> sole optimization being that we carry over any unchanged toasted fields.
> So in particular, any changed toasted fields will need to be marked
> deleted in the TOAST table.
>

Thanks!

> It would be nice to push the TOAST deletions off to become the
> responsibility of VACUUM, but I'm not entirely sure how to do that
> without giving up the UPDATE optimization of shared values.
>

That could be optimised, but there are advantages to removing TOASTed values
immediately since they are potentially big space hogs, so I'm actually glad
to hear that it works that way.

Best Regards, Simon Riggs

Joe Conway wrote:
> James William Pye wrote:
>
>> Looking at get_python_lib(), I'm not so sure that I'm correct:
>>
>
> It does seem to work for me:
>
> # python -c "from distutils.sysconfig import get_python_lib as f; import
> os; print os.path.join(f(plat_specific=1,standard_lib=1),'config')"
> /usr/lib64/python2.3/config
>
> Any other proposals? If not, any objections to the attached patch?

No one, we have to check also for the presence of distutils package installation.
It's not installed by default and I was bitten by it during the RPM building for
RH AS 2.1

Regards
Gaetano Mendola

Greg Stark wrote:

> "Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
>
>
>>I've just been asked to clarify what actually happens when a DELETE takes
>>place, and what happens to TOASTed data.
>
>
> What does this have to do with "x86_64 configure problem"?

I believe this was a new thread but Outlook or the new server had a glitch.

Regards
Gaetano Mendola

On Fri, Sep 10, 2004 at 02:10:08AM +0200, Gaetano Mendola wrote:
> Greg Stark wrote:
>
> >"Simon Riggs" <simon(at)2ndquadrant(dot)com> writes:
> >
> >
> >>I've just been asked to clarify what actually happens when a DELETE takes
> >>place, and what happens to TOASTed data.
> >
> >What does this have to do with "x86_64 configure problem"?
>
> I believe this was a new thread but Outlook or the new server had a glitch.

No, Simon used another thread. The Reply-To: header was present.

--
Alvaro Herrera (<alvherre[a]dcc.uchile.cl>)
"Uno combate cuando es necesario... ¡no cuando está de humor!
El humor es para el ganado, o para hacer el amor, o para tocar el
baliset. No para combatir." (Gurney Halleck)

Gaetano Mendola wrote:
> Joe Conway wrote:
>> # python -c "from distutils.sysconfig import get_python_lib as f;
>> import os; print
>> os.path.join(f(plat_specific=1,standard_lib=1),'config')"
>> /usr/lib64/python2.3/config
>>
>> Any other proposals? If not, any objections to the attached patch?
>
> No one, we have to check also for the presence of distutils package
> installation.
> It's not installed by default and I was bitten by it during the RPM
> building for
> RH AS 2.1

What version of python comes with RH2.1?

In any case, the online documentation for python suggests that it is
reasonable to expect that distutils is already installed (by default for
python >= 1.6, and by the user for python 1.5.2). See:
http://www.python.org/doc/2.2.3/dist/intro.html

If there are no other objections, I'll commit the attached in about 24
hours.

Thanks,

Joe

Joe Conway wrote:

> Gaetano Mendola wrote:
>
>> Joe Conway wrote:
>>
>>> # python -c "from distutils.sysconfig import get_python_lib as f;
>>> import os; print
>>> os.path.join(f(plat_specific=1,standard_lib=1),'config')"
>>> /usr/lib64/python2.3/config
>>>
>>> Any other proposals? If not, any objections to the attached patch?
>>
>>
>> No one, we have to check also for the presence of distutils package
>> installation.
>> It's not installed by default and I was bitten by it during the RPM
>> building for
>> RH AS 2.1
>
>
> What version of python comes with RH2.1?

1.5.2

> In any case, the online documentation for python suggests that it is
> reasonable to expect that distutils is already installed (by default for
> python >= 1.6, and by the user for python 1.5.2). See:
> http://www.python.org/doc/2.2.3/dist/intro.html
>
> If there are no other objections, I'll commit the attached in about 24
> hours.

Are you going also to change the configure so it will check the distutils
package installation ?

Regards
Gaetano Mendola

Gaetano Mendola wrote:
> Joe Conway wrote:
>> What version of python comes with RH2.1?
>
> 1.5.2
>
>> In any case, the online documentation for python suggests that it is
>> reasonable to expect that distutils is already installed (by default
>> for python >= 1.6, and by the user for python 1.5.2). See:
>> http://www.python.org/doc/2.2.3/dist/intro.html
>>
>> If there are no other objections, I'll commit the attached in about 24
>> hours.
>
> Are you going also to change the configure so it will check the distutils
> package installation ?

I wasn't going to -- python's own docs say I shouldn't need to. From the
link above:

"the Distutils work just fine with Python 1.5.2, and it is reasonable
(and expected to become commonplace) to expect users of Python 1.5.2 to
download and install the Distutils separately before they can install
your modules"

Joe

Joe Conway wrote:
> Gaetano Mendola wrote:
>
>> Joe Conway wrote:
>>
>>> What version of python comes with RH2.1?
>>
>>
>> 1.5.2
>>
>>> In any case, the online documentation for python suggests that it is
>>> reasonable to expect that distutils is already installed (by default
>>> for python >= 1.6, and by the user for python 1.5.2). See:
>>> http://www.python.org/doc/2.2.3/dist/intro.html
>>>
>>> If there are no other objections, I'll commit the attached in about
>>> 24 hours.
>>
>>
>> Are you going also to change the configure so it will check the distutils
>> package installation ?
>
>
> I wasn't going to -- python's own docs say I shouldn't need to. From the
> link above:
>
> "the Distutils work just fine with Python 1.5.2, and it is reasonable
> (and expected to become commonplace) to expect users of Python 1.5.2 to
> download and install the Distutils separately before they can install
> your modules"

I'm not native english speakers but that sentence above doesn't imply
Python 1.5.2 => Distutils. So instead of an error like this:

checking Python installation directories... Traceback (innermost last):
File "<string>", line 1, in ?
ImportError: No module named distutils
Traceback (innermost last):
File "<string>", line 1, in ?
ImportError: No module named distutils
/usr/
checking how to link an embedded Python application... no
configure: error: Python Makefile not found
error: Bad exit status from /var/tmp/rpm-tmp.56347 (%build)

is better have:

Error: you need to have Distutil installed

Regards
Gaetano Mendola

Gaetano Mendola wrote:
> Joe Conway wrote:
>> I wasn't going to -- python's own docs say I shouldn't need to. From
>> the link above:
>>
>> "the Distutils work just fine with Python 1.5.2, and it is reasonable
>> (and expected to become commonplace) to expect users of Python 1.5.2
>> to download and install the Distutils separately before they can
>> install your modules"
>
> I'm not native english speakers but that sentence above doesn't imply
> Python 1.5.2 => Distutils. So instead of an error like this:
>
> checking Python installation directories... Traceback (innermost last):
> File "<string>", line 1, in ?
> ImportError: No module named distutils
> Traceback (innermost last):
> File "<string>", line 1, in ?
> ImportError: No module named distutils
> /usr/
> checking how to link an embedded Python application... no
> configure: error: Python Makefile not found
> error: Bad exit status from /var/tmp/rpm-tmp.56347 (%build)
>
> is better have:
>
> Error: you need to have Distutil installed

Perhaps. The first error message seems clear enough to me, but if you
want to send me the necessary distutils test, I can incorporate it. Any
other opinions out there?

Joe

On Sat, 2004-09-11 at 14:05, Joe Conway wrote:
> Perhaps. The first error message seems clear enough to me, but if you
> want to send me the necessary distutils test, I can incorporate it. Any
> other opinions out there?

AC_MSG_CHECKING([for Python distutils module])
if "${PYTHON}" 2>&- -c 'import distutils'
then
AC_MSG_RESULT(yes)
else
AC_MSG_RESULT(no)
AC_MSG_ERROR([distutils module not found])
fi

That should work.

While I'm of the opinion that installations without distutils are
antiquated, I see no reason why we couldn't check for it anyways.
(Gaetano requested it so there would probably be others who would
appreciate it.)

--
Regards,
James William Pye

Joe Conway wrote:

> Gaetano Mendola wrote:
>
>> Joe Conway wrote:
>>
>>> I wasn't going to -- python's own docs say I shouldn't need to. From
>>> the link above:
>>>
>>> "the Distutils work just fine with Python 1.5.2, and it is reasonable
>>> (and expected to become commonplace) to expect users of Python 1.5.2
>>> to download and install the Distutils separately before they can
>>> install your modules"
>>
>>
>> I'm not native english speakers but that sentence above doesn't imply
>> Python 1.5.2 => Distutils. So instead of an error like this:
>>
>> checking Python installation directories... Traceback (innermost last):
>> File "<string>", line 1, in ?
>> ImportError: No module named distutils
>> Traceback (innermost last):
>> File "<string>", line 1, in ?
>> ImportError: No module named distutils
>> /usr/
>> checking how to link an embedded Python application... no
>> configure: error: Python Makefile not found
>> error: Bad exit status from /var/tmp/rpm-tmp.56347 (%build)
>>
>> is better have:
>>
>> Error: you need to have Distutil installed
>
>
> Perhaps. The first error message seems clear enough to me, but if you
> want to send me the necessary distutils test, I can incorporate it. Any
> other opinions out there?

python -c "from distutils import *" > /dev/null 2>&1 || (echo "You need distutils installed"; exit 1)

Regards
Gaetano Mendola

Joe Conway wrote:
> Perhaps. The first error message seems clear enough to me, but if you
> want to send me the necessary distutils test, I can incorporate it.
> Any other opinions out there?

It looks like a waste of space and time to me. We can't really check
for everything, let alone preconditions of what we're really interested
in. We want to find the Python makefile, so let's look for that. The
error message if it's not found is perfectly clear.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Joe Conway wrote:
>
>>Perhaps. The first error message seems clear enough to me, but if you
>>want to send me the necessary distutils test, I can incorporate it.
>>Any other opinions out there?
>
>
> It looks like a waste of space and time to me.

Are you serious ? May you quantify the space and the time lost ?

> The error message if it's not found is perfectly clear.

Is it ?
The error say "no modules named distutils", is the module an external
one that I need to have installed or is a bogus error due the previous
line: File "<string>", line 1, in ?

And most of all why when I wrote about the inability to create the RPM for
RH2.1 no one wrote: you need to install the distutils package, after
having seen this error:

checking Python installation directories... Traceback (innermost last):
File "<string>", line 1, in ?
ImportError: No module named distutils
Traceback (innermost last):
File "<string>", line 1, in ?
ImportError: No module named distutils
/usr/
checking how to link an embedded Python application... no
configure: error: Python Makefile not found
error: Bad exit status from /var/tmp/rpm-tmp.56347 (%build)

may be because the message after all is not concise.

Regards
Gaetano Mendola

Gaetano Mendola wrote:
> python -c "from distutils import *" > /dev/null 2>&1 || (echo "You need
> distutils installed"; exit 1)
>

Sorry for the delay -- things got busy around here all of a sudden.

Attached is a version of the patch with James Pye's distutils checking
code. Gaetano, please verify it works properly for you.

Although Peter was clearly not in favor of this, Gaetano and James both
are, and I don't feel strongly either way. So barring any other
objections I'll apply this in a day or so.

One procedural issue did occur to me regarding this kind of change. It
requires someone to run autoconf after applying -- how is that normally
handled?

Thanks,

Joe

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Joe Conway wrote:
| Gaetano Mendola wrote:
|
|> python -c "from distutils import *" > /dev/null 2>&1 || (echo "You
|> need distutils installed"; exit 1)
|>
|
| Sorry for the delay -- things got busy around here all of a sudden.
|
| Attached is a version of the patch with James Pye's distutils checking
| code. Gaetano, please verify it works properly for you.

Patch applied and it works. I did the negative test and the positive as
well.

Thanks.

Regards
Gaetano Mendola

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (MingW32)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBSK5K7UpzwH2SGd4RAtUYAJ0dEjiGUgPhU0Qyb9PqRK+5nJct7wCgvMti
A/J0qMMGqIVQH3E4BdV0Tu8=
=T+77
-----END PGP SIGNATURE-----

Joe Conway wrote:
> One procedural issue did occur to me regarding this kind of change.
> It requires someone to run autoconf after applying -- how is that
> normally handled?

You run autoconf before you commit and then check it in. Please use
version 2.53.

--
Peter Eisentraut
http://developer.postgresql.org/~petere/

Peter Eisentraut wrote:
> Joe Conway wrote:
>>One procedural issue did occur to me regarding this kind of change.
>>It requires someone to run autoconf after applying -- how is that
>>normally handled?
>
> You run autoconf before you commit and then check it in. Please use
> version 2.53.
>

Thanks. Attached patch applied.

BTW, if anyone is interested in an autoconf253 source RPM that will
build on Fedora Core 2, let me know.

Joe