Re: Making PostgreSQL 7.4 (CVS) work properly on OS X 10.3 (7B85)

Here's my notes on how to build PostgreSQL 7.4 (CVS) on OS X 10.3
(7B85) with (seems to be working, but I haven't really ran any tests)
python, tcl, perl, readline. I noticed today that you guys are just
about to release, so I figure I should bring this up right now. Please
feel free to email me if you have any questions/comments, copy me
off-list if possible because I probably won't stay subscribed for very
long.

Notes:
----------
* CVS co pgsql at about Nov 7 2003 5pm EST
* These instructions PROBABLY DO NOT WORK ON 10.2! If these changes to
into a configure script, make sure to trap them against darwin >= 7.0.0
* downloaded and built static readline
* downloaded and built latest bison 1.875
* /usr/local/bin comes before /usr/bin in my path! (this matters for
bison, probably)
* Probably a good idea to include kerberos support as well, since OS X
Server 10.3 uses it a lot, but I don't know enough about it to try..
* I have set env MAC_OS_X_DEPLOYMENT_TARGET 10.3
* I ditched the system.c hack, assuming Apple has fixed them by 10.3 --
because it breaks tcl and python if you do.. I did:

#include <AccessibilityMacros.h>
#ifdef MAC_OS_X_10_3
... original code
#endif

This can be done nicer, of course.

It seems that you can't sysctl some things after they've been set
once.. so I had to sudo vi /etc/rc file, changed the following two
parameters
sysctl -w kern.sysv.shmmax=167772160
sysctl -w kern.sysv.shmall=65536

Then rebooted.

Shell commands:
------------------------
cd pgsql
mkdir build_osx
cd build_osx
../configure --with-tcl --without-tk
--with-tclconfig=/Library/Frameworks/Tcl.framework/Versions/8.4
--with-perl --with-python --without-java --with-rendezvous
--with-openssl
make

Problems:
--------------
*** Cannot build PL/Python because libpython is not a shared library.
*** You might have to rebuild your Python installation. Refer to
*** the documentation for details.

Ok, that's not quite true, so:
I edited src/Makefile.global to have
python_libspec = -framework Python

You could probably just grep the LINKFORSHARED line out of
/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/
config/Makefile, but it links to things that aren't necessary for the
plugin bundle.

Then I edited src/pl/plpython/Makefile and changed the "ifneq" that
makes it not build to "ifneq (justbuildit, ....." so it would try and
build it anyways. This is a stupid hack.

These things should be fixed in the configure process, but I'm too lazy
to hack that at the moment.

Shell commands:
------------------------
make
sudo make install

Shell commands:
------------------------
sudo mkdir /usr/local/pgsql/data
sudo chown mysql /usr/local/pgsql/data
sudo -u mysql /usr/local/pgsql/bin/initdb -D /usr/local/pgsql/data
sudo -u mysql /usr/local/pgsql/bin/pg_ctl -D /usr/local/pgsql/data -l
/usr/local/pgsql/data/postgres.log start
sudo -u mysql /usr/local/pgsql/bin/createdb test
sudo -u mysql /usr/local/pgsql/bin/createlang pltcl test
sudo -u mysql /usr/local/pgsql/bin/createlang plpythonu test
sudo -u mysql /usr/local/pgsql/bin/createlang plperl test

Notes:
---------
* Yes I know it's blasphemous to use mysql as the username, but OS X
comes with a mysql account, just think of it as an "easy to use uid
74". Yes, I'm lazy.
* Yes, my only test is createlang.. but that crashes for tcl and
python w/o the system fix.

Bob Ippolito <bob(at)redivi(dot)com> writes:
> Here's my notes on how to build PostgreSQL 7.4 (CVS) on OS X 10.3
> (7B85) with (seems to be working, but I haven't really ran any tests)
> python, tcl, perl, readline.

I have just in the past couple hours realized that ps_status.c is
seriously broken on OS X 10.3. It appears that Apple has randomly
decided to start #define'ing "BSD", which they did not do in any
prior OS X release, and this confuses ps_status.c into choosing the
wrong method of updating the ps-visible command line. Aside from
causing ps to not show any useful info about postmaster child processes,
this seems to cause dynamic loading of libraries to fail in some
cases :-( ... still trying to understand what's happening there ...

> I noticed today that you guys are just
> about to release, so I figure I should bring this up right now.

Indeed. We need to fix this yesterday.

> * I ditched the system.c hack, assuming Apple has fixed them by 10.3 --
> because it breaks tcl and python if you do.

I don't see why system.c would affect the problem I'm seeing --- does
this really fix pltcl for you?

regards, tom lane

On Nov 8, 2003, at 12:31 AM, Tom Lane wrote:

> Bob Ippolito <bob(at)redivi(dot)com> writes:
>> Here's my notes on how to build PostgreSQL 7.4 (CVS) on OS X 10.3
>> (7B85) with (seems to be working, but I haven't really ran any tests)
>> python, tcl, perl, readline.
>
> I have just in the past couple hours realized that ps_status.c is
> seriously broken on OS X 10.3. It appears that Apple has randomly
> decided to start #define'ing "BSD", which they did not do in any
> prior OS X release, and this confuses ps_status.c into choosing the
> wrong method of updating the ps-visible command line. Aside from
> causing ps to not show any useful info about postmaster child
> processes,
> this seems to cause dynamic loading of libraries to fail in some
> cases :-( ... still trying to understand what's happening there ...

__APPLE__ is usually the only define you should really be depending on
(other than things you pass in yourself via configure).

>> I noticed today that you guys are just
>> about to release, so I figure I should bring this up right now.
>
> Indeed. We need to fix this yesterday.
>
>> * I ditched the system.c hack, assuming Apple has fixed them by 10.3
>> --
>> because it breaks tcl and python if you do.
>
> I don't see why system.c would affect the problem I'm seeing --- does
> this really fix pltcl for you?

If you're getting the NSLinkModule -> mmap crash, then yes. It's the
multiply defined symbol that kills it (system in postmaster, system in
libSystem). The other crashes I saw were related to the shared mem
limit that you can only set *once*.. so you have to sudo vi /etc/rc
instead of /etc/sysctl.conf :(

-bob

Bob Ippolito <bob(at)redivi(dot)com> writes:
> On Nov 8, 2003, at 12:31 AM, Tom Lane wrote:
>> I have just in the past couple hours realized that ps_status.c is
>> seriously broken on OS X 10.3. It appears that Apple has randomly
>> decided to start #define'ing "BSD",

> __APPLE__ is usually the only define you should really be depending on

It is not that we are failing to recognize Darwin, it is that Darwin is
falsely claiming to be something else.

>> I don't see why system.c would affect the problem I'm seeing --- does
>> this really fix pltcl for you?

> If you're getting the NSLinkModule -> mmap crash, then yes.

No, I'm seeing an NSLinkModule -> memcmp crash. memcmp() is crashing
because it is being passed a null pointer, and the pointer in question
is null because ps_status.c zeroed it via "argv[1] = NULL".

I modified ps_status to use the correct status update method
(PS_CLOBBER_ARGV) but it still failed :-(. It sorta looks like
something in the dynamic loader may be relying on a stale copy of
the "environ" pointer ... does that ring a bell at all?

regards, tom lane

On Nov 8, 2003, at 1:00 AM, Tom Lane wrote:

> Bob Ippolito <bob(at)redivi(dot)com> writes:
>> On Nov 8, 2003, at 12:31 AM, Tom Lane wrote:
>>> I have just in the past couple hours realized that ps_status.c is
>>> seriously broken on OS X 10.3. It appears that Apple has randomly
>>> decided to start #define'ing "BSD",
>
>> __APPLE__ is usually the only define you should really be depending on
>
> It is not that we are failing to recognize Darwin, it is that Darwin is
> falsely claiming to be something else.
>
>>> I don't see why system.c would affect the problem I'm seeing --- does
>>> this really fix pltcl for you?
>
>> If you're getting the NSLinkModule -> mmap crash, then yes.
>
> No, I'm seeing an NSLinkModule -> memcmp crash. memcmp() is crashing
> because it is being passed a null pointer, and the pointer in question
> is null because ps_status.c zeroed it via "argv[1] = NULL".
>
> I modified ps_status to use the correct status update method
> (PS_CLOBBER_ARGV) but it still failed :-(. It sorta looks like
> something in the dynamic loader may be relying on a stale copy of
> the "environ" pointer ... does that ring a bell at all?

Er... I meant memcmp.. Have you tried removing the system.c hack?
That's what fixed it for me. I'm pretty sure that the multiply defined
_system is messing with things. Is it possible to use execve to
rewrite argv[0]? I've had better luck with that.

-bob

Bob Ippolito <bob(at)redivi(dot)com> writes:
> On Nov 8, 2003, at 12:31 AM, Tom Lane wrote:
>> I have just in the past couple hours realized that ps_status.c is
>> seriously broken on OS X 10.3.

> Er... I meant memcmp.. Have you tried removing the system.c hack?
> That's what fixed it for me.

AFAICT system.c hasn't got anything to do with the problem that I'm
seeing; it's purely a matter of ps_status.c clobbering argv[] contents
that the dynamic loader depends on for some weird reason. It looks
like Apple's implementation stores a copy of the original argc count,
and there is a bit of code in the loader that for some reason is
examining each argv string from 0..original_argc-1. Who knows why :-(
... but where we set save_argv[1] to NULL, we create a null pointer
crash in the loader. Take that out, no crash. You would not see this
crash if you started the postmaster with no command-line arguments, btw.

I'm planning to change ps_status so that instead of zeroing
save_argv[1], it makes all the original argv strings be pointers to ""
except for argv[0].

As for getting rid of system.c, I am not eager to do that since it would
certainly break compatibility with OS X 10.1. We could conditionally
compile it out perhaps. Do you know what #define symbol we could test
for to determine which OS X version we are on?

regards, tom lane

On Nov 8, 2003, at 1:13 PM, Tom Lane wrote:

> Bob Ippolito <bob(at)redivi(dot)com> writes:
>> On Nov 8, 2003, at 12:31 AM, Tom Lane wrote:
>>> I have just in the past couple hours realized that ps_status.c is
>>> seriously broken on OS X 10.3.
>
>> Er... I meant memcmp.. Have you tried removing the system.c hack?
>> That's what fixed it for me.
>
> AFAICT system.c hasn't got anything to do with the problem that I'm
> seeing; it's purely a matter of ps_status.c clobbering argv[] contents
> that the dynamic loader depends on for some weird reason. It looks
> like Apple's implementation stores a copy of the original argc count,
> and there is a bit of code in the loader that for some reason is
> examining each argv string from 0..original_argc-1. Who knows why :-(
> ... but where we set save_argv[1] to NULL, we create a null pointer
> crash in the loader. Take that out, no crash. You would not see this
> crash if you started the postmaster with no command-line arguments,
> btw.
>
> I'm planning to change ps_status so that instead of zeroing
> save_argv[1], it makes all the original argv strings be pointers to ""
> except for argv[0].

It may be causing problems because dyld does this thing called
@executable_path substitution so it can find dylibs relative to the
executable. Also, the WindowServer and several other things
(CoreFoundation, Foundation) use argv[0] to determine whether the
executable is inside a bundle or not.

I don't think OS X people would make a big fuss about argv[0] not being
as useful as it is on other platforms, personally I'd just take it out
if it's not working.

> As for getting rid of system.c, I am not eager to do that since it
> would
> certainly break compatibility with OS X 10.1. We could conditionally
> compile it out perhaps. Do you know what #define symbol we could test
> for to determine which OS X version we are on?

See /usr/include/AvailabilityMacros.h

-bob

Bob Ippolito <bob(at)redivi(dot)com> writes:
> On Nov 8, 2003, at 1:13 PM, Tom Lane wrote:
>> As for getting rid of system.c, I am not eager to do that since it
>> would
>> certainly break compatibility with OS X 10.1. We could conditionally
>> compile it out perhaps. Do you know what #define symbol we could test
>> for to determine which OS X version we are on?

> See /usr/include/AvailabilityMacros.h

I don't see anything there that we can use in the form

#ifdef OSX_VERSION_10_2
or
#if OSX_VERSION >= something

My 10.2.6 copy already has MAC_OS_X_VERSION_10_3 in it, so they are
obviously not intending that the highest defined symbol of that
series is the OS version.

regards, tom lane

On Nov 8, 2003, at 1:44 PM, Tom Lane wrote:

> Bob Ippolito <bob(at)redivi(dot)com> writes:
>> On Nov 8, 2003, at 1:13 PM, Tom Lane wrote:
>>> As for getting rid of system.c, I am not eager to do that since it
>>> would
>>> certainly break compatibility with OS X 10.1. We could conditionally
>>> compile it out perhaps. Do you know what #define symbol we could
>>> test
>>> for to determine which OS X version we are on?
>
>> See /usr/include/AvailabilityMacros.h
>
> I don't see anything there that we can use in the form
>
> #ifdef OSX_VERSION_10_2
> or
> #if OSX_VERSION >= something
>
> My 10.2.6 copy already has MAC_OS_X_VERSION_10_3 in it, so they are
> obviously not intending that the highest defined symbol of that
> series is the OS version.

MAC_OS_X_VERSION_MIN_REQUIRED

* If min OS not specified, assume 10.0
* Note: gcc driver may set MAC_OS_X_VERSION_MIN_REQUIRED based on
MACOSX_DEPLOYMENT_TARGET environment variable

Compiling software for 10.3 should setenv MACOSX_DEPLOYMENT_TARGET 10.3

-bob

Bob Ippolito <bob(at)redivi(dot)com> writes:
> * I ditched the system.c hack, assuming Apple has fixed them by 10.3 --
> because it breaks tcl and python if you do.. I did:

After fixing the ps_status problems, I cannot observe any problem, with
or without system.c. However, I agree that it's a bad idea to propagate
that hack forward when it's no longer needed. I've #ifdef'd out
system.c for OS X 10.3 and later. (I was tempted to disable it on
10.2.* as well, but I can only prove system() works in 10.2.6, not
having older OS X handy to test.)

> Ok, that's not quite true, so:
> I edited src/Makefile.global to have
> python_libspec = -framework Python

I'm not going to touch that at this point in the release cycle. It'll
just have to wait for a future release.

regards, tom lane

On Nov 8, 2003, at 3:19 PM, Tom Lane wrote:

> Bob Ippolito <bob(at)redivi(dot)com> writes:
>> * I ditched the system.c hack, assuming Apple has fixed them by 10.3
>> --
>> because it breaks tcl and python if you do.. I did:
>
> After fixing the ps_status problems, I cannot observe any problem, with
> or without system.c. However, I agree that it's a bad idea to
> propagate
> that hack forward when it's no longer needed. I've #ifdef'd out
> system.c for OS X 10.3 and later. (I was tempted to disable it on
> 10.2.* as well, but I can only prove system() works in 10.2.6, not
> having older OS X handy to test.)

Sounds great

>> Ok, that's not quite true, so:
>> I edited src/Makefile.global to have
>> python_libspec = -framework Python
>
> I'm not going to touch that at this point in the release cycle. It'll
> just have to wait for a future release.

That's fine, does anyone even use plpython? I only compiled it because
I knew how to fix it..

-bob

On 8 Nov 2003, at 22:19, Tom Lane wrote:
> After fixing the ps_status problems, I cannot observe any problem, with
> or without system.c. However, I agree that it's a bad idea to
> propagate
> that hack forward when it's no longer needed. I've #ifdef'd out
> system.c for OS X 10.3 and later. (I was tempted to disable it on
> 10.2.* as well, but I can only prove system() works in 10.2.6, not
> having older OS X handy to test.)

As a general rule of thumb, I don't think it's necessary to support
anything other than the latest revision in 10.2.x and 10.3.x at
any time. The revisions are free, they mainly fix bugs, they don't
affect hardware requirements and other software vendors see
requiring them as a non-issue.

This, at the very least, applies to major pgsql versions such
as 7.4 and 7.5 -- you might disagree about a OS version
requirement changing in a point release.

But in my opinion, if system() is fixed in 10.2.6, you can require
Jaguar users to upgrade to that version for 7.4.

mk

Marko Karppinen <marko(at)karppinen(dot)fi> writes:
> But in my opinion, if system() is fixed in 10.2.6, you can require
> Jaguar users to upgrade to that version for 7.4.

[shrug] If there were any need to, I'd do so. But we have no evidence
of any trouble caused by the system() hack in 10.2, so I think the
most prudent course is just to leave it alone.

regards, tom lane