Re: OS X shared memory documentation

I finally figured out what the issue was with OS X blowing away shared
memory settings from /etc/sysctl.conf. I added a note to the 8.1 docs
already, and here's a patch against -HEAD.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Thanks for this Jim! We'll test it and report results.

- Luke

________________________________

From: pgsql-patches-owner(at)postgresql(dot)org on behalf of Jim C. Nasby
Sent: Sat 2/11/2006 9:26 PM
To: pgsql-patches(at)postgresql(dot)org
Subject: [PATCHES] OS X shared memory documentation

I finally figured out what the issue was with OS X blowing away shared
memory settings from /etc/sysctl.conf. I added a note to the 8.1 docs
already, and here's a patch against -HEAD.
--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

"Jim C. Nasby" <jnasby(at)pervasive(dot)com> writes:
> + In OS X 10.3 and later, you can simply place the settings into
> + <filename>/etc/sysctl.conf</>, with one caveat: all five of these
> + parameters are linked together, so you must specify all five in
> + <filename>/etc/sysctl.conf</>. If you do not specify all five the
> + defaults that are hard-coded into <filename>/etc/rc</> will be used
> + instead. In all versions, you'll need to reboot to make

It'd be nice if it were as simple as that, but my experiments say that
it isn't.

The conclusion I had come to (which is in our docs now) is that the real
gotcha in the mix is that 10.4 ignores SHMMAX settings that aren't an
exact multiple of the page size.

regards, tom lane

I wrote:
> It'd be nice if it were as simple as that, but my experiments say that
> it isn't.

It took me a little bit to dig up the results of my previous
experiments, but here they are:
http://archives.postgresql.org/pgsql-hackers/2005-10/msg00684.php

AFAICT, setting all vs some of the variables isn't a factor; at least
not in 10.4 (couldn't say about 10.3 anymore).

regards, tom lane

The definitive answer can be found in the source code for the Darwin
kernel, xnu (links are below):

1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be
changed after the shared memory system is initialized

2. The shared memory system is initialized immediately after all 5
settings have been configured

3. The shmall setting must be a multiple of the page size (on both
10.3 and 10.4)

Here are excerpts from the sysctl_shminfo() function found in
sysv_shm.c:

[1]
/* Set the values only if shared memory is not initialised */
if (!shm_inited) {
if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t)))
!= 0) {
sysctl_shminfo_ret = error;
goto sysctl_shminfo_out;
}

[2]
/* Initialize only when all values are set */
if ((shminfo.shmmax != (user_ssize_t)-1) &&
(shminfo.shmmin != (user_ssize_t)-1) &&
(shminfo.shmmni != (user_ssize_t)-1) &&
(shminfo.shmseg != (user_ssize_t)-1) &&
(shminfo.shmall != (user_ssize_t)-1)) {
shminit(NULL);
}

[3]
(10.3)
if (arg1 == &shminfo.shmmax) {
if (shminfo.shmmax & PAGE_MASK) {
shminfo.shmmax = -1;
return(EINVAL);
}
}

(10.4)
if (arg1 == &shminfo.shmmax) {
if (shminfo.shmmax & PAGE_MASK_64) {
shminfo.shmmax = -1;
return(EINVAL);
}
}

PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned
long long)PAGE_MASK"

/etc/rc contains commands to set all 5 of the parameters. In 10.3.9
and later, you can place your own customized settings inside /etc/
sysctl.conf that will be executed before /etc/rc's commands. But if
you don't set them all, the shared memory system isn't initialized,
and /etc/rc will overwrite them.

Does that make sense?

In a nutshell, if you want to customize the shared memory settings,
you must:

(On 10.3.9 and later)
Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax
is a multiple of the page size (4096?). The page size is a runtime
variable that's set by kernel bootstrap mechanisms calling
vm_set_page_size() (I haven't tracked those down yet).

(On earlier systems)
Modify the desired settings inside /etc/rc, and be aware that OS
updates will overwrite your changes. If you upgrade to 10.3.9,
migrate to using /etc/sysctl.conf.

In my own OS X products that use a PostgreSQL server, I install an /
etc/sysctl.conf file on all servers. If installing on anything
earlier than 10.3, my installer modifies /etc/rc. That way, when the
customer upgrades their server to 10.3.9 and the /etc/rc file is
overwritten, the /etc/sysctl.conf file is already in place.

Thanks!

- Chris

Referenced source code:

Mac OS X 10.4.4:
http://www.opensource.apple.com/darwinsource/10.4.4.ppc/
xnu-792.6.56/bsd/kern/sysv_shm.c

Mac OS X 10.3.9:
http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/
bsd/kern/sysv_shm.c

Mac OS X 10.3:
http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/
kern/sysv_shm.c

Outstanding writeup! Mind if I post it in our knowledgebase at
http://www.pervasivepostgres.com/instantkb13/? (Unless someone has a
better idea of where this could go...)

On Sun, Feb 12, 2006 at 04:33:27PM -0500, Chris Campbell wrote:
> The definitive answer can be found in the source code for the Darwin
> kernel, xnu (links are below):
>
> 1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be
> changed after the shared memory system is initialized
>
> 2. The shared memory system is initialized immediately after all 5
> settings have been configured
>
> 3. The shmall setting must be a multiple of the page size (on both
> 10.3 and 10.4)
>
> Here are excerpts from the sysctl_shminfo() function found in
> sysv_shm.c:
>
> [1]
> /* Set the values only if shared memory is not initialised */
> if (!shm_inited) {
> if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t)))
> != 0) {
> sysctl_shminfo_ret = error;
> goto sysctl_shminfo_out;
> }
>
> [2]
> /* Initialize only when all values are set */
> if ((shminfo.shmmax != (user_ssize_t)-1) &&
> (shminfo.shmmin != (user_ssize_t)-1) &&
> (shminfo.shmmni != (user_ssize_t)-1) &&
> (shminfo.shmseg != (user_ssize_t)-1) &&
> (shminfo.shmall != (user_ssize_t)-1)) {
> shminit(NULL);
> }
>
>
> [3]
> (10.3)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> (10.4)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK_64) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned
> long long)PAGE_MASK"
>
>
> /etc/rc contains commands to set all 5 of the parameters. In 10.3.9
> and later, you can place your own customized settings inside /etc/
> sysctl.conf that will be executed before /etc/rc's commands. But if
> you don't set them all, the shared memory system isn't initialized,
> and /etc/rc will overwrite them.
>
> Does that make sense?
>
> In a nutshell, if you want to customize the shared memory settings,
> you must:
>
> (On 10.3.9 and later)
> Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax
> is a multiple of the page size (4096?). The page size is a runtime
> variable that's set by kernel bootstrap mechanisms calling
> vm_set_page_size() (I haven't tracked those down yet).
>
> (On earlier systems)
> Modify the desired settings inside /etc/rc, and be aware that OS
> updates will overwrite your changes. If you upgrade to 10.3.9,
> migrate to using /etc/sysctl.conf.
>
> In my own OS X products that use a PostgreSQL server, I install an /
> etc/sysctl.conf file on all servers. If installing on anything
> earlier than 10.3, my installer modifies /etc/rc. That way, when the
> customer upgrades their server to 10.3.9 and the /etc/rc file is
> overwritten, the /etc/sysctl.conf file is already in place.
>
> Thanks!
>
> - Chris
>
>
> Referenced source code:
>
> Mac OS X 10.4.4:
> http://www.opensource.apple.com/darwinsource/10.4.4.ppc/
> xnu-792.6.56/bsd/kern/sysv_shm.c
>
> Mac OS X 10.3.9:
> http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/
> bsd/kern/sysv_shm.c
>
> Mac OS X 10.3:
> http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/
> kern/sysv_shm.c
>

--
Jim C. Nasby, Sr. Engineering Consultant jnasby(at)pervasive(dot)com
Pervasive Software http://pervasive.com work: 512-231-6117
vcard: http://jim.nasby.net/pervasive.vcf cell: 512-569-9461

Jim C. Nasby wrote:
> Outstanding writeup! Mind if I post it in our knowledgebase at
> http://www.pervasivepostgres.com/instantkb13/? (Unless someone has a
> better idea of where this could go...)

Some of this information should be added to the documentation on how to
configure the shared memory limits for the different Unixen (16.4.1 of
the main docs).

-Neil

Great. I have applied the following documentation patch for CVS HEAD
and 8.1.X.

It seems you are saying that shmmax and shmall must be multiples of 4k,
and I added that to the docs. It previously only mentioned shmmax in
that regard.

---------------------------------------------------------------------------

Chris Campbell wrote:
> The definitive answer can be found in the source code for the Darwin
> kernel, xnu (links are below):
>
> 1. The shmmax, shmmin, shmmni, shmseg, and shmall settings cannot be
> changed after the shared memory system is initialized
>
> 2. The shared memory system is initialized immediately after all 5
> settings have been configured
>
> 3. The shmall setting must be a multiple of the page size (on both
> 10.3 and 10.4)
>
> Here are excerpts from the sysctl_shminfo() function found in
> sysv_shm.c:
>
> [1]
> /* Set the values only if shared memory is not initialised */
> if (!shm_inited) {
> if ((error = SYSCTL_IN(req, arg1, sizeof(user_ssize_t)))
> != 0) {
> sysctl_shminfo_ret = error;
> goto sysctl_shminfo_out;
> }
>
> [2]
> /* Initialize only when all values are set */
> if ((shminfo.shmmax != (user_ssize_t)-1) &&
> (shminfo.shmmin != (user_ssize_t)-1) &&
> (shminfo.shmmni != (user_ssize_t)-1) &&
> (shminfo.shmseg != (user_ssize_t)-1) &&
> (shminfo.shmall != (user_ssize_t)-1)) {
> shminit(NULL);
> }
>
>
> [3]
> (10.3)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> (10.4)
> if (arg1 == &shminfo.shmmax) {
> if (shminfo.shmmax & PAGE_MASK_64) {
> shminfo.shmmax = -1;
> return(EINVAL);
> }
> }
>
> PAGE_MASK is "(PAGE_SIZE - 1)", and PAGE_MASK_64 is simply "(unsigned
> long long)PAGE_MASK"
>
>
> /etc/rc contains commands to set all 5 of the parameters. In 10.3.9
> and later, you can place your own customized settings inside /etc/
> sysctl.conf that will be executed before /etc/rc's commands. But if
> you don't set them all, the shared memory system isn't initialized,
> and /etc/rc will overwrite them.
>
> Does that make sense?
>
> In a nutshell, if you want to customize the shared memory settings,
> you must:
>
> (On 10.3.9 and later)
> Set ALL 5 settings inside /etc/sysctl.conf, and make sure that shmmax
> is a multiple of the page size (4096?). The page size is a runtime
> variable that's set by kernel bootstrap mechanisms calling
> vm_set_page_size() (I haven't tracked those down yet).
>
> (On earlier systems)
> Modify the desired settings inside /etc/rc, and be aware that OS
> updates will overwrite your changes. If you upgrade to 10.3.9,
> migrate to using /etc/sysctl.conf.
>
> In my own OS X products that use a PostgreSQL server, I install an /
> etc/sysctl.conf file on all servers. If installing on anything
> earlier than 10.3, my installer modifies /etc/rc. That way, when the
> customer upgrades their server to 10.3.9 and the /etc/rc file is
> overwritten, the /etc/sysctl.conf file is already in place.
>
> Thanks!
>
> - Chris
>
>
> Referenced source code:
>
> Mac OS X 10.4.4:
> http://www.opensource.apple.com/darwinsource/10.4.4.ppc/
> xnu-792.6.56/bsd/kern/sysv_shm.c
>
> Mac OS X 10.3.9:
> http://www.opensource.apple.com/darwinsource/10.3.9/xnu-517.12.7/
> bsd/kern/sysv_shm.c
>
> Mac OS X 10.3:
> http://www.opensource.apple.com/darwinsource/10.3/xnu-517/bsd/
> kern/sysv_shm.c
>

--
Bruce Momjian http://candle.pha.pa.us
SRA OSS, Inc. http://www.sraoss.com

+ If your life is a hard drive, Christ can be your backup. +

On Mar 2, 2006, at 15:32, Bruce Momjian wrote:

> It seems you are saying that shmmax and shmall must be multiples of
> 4k,
> and I added that to the docs. It previously only mentioned shmmax in
> that regard.

That was a typo in my comment before the code (the nutshell
descriptions after the code are correct). Only shmmax needs to be
multiples of the page size (at least, that's how I interpret the
Darwin code).

Thanks!

- Chris

Patch applied. Thanks. Your documentation changes can be viewed in
five minutes using links on the developer's page,
http://www.postgresql.org/developer/testing.

---------------------------------------------------------------------------

Chris Campbell wrote:
> On Mar 2, 2006, at 15:32, Bruce Momjian wrote:
>
> > It seems you are saying that shmmax and shmall must be multiples of
> > 4k,
> > and I added that to the docs. It previously only mentioned shmmax in
> > that regard.
>
> That was a typo in my comment before the code (the nutshell
> descriptions after the code are correct). Only shmmax needs to be
> multiples of the page size (at least, that's how I interpret the
> Darwin code).
>
> Thanks!
>
> - Chris
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend
>

--
Bruce Momjian http://candle.pha.pa.us
SRA OSS, Inc. http://www.sraoss.com

+ If your life is a hard drive, Christ can be your backup. +