dynamic shared memory: wherein I am punished for good intentions

Since, as has been previously discussed in this forum on multiple
occasions [citation needed], the default System V shared memory limits
are absurdly low on many systems, the dynamic shared memory patch
defaults to POSIX shared memory, which has often been touted as a
superior alternative [citation needed]. Unfortunately, the buildfarm
isn't entirely happy with this decision. On buildfarm member anole
(HP-UX B.11.31), allocation of dynamic shared memory fails with a
"Permission denied" error, and on smew (Debian GNU/Linux 6.0), it
fails with "Function not implemented", which according to a forum
post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
on that box.

What shall we do about this? I see a few options.

(1) Define the issue as "not our problem". IOW, as of now, if you
want to use PostgreSQL, you've got to either make POSIX shared memory
work on your machine, or change the GUC that selects the type of
dynamic shared memory used.

(2) Default to using System V shared memory. If people want POSIX
shared memory, let them change the default.

(3) Add a new setting that auto-probes for a type of shared memory
that works. Try POSIX first, then System V. Maybe even fall back to
mmap'd files if neither of those works.

(4) Remove the option to use POSIX shared memory. System V FTW!

After some consideration, I think my vote is for option #2. Option #1
seems too user-hostile, especially for a facility that has no in-core
users yet, though I can imagine we might want to go that way
eventually, especially if we at some point try to dump System V shared
memory altogether, as has been proposed. Option #4 seems sad; we know
that System V shared memory limits are a problem for plenty of people,
so it'd be a shame not to have a way to use the POSIX facilities if
they're available. Option #3 is fine as far as it goes, but it adds a
significant amount of complexity I'd rather not deal with.

Other votes? Other ideas?

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

[1] http://forums.justlinux.com/showthread.php?142429-shm_open-problem

Robert Haas wrote
> Unfortunately, the buildfarm
> isn't entirely happy with this decision. On buildfarm member anole
> (HP-UX B.11.31), allocation of dynamic shared memory fails with a
> "Permission denied" error, and on smew (Debian GNU/Linux 6.0), it
> fails with "Function not implemented", which according to a forum
> post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
> on that box.
>
> What shall we do about this? I see a few options.

Is this something that rightly falls into being a distro/package specific
setting? If so then the first goal should be to ensure the maximum number
of successful basic installation scenarios - namely someone installing
PostgreSQL and connect to the running postgres database without encountering
an error.

As a default I would presume the current System V behavior is sufficient to
accomplish this goal. If package maintainers can then guarantee that
changing the default will improve the user experience they should be
supported and encouraged to do so but if they are at all unsure they should
leave the default in place.

As long as a new user is able to get a running database on their machine
if/when they run up against the low defaults of System V memory they will at
least be able to focus on that single problem as opposed to having a failed
initial install and being unsure exactly what they may have done wrong.

Thus option # 2 seems sufficient. I do think that having some kind of
shared-memory-manager utility could have value but I'd rather see that be a
standalone utility as opposed to something magical done inside the bowels of
the database. While probably harder to code and learn such a utility would
provide for a much greater UX if implemented well.

David J.

--
View this message in context: http://postgresql.1045698.n5.nabble.com/dynamic-shared-memory-wherein-I-am-punished-for-good-intentions-tp5774055p5774080.html
Sent from the PostgreSQL - hackers mailing list archive at Nabble.com.

On Thu, Oct 10, 2013 at 1:13 PM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> (1) Define the issue as "not our problem". IOW, as of now, if you
> want to use PostgreSQL, you've got to either make POSIX shared memory
> work on your machine, or change the GUC that selects the type of
> dynamic shared memory used.
>
> (2) Default to using System V shared memory. If people want POSIX
> shared memory, let them change the default.
>
> (3) Add a new setting that auto-probes for a type of shared memory
> that works. Try POSIX first, then System V. Maybe even fall back to
> mmap'd files if neither of those works.
>
> (4) Remove the option to use POSIX shared memory. System V FTW!
>
> After some consideration, I think my vote is for option #2. Option #1
> seems too user-hostile, especially for a facility that has no in-core
> users yet, though I can imagine we might want to go that way
> eventually, especially if we at some point try to dump System V shared
> memory altogether, as has been proposed. Option #4 seems sad; we know
> that System V shared memory limits are a problem for plenty of people,
> so it'd be a shame not to have a way to use the POSIX facilities if
> they're available. Option #3 is fine as far as it goes, but it adds a
> significant amount of complexity I'd rather not deal with.
>
> Other votes? Other ideas?

I believe option 2 is not only good for now, but also a necessary
previous step in the way to option 3, which I believe should be the
goal.

So, ship a version with option 2, then implement 3, and make it the
default when enough people (using option 2) have successfully tested
pg's implementation of POSIX shared memory.

On 10/10/2013 12:13 PM, Robert Haas wrote:
> Since, as has been previously discussed in this forum on multiple
> occasions [citation needed], the default System V shared memory limits
> are absurdly low on many systems, the dynamic shared memory patch
> defaults to POSIX shared memory, which has often been touted as a
> superior alternative [citation needed]. Unfortunately, the buildfarm
> isn't entirely happy with this decision. On buildfarm member anole
> (HP-UX B.11.31), allocation of dynamic shared memory fails with a
> "Permission denied" error, and on smew (Debian GNU/Linux 6.0), it
> fails with "Function not implemented", which according to a forum
> post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
> on that box.
>
> What shall we do about this? I see a few options.
>
> (1) Define the issue as "not our problem". IOW, as of now, if you
> want to use PostgreSQL, you've got to either make POSIX shared memory
> work on your machine, or change the GUC that selects the type of
> dynamic shared memory used.
>
> (2) Default to using System V shared memory. If people want POSIX
> shared memory, let them change the default.
>
> (3) Add a new setting that auto-probes for a type of shared memory
> that works. Try POSIX first, then System V. Maybe even fall back to
> mmap'd files if neither of those works.
>
> (4) Remove the option to use POSIX shared memory. System V FTW!
>
> After some consideration, I think my vote is for option #2. Option #1
> seems too user-hostile, especially for a facility that has no in-core
> users yet, though I can imagine we might want to go that way
> eventually, especially if we at some point try to dump System V shared
> memory altogether, as has been proposed. Option #4 seems sad; we know
> that System V shared memory limits are a problem for plenty of people,
> so it'd be a shame not to have a way to use the POSIX facilities if
> they're available. Option #3 is fine as far as it goes, but it adds a
> significant amount of complexity I'd rather not deal with.
>
> Other votes? Other ideas?
>

5) test and set it in initdb.

cheers

andrew

On Thu, Oct 10, 2013 at 2:21 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>> Other votes? Other ideas?
>
> 5) test and set it in initdb.

Are you advocating for that option, or just calling out that it's
possible? I'd say that's closely related to option #3, except at
initdb time rather than run-time - and it might be preferable to #3
for some of the same reasons discussed on the thread about tuning
work_mem, namely, that having it change from one postmaster lifetime
to the next might lead to user astonishment.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On Thu, Oct 10, 2013 at 9:13 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> (2) Default to using System V shared memory. If people want POSIX
> shared memory, let them change the default.

> After some consideration, I think my vote is for option #2.

Wouldn't that become the call of packagers? Wasn't there already some
reason why it was advantageous for FreeBSD to continue to use System V
shared memory?

--
Peter Geoghegan

On Thu, Oct 10, 2013 at 2:36 PM, Peter Geoghegan <pg(at)heroku(dot)com> wrote:
> On Thu, Oct 10, 2013 at 9:13 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>> (2) Default to using System V shared memory. If people want POSIX
>> shared memory, let them change the default.
>
>> After some consideration, I think my vote is for option #2.
>
> Wouldn't that become the call of packagers?

Packagers can certainly override whatever we do, but we still need to
make the buildfarm green again.

> Wasn't there already some
> reason why it was advantageous for FreeBSD to continue to use System V
> shared memory?

Yes, but this code doesn't affect the main shared memory segment, so I
think that's sort of a separate point.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

On 10/10/2013 02:35 PM, Robert Haas wrote:
> On Thu, Oct 10, 2013 at 2:21 PM, Andrew Dunstan <andrew(at)dunslane(dot)net> wrote:
>>> Other votes? Other ideas?
>> 5) test and set it in initdb.
> Are you advocating for that option, or just calling out that it's
> possible? I'd say that's closely related to option #3, except at
> initdb time rather than run-time - and it might be preferable to #3
> for some of the same reasons discussed on the thread about tuning
> work_mem, namely, that having it change from one postmaster lifetime
> to the next might lead to user astonishment.

Mainly just to throw it into the mix, But like you I think it's probably
a better option than #3 for the reason you give. It also has the
advantage of keeping any probing code out of the backend.

cheers

andrew

* Robert Haas (robertmhaas(at)gmail(dot)com) wrote:
> On Thu, Oct 10, 2013 at 2:36 PM, Peter Geoghegan <pg(at)heroku(dot)com> wrote:
> > On Thu, Oct 10, 2013 at 9:13 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> >> (2) Default to using System V shared memory. If people want POSIX
> >> shared memory, let them change the default.
> >
> >> After some consideration, I think my vote is for option #2.
> >
> > Wouldn't that become the call of packagers?
>
> Packagers can certainly override whatever we do, but we still need to
> make the buildfarm green again.

While I agree that making the buildfarm green is valuable, I really
wonder about a system where /dev/shm is busted.

Personally, I like Andrew's suggestion to test and set accordingly, with
the default being POSIX if it's available and a fall-back to SysV (maybe
with a warning..). Going back to the situation where our default set-up
limits us to the ridiculously small SysV value would *really* suck; even
if we don't have any users today, we're certainly going to have some
soon and I don't think they'll be happy with a 24MB (or whatever) limit.

Thanks,

Stephen

On Thu, Oct 10, 2013 at 11:13 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
> Since, as has been previously discussed in this forum on multiple
> occasions [citation needed], the default System V shared memory limits
> are absurdly low on many systems, the dynamic shared memory patch
> defaults to POSIX shared memory, which has often been touted as a
> superior alternative [citation needed]. Unfortunately, the buildfarm
> isn't entirely happy with this decision. On buildfarm member anole
> (HP-UX B.11.31), allocation of dynamic shared memory fails with a
> "Permission denied" error, and on smew (Debian GNU/Linux 6.0), it
> fails with "Function not implemented", which according to a forum
> post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
> on that box.
>
> What shall we do about this? I see a few options.
>
> (1) Define the issue as "not our problem". IOW, as of now, if you
> want to use PostgreSQL, you've got to either make POSIX shared memory
> work on your machine, or change the GUC that selects the type of
> dynamic shared memory used.
>
> (2) Default to using System V shared memory. If people want POSIX
> shared memory, let them change the default.

Doesn't #2 negate all advantages of this effort? Bringing sysv
management back on the table seems like a giant step backwards -- or
am I missing something?

http://www.postgresql.org/docs/9.3/interactive/kernel-resources.html#SYSVIPC

merlin

On 10/10/2013 02:45 PM, Robert Haas wrote:
> On Thu, Oct 10, 2013 at 2:36 PM, Peter Geoghegan <pg(at)heroku(dot)com> wrote:
>> On Thu, Oct 10, 2013 at 9:13 AM, Robert Haas <robertmhaas(at)gmail(dot)com> wrote:
>>> (2) Default to using System V shared memory. If people want POSIX
>>> shared memory, let them change the default.
>>> After some consideration, I think my vote is for option #2.
>> Wouldn't that become the call of packagers?
> Packagers can certainly override whatever we do, but we still need to
> make the buildfarm green again.
>
>

I really dislike throwing things over the wall to packagers like this,
anyway. Quite apart from anything else, not everyone uses pre-built
packages, and we should make things as as easy as possible for those who
don't, too.

cheers

andrew

On Thu, Oct 10, 2013 at 12:13:20PM -0400, Robert Haas wrote:
> Since, as has been previously discussed in this forum on multiple
> occasions [citation needed], the default System V shared memory limits
> are absurdly low on many systems, the dynamic shared memory patch
> defaults to POSIX shared memory, which has often been touted as a
> superior alternative [citation needed]. Unfortunately, the buildfarm
> isn't entirely happy with this decision. On buildfarm member anole
> (HP-UX B.11.31), allocation of dynamic shared memory fails with a
> "Permission denied" error, and on smew (Debian GNU/Linux 6.0), it
> fails with "Function not implemented", which according to a forum
> post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
> on that box.
>
> What shall we do about this? I see a few options.
>
> (1) Define the issue as "not our problem". IOW, as of now, if you
> want to use PostgreSQL, you've got to either make POSIX shared memory
> work on your machine, or change the GUC that selects the type of
> dynamic shared memory used.

+1 for this.

Cheers,
David.
--
David Fetter <david(at)fetter(dot)org> http://fetter.org/
Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter
Skype: davidfetter XMPP: david(dot)fetter(at)gmail(dot)com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

On Thu, Oct 10, 2013 at 4:00 PM, Merlin Moncure <mmoncure(at)gmail(dot)com> wrote:
>> (2) Default to using System V shared memory. If people want POSIX
>> shared memory, let them change the default.
>
> Doesn't #2 negate all advantages of this effort? Bringing sysv
> management back on the table seems like a giant step backwards -- or
> am I missing something?

Not unless there's no difference between "the default" and "the only option".

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

Robert,

>> Doesn't #2 negate all advantages of this effort? Bringing sysv
>> management back on the table seems like a giant step backwards -- or
>> am I missing something?
>
> Not unless there's no difference between "the default" and "the only option".

Well, per our earlier discussion about "the first 15 minutes", there
actually isn't a difference.

We got rid of SHMMAX for the majority of our users for 9.3. We should
NOT revert that just so we can support older platforms -- especially
since, if anything, Kernel.org is going to cripple SysV support even
further in the future. The platforms where it does work represent the
vast majority of our users, and are only on the increase.

I can only see two reasonable alternatives:

This one:
> (3) Add a new setting that auto-probes for a type of shared memory
> that works. Try POSIX first, then System V. Maybe even fall back to
> mmap'd files if neither of those works.

Or:

(5) Default to POSIX, and allow for SysV as a compile-time option for
platforms with poor POSIX memory support.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On 2013-10-10 12:13:20 -0400, Robert Haas wrote:
> and on smew (Debian GNU/Linux 6.0), it
> fails with "Function not implemented", which according to a forum
> post[1] I found probably indicates that /dev/shm doesn't mount a tmpfs
> on that box.

It would be nice to get confirmed what the reason for that is - afaik
debian has mounted /dev/shm for ages. The most likely issue I can see is
an incorrectly setup chroot.
If it's just that I wouldn't be too concerned about it. That's a
scenario that won't happen to many users and relatively easily fixed

Greetings,

Andres Freund

--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

On Thu, Oct 10, 2013 at 5:14 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>>> Doesn't #2 negate all advantages of this effort? Bringing sysv
>>> management back on the table seems like a giant step backwards -- or
>>> am I missing something?
>>
>> Not unless there's no difference between "the default" and "the only option".
>
> Well, per our earlier discussion about "the first 15 minutes", there
> actually isn't a difference.

Harsh. :-)

> I can only see two reasonable alternatives:
>
> This one:
>> (3) Add a new setting that auto-probes for a type of shared memory
>> that works. Try POSIX first, then System V. Maybe even fall back to
>> mmap'd files if neither of those works.
>
> Or:
>
> (5) Default to POSIX, and allow for SysV as a compile-time option for
> platforms with poor POSIX memory support.

OK, I did #5. Let's see how that works.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

>> (5) Default to POSIX, and allow for SysV as a compile-time option for
>> platforms with poor POSIX memory support.
>
> OK, I did #5. Let's see how that works.

Andrew pointed out upthread that, since platforms are unlikely to change
what they support dynamically, we could do this at initdb time instead.

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

On Thu, Oct 10, 2013 at 7:59 PM, Josh Berkus <josh(at)agliodbs(dot)com> wrote:
>>> (5) Default to POSIX, and allow for SysV as a compile-time option for
>>> platforms with poor POSIX memory support.
>>
>> OK, I did #5. Let's see how that works.
>
> Andrew pointed out upthread that, since platforms are unlikely to change
> what they support dynamically, we could do this at initdb time instead.

Oh, sorry, that's what I did. I missed the fact that your #5 and his
#5 were different.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company