Linux2.6 overcommit behaviour

Hi all,

Following is from Documentation/vm/overcommit-accounting
-------------
2 - (NEW) strict overcommit. The total address space commit
for the system is not permitted to exceed swap + a
configurable percentage (default is 50) of physical RAM.
Depending on the percentage you use, in most situations
this means a process will not be killed while accessing
pages but will receive errors on memory allocation as
appropriate.
-------------

Looks like it's been taken care once for all.

Shridhar

Yes, in 2.6, which is not yet released. Even after it is released I
expect it to take some time to bed down and make its way into vendor
releases, if the history of 2.4 is anything to go by.

Incidentally, it looks to me like it is only in 2.6 if your kernel is
built with CONFIG_SECURITY, which I expect most will be.

andrew

Shridhar Daithankar wrote:

>Hi all,
>
>Following is from Documentation/vm/overcommit-accounting
>-------------
>2 - (NEW) strict overcommit. The total address space commit
> for the system is not permitted to exceed swap + a
> configurable percentage (default is 50) of physical RAM.
> Depending on the percentage you use, in most situations
> this means a process will not be killed while accessing
> pages but will receive errors on memory allocation as
> appropriate.
>-------------
>
>Looks like it's been taken care once for all.
>
> Shridhar
>
>
>---------------------------(end of broadcast)---------------------------
>TIP 4: Don't 'kill -9' the postmaster
>
>
>

On Thursday 28 August 2003 17:11, Andrew Dunstan wrote:
> Yes, in 2.6, which is not yet released. Even after it is released I
> expect it to take some time to bed down and make its way into vendor
> releases, if the history of 2.4 is anything to go by.

Better late than never. I sincerely hope that it lives by the expectations..

> Incidentally, it looks to me like it is only in 2.6 if your kernel is
> built with CONFIG_SECURITY, which I expect most will be.

No. It isn't. Just checked the config file. It isn't set. But I haven't tried
setting overcommit as well.

BTW, what is the sway of switching disk IO scheduler in 2.6? Could not find
any references to sysctl switching. Andrew Morton's TODO list still list it
as TODO.

So am I stuck with whatever disk scheduler it's bundled with?

Shridhar

I take that last remark back - it is there whether or not
CONFIG_SECURITY is defined or not. The code is in 2 places - ugh.

andrew

Andrew Dunstan wrote:

>
> Yes, in 2.6, which is not yet released. Even after it is released I
> expect it to take some time to bed down and make its way into vendor
> releases, if the history of 2.4 is anything to go by.
>
> Incidentally, it looks to me like it is only in 2.6 if your kernel is
> built with CONFIG_SECURITY, which I expect most will be.
>
> andrew
>
> Shridhar Daithankar wrote:
>
>> Hi all,
>>
>> Following is from Documentation/vm/overcommit-accounting
>> -------------
>> 2 - (NEW) strict overcommit. The total address space commit
>> for the system is not permitted to exceed swap + a
>> configurable percentage (default is 50) of physical RAM.
>> Depending on the percentage you use, in most situations
>> this means a process will not be killed while accessing
>> pages but will receive errors on memory allocation as
>> appropriate.
>> -------------
>>
>> Looks like it's been taken care once for all.
>>
>> Shridhar
>>
>>
>> ---------------------------(end of broadcast)---------------------------
>> TIP 4: Don't 'kill -9' the postmaster
>>
>>
>>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 1: subscribe and unsubscribe commands go to majordomo(at)postgresql(dot)org
>

On Thu, Aug 28, 2003 at 05:22:36PM +0530, Shridhar Daithankar wrote:
> BTW, what is the sway of switching disk IO scheduler in 2.6? Could not find
> any references to sysctl switching. Andrew Morton's TODO list still list it
> as TODO.

Sorry, I was mistaken: you can switch I/O schedulers by specifying a
flag at boot-time:

http://marc.theaimsgroup.com/?l=linux-kernel&m=105743728122143&w=2

-Neil

Shridhar Daithankar wrote:
> Hi all,
>
> Following is from Documentation/vm/overcommit-accounting
> -------------
> 2 - (NEW) strict overcommit. The total address space commit
> for the system is not permitted to exceed swap + a
> configurable percentage (default is 50) of physical RAM.
> Depending on the percentage you use, in most situations
> this means a process will not be killed while accessing
> pages but will receive errors on memory allocation as
> appropriate.

It is strange to choose 50% of RAM plus swap (what if your spam is
small). I thought it would be 100% of RAM plus the swap that exceeds RAM
size.

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

You need to allow some head room, I should think. Actually, the
equivalent of the previously discussed paranoid mode would be to set the
percentage to 0, i.e. ensure you can put every page in swap. If you say
50% then the chances of your running out of room are exceedingly small.

andrew

Bruce Momjian wrote:

>Shridhar Daithankar wrote:
>
>
>>Hi all,
>>
>>Following is from Documentation/vm/overcommit-accounting
>>-------------
>>2 - (NEW) strict overcommit. The total address space commit
>> for the system is not permitted to exceed swap + a
>> configurable percentage (default is 50) of physical RAM.
>> Depending on the percentage you use, in most situations
>> this means a process will not be killed while accessing
>> pages but will receive errors on memory allocation as
>> appropriate.
>>
>>
>
>It is strange to choose 50% of RAM plus swap (what if your spam is
>small). I thought it would be 100% of RAM plus the swap that exceeds RAM
>size.
>
>
>

Bruce Momjian wrote:

>Shridhar Daithankar wrote:
>
>
>>Hi all,
>>
>>Following is from Documentation/vm/overcommit-accounting
>>-------------
>>2 - (NEW) strict overcommit. The total address space commit
>> for the system is not permitted to exceed swap + a
>> configurable percentage (default is 50) of physical RAM.
>> Depending on the percentage you use, in most situations
>> this means a process will not be killed while accessing
>> pages but will receive errors on memory allocation as
>> appropriate.
>>
>>
>
>It is strange to choose 50% of RAM plus swap (what if your spam is
>small). I thought it would be 100% of RAM plus the swap that exceeds RAM
>size.
>
>
Linux doesn't release the swap file page when a page is read back: If a
page is only read by the user space app, then the swapped out page
remains valid, and thus the kernel can skip the write to disk on the
next swapout. Thus if you are paranoid, you must limit the total address
space to the size of your swap files.
If your swap space (your wrote "spam" - I assume a typo) is small, then
you'll run into problems. It's recommended that your swap space should
be 2*physical memory. I assume that many oom killer reports are from
system with too small swap files, and then an updatedb run pushes the
system into oom.

--
Manfred

Manfred Spraul wrote:

>>
>> It is strange to choose 50% of RAM plus swap (what if your spam is
>> small). I thought it would be 100% of RAM plus the swap that exceeds RAM
>> size.
>>
>>
> Linux doesn't release the swap file page when a page is read back: If
> a page is only read by the user space app, then the swapped out page
> remains valid, and thus the kernel can skip the write to disk on the
> next swapout. Thus if you are paranoid, you must limit the total
> address space to the size of your swap files.
> If your swap space (your wrote "spam" - I assume a typo) is small,
> then you'll run into problems. It's recommended that your swap space
> should be 2*physical memory. I assume that many oom killer reports are
> from system with too small swap files, and then an updatedb run pushes
> the system into oom.

I believe that the swap slot can be subsequently freed, though. In
theory your available virtual memory should be (almost) RAM+swap. In
practice, Linux can run too close to that limit, (or way over it if you
turn the checks off). But restricting the maximum possible pages to
RAM/2 + swap should normally be fine. IANAKH, though.

Also note that the truly bad thing about the OOM killer is that it can
affect a process that is not making any new memory demands at all.

cheers

andrew

Andrew Dunstan wrote:
> I believe that the swap slot can be subsequently freed, though. In
> theory your available virtual memory should be (almost) RAM+swap. In
> practice, Linux can run too close to that limit, (or way over it if you
> turn the checks off). But restricting the maximum possible pages to
> RAM/2 + swap should normally be fine. IANAKH, though.
>
> Also note that the truly bad thing about the OOM killer is that it can
> affect a process that is not making any new memory demands at all.

How does the OOM killer kill processes, kill -9 or kill -1 and wait?

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073

Bruce Momjian wrote:

>Andrew Dunstan wrote:
>
>
>>I believe that the swap slot can be subsequently freed, though. In
>>theory your available virtual memory should be (almost) RAM+swap. In
>>practice, Linux can run too close to that limit, (or way over it if you
>>turn the checks off). But restricting the maximum possible pages to
>>RAM/2 + swap should normally be fine. IANAKH, though.
>>
>>Also note that the truly bad thing about the OOM killer is that it can
>>affect a process that is not making any new memory demands at all.
>>
>>
>
>How does the OOM killer kill processes, kill -9 or kill -1 and wait?
>
>

It sends a SIGKILL (9) unless the process is doing raw io, in which case
it sends SIGTERM (15). It can't really wait - at this stage the kernel
is in trouble - it can either kill processes or panic. The whole idea of
strict accounting is not to let it get to this stage in the first place.

see mm/oom_kill.c

cheers

andrew

Andrew Dunstan wrote:
> It sends a SIGKILL (9) unless the process is doing raw io, in which case
> it sends SIGTERM (15). It can't really wait - at this stage the kernel
> is in trouble - it can either kill processes or panic. The whole idea of
> strict accounting is not to let it get to this stage in the first place.
>
> see mm/oom_kill.c

So it brings down all the backends --- I see. :-(

--
Bruce Momjian | http://candle.pha.pa.us
pgman(at)candle(dot)pha(dot)pa(dot)us | (610) 359-1001
+ If your life is a hard drive, | 13 Roberts Road
+ Christ can be your backup. | Newtown Square, Pennsylvania 19073