modperl-loss_of_shared_memory_in_parent_httpd_mlockall

This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.



To: Bill Marrs <bill@apocalypse.org>
From: Ed Grimm <ed@tgape.org>
Subject: RE: loss of shared memory in parent httpd
Date: Sat, 16 Mar 2002 16:16:45 -0500 (EST)


On Thu, 14 Mar 2002, Bill Marrs wrote:

> >It's copy-on-write.  The swap is a write-to-disk.
> >There's no such thing as sharing memory between one process on disk(/swap)
> >and another in memory.
> 
> agreed.   What's interesting is that if I turn swap off and back on again, 
> the sharing is restored!  So, now I'm tempted to run a crontab every 30 
> minutes that  turns the swap off and on again, just to keep the httpds 
> shared.  No Apache restart required!
>
> Seems like a crazy thing to do, though.
> 
> >You'll also want to look into tuning your paging algorithm.
> 
> Yeah... I'll look into it.  If I had a way to tell the kernel to never swap 
> out any httpd process, that would be a great solution.  The kernel is 
> making a bad choice here.  By swapping, it triggers more memory usage 
> because sharing removed on the httpd process group (thus multiplied)...
> 
> I've got MaxClients down to 8 now and it's still happening.  I think my 
> best course of action may be a crontab swap flusher.

I believe I have the answer...

The problem is that the parent httpd swaps, and any new children it
creates load the portion of memory that was swaped from swap, which does
not make it copy-on-write.  The really annoying thing - when memory gets
tight, the parent is the most likely httpd process to swap, because its
memory is 99% idle.  This issue aflicts Linux, Solaris, and a bunch of
other OSes.

The solution is mlockall(2), available under Linux, Solaris, and other
POSIX.1b compliant OSes.  I've not experimented with calling it from
perl, and I've not looked at Apache enough to consider patching it
there, but this system call, if your process is run as root, will
prevent any and all swapping of your process's memory.  If your process
is not run as root, it returns an error.


The reason turning off swap works is because it forces the memory from
the parent process that was swapped out to be swapped back in.  It will
not fix those processes that have been sired after the shared memory
loss, as of Linux 2.2.15 and Solaris 2.6.  (I have not checked since
then for behavior in this regard, nor have I checked on other OSes.)

===
To: Ed Grimm <ed@tgape.org>
From: Adi Fairbank <adi@adiraj.org>
Subject: Re: loss of shared memory in parent httpd
Date: Sat, 16 Mar 2002 14:36:43 -0800

Yes, this is my theory also.  I figured this out a while back, and started a
thread on this list, but since then haven't had enough time to investigate
it further.

The thread is here:

  http://mathforum.org/epigone/modperl/wherdtharvoi

which includes some helpful hints from Doug on how to call mlockall() from
the mod_perl parent process.

HTH.. I'm very curious to know if this works.

-Adi

===

To: Ed Grimm <ed@tgape.org>
From: Bill Marrs <bill@apocalypse.org>
Subject: RE: loss of shared memory in parent httpd
Date: Sat, 16 Mar 2002 22:29:21 -0500

>The reason turning off swap works is because it forces the memory from
>the parent process that was swapped out to be swapped back in.  It will
>not fix those processes that have been sired after the shared memory
>loss, as of Linux 2.2.15 and Solaris 2.6.  (I have not checked since
>then for behavior in this regard, nor have I checked on other OSes.)

In my case, I'm using Linux 2.4.17, when I turn off swap and turn it back 
on again, it restores the shared memory of both the parent and the children 
Apache processes.

This seems counter-intuitive, as it would seem the kernel memory manager 
would have to bend over backwards to accomplish this re-binding of the 
swapped-out shared memory pages.

Thus, it leads ones to wonder if some of our assumptions or tools used to 
monitor memory are inaccurate or we're misinterpreting them.

===
To: Bill Marrs <bill@apocalypse.org>
From: Ged Haywood <ged@www2.jubileegroup.co.uk>
Subject: RE: loss of shared memory in parent httpd
Date: Sun, 17 Mar 2002 09:53:39 +0000 (GMT)

Hi all,

On Sat, 16 Mar 2002, Bill Marrs wrote:

> leads ones to wonder if some of our assumptions or tools used to 
> monitor memory are inaccurate or we're misinterpreting them.

Well 'top' on Linux is rubbish for sure.

===

To: Ed Grimm <ed@tgape.org>
From: Eric Frazier <ef@kwinternet.com>
Subject: RE: loss of shared memory in parent httpd
Date: Sun, 17 Mar 2002 11:04:08 -0500

I had hoped that FreeBSD would be immune, but it seems not. I have been
bashing it with http_load and all of a sudden(after a LOT of bashing and
swaping)  all of my processes had zero shared. It did take me days of
fiddling to run into this though. 

===

To: Elizabeth Mattijsen <liz@dijkmat.nl>
From: Eric Frazier <ef@kwinternet.com>
Subject: Re: loss of shared memory in parent httpd (2)
Date: Mon, 18 Mar 2002 00:07:09 -0500

Hi,

This may be totaly ignorate crap, but I noticed this when I was reading the
ps man page on BSD 4.5 about sys/proc.h flags

This one I noticed.. 

P_SYSTEM       0x00200        System proc: no sigs, stats or swapping


Could this mean what I think it means? That a process with this flag set,
won't be swaped out?? 

===

To: Eric Frazier <ef@kwinternet.com>
From: Stas Bekman <stas@stason.org>
Subject: Re: loss of shared memory in parent httpd (2)
Date: Mon, 18 Mar 2002 17:23:42 +0800

Eric Frazier wrote:
> Hi,
> 
> This may be totaly ignorate crap, but I noticed this when I was reading the
> ps man page on BSD 4.5 about sys/proc.h flags
> 
> This one I noticed.. 
> 
> P_SYSTEM       0x00200        System proc: no sigs, stats or swapping
> 
> 
> Could this mean what I think it means? That a process with this flag set,
> won't be swaped out?? 

I've spent some time with our friend google and here is what I came up 
with (it's hard to search for things which can be called in many 
different ways):

I've searched for P_SYSTEM and it seems that it's a *BSD thing and also 
when you set it you don't get sigs delivered. e.g. see the discussion here:
http://mail-index.netbsd.org/tech-kern/1998/09/08/0004.html

There is also: madvise(2) - give advice about use of memory
Has anybody tried to use it? Can this help? There is some discussion here:
http://lists.insecure.org/linux-kernel/2001/Oct/0877.html

Here is another observation and explanation of the swapping/mem 
unsharing phenomena on linux.
http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0324.html
http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0307.html

Finally, apparently it's relatively easy to patch the linux kernel to 
enabling mlock for non-root processes:
http://www.uwsg.iu.edu/hypermail/linux/kernel/9608.2/0280.html

===


the rest of The Pile (a partial mailing list archive)

doom@kzsu.stanford.edu