This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
To: Bill Marrs <bill@apocalypse.org> From: Ed Grimm <ed@tgape.org> Subject: RE: loss of shared memory in parent httpd Date: Sat, 16 Mar 2002 16:16:45 -0500 (EST) On Thu, 14 Mar 2002, Bill Marrs wrote: > >It's copy-on-write. The swap is a write-to-disk. > >There's no such thing as sharing memory between one process on disk(/swap) > >and another in memory. > > agreed. What's interesting is that if I turn swap off and back on again, > the sharing is restored! So, now I'm tempted to run a crontab every 30 > minutes that turns the swap off and on again, just to keep the httpds > shared. No Apache restart required! > > Seems like a crazy thing to do, though. > > >You'll also want to look into tuning your paging algorithm. > > Yeah... I'll look into it. If I had a way to tell the kernel to never swap > out any httpd process, that would be a great solution. The kernel is > making a bad choice here. By swapping, it triggers more memory usage > because sharing removed on the httpd process group (thus multiplied)... > > I've got MaxClients down to 8 now and it's still happening. I think my > best course of action may be a crontab swap flusher. I believe I have the answer... The problem is that the parent httpd swaps, and any new children it creates load the portion of memory that was swaped from swap, which does not make it copy-on-write. The really annoying thing - when memory gets tight, the parent is the most likely httpd process to swap, because its memory is 99% idle. This issue aflicts Linux, Solaris, and a bunch of other OSes. The solution is mlockall(2), available under Linux, Solaris, and other POSIX.1b compliant OSes. I've not experimented with calling it from perl, and I've not looked at Apache enough to consider patching it there, but this system call, if your process is run as root, will prevent any and all swapping of your process's memory. If your process is not run as root, it returns an error. The reason turning off swap works is because it forces the memory from the parent process that was swapped out to be swapped back in. It will not fix those processes that have been sired after the shared memory loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since then for behavior in this regard, nor have I checked on other OSes.) === To: Ed Grimm <ed@tgape.org> From: Adi Fairbank <adi@adiraj.org> Subject: Re: loss of shared memory in parent httpd Date: Sat, 16 Mar 2002 14:36:43 -0800 Yes, this is my theory also. I figured this out a while back, and started a thread on this list, but since then haven't had enough time to investigate it further. The thread is here: http://mathforum.org/epigone/modperl/wherdtharvoi which includes some helpful hints from Doug on how to call mlockall() from the mod_perl parent process. HTH.. I'm very curious to know if this works. -Adi === To: Ed Grimm <ed@tgape.org> From: Bill Marrs <bill@apocalypse.org> Subject: RE: loss of shared memory in parent httpd Date: Sat, 16 Mar 2002 22:29:21 -0500 >The reason turning off swap works is because it forces the memory from >the parent process that was swapped out to be swapped back in. It will >not fix those processes that have been sired after the shared memory >loss, as of Linux 2.2.15 and Solaris 2.6. (I have not checked since >then for behavior in this regard, nor have I checked on other OSes.) In my case, I'm using Linux 2.4.17, when I turn off swap and turn it back on again, it restores the shared memory of both the parent and the children Apache processes. This seems counter-intuitive, as it would seem the kernel memory manager would have to bend over backwards to accomplish this re-binding of the swapped-out shared memory pages. Thus, it leads ones to wonder if some of our assumptions or tools used to monitor memory are inaccurate or we're misinterpreting them. === To: Bill Marrs <bill@apocalypse.org> From: Ged Haywood <ged@www2.jubileegroup.co.uk> Subject: RE: loss of shared memory in parent httpd Date: Sun, 17 Mar 2002 09:53:39 +0000 (GMT) Hi all, On Sat, 16 Mar 2002, Bill Marrs wrote: > leads ones to wonder if some of our assumptions or tools used to > monitor memory are inaccurate or we're misinterpreting them. Well 'top' on Linux is rubbish for sure. === To: Ed Grimm <ed@tgape.org> From: Eric Frazier <ef@kwinternet.com> Subject: RE: loss of shared memory in parent httpd Date: Sun, 17 Mar 2002 11:04:08 -0500 I had hoped that FreeBSD would be immune, but it seems not. I have been bashing it with http_load and all of a sudden(after a LOT of bashing and swaping) all of my processes had zero shared. It did take me days of fiddling to run into this though. === To: Elizabeth Mattijsen <liz@dijkmat.nl> From: Eric Frazier <ef@kwinternet.com> Subject: Re: loss of shared memory in parent httpd (2) Date: Mon, 18 Mar 2002 00:07:09 -0500 Hi, This may be totaly ignorate crap, but I noticed this when I was reading the ps man page on BSD 4.5 about sys/proc.h flags This one I noticed.. P_SYSTEM 0x00200 System proc: no sigs, stats or swapping Could this mean what I think it means? That a process with this flag set, won't be swaped out?? === To: Eric Frazier <ef@kwinternet.com> From: Stas Bekman <stas@stason.org> Subject: Re: loss of shared memory in parent httpd (2) Date: Mon, 18 Mar 2002 17:23:42 +0800 Eric Frazier wrote: > Hi, > > This may be totaly ignorate crap, but I noticed this when I was reading the > ps man page on BSD 4.5 about sys/proc.h flags > > This one I noticed.. > > P_SYSTEM 0x00200 System proc: no sigs, stats or swapping > > > Could this mean what I think it means? That a process with this flag set, > won't be swaped out?? I've spent some time with our friend google and here is what I came up with (it's hard to search for things which can be called in many different ways): I've searched for P_SYSTEM and it seems that it's a *BSD thing and also when you set it you don't get sigs delivered. e.g. see the discussion here: http://mail-index.netbsd.org/tech-kern/1998/09/08/0004.html There is also: madvise(2) - give advice about use of memory Has anybody tried to use it? Can this help? There is some discussion here: http://lists.insecure.org/linux-kernel/2001/Oct/0877.html Here is another observation and explanation of the swapping/mem unsharing phenomena on linux. http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0324.html http://www.uwsg.iu.edu/hypermail/linux/kernel/0110.3/0307.html Finally, apparently it's relatively easy to patch the linux kernel to enabling mlock for non-root processes: http://www.uwsg.iu.edu/hypermail/linux/kernel/9608.2/0280.html ===