This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Date: Wed, 04 Oct 2000 11:53:28 -0700
From: ed phillips <artlore@sirius.com>
To: "David E. Wheeler" <David@Wheeler.net>
Subject: Re: Forking in mod_perl?
Hi David,
Check out the guide at
http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess
The Eagle book also covers the C API subprocess details on page 622-631.
Let us know if the guide is unclear to you, so we can improve it.
Ed
"David E. Wheeler" wrote:
> Hi All,
>
> Quick question - can I fork off a process in mod_perl? I've got a piece
> of code that needs to do a lot of processing that's unrelated to what
> shows up in the browser. So I'd like to be able to fork the processing
> off and return data to the browser, letting the forked process handle
> the extra processing at its leisure. Is this doable? Is forking a good
> idea in a mod_perl environment? Might there be another way to do it?
>
> TIA for the help!
>
===
From: Geoffrey Young <gyoung@laserlink.net>
To: "'David E. Wheeler'" <David@Wheeler.net>, modperl@apache.org
Subject: RE: Forking in mod_perl?
Date: Wed, 4 Oct 2000 15:40:44 -0400
David E. Wheeler [mailto:David@Wheeler.net] wrote:
> Quick question - can I fork off a process in mod_perl? I've got a piece
> of code that needs to do a lot of processing that's unrelated to what
> shows up in the browser. So I'd like to be able to fork the processing
> off and return data to the browser, letting the forked process handle
> the extra processing at its leisure. Is this doable? Is forking a good
> idea in a mod_perl environment? Might there be another way to do it?
http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess
the cleanup phase is also a good place to do extended processing. It does
tie up the child until the processing finishes, but it at least make the
client think that the response is finished (so that little moving thingy in
netscape stops moving around)
===
Date: Wed, 04 Oct 2000 12:44:29 -0700
From: "David E. Wheeler" <David@Wheeler.net>
To: modperl@apache.org
Subject: Forking in mod_perl?
Hi All,
Quick question - can I fork off a process in mod_perl? I've got a piece
of code that needs to do a lot of processing that's unrelated to what
shows up in the browser. So I'd like to be able to fork the processing
off and return data to the browser, letting the forked process handle
the extra processing at its leisure. Is this doable? Is forking a good
idea in a mod_perl environment? Might there be another way to do it?
TIA for the help!
===
Date: Wed, 04 Oct 2000 12:47:59 -0700
From: ed phillips <artlore@sirius.com>
To: "David E. Wheeler" <David@Wheeler.net>
Subject: Re: Forking in mod_perl?
"David E. Wheeler" wrote:
> ed phillips wrote:
> > Check out the guide at
> >
> > http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess
> >
> > The Eagle book also covers the C API subprocess details on page 622-631.
> >
> > Let us know if the guide is unclear to you, so we can improve it.
>
> Yeah, it's a bit unclear. If I understand correctly, it's suggesting
> that I do a system() call and have the perl script called detach itself
> from Apache, yes? I'm not too sure I like this approach. I was hoping
> for something a little more integrated. And how much overhead are we
> talking about getting taken up by this approach?
>
> Using the cleanup phase, as Geoffey Young suggests, might be a bit
> nicer, but I'll have to look into how much time my processing will
> likely take, hogging up an apache fork while it finishes.
>
> Either way, I'll have to think about various ways to handle this stuff,
> since I'm writing it into a regular Perl module that will then be called
> from mod_perl...
I hope it is clear that you don't want fork the whole server!
Mod_cgi goes to great pains to effectively fork a subprocess, and
was the major impetus I believe for the development of
the C subprocess API. It (the source code for
mod_cgi) is a great place to learn some of the
subtleties as the Eagle book points out. As the Eagle book
says, Apache is a complex beast. Mod_perl gives
you the power to use the beast to your best advantage.
Now you are faced with a trade off. Is it more expensive to
detach a subprocess, or use the child cleanup phase to do
some extra processing? I'd have to know more specifics to answer
that with any modicum of confidence.
Cheers,
Ed
===
Date: Wed, 4 Oct 2000 15:12:26 -0500 (CDT)
From: Jay Jacobs <jay@lach.net>
To: "'David E. Wheeler'" <David@Wheeler.net>
Subject: RE: Forking in mod_perl?
I was just going to post that url to the guide also... But another option
I've come up with not listed in the guide is to use the *nix "at" command.
If I need to run some processor intensive application that doesn't need
apache_anything, I'll do a system call to "at" to schedule it to run
(usually I pass in "now"). However, the drawbacks are that it's a
complete seperate process and passing complicated structures isn't worth
the time to think about using at.
Jay
On Wed, 4 Oct 2000, Geoffrey Young wrote:
>
>
> > -----Original Message-----
> > From: David E. Wheeler [mailto:David@Wheeler.net]
> > Sent: Wednesday, October 04, 2000 3:44 PM
> > To: modperl@apache.org
> > Subject: Forking in mod_perl?
> >
> >
> > Hi All,
> >
> > Quick question - can I fork off a process in mod_perl? I've
> > got a piece
> > of code that needs to do a lot of processing that's unrelated to what
> > shows up in the browser. So I'd like to be able to fork the processing
> > off and return data to the browser, letting the forked process handle
> > the extra processing at its leisure. Is this doable? Is forking a good
> > idea in a mod_perl environment? Might there be another way to do it?
>
> http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subproce
> ss
>
> the cleanup phase is also a good place to do extended processing. It does
> tie up the child until the processing finishes, but it at least make the
> client think that the response is finished (so that little moving thingy in
> netscape stops moving around)
>
> HTH
>
> --Geoff
>
> >
> > TIA for the help!
> >
> > David
> >
> > --
> > David E. Wheeler
> > Software Engineer
> > Salon Internet ICQ: 15726394
> > dw@salon.com AIM: dwTheory
> >
>
===
Date: Wed, 04 Oct 2000 13:24:57 -0700
From: "David E. Wheeler" <David@Wheeler.net>
To: ed phillips <artlore@sirius.com>
Subject: Re: Forking in mod_perl?
ed phillips wrote:
>
> Hi David,
>
> Check out the guide at
>
> http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess
>
> The Eagle book also covers the C API subprocess details on page 622-631.
>
> Let us know if the guide is unclear to you, so we can improve it.
Yeah, it's a bit unclear. If I understand correctly, it's suggesting
that I do a system() call and have the perl script called detach itself
from Apache, yes? I'm not too sure I like this approach. I was hoping
for something a little more integrated. And how much overhead are we
talking about getting taken up by this approach?
Using the cleanup phase, as Geoffey Young suggests, might be a bit
nicer, but I'll have to look into how much time my processing will
likely take, hogging up an apache fork while it finishes.
Either way, I'll have to think about various ways to handle this stuff,
since I'm writing it into a regular Perl module that will then be called
from mod_perl...
Thanks,
David
===
Date: Wed, 4 Oct 2000 17:10:08 -0400 (EDT)
From: Billy Donahue <billy@dadadada.net>
To: modperl@apache.org
Subject: Re: Forking in mod_perl?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
On Wed, 4 Oct 2000, ed phillips wrote:
> Now you are faced with a trade off. Is it more expensive to
> detach a subprocess, or use the child cleanup phase to do
> some extra processing? I'd have to know more specifics to answer
> that with any modicum of confidence.
He might try a daemon coprocesses using some IPC to communicate with
Apache, which is my favorite way to do it..
===
Date: Wed, 04 Oct 2000 14:40:22 -0700
From: "David E. Wheeler" <David@Wheeler.net>
To: ed phillips <artlore@sirius.com>
Subject: Re: Forking in mod_perl?
ed phillips wrote:
>
> I hope it is clear that you don't want fork the whole server!
>
> Mod_cgi goes to great pains to effectively fork a subprocess, and
> was the major impetus I believe for the development of
> the C subprocess API. It (the source code for
> mod_cgi) is a great place to learn some of the
> subtleties as the Eagle book points out. As the Eagle book
> says, Apache is a complex beast. Mod_perl gives
> you the power to use the beast to your best advantage.
Yeah, but I don't speak C. Just Perl. And it looks like the way to do it
in Perl is to call system() and then detach the called script. I was
trying to keep this all nice and tidy in modules, but I don't know if
it'll be possible.
> Now you are faced with a trade off. Is it more expensive to
> detach a subprocess, or use the child cleanup phase to do
> some extra processing? I'd have to know more specifics to answer
> that with any modicum of confidence.
I think I can probably evaluate that with a few tests.
Thanks!
David
===
Date: Wed, 04 Oct 2000 14:42:50 -0700
From: "David E. Wheeler" <David@Wheeler.net>
To: Billy Donahue <billy@dadadada.net>
Subject: Re: Forking in mod_perl?
Billy Donahue wrote:
> > Now you are faced with a trade off. Is it more expensive to
> > detach a subprocess, or use the child cleanup phase to do
> > some extra processing? I'd have to know more specifics to answer
> > that with any modicum of confidence.
>
> He might try a daemon coprocesses using some IPC to communicate with
> Apache, which is my favorite way to do it..
Yeah, I was thinking something along these lines. Don't know if I need
something as complex as IPC. I was thinking of perhaps a second Apache
server set up just to handle long-term processing. Then the first server
could send a request to the second with the commands it needs to execute
in a header. The second server processes those commands independantly of
the first server, which then returns data to the browser.
But maybe that's overkill. I'll have to weigh the heft of the
post-request processing I need to do.
Thanks for the suggestion!
David
===
Date: Wed, 4 Oct 2000 18:23:11 -0400
To: modperl@apache.org
Subject: Re: Forking in mod_perl?
From: Neil Conway <nconway@klamath.dyndns.org>
--0OAP2g/MAC+5xKAE
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Wed, Oct 04, 2000 at 02:42:50PM -0700, David E. Wheeler wrote:
> Yeah, I was thinking something along these lines. Don't know if I need
> something as complex as IPC. I was thinking of perhaps a second Apache
> server set up just to handle long-term processing. Then the first server
> could send a request to the second with the commands it needs to execute
> in a header. The second server processes those commands independantly of
> the first server, which then returns data to the browser.
In a pinch, I'd just use something like a 'queue' directory. In other
words, when your mod_perl code gets some info to process, it writes
this into a file in a certain directory (name it with a timestamp /
cksum to ensure the filename is unique). Every X seconds, have a
daemon poll the directory; if it finds a file, it processes it.
If not, it goes back to sleep for X seconds. I guess it's poor
man's IPC. But it runs over NFS nicely, it's *very* simple, it's
portable, and I've never needed anything more complex. You also
don't need to fork the daemon or startup a new script every
processing request. But if you need to do the processing in realtime,
waiting up to X seconds for the results might be unacceptable.
How does this sound?
===
Date: Wed, 4 Oct 2000 19:31:45 -0400 (EDT)
From: "C. Jon Larsen" <jlarsen@richweb.com>
To: Neil Conway <nconway@klamath.dyndns.org>
Subject: Re: Forking in mod_perl?
I use a database table for the queue. No file locking issues, atomic
transactions, you can sort and order the jobs, etc . . . you can wrap the
entire "queue" library in a module. Plus, the background script that
processes the queue can easily run with higher permissions, and you don't
have to worry as much with setuid issues when forking from a parent
process (like your apache) running as a user with less priviledges than
what you (may) need. You can pass all the args you need to via a column in
the db, and, if passing data back and forth is a must, serialize your data
using Storable and have the queue runner thaw it back out. Very simple,
very fast, very powerful.
On Wed, 4 Oct 2000, Neil Conway wrote:
> On Wed, Oct 04, 2000 at 02:42:50PM -0700, David E. Wheeler wrote:
> > Yeah, I was thinking something along these lines. Don't know if I need
> > something as complex as IPC. I was thinking of perhaps a second Apache
> > server set up just to handle long-term processing. Then the first server
> > could send a request to the second with the commands it needs to execute
> > in a header. The second server processes those commands independantly of
> > the first server, which then returns data to the browser.
>
> In a pinch, I'd just use something like a 'queue' directory. In other
> words, when your mod_perl code gets some info to process, it writes
> this into a file in a certain directory (name it with a timestamp /
> cksum to ensure the filename is unique). Every X seconds, have a
> daemon poll the directory; if it finds a file, it processes it.
> If not, it goes back to sleep for X seconds. I guess it's poor
> man's IPC. But it runs over NFS nicely, it's *very* simple, it's
> portable, and I've never needed anything more complex. You also
> don't need to fork the daemon or startup a new script every
> processing request. But if you need to do the processing in realtime,
> waiting up to X seconds for the results might be unacceptable.
>
> How does this sound?
>
> HTH,
>
> Neil
>
> --
> Neil Conway <neilconway@home.com>
> Get my GnuPG key from: http://klamath.dyndns.org/mykey.asc
> Encrypted mail welcomed
>
> It is dangerous to be right when the government is wrong.
> -- Voltaire
>
===
From: Jim Woodgate <woody@realtime.net>
Date: Wed, 4 Oct 2000 23:23:10 -0500 (CDT)
To: "David E. Wheeler" <David@Wheeler.net>
Cc: ed phillips <artlore@sirius.com>, modperl@apache.org
Subject: Re: Forking in mod_perl?
David E. Wheeler writes:
> Using the cleanup phase, as Geoffey Young suggests, might be a bit
> nicer, but I'll have to look into how much time my processing will
> likely take, hogging up an apache fork while it finishes.
I've wondered about this as well. I really like the cleanup handler,
and thought that in general it would be better to tie up the httpd
process and let apache decide when a new process is needed rather than
always forking.
For the most part I use the cleanup handlers to handle something that
takes alot of time, but doesn't happen very often. If I had something
that took alot of time every time someone hit a page I still don't
think I'd fork, instead I'd pass off the information to another
process and let that process run through the data asynchronously like
a spooler...
===
Date: Thu, 5 Oct 2000 07:41:35 -0400 (EDT)
From: "Sean D. Cook" <sdc@edutest.com>
To: modperl@apache.org
Subject: Re: Forking in mod_perl?
> On Wed, Oct 04, 2000 at 02:42:50PM -0700, David E. Wheeler wrote:
> > Yeah, I was thinking something along these lines. Don't know if I need
> > something as complex as IPC. I was thinking of perhaps a second Apache
> > server set up just to handle long-term processing. Then the first server
> > could send a request to the second with the commands it needs to execute
> > in a header. The second server processes those commands independantly of
> > the first server, which then returns data to the browser.
>
> In a pinch, I'd just use something like a 'queue' directory. In other
> words, when your mod_perl code gets some info to process, it writes
> this into a file in a certain directory (name it with a timestamp /
> cksum to ensure the filename is unique). Every X seconds, have a
It might be safer to do this in a db rather than the file system. That
way there is less chance for colision and you don't have to worry about
the file being half written when the daemon comes along and tries to read
the file while mod_perl/apache is trying to write it. Let the DB do the
storage side and let the damon do a select to gather the info.
===
Date: Thu, 05 Oct 2000 12:37:48 -0700
To: modperl@apache.org
From: Bill Moseley <moseley@hank.org>
Subject: Re: Forking in mod_perl? (benchmarking)
I'm working with the Swish search engine (www.apache.org and the guide use
it).
Until this month, SWISH could only be called via a fork/exec. Now there's
an early C library for swish that I've built into a perl module for use
with mod_perl.
Yea! No forking!
I decided to do some quick benchmarking with ab. I'm rotten at
benchmarking anything, so any suggestions are welcome.
My main question was this: With the library version you first call a
routine to open the index files. This reads in header info and gets ready
for the search. Then you run the query, and then you call a routine to
close the index.
OR, you can open the index file, and do multiple queries without opening
and closing the index each time. Somewhat like caching a DBI connection, I
suppose.
So I wanted to see how much faster it is to keep the index file open.
I decided to start Apache with only one child, so it would handle ALL the
requests. I'm running ab on the same machine, and only doing 100 requests.
Running my mod_perl program without asking for a query I can get almost 100
requests per second. That's just writing from memory and logging to an
open file.
Now comparing the two methods of calling SWISH I got about 7.7 request per
second leaving the index file open between requests, and 6.5 per second
opening each time. My guess is Linux is helping buffer the file contents
quite a bit since this machine isn't doing anything else at the time, so
there might be a wider gap if the machine was busy.
Now, here's why this post is under this subject thread:
For fun I changed over to forking Apache and exec'ing SWISH each request,
and I got just over 6 requests per second. I guess I would have expected
much worse, but again, I think Linux is helping out quite a bit in the fork.
And for more fun, the "same" program under mod_cgi: 0.90 requests/second
Bill Moseley
mailto:moseley@hank.org
===
Date: Thu, 5 Oct 2000 14:39:35 -0700 (PDT)
From: Tim Bishop <timb@activespace.com>
To: "Sean D. Cook" <sdc@edutest.com>
Subject: Re: Forking in mod_perl?
On Thu, 5 Oct 2000, Sean D. Cook wrote:
> > On Wed, Oct 04, 2000 at 02:42:50PM -0700, David E. Wheeler wrote:
> > > Yeah, I was thinking something along these lines. Don't know if I need
> > > something as complex as IPC. I was thinking of perhaps a second Apache
> > > server set up just to handle long-term processing. Then the first server
> > > could send a request to the second with the commands it needs to execute
> > > in a header. The second server processes those commands independantly of
> > > the first server, which then returns data to the browser.
> >
> > In a pinch, I'd just use something like a 'queue' directory. In other
> > words, when your mod_perl code gets some info to process, it writes
> > this into a file in a certain directory (name it with a timestamp /
> > cksum to ensure the filename is unique). Every X seconds, have a
>
> It might be safer to do this in a db rather than the file system. That
> way there is less chance for colision and you don't have to worry about
> the file being half written when the daemon comes along and tries to read
> the file while mod_perl/apache is trying to write it. Let the DB do the
> storage side and let the damon do a select to gather the info.
If you don't have a db easily available, I've had good luck using temp
files. You can avoid partially written file errors by exploiting the
atomic nature of moving (renaming) files. NFS does *not* have this nice
behavior, however.
===
From: "Perrin Harkins" <perrin@elem.com>
To: <kirk.rogers@verizon.net>, <modperl@apache.org>
Subject: Re: Forking another process in Apache?
Date: Sun, 20 Jan 2002 12:46:15 -0500
> I have a requirement to spin off a SQL loader process
> after a web page (a form which is qualified and accepted)
> has been submitted. Does it make sense, or more
> importantly, is it dangerous to apply a "fork" at the end
> of a module such as this:
You're probably better off using a cleanup handler to do this after
disconnecting form the client. See the guide for more details:
http://perl.apache.org/guide/performance.html#Forking_and_Executing_Subprocess
===
From: "eCap" <kirk.rogers@verizon.net>
To: <modperl@apache.org>
Subject: Forking another process in Apache?
Date: Mon, 21 Jan 2002 09:45:37 -0800
A newbie question here...
I have a requirement to spin off a SQL loader process after a web page (a
form which is qualified and accepted) has been submitted. Does it make
sense, or more importantly, is it dangerous to apply a "fork" at the end of
a module such as this:
sub handler {
# do some qualification stuff here and accept the form submission...
if ($pid = fork) {
# parent
# ...whatever i need to accomplish to deliver return html code
return OK
} elsif {
# child
exec($sql_loader);
} else {
# ...whatever i ned to do to recover errors
return DECLINED
}
}
Are there any dangers in doing something like this? Or is there a more
efficient way to accomplish the same thing?
Thanks for the advice,
Kirk
===
Date: Tue, 22 Jan 2002 12:04:05 +0500
From: "Mike P. Mikhailov" <mike@sibtel.ru>
To: eCap <kirk.rogers@verizon.net>
Subject: Re: Forking another process in Apache?
Hello eCap,
Monday, January 21, 2002, 10:45:37 PM, you wrote:
e> A newbie question here...
e> I have a requirement to spin off a SQL loader process after a web page (a
e> form which is qualified and accepted) has been submitted. Does it make
e> sense, or more importantly, is it dangerous to apply a "fork" at the end of
e> a module such as this:
e> sub handler {
e> # do some qualification stuff here and accept the form submission...
e> if ($pid = fork) {
e> # parent
e> # ...whatever i need to accomplish to deliver return html code
e> return OK
e> } elsif {
e> # child
e> exec($sql_loader);
e> } else {
e> # ...whatever i ned to do to recover errors
e> return DECLINED
e> }
e> }
e> Are there any dangers in doing something like this? Or is there a more
e> efficient way to accomplish the same thing?
e> Thanks for the advice,
e> Kirk
I'm recently implement exactly such loader. From the client HTTP
request I'm starting loader with double fork approach. I'm loading
posssible large enough (about 100 - 150 MB) data from DBF flat files
into Oracle in single transaction (I'm must provide consistency).
Loader process takes about 40-50 min to complete and consumes many
resources (CPU and RAM). But it works !
===
From: "Chris Hutchinson" <chris@hutchinsonsoftware.com>
To: "Mike P. Mikhailov" <mike@sibtel.ru>
Cc: <modperl@apache.org>
Subject: RE: Forking another process in Apache?
Date: Tue, 22 Jan 2002 21:07:06 +1100
> Mike P. Mikhailov [mailto:mike@sibtel.ru] wrote:
>
> eCap wrote:
>
> e> A newbie question here...
> e> I have a requirement to spin off a SQL loader process after a
> web page (a
> e> form which is qualified and accepted) has been submitted. Does it make
> e> sense, or more importantly, is it dangerous to apply a "fork"
> at the end of
> e> a module such as this:
>
>
> e> sub handler {
> e> # do some qualification stuff here and accept the form submission...
> e> if ($pid = fork) {
> e> # parent
> e> # ...whatever i need to accomplish to deliver return html code
> e> return OK
> e> } elsif {
> e> # child
> e> exec($sql_loader);
> e> } else {
> e> # ...whatever i ned to do to recover errors
> e> return DECLINED
> e> }
>
> e> }
>
> e> Are there any dangers in doing something like this? Or is there a more
> e> efficient way to accomplish the same thing?
>
> e> Thanks for the advice,
> e> Kirk
>
>
> I'm recently implement exactly such loader. From the client HTTP
> request I'm starting loader with double fork approach. I'm loading
> posssible large enough (about 100 - 150 MB) data from DBF flat files
> into Oracle in single transaction (I'm must provide consistency).
> Loader process takes about 40-50 min to complete and consumes many
> resources (CPU and RAM). But it works !
We've had a fair amount of success handing long processing jobs off to
daemons (written with Net:Daemon, in most cases passing across args using
Net::Telnet), using pages with reloading redirects to check the daemon
status in a table.
Avoids much work in httpd, and allows user to hang up web connection and
return later to continue viewing status.
===
From: Rob Nagler <nagler@bivio.biz>
Date: Tue, 22 Jan 2002 11:42:05 -0700
To: modperl@apache.org
Subject: RE: Forking another process in Apache?
Chris Hutchinson writes:
> Avoids much work in httpd, and allows user to hang up web connection and
> return later to continue viewing status.
We used to do this, but found it more complex (more protocols and
server types) than simply letting Apache/mod_perl handle the job. I
guess this depends on the frequency of long requests, but in our case
the mix is handle nicely with a single common server using http as the
only protocol.
The idea is that all the work is handled by the middle tier. This
includes processing incoming mail messages, long running jobs, and
credit card processing. There's a lot of common code between all
these tasks, so memory is shared efficiently.
One trick for long running jobs started by an http request is to reply
to the user as normal and do the long part in a PerlCleanupHandler.
This avoids a fork of a large process, which keeps the memory usage
relatively constant. This simplifies resource allocation.
Just another way to do it.
Rob
===
Date: Tue, 22 Jan 2002 16:42:16 -0800
To: modperl@apache.org
From: Bill Moseley <moseley@hank.org>
Subject: RE: Forking another process in Apache?
At 11:42 AM 01/22/02 -0700, Rob Nagler wrote:
>One trick for long running jobs started by an http request is to reply
>to the user as normal and do the long part in a PerlCleanupHandler.
>This avoids a fork of a large process, which keeps the memory usage
>relatively constant. This simplifies resource allocation.
But if that child is tied up Apache will fork a new child anyway, if it's
needed, so I don't see the gain. And then you would have your long running
process the size of the unshared memory of the mod_perl process, instead of
a perhaps smaller exec'd replacement for the mod_perl process.
I'd love to be corrected if this is a misunderstanding of things on my part.
===
From: Rob Nagler <nagler@bivio.biz>
Date: Tue, 22 Jan 2002 18:34:13 -0700
To: modperl@apache.org
Subject: RE: Forking another process in Apache?
Bill Moseley writes:
> But if that child is tied up Apache will fork a new child anyway, if
> it's needed, so I don't see the gain.
We set:
StartServers 10
MinSpareServers 1
MaxSpareServers 10
MaxClients 10
Or whatever the number servers we want running on the system. We
avoid dynamic resource allocation whenever possible.
> And then you would have your long running process the size of the
> unshared memory of the mod_perl process, instead of a perhaps
> smaller exec'd replacement for the mod_perl process.
We write everything in Perl. All state is stored in Oracle. We reuse
heavily.
For example, our credit card processing is handled by two classes
(about 100 lines). They use exactly the same classes (models,
object-relational mapping, etc.) as our front-end credit card entry
forms. The additional overhead of the two classes is miniscule in
comparison to the site (~300 tasks) and server (70MB).
All servers share the same log files, so we only need one log
monitor/parser. The process list is static, and easily monitored.
Server startup order is trivial. We only need to know one network
protocol (tcp/http). Apache has to be one of the most reliable
middleware platforms available. Keys to secure data (tax ids, credit
cards, etc.) reside only on the middle tier, and are entered by hand.
The side benefits really add up to quite a bit.
We still use cron for certain jobs, e.g. log processing. It's not a
panacea, but it certainly is worth looking into as an alternative to
the classical "create a server to do one job" model.
Rob
===
Date: Tue, 22 Jan 2002 17:44:11 -0800
To: Rob Nagler <nagler@bivio.biz>, modperl@apache.org
From: Bill Moseley <moseley@hank.org>
Subject: RE: Forking another process in Apache?
At 06:34 PM 01/22/02 -0700, Rob Nagler wrote:
>Bill Moseley writes:
>> But if that child is tied up Apache will fork a new child anyway, if
>> it's needed, so I don't see the gain.
>
>We set:
>
>StartServers 10
>MinSpareServers 1
>MaxSpareServers 10
>MaxClients 10
>
>Or whatever the number servers we want running on the system. We
>avoid dynamic resource allocation whenever possible.
That all seems reasonable. I was thinking of forking off long running
process, but it probably doesn't make any difference if you have the RAM.
Might as well stay in the apache process.
I'd like to see more configs like this written up in some detail. Make
great reading on the mod_perl site.
Thanks,
===
Date: Wed, 23 Jan 2002 09:24:27 +0000 (GMT)
From: Ged Haywood <ged@www2.jubileegroup.co.uk>
To: Bill Moseley <moseley@hank.org>
Subject: RE: Forking another process in Apache?
Hi all,
On Tue, 22 Jan 2002, Bill Moseley wrote:
> At 06:34 PM 01/22/02 -0700, Rob Nagler wrote:
> [snip]
>
> >We set:
> >
> >StartServers 10
> >MinSpareServers 1
> >MaxSpareServers 10
> >MaxClients 10
> >
> >Or whatever the number servers we want running on the system.
> >We avoid dynamic resource allocation whenever possible.
>
> [snip]
>
> I'd like to see more configs like this written up in some detail.
> Make great reading on the mod_perl site.
I second that.
73,
Ged.
===
From: "Perrin Harkins" <perrin@elem.com>
To: <modperl@apache.org>, "Bill Moseley" <moseley@hank.org>
Subject: Re: Forking another process in Apache?
Date: Wed, 23 Jan 2002 10:47:57 -0500
> At 11:42 AM 01/22/02 -0700, Rob Nagler wrote:
> >One trick for long running jobs started by an http request is to reply
> >to the user as normal and do the long part in a PerlCleanupHandler.
> >This avoids a fork of a large process, which keeps the memory usage
> >relatively constant. This simplifies resource allocation.
>
> But if that child is tied up Apache will fork a new child anyway, if it's
> needed, so I don't see the gain. And then you would have your long
running
> process the size of the unshared memory of the mod_perl process, instead
of
> a perhaps smaller exec'd replacement for the mod_perl process.
There are situations where this is true, but keep in mind that in an exec'ed
process nothing is shared (well, no Perl bytecode anyway), so it may very
well end up bigger than the existing one if it's using many of the same
libraries.
Also, if you run a busy site it can be dangerous to go exec'ing processes
willy nilly. Apache provides you with some controls to limit the overall
memory used. With MaxClients and Apache::SizeLimit, you can keep a pretty
good lock on the total amount of RAM in use and prevent swap. If you exec a
whole bunch of processes, you might swamp your server. Repeated requests to
your exec'ing URL would be an effective DoS attack.
This is why queuing a job to be picked up by a cron script or daemon is a
good approach when you can get away with it. It effective controls the
resources being used.
===