This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
From: "Andrew Dunstan" <andrewd@writeme.com> To: "Modperl" <modperl@apache.org> Subject: Re: open(FH,'|qmail-inject') fails Date: Thu, 7 Sep 2000 21:59:04 -0400 Could someone please explain to me why everybody seems so intent on having a mod_perl handler fork in order to send mail? Why not just use the very common Net::SMTP package which just talks on an SMTP socket to whatever mailhost you have (localhost or other). There are other packages on CPAN which perhaps have more power, but still don't fork, if that's what you need. Every benchmark I've done (quite a few ;-) shows that this is far faster way of sending mail. My understanding (correct me if I'm wrong) is that in general having a mod_perl handler fork is a Bad Thing (tm). (and of course there is even less danger with funky email addresses with shell metacharacters that way, too) I recall with some fondness Randal's "useless use of cat" awards - maybe we need to create a "useless use of fork" award :-) === From: "David Harris" <dharris@drh.net> To: "Stas Bekman" <stas@stason.org>, "Perrin Harkins" <perrin@primenet.com> Cc: "Modperl" <modperl@apache.org> Subject: RE: open(FH,'|qmail-inject') fails Date: Fri, 8 Sep 2000 19:26:45 -0400 Stas wrote: > On Fri, 8 Sep 2000, Perrin Harkins wrote: > > Nevertheless, in benchmarks we ran we found forking qmail-inject to be > > quite a bit faster than Net::SMTP. I'd say that at least from a > > command-line script qmail-inject is a more scalable approach. > > Quite possible, I was talking about the fat sendmail binaries :) Yes, quite possible. Using SMTP and qmail-inject both have the overhead of a fork, because the SMTP tcpserver will fork off a copy of qmail-smtpd to handle the request. Additionally, the SMTP tcpserver is probably doing a reverse DNS lookup and probably an ident lookup which would probably cause another fork for identd. (Both reverse DNS and ident lookup are enabled by default in ucspi-tcp-0.84.) This network activity and possibly another fork will cause delay. The overhead of forking directly off from mod_perl does not seem so bad when you look at copy-on-write memory managers. The fork: (a) does not cause copying of the big mod_perl process at fork thanks to copy-on-write, and (b) there will be virtually no dirtying of pages and copying because a exec() will be immediately done. A possible problem is qmail-inject inheriting a bunch of filehandles from mod_perl, but they should all be marked close-on-exec. === From: "Andrew Dunstan" <andrewd@writeme.com> To: "Modperl" <modperl@apache.org> Subject: Re: open(FH,'|qmail-inject') fails Date: Sat, 9 Sep 2000 00:52:24 -0400 Regarding cost of forking etc.: Your mileage will undoubtedly vary, according to OS and MTA. Last time I did work on this was about a year ago on Solaris 2.6, with sendmail and postfix. In both cases using Net::SMTP was far faster. IIRC, with postfix there is no forking cost at all, as its daemon does not fork on connect (it uses a select() loop instead). Talking on the SMTP port is actually Wietse Venema's recommended method for fastest injection into the postfix queue. It also has the advantage over other methods that it is totally MTA independent. === From: Shane Adams <shane@viralon.com> To: Andrew Dunstan <andrewd@writeme.com>, Modperl <modperl@apache.org> Subject: RE: open(FH,'|qmail-inject') fails Date: Fri, 8 Sep 2000 22:01:39 -0700 This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_001_01C01A1B.0987037A Content-Type: text/plain; charset="iso-8859-1" Another approach to is to write the email directly into the queue. I've used this approach and it's very fast. After you write your email to the qmail queue, you write a value of 1 to a named pipe that qmail reads off of. This causes a qmail process (there are like 20 different ones and I forget which is which - check the docs) to wake up and drain the queue. If you want anymore speed then that, you have to either install ram disks or seriously write your own mta. We installed ram disks =) === Date: Sat, 09 Sep 2000 12:29:36 -0700 To: Modperl <modperl@apache.org> From: Bill Moseley <moseley@hank.org> Subject: Re: open(FH,'|qmail-inject') fails At 11:15 PM 09/08/00 +0200, Stas Bekman wrote: >On Fri, 8 Sep 2000, Bill Moseley wrote: >> I just looked at my old mail sending module a few days ago that uses >> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it >> was running on Win at one point, argh!). I just removed the Net::SMTP >> part. Are you saying that I removed the wrong code? >> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: > >As Perrin has suggested, benchmark it an see what's faster. It's so >simple. <somewhat joking> Benchmark: timing 1 iterations of SMTP, Sendmail... SMTP: 60 wallclock secs ( 0.03 usr + 0.00 sys = 0.03 CPU) @ 33.33/s (n=1) (warning: too few iterations for a reliable count) Sendmail: 0 wallclock secs ( 0.00 usr 0.00 sys + 0.01 cusr 0.01 csys = 0.02 CPU) (warning: too few iterations for a reliable count) I guess it isn't a fair test ;) I had my DNS servers unplugged and used -odd with sendmail. And I guess if I set a timeout of 60 seconds on Net::SMTP I better be ready to wait that long. </somewhat joking> Seriously, though, I can't imagine how to benchmark such a complicated thing as sending mail. I can do it on my machine, I guess, but that doesn't tell me how it will work on the live machine that's also sending out a mailing list or running a faulty nscd on Solaris that hangs DNS requests, or mailing to foo@a.b.c.d.e.f.com and DNS is hopping from DNS to DNS to do the lookup. The nscd thing is the reason I moved from using Net::SMTP; SMTP worked great most of the time, but really messed things up when the DNS was not working. Forking may or may not be slower, but it like more of a sure bet. === Date: Sat, 9 Sep 2000 17:26:37 -0400 From: Richard Chen <qchen@snet.net> To: Bill Moseley <moseley@hank.org> Cc: modperl@apache.org Subject: Re: open(FH,'|qmail-inject') fails On Sat, Sep 09, 2000 at 12:29:36PM -0700, Bill Moseley wrote: > At 11:15 PM 09/08/00 +0200, Stas Bekman wrote: > >On Fri, 8 Sep 2000, Bill Moseley wrote: > >> I just looked at my old mail sending module a few days ago that uses > >> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it > >> was running on Win at one point, argh!). I just removed the Net::SMTP > >> part. Are you saying that I removed the wrong code? > > >> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote: > > > >As Perrin has suggested, benchmark it an see what's faster. It's so > >simple. > > <somewhat joking> > > Benchmark: timing 1 iterations of SMTP, Sendmail... > SMTP: 60 wallclock secs ( 0.03 usr + 0.00 sys = 0.03 CPU) @ 33.33/s > (n=1) > (warning: too few iterations for a reliable count) > Sendmail: 0 wallclock secs ( 0.00 usr 0.00 sys + 0.01 cusr 0.01 csys = > 0.02 CPU) > (warning: too few iterations for a reliable count) > > I guess it isn't a fair test ;) I had my DNS servers unplugged and used > -odd with sendmail. And I guess if I set a timeout of 60 seconds on > Net::SMTP I better be ready to wait that long. > > </somewhat joking> > > Seriously, though, I can't imagine how to benchmark such a complicated > thing as sending mail. I can do it on my machine, I guess, but that > doesn't tell me how it will work on the live machine that's also sending > out a mailing list or running a faulty nscd on Solaris that hangs DNS > requests, or mailing to foo@a.b.c.d.e.f.com and DNS is hopping from DNS to > DNS to do the lookup. > > The nscd thing is the reason I moved from using Net::SMTP; SMTP worked > great most of the time, but really messed things up when the DNS was not > working. Forking may or may not be slower, but it like more of a sure bet. > > Have fun, > > > > > Bill Moseley > mailto:moseley@hank.org All your objections about smtp will be gone if you set up a dedicated local account just for processing emails from modperl. Since the account is local, Net::SMTP will get done instantly and the modperl can get back doing what it is supposed to be doing instead of waiting for dns resolution, nscd and what not. So in modperl, you write the real email as the body of email to the local account, giving it a special subject line such as "send out the body text". Then you set up procmail or .forward for that local account so that it will do "sendmail -oi -t" or whatever to actually send the body of text. It can take as much time as it wants without interfering the website service. === Date: Sat, 09 Sep 2000 11:45:18 +0000 From: Greg Cope <gjjc@rubberplant.freeserve.co.uk> To: Perrin Harkins <perrin@primenet.com> Subject: Re: open(FH,'|qmail-inject') fails Perrin Harkins wrote: > > On Fri, 8 Sep 2000, Stas Bekman wrote: > > > As far as benchmarks are concerned, I'm sending one mail after having > > > displayed the page, so it shoul'dnt matter much ... > > > > Yeah, and everytime you get 1M process fired up... > > Nevertheless, in benchmarks we ran we found forking qmail-inject to be > quite a bit faster than Net::SMTP. I'd say that at least from a > command-line script qmail-inject is a more scalable approach. Or even better would be to use qmail-remote and avoid the queue, but thats a qmail and not a mod_perl thing. BTW I'm looking at this at the mo and have decided to go back to a dump email to db -> crond robot sends mail to port 25 -> mail delt with there. Why ? Well this is the most scalable, and its cross platform, talking directly via pipes / forks etc usually has a limit somewhere, and all too often I may reach it far sooner than I thought. ===