modperl-Net::SMTP_avoids_need_to_fork_off_a_mailhandler_or_does_it

This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.



From: "Andrew Dunstan" <andrewd@writeme.com>
To: "Modperl" <modperl@apache.org>
Subject: Re: open(FH,'|qmail-inject') fails
Date: Thu, 7 Sep 2000 21:59:04 -0400


Could someone please explain to me why everybody seems so
intent on having a mod_perl handler fork in order to send
mail? Why not just use the very common Net::SMTP package
which just talks on an SMTP socket to whatever mailhost you
have (localhost or other). There are other packages on CPAN
which perhaps have more power, but still don't fork, if
that's what you need. Every benchmark I've done (quite a few
;-) shows that this is far faster way of sending mail.

My understanding (correct me if I'm wrong) is that in
general having a mod_perl handler fork is a Bad Thing (tm).

(and of course there is even less danger with funky email
addresses with shell metacharacters that way, too)

I recall with some fondness Randal's "useless use of cat"
awards - maybe we need to create a "useless use of fork"
award :-)

===

From: "David Harris" <dharris@drh.net>
To: "Stas Bekman" <stas@stason.org>, "Perrin Harkins" <perrin@primenet.com>
Cc: "Modperl" <modperl@apache.org>
Subject: RE: open(FH,'|qmail-inject') fails
Date: Fri, 8 Sep 2000 19:26:45 -0400


Stas wrote:
> On Fri, 8 Sep 2000, Perrin Harkins wrote:
> > Nevertheless, in benchmarks we ran we found forking qmail-inject to be
> > quite a bit faster than Net::SMTP.  I'd say that at least from a
> > command-line script qmail-inject is a more scalable approach.
>
> Quite possible, I was talking about the fat sendmail binaries :)

Yes, quite possible.

Using SMTP and qmail-inject both have the overhead of a fork, because the SMTP
tcpserver will fork off a copy of qmail-smtpd to handle the request.

Additionally, the SMTP tcpserver is probably doing a reverse DNS lookup and
probably an ident lookup which would probably cause another fork for identd.
(Both reverse DNS and ident lookup are enabled by default in ucspi-tcp-0.84.)
This network activity and possibly another fork will cause delay.

The overhead of forking directly off from mod_perl does not seem so bad when
you look at copy-on-write memory managers. The fork: (a) does not cause copying
of the big mod_perl process at fork thanks to copy-on-write, and (b) there will
be virtually no dirtying of pages and copying because a exec() will be
immediately done. A possible problem is qmail-inject inheriting a bunch of
filehandles from mod_perl, but they should all be marked close-on-exec.

===

From: "Andrew Dunstan" <andrewd@writeme.com>
To: "Modperl" <modperl@apache.org>
Subject: Re: open(FH,'|qmail-inject') fails
Date: Sat, 9 Sep 2000 00:52:24 -0400


Regarding cost of forking etc.:

Your mileage will undoubtedly vary, according to OS and MTA.

Last time I did work on this was about a year ago on Solaris 
2.6, with sendmail and postfix. In both cases using Net::SMTP 
was far faster. IIRC, with postfix there is no forking cost at all, 
as its daemon does not fork on connect (it uses a select() loop 
instead). Talking on the SMTP port is actually Wietse Venema's 
recommended method for fastest injection into the postfix queue.

It also has the advantage over other methods that it is totally 
MTA independent.

===

From: Shane Adams <shane@viralon.com>
To: Andrew Dunstan <andrewd@writeme.com>, Modperl <modperl@apache.org>
Subject: RE: open(FH,'|qmail-inject') fails
Date: Fri, 8 Sep 2000 22:01:39 -0700 

This message is in MIME format. Since your mail reader does not understand
this format, some or all of this message may not be legible.

------_=_NextPart_001_01C01A1B.0987037A
Content-Type: text/plain;
	charset="iso-8859-1"

Another approach to is to write the email directly into the queue.  I've
used this approach and it's very fast.  After you write your email to
the qmail queue, you write a value of 1 to a named pipe that qmail reads
off of.  This causes a qmail process (there are like 20 different ones
and I forget which is which - check the docs) to wake up and drain the
queue.

If you want anymore speed then that, you have to either install ram
disks or seriously write your own mta.  We installed ram disks =)

===

Date: Sat, 09 Sep 2000 12:29:36 -0700
To: Modperl <modperl@apache.org>
From: Bill Moseley <moseley@hank.org>
Subject: Re: open(FH,'|qmail-inject') fails

At 11:15 PM 09/08/00 +0200, Stas Bekman wrote:
>On Fri, 8 Sep 2000, Bill Moseley wrote:
>> I just looked at my old mail sending module a few days ago that uses
>> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it
>> was running on Win at one point, argh!).  I just removed the Net::SMTP
>> part.  Are you saying that I removed the wrong code?

>> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote:
>
>As Perrin has suggested, benchmark it an see what's faster. It's so
>simple.

<somewhat joking>

Benchmark: timing 1 iterations of SMTP, Sendmail...
    SMTP: 60 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU) @ 33.33/s
(n=1)
         (warning: too few iterations for a reliable count)
Sendmail:  0 wallclock secs ( 0.00 usr  0.00 sys +  0.01 cusr  0.01 csys =
         0.02 CPU)
         (warning: too few iterations for a reliable count)

I guess it isn't a fair test ;)  I had my DNS servers unplugged and used
-odd with sendmail.  And I guess if I set a timeout of 60 seconds on
Net::SMTP I better be ready to wait that long.

</somewhat joking>

Seriously, though, I can't imagine how to benchmark such a complicated
thing as sending mail.  I can do it on my machine, I guess, but that
doesn't tell me how it will work on the live machine that's also sending
out a mailing list or running a faulty nscd on Solaris that hangs DNS
requests, or mailing to foo@a.b.c.d.e.f.com and DNS is hopping from DNS to
DNS to do the lookup.  

The nscd thing is the reason I moved from using Net::SMTP; SMTP worked
great most of the time, but really messed things up when the DNS was not
working.  Forking may or may not be slower, but it like more of a sure bet.

===

Date: Sat, 9 Sep 2000 17:26:37 -0400
From: Richard Chen <qchen@snet.net>
To: Bill Moseley <moseley@hank.org>
Cc: modperl@apache.org
Subject: Re: open(FH,'|qmail-inject') fails

On Sat, Sep 09, 2000 at 12:29:36PM -0700, Bill Moseley wrote:
> At 11:15 PM 09/08/00 +0200, Stas Bekman wrote:
> >On Fri, 8 Sep 2000, Bill Moseley wrote:
> >> I just looked at my old mail sending module a few days ago that uses
> >> sendmail and would fallback to Net::SMTP if sendmail wasn't available (it
> >> was running on Win at one point, argh!).  I just removed the Net::SMTP
> >> part.  Are you saying that I removed the wrong code?
> 
> >> At 10:31 AM 09/08/00 +0200, Stas Bekman wrote:
> >
> >As Perrin has suggested, benchmark it an see what's faster. It's so
> >simple.
> 
> <somewhat joking>
> 
> Benchmark: timing 1 iterations of SMTP, Sendmail...
>     SMTP: 60 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU) @ 33.33/s
> (n=1)
>          (warning: too few iterations for a reliable count)
> Sendmail:  0 wallclock secs ( 0.00 usr  0.00 sys +  0.01 cusr  0.01 csys =
>          0.02 CPU)
>          (warning: too few iterations for a reliable count)
> 
> I guess it isn't a fair test ;)  I had my DNS servers unplugged and used
> -odd with sendmail.  And I guess if I set a timeout of 60 seconds on
> Net::SMTP I better be ready to wait that long.
> 
> </somewhat joking>
> 
> Seriously, though, I can't imagine how to benchmark such a complicated
> thing as sending mail.  I can do it on my machine, I guess, but that
> doesn't tell me how it will work on the live machine that's also sending
> out a mailing list or running a faulty nscd on Solaris that hangs DNS
> requests, or mailing to foo@a.b.c.d.e.f.com and DNS is hopping from DNS to
> DNS to do the lookup.  
> 
> The nscd thing is the reason I moved from using Net::SMTP; SMTP worked
> great most of the time, but really messed things up when the DNS was not
> working.  Forking may or may not be slower, but it like more of a sure bet.
> 
> Have fun,
> 
> 
> 
> 
> Bill Moseley
> mailto:moseley@hank.org

All your objections about smtp will be gone if you set up a
dedicated local account just for processing emails from modperl.
Since the account is local, Net::SMTP will get done instantly
and the modperl can get back doing what it is supposed to be doing
instead of waiting for dns resolution, nscd and what not.

So in modperl, you write the real email as the body of email
to the local account, giving it a special subject line such
as "send out the body text". Then you set up procmail or .forward
for that local account so that it will do "sendmail -oi -t" or
whatever to actually send the body of text. It can take as much time
as it wants without interfering the website service.

===
Date: Sat, 09 Sep 2000 11:45:18 +0000
From: Greg Cope <gjjc@rubberplant.freeserve.co.uk>
To: Perrin Harkins <perrin@primenet.com>
Subject: Re: open(FH,'|qmail-inject') fails

Perrin Harkins wrote:
> 
> On Fri, 8 Sep 2000, Stas Bekman wrote:
> > > As far as benchmarks are concerned, I'm sending one mail after having
> > > displayed the page, so it shoul'dnt matter much ...
> >
> > Yeah, and everytime you get 1M process fired up...
> 
> Nevertheless, in benchmarks we ran we found forking qmail-inject to be
> quite a bit faster than Net::SMTP.  I'd say that at least from a
> command-line script qmail-inject is a more scalable approach.

Or even better would be to use qmail-remote and avoid the queue, but
thats a qmail and not a mod_perl thing.

BTW I'm looking at this at the mo and have decided to go back to a dump
email to db -> crond robot sends mail to port 25 -> mail delt with
there.  Why ? Well this is the most scalable, and its cross platform,
talking directly via pipes / forks etc usually has a limit somewhere,
and all too often I may reach it far sooner than I thought.

===

the rest of The Pile (a partial mailing list archive)

doom@kzsu.stanford.edu