This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Subject: enterprise mod_perl architectures From: "Ed Park" <epark@athenahealth.com> Date: Wed, 6 Dec 2000 01:33:29 -0500 I've been using mod_perl for two years, and I'm currently particularly interested in: 1) Definitively establishing mod_perl as a credible player in the enterprise space. 2) Discussing enterprise-level architecture considerations, performance benchmarks, development methodologies, etc. By "enterprise", I mean folks who have serious money to throw at mod_perl-- folks who have racks of quad-Xeon, 4GB servers at Exodus serving millions of pages per day, and who have been asked (or are being asked) by the company to make their application infinitely scalable. I know that there are several of you out there. As a bit of background, I started out using mod_perl/Embperl because pound for pound, it allowed me to develop a high-quality production application in the shortest amount of time on a stable platform. At that point, WebLogic and EJB were at their version .9 spec (and were highly unstable besides), PHP did not have very rich third-party module support. I have continued to use mod_perl because of its outstanding performance, stability, and architectural characteristics-- in the two years that our site has been up in production, I have had exactly zero problems that could be traced to instabilities in mod_perl/Embperl. During those two years, we grew from an angel-backed garage company to a company that is now backed by Draper Fisher Jurvetson, Venrock, Cardinal Health Partners, and (most recently) Oak Investment Partners. Guy Kawasaki (CEO of garage.com) was once quoted as saying that the amount of pressure increases with the square of the amount money that you have, and I have certainly found that to true. We are now at the hockey stick of growth. Currently, our application serves 300,000 dynamic page requests per day, each of which executes on the average of about 20 SQL statements against an Oracle RDBMS running on Linux. By the end of next year, the application needs to be able to support at least 8 million dynamic page requests per day; and the year after that, 30+ million per day. The project that I propose is simple and concrete: create an open forum in which all of the folks who are currently undergoing the same growing pains that we are, or who have been through them already, or who are otherwise interested in the project, to address the following questions, specifically as they relate to mod_perl + Oracle (or perhaps some other RDBMS): 1) What are the software building blocks that you use? I'm interested specifically in mod_perl + Oracle, but I'd also welcome questions about mod_perl + apache. 2) What is the logical scaling architecture that you use? i.e., how have you architected your mod_perl application so that it will scale linearly with additional hardware? 3) What is the physical scaling and high-availability architecture that you have used? e.g., what is the best way to load-balance across a set of mod_perl servers? Do folks use LVS, or mod_proxy, or F5 BigIPs, or Arrowpoint switches... etc; what hardware are folks using? Dell, Compaq, VA Linux, Penguin, etc.-- what vendor did you choose, and why? Have you had any problems with the vendor? Are folks running mod_perl in production on Linux, or have folks found BSD or Solaris to be better servers, and for what reasons? 4) What tools do folks use to monitor your mod_perl servers? For example, I have written apache modules that simply time how long it takes to execute a page, and a script that does a quick https 'GET' from all production servers to make certain they are up. 5) What tools do folks use to profile the performance of the application? I have also written scripts that essentially do an 'explain plan' in Oracle on every query executed for a given pageview. (This was done very painfully, by hacking dbdimp.c... in retrospect, there may have been an easier way of hacking it at the DBD:: layer, but I wasn't sufficently familiar with the mechanics of XS to do that). I have also tried DProf, but I was unable to get it to work in any meaningful way (don't remember exactly why now) 5) How do you estimate server capacity? For example, has anyone used tools that proxy all requests for a given day and play them back, or take the apache logs for a server for a given day and play them back at varying speeds, simulating varying bandwidth? 6) etc. As you can probably tell, a good deal of this will be mod_perl-specific, and a good deal will simply be a forum for folks to talk _from experience_ about the relative pros and cons of various production architectures that may or may not have anything to do with mod_perl. This is a conversation that may in places go off-topic, and in some places -way- off-topic. However, it is a conversation that I think should ultimately benefit the mod_perl community because it should eventually provide examples of mod_perl architectures, in production, that scale to tens or hundreds of millions of requests per day, and should help to ease some of the tensions that are generated when top management and VCs come knocking with questions. In this way, it should dovetail nicely with the mod_perl advocacy project. I am not yet certain whether the best forum for this is this mailing list, or whether I should try to create a private list of names for folks who are interested. Relevant considerations include: -The possible very off-topicness of pieces of the discussion. -At some point, some of us may want opinions from other folks on sensitive information (network diagrams, etc.) that Corporate won't allow us to show to the outside world except under NDA; if all the folks on a list signed an NDA, then we could speak freely all the time. -At any rate, I'd like to publish any methodologies we use and put any monitoring tools, performance benching tools, etc. into open-source. To that end, I'll be creating a page that publishes any code we come up with and summarizes our thoughts. I'd be happy to publish that page myself, but I could also just add it as a page-- 'Enterprise mod_perl architectures'-- to Matt's new site (modperl.sergeant.org). So, I'd like to get folks' thoughts on this project. Again, I am staking out very high ground on this project-- multimillion-dollar companies with multimillion-dollar budgets. I'm doing this not because I'm disparaging other companies, but because part of the reason behind doing this project is to establish mod_perl's credibility as an enterprise web platform and to describe some of the pitfalls and workarounds that allow mod_perl to scale to that level. To that end, I'd like to get a list of interested parties. In general, this should include the chief architects, CTOs, and/or senior engineers at different shops using mod_perl. Some of those folks don't read this list regularly, and in that case, I'd be happy to email them/call them directly if people could just point them my way. If any subset of folks are interested, I'd be more than happy to drive this project forward. This is a project that really describes one of my core responsibilities in my company right now, so I actually have a lot of time and the resources to devote to this as part of my job. Anyways, not to belabor the point-- I'd like y'alls input on this, specifically: 1) What do folks thing about the project in general? 2) Should we keep it on this list, or should we create a separate mailing list for interested parties, or should we do a combination of the two? 3) Is there anyone who'd like to volunteer virtual space to host this? e.g. ftp, web, creating a mailing list, etc. I am not yet interested in specifics about peoples' architectures; I think that we need to frame the general discussion and create some infrastructure before we go into that. === Subject: Re: enterprise mod_perl architectures From: "Gerald Richter" <richter@ecos.de> Date: Wed, 6 Dec 2000 08:03:10 +0100 Some of those folks don't read > this list regularly, and in that case, I'd be happy to email them/call them > directly if people could just point them my way. > I know that iii in London (www.iii.co.uk) use Embperl/mod_perl for there site and they generate about 1 million dynamic pages a day. For them I currently working on speeding up Embperl which will published as Embperl 2.0. I know they are working together with Tim Bunce to improve the database side. Maybe they are interessed in your project. > 3) Is there anyone who'd like to volunteer virtual space to host this? e.g. > ftp, web, creating a mailing list, etc. > I can do this, but I guess it also wouldn't be a problem to host this on perl.apache.org Gerald === Subject: Re: enterprise mod_perl architectures From: Stas Bekman <stas@stason.org> Date: Wed, 6 Dec 2000 12:21:32 +0100 (CET) On 6 Dec 2000, David Hodgkinson wrote: > "Gerald Richter" <richter@ecos.de> writes: > > > > 3) Is there anyone who'd like to volunteer virtual space to host this? > > e.g. > > > ftp, web, creating a mailing list, etc. > > > > > > > I can do this, but I guess it also wouldn't be a problem to host this on > > perl.apache.org > > This is, IMHO, a mod_perl advocacy effort and should sit on that site > quite nicely... We even have a special list for that, which wasn't really used because people didn't signup. So we can do it now. (well I'm there :) http://perl.apache.org/#advocacy-list === Subject: Re: enterprise mod_perl architectures From: barries <barries@slaysys.com> Date: Wed, 6 Dec 2000 07:53:29 -0500 On Wed, Dec 06, 2000 at 01:33:29AM -0500, Ed Park wrote: > I've been using mod_perl for two years, and I'm currently particularly > interested in: > 1) Definitively establishing mod_perl as a credible player in the enterprise > space. > 2) Discussing enterprise-level architecture considerations, performance > benchmarks, development methodologies, etc. You might be interested in the scalable@arctic.org list Ask mentioned a while ago. It's pretty low traffic, but this lot has the ability to change all of that... http://www.mail-archive.com/modperl%40apache.org/msg11699.html I also am interested in learning from others' battle scars and victories in these areas and a scalable@perl.apache.org (as opposed to the advocacy list Stas mentioned) would be interesting, or seeing it here. === Subject: Re: enterprise mod_perl architectures From: Ask Bjoern Hansen <ask@valueclick.com> Date: Wed, 6 Dec 2000 10:28:58 -0800 (PST) On Wed, 6 Dec 2000, Ed Park wrote: > The project that I propose is simple and concrete: create an open forum in > which all of the folks who are currently undergoing the same growing pains > that we are, or who have been through them already, or who are otherwise [...] The scalable list is for you. try scalable-subscribe@arctic.org and look at http://archive.develooper.com/scalable@arctic.org/ ===