This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Date: Wed, 17 Apr 2002 00:12:33 -0400 From: Benjamin Elbirt <benelb@nac.net> To: modperl@perl.apache.org Subject: Sharing Variable Across Apache Children I've gotten it so that I can define a variable across apache children, however when I update the variable, it only updates for the active child. Is there any way to share a variable across children so no matter what child is doing the update, the variable is updated for all children? === Date: Wed, 17 Apr 2002 12:22:23 +0800 From: Stas Bekman <stas@stason.org> To: benelb@nac.net Cc: modperl@perl.apache.org Subject: Re: Sharing Variable Across Apache Children Benjamin Elbirt wrote: > I've gotten it so that I can define a variable across > apache children, however when I update the variable, it > only updates for the active child. Is there any way to > share a variable across children so no matter what child > is doing the update, the variable is updated for all > children? You cannot do that unless you use IPC, which is usually only useful if the variable is small. DBM file is another solution if you need share a hash variable. some info is here:http://perl.apache.org/preview/modperl-docs/dst_html/docs/1.0/guide/porting.html#Sharing_variables_between_processes This section needs work. Can anybody help me improve this section? Thanks! === Date: Wed, 17 Apr 2002 08:49:17 -0400 From: Perrin Harkins <perrin@elem.com> To: Stas Bekman <stas@stason.org> Subject: Re: Sharing Variable Across Apache Children Stas Bekman wrote: > You cannot do that unless you use IPC, which is usually only useful if > he variable is small. DBM file is another solution if you need share a > hash variable. Yes, and if you use the tied interface of MLDM::Sync, it can look just like a normal hash, but be shared across all apache children. > some info is > here:http://perl.apache.org/preview/modperl-docs/dst_html/docs/1.0/guide/porting.html#Sharing_variables_between_processes > > > This section needs work. Can anybody help me improve this section? I'm writing a whole article on this subject for my presentation at TPC. When it's done, I'll contribute it. === Date: Wed, 17 Apr 2002 21:59:47 +0800 From: Stas Bekman <stas@stason.org> To: Perrin Harkins <perrin@elem.com>, modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children Perrin Harkins wrote: > Stas Bekman wrote: > >> You cannot do that unless you use IPC, which is usually only useful if >> he variable is small. DBM file is another solution if you need share a >> hash variable. > > > Yes, and if you use the tied interface of MLDM::Sync, it can look just > like a normal hash, but be shared across all apache children. But won't it be ineffective if the data set is big? >> some info is >> here:http://perl.apache.org/preview/modperl-docs/dst_html/docs/1.0/guide/porting.html#Sharing_variables_between_processes >> >> >> This section needs work. Can anybody help me improve this section? > > > I'm writing a whole article on this subject for my presentation at TPC. > When it's done, I'll contribute it. Great! Thanks Perrin! === Date: Wed, 17 Apr 2002 10:14:29 -0400 From: DeWitt Clinton <dewitt@unto.net> To: Stas Bekman <stas@stason.org> Cc: Perrin Harkins <perrin@elem.com>, modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children On Wed, Apr 17, 2002 at 09:59:47PM +0800, Stas Bekman wrote: > >Yes, and if you use the tied interface of MLDM::Sync, it can look just > >like a normal hash, but be shared across all apache children. > > But won't it be ineffective if the data set is big? Or, sacrifice the tied interface and use something like Cache::Cache, particularly FileCache, which handles the sharing of large data sets rather well. You'll have to call cache->set and cache->get to keep data current, but that degree of control may be desirable anyway. Cheers, -DeWitt PS: Perrin -- where'd you go? You normally suggest this! === Date: Wed, 17 Apr 2002 10:18:29 -0400 From: Perrin Harkins <perrin@elem.com> To: Stas Bekman <stas@stason.org> Subject: Re: Sharing Variable Across Apache Children Stas Bekman wrote: >> Yes, and if you use the tied interface of MLDM::Sync, it can look just >> like a normal hash, but be shared across all apache children. > > > But won't it be ineffective if the data set is big? Typo, that should have been MLDBM::Sync. It's good even with a large data set. It will be slowed down if you put a huge chunk of data in a single hash entry though. === Date: Wed, 17 Apr 2002 10:23:22 -0400 From: Perrin Harkins <perrin@elem.com> To: DeWitt Clinton <dewitt@unto.net> Subject: Re: Sharing Variable Across Apache Children DeWitt Clinton wrote: > Or, sacrifice the tied interface and use something like Cache::Cache, > particularly FileCache, which handles the sharing of large data sets > rather well. You'll have to call cache->set and cache->get to keep > data current, but that degree of control may be desirable anyway. > > Cheers, > > -DeWitt > > PS: Perrin -- where'd you go? You normally suggest this! Cache::Cache would of course work as well. This guy seemed to want a very simple interface that wouldn't change his code much, so a tied interface might be better for him. It is slower to use a tied interface though. On this subject, it would be nice to have a supported public interface to the hash-like storage of the Cache::Cache modules without the actual cache stuff on top. Most people don't know that they can use the file backend separately if they don't want any expiration stuff. === Date: Wed, 17 Apr 2002 11:15:36 -0400 From: Benjamin Elbirt <benelb@nac.net> To: modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children Perrin Harkins wrote: > DeWitt Clinton wrote: > > Or, sacrifice the tied interface and use something like Cache::Cache, > > particularly FileCache, which handles the sharing of large data sets > > rather well. You'll have to call cache->set and cache->get to keep > > data current, but that degree of control may be desirable anyway. > > > > PS: Perrin -- where'd you go? You normally suggest this! > > Cache::Cache would of course work as well. This guy seemed to want a > very simple interface that wouldn't change his code much, so a tied > interface might be better for him. It is slower to use a tied interface > though. > > On this subject, it would be nice to have a supported public interface > to the hash-like storage of the Cache::Cache modules without the actual > cache stuff on top. Most people don't know that they can use the file > backend separately if they don't want any expiration stuff. First, let me thank you all for the responses. I'm looking into everything you guys said, and I've also picked up a book on Perl Network Programming (and I'm trying to get the Mod_Perl Cookbook as well). Let me explain in more detail what I'm doing. Basically, I've created a mod_perl application environment that is horizontally scalable. In fact, I have it running on 5 machines connected to a single oracle database on the back end and I use DNS to load balance the machines. It looks something like: [USER] | [LOAD BALANCE] | ------------------------------------------ | | | | | MP1 MP2 MP3 MP4 MP5 | | | | | ------------------------------------------ | [ORACLE] So, what I've also done is added to the mod_perl system an e-mail notification option for specific errors (like the database being unavailable or a template file being unavailable) so that I can keep an eye on things remotely. However, because I have 5 systems, and I'm receiving about 450,000 requests a day across the 5 machines, I get TONS of e-mails on a constant basis when things like the Database Dropping occurs. To solve this problem, I came up with the idea that I will have a global variable, called $AVAILABILITY, for each of the 5 machines. When something fatal goes wrong, it sends the first e-mail and then modifies $AVAILABILITY to have something like: $AVAILABILITY = { 'E-MAIL NOTIFY' => { 'DB FAILURE' => { 'LAST ACTIVITY' => 385903583, 'TIME TO LIVE' => 1800 }, } }; The next time the system goes to run the e-mail for the problem, it will look up in the $AVAILABILITY->{'E-MAIL NOTIFY'} hash ref the 'DB FAILURE' error key... if it finds it, it will use the LAST ACTIVITY (time stamp) + the TIME TO LIVE and see if it's >= the current time stamp. If it is, then it will NOT send another e-mail. If it is < the current time stamp, reset the LAST ACTIVITY to the current time stamp and send 1 more e-mail. On a side note, there's actually more to this as their is a clean-up method that runs to remove anything that has passed the TIME TO LIVE and hasn't received further e-mail attempts (thus reducing memory, etc.). Also, for fatal errors, I set something up called NA (Not Available) that actually forces the code to tell users the system is down for maintenance (thus never actually attempting to do the rest of the code / send e-mails, etc.) The setup I have now is not sharing the $AVAILABILITY across the children. It creates a separate instance for each child and any modifications are unique to that child. Clearly this defeats the purpose as having 50 open children per server, 5 servers, results in up to 250 e-mails before it stops. Now, that's better than the 2000 every 10 minutes I was getting at one point, but I'd rather it be 5 e-mails and be done with it. Now, using the Database is not an option as it may crash (thus the $AVAILABILITY becomes unavailable). Using flat file is also possible but I figured the read/writes would make it slow and possibly cause over-lapping write problems. Hence the desire to have the shared memory. Thanks for the help, again. Sorry for the lengthy reply, but I figured it'd be easier to provide a solid answer this way. === Date: Wed, 17 Apr 2002 11:41:54 -0400 From: Perrin Harkins <perrin@elem.com> To: benelb@nac.net Subject: Re: Sharing Variable Across Apache Children Benjamin Elbirt wrote: > Using flat file is also possible but I > figured the read/writes would make it slow and possibly cause over-lapping > write problems. Hence the desire to have the shared memory. Actually flat files are faster than shared memory in most situations on a modern OS. It's a common mistake to assume shared memory will be faster. As for your larger problem, there are a few things you could try. One is to have something like Mail::Audit process that mailbox and remove dupes within a certain time fram. Another is to switch as much of this as possible to an external monitoring tool like Mon: http://ftp.kernel.org/software/mon/ === Date: Wed, 17 Apr 2002 08:46:04 -0700 (PDT) From: Andrew Ho <andrew@tellme.com> To: Benjamin Elbirt <benelb@nac.net> Cc: modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children BE>Let me explain in more detail what I'm doing. So if the situation you explain is the only reason you want a variable shared load balanced machines, I'd suggest a totally different way of doing this altogether. Best would be to use an already shared persistent storage mechanism (NFS or Oracle) but it looks like Oracle warnings are precisely what you want distinct alerts on (why are you getting so many Oracle errors anyway? that might be the first thing you want to check). I'm assuming the number of warning e-mails you get is less than 450,000 / 5 == 90,000 each day, so that if each warning e-mail were a web request, a single box could handle them (if more than 1/5 of your requests result in errors, you REALLY want to just fix the problem first). So put up a single webserver box to serve as an error reporter and logger. You could either use distributed logging (like mod_log_spread) or simpler, just set up another webserver that takes requests like /record_error.pl?error_msg=foo&remote_addr=bar or whatever. Your error handlers on your five load-balanced boxes send an HTTP request to this error handling box. All error e-mails can originate from this box, and the box can internally keep a lookup table (using any of the fine techniques discussed by the folks here) to avoid sending duplicate errors. In this scenario error handling is offloaded to another box, and as a bonus you can track the aggregate number of errors each day in an automated way and run reports and such (without having to count e-mails in your inbox). === Date: Wed, 17 Apr 2002 11:58:15 -0400 From: Perrin Harkins <perrin@elem.com> To: Andrew Ho <andrew@tellme.com> Subject: Re: Sharing Variable Across Apache Children Andrew Ho wrote: > Your error handlers on your five load-balanced boxes send an HTTP request > to this error handling box. That sounds kind of slow, since it requires a network connection to be created every time and some synchronous processing on the other end. It also depends on that box always staying up. I think e-mail is actually a good approach, since it's a robust message queuing system and if you use something like qmail-inject to send the e-mail it takes almost no time at all for mod_perl to finish with it and move on. You just need to process those messages on the other end instead of looking at the raw output, i.e. use Mail::Audit to keep track of the current state and remove duplicate messages. Matt posted something about PPerl yesterday, which could make a Mail::Audit script more efficient by keeping it persistent. === Date: Wed, 17 Apr 2002 11:56:36 -0400 From: Benjamin Elbirt <benelb@nac.net> To: modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children Perrin Harkins wrote: > Andrew Ho wrote: > > Your error handlers on your five load-balanced boxes send an HTTP request > > to this error handling box. > > That sounds kind of slow, since it requires a network connection to be > created every time and some synchronous processing on the other end. It > also depends on that box always staying up. I think e-mail is actually > a good approach, since it's a robust message queuing system and if you > use something like qmail-inject to send the e-mail it takes almost no > time at all for mod_perl to finish with it and move on. You just need > to process those messages on the other end instead of looking at the raw > output, i.e. use Mail::Audit to keep track of the current state and > remove duplicate messages. > > Matt posted something about PPerl yesterday, which could make a > Mail::Audit script more efficient by keeping it persistent. I never expected the response I got! Well, lets assume that I were to go with the shared memory option anyway... what would the pitfalls be / concerns? The truth is, I don't want a separate system (as per the e-mail about having an error handling server), and I don't want to have to manage the e-mail on the receiving end because I'm not the only person who receives it (didn't mention it, but I guess that's important). Further, I have no control over the mail server that handles the incoming mail so I'd have to handle it on the mail client (Outlook / Netscape Mail) resulting in the same problem I have now. === Date: Wed, 17 Apr 2002 12:19:50 -0500 From: Tom Brown <thecap@mail.utexas.edu> To: Benjamin Elbirt <benelb@nac.net> Cc: modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children On Wed, Apr 17, 2002 at 11:56:36AM -0400, Benjamin Elbirt wrote: > I never expected the response I got! Well, lets assume that I were to go with > the shared memory option anyway... what would the pitfalls be / concerns? The > truth is, I don't want a separate system (as per the e-mail about having an > error handling server), and I don't want to have to manage the e-mail on the > receiving end because I'm not the only person who receives it (didn't mention > it, but I guess that's important). Further, I have no control over the mail > server that handles the incoming mail so I'd have to handle it on the mail > client (Outlook / Netscape Mail) resulting in the same problem I have now. Is the webserver useful if you have an error that warrants sending a mail? If sending an email means the server is broken having a flood of mails may be a feature. It will be incentive to fix whatever is breaking your server/db. Also, I would strongly recommend keeping your warning system as simple as possible. Why not have the servers output an error message to a file on a single nfs file system and then setup a crontab to watch the file(s)? One machine could run the crontab every minute. 60 mails/min isn't too much unless you are forwarding it to a beeper in which case you could write a quick script to only mail once per unit time. === Date: Wed, 17 Apr 2002 13:26:52 -0400 From: Perrin Harkins <perrin@elem.com> To: benelb@nac.net Subject: Re: Sharing Variable Across Apache Children Benjamin Elbirt wrote: > Well, lets assume that I were to go with > the shared memory option anyway... what would the pitfalls be / concerns? As mentioned before, you'd probably be better off with MLDBM::Sync or Cache::Cache. You can try IPC::Shareable, but a lot of people seem to have trouble getting it to work. Note that this is not shared acorss all the machine in the cluster unless you used a shared filesystem like NFS. === From: "Vuillemot, Ward W" <ward.w.vuillemot@boeing.com> To: modperl list <modperl@apache.org> Subject: RE: Sharing Variable Across Apache Children Date: Wed, 17 Apr 2002 14:21:07 -0700 Sam Tregar [mailto:sam@tregar.com] wrote: > On Wed, 17 Apr 2002, Perrin Harkins wrote: > > > Benjamin Elbirt wrote: > > > Well, lets assume that I were to go with > > > the shared memory option anyway... what would the > pitfalls be / concerns? > > > > As mentioned before, you'd probably be better off with > MLDBM::Sync or > > Cache::Cache. You can try IPC::Shareable, but a lot of > people seem to > > have trouble getting it to work. > > I agree with you 100% - file-based caches are generally > as-fast and far > easier to manage. Still, I can't resist the urge to plug my > IPC::SharedCache module. It's much easier than using > IPC::Shareable (or > even the better alternative, IPC::ShareLite). I am desperate enough to use this thread to try to get an answer to my own IPC question. ;) I keep getting the following error using OPEN3 along with the appropriate modifications to my code for using mod_perl with Perl 5.6.1. The offending (and it is very offensive to me!) is: [error] open3: Can't call method "close" on an undefined value at C:/Perl/lib/IPC/Open3.pm line 327, <_GEN_0> line 154. [error] [client 127.0.0.1] [libapreq] remove error on 'C:\WINNT\TEMP\apreq15' Okay, it is two errors. But I am quite positive they are related. I am trying to run this on Win32. When I look at the processes, I found a bunch of abandoned children that I had to kill off. Any ideas? === Date: Wed, 17 Apr 2002 20:44:11 -0700 From: Medi Montaseri <medi@cybershell.com> To: Perrin Harkins <perrin@elem.com> Subject: Re: Sharing Variable Across Apache Children Perrin Harkins wrote: > Stas Bekman wrote: > > You cannot do that unless you use IPC, which is usually only useful if > > he variable is small. DBM file is another solution if you need share a > > hash variable. > > Yes, and if you use the tied interface of MLDM::Sync, it can look just > like a normal hash, but be shared across all apache children. > > > some info is > > here:http://perl.apache.org/preview/modperl-docs/dst_html/docs/1.0/guide/porting.html#Sharing_variables_between_processes > > > > > > This section needs work. Can anybody help me improve this section? > > I'm writing a whole article on this subject for my presentation at TPC. > When it's done, I'll contribute it. Abstracting access to data is only half of the story....the more challenging part is race conditions and lock management... === Date: Wed, 17 Apr 2002 21:08:00 -0700 From: Medi Montaseri <medi@cybershell.com> To: Andrew Ho <andrew@tellme.com> Subject: Re: Sharing Variable Across Apache Children Andrew Ho wrote: > Hello, > > BE>Let me explain in more detail what I'm doing. > > So if the situation you explain is the only reason you want a variable > shared load balanced machines, I'd suggest a totally different way of > doing this altogether. Best would be to use an already shared persistent > storage mechanism (NFS or Oracle) but it looks like Oracle warnings are > precisely what you want distinct alerts on (why are you getting so many > Oracle errors anyway? that might be the first thing you want to check). > > I'm assuming the number of warning e-mails you get is less than 450,000 / > 5 == 90,000 each day, so that if each warning e-mail were a web request, a > single box could handle them (if more than 1/5 of your requests result in > errors, you REALLY want to just fix the problem first). So put up a single > webserver box to serve as an error reporter and logger. You could either > use distributed logging (like mod_log_spread) or simpler, just set up > another webserver that takes requests like > /record_error.pl?error_msg=foo&remote_addr=bar or whatever. > > Your error handlers on your five load-balanced boxes send an HTTP request > to this error handling box. All error e-mails can originate from this box, > and the box can internally keep a lookup table (using any of the fine > techniques discussed by the folks here) to avoid sending duplicate errors. > > In this scenario error handling is offloaded to another box, and as a > bonus you can track the aggregate number of errors each day in an > automated way and run reports and such (without having to count e-mails in > your inbox). You had us going for a while....I thought you are talking about some distributed session management (accross different boxes).... Another suggestion is to use lbnamed. lbnamed is a DNS server and Load Balancing server that listens to port 53 and resolves IPs, but on the other side of its personality it talks to bunch of agents who are running on workers. You get to set what the parameters or criteria is and assign a cost factor for a worker. lbnamed then distributes the work on the lightest/least cost worker. In this scenario, whence a box is out (or its critical piece like Oracle, or HTTP server) then no further work is routed to it . Also, be aware that using CNAME in DNS does not provide a uniform distribution of load. Imagine a web page having 20 images and another 5 images. You'll not know with good certainty that if your heavy work like database access is really being distributed. With CNAME you do have a chance of developing harmonics. CNAME (aka round robin) is totally unaware of the load on the worker. Maybe that's why your boxes are bulking.... See http://www.stanford.edu/~schemers/docs/lbnamed/lbnamed.html === Date: Wed, 17 Apr 2002 21:20:57 -0700 From: Medi Montaseri <medi@cybershell.com> To: benelb@nac.net Subject: Re: Sharing Variable Across Apache Children Unfortunately everyone wants to use email for everything.....the proper way would be to use some facilities that have proven track record, like syslogd(1d) on each box. As you know syslogd() allows you to route a message to a remote box. So, you assign one of your boxes as the final or central point. All messages would be routed to this box automatically for you. Here you can have a process, person or a file to receive such cumalated messages. Here you have the opportunity to visit last 100 message and look for corrolations. The point is, process these reports first, prioritize and then inform. By the way, your exact problem has been solved by syslogd().... Try the following on your unix box..... Find an entry in your /etc/syslog.conf that writes to /var/log/message or somel file. I will use cron.none , then do logger -p cron.none -t Test "this is a test" logger -p cron.none -t Test "this is a test" logger -p cron.none -t Test "this is a test" etc, etc You'll see that syslog will say last message repeated 4 times Cheers.... === From: "Perrin Harkins" <perrin@elem.com> To: "Medi Montaseri" <medi@cybershell.com> Cc: <modperl@perl.apache.org> Subject: Re: Sharing Variable Across Apache Children Date: Thu, 18 Apr 2002 00:51:42 -0400 > Abstracting access to data is only half of the > story....the more challenging part is race conditions and > lock management... All of the data sharing tools take the logistics of multi-process read/write situations into account, although they do it in different ways. Some use file locking, others use semaphores, and still others use tricks like atomic file renames. The ones I mentioned in this thead (MLDBM::Sync and Cache::Cache) are both written to provide safe multi-process access. === Date: Thu, 18 Apr 2002 12:52:11 +0800 From: Stas Bekman <stas@stason.org> To: Medi Montaseri <medi@cybershell.com> Cc: benelb@nac.net, modperl list <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children Medi Montaseri wrote: > Unfortunately everyone wants to use email for > everything.....the proper way would be to use some > facilities that have proven track record, like syslogd(1d) > on each box. syslog(3) is not a very scalable solution because it's slow. It's also unreliable because it uses UDP to send the data, which doesn't ensure that the data will reach its destination. This solution is also unicast. If the master host goes down the logs will get lost. === Date: Thu, 18 Apr 2002 12:54:31 +0800 From: Stas Bekman <stas@stason.org> Cc: Medi Montaseri <medi@cybershell.com>, benelb@nac.net, Subject: Re: Sharing Variable Across Apache Children Stas Bekman wrote: > Medi Montaseri wrote: > >> Unfortunately everyone wants to use email for everything.....the >> proper way would >> be to use some facilities that have proven track record, like >> syslogd(1d) on each >> box. > > > syslog(3) is not a very scalable solution because it's slow. > It's also unreliable because it uses UDP to send the data, which doesn't > ensure that the data will reach its destination. This solution is also > unicast. If the master host goes down the logs will get lost. And a better replacement is mod_log_spread. Based on the group communications toolkit Spread using IP multicast. mod_log_spread provides reliable, scalable centralized logging whith minimal performance impact on the web servers. For more information see http://www.backhand.org/mod_log_spread/. === From: "Peter Bi" <mod_perl@att.net> To: "Medi Montaseri" <medi@cybershell.com> Cc: "modperl list" <modperl@apache.org> Subject: Re: Sharing Variable Across Apache Children Date: Wed, 17 Apr 2002 23:42:08 -0700 "Medi Montaseri" <medi@cybershell.com> wrote: > You had us going for a while....I thought you are talking > about some distributed session management (accross > different boxes).... > Another suggestion is to use lbnamed. lbnamed is a DNS > server and Load Balancing server that listens to port 53 > and resolves IPs, but on the other side of its personality > it talks to bunch of agents who are running on > workers. You get to set what the parameters or criteria is > and assign a cost factor for a worker. lbnamed then > distributes the work on the lightest/least cost worker. > In this scenario, whence a box is out (or its critical > piece like Oracle, or HTTP server) then no further work is > routed to it . > Also, be aware that using CNAME in DNS does not provide a > uniform distribution of load. Imagine a web page having 20 > images and another 5 images. You'll not know with good > certainty that if your heavy work like database access is > really being distributed. With CNAME you do have a chance > of developing harmonics. CNAME (aka round robin) is > totally unaware of the load on the worker. Maybe that's > why your boxes are bulking.... > See http://www.stanford.edu/~schemers/docs/lbnamed/lbnamed.html What will happen if the client's DNS caches the domain name to an IP, which is then dead ? If I understand it corrently, the current system can work only if there is something like NAT in front of the machines, which dynaimcally forward each request. Won't it ? === Date: Wed, 17 Apr 2002 23:50:34 -0700 From: Medi Montaseri <medi@cybershell.com> To: Peter Bi <mod_perl@att.net> Subject: Re: Sharing Variable Across Apache Children Peter Bi wrote: > What will happen if the client's DNS caches the domain name to an IP, which > is then dead ? If I understand it corrently, the current system can work > only if there is something like NAT in front of the machines, which > dynaimcally forward each request. Won't it ? True, but much better than Round Robin DNS.....also one can configure those A records to expire immediately.... ===