This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Date: Thu, 31 Aug 2000 08:47:14 +0100 (BST)
From: Matt Sergeant <matt@sergeant.org>
To: Rob Tanner <rtanner@onlinemac.com>
Subject: Re: OT -- BerkeleyDB/Perl/Solaris
On Wed, 30 Aug 2000, Rob Tanner wrote:
> I know this is off topic but I've gotten zero reponses off normal
> channels nor have I heard back when I emailed the module author (and I
> know there's at least one other person in the same predicament), so I'm
> hoping I might get some help here.
>
> The problem is building the perl BerkeleyDB module on Solaris. I've
> installed BerkeleyDB-3.1.14 from Sleepycat. It comes with a huge
> battery of tests, and some 14 hours later, they finished and the
> BerkeleyDB build passed with zero errors.
>
> Now comes the perl module BerkeleyDB-0.12. It build without complaint,
> but can't get to first base in 'make test'. I have included the full
> set of results below.
I heard on IRC the other day that BDB still isn't 100% happy on perl 5.6
--
<Matt/>
Fastnet Software Ltd. High Performance Web Specialists
Providing mod_perl, XML, Sybase and Oracle solutions
Email for training and consultancy availability.
http://sergeant.org | AxKit: http://axkit.org
===
From: "Mark Matthews" <mark.matthews@citcomm.com>
To: <modperl@perl.apache.org>
Subject: BerkeleyDB Problems
Date: Tue, 12 Mar 2002 11:06:00 -0500
Hello,
I am moving a website that now resides on a i686 server running RedHat
6.2 with perl v5.005_03 to another i686 server running Suse 7.1 with
perl v5.6.1.
The website uses a number of cgi scripts that read and write from
BerkeleyDB files using the tie function.
The site is currently running fine on the RedHat server, but when
testing the scripts on the Suse box I am finding the the scripts are
failing complaining that the db file cannot be opened.
The function calling the script is as follows...
$db = "blurb";
tie(%BLURB, "DB_File", $db, O_RDONLY, 0664) || die("Error: could not
open $db: $!\n");
Things I have tried so far..
- I have checked that the BerkeleyDB file (blurb) in the right path, and
is readable/writable.
- I have checked that the DB file is not corrupt by ftping it back to
the RedHat box and testing it.. Works fine..
- the command "file blurb" shows that the db file is Berkeley DB (Hash,
version 5, native byte-order) and my guess is the version of DB_File
cannot read that DB version. I have installed earlier versions of
DB_File on the Suse box with no luck.
- I have successfully created a new db file using tie. The file created
is version 7.
Since these scripts do infact work on the RedHat server, what do I need
to do to get them to work on the Suse server
Any help would be greatly appreciated..
Mark Matthews
===
Date: Tue, 12 Mar 2002 08:12:05 -0800
From: Paul Lindner <lindner@inuus.com>
To: Mark Matthews <mark.matthews@citcomm.com>
Cc: modperl@perl.apache.org
Subject: Re: BerkeleyDB Problems
On Tue, Mar 12, 2002 at 11:06:00AM -0500, Mark Matthews wrote:
> Hello,
>
> I am moving a website that now resides on a i686 server running RedHat
> 6.2 with perl v5.005_03 to another i686 server running Suse 7.1 with
> perl v5.6.1.
> The website uses a number of cgi scripts that read and write from
> BerkeleyDB files using the tie function.
>
> The site is currently running fine on the RedHat server, but when
> testing the scripts on the Suse box I am finding the the scripts are
> failing complaining that the db file cannot be opened.
> The function calling the script is as follows...
> $db = "blurb";
> tie(%BLURB, "DB_File", $db, O_RDONLY, 0664) || die("Error: could not
> open $db: $!\n");
>
> Things I have tried so far..
> - I have checked that the BerkeleyDB file (blurb) in the right path, and
> is readable/writable.
> - I have checked that the DB file is not corrupt by ftping it back to
> the RedHat box and testing it.. Works fine..
> - the command "file blurb" shows that the db file is Berkeley DB (Hash,
> version 5, native byte-order) and my guess is the version of DB_File
> cannot read that DB version. I have installed earlier versions of
> DB_File on the Suse box with no luck.
> - I have successfully created a new db file using tie. The file created
> is version 7.
>
> Since these scripts do infact work on the RedHat server, what do I need
> to do to get them to work on the Suse server
DB_File is usually implemented on top of whatever the latest Berkeley
DB is available. The file formats are usually not compatible from
major version to another major version. I believe RH 6.2 uses bdb v2
and Suse uses a newer version 3 or 3.1. Use the rpm -q -a command to
find out which versions are which.
Anyway, you need to try out the db_upgrade command, it should upgrade
the db file to the latest format. It might not be installed by
default, so check your local docs. (I think it's in db3-utils, or
some such..)
You might also try out the BerkeleyDB module for access to more
interesting features provided by the later versions of this library.
In particular the transactions subsystem is very, very cool.
Good Luck
--
Paul Lindner lindner@inuus.com ||||| | | | | | | | | |
mod_perl Developer's Cookbook http://www.modperlcookbook.org/
Human Rights Declaration http://www.unhchr.ch/udhr/
===
Date: Wed, 13 Mar 2002 00:26:22 +0800
From: Stas Bekman <stas@stason.org>
To: Mark Matthews <mark.matthews@citcomm.com>
Cc: modperl@perl.apache.org
Subject: Re: BerkeleyDB Problems
Mark Matthews wrote:
> Hello,
>
> I am moving a website that now resides on a i686 server running RedHat
> 6.2 with perl v5.005_03 to another i686 server running Suse 7.1 with
> perl v5.6.1.
>
> The website uses a number of cgi scripts that read and write from
> BerkeleyDB files using the tie function.
>
> The site is currently running fine on the RedHat server, but when
> testing the scripts on the Suse box I am finding the the scripts are
> failing complaining that the db file cannot be opened.
>
> The function calling the script is as follows...
> $db = "blurb";
> tie(%BLURB, "DB_File", $db, O_RDONLY, 0664) || die("Error: could not
> open $db: $!\n");
>
> Things I have tried so far..
> - I have checked that the BerkeleyDB file (blurb) in the right path, and
> is readable/writable.
> - I have checked that the DB file is not corrupt by ftping it back to
> the RedHat box and testing it.. Works fine..
> - the command "file blurb" shows that the db file is Berkeley DB (Hash,
> version 5, native byte-order) and my guess is the version of DB_File
> cannot read that DB version. I have installed earlier versions of
> DB_File on the Suse box with no luck.
> - I have successfully created a new db file using tie. The file created
> is version 7.
>
> Since these scripts do infact work on the RedHat server, what do I need
> to do to get them to work on the Suse server
Mark, I doubt this has anything to do with mod_perl.
Have you tested the functionality as a standalone script or even a one
liner?
In any case make sure that you have the same versions of BerkeleyDB and
DB_File installed on both machines. Notice that there are three
different major versions of BerkeleyDB. The latest BerkeleyDB versions 2
and 3 require a different Perl module: BerkeleyDB.pm and it won't work
with older 1st version of BerkeleyDB lib.
_____________________________________________________________________
Stas Bekman JAm_pH -- Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide http://perl.apache.org/guide
mailto:stas@stason.org http://ticketmaster.com http://apacheweek.com
http://singlesheaven.com http://perl.apache.org http://perlmonth.com/
===
Date: Tue, 12 Mar 2002 10:04:09 -0800 (PST)
From: Michael Robinton <michael@bizsystems.com>
To: <modperl@perl.apache.org>
Subject: Re: BerkeleyDB Problems
> On Tue, Mar 12, 2002 at 11:06:00AM -0500, Mark Matthews wrote:
> > Hello,
> >
> > I am moving a website that now resides on a i686 server running RedHat
> > 6.2 with perl v5.005_03 to another i686 server running Suse 7.1 with
> > perl v5.6.1.
>
> > The website uses a number of cgi scripts that read and write from
> > BerkeleyDB files using the tie function.
> >
> > The site is currently running fine on the RedHat server, but when
> > testing the scripts on the Suse box I am finding the the scripts are
> > failing complaining that the db file cannot be opened.
>
> > The function calling the script is as follows...
> > $db = "blurb";
> > tie(%BLURB, "DB_File", $db, O_RDONLY, 0664) || die("Error: could not
> > open $db: $!\n");
> >
> > Things I have tried so far..
> > - I have checked that the BerkeleyDB file (blurb) in the right path,
and
> > is readable/writable.
> > - I have checked that the DB file is not corrupt by ftping it back to
> > the RedHat box and testing it.. Works fine..
> > - the command "file blurb" shows that the db file is Berkeley DB
(Hash,
> > version 5, native byte-order) and my guess is the version of DB_File
> > cannot read that DB version. I have installed earlier versions of
> > DB_File on the Suse box with no luck.
> > - I have successfully created a new db file using tie. The file
created
> > is version 7.
> >
> > Since these scripts do infact work on the RedHat server, what do I
need
> > to do to get them to work on the Suse server
>
> DB_File is usually implemented on top of whatever the latest
> Berkeley DB is available. The file formats are usually not
> compatible from major version to another major version. I believe
> RH 6.2 uses bdb v2 and Suse uses a newer version 3 or 3.1. Use the
> rpm -q -a command to find out which versions are which.
>
> Anyway, you need to try out the db_upgrade command, it should
> upgrade the db file to the latest format. It might not be installed
> by default, so check your local docs. (I think it's in db3-utils,
> or some such..)
>
> You might also try out the BerkeleyDB module for access to more
> interesting features provided by the later versions of this library.
> In particular the transactions subsystem is very, very cool.
>
> Good Luck
>
> --
It's more complicated than that :-(
Newer versions of Linux (e.g. RedHat 6, SuSe 6) ship with a C library
that has version 2.x of Berkeley DB linked into it. My particular
version has 2.x with header files for 3.x. To make matters worse,
prior to Perl 5.6.1, the perl binary itself included the Berkeley DB
library. This has caused me some headaches which I've solved by
building and installing BDB v 4.x and including the following at the
beginning of the apachecel / httpsdctl file.
# fix up problem with C-lib database
export LD_PRELOAD=/usr/local/BerkeleyDB.4.0/lib/libdb.so
The LD_PRELOAD environment variable to point to the new shared
library, Perl will use it instead of the version of Berkeley DB that
shipped with your Linux distribution.
Maybe this will work for you as well. If you use DBD, make sure and
re-install the DBD / DBI modules with the new pointers to the
database lib.
Michael Robinton
BizSystems
4600 El Camino Real - Ste 206
Los Altos, CA 94022
650-947-3351
===
Date: Thu, 21 Mar 2002 12:47:54 -0500
To: modperl@perl.apache.org
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Hello,
I have run into a problem specific to Berkeley 4.0.14 that I hope you
can help me to diagnose. There are enough other products involved
that I'm not sure if it's DB's fault or mod_perl's. Here's the setup:
Linux 7.2 i386
perl 5.6.1
apache 1.3.19 with mod_perl 1.25
DB 4.0.14
BerkeleyDB 0.18
I'm testing with the Perl script below, with the filename ending
".mperl" (which, in my configuration, causes it to run as a mod_perl
registry script).
The problem: Under DB 4.0.14, five lockers get allocated, one during
each iteration of the loop. The test program will show "Current
number of lockers" and "Maximum number of lockers so far" both to be
5. In a production environment, this behavior eventually leads to an
error, because the maximum number of lockers is reached.
If I either use DB 3.x or even run this from the commandline
(bypassing mod_perl) under DB 4 the problem goes away: only one
locker is allocated per loop, and therefore the total number used
does not increase unexpectedly.
I have already contacted Sleepycat (the makers of DB), but they
cannot help much, since their experience with mod_perl is limited.
--------- TEST SCRIPT ---------
#!/usr/bin/perl
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
# Change me to something appropriate for your system
my $dir='/home/httpd/some/directory';
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
===
Date: Thu, 21 Mar 2002 13:32:35 -0500
From: Perrin Harkins <perrin@elem.com>
To: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Dan Wilga wrote:
> If I either use DB 3.x or even run this from the commandline (bypassing
> mod_perl) under DB 4 the problem goes away: only one locker is allocated
> per loop, and therefore the total number used does not increase
> unexpectedly.
This sort of begs the question: why not use DB 3.x? Is there some new
feature you need in DB 4?
It's been a little while since I messed with this stuff, but I don't
think you need to tear down the Env each time. I think you can just
create it the first time and reuse it after that. Maybe that will help.
(Ah, Mount Holyoke. As a Five College alumn, I remember it well.)
- Perrin
===
Date: Thu, 21 Mar 2002 13:46:03 -0500
To: Perrin Harkins <perrin@elem.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
At 1:32 PM -0500 3/21/02, Perrin Harkins wrote:
>Dan Wilga wrote:
>>If I either use DB 3.x or even run this from the commandline
>>(bypassing mod_perl) under DB 4 the problem goes away: only one
>>locker is allocated per loop, and therefore the total number used
>>does not increase unexpectedly.
>
>This sort of begs the question: why not use DB 3.x? Is there some
>new feature you need in DB 4?
I'm having problems with minor DB corruption in the last release of
3.x (3.3.11), so I wanted to try 4.
>It's been a little while since I messed with this stuff, but I don't
>think you need to tear down the Env each time. I think you can just
>create it the first time and reuse it after that. Maybe that will
>help.
I tried that, believe me :-). I only did it this way in the test
script to show that the problem occurs despite that bit of extra
caution. (Even the "undefs" are probably not needed, but are there
for clarity.)
What surprises me is that all I have to do to introduce the problem
is run it under mod_perl. It acts normally when run from the
commandline.
>(Ah, Mount Holyoke. As a Five College alumn, I remember it well.)
:-)
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
===
Date: Thu, 21 Mar 2002 13:55:53 -0500
From: Perrin Harkins <perrin@elem.com>
To: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Dan Wilga wrote:
> What surprises me is that all I have to do to introduce the problem is
> run it under mod_perl. It acts normally when run from the commandline.
Well, let's see, what would be different...
Is it possible that the problem is concurrency from multiple mod_perl
processes? What happens when you run it with httpd -X ?
Could there be a permissions issue involved?
Are you sure you're only invoking the script once under mod_perl? If
you raise the number of repetitions, does the command-line version
exhibit the problem?
I don't see any scoping issues in your script, but are you running it
under Apache::Registry? Maybe the nested subroutines that creates are
causing problems.
- Perrin
===
To: modperl@apache.org
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
From: Michael Alan Dorman <mdorman@debian.org>
Date: 21 Mar 2002 14:01:49 -0500
Perrin Harkins <perrin@elem.com> writes:
> This sort of begs the question: why not use DB 3.x? Is there some new
> feature you need in DB 4?
Anecdotaly, I believe the OpenLDAP and Cyrus projects have both found
DB4 to be more reliable under load than DB3.
Mike.
===
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
From: Aaron Ross <ross@alias-i.com>
To: Dan Wilga <dwilga@MtHolyoke.edu>
Cc: modperl@perl.apache.org
Date: 21 Mar 2002 14:03:26 -0500
> I'm testing with the Perl script below, with the filename ending
> ".mperl" (which, in my configuration, causes it to run as a mod_perl
> registry script).
I would re-write it as a handler and see if Apache::Registry is partly
to blame.
hth, aaron
===
Date: Thu, 21 Mar 2002 15:45:02 -0500
To: Perrin Harkins <perrin@elem.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
At 1:55 PM -0500 3/21/02, Perrin Harkins wrote:
>Dan Wilga wrote:
>>What surprises me is that all I have to do to introduce the problem
>>is run it under mod_perl. It acts normally when run from the
>>commandline.
>
>Well, let's see, what would be different...
>
>Is it possible that the problem is concurrency from multiple
>mod_perl processes? What happens when you run it with httpd -X ?
The same (bad) results.
>Could there be a permissions issue involved?
Nope, all the files get deleted and re-created in the script (and
they do get created correctly). Permissions also wouldn't explain why
the same thing works in 3.x.
>Are you sure you're only invoking the script once under mod_perl?
Yes, I'm doing it by a direct URL.
> If you raise the number of repetitions, does the command-line
>version exhibit the problem?
No, the commandline version never does. When run under mod_perl, the
"Current number of lockers" reported by db_stat is always equal to
the number of iterations, no matter how many are used.
>I don't see any scoping issues in your script, but are you running
>it under Apache::Registry? Maybe the nested subroutines that
>creates are causing problems.
Ack. Ptooey. I just renamed the file to end in ".cgi" and the problem
persists. I even removed mod_perl from my configuration files, and it
still fails. I thought I had tested the before, but I guess not.
This can only mean mod_perl isn't at fault. Sorry for the false
alarm. I'll have to go back to Sleepycat with this new information.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"
===
Date: Thu, 21 Mar 2002 15:49:09 -0500
To: Aaron Ross <ross@alias-i.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>
>> I'm testing with the Perl script below, with the filename ending
>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>> registry script).
>
> I would re-write it as a handler and see if Apache::Registry is partly
>to blame.
Sorry, I figured out that the problem persists even when mod_perl is
completely disabled, so the problem must lie in either Apache or
Berkeley DB. (As a diagnostic, I think I'll try disabling
mod_auth_db.* and mod_ssl, since they both use DB.)
Thanks for the quick responses, folks!
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
===
Date: Thu, 21 Mar 2002 16:00:16 -0500
To: Perrin Harkins <perrin@elem.com>, Aaron Ross <ross@alias-i.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
OK, I'm an idiot who can't read. I was reading the wrong part of
db_stat's output when I thought that running it with the .cgi ending
also caused the problem. Turns out it doesn't.
So I have not absolved mod_perl after all. It's definitely an
interaction between it and DB 4.
This is one of those times I really wish I could go back and edit my
posts :-). Anyone reading this thread will now be seriously confused.
So I'll try Aaron's suggestion of writing it as a handler. Any other
suggestions are welcome.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
===
Date: Thu, 21 Mar 2002 16:46:27 -0500
To: Aaron Ross <ross@alias-i.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>
>> I'm testing with the Perl script below, with the filename ending
>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>> registry script).
>
> I would re-write it as a handler and see if Apache::Registry is partly
>to blame.
I tried doing it as a handler, using the configuration below (and the
appropriate changes in the source) and the problem persists. So it
doesn't seem to be Registry's fault.
<Location /dan>
SetHandler perl-script
PerlHandler DanTest
</Location>
---- source code ----
#!/usr/bin/perl
package DanTest;
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
my $dir='/home/httpd/some/path';
sub handler {
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
}
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
1;
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
===
Date: Fri, 22 Mar 2002 00:27:11 -0500 (EST)
From: Ed Grimm <ed@tgape.org>
To: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Does shutting down apache free up your locks?
(As an aside, I'm not sure I'll ever get over undef being proper closing
of a database connection; it seems so synonomous to free([23]). I
expect something like $db->db_close() or something.)
Ed
On Thu, 21 Mar 2002, Dan Wilga wrote:
> At 2:03 PM -0500 3/21/02, Aaron Ross wrote:
>>
>>> I'm testing with the Perl script below, with the filename ending
>>> ".mperl" (which, in my configuration, causes it to run as a mod_perl
>>> registry script).
>>
>> I would re-write it as a handler and see if Apache::Registry is partly
>>to blame.
>
> I tried doing it as a handler, using the configuration below (and the
> appropriate changes in the source) and the problem persists. So it
> doesn't seem to be Registry's fault.
>
> <Location /dan>
> SetHandler perl-script
> PerlHandler DanTest
> </Location>
>
> ---- source code ----
>
> #!/usr/bin/perl
>
> package DanTest;
>
> use strict;
> use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
>
> my $dir='/home/httpd/some/path';
>
> sub handler {
> system( "rm $dir/__db* $dir/TESTdb" );
>
> foreach( 1..5 ) {
> my $env = open_env($dir);
> my %hash;
> my $db = open_db( "TESTdb", \%hash, $env );
> untie %hash;
> undef $db;
> undef $env;
> }
> print "HTTP/1.1 200\nContent-type: text/plain\n\n";
> print `db_stat -c -h $dir`;
> print "\n";
> }
>
> sub open_env {
> my $env = new BerkeleyDB::Env(
> -Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
> -Home=> $_[0],
> );
> die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
> return $env;
> }
>
> sub open_db {
> my( $file, $Rhash, $env ) = @_;
> my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> -Flags=>DB_CREATE,
> -Filename=>$file,
> -Env=>$env );
> die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> return $db_key;
> }
>
> 1;
>
>
> Dan Wilga dwilga@mtholyoke.edu
> Web Technology Specialist http://www.mtholyoke.edu
> Mount Holyoke College Tel: 413-538-3027
> South Hadley, MA 01075 "Seduced by the chocolate side of the Force"
>
===
Date: Fri, 22 Mar 2002 00:08:41 -0600
From: Aaron Ross <ross@alias-i.com>
To: Dan Wilga <dwilga@MtHolyoke.edu>
Cc: modperl@perl.apache.org
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
> >> I'm testing with the Perl script below, with the filename ending
> >> ".mperl" (which, in my configuration, causes it to run as a mod_perl
> >> registry script).
> >
> > I would re-write it as a handler and see if Apache::Registry is partly
> >to blame.
>
> I tried doing it as a handler, using the configuration below (and the
> appropriate changes in the source) and the problem persists. So it
> doesn't seem to be Registry's fault.
Bummer. I wish I had a machine to test this code on, but i won't until the
end of next week. it's ugly, but you could use Devel::Peek to look at why
garbage collection isn't working, maybe there is a circular reference in
BerkeleyDB.
Ugh. Please keep me informed.
-- Aaron
===
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
From: Aaron Ross <ross@alias-i.com>
To: Ed Grimm <ed@tgape.org>
Cc: mod_perl list <modperl@perl.apache.org>
Date: 22 Mar 2002 08:59:54 -0500
> (As an aside, I'm not sure I'll ever get over undef being proper closing
> of a database connection; it seems so synonomous to free([23]). I
> expect something like $db->db_close() or something.)
You mean like $db->db_close()? :)
http://sleepycat.com/docs/api_c/db_close.html
> > my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> > -Flags=>DB_CREATE,
> > -Filename=>$file,
> > -Env=>$env );
> > die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> > return $db_key;
I was wondering if using tie() instead of just the OO interface was
causing a problem. Maybe rewriting the code to just use new() would
help? Of course that means losing the hash interface.
grasping at straws, aaron
===
Date: Fri, 22 Mar 2002 09:32:00 -0500
To: Ed Grimm <ed@tgape.org>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
At 12:27 AM -0500 3/22/02, Ed Grimm wrote:
>Does shutting down apache free up your locks?
Nope. They are held open in the environment files (__db00*). When the
server is restarted, the count increases where it left off.
Just to be clear, the items left allocated are what DB calls
"lockers", which are different from actual "locks". A locker is just
a slot for data concerning an active lock (think of it as an array
index).
Normally, DB will grow the array by one, store info about the lock in
it, and then when the lock is freed, shrink the array. Instead,
what's happening is that the lock itself is freed, but the array
doesn't shrink (the locker isn't freed). So the next time a new lock
is required, it allocates another locker. If you do this enough (1000
times), the environment file runs out of room, and opens fail.
>(As an aside, I'm not sure I'll ever get over undef being proper closing
>of a database connection; it seems so synonomous to free([23]). I
>expect something like $db->db_close() or something.)
Agreed!
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "I have a bold and cunning plan" - Baldric
===
Date: Fri, 22 Mar 2002 17:20:42 -0500
To: Aaron Ross <ross@alias-i.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
Aaron Ross wrote:
> > > my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
> > > -Flags=>DB_CREATE,
> > > -Filename=>$file,
> > > -Env=>$env );
> > > die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
> > > return $db_key;
>
>I was wondering if using tie() instead of just the OO interface was
>causing a problem. Maybe rewriting the code to just use new() would
>help? Of course that means losing the hash interface.
Of course, that would avoid the problem, because I wouldn't be using
DB's built-in locking anymore :-) Using tie() means losing the
environment, which is needed for DB's internal locking.
In a production environment with thousands of hits per hour coming in
on multiple threads, I really don't trust flock() to do the job.
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Who left the cake out in the rain?"
===
Date: Mon, 25 Mar 2002 12:12:31 -0600
From: Jim Smith <jgsmith@moya.tamu.edu>
To: Michael Alan Dorman <mdorman@debian.org>
Cc: modperl@apache.org
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
On Thu, Mar 21, 2002 at 02:01:49PM -0500, Michael Alan Dorman wrote:
> Perrin Harkins <perrin@elem.com> writes:
> > This sort of begs the question: why not use DB 3.x? Is there some new
> > feature you need in DB 4?
>
> Anecdotaly, I believe the OpenLDAP and Cyrus projects have both found
> DB4 to be more reliable under load than DB3.
To add to the anecdotes...
We have been using the predecessor of Sendmail, Inc's SAMS product
(ultimately derived from Cyrus, I believe) and have seen some
problems with locking and sleepycat db's resulting in hanging IMAP
servers or database corruption. I would not blame mod_perl first --
it's not responsible for the database.
I'm not certain that the problem is in the IMAP server either -- but
the environment consists of the IMAP server, POP server, tcl scripts,
and web interface all trying to access a sleepycat database.
Sendmail, Inc. has mentioned (I believe privately) that they feel the
problems have been resolved in the newer versions of the sleepycat
db. I'm not trying to put words in their mouth, so I could be
mistaken, but this is afaik.
Before blaming mod_perl, I would run the server with the -X option to
make sure the script is only being called once within the mod_perl
environment (just being paranoid). Multiple (almost) simultaneous
invocations might be causing some of the locking problems. They
wouldn't show up then when being run from the command line.
my $0.02
--jim
===
Date: Wed, 3 Apr 2002 11:29:00 -0500
To: Aaron Ross <ross@alias-i.com>
From: Dan Wilga <dwilga@MtHolyoke.edu>
Subject: Re: Berkeley DB 4.0.14 not releasing lockers under mod_perl
Cc: modperl@perl.apache.org
Based on others' comments, I found the solution to the problem. I
just added a db_close(), and the problem is really, truly gone. For
good measure, I put a db_sync() in as well.
Why db_close isn't necessary with 3.x, but is with 4.0 is the weird
part. Maybe this suggests a bug in 4.0? Or is it just bad style to
have left it out in the first place? I was under the impression that
the untie/undef should have accomplished this implicitly within
BerkeleyDB.pm.
Here is the modified test script:
--------- TEST SCRIPT ---------
#!/usr/bin/perl
use strict;
use BerkeleyDB qw( DB_CREATE DB_INIT_MPOOL DB_INIT_CDB );
# Change me to something appropriate for your system
my $dir='/home/httpd/some/directory';
system( "rm $dir/__db* $dir/TESTdb" );
foreach( 1..5 ) {
my $env = open_env($dir);
my %hash;
my $db = open_db( "TESTdb", \%hash, $env );
$db->db_sync(); ### NEW
$db->db_close(); ### NEW
untie %hash;
undef $db;
undef $env;
}
print "HTTP/1.1 200\nContent-type: text/plain\n\n";
print `db_stat -c -h $dir`;
print "\n";
sub open_env {
my $env = new BerkeleyDB::Env(
-Flags=>DB_INIT_MPOOL|DB_INIT_CDB|DB_CREATE,
-Home=> $_[0],
);
die "Could not create env: $! ".$BerkeleyDB::Error. "\n" if !$env;
return $env;
}
sub open_db {
my( $file, $Rhash, $env ) = @_;
my $db_key = tie( %{$Rhash}, 'BerkeleyDB::Btree',
-Flags=>DB_CREATE,
-Filename=>$file,
-Env=>$env );
die "Can't open $file: $! ".$BerkeleyDB::Error."\n" if !$db_key;
return $db_key;
}
Dan Wilga dwilga@mtholyoke.edu
Web Technology Specialist http://www.mtholyoke.edu
Mount Holyoke College Tel: 413-538-3027
South Hadley, MA 01075 "Seduced by the chocolate side of the Force"