This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
Subject: Re: perl's memory leak
From: Stas Bekman <stas@stason.org>
Date: Thu, 7 Dec 2000 19:58:41 +0100 (CET)
On Thu, 7 Dec 2000, Ivan E. Panchenko wrote:
>
>
> Today I discovered a strange behaiviour of perl,
> and I wonder if anybody can tell me what to do with it.
>
> The matter is that perl DOES NOT REUSE MEMORY allocated for
> intermediate calculation results. This is specially harmful to
> data-intensive modperl applications where one perl process processes
> many queries and can leak great amount of memory.
When you will rerun the same code again (the same request) in the same
httpd process, the memory won't grow. There is no leak, that's the way
Perl optimizes memory allocation.
On the contrary, when you will rerun the same code in the same process,
it'll be executed even faster since the memory allocation for lexically
scoped variables $a and $b was done in the first execution. So even if you
undef them the memory is still allocated.
You can easily test it with httpd -X. See more examples of this kind of
testing with gtop and Devel::Peek in the performance chapter in the guide.
Of course if you don't like it, you can complain to p5p list :)
> The example is below:
>
> use BSD::Resource;
> my $cc = 'a' x 20000000 ; # alocates 20Mb for the right part and
> # 20Mb for $a
> &p;
> { my $a = $cc.'x'; # allocates 20 more Mb for right part
> # and 20 for a
> &p;
> undef $a; # deallocates $a
> }
> &p;
> { my $b = $cc.'y'; # allocates 20 more Mb for right part
> # and reuses deallocated 20Mb for b
> &p;
> undef $b;
> }
> &p;
>
> sub p {
> print STDERR "used memory = ".(BSD::Resource::getrusage)[2]."\n"
> }
>
> # end of example.
> Output:
> used memory = 40772
> used memory = 79804
> used memory = 80068
> used memory = 99676
> used memory = 99700
> ##
> Here I used BSD:Resource to measure consumed memory. Its result seems to
> be correlated with the amount of memory taken by the process from the OS.
> #
>
> This was checked on FreeBSD 3.4 and 4.2 ; and perl5.00405 5.00503 .
> Same things where noticed on Linux and probably on Solaris too.
===
Subject: Re: perl's memory leak
From: Perrin Harkins <perrin@primenet.com>
Date: Thu, 7 Dec 2000 11:24:00 -0800 (PST)
On Thu, 7 Dec 2000, Ivan E. Panchenko wrote:
> Today I discovered a strange behaiviour of perl,
> and I wonder if anybody can tell me what to do with it.
>
> The matter is that perl DOES NOT REUSE MEMORY allocated for
> intermediate calculation results. This is specially harmful to
> data-intensive modperl applications where one perl process processes
> many queries and can leak great amount of memory.
This is known and it's not really a leak. Perl knows about that memory
and does re-use it, the next time it needs that lexical variable. It's a
performance optimization. Try running your code multiple times and you
should see memory stay at the same level after the first run.
There has been discussion about this on the mailing list which you can
find in the archives. There has also been talk about changing this
behavior for mod_perl 2, which Doug is working on.
Anyway, if you just want the memory back, undef your lexicals after you
finish with them.
===
Subject: [OT] Re: perl's memory leak
From: Andrew Wyllie <wyllie@dilex.net>
Date: Thu, 7 Dec 2000 11:58:54 -0800
Hi Ivan,
It's not really mod_perl, but is relevant to people on the list I guess...
If you really play aorund with this, you'll find some interesting variations.
If I assign $cc using a for loop
my $c;
for ( 1..20000000) { $cc .= 'a'; }
it's a lot slower, but only uses half as much ram
also if I do the assignment first and then add a char:
my $a = $cc; # allocates 20 more Mb for right part
p( "assigned a" );
$a .= 'x';
p( "changed a" );
undef $a; # deallocates $a
p( "undefed a" );
my $b = $cc; # allocates 20 more Mb for right part
$b .= 'a';
# and reuses deallocated 20Mb for b
p( "defined b" );
undef $b;
p( "undefed b");
sub p {
my $mesg = shift;
print "$mesg used memory = ".(BSD::Resource::getrusage)[2]."\n"
}
I get:
assigned a used memory = 40212
changed a used memory = 40212
undefed a used memory = 40212
defined b used memory = 40712
undefed b used memory = 40712
Which is more what you would hope to see right?
===
Subject: Re: perl's memory leak
From: Stas Bekman <stas@stason.org>
Date: Thu, 7 Dec 2000 22:39:38 +0100 (CET)
On Thu, 7 Dec 2000 newsreader@mediaone.net wrote:
>
> The output I get is
>
> used memory = 0
> used memory = 0
> used memory = 0
> used memory = 0
> used memory = 0
I get the same under perl 5.6.0 on linux, looks like BSD::Resource doesn't
work there :( Anyone?
Use gtop instead (if you have it):
use GTop ();
print GTop->new->proc_mem($$)->size,"\n";
>
>
> I'm interested in how many leaks are possible in mod_perl
> though because my mod_perl processes are getting bigger
> with time -- about 200 requests is making the process
> fatter by 1mb on the average. I'm watching to see if
> they will max out around the current level of 10 mb per child.
>
===
Subject: Re: perl's memory leak
From: "Ivan E. Panchenko" <ivan@xray.sai.msu.ru>
Date: Fri, 8 Dec 2000 02:25:12 +0300 (MSK)
On Thu, 7 Dec 2000 newsreader@mediaone.net wrote:
>
> The output I get is
>
> used memory = 0
> used memory = 0
> used memory = 0
> used memory = 0
> used memory = 0
>
>
> I'm interested in how many leaks are possible in mod_perl
> though because my mod_perl processes are getting bigger
> with time -- about 200 requests is making the process
> fatter by 1mb on the average. I'm watching to see if
> they will max out around the current level of 10 mb per child.
You probably tried this script on linux or some other not fully BSD
compartible system. We obtained same zeros on linux, where getrusage()
means something else than on FreeBSD,
but if you try measuring memory sizes with ps or top, you should
observe the mentioned leak. Please insert sleep(10) after print STDERR
and watch "top".
Ivan
===
Subject: Re: perl's memory leak
From: Stas Bekman <stas@stason.org>
Date: Fri, 8 Dec 2000 19:39:57 +0100 (CET)
On Fri, 8 Dec 2000, mark warren bracher wrote:
> it seems as if most (if not all) the techniques for checking the size of
> the current process are _very_ platform specific. on linux you can use
>
> Apache::SizeLimit::linux_size_check
If you have linux you have (or can have GTop), which gives you an API to
do this and many other things. Apache::SizeLimit::linux_size_check is just
a custom function that you cannot really re-use (unless you put it into
some other module...
===
Subject: Re: perl's memory leak
From: Perrin Harkins <perrin@primenet.com>
Date: Fri, 8 Dec 2000 10:57:15 -0800 (PST)
On Fri, 8 Dec 2000, Stas Bekman wrote:
> If you have linux you have (or can have GTop), which gives you an API to
> do this and many other things. Apache::SizeLimit::linux_size_check is just
> a custom function that you cannot really re-use (unless you put it into
> some other module...
Unfortunately, GTop is kind of a pain to compile. It seems to depend on
some Gnome stuff. We use Apache::SizeLimit for this reason, and it works
well.
===
Subject: Re: perl's memory leak
From: "Ivan E. Panchenko" <ivan@xray.sai.msu.ru>
Date: Sat, 9 Dec 2000 19:53:38 +0300 (MSK)
No, i did not mean freeing memory from lexicals. I meant the memory
allocated for the temporary results, such as
my $a = 'x' x 1000000
Here perl allocates 1M for $a and 1M for evaluating the right part,
after that it is possible to undef $a and reuse its memory (1M),
but the right part memory (one more 1M) can be used nowhere except the
same line of code.
This is strange. I understand that it can be an optimization trick,
but when you frequently operate with megabytes, this becomes a
pleasant feature which looks and behaves like a memory leak :)
===
Subject: Re: perl's memory leak
From: mark warren bracher <bracher@citysearch.com>
Date: Fri, 08 Dec 2000 10:22:39 -0800
Stas Bekman wrote:
> On Thu, 7 Dec 2000 newsreader@mediaone.net wrote:
>
>
>> The output I get is
>>
>> used memory = 0
>> used memory = 0
>> used memory = 0
>> used memory = 0
>> used memory = 0
>
>
> I get the same under perl 5.6.0 on linux, looks like BSD::Resource doesn't
> work there :( Anyone?
>
> Use gtop instead (if you have it):
> use GTop ();
> print GTop->new->proc_mem($$)->size,"\n";
[snip]
it seems as if most (if not all) the techniques for checking the size of
the current process are _very_ platform specific. on linux you can use
Apache::SizeLimit::linux_size_check
which is just parsing /proc/self/status.
- mark
===
Subject: Re: perl's memory leak
From: Tim Bunce <Tim.Bunce@ig.co.uk>
Date: Sun, 10 Dec 2000 17:35:31 +0000
The long time mythical
use less qw(memory);
pragma was always intended to address issues like that. It would be
fairly straightforward to implement (set a flag like use strict and
check it wherever memory could usefully be freed).
===