This is part of The Pile, a partial archive of some open source mailing lists and newsgroups.
To: "Ken Williams" <ken@forum.swarthmore.edu>
From: Alex Krohn <alex@gossamer-threads.com>
Subject: Re: memory leaking with closures
Date: Fri, 07 Sep 2001 00:27:58 -0700
Hi,
> >> With this simple test script:
> >>
> >> print "Content-type: text/html\n\n";
> >> my $var = 'x' x 500000;
> >
> > our $var = 'x' x 500000;
> >
> >>
> >> my $sub = sub { my $sub2 = sub { $var; }; };
> >> print "Done\n";
> >>
> >> $var does not get freed, and the process grows each request. Has anyone
> >> seen this sort of behavior and have any ideas/workarounds (besides don't
> >
> > yes. this is the documented behavior for closures. the anonymous sub
> > must have it's own copy of $var which happens to be 500k plus perl
> > overhead on each invocation.
>
> I think the problem is that $var doesn't get freed even when $var and
> $sub and $sub2 have all gone out of scope. That's a perl bug for which I
> can fathom no workaround.
>
> If any of those variables are still in scope, then of course $var can't
> get freed.
Yes, upon further investigation, this perl script:
while (1) {
{
my $var = 'x' x 500000;
my $sub = sub { my $sub2 = sub { $var; } };
}
# $var and $sub should be gone, but memory is never freed
sleep 1; # Don't crash things =)
}
will grow forever as $var never seems to be freed even when everything
should go out of scope, definately a perl bug, not a mod_perl one. =)
Oddly, if you just do
my $sub = sub { $var; };
it does not grow, definately something strange going on. Happens on
perl 5.004_04, 5.005_03 and 5.6.1.
===
To: Alex Krohn <alex@gossamer-threads.com>
From: Stas Bekman <stas@stason.org>
Subject: Re: memory leaking with closures
Date: Fri, 7 Sep 2001 18:16:47 +0800 (SGT)
On Fri, 7 Sep 2001, Alex Krohn wrote:
> Hi,
>
> > >> With this simple test script:
> > >>
> > >> print "Content-type: text/html\n\n";
> > >> my $var = 'x' x 500000;
> > >
> > > our $var = 'x' x 500000;
> > >
> > >>
> > >> my $sub = sub { my $sub2 = sub { $var; }; };
> > >> print "Done\n";
> > >>
> > >> $var does not get freed, and the process grows each request. Has anyone
> > >> seen this sort of behavior and have any ideas/workarounds (besides don't
> > >
> > > yes. this is the documented behavior for closures. the anonymous sub
> > > must have it's own copy of $var which happens to be 500k plus perl
> > > overhead on each invocation.
> >
> > I think the problem is that $var doesn't get freed even when $var and
> > $sub and $sub2 have all gone out of scope. That's a perl bug for which I
> > can fathom no workaround.
> >
> > If any of those variables are still in scope, then of course $var can't
> > get freed.
>
> Yes, upon further investigation, this perl script:
>
> while (1) {
> {
> my $var = 'x' x 500000;
> my $sub = sub { my $sub2 = sub { $var; } };
> }
> # $var and $sub should be gone, but memory is never freed
> sleep 1; # Don't crash things =)
> }
>
> will grow forever as $var never seems to be freed even when everything
> should go out of scope, definately a perl bug, not a mod_perl one. =)
Look like memory leaking to me. The reference count is done incorrectly,
which leads to memory leaking:
use GTop;
use Devel::Peek;
my $gtop = GTop->new;
for (1..3) {
my $a = 'x' x 500000;
{
sub { $a; };
}
printf "RERCNT %s: \$a=%d\n",
GTop::size_string($gtop->proc_mem($$)->rss),
Devel::Peek::SvREFCNT($a);
}
prints:
RERCNT 3.5M: $a=1
RERCNT 3.5M: $a=1
RERCNT 3.5M: $a=1
all fine!
use GTop;
use Devel::Peek;
my $gtop = GTop->new;
for (1..3) {
my $a = 'x' x 500000;
{
sub { my $sub2 = sub { $a; }; };
}
printf "RERCNT %s: \$a=%d\n",
GTop::size_string($gtop->proc_mem($$)->rss),
Devel::Peek::SvREFCNT($a);
}
prints:
RERCNT 3.5M: $a=4
RERCNT 4.0M: $a=3
RERCNT 4.5M: $a=3
1. the reference counting is wrong
2. memory leaks (because of 1)
This is a hack to fix the leaking (well, almost):
use GTop;
use Devel::Peek;
my $gtop = GTop->new;
for (1..3) {
my $a = 'x' x 500000;
{
sub { my $sub2 = sub { $a; }; };
Devel::Peek::SvREFCNT_dec($a);
Devel::Peek::SvREFCNT_dec($a);
}
printf "RERCNT %s: \$a=%d\n",
GTop::size_string($gtop->proc_mem($$)->rss),
Devel::Peek::SvREFCNT($a);
}
prints:
RERCNT 3.5M: $a=2
RERCNT 4.0M: $a=1
RERCNT 4.0M: $a=1
> Oddly, if you just do
>
> my $sub = sub { $var; };
>
> it does not grow, definately something strange going on. Happens on
> perl 5.004_04, 5.005_03 and 5.6.1.
You mean this works for you?:
while (1) {
{
my $var = 'x' x 500000;
my $sub = sub { $var; };
}
# $var and $sub should be gone, but memory is never freed
sleep 1; # Don't crash things =)
}
that's because your $sub goes out of scope. the other one seems like a
bug.
If I didn't miss something, it seems that we need to run this through p5p.
===
To: "Stas Bekman" <stas@stason.org>, "Alex Krohn"
<alex@gossamer-threads.com>
From: "Perrin Harkins" <perrin@elem.com>
Subject: Re: memory leaking with closures
Date: Fri, 7 Sep 2001 10:23:38 -0400
> > Oddly, if you just do
> >
> > my $sub = sub { $var; };
> >
> > it does not grow, definately something strange going on. Happens on
> > perl 5.004_04, 5.005_03 and 5.6.1.
>
> You mean this works for you?:
>
> while (1) {
> {
> my $var = 'x' x 500000;
> my $sub = sub { $var; };
> }
> # $var and $sub should be gone, but memory is never freed
> sleep 1; # Don't crash things =)
> }
>
> that's because your $sub goes out of scope.
The leaking behavior only happens when you use nested anonymous subs. It
should always be possible to avoid it, although your code may look uglier.
> the other one seems like a
> bug.
>
> If I didn't miss something, it seems that we need to run this through p5p.
It's been known about for a while. I'm not sure what the status of getting
a fix is. I assume that it must be difficult to fix or it would have been
changed a long time ago.
===
To: Perrin Harkins <perrin@elem.com>
From: Stas Bekman <stas@stason.org>
Subject: Re: memory leaking with closures
Date: Fri, 7 Sep 2001 22:47:56 +0800 (SGT)
On Fri, 7 Sep 2001, Perrin Harkins wrote:
> > > Oddly, if you just do
> > >
> > > my $sub = sub { $var; };
> > >
> > > it does not grow, definately something strange going on. Happens on
> > > perl 5.004_04, 5.005_03 and 5.6.1.
> >
> > You mean this works for you?:
> >
> > while (1) {
> > {
> > my $var = 'x' x 500000;
> > my $sub = sub { $var; };
> > }
> > # $var and $sub should be gone, but memory is never freed
> > sleep 1; # Don't crash things =)
> > }
> >
> > that's because your $sub goes out of scope.
>
> The leaking behavior only happens when you use nested anonymous subs. It
> should always be possible to avoid it, although your code may look uglier.
It only happens if you use double-nested subs, as far as I've tested with
Devel::Peek. Is it documented somewhere in the perl docs?
> > the other one seems like a
> > bug.
> >
> > If I didn't miss something, it seems that we need to run this through p5p.
>
> It's been known about for a while. I'm not sure what the status of getting
> a fix is. I assume that it must be difficult to fix or it would have been
> changed a long time ago.
Do you say that it would be a waste to raise this issue on p5p?
It's very hard to keep up with the traffic on that list along with
dev@httpd.apache.org, so I'm not aware of many issues discussed there.
unfortunately :(
===
To: Alex Krohn <alex@gossamer-threads.com>
From: Barrie Slaymaker <barries@slaysys.com>
Subject: Re: memory leaking with closures
Date: Fri, 7 Sep 2001 10:50:13 -0400
On Fri, Sep 07, 2001 at 12:27:58AM -0700, Alex Krohn wrote:
> while (1) {
> {
> my $var = 'x' x 500000;
> my $sub = sub { my $sub2 = sub { $var; } };
> }
> # $var and $sub should be gone, but memory is never freed
> sleep 1; # Don't crash things =)
> }
>
> will grow forever
> it does not grow, definately something strange going on. Happens on
> perl 5.004_04, 5.005_03 and 5.6.1.
confirmed in bleadperl (patch 11936). CCing p5p.
atleast one sub { sub{} } leak was fixed recently, but not this one.
===
To: "Arthur Bergman" <arthur@contiller.se>
From: Gurusamy Sarathy <gsar@ActiveState.com>
Subject: Re: memory leaking with closures
Date: Fri, 07 Sep 2001 11:00:30 -0700
On Fri, 07 Sep 2001 18:06:17 +0200, "Arthur Bergman" wrote:
>> On Fri, Sep 07, 2001 at 12:27:58AM -0700, Alex Krohn wrote:
>> > while (1) {
>> > {
>> > my $var = 'x' x 500000;
>> > my $sub = sub { my $sub2 = sub { $var; } };
>> > }
>> > # $var and $sub should be gone, but memory is never freed
>> > sleep 1; # Don't crash things =)
>> > }
>> >
>> > will grow forever
>>
>> > it does not grow, definately something strange going on. Happens on
>> > perl 5.004_04, 5.005_03 and 5.6.1.
>>
>> confirmed in bleadperl (patch 11936). CCing p5p.
>>
>> atleast one sub { sub{} } leak was fixed recently, but not this one.
>>
>> - Barrie
>
>Seems like we are not properly freeing the prototype CV which is cloned.
Not likely, since there are always a fixed number of closure prototypes
when there is no eval"" to create new ones.
It is more likely that the reference loop between the inner and outer
CVs is preventing the freeing of either of them. I'm not in fact sure
that the reference loop *can* be eliminated trivially, given these two
CVs can have different lifetimes. Perhaps the right solution is to
move to using weakrefs for CvOUTSIDE(), I dunno.
===
To: "Barrie Slaymaker" <barries@slaysys.com>,
From: "Arthur Bergman" <arthur@contiller.se>
Subject: RE: memory leaking with closures
Date: Fri, 7 Sep 2001 18:06:17 +0200
> On Fri, Sep 07, 2001 at 12:27:58AM -0700, Alex Krohn wrote:
> > while (1) {
> > {
> > my $var = 'x' x 500000;
> > my $sub = sub { my $sub2 = sub { $var; } };
> > }
> > # $var and $sub should be gone, but memory is never freed
> > sleep 1; # Don't crash things =)
> > }
> >
> > will grow forever
>
> > it does not grow, definately something strange going on. Happens on
> > perl 5.004_04, 5.005_03 and 5.6.1.
>
> confirmed in bleadperl (patch 11936). CCing p5p.
>
> atleast one sub { sub{} } leak was fixed recently, but not this one.
>
> - Barrie
Seems like we are not properly freeing the prototype CV which is cloned.
===
To: "Gurusamy Sarathy" <gsar@ActiveState.com>
From: "Arthur Bergman" <arthur@contiller.se>
Subject: SV: memory leaking with closures
Date: Fri, 7 Sep 2001 20:18:28 +0200
> On Fri, 07 Sep 2001 18:06:17 +0200, "Arthur Bergman" wrote:
> >> On Fri, Sep 07, 2001 at 12:27:58AM -0700, Alex Krohn wrote:
> >> > while (1) {
> >> > {
> >> > my $var = 'x' x 500000;
> >> > my $sub = sub { my $sub2 = sub { $var; } };
> >> > }
> >> > # $var and $sub should be gone, but memory is never freed
> >> > sleep 1; # Don't crash things =)
> >> > }
> >> >
> >> > will grow forever
> >>
> >> > it does not grow, definately something strange going on. Happens on
> >> > perl 5.004_04, 5.005_03 and 5.6.1.
> >>
> >> confirmed in bleadperl (patch 11936). CCing p5p.
> >>
> >> atleast one sub { sub{} } leak was fixed recently, but not this one.
> >>
> >> - Barrie
> >
> >Seems like we are not properly freeing the prototype CV which is cloned.
>
> Not likely, since there are always a fixed number of closure prototypes
> when there is no eval"" to create new ones.
Silly me.
> It is more likely that the reference loop between the inner and outer
> CVs is preventing the freeing of either of them. I'm not in fact sure
> that the reference loop *can* be eliminated trivially, given these two
> CVs can have different lifetimes. Perhaps the right solution is to
> move to using weakrefs for CvOUTSIDE(), I dunno.
>
Sounds like an idea anyway. I will give it a try. Only
problem I see it will be messed up after perl_clone.
===