Severe Memory Leak

RT Developers
1

I am experiencing quite a severe memory leak with RT. We have the latest
version of RT (3.0.5) and the Latest RTFM (2.0.0RC4) installed on an
Apache Server version 1.3.27 with ModPerl1.

Perl is version 5.8.0 on kernel 2.4.20-18.7. with all pre-requisite
modules up to date as per installation scripts (all modules downloaded
within the last 2 weeks)

MySQL version mysql-max-4.0.15

The system is lightly loaded with perhaps 50 tickets in total. If a
browser is left with ‘Support at a glance’ page showing, and auto
refresh set to every 2 minutes, the system leaks memory every refresh –
anywhere between 100 bytes and 4 or 5k per refresh.

This slowly burns all physical memory, then starts to eat virtual memory
and eventually takes the server down.

If I do a stop of apache, wait a couple of minutes for the system to
stabilise and then re-start apache, lots of memory is returned to the
system

Has anyone else seen this/got any fantastic suggestions as to how I may
debug it ? It is clearly killing my users perception of RT as a reliable
system if the server keeps going down,

Thanks,

Simon

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

2

I am experiencing quite a severe memory leak with RT. We have the latest
version of RT (3.0.5) and the Latest RTFM (2.0.0RC4) installed on an
Apache Server version 1.3.27 with ModPerl1.

I’d be curious to hear if you can reproduce this with RT’s fastcgi
handler. (Just so we can try to isolate the issue.)

Also, what modperl_1?

The system is lightly loaded with perhaps 50 tickets in total. If a
browser is left with ‘Support at a glance’ page showing, and auto
refresh set to every 2 minutes, the system leaks memory every refresh –
anywhere between 100 bytes and 4 or 5k per refresh.

Does constant reload of other pages trigger the same apparent leak?

This slowly burns all physical memory, then starts to eat virtual memory
and eventually takes the server down.

If I do a stop of apache, wait a couple of minutes for the system to
stabilise and then re-start apache, lots of memory is returned to the
system

Well, that’s always going to be the case, whether there’s a leak or not.
Perl doesn’t “return” memory to the system while a given process is
running.

Has anyone else seen this/got any fantastic suggestions as to how I may
debug it? It is clearly killing my users perception of RT as a reliable
system if the server keeps going down,

In the meantime, why don’t you tune-down the max # of clients that a
process serves before being recycled? That should free up memory that’s
being leaked until we get to the bottom of what’s going on.

-j

Thanks,

Simon

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

3

One additional thing I have just noticed is that the system seems to be
accumulating lots of MySQL connections, rather than re-using
connections. This could certainly be contributing to the problems ?

In answer to your other questions, it is modperl_1

Also, what modperl_1?
Modperl-1.28

I’d be curious to hear if you can reproduce this with RT’s fastcgi
handler.
Will happily try this if necessary

Does constant reload of other pages trigger the same apparent leak?
The worst offender is the Support at a glance, but other pages do
exhibit similar behaviour

In the meantime, why don’t you tune-down the max # of clients that a
process serves before being recycled?
That should free up memory that’s being leaked until we get to the
bottom of what’s going on.

Good suggestion for a workaround, thanks

SimonFrom: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 17:50
To: Simon Talbot
Cc: rt-devel@lists.fsck.com
Subject: Re: [rt-devel] Severe Memory Leak

I am experiencing quite a severe memory leak with RT. We have the
latest version of RT (3.0.5) and the Latest RTFM (2.0.0RC4) installed
on an Apache Server version 1.3.27 with ModPerl1.

I’d be curious to hear if you can reproduce this with RT’s fastcgi
handler. (Just so we can try to isolate the issue.)

Also, what modperl_1?

The system is lightly loaded with perhaps 50 tickets in total. If a
browser is left with ‘Support at a glance’ page showing, and auto
refresh set to every 2 minutes, the system leaks memory every refresh
– anywhere between 100 bytes and 4 or 5k per refresh.

Does constant reload of other pages trigger the same apparent leak?

This slowly burns all physical memory, then starts to eat virtual
memory and eventually takes the server down.

If I do a stop of apache, wait a couple of minutes for the system to
stabilise and then re-start apache, lots of memory is returned to the
system

Well, that’s always going to be the case, whether there’s a leak or not.
Perl doesn’t “return” memory to the system while a given process is
running.

Has anyone else seen this/got any fantastic suggestions as to how I
may debug it? It is clearly killing my users perception of RT as a
reliable system if the server keeps going down,

In the meantime, why don’t you tune-down the max # of clients that a
process serves before being recycled? That should free up memory that’s
being leaked until we get to the bottom of what’s going on.

-j

Thanks,

Simon

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

4

Try removing Apache::DBI from the mix?On Wed, Sep 24, 2003 at 06:13:05PM +0100, Simon Talbot wrote:

Yes, as per instructions – stock install

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:10
To: Simon Talbot
Subject: Re: [rt-devel] Severe Memory Leak

On Wed, Sep 24, 2003 at 06:04:08PM +0100, Simon Talbot wrote:

One additional thing I have just noticed is that the system seems to
be accumulating lots of MySQL connections, rather than re-using
connections. This could certainly be contributing to the problems ?

That could be the entirety of the problem. Are you using Apache::DBI?

–
Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

5

How would I do that easily ? Surely there are way too many dependencies
?

Just comment it out in your apache config. Apache::DBI frobs the symbol
table to get itself into the mix.

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

The information contained in this e-mail and any attachments are private
and confidential and may be legally privileged.

It is intended for the named addressee(s) only. If you are not the
intended recipient(s), you must not read, copy or use the information
contained in any way. If you receive this email or any attachments in
error, please notify us immediately by e-mail or by telephoning +44
(01206) 216400.

Net Solutions Europe Limited accepts no responsibility for any loss or
damages whatsoever arising in any way from receipt or use of this e-mail
or any attachments. This e-mail is not intended to create legally
binding commitments on behalf of Net Solutions Europe Limited, nor do
its comments reflect the corporate views or policies of Net Solutions
Europe Limited.

-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:18
To: Simon Talbot
Cc: rt-devel@fsck.com
Subject: Re: [rt-devel] Severe Memory Leak

Try removing Apache::DBI from the mix?

Yes, as per instructions – stock install

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:10
To: Simon Talbot
Subject: Re: [rt-devel] Severe Memory Leak

One additional thing I have just noticed is that the system seems to

be accumulating lots of MySQL connections, rather than re-using
connections. This could certainly be contributing to the problems
?

That could be the entirety of the problem. Are you using Apache::DBI?

–
Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

–
Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

6

Have done so and will monitor the server – although it is still
creating 8 or 10 SQL connections – and increasing, so seems to be
behaving in a fairly similar manner,

How many connections should the RT system create as it runs ?

Thanks,

SimonFrom: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:22
To: Simon Talbot
Cc: rt-devel@lists.fsck.com
Subject: Re: [rt-devel] Severe Memory Leak

How would I do that easily ? Surely there are way too many
dependencies ?

Just comment it out in your apache config. Apache::DBI frobs the symbol
table to get itself into the mix.

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

The information contained in this e-mail and any attachments are
private and confidential and may be legally privileged.

It is intended for the named addressee(s) only. If you are not the
intended recipient(s), you must not read, copy or use the information
contained in any way. If you receive this email or any attachments in
error, please notify us immediately by e-mail or by telephoning +44
(01206) 216400.

Net Solutions Europe Limited accepts no responsibility for any loss or

damages whatsoever arising in any way from receipt or use of this
e-mail or any attachments. This e-mail is not intended to create
legally binding commitments on behalf of Net Solutions Europe Limited,

nor do its comments reflect the corporate views or policies of Net
Solutions Europe Limited.

-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:18
To: Simon Talbot
Cc: rt-devel@fsck.com
Subject: Re: [rt-devel] Severe Memory Leak

Try removing Apache::DBI from the mix?

Yes, as per instructions – stock install

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 216400
Fax: 01206 216420

-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: 24 September 2003 18:10
To: Simon Talbot
Subject: Re: [rt-devel] Severe Memory Leak

One additional thing I have just noticed is that the system seems
to

be accumulating lots of MySQL connections, rather than re-using
connections. This could certainly be contributing to the
problems
?

That could be the entirety of the problem. Are you using
Apache::DBI?

–
Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

–
Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

7

Have done so and will monitor the server – although it is still
creating 8 or 10 SQL connections – and increasing, so seems to be
behaving in a fairly similar manner,

How many connections should the RT system create as it runs ?

Roughly, one per httpd server thread.

Thanks,

Simon

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

8

That is what I thought – so what I don’t get, is why when I am using a
single browser with no other clients connected I can generate 10 or 12
rt_user connections to SQL server just by refreshing a single page, I
also seem to be creating pairs of httpd processes as though the apache
server is detecting my page refreshes as new clients and starting new
httpd process with their own SQL connections etc… If you see what I
mean

I can’t speak to how apache decides whether to allocate an existing
process to your request or a new one. That, at least, is very much
outside RT’s hands.

-jesse

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

9

“ST” == Simon Talbot simont@nse.co.uk writes:

ST> The system is lightly loaded with perhaps 50 tickets in total. If a
ST> browser is left with ‘Support at a glance’ page showing, and auto
ST> refresh set to every 2 minutes, the system leaks memory every refresh –
ST> anywhere between 100 bytes and 4 or 5k per refresh.

Perhaps switching to a process model where the processes are not
persistent for so long will help. The thing with mod_perl is that the
perl interpreter is persistent, so any memory it allocates will not be
released to the system upon termination of the request. This is not a
memory leak – it is just how it works. The perl process has gotten
the memory from the system and can reuse it many times to satisfy the
memory allocation needs of the code.

Your options are to either reduce the number of requests each mod_perl
process handles (MaxRequests in httpd.conf) or to use something like
speedy cgi with a kill idle processes after X minutes of idle time so
that the memory is released back to the system.

10

Perhaps switching to a process model where the processes are not
persistent for so long will help. The thing with mod_perl is that the
perl interpreter is persistent, so any memory it allocates will not be
released to the system upon termination of the request. This is not a
memory leak – it is just how it works. The perl process has gotten
the memory from the system and can reuse it many times to satisfy the
memory allocation needs of the code.

At the same time, we do want to track down cases where RT or its’
dependencies are being anti-social :wink:

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

11

I hear what you say, but this would not account for the system getting
to a point where it runs out of physical and virtual memory and you end
with a kernel oops – something is definitely leaking and of course you
can fix it by brute force, but isolating the errant module/area of code
has to be preferable. This certainly does provide me with another
workaround whilst we investigate further,

Thanks,

Simon

Simon Talbot MEng, ACGI
(Chief Engineer)
Net Solutions Europe Limited
Tel: 01206 274210
Fax: 01255 233305-----Original Message-----
From: Vivek Khera [mailto:khera@kcilink.com]
Sent: 24 September 2003 19:42
To: rt-devel@lists.fsck.com
Subject: Re: [rt-devel] Severe Memory Leak

“ST” == Simon Talbot simont@nse.co.uk writes:

ST> The system is lightly loaded with perhaps 50 tickets in total. If a
ST> browser is left with ‘Support at a glance’ page showing, and auto
ST> refresh set to every 2 minutes, the system leaks memory every
refresh –
ST> anywhere between 100 bytes and 4 or 5k per refresh.

Perhaps switching to a process model where the processes are not
persistent for so long will help. The thing with mod_perl is that the
perl interpreter is persistent, so any memory it allocates will not be
released to the system upon termination of the request. This is not a
memory leak – it is just how it works. The perl process has gotten
the memory from the system and can reuse it many times to satisfy the
memory allocation needs of the code.

Your options are to either reduce the number of requests each mod_perl
process handles (MaxRequests in httpd.conf) or to use something like
speedy cgi with a kill idle processes after X minutes of idle time so
that the memory is released back to the system.
rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

12

At Wed, 24 Sep 2003 21:02:56 +0100,
Simon Talbot wrote:

I hear what you say, but this would not account for the system getting
to a point where it runs out of physical and virtual memory and you end
with a kernel oops – something is definitely leaking and of course you
can fix it by brute force, but isolating the errant module/area of code
has to be preferable. This certainly does provide me with another
workaround whilst we investigate further,

The kernel oops points to something else going on.

- Do you have memory overcommit activated?  If so, turn it off.
 ~$ cat /proc/sys/vm/overcommit_memory 
 0

  The kernel should kill the big process, which you didn't
  identify.  Is it apache?  Is it mysql?  Is it
  netscape/mozilla/opera/konqueror?

- You may have bad ram.  Download memtest86 (if you are running on
  the appropriate hardware) and test.

- You did not tell us what the oops is.  I don't care what the CPU
  registers were, but it would be nice to know where the oops
  happened.

I highly doubt that this is a user-space configuration problem.

-R

13

Jesse,

The server seems to be considerably more stable with DBI removed from
the configuration, the system seems to do much better garbage
collection. Have you experienced problems like this before ?

Secondly, is there a limit to the size of attachments that RT allows in
a ticket, it seems that on our systems, small attachments are fine, but
larger attachments (2MB ish) just get ignored ? Is this a soft limit –
I can’t seem to find reference to it anywhere ?

Simon

14

Secondly, is there a limit to the size of attachments that RT allows in
a ticket, it seems that on our systems, small attachments are fine, but
larger attachments (2MB ish) just get ignored ? Is this a soft limit –
I can’t seem to find reference to it anywhere ?

Regardless of possible other reasons, if you’re running MySQL checkout the
max_allowed_packet parameter. That is a “hardcoded” limit.

Koos Pol
Release Engineering - Compuware Europe B.V.
T: +31 20 3116122 E: koos_pol@nl.compuware.com

15
  • Simon Talbot [2003-09-29 07:36]:

Secondly, is there a limit to the size of attachments that RT allows
in a ticket, it seems that on our systems, small attachments are fine,
but larger attachments (2MB ish) just get ignored ? Is this a soft
limit – I can’t seem to find reference to it anywhere ?

Are you referring to the $MaxAttachmentSize variable?

In etc/RT_Config.pm, I see:

$MaxAttachmentSize sets the maximum size (in bytes) of attachments stored

in the database.

For mysql and oracle, we set this size at 10 megabytes.

If you’re running a postgres version earlier than 7.1, you will need

to drop this to 8192. (8k)

Set($MaxAttachmentSize , 10000000);

Try fiddling with that (after checking for database-imposed limits, as
someone else suggested).

(darren)

To me vi is Zen. To use vi is to practice zen.
Every command is a koan. Profound to the user,
unintelligible to the uninitiated. You discover
truth everytime you use it.
– Reddy