Memory leak (RT 4.0.5)

RT Users
1

System:
RT version 4.0.5
Apache version 2.2.16
PostgreSQL version 8.4.11
Debian version 6.0.4
Kernel version 2.6.24-25-xen (64-bit)

All software is as supplied in the standard Debian repositories.

RT installation was upgraded from 3.8.7 using the rt-setup-database-4
script.

This is a Xen guest with 2Gb RAM allocated.

At the time of writing there are only 11,303 tickets.

As the system runs the available memory is slowly eaten up by Apache.
Even with just one active user, it can go from 1.1Gb free to needing
swap in under 2 hours, and sometimes less than 1 hour.

Most of the time the memory can be freed with apachectl graceful,
however, sometimes a full apachectl restart
is needed.

Here is some sample data before and after a reload of Apache:

Before reload:
PID DRS RSS %MEM
2914 628234 400104 19.0
19202 628234 393904 18.7
19220 1089194 575868 27.4
19248 1138122 554844 26.4
Total = 91.5

After reload:
PID DRS RSS %MEM
2914 628234 400104 19.0
19556 628234 393896 18.7
Total = 37.7

I’ve GTFW and not found any solutions.

Can anyone here point me in the right direction?

Note - I have a previous installation using 3.8.7 that runs fine.

Let me know if you need any more information.

Thanks.

Regards,
Austin.
Austin L. Denyer.
I. T. Server & Security Administrator.

signature.asc (262 Bytes)

2

System:
RT version 4.0.5
Apache version 2.2.16
PostgreSQL version 8.4.11
Debian version 6.0.4
Kernel version 2.6.24-25-xen (64-bit)

I’ve not used the Debian packaged version - is it using mod_perl, or mod_fastcgi or mod_fcgid? I found that using mod_fastcgi we had severe memory leak problems, so yes there does seem to be a memory leak in RT, but I don’t know where. However, I mitigated the problem by switching to mod_fcgid in the Apache config, because it has useful knobs you can tweak to tell it to start new CGI processes every so often, or after a certain number of requests have been handled. Thus no individual RT server process now lives long enough for the memory leak to be a significant problem. Here’s my /etc/apache2/mods-enabled/fcgid.conf:

AddHandler fcgid-script .fcgi FcgidConnectTimeout 20 FcgidIdleTimeout 300 FcgidProcessLifetime 3600 FcgidMaxRequestsPerProcess 1000

My RT web server is an Ubuntu 10.04 VM with 2GB RAM also, on a setup with about a quarter of a million tickets.

We run the MySQL database server on a separate VM.

Tim

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

3

Thanks for your response!

System:
RT version 4.0.5
Apache version 2.2.16
PostgreSQL version 8.4.11
Debian version 6.0.4
Kernel version 2.6.24-25-xen (64-bit)

I’ve not used the Debian packaged version - is it using mod_perl, or
mod_fastcgi or mod_fcgid?

It is using mod_perl. It didn’t have mod_fastcgi or mod_fcgid installed.

I found that using mod_fastcgi we had
severe memory leak problems, so yes there does seem to be a memory
leak in RT, but I don’t know where. However, I mitigated the
problem by switching to mod_fcgid in the Apache config, because it
has useful knobs you can tweak to tell it to start new CGI processes
every so often, or after a certain number of requests have been
handled. Thus no individual RT server process now lives long enough
for the memory leak to be a significant problem. Here’s my
/etc/apache2/mods-enabled/fcgid.conf:

AddHandler fcgid-script .fcgi FcgidConnectTimeout 20 FcgidIdleTimeout 300 FcgidProcessLifetime 3600 FcgidMaxRequestsPerProcess 1000

Cool. I’ve just installed mod_fcgid, made the above changes to
/etc/apache2/mods-enabled/fcgid.conf and restarted Apache, though I’m
not sure how to get RT to use it.

Regards,
Austin.
Austin L. Denyer.
I. T. Server & Security Administrator.

signature.asc (262 Bytes)

4

Cool. I’ve just installed mod_fcgid, made the above changes to
/etc/apache2/mods-enabled/fcgid.conf and restarted Apache, though I’m
not sure how to get RT to use it.

The docs supplied with RT give you the right runes to put in your apache configuration. It also depends on whether the debian-supplied RT was configured with fastcgi support or not. Dominic will be able to tell us…

Tim

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

5

System:
RT version 4.0.5
Apache version 2.2.16
PostgreSQL version 8.4.11
Debian version 6.0.4
Kernel version 2.6.24-25-xen (64-bit)

I’ve not used the Debian packaged version - is it using mod_perl, or mod_fastcgi or mod_fcgid? I found that using mod_fastcgi we had severe memory leak problems, so yes there does seem to be a memory leak in RT, but I don’t know where. However, I mitigated the problem by switching to mod_fcgid in the Apache config, because it has useful knobs you can tweak to tell it to start new CGI processes every so often, or after a certain number of requests have been handled. Thus no individual RT server process now lives long enough for the memory leak to be a significant problem. Here’s my /etc/apache2/mods-enabled/fcgid.conf:

AddHandler fcgid-script .fcgi FcgidConnectTimeout 20 FcgidIdleTimeout 300 FcgidProcessLifetime 3600 FcgidMaxRequestsPerProcess 1000

My RT web server is an Ubuntu 10.04 VM with 2GB RAM also, on a setup with about a quarter of a million tickets.

We run the MySQL database server on a separate VM.

Tim

You can set MaxRequestsPerChild for the mpm_worker_module.
This should also affect RT if you run it with mod_perl.

-Chris

6

You can set MaxRequestsPerChild for the mpm_worker_module.
This should also affect RT if you run it with mod_perl.

Thanks for your input.

I tried changing this but it didn’t seem to help any. (I did a full
apache2 restart after changing.)

I’ll try tweaking a few other settings in there to see if it makes a
difference.

Thanks.

Regards,
Austin.
Austin L. Denyer.
I. T. Server & Security Administrator.

signature.asc (262 Bytes)

7

You can set MaxRequestsPerChild for the mpm_worker_module.
This should also affect RT if you run it with mod_perl.

Thanks for your input.

I tried changing this but it didn’t seem to help any. (I did a full
apache2 restart after changing.)

I’ll try tweaking a few other settings in there to see if it makes a
difference.

If you can figure out replicable steps to make a vanilla install of RT
leak, please file a bug. Unfortunately, to solve leaks, we know we need
repeatable steps otherwise things tend to be very data/site specific.

-kevin