RT Runs VERY Slowly

Hey Guys,

Another question… this time about performance. My RT box runs SLOW all
the time and I know this might be an Apache tweaking problem but I was
hoping someone might have an idea on how to tweak Apache to run better for
RT.

RT is the only app I run on this box and the box is a Dual Proc (CPU is
pretty idle) with 512MB RAM and slowly but surely all the RAM gets eaten up
and RT just crawls… below is the output from top about 5 minutes after
restarting apache - you can see that 411MB are already used up… and over
that that number will just increase and increase!

Can anyone throw me a bone here?

Stevo

12:10pm up 29 days, 0 min, 3 users, load average: 0.12, 0.24, 0.17
63 processes: 62 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.0% user, 0.5% system, 0.0% nice, 99.0% idle
CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Mem: 513476K av, 411880K used, 101596K free, 0K shrd, 39128K
buff
Swap: 257032K av, 6064K used, 250968K free 244728K
cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
3567 nobody 9 0 26636 26M 2720 S 0.0 5.1 0:11 httpd
3565 nobody 9 0 25452 24M 2716 S 0.0 4.9 0:03 httpd
3569 nobody 11 0 25320 24M 2696 S 0.0 4.9 0:03 httpd
3571 nobody 9 0 22344 21M 2628 S 0.0 4.3 0:00 httpd
3566 nobody 9 0 22336 21M 2624 S 0.0 4.3 0:00 httpd
3568 nobody 9 0 22336 21M 2624 S 0.0 4.3 0:00 httpd
3573 nobody 9 0 21148 20M 2388 S 0.0 4.1 0:00 httpd
3574 nobody 9 0 21148 20M 2388 S 0.0 4.1 0:00 httpd
3564 root 9 0 21072 20M 2328 S 0.0 4.1 0:00 httpd
13145 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13147 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13148 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13149 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13150 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13151 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13152 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13153 mysql 10 0 22344 18M 2532 S 0.0 3.6 0:39 mysqld
13154 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13155 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3570 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3572 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3575 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3576 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3577 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3578 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
29490 jferong 9 0 2220 2180 1804 S 0.0 0.4 0:00 sshd
29491 jferong 9 0 1492 1492 1084 S 0.0 0.2 0:00 bash
2403 root 9 0 1488 1488 1104 S 0.0 0.2 0:00 bash
2363 root 11 0 1484 1484 1104 S 0.0 0.2 0:00 bash
581 root 9 0 1428 1348 1188 S 0.0 0.2 0:11 sshd
3579 root 15 0 1036 1036 800 R 0.3 0.2 0:00 top
13121 root 9 0 1140 984 980 S 0.0 0.1 0:00 mysqld_safe
21490 root 9 0 688 608 568 S 0.0 0.1 0:00 sendmail
485 rpcuser 9 0 676 592 592 S 0.0 0.1 0:00 rpc.statd
629 root 9 0 580 556 512 S 0.0 0.1 0:03 crond
445 root 9 0 568 544 492 S 0.0 0.1 0:07 syslogd

There has been a lot on the list about performance over the last few months.

Release 3.0.10 has made some improvement for us.
Also making sure our database is vacuumed daily.

The following also made a big difference.

[ From http://lists.fsck.com/pipermail/rt-users/2002-August/009562.html ]

Apache/mod_perl is often a culprit, well, mostly mod_perl, I think.
Make sure you have Apache processes set to die after a certain
number of requests. If you set the number to 0, they won’t die,
and it’ll just bog down. Our installation used to run relatively
quickly at first, but as those processes lived on, the machine
slowed to a crawl; changing this setting fixed this gradual
slowdown. Not sure if this is in the docs, but it should be…

Modified /usr/local/apache/conf/httpd.conf and changed:
MaxRequestsPerChild 0
to:
MaxRequestsPerChild 100
then restarted Apache. 100 might be too low - I’m not sure - we will
see how performance goes in the next few days.

Thanks Mike - would RT run better with FastCGI?From: “Mike Husband” m.husband@leadup.com.au
To: “‘Stevo’” checkpoint@ozbergs.com; RT-Users@lists.bestpractical.com
Sent: Thursday, April 29, 2004 4:39 PM
Subject: RE: [rt-users] RT Runs VERY Slowly…

There has been a lot on the list about performance over the last few
months.

Release 3.0.10 has made some improvement for us.
Also making sure our database is vacuumed daily.

The following also made a big difference.

[ From http://lists.fsck.com/pipermail/rt-users/2002-August/009562.html ]

Apache/mod_perl is often a culprit, well, mostly mod_perl, I think.
Make sure you have Apache processes set to die after a certain
number of requests. If you set the number to 0, they won’t die,
and it’ll just bog down. Our installation used to run relatively
quickly at first, but as those processes lived on, the machine
slowed to a crawl; changing this setting fixed this gradual
slowdown. Not sure if this is in the docs, but it should be…

Modified /usr/local/apache/conf/httpd.conf and changed:
MaxRequestsPerChild 0
to:
MaxRequestsPerChild 100
then restarted Apache. 100 might be too low - I’m not sure - we will
see how performance goes in the next few days.

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Stevo
Sent: Thursday, 29 April 2004 4:15 AM
To: RT-Users@lists.bestpractical.com
Subject: [rt-users] RT Runs VERY Slowly…

Hey Guys,

Another question… this time about performance. My RT box
runs SLOW all
the time and I know this might be an Apache tweaking problem but I was
hoping someone might have an idea on how to tweak Apache to
run better for
RT.

RT is the only app I run on this box and the box is a Dual
Proc (CPU is
pretty idle) with 512MB RAM and slowly but surely all the RAM
gets eaten up
and RT just crawls… below is the output from top about 5
minutes after
restarting apache - you can see that 411MB are already used
up… and over
that that number will just increase and increase!

Can anyone throw me a bone here?

Stevo

12:10pm up 29 days, 0 min, 3 users, load average: 0.12, 0.24, 0.17
63 processes: 62 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.0% user, 0.5% system, 0.0% nice, 99.0% idle
CPU1 states: 0.0% user, 0.0% system, 0.0% nice, 100.0% idle
Mem: 513476K av, 411880K used, 101596K free, 0K
shrd, 39128K
buff
Swap: 257032K av, 6064K used, 250968K free
244728K
cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
3567 nobody 9 0 26636 26M 2720 S 0.0 5.1 0:11 httpd
3565 nobody 9 0 25452 24M 2716 S 0.0 4.9 0:03 httpd
3569 nobody 11 0 25320 24M 2696 S 0.0 4.9 0:03 httpd
3571 nobody 9 0 22344 21M 2628 S 0.0 4.3 0:00 httpd
3566 nobody 9 0 22336 21M 2624 S 0.0 4.3 0:00 httpd
3568 nobody 9 0 22336 21M 2624 S 0.0 4.3 0:00 httpd
3573 nobody 9 0 21148 20M 2388 S 0.0 4.1 0:00 httpd
3574 nobody 9 0 21148 20M 2388 S 0.0 4.1 0:00 httpd
3564 root 9 0 21072 20M 2328 S 0.0 4.1 0:00 httpd
13145 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13147 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13148 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13149 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13150 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13151 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13152 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13153 mysql 10 0 22344 18M 2532 S 0.0 3.6 0:39 mysqld
13154 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
13155 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3570 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3572 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3575 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3576 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3577 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
3578 mysql 9 0 22344 18M 2532 S 0.0 3.6 0:00 mysqld
29490 jferong 9 0 2220 2180 1804 S 0.0 0.4 0:00 sshd
29491 jferong 9 0 1492 1492 1084 S 0.0 0.2 0:00 bash
2403 root 9 0 1488 1488 1104 S 0.0 0.2 0:00 bash
2363 root 11 0 1484 1484 1104 S 0.0 0.2 0:00 bash
581 root 9 0 1428 1348 1188 S 0.0 0.2 0:11 sshd
3579 root 15 0 1036 1036 800 R 0.3 0.2 0:00 top
13121 root 9 0 1140 984 980 S 0.0 0.1 0:00
mysqld_safe
21490 root 9 0 688 608 568 S 0.0 0.1 0:00 sendmail
485 rpcuser 9 0 676 592 592 S 0.0 0.1 0:00
rpc.statd
629 root 9 0 580 556 512 S 0.0 0.1 0:03 crond
445 root 9 0 568 544 492 S 0.0 0.1 0:07 syslogd


The rt-users Archives

RT Developer and Administrator training is coming to LA, DC
and Frankfurt this spring and summer.
http://bestpractical.com/services/training.html

Sign up early, as class space is limited.

Anything thing I noticed is that I have plenty of swap space and NONE of
it’s being used… is there a way to get mod_perl to utilize all that free
swap?!!

–StevoFrom: “Jamie Lawrence” jal@jal.org
To: “Mike Husband” m.husband@leadup.com.au
Cc: “‘Stevo’” checkpoint@ozbergs.com; RT-Users@lists.bestpractical.com
Sent: Thursday, April 29, 2004 10:20 PM
Subject: Re: [rt-users] RT Runs VERY Slowly…

There has been a lot on the list about performance over the last few
months.

Release 3.0.10 has made some improvement for us.
Also making sure our database is vacuumed daily.

The following also made a big difference.

[ From
http://lists.fsck.com/pipermail/rt-users/2002-August/009562.html ]

Apache/mod_perl is often a culprit, well, mostly mod_perl, I think.
Make sure you have Apache processes set to die after a certain
number of requests. If you set the number to 0, they won’t die,
and it’ll just bog down. Our installation used to run relatively
quickly at first, but as those processes lived on, the machine
slowed to a crawl; changing this setting fixed this gradual
slowdown. Not sure if this is in the docs, but it should be…

Modified /usr/local/apache/conf/httpd.conf and changed:
MaxRequestsPerChild 0
to:
MaxRequestsPerChild 100
then restarted Apache. 100 might be too low - I’m not sure - we will
see how performance goes in the next few days.

I don’t know much about your situation, but one thing many people
underestimate with mod_perl is the cost of memory. It starts out mostly
shared, but copy-on-write over time kills that. Too many processes and
you’re very, very quickly swapping. RT is a rather heavyweight app,
memory-wise, and seems to behave in a non-memory-conservative fashion,
so you’ll feel that quickly.

-j


Jamie Lawrence jal@jal.org
Fascists divide in two categories: the fascists and the
anti-fascists.

  • Ennio Flaiano

Anything thing I noticed is that I have plenty of swap space and NONE of
it’s being used… is there a way to get mod_perl to utilize all that free
swap?!!

Uh, maybe I missed something, but why would you want mod_perl to use
swap space if performance is the issue? Unless you have a very
interesting system your swap space is on disk which is many orders
of magnitude slower than RAM.

Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie

Hey Steve,

Well I’ve got 512MB ram in the system and about 10MB free and my Swap
partition is 1Gig with 1Gig free… so it seems that RT could be making use
of that extra space for mod_perl… the system used to be at 256MB and it
sucked all that up pretty quick…

I was just wondering! :slight_smile:

–StevoFrom: “Steve Wampler” sbw@tapestry.tucson.az.us
To: “Stevo” checkpoint@ozbergs.com
Cc: “Jamie Lawrence” jal@jal.org; RT-Users@lists.bestpractical.com
Sent: Friday, April 30, 2004 9:19 AM
Subject: Re: [rt-users] RT Runs VERY Slowly…

Anything thing I noticed is that I have plenty of swap space and NONE of
it’s being used… is there a way to get mod_perl to utilize all that
free
swap?!!

Uh, maybe I missed something, but why would you want mod_perl to use
swap space if performance is the issue? Unless you have a very
interesting system your swap space is on disk which is many orders
of magnitude slower than RAM.


Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie

:slight_smile: I think this is obviously not a normal operation mode for RT expept if, maybe, Jessy has a lot of actions in RAM business :wink:

And Steeve, the other one (not Steevo), you may want to be a little tolerant to ideas, specialy those none sense thoeories/ideas… Because you could end up beeing into trouble some time, in such trouble then even silly ideas looks tempting :slight_smile:

BTW good luck !!

SLFrom: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Stevo
Sent: Friday, April 30, 2004 12:28 PM
To: Steve Wampler
Cc: RT-Users@lists.bestpractical.com
Subject: Re: [rt-users] RT Runs VERY Slowly…

Hey Steve,

Well I’ve got 512MB ram in the system and about 10MB free and my Swap
partition is 1Gig with 1Gig free… so it seems that RT could be making use
of that extra space for mod_perl… the system used to be at 256MB and it
sucked all that up pretty quick…

I was just wondering! :slight_smile:

–Stevo

From: “Steve Wampler” sbw@tapestry.tucson.az.us
To: “Stevo” checkpoint@ozbergs.com
Cc: “Jamie Lawrence” jal@jal.org; RT-Users@lists.bestpractical.com
Sent: Friday, April 30, 2004 9:19 AM
Subject: Re: [rt-users] RT Runs VERY Slowly…

Anything thing I noticed is that I have plenty of swap space and NONE of
it’s being used… is there a way to get mod_perl to utilize all that
free
swap?!!

Uh, maybe I missed something, but why would you want mod_perl to use
swap space if performance is the issue? Unless you have a very
interesting system your swap space is on disk which is many orders
of magnitude slower than RAM.


Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie

http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

RT Developer and Administrator training is coming to LA, DC and Frankfurt this spring and summer.
http://bestpractical.com/services/training.html

Sign up early, as class space is limited.

Hey Steve,

Well I’ve got 512MB ram in the system and about 10MB free and my Swap
partition is 1Gig with 1Gig free… so it seems that RT could be making use
of that extra space for mod_perl… the system used to be at 256MB and it
sucked all that up pretty quick…

(I came into this discussion late, so please let me know if this has
been covered!)

It’s unlikely that letting mod_perl go into swap space is going to
help (most likely, things will slow down even more). It is a little
surprising that nothing is being moved into swap if your free RAM is
so small, unless most of that is bufferspace. What does ‘free’ show?
(Even better, what does ‘cat /proc/meminfo’ show?)

The problem is probably elsewhere and adding more memory (RAM or otherwise)
isn’t likely to help. 512MB should be more than enough - RT on one of
my systems (with 512MB) takes up quite a bit less that what you’re seeing
(typically a few dozen MB, except for buffers).

Good luck!
Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie

Here’s the output… I’d love to get to the root of this…

    total:    used:    free:  shared: buffers:  cached:

Mem: 525799424 514760704 11038720 0 85594112 223989760
Swap: 1068498944 8089600 1060409344
MemTotal: 513476 kB
MemFree: 10780 kB
MemShared: 0 kB
Buffers: 83588 kB
Cached: 214824 kB
SwapCached: 3916 kB
Active: 355092 kB
ActiveAnon: 139360 kB
ActiveCache: 215732 kB
Inact_dirty: 0 kB
Inact_laundry: 76368 kB
Inact_clean: 7244 kB
Inact_target: 87740 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 513476 kB
LowFree: 10780 kB
SwapTotal: 1043456 kB
SwapFree: 1035556 kBFrom: “Steve Wampler” sbw@tapestry.tucson.az.us
To: “Stevo” checkpoint@ozbergs.com
Cc: “Steve Wampler” sbw@tapestry.tucson.az.us;
RT-Users@lists.bestpractical.com
Sent: Friday, April 30, 2004 10:33 AM
Subject: Re: [rt-users] RT Runs VERY Slowly…

Hey Steve,

Well I’ve got 512MB ram in the system and about 10MB free and my Swap
partition is 1Gig with 1Gig free… so it seems that RT could be making
use
of that extra space for mod_perl… the system used to be at 256MB and
it
sucked all that up pretty quick…

(I came into this discussion late, so please let me know if this has
been covered!)

It’s unlikely that letting mod_perl go into swap space is going to
help (most likely, things will slow down even more). It is a little
surprising that nothing is being moved into swap if your free RAM is
so small, unless most of that is bufferspace. What does ‘free’ show?
(Even better, what does ‘cat /proc/meminfo’ show?)

The problem is probably elsewhere and adding more memory (RAM or
otherwise)
isn’t likely to help. 512MB should be more than enough - RT on one of
my systems (with 512MB) takes up quite a bit less that what you’re seeing
(typically a few dozen MB, except for buffers).

Good luck!

Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie

Here’s the output… I’d love to get to the root of this…

    total:    used:    free:  shared: buffers:  cached:

Mem: 525799424 514760704 11038720 0 85594112 223989760
Swap: 1068498944 8089600 1060409344
MemTotal: 513476 kB
MemFree: 10780 kB
MemShared: 0 kB
Buffers: 83588 kB
Cached: 214824 kB
SwapCached: 3916 kB
Active: 355092 kB
ActiveAnon: 139360 kB
ActiveCache: 215732 kB
Inact_dirty: 0 kB
Inact_laundry: 76368 kB
Inact_clean: 7244 kB
Inact_target: 87740 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 513476 kB
LowFree: 10780 kB
SwapTotal: 1043456 kB
SwapFree: 1035556 kB

Actually, that looks OK - and pretty typical. I doubt that the
problem is memory related. Of your ~500MB of used memory, over
half (about 288MB) is buffer/cache. The kernel has a lot of
freedom in how that memory is allocated and has room to balance
buffers and cache as needed. The fact that you only have 10MB
free is a Good Thing - it means the kernel is doing a good job
managing your memory and using it efficiently. (The mantra of
the Linux kernel is “free memory is wasted memory”.)

And there is a at least a little in the swap - just under 8MB.
So swap is being used when memory does start getting tight [I’d guess that the machine hasn’t been very long, as the kernel tends to migrate more idle processes to swap as time goes on.]

So, my best guess is that something else is the culprit of
the slow RT performance. I tend to blame the database for
such things, but that’s really just because I like blaming
mysql [I use postgresql] and not for any justifiable reason.
You’re probably better off listening to the advice of others
at this point…

Good luck!
Steve

Steve Wampler {sbw@tapestry.tucson.az.us}
The gods that smiled upon your birth are laughing now. – fortune cookie