Performance Issues

RT Users
1

We’re currently running RT 3.0.3 with perl 5.6.1 on a RH 7.3 box and
using postgresql 7.1.3. When creating or modifying tickets, it can take
30-60 seconds to complete. It looks like the database connection/query
takes only about 3 seconds. I can’t figure out what’s causing it to be
so slow. I noticed a few other threads on this topic recently (from
August) but no obvious answers. Any ideas? Thanks.

Note: I have not upgraded to 3.0.5 because of the perl 5.8 requirement.
If this is the solution to my performance issues, I will pursue this
option though.
Amy Tanner
amy@real-time.com

2

We’re currently running RT 3.0.3 with perl 5.6.1 on a RH 7.3 box and
using postgresql 7.1.3. When creating or modifying tickets, it can take
30-60 seconds to complete. It looks like the database connection/query
takes only about 3 seconds. I can’t figure out what’s causing it to be
so slow. I noticed a few other threads on this topic recently (from
August) but no obvious answers. Any ideas? Thanks.

One thing I noticed, without doing any tuning of httpd or the DB (in my
case, mysql), is that throwing more memory at it helps. The box went from
512MB to 768MB and there was marked improvement in RT 3.0.4 performance.

I can’t comment on bumping it up to 1GB, because in that same round, I
moved to mysql 4.0.x, which by itself improved performance over mysql 3.x
– but that doesn’t help you. :slight_smile:

This is also on a 7.3 system.

3

Try running top sorted by either top cpu or memory while modifying your
ticket and see whats going on. Also, are there a lot of users modifying
and creating new tickets as well? Is the system heavily used by other
services?

Michael-----Original Message-----
From: Amy Tanner [mailto:amy@real-time.com]
Posted At: Thursday, September 11, 2003 10:23 AM
Posted To: RT
Conversation: [rt-users] Performance Issues
Subject: [rt-users] Performance Issues

We’re currently running RT 3.0.3 with perl 5.6.1 on a RH 7.3 box and
using postgresql 7.1.3. When creating or modifying tickets, it can take
30-60 seconds to complete. It looks like the database connection/query
takes only about 3 seconds. I can’t figure out what’s causing it to be
so slow. I noticed a few other threads on this topic recently (from
August) but no obvious answers. Any ideas? Thanks.

Note: I have not upgraded to 3.0.5 because of the perl 5.8 requirement.
If this is the solution to my performance issues, I will pursue this
option though.
Amy Tanner
amy@real-time.com
rt-users mailing list
rt-users@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

4

Try running top sorted by either top cpu or memory while modifying your
ticket and see whats going on. Also, are there a lot of users modifying
and creating new tickets as well? Is the system heavily used by other
services?

We don’t have many users yet so it’s not being used simulataneously by
many users.

The system is running a few other services, but nothing too
resource-intensive. It has 512Mb memory and 512Mb swap, but most of the
swap space is free. Here’s a sample:

10:01am up 34 days, 3:07, 6 users, load average: 0.27, 0.14, 0.03
142 processes: 141 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 10.3% user, 7.1% system, 0.0% nice, 82.0% idle
CPU1 states: 32.0% user, 7.0% system, 0.0% nice, 60.4% idle
Mem: 513524K av, 471284K used, 42240K free, 0K shrd, 69136K buff
Swap: 526296K av, 21836K used, 504460K free 199104K cached

PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
18013 webusr 23 0 29556 28M 15080 S 34.7 5.7 0:09 httpd

A few more details:

o dual PIII 500Mhz box
o apache 1.3.27
o postgres is running on another box - RHAS 2.1, 1Gb RAM, dual PIII
500Mhz

Running top on the postgres box, I see postmaster taking up 98% of the
CPU when I’m modifying tickets and it seems to stay that way until the
modify is complete. So, maybe it is postgres afterall.

For now, I’ve resorted to using the email gateway as much as possible to
avoid the slow web interface.

Amy Tanner
amy@real-time.com

5

After the thread regarding the performance issues. I have started
looking at FastCGI and HTML Mason’s caching. Since I don’t seem to have
a problem with the DB ( MySQL ) being slow.
I did get 3 entries in the Slow log that took 5 seconds but this isn’t
an issue to me. I did notice that there is 1.5 million entries on the
table where the query was generated.

In the threads some people suggested that I check the Caching. Now I
don’t know HTML mason at all and I don’t really know that much about the
FastCGI, but I tried looking for more info. All I found was something
call m->{cache} ( or something like that) which didn’t mean much to me.

I just want to know if anyone knows where to check the cache size in
memory and on disk of HTML Mason and the FastCGI.

When using an progressive browser like Phoenix I see the entries appear,
slowly, one entry at a time, but sometimes it takes for ever for the
first entry to appear and any entry after that as well.

I am really running out of options what to check and my boss is jumping
on my head wanting to know why it is slow.

I hope someone can help with this. I know there are lots of other people
out there that are experiencing similar problems and I would like use to
try and resolve this as quick as we can.

Leon

6

Hans Kristian Rosbach wrote:
Thanks. I have just added this to my Httpd.conf
a nd i am testing this.
I think this will help, since I was only running one instance .
The Idle-timeout isn’t accepted though. It doesn’t recognize the
parameter. I just left it out and I am testing it.

I hope this helps some other people too :stuck_out_tongue_winking_eye:

Leon

7

Guys. Please. Fill this info on wiki.bestpractical.com with descriptions
and may be links to archive of this or other similar discussion.

‘-processes X’ is not optional record in config but required IMHO. When
you run only one FastCGI process then it’s something like running RT
under mod_perl in ‘httpd -X’ mode.

			Best regards. Ruslan.

Leon wrote:

8

Guys. Please. Fill this info on wiki.bestpractical.com with descriptions
and may be links to archive of this or other similar discussion.

‘-processes X’ is not optional record in config but required IMHO. When
you run only one FastCGI process then it’s something like running RT
under mod_perl in ‘httpd -X’ mode.

Another data point: this helped us greatly. I added the following to the
FastCGIFAQ page, and changed the examples below to match – can someone
fact-check me please?

Q: Why is RT under FastCGI so sloooooooow? When using a progressive-rending
browser like Firefox I see the entries appear slowly, one at a time, but
the server doesn't appear to be overloaded and the database is not large.

A: You might only have spawned one mason_handler.fcgi, which would cause
all the httpd processes to block, waiting for it to service them. Make sure
you use an appropriate "-processes X" option in your FastCgiServer
directive, as shown below. You should have as many mason_handler.fcgi
processes as the nominal number of httpds that are usually running
(StartServers <= "-processes" <= MaxSpareServers) 


Eric Sorenson - EXPLOSIVE Networking - http://explosive.net
9

Guys. Please. Fill this info on wiki.bestpractical.com with descriptions
and may be links to archive of this or other similar discussion.

‘-processes X’ is not optional record in config but required IMHO. When
you run only one FastCGI process then it’s something like running RT
under mod_perl in ‘httpd -X’ mode.

Another data point: this helped us greatly. I added the following to the
FastCGIFAQ page, and changed the examples below to match – can someone
fact-check me please?

Q: Why is RT under FastCGI so sloooooooow? When using a progressive-rendin
g
browser like Firefox I see the entries appear slowly, one at a time, but
the server doesn’t appear to be overloaded and the database is not large.

A: You might only have spawned one mason_handler.fcgi, which would cause
all the httpd processes to block, waiting for it to service them. Make sur
e
you use an appropriate “-processes X” option in your FastCgiServer
directive, as shown below. You should have as many mason_handler.fcgi
processes as the nominal number of httpds that are usually running
(StartServers <= “-processes” <= MaxSpareServers)

Are you sure about your recommended number of processes? My
understanding is that apache spawns StartServers processes
when it launches, and that it continues to spawn processes,
keeping between MinSpareServices and MaxSpareServers idle,
just to handle any unanticipated surge. It's quite possible
to have MaxSpareServers=10, yet to have 100 running processes
on a loaded server.

Too many fcgi processes would eat memory, too few won't perform.

I also have a question about your explantion.  I think it's
true that if my request is being processed by the only
fcgi process, yours will block until mine is complete.
But why would my request, once processing starts, be slow?
I would have thought the symptom would be a long latency
followed by a fast full page.

    bobg
10

Bob Goldstein wrote:

Guys. Please. Fill this info on wiki.bestpractical.com with descriptions
and may be links to archive of this or other similar discussion.

‘-processes X’ is not optional record in config but required IMHO. When
you run only one FastCGI process then it’s something like running RT
under mod_perl in ‘httpd -X’ mode.

Another data point: this helped us greatly. I added the following to the
FastCGIFAQ page, and changed the examples below to match – can someone
fact-check me please?

Q: Why is RT under FastCGI so sloooooooow? When using a progressive-rendin
g
browser like Firefox I see the entries appear slowly, one at a time, but
the server doesn’t appear to be overloaded and the database is not large.

A: You might only have spawned one mason_handler.fcgi, which would cause
all the httpd processes to block, waiting for it to service them. Make sur
e
you use an appropriate “-processes X” option in your FastCgiServer
directive, as shown below. You should have as many mason_handler.fcgi
processes as the nominal number of httpds that are usually running
(StartServers <= “-processes” <= MaxSpareServers)
As I wrote in FAQ other FastCGI guys had reported that one FastCGI
process can handle 2-3 httpd process well.

Also this should be mentioned on FastCGIConfiguration wiki page.

Are you sure about your recommended number of processes? My
understanding is that apache spawns StartServers processes
when it launches, and that it continues to spawn processes,
keeping between MinSpareServices and MaxSpareServers idle,
just to handle any unanticipated surge. It's quite possible
to have MaxSpareServers=10, yet to have 100 running processes
on a loaded server.

Too many fcgi processes would eat memory, too few won't perform.

I also have a question about your explantion.  I think it's
true that if my request is being processed by the only
fcgi process, yours will block until mine is complete.
But why would my request, once processing starts, be slow?
I would have thought the symptom would be a long latency
followed by a fast full page.

Not right. Nobody say that RT is fast like a bullet. And if avg request
time 3 seconds it does make sense if you wait 3 or 6(2 concurent
requests) or 9 or …

Also you can wait even more if requests for images(static images and
other files) goes through FastCGI/mod_perl too :slight_smile: As I remember this
topic is highlighted on ML only.

11

Hi all -

I am new to the list and reasonably new to RT.

I saw the thread with Jon Masters et. al. last month talking about
performance issues with RT 3.

I’ve got a brand-new installation of RT 3.2.1 running on Fedora
Core 1, with apache 2 & FastCGI, with a mysql backend. We have
fewer than 100 tickets in our database. In general all the tools
are those from Fedora - I think I had to build FastCGI by hand.

The hardware is a dual-CPU pentium iii 600MHz. SCSI disks and 1 gig
of RAM. Nothing else is happening on this box.

things were horribly slow with only one mason_handler.fcgi process.
(it would take 5-10 seconds to display a ticket - generally most of the
wait was for the History section to come up). things are definitely
better now that i’m starting 10 fcgi processes but it still seems slow.
I can move to faster hardware but it is worrisome that this is a
problem already.

Any updates?

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

12

Hi all -

I am new to the list and reasonably new to RT.

I saw the thread with Jon Masters et. al. last month talking about
performance issues with RT 3.

I’ve got a brand-new installation of RT 3.2.1 running on Fedora
Core 1, with apache 2 & FastCGI, with a mysql backend. We have
fewer than 100 tickets in our database. In general all the tools

Dan,

It might help for you to talk about how you've configured mysql

to make use of the system’s ram, made sure that the system isn’t
swapping, that most of the CPU time seems to be in the database or in
perl, what versions of things other than RT you’re running, etc. Lately,
we’ve been seeing performance numbers of between .6 seconds and 2
seconds for ticket display in systems with about 10,000 tickets in them.

Jesse
13

It might help for you to talk about how you’ve configured mysql
to make use of the system’s ram, made sure that the system isn’t
swapping, that most of the CPU time seems to be in the database or in
perl, what versions of things other than RT you’re running, etc. Lately,
we’ve been seeing performance numbers of between .6 seconds and 2
seconds for ticket display in systems with about 10,000 tickets in them.

make sure the system is not swapping? Will the speed increase with swapping turned off with RT 3.0.11, perl 5.8.3, and Fastcgi?

This electronic mail message contains information belonging to PaymentOne, which may be confidential and/or legal privileged. The information is intended only for the use of the individual or entity named above. If you are not the intended recipient, you are hereby notified that any disclosure, printing, copying, distribution, or the taking of any action in reliance on the contents of this electronically mailed information is strictly prohibited. If you receive this message in error, please immediately notify us by electronic mail and delete this message.

14

HelloOn Wed, Sep 08, 2004 at 03:55:00PM -0400, Dan Pritts wrote:

I’ve got a brand-new installation of RT 3.2.1 running on Fedora
Core 1, with apache 2 & FastCGI, with a mysql backend. We have
fewer than 100 tickets in our database. In general all the tools
are those from Fedora - I think I had to build FastCGI by hand.

The hardware is a dual-CPU pentium iii 600MHz. SCSI disks and 1 gig
of RAM. Nothing else is happening on this box.

things were horribly slow with only one mason_handler.fcgi process.
(it would take 5-10 seconds to display a ticket - generally most of the

This is exactly the same as I reported 1-2 weeks ago, I run RT3.2 and
RT3 on Debian on two different machines (Celeron class w/ 768MB RAM,
idle and I normally know how to setup a server) and with only 1 ticket
and still got these 5-10 seconds with either fcgi, scgi and mod_perl on
Apache1.3. I also checked for timeouting DNS queries but with no luck.

I would really love to use RT but these numbers are of course not usable
for production use. I’m glad another user just reported <1s per ticket
which means that it is not broken by design.

So Any ideas what could cause such a delay?

thanks,

-christian-

Christian Hammers WESTEND GmbH | Internet-Business-Provider
Technik CISCO Systems Partner - Authorized Reseller
Lï¿œtticher Straï¿œe 10 Tel 0241/701333-11
ch@westend.com D-52064 Aachen Fax 0241/911879

15

So Any ideas what could cause such a delay?

For the record we’re on 3.2.1 and while it is slower than 3.0.11 which we were using before we’re still getting load times of around 1 second on very modest hardware - (AMD Athlon 2000XP, 256MB RAM, IDE HDDs).

We’re up to ~5000 tickets at present.

Cheers

Alex

16

It might help for you to talk about how you’ve configured mysql
to make use of the system’s ram, made sure that the system isn’t
swapping, that most of the CPU time seems to be in the database or in
perl, what versions of things other than RT you’re running, etc.
Lately,
we’ve been seeing performance numbers of between .6 seconds and 2
seconds for ticket display in systems with about 10,000 tickets in
them.

Jesse,

You may want to further quantify what “ticket display” really means.

Full ticket display
Default ticket listing on first page

Etc.

And on what type of hardware this performance is seen on.

Nate Duehr, nate@natetech.com

17

Hi all,

Is ModifySelf an enough setting for users (group everyone) to be able to
change their password? Or am I giving them too much of a happy
modifications to their profile? I want them to be able to change
password only.

Michael

18 
It might help for you to talk about how you've configured mysql

to make use of the system’s ram, made sure that the system isn’t
swapping, that most of the CPU time seems to be in the database or in
perl, what versions of things other than RT you’re running, etc. Lately,
we’ve been seeing performance numbers of between .6 seconds and 2
seconds for ticket display in systems with about 10,000 tickets in them.

make sure the system is not swapping? Will the speed increase with swapping turned off with RT 3.0.11, perl 5.8.3, and Fastcgi?

I didn’t mean “configured without swap” so much as “not actively using
swap while trying to serve out requests.”

19

we’ve been seeing performance numbers of between .6 seconds and 2
seconds for ticket display in systems with about 10,000 tickets in
them.

For me, the biggest indicator of how long it will take to display a
ticket is how much correspondence it has, and if any of those include
large attachments such as files. There is a direct correlation between
number and size with the time to display.

smime.p7s (2.42 KB)

20

For me, the biggest indicator of how long it will take to display a
ticket is how much correspondence it has, and if any of those include
large attachments such as files. There is a direct correlation between
number and size with the time to display.

That’s something that should have been wildly improved in RT 3.0.11 or
so. Certainly here, we saw page loads drop from 5+ minutes for long
tickets with multi-megabyte attachments to about 10 seconds for the
longest ticket history pages.

Jesse