Mod_perl vs fastcgi, which performs best, which is more scalable?

RT Users
1

I have an associate who has been running RT for quite some time, and has
literally tens of thousands of tickets in the database. He mentioned
that sometimes loading a ticket can take an extremly long time (I assume
if it has other parent/child tickets that it has to find). I do not want
this to eventually happen to my RT instance, so I am wondering what the
optimal setup is. He is using mod_perl, I am using fastcgi. His apache
and mysql are on solaris, mine are on linux. Both architectures are
SMP, but I really am more interested in optimal setups and tweaks for RT
on the software side, rather than the OS and hardware. In other words,
we can probably discuss at length whether apache runs better on a
multi-proc Solaris box or a Linux box, as well as 64bit Mysql advantages
etc.

What I am wondering exactly is:

  1. Which performs better in general, mod_perl or fastcgi? I googled for
    some benchmarks and didn’t really find anything enlightening.
  2. Is there anything that can be done to the RT database itself, to
    speed up a large database, short of actually deleting older tickets?
    Things like table indexes, ensuring all the numeric tables are numeric
    types, etc?
  3. How aware are the RT developers of issues later down the road when
    the database gets huge? In other words, is RT3 as optimal as it can be
    as far as making the db queries as efficient as possible, or is there a
    lot of work and improvment that can/will be done in future versions?

Thanks,
-Charles Jones

2
  1. How aware are the RT developers of issues later down the road when
    the database gets huge? In other words, is RT3 as optimal as it can be
    as far as making the db queries as efficient as possible, or is there a
    lot of work and improvment that can/will be done in future versions?

What’s huge? We’ve got sites with 750,000 tickets in their database.
Each and every site will have different scalabiltiy concerns. If your
site is getting huge, you’re going to want to spend time tuning the
system to meet your needs or have us do so on your behalf.

3

Jesse Vincent wrote:

  1. How aware are the RT developers of issues later down the road when
    the database gets huge? In other words, is RT3 as optimal as it can be
    as far as making the db queries as efficient as possible, or is there a
    lot of work and improvment that can/will be done in future versions?

What’s huge? We’ve got sites with 750,000 tickets in their database.
Each and every site will have different scalabiltiy concerns. If your
site is getting huge, you’re going to want to spend time tuning the
system to meet your needs or have us do so on your behalf.

I’m not sure how big our instance will get, but I was concerned about my
colleagues issues with the slow load times. They may (in fact probably
are) be due to his particular hardware setup, but I just wanted to do
some fact finding so I could have optimal setup from the get-go, instead
of finding out 6 months from now that I need to switch to mod_perl to
keep the performance level up. I don’t mind tuning things to get things
running optimally (in fact I enjoy it!)…just wondering what other RT
users (that have large dbs) experiences are like.

P.S. RT is great :slight_smile:

-Charles Jones

4

I’ve been running on FastCGI since 3.0.7. Starting from 3.0.9 we have seen some horrendous performance problems (CPU pegging and effectively hanging the application for upwards of a minute) even on modern hardware with minimal workloads. We were able to work around them by pre-starting multiple server processes (the perl process is actually what seems to hang), and performance has been pretty good since then.
I did hear from one mod_perl user who said he had a similar problem, but I’ve never tried that configuration myself.
For comparison, I’ve got a dual Xeon server with 2GB RAM and hardware mirrored SATA drives running RHEL 3.-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com on behalf of Charles Jones
Sent: Tue 6/15/2004 5:14 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] mod_perl vs fastcgi, which performs best,which is more scalable?

I have an associate who has been running RT for quite some time, and has
literally tens of thousands of tickets in the database. He mentioned
that sometimes loading a ticket can take an extremly long time (I assume
if it has other parent/child tickets that it has to find). I do not want
this to eventually happen to my RT instance, so I am wondering what the
optimal setup is. He is using mod_perl, I am using fastcgi. His apache
and mysql are on solaris, mine are on linux. Both architectures are
SMP, but I really am more interested in optimal setups and tweaks for RT
on the software side, rather than the OS and hardware. In other words,
we can probably discuss at length whether apache runs better on a
multi-proc Solaris box or a Linux box, as well as 64bit Mysql advantages
etc.

What I am wondering exactly is:

  1. Which performs better in general, mod_perl or fastcgi? I googled for
    some benchmarks and didn’t really find anything enlightening.
  2. Is there anything that can be done to the RT database itself, to
    speed up a large database, short of actually deleting older tickets?
    Things like table indexes, ensuring all the numeric tables are numeric
    types, etc?
  3. How aware are the RT developers of issues later down the road when
    the database gets huge? In other words, is RT3 as optimal as it can be
    as far as making the db queries as efficient as possible, or is there a
    lot of work and improvment that can/will be done in future versions?

Thanks,
-Charles Jones
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

RT Developer and Administrator training is coming to LA, DC and Frankfurt this spring and summer.
http://bestpractical.com/services/training.html

Sign up early, as class space is limited.

5

I’ve been running on FastCGI since 3.0.7. Starting from 3.0.9 we have seen some horrendous performance problems (CPU pegging and effectively hanging the application for upwards of a minute) even on modern hardware with minimal workloads. We were able to work around them by pre-starting multiple server processes (the perl process is actually what seems to hang), and performance has been pretty good since then.

Was the perl process waiting on anything? Was there a blocking database
query? Did you attempt any sort of debugging of the actual problem?

Thanks,
Jesse
6

I posted the details back when it happened, but nobody seemed to have any ideas. I assumed that there was something odd about our configuration that triggered the problem. MySQL was more or less idle and completely responsive at all times. Off the top of my head, the perl process seemed to be doing some sort of rapid polling that didn’t produce any results. Typically we would see perl lock up for about 30-120 seconds 10-20 times per day. This was with five active administrators and about 100 tickets per day.
I would guess that having multiple backend processes doesn’t actually prevent the hangups, but it does allow other people to keep working without any visible problems.-----Original Message-----
From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: Tue 6/15/2004 9:46 PM
To: Derek Suzuki
Cc: Charles Jones; rt-users@lists.bestpractical.com
Subject: Re: [rt-users] mod_perl vs fastcgi, which performs best, which is more scalable?

On Tue, Jun 15, 2004 at 09:43:57PM -0700, Derek Suzuki wrote:

I’ve been running on FastCGI since 3.0.7. Starting from 3.0.9 we have seen some horrendous performance problems (CPU pegging and effectively hanging the application for upwards of a minute) even on modern hardware with minimal workloads. We were able to work around them by pre-starting multiple server processes (the perl process is actually what seems to hang), and performance has been pretty good since then.

Was the perl process waiting on anything? Was there a blocking database
query? Did you attempt any sort of debugging of the actual problem?

Thanks,
Jesse
7

I posted the details back when it happened, but nobody seemed to have any ideas. I assumed that there was something odd about our configuration that triggered the problem. MySQL was more or less idle and completely responsive at all times. Off the top of my head, the perl process seemed to be doing some sort of rapid polling that didn’t produce any results. Typically we would see perl lock up for about 30-120 seconds 10-20 times per day. This was with five active administrators and about 100 tickets per day.
I would guess that having multiple backend processes doesn’t actually prevent the hangups, but it does allow other people to keep working without any visible problems.

That sounds like users were asking for pages while still rendering
others. The user session handler puts a lock on the session while
a page is being generated to stop other processes from clobbering that
session. The symptom of a second process polling for a lock to be
released would look much like what you’re seeing. But you’d want to use
a tool like “mytop” to see if there were actually mysql processes
spinning on locks.

8
  1. How aware are the RT developers of issues later down the road when
    the database gets huge? In other words, is RT3 as optimal as it can be
    as far as making the db queries as efficient as possible, or is there a
    lot of work and improvment that can/will be done in future versions?

What’s huge? We’ve got sites with 750,000 tickets in their database.
Each and every site will have different scalabiltiy concerns. If your
site is getting huge, you’re going to want to spend time tuning the
system to meet your needs or have us do so on your behalf.

We’ve just hit 200,000 tickets in our RT database, and our install is
very responsive with 10 - 15 operators using RT at any given time;

Logging in & displaying stats for 50 queues  :  2 - 3 secs
Entering a queue with 20 tickets             :  2 secs
Displaying a ticket with several responses   :  1 - 2 secs

Our system specs aren’t anything very special;
CPU: 2.8ghz Intel P4
RAM: 2gb
RT: 3.0.11
Apache: 1.3.31
mod_perl: 1.29

We hit major performance problems once we upgraded from the RT 2.0.x to
3.0.x series. The following are things that seem to have helped in
speeding up the system;

  1. We were running the system on IDE drives. We migrated the DB data
    onto SCSI drives.

  2. Upgraded the amount of RAM from 1gb to 2gb.

  3. Tweaking of MySQL config - trial & error to some extent.

  4. SWAP was running on a single IDE drive - we defined several smaller
    swap partitions, one on each physical drive.

After the above 4, and a few RT upgrades as newer versions were
released, the system was running very nicely. Of the above 4 changes, I
think we saw the most performance gain from the tweaking of MySQL &
additional RAM.

The only remaining problem was that RT would get less & less responsive
throughout the day, until apache was restarted. Changing
MaxRequestsPerChild in apache from 0 to 100 has fixed this problem for
us.

I’m sure we made a few additional tweaks, but I cannot recall these off
the top of my head.

The next step for us, is to remove all deleted tickets from our
database. Our management’s decision is that we cannot outright delete
anything for 5 years, so we have to transfer all deleted tickets to a
holding database, where each ticket can be permanently purged after it’s
reached 5 years… fun!

Regards,

Peter

signature.asc (189 Bytes)

9

Peter Silver wrote:

Our system specs aren’t anything very special;
CPU: 2.8ghz Intel P4
RAM: 2gb
RT: 3.0.11
Apache: 1.3.31
mod_perl: 1.29

Peter,

What OS are you running?

Really curious,
mike

10

Peter Silver wrote:

Our system specs aren’t anything very special;
CPU: 2.8ghz Intel P4
RAM: 2gb
RT: 3.0.11
Apache: 1.3.31
mod_perl: 1.29

What OS are you running?

Our system runs on RHEL 3 ES.

Regards,

Peter

signature.asc (189 Bytes)