Performance issues with RT3 (all versions)

RT Developers
1

Hi all,

For quite some time we have been running RT3 to track support issues,
following my recommendation of the software. Although various
configurations have been used, I am currently debugging an instance
running on a Debian testing box with the specification below because the
faster production machine (running Mandrake release 9) cannot be
tinkered with for the moment and performace problems plague both.

Performance is…well horrible. Painfully unpleasant and this is not
with the 40,000 tickets quoted in RT documentation. This is with a
database not exceeding about 50-100MB in size and with only a handfull
of tickets in it. In other words there are almost no tickets.

I have tried various different configurations based around RT3 (various
versions including but not limited to 3.2.0), MySQL 4, and Apache 1 and
2 running every possible configuration of mod_perl 1,2 and FastCGI.
Essentially I have tried every possible choice available.

Query time is so painful that the people here are having to resort to
using regular email and just have RT keep a copy for auditing. I do
unfortunately not have time to delve much in to the bowls of RT (I do
embedded Linux kernel hacking and am not usually a perl person) but have
installed various different profiling options in Apache to at least give
a feel for where this is falling over. I have debug traces of apache and
various other output here and am willing to provide various additional
information and test solutions if I can help to correct this issue.

Clearly the testing machine is not the fastest in the world but it is an
order of magnitude too slow and there is something more fundamental at
fault here. It does not look like MySQL is at fault - it looks like it
is SearchBuilder spewing out way too many searches or something. As I
have read on your mailing list archives and on other online lists,
numerous people report problems with attachments - some of these tickets
have large numbers of attachments but not all, and even then even
listing the queue takes nearly all day.

I am not trying to rant here but there is clearly something very weird
going on which I would like to help you guys to fix. You never know, it
might even be possible to convince me that perl isn’t such a slow
scripting language after all.

Cheers,

Jon.

— Specification —

Linux kernel 2.6.6 (Not stock, Debian kernel release 2.6.6-1-686)
running on an Intel Celeron 600MHz CPU. This test box is not the fastest
machine in the world but it runs nothing else other than RT and a few
general system daemons. No windowing system cruft. System has been tuned
using sysctls and ulimits adjusted all in vain, IDE DMA is on, etc. etc.
The production machine exhibits the same problems and is much faster.

(So please assume that the base Debian is configured correctly).

To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).

I have also installed RT 3.2.0 as well as other versions.

I have performed database OPTIMIZE, ANALYZE, etc. etc.

etc. etc. etc. etc.

— dprofpp output —

Storable::nfreeze has 1 unstacked calls in outer
Storable::_freeze has 1 unstacked calls in outer
Exporter::import has -2 unstacked calls in outer
RT::Mason::handler has -1 unstacked calls in outer
CGI::delete has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Storable::thaw has 1 unstacked calls in outer
Exporter::Heavy::heavy_export has 2 unstacked calls in outer
Total Elapsed Time = 196.2333 Seconds
User+System Time = 72.21333 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
40.4 29.24 29.418 4647 0.0063 0.0063
DBIx::SearchBuilder::Record::ANO
N
9.20 6.641 71.487 2297 0.0029 0.0311 HTML::Mason::Request::comp
8.09 5.844 5.844 10197 0.0006 0.0006 HTML::Mason::Request::print
6.79 4.902 4.974 52 0.0943 0.0957
RT::Transaction::BriefDescription
6.58 4.753 4.805 41 0.1159 0.1172 RT::Attachment::ContentLength
4.35 3.139 3.178 25 0.1256 0.1271 RT::Attachment::Content
3.52 2.539 2.639 25 0.1016 0.1055 Text::Quoted::extract
2.28 1.644 1.658 41 0.0401 0.0404 RT::Attachment::Headers
1.65 1.195 1.211 53 0.0226 0.0228 RT::Transactions::Next
1.64 1.186 1.182 52 0.0228 0.0227 RT::Record::CreatedAsString
1.52 1.095 1.119 25 0.0438 0.0448 RT::Transaction::IsInbound
1.37 0.986 1.011 42 0.0235 0.0241 RT::Attachments::Next
0.97 0.697 0.695 50 0.0139 0.0139 RT::Transaction::TicketObj
0.87 0.628 0.635 52 0.0121 0.0122 RT::Record::CreatorObj
0.51 0.369 0.367 20 0.0184 0.0184
DBIx::SearchBuilder::Record::AUTOL
OAD

2

Jon Masters wrote:

Hi all,

[snip]

Cheers,

Jon.

— Specification —

Linux kernel 2.6.6 (Not stock, Debian kernel release 2.6.6-1-686)
running on an Intel Celeron 600MHz CPU. This test box is not the fastest
machine in the world but it runs nothing else other than RT and a few
general system daemons. No windowing system cruft. System has been tuned
using sysctls and ulimits adjusted all in vain, IDE DMA is on, etc. etc.
The production machine exhibits the same problems and is much faster.

(So please assume that the base Debian is configured correctly).
so as I understand no memory presure.

To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).
Turn on log of slow queries with limit 1 second.
Request Tracker Wiki
Also can you create full SQL log for one request to Ticket/Display page?

I have also installed RT 3.2.0 as well as other versions.
Does it make any sense if you switch to 3.0.x.

I have performed database OPTIMIZE, ANALYZE, etc. etc.
It doesn’t help if you are on InnoDB. Are you?

etc. etc. etc. etc.

You didn’t report what versions of modules and perl do you have?
mod_perl 1.x or 2.x? if 1.x then DSO or Static?

This output isn’t normal

— dprofpp output —

Total Elapsed Time = 196.2333 Seconds
User+System Time = 72.21333 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
40.4 29.24 29.418 4647 0.0063 0.0063
DBIx::SearchBuilder::Record::ANO
N
9.20 6.641 71.487 2297 0.0029 0.0311 HTML::Mason::Request::comp
8.09 5.844 5.844 10197 0.0006 0.0006 HTML::Mason::Request::print
6.79 4.902 4.974 52 0.0943 0.0957
RT::Transaction::BriefDescription
6.58 4.753 4.805 41 0.1159 0.1172 RT::Attachment::ContentLength
4.35 3.139 3.178 25 0.1256 0.1271 RT::Attachment::Content
3.52 2.539 2.639 25 0.1016 0.1055 Text::Quoted::extract
2.28 1.644 1.658 41 0.0401 0.0404 RT::Attachment::Headers
1.65 1.195 1.211 53 0.0226 0.0228 RT::Transactions::Next
1.64 1.186 1.182 52 0.0228 0.0227 RT::Record::CreatedAsString
1.52 1.095 1.119 25 0.0438 0.0448 RT::Transaction::IsInbound
1.37 0.986 1.011 42 0.0235 0.0241 RT::Attachments::Next
0.97 0.697 0.695 50 0.0139 0.0139 RT::Transaction::TicketObj
0.87 0.628 0.635 52 0.0121 0.0122 RT::Record::CreatorObj
0.51 0.369 0.367 20 0.0184 0.0184
DBIx::SearchBuilder::Record::AUTOL
OAD
dprofpp: Ticket/Display for ticket with 141 attachments.
Total Elapsed Time = 116.6960 Seconds
User+System Time = 37.64604 Seconds
Exclusive Times
%Time ExclSec CumulS #Calls sec/call Csec/c Name
8.22 3.095 5.953 80847 0.0000 0.0001
DBIx::SearchBuilder::Record::Cachable::_gen_primary_cache_key
8.17 3.077 3.077 9457 0.0003 0.0003 RT::User::_ClassAccessible
6.77 2.547 12.232 80251 0.0000 0.0002 RT::Record::__Value
5.86 2.206 8.807 80251 0.0000 0.0001
DBIx::SearchBuilder::Record::Cachable::__Value
4.92 1.853 11.355 9264 0.0002 0.0012 RT::Principal::HasRight
4.86 1.831 8.322 44702 0.0000 0.0002
DBIx::SearchBuilder::Record::Cachable::LoadByCols
4.55 1.714 2.936 44702 0.0000 0.0001
DBIx::SearchBuilder::Record::Cachable::_lookup_primary_cache_key
4.30 1.617 1.911 44702 0.0000 0.0000
DBIx::SearchBuilder::Record::Cachable::_gen_alternate_cache_key
3.99 1.503 1.503 402077 0.0000 0.0000
DBIx::SearchBuilder::Record::Cachable::_RecordCache
3.97 1.494 10.414 44702 0.0000 0.0002 RT::Record::LoadByCols
3.82 1.439 9.439 37661 0.0000 0.0003
DBIx::SearchBuilder::Record::new
3.45 1.298 1.298 9263 0.0001 0.0001 RT::Queue::_ClassAccessible
3.43 1.293 11.158 37661 0.0000 0.0003
DBIx::SearchBuilder::Record::Cachable::new
3.31 1.245 1.245 184179 0.0000 0.0000
DBIx::SearchBuilder::Record::id
3.27 1.232 1.307 86734 0.0000 0.0000
DBIx::SearchBuilder::Record::__Value

3

Jon Masters wrote:

Hi all,
[snip]

— dprofpp output —

You didn’t say which options you were using.

		Best regards. Ruslan.
4

[Please don’t cross-post to rt-users and rt-devel. Most folks on rt-devel
are also on RT-Users]

Hi all,

For quite some time we have been running RT3 to track support issues,
following my recommendation of the software. Although various
configurations have been used, I am currently debugging an instance
running on a Debian testing box with the specification below because the
faster production machine (running Mandrake release 9) cannot be
tinkered with for the moment and performace problems plague both.

So. How much RAM do you have in these machines?

Performance is…well horrible. Painfully unpleasant and this is not
with the 40,000 tickets quoted in RT documentation. This is with a
database not exceeding about 50-100MB in size and with only a handfull
of tickets in it. In other words there are almost no tickets.

How slow is slow? Quoting actual numbers, rather saying “painfully
unpleasant” will allow folks to better understand the magnitude of your
issue.

5

Hi all,

For quite some time we have been running RT3 to track support issues,
following my recommendation of the software. Although various
configurations have been used, I am currently debugging an instance
running on a Debian testing box with the specification below because
the faster production machine (running Mandrake release 9) cannot be
tinkered with for the moment and performace problems plague both.

An upgrade to Perl 5.8 just hit my Debian Testing machine recently. I
saw notes on the lists here that it made a major difference. I have a
3500 ticket system with lots of file attachments, and InnoDB turned on
in MySQL (it’s off by default if you upgraded to Testing and didn’t
turn it on). The machine is an Athlon 2500 with 512 MB RAM, and a
software RAID1 root filesystem. I’m running apache1 with mod_perl.

I’d say performance on my system is “usable” but not speedy. On a
pretty big box. But RT is also doing a LOT of stuff under the hood, so
I have no complaints. I also hear that apache1/mod_perl is slower than
other options, but haven’t had time to fiddle with other setups.

You didn’t mention ANY of the above variables, so it’s really hard to
make a remote guess at what your problem might be, but I’d say the
Celeron 600 is probably the biggest bottleneck. When you’re doing live
queries, etc… what does the load average look like on the machine and
which processes are taking up CPU? How much RAM do you have? (I
noticed large performance increases when I tweaked MySQL to allow for
bigger queries and larger result sets as documented here on the lists a
few times, but the trade-off is that 512MB RAM really isn’t enough –
the machine is hitting swap from time to time as it handles other
services also, and I need to add RAM to alleviate that.)

Gotta run, but we all probably need more info. If your complaint is a
general one regarding performance I’m sure Jesse and gang would happily
accept any thoughts on how to improve it.

Nate Duehr, nate@natetech.com

6

For quite some time we have been running RT3 to track support issues,
following my recommendation of the software. Although various
configurations have been used, I am currently debugging an instance
running on a Debian testing box with the specification below because the
faster production machine (running Mandrake release 9) cannot be
tinkered with for the moment and performace problems plague both.

How much RAM? and post your my.cnf - tweaking it can make a huge
difference.

Joe Micciche

7

I have also had major issues with performance. We have a quad Xeon with
a 1 gig ram, and 15000RPM SCSI drives, that we use for development. We
have about 2000 tickets on our system and the attachments table is
about 100 to 150 megs already. We are also using FastCGI,Apache MySQL 4
, RT 3.2.1.

When I run IE against RT it sometimes take up to a minute before the
ticket is shown.
I have tried Firefox and found that as the info is retrieved fro the DB,
it is displayed. The header will sometime appear a bit slow ( about 5 to
10 seconds) and then the rest of the info comes through one reply or
comment at a time. Still taking about 30 to 40 seconds. When running
top I can see the MasonHandler.fcgi runs at 99% for the duration of the
ticket retrieval.
If another session tries to retrieve a ticket at the some time the
duration of all the ticket retrievals increase.

I have applied the Fast CGI and MYSQL config changes in the WIKIs and no
real improvements.

Leon
Jon Masters wrote:

8

Guys, a lot of words, but no data.
For example DBI profile or perl profile.
MySQL slow log.

Leon wrote:

I have also had major issues with performance. We have a quad Xeon with
a 1 gig ram, and 15000RPM SCSI drives, that we use for development. We
have about 2000 tickets on our system and the attachments table is
about 100 to 150 megs already. We are also using FastCGI,Apache MySQL 4
, RT 3.2.1.
We have 7GB of data on IDE with one CPU. Apache1+mp1+MySQL4.0.x. And
ticket loads less then 15 seconds in most cases.

When I run IE against RT it sometimes take up to a minute before the
ticket is shown.
I have tried Firefox and found that as the info is retrieved fro the DB,
it is displayed. The header will sometime appear a bit slow ( about 5 to
10 seconds) and then the rest of the info comes through one reply or
comment at a time. Still taking about 30 to 40 seconds. When running
top I can see the MasonHandler.fcgi runs at 99% for the duration of the
ticket retrieval.
top is not an answer. For example if MySQL can’t do task in memory and
start use tmptables then you can’t see big CPU usage by MySQL, but HDD
activity is high.

If another session tries to retrieve a ticket at the some time the
duration of all the ticket retrievals increase.

I have applied the Fast CGI and MYSQL config changes in the WIKIs and no
real improvements.
Which? I don’t remember any MySQL tricks on WiKi.

9

I Will try the debugging tomorrow at the office.

The MySQL entry was related to large attachments that wasn’t processed
correctly and I changed ti the InnoDB format.
[Request Tracker Wiki]

Thanks for the Debug link.

Leon
Ruslan U. Zakirov wrote:

10

I’m running RT 3.0.3 on a Sun V60x, 2x2.8 GHz CPUs, 4 GB RAM, 10k RPM
internal disk, RHEL 2.1WS, apache 1.3.27 w/mod_perl as a DSO, perl
5.6.1, pgsql 7.3.3–all as supplied with RHEL. Required Perl mods
added when the server was built (not by me, though) via CPAN. Nothing
was built by us (trying to use off-the-shelf software).

I also see poor performance, with the time to display the Update page
being almost 90 seconds from the time of clicking the Reply link at the
top of a ticket display page. I have OS kernel & pgsql config settings
as per the pgsql manual. I turned query logging on for the duration of
bringing up the Update page for a ticket. I can supply the output of
that (it’s about 130 K, so I didn’t want to clog the list with it).

We started seeing the slowdown around ticket 80-100; we’re up to about
1100 now. The slowdown was progressively worse, too. I finally got
permission to address it around ticket 1000, when I made my query log.
I found the shared memory in both the OS & pgsql at their defaults, so
I cranked 'em up to the recommended values, but that didn’t help any at
all. So I joined this list to find out what we’re doing wrong (it’s a
popular product, so it can’t really be this slow; ergo, it must be our
configurations that are preventing it from shining).

I’ll gladly supply any hard data you need to help debug this. I’m not
a Perl programmer at all (can’t write a lick, but can understand
well-written code; I used to write C), so please be concise in asking
for what you need. Thanks!

-Guy

11

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Vincent wrote:

| [Please don’t cross-post to rt-users and rt-devel. Most folks on rt-devel
| are also on RT-Users]

Noted.

| So. How much RAM do you have in these machines?

Sufficient level that vmstat monitoring confirms that swapping is not
the issue here Jesse. Production box is not paging out at all.

Production: 1GB. The box has a load due to mail connections however this
is negligable compared with the load occuring due to RT perl.
Development: Box down due to power failure in office and I am at home
now (01:08). I believe it has something in the region of 256MB RAM.

|>Performance is…well horrible. Painfully unpleasant and this is not
|>with the 40,000 tickets quoted in RT documentation. This is with a
|>database not exceeding about 50-100MB in size and with only a handfull
|>of tickets in it. In other words there are almost no tickets.

| How slow is slow?

10 seconds or more to display a queue and then 30 seconds or longer to
display the larger tickets but always more than would seem sane.

Jon.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBECn9eTyyexZHHxERAjiWAKCcUtd6rvehhACj3OSAuQjHdST3WACeILBe
pVUVSQSqhXqIK5CwXZKRYdY=
=qEkZ
-----END PGP SIGNATURE-----

12

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ruslan U. Zakirov wrote:

jcm>> Debian is configured correctly).

| so as I understand no memory presure.

None.

jcm>> To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).

| Turn on log of slow queries with limit 1 second.

I will look in to this.

| Request Tracker Wiki
| Also can you create full SQL log for one request to Ticket/Display page?

The problem here again will be producing some fake data to send to you
so I will need some time since I am at a conference tomorrow.

|> I have also installed RT 3.2.0 as well as other versions.
|
| Does it make any sense if you switch to 3.0.x.

None.

jcm>> I have performed database OPTIMIZE, ANALYZE, etc. etc.

| It doesn’t help if you are on InnoDB. Are you?

I tested my configurations with each MySQL table type in turn and each
time re-installed and reconfigured mysql from a dump.

| You didn’t report what versions of modules and perl do you have?
| mod_perl 1.x or 2.x? if 1.x then DSO or Static?

Tested on mod_perl 1.x and 2.x compiled as DSO from Debian.

| This output isn’t normal

I know. Hence the reason for posting it - do you think compiling apache
up from source is going to make that much performance difference?

The trace output suggests that SearchBuilder has some pathological
problem but that output came from a release from CPAN yesterday.

Cheers,

Jon.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBECwNeTyyexZHHxERAsFCAJwPV/A9+6jOKAEnCyvlb+yOF6e0ZQCgnCJl
bxJbCamUn38WCbe2EkicLcY=
=auBK
-----END PGP SIGNATURE-----

13

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[ apologies for breaking threading for some folks here. Please CC me
with replies as I am subscribed but not currently filing your list. ]

Nate Duehr writes:

| An upgrade to Perl 5.8 just hit my Debian Testing machine recently.

This issue has been ongoing for many months now.

| I have a 3500 ticket system with lots of file attachments

We get lots of attachments with customers sending in multiple data sets
for analysis (I work at an instrumentation company). The problem
actually is exacerrated by a large quantity of mail attachments however
this does not appear to be the root cause.

| and InnoDB turned on in MySQL (it’s off by default if you upgraded to
| Testing and didn’t turn it on).

Oh but I did. In fact I tried a number of different MySQL configurations
and even tinkered with the indexing that RT had chosen. I am not now
thinking that MySQL is at fault here but will run the slow query logging
to provide some figures.

| The machine is an Athlon 2500 with 512 MB RAM

The processor in the production box is something over 1GHz with 1GB RAM:

AMD Duron™ processor 1104.978 64 BK cache size.

| and asoftware RAID1 root filesystem.

No software RAID on those boxen to slow down write or speed up read IO.
Note that both kernel 2.4 and 2.6 are running on separate machines.

| I’m running apache1 with mod_perl.

Currently so am I. I have tried apache2 and numerous other possible
configurations which might help.

| I’d say performance on my system is “usable” but not speedy.

I get people moaning on a daily basis that RT is various four letter
words that I will refrain from posting here. It is gets annoying.

| I’d say the Celeron 600 is probably the biggest bottleneck

I noted that this is just for the test box I am using. The test box was
installed in order to agressively screw with RT without having to do
things on a box people were trying to use to deal with customers.
Loading times over 5-10 seconds are out of whack in any case.

| what does the load average look like on the machine and
| which processes are taking up CPU?

There is only one process running most of the time and that is whatever
is running the perl - be it apache modperl or whatever.

| How much RAM do you have?

Plenty.

| I noticed large performance increases when I tweaked MySQL

I have tried every standard my.cnf provided for different workloads and
have tweaked numerous tunables without any performance improvement.

| the machine is hitting swap from time to time

This machine is not paging out to disk under normal circumstances. I am
fully aware of standard UNIX sysadmin tuning and so on.

Jon.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBEDQ4eTyyexZHHxERAvmfAJ0f5ouk1jjrUIqBk3ugul0LYy9S8wCgkQpO
aTpFV8tn2mPZwf3tbcNozcI=
=BbcB
-----END PGP SIGNATURE-----

14

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Leon writes:

| When I run IE against RT it sometimes take up to a minute before
| the ticket is shown.

Sounds familiar. Not quite at the punch-the-monitor-hard stage but some
folks are definately not happy with the software at the moment.

| I have tried Firefox and found that as the info is retrieved fro the
| DB, it is displayed.

Indeed. That is not really a fix though it helps me out sometimes.

| The header will sometime appear a bit slow ( about 5 to
| 10 seconds)

Indeed.

| and then the rest of the info comes through one reply or
| comment at a time. Still taking about 30 to 40 seconds.

Indeed.

| When running top I can see the MasonHandler.fcgi runs at 99% for the
| duration of the ticket retrieval.

Indeed. However it is not actually issuing many system calls other than
occasional poll and block on a socket which is probably connected to the
MySQL server. Perhaps there is a problem at the MySQL end but queries
seem to be running - although I will do the slow queries logging.

Jon.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBEDWeeTyyexZHHxERAnicAJ9cJzIuQE86Fnk/CbRUdL9ioUmxqwCgiPah
tZykyMym7n75Lruz2J0gGkQ=
=GbZj
-----END PGP SIGNATURE-----

15

Jon Masters wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ruslan U. Zakirov wrote:

jcm>> Debian is configured correctly).

| so as I understand no memory presure.

None.

jcm>> To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).

| Turn on log of slow queries with limit 1 second.

I will look in to this.

| http://wiki.bestpractical.com/?Debug
| Also can you create full SQL log for one request to Ticket/Display page?

The problem here again will be producing some fake data to send to you
so I will need some time since I am at a conference tomorrow.

|> I have also installed RT 3.2.0 as well as other versions.
|
| Does it make any sense if you switch to 3.0.x.

None.

jcm>> I have performed database OPTIMIZE, ANALYZE, etc. etc.

| It doesn’t help if you are on InnoDB. Are you?

I tested my configurations with each MySQL table type in turn and each
time re-installed and reconfigured mysql from a dump.

| You didn’t report what versions of modules and perl do you have?
| mod_perl 1.x or 2.x? if 1.x then DSO or Static?
mod_perl1 as DSO(is it working? :slight_smile: )
see special notes about MP1+DSO.
Request Tracker Wiki (mod_perl section)
Request Tracker Wiki

Tested on mod_perl 1.x and 2.x compiled as DSO from Debian.

| This output isn’t normal

I know. Hence the reason for posting it - do you think compiling apache
up from source is going to make that much performance difference?
Sometimes users report that if perl is compiled with own malloc
implementation then mod_perl works much faster.

static mod_perl1 could give you up to 10% perfomance bust.

The trace output suggests that SearchBuilder has some pathological
problem but that output came from a release from CPAN yesterday.
As far as I remember SB is stable(perfomance context) long time(mysql)
users.

HTML::Mason::Request::comp is very slow. May be you have some problems
with Mason cache? Something that force Mason to build each time
component from scratch.

16

I’m running RT 3.0.3 on a Sun V60x, 2x2.8 GHz CPUs, 4 GB RAM, 10k RPM
internal disk, RHEL 2.1WS, apache 1.3.27 w/mod_perl as a DSO, perl

I’ll gladly supply any hard data you need to help debug this. I’m not
a Perl programmer at all (can’t write a lick, but can understand
well-written code; I used to write C), so please be concise in asking
for what you need. Thanks!

We’ve done a lot of tuning within the RT 3.0.x series. It’s worth
going up to 3.0.11 before spending time trying to tune or profile.

17

Hi Jon,

[ apologies for breaking threading for some folks here. Please CC me
with replies as I am subscribed but not currently filing your list. ]

Done.

Hmm… I just started out with some “basics” - noted that you understand how
to do Unix admin – some don’t, and I’ve spent a lot of time in the last
three years helping newbies even load Linux, so have built up some bad
habits of dumbing everything down to the lowest common denominator. Sorry
about that!

| and InnoDB turned on in MySQL (it’s off by default if you upgraded to
| Testing and didn’t turn it on).

Oh but I did. In fact I tried a number of different MySQL configurations
and even tinkered with the indexing that RT had chosen. I am not now
thinking that MySQL is at fault here but will run the slow query logging
to provide some figures.

Yeah, I saw some of that thread - haven’t run into MySQL being the culprit of
slowdowns on my system that I can tell, anyway.

I’ll paste the appropriate parts of the my.cnf here, but I warn you (and
anyone reading) these settings weren’t gained via any scientific or
knowledgeable method – It was 2AM and I was tinkering and barely reading
documentation and I’m not a MySQL guru or anything close to it. (There are
folks on this list who are MUCH better MySQL tweakers than I am, I’m sure.)

These numbers were STRICTLY a result of a “hey, let’s set these values
insanely high and let the OS sort out the dang RAM issues!” in a
mess-around-with-the-server session one night, very late – and they seemed
to work, so I left them in place.

Heck, I might have some of these values so messed up they’re causing me harm,
I don’t know… this RT setup here isn’t a business or “production” system
other than a group of ten or so volunteer Linux folks would be discouraged if
it broke down.

skip-locking
skip-networking
key_buffer = 256M
max_allowed_packet = 10M
thread_stack = 1M
query_cache_limit = 128M
query_cache_size = 512M
query_cache_type = 1

Let’s see… Debian “testing” branch… Kernel: 2.4.26-1-k7 (it does help to
use the pre-compiled Debian kernel or your own compiled kernel that’s built
for your particular processor… how much I can’t say, but it helped quite a
bit on this Athlon for various other things it does besides RT).

Athlon 2500, 512MB RAM (PC2700), Dual 7200 RPM IDE disks on opposite IDE
controllers (hdparm -t says the buffered read speed from the md device
wanders from 8 MB/s to 12 MB/s… haven’t ever bothered with any bonnie++
tests.)

The processor in the production box is something over 1GHz with 1GB RAM:

Note: Athlon 2500 is a native clock speed of 1.8 GHz, if I remember correctly.

I get people moaning on a daily basis that RT is various four letter
words that I will refrain from posting here. It is gets annoying.

Just a side-thought – have you investigated network overload issues or
bottlenecks in the network getting to/from the RT machine during the
timeframe people are complaining? How many users do you typically have
hitting it simultaneously? I rarely see more than 2 users at a time on
this system but it is doing a number of other things, probably the biggest
one is it also runs spamc for a medium-load mail server.

Loading times over 5-10 seconds are out of whack in any case.

Loading times of what – the queue list, a ticket, a bunch of tickets in a
search? It will probably make a difference. Also how often are “idling”
clients reloading the main page?

There is only one process running most of the time and that is whatever
is running the perl - be it apache modperl or whatever.

I usually only see apache itself jump to the top of a quickly-updating (1
second refresh) top screen, rarely do I see MySQL pop up there long enough to
matter, and I never see anything named “perl”, although of course, I know
it’s running.

Let’s see… what else… um, this is RT 3.0.11 from Debian… can’t remember
if you said which version you were on.

I’m sitting here (yet another late night e-mail/hacking session) trying to
think of other parameters to either share that might help you find your
performance issue, or ones to ask you about. Hmmm.

Just for comparison, I did a search against our tickets that returned 2979
tickets in a search list and I selected “unlimited” number of tickets
displayed at one time. It took 147 seconds for apache to finish handing it
all out to my laptop over 802.11g wireless with konqueror showing an average
of 12.0 Kb/s throughput to the browser during the transfer. Apache and
MySQL fought over the single CPU with apache always much heavier than mysql
– typical numbers were 80% apache, <20% mysql with other sundry apps running
at the same time as well.

Like I said, “usable” but definitely not “speedy”. It’s really rare that
anyone would ever need to show all those tickets at the same time in a search
like that.

For the more typical “show 50 tickets” the query was over and the data
displayed in less than 3 seconds. Closer to 2, but too fast for me to really
measure easily.

Pulling up one of those tickets with a 43.5 K file attachment was also too
fast to measure realistically.

Okay, I’m fading fast… time for bed… hopefully something here helps.
Nothing here was done to the level of professionalism I have to deal with at
work – this box is just a hodgepodge of various tools and toys for home
stuff.

Nate Duehr, nate@natetech.com

18

Nate Duehr wrote:

Hmm… I just started out with some “basics”

This is why I always feel compelled to write extremely verbose initial
messages on these lists - I have to instantly demonstrate to you that as
someone who spends most of the day hacking on embedded boxes and does
occasional sysadmin we can just get right down to the point :-).

haven’t run into MySQL being the culprit

In my experience on small scale systems MySQL is ressonable.

Just a side-thought – have you investigated network overload issues or
bottlenecks in the network getting to/from the RT machine

There are no problems.

during the timeframe people are complaining?

Continuously.

How many users do you typically have hitting it simultaneously?

One or two users. This is what I would call a small installation which
should not push RT at all but performance is dreadful.

jcm>>Loading times over 5-10 seconds are out of whack in any case.

Loading times of what – the queue list

That takes at least 5-10 seconds but perhaps much longer.

a ticket

That takes perhaps even a minute.

Also how often are “idling” clients reloading the main page?

Usually one person uses RT and rarely will someone else in the building
have a ticket to look at or be covering if our support guy is away.

There is only one process running most of the time and that is whatever
is running the perl - be it apache modperl or whatever.

I usually only see apache itself jump to the top of a quickly-updating (1
second refresh) top screen

That is my point.

rarely do I see MySQL pop up there long enough to matter

Which of course means nothing since it could be blocking on IO and
apache might just be sitting in a tight pointless loop.

and I never see anything named “perl”

However you will not because you are using mod_perl as am I.

Let’s see… what else… um, this is RT 3.0.11 from Debian… can’t remember
if you said which version you were on.

Pretty much every release in the 3.x series from bestpractical.com.

For the more typical “show 50 tickets” the query was over and the data
displayed in less than 3 seconds.

That would be fine.

Pulling up one of those tickets with a 43.5 K file attachment was also too
fast to measure realistically.

That would be fine.

Jon.

19

Athlon 2500, 512MB RAM (PC2700), Dual 7200 RPM IDE disks on opposite IDE
controllers (hdparm -t says the buffered read speed from the md device
wanders from 8 MB/s to 12 MB/s… haven’t ever bothered with any bonnie++
tests.)

That’s pretty slow for modern equpment. I usually see 60+MB on single
or software-mirrored scsi and 40-60MB on IDE (don’t have
any raided IDE…). Be sure you have good 80-wire cables and
you see something like:
hda: 195371568 sectors (100030 MB) w/2048KiB Cache, CHS=12161/255/63,
UDMA(100)
in dmesg.

That may not be the real problem, but it is something you
should be able to improve.

Les Mikesell
les@futuresource.com

20

I have a related question regarding RT performance.

Due to many interventions in the code base, a company I’m consulting to
is stuck with rt2.

I want to use mod_proxy to speed up the performance. However, I have
read that for caching using mod_proxy:

You have to produce correct Content-Length, Last-Modified and Expires
http headers for it to work.

As far as I can see, RT2 is not sending these headers.

Has any got this working (apache, mod_perl, rt + mod_proxy)? Where can
the headers safely be calculated and sent? Are there any issues with
this setup? Links to documentation?

Thanks.

Jason Armstrong