Httpd speed on rt3

RT Users
1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I appologize if this has been covered in the past; please let me know if it
has.

I’m running a Dual P3/1.26GHz w/ 1G of RAM on Fedora Core 1, with Apache
1.3.29 and mod_perl 1.29 compiled in and MySQL 4.0.17 RPMs, using RT 3.0.8.
Viewing tickets seems especially slow. Most tickets take 8 seconds or longer
to load, and one ticket in particular takes about 16 seconds. On my much
humbler box running RT2, it takes about half that time. During this time, it
doesn’t appear to be MySQL that’s using CPU or disk, but httpd that’s hogging
up 100% of one CPU.

I’d really appreciate it if anybody has any ideas on how to reduce the load
time for tickets, or what I can do to track down why tickets are taking so
long to display.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAB0YDJV36su0A0xIRAgaUAJ4oSNWnu+6/bFqpmBIIacdRsd+UfACgy4gc
FQyHgaosDFe1erCgK1Uj6+s=
=U52p
-----END PGP SIGNATURE-----

2

I’m seeing similar performance problems. Pls check out the thread @
rt-devel from Jan 15th.

I run fastcgi, so I see the Perl process eating CPU cycles, since you run
mod_perl, you see the httpd process hogging, since that’s where perl runs.
It’s definitely perl doing some bad stuff.

I’ve no idea what happens yet, but just wanted to tell you that you’re not
alone, FWIW. :slight_smile:

Palle–On torsdag, januari 15, 2004 21.01.39 -0500 “Douglas E. Warner” dwarner@ctinetworks.com wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I appologize if this has been covered in the past; please let me know if
it has.

I’m running a Dual P3/1.26GHz w/ 1G of RAM on Fedora Core 1, with Apache
1.3.29 and mod_perl 1.29 compiled in and MySQL 4.0.17 RPMs, using RT
3.0.8. Viewing tickets seems especially slow. Most tickets take 8
seconds or longer to load, and one ticket in particular takes about 16
seconds. On my much humbler box running RT2, it takes about half that
time. During this time, it doesn’t appear to be MySQL that’s using CPU
or disk, but httpd that’s hogging up 100% of one CPU.

I’d really appreciate it if anybody has any ideas on how to reduce the
load time for tickets, or what I can do to track down why tickets are
taking so long to display.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAB0YDJV36su0A0xIRAgaUAJ4oSNWnu+6/bFqpmBIIacdRsd+UfACgy4gc
FQyHgaosDFe1erCgK1Uj6+s=
=U52p
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

3

Consider that MySQL may not be optimized on the RT3 machine, and what you
are seeing with Perl is actually just Perl waiting on the records to be
retrieved. There are quite a few joins in the SQL queries, so an
unoptimized database could very well be slowing things down.

In my personal experience, as well, MySQL 3.23.xx always seemed to run
better for me than did 4.0.xx, regardless of what I did to improve the
speed. Maybe it was just me but that was my experience.From: Palle Girgensohn [mailto:girgen@pingpong.net]
Sent: Saturday, January 17, 2004 11:15 PM
To: Douglas E. Warner; rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

I’m seeing similar performance problems. Pls check out the thread @
rt-devel from Jan 15th.

I run fastcgi, so I see the Perl process eating CPU cycles, since you run
mod_perl, you see the httpd process hogging, since that’s where perl runs.
It’s definitely perl doing some bad stuff.

I’ve no idea what happens yet, but just wanted to tell you that you’re not
alone, FWIW. :slight_smile:

Palle

4

I use postgresql. And, I did a check as to how much time was spent by
postgresql handling the queries, and althogh the same question was asked
over a hundred times, which surely is a bug, postgresql spent around 800 ms
doing queries. The request tooks six seconds. And, perl waiting for
postgresql hardly eats CPU, it’s a network connection (or in this case unix
domain socket) waiting for data. Most of the queries where pretty simple,
only a few took more than a couple of milliseconds.

Perl probably goes into a loop handling this same data it gets more than a
hundred times, but it hardly eats cycles while waiting for it to come.

As for MySQL, I have no experience, but isn’t 4.x much more competent in
terms of what SQL it handles, compared to 3.x? This perhaps comes with a
price, a performance penalty? But as I said, I have no experience with
MySQL, been using postgresql since 1995… :slight_smile:

Palle–On tisdag, januari 20, 2004 09.34.46 -0500 Mike Frazer Michael.Frazer@InterCept.Net wrote:

Consider that MySQL may not be optimized on the RT3 machine, and what you
are seeing with Perl is actually just Perl waiting on the records to be
retrieved. There are quite a few joins in the SQL queries, so an
unoptimized database could very well be slowing things down.

In my personal experience, as well, MySQL 3.23.xx always seemed to run
better for me than did 4.0.xx, regardless of what I did to improve the
speed. Maybe it was just me but that was my experience.

-----Original Message-----
From: Palle Girgensohn [mailto:girgen@pingpong.net]
Sent: Saturday, January 17, 2004 11:15 PM
To: Douglas E. Warner; rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

I’m seeing similar performance problems. Pls check out the thread @
rt-devel from Jan 15th.

I run fastcgi, so I see the Perl process eating CPU cycles, since you run
mod_perl, you see the httpd process hogging, since that’s where perl
runs. It’s definitely perl doing some bad stuff.

I’ve no idea what happens yet, but just wanted to tell you that you’re
not alone, FWIW. :slight_smile:

Palle

–On torsdag, januari 15, 2004 21.01.39 -0500 “Douglas E. Warner” dwarner@ctinetworks.com wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I appologize if this has been covered in the past; please let me know if
it has.

I’m running a Dual P3/1.26GHz w/ 1G of RAM on Fedora Core 1, with Apache
1.3.29 and mod_perl 1.29 compiled in and MySQL 4.0.17 RPMs, using RT
3.0.8. Viewing tickets seems especially slow. Most tickets take 8
seconds or longer to load, and one ticket in particular takes about 16
seconds. On my much humbler box running RT2, it takes about half that
time. During this time, it doesn’t appear to be MySQL that’s using CPU
or disk, but httpd that’s hogging up 100% of one CPU.

I’d really appreciate it if anybody has any ideas on how to reduce the
load time for tickets, or what I can do to track down why tickets are
taking so long to display.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAB0YDJV36su0A0xIRAgaUAJ4oSNWnu+6/bFqpmBIIacdRsd+UfACgy4gc
FQyHgaosDFe1erCgK1Uj6+s=
=U52p
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

5

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1On Tuesday 20 January 2004 09:34, Mike Frazer wrote:

Consider that MySQL may not be optimized on the RT3 machine, and what you
are seeing with Perl is actually just Perl waiting on the records to be
retrieved. There are quite a few joins in the SQL queries, so an
unoptimized database could very well be slowing things down.

In my personal experience, as well, MySQL 3.23.xx always seemed to run
better for me than did 4.0.xx, regardless of what I did to improve the
speed. Maybe it was just me but that was my experience.

Well, as I said, I captured the queries that the page loads were doing, and
could replay them in about half a second (out of 8 seconds for the page
load). On top of that, if MySQL were busy doing a join I would see it eating
CPU, but I don’t. Only httpd. I think Palle is on the money that it’s perl.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADqfpJV36su0A0xIRAqyKAJ9I0KqctKTuGNCmpNQ/F6Qv2HpoWgCgrYWY
DLcIYkrGGeAfbs75Ak0Vy4Q=
=5An2
-----END PGP SIGNATURE-----

6

Well, as I said, I captured the queries that the page loads were doing, and
could replay them in about half a second (out of 8 seconds for the page
load). On top of that, if MySQL were busy doing a join I would see it eating
CPU, but I don’t. Only httpd. I think Palle is on the money that it’s perl.

Can you replay them in half a second on a freshly restarted mysql
database? Mysql’s query cache should make sure that the queries are
lightning fast if they’ve already been run in recent memory.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

7

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1On Wednesday 21 January 2004 11:46, Jesse Vincent wrote:

Can you replay them in half a second on a freshly restarted mysql
database? Mysql’s query cache should make sure that the queries are
lightning fast if they’ve already been run in recent memory.

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like Palle was; there 227
queries for the page load, but there’s 88 transactions (that includes
comments, correspondance, and actions).

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

8

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like Palle was; there 227
queries for the page load, but there’s 88 transactions (that includes
comments, correspondance, and actions).

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t displayed inline. A
customer asked us to quote a fix for them. We’ve sent them the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

Jesse
  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

9

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1On Wednesday 21 January 2004 12:25, Jesse Vincent wrote:

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t displayed inline. A
customer asked us to quote a fix for them. We’ve sent them the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

I know there isn’t any binary attachments in this ticket; it’s all just ASCII
correspondance and comments, and they’re usually < 1k (they’re only a
paragraph).
Other tickets are a good bit faster (usually only 1-3 seconds); I was just
trying to figure out what was going on for a couple of these larger tickets.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrrpJV36su0A0xIRAnQ7AKDVe6+keVrnE750ra0vbrRX44BK5QCeIu8Q
l9fDcAyAvvTzYYp2Oyx/saA=
=2gAN
-----END PGP SIGNATURE-----

10

I know there isn’t any binary attachments in this ticket; it’s all just ASCII
correspondance and comments, and they’re usually < 1k (they’re only a
paragraph).
Other tickets are a good bit faster (usually only 1-3 seconds); I was just
trying to figure out what was going on for a couple of these larger tickets.

Ok. There are two tools that it’s worth using to attempt to debug what
code the performance bottleneck is in: MasonX::Profiler and
Apache::DProf. I’d love to see traces for your longer tickets.

One possible culprit here is the perl module that colorizes indented
text.

-j
  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrrpJV36su0A0xIRAnQ7AKDVe6+keVrnE750ra0vbrRX44BK5QCeIu8Q
l9fDcAyAvvTzYYp2Oyx/saA=
=2gAN
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

11

Have you enabled ExtendedStatus in apache and taken a peek at what it is
doing?

Also there is:

<Location /perl-status>
SetHandler perl-script
PerlHandler Apache::Status
PerlSetVar StatusOptionsAll On
Order Deny,Allow
Deny from All
Allow from

That will show you the Perl config, Memory, Loaded Modules, the stack, etc…

Out of curiosity, as well as CPU, how does the load average and per process
memory consumption on the box look? Top can give you those, and vmstat can
at least give you an idea if processes are getting blocked for some reason.
Also for linux, the following can give you memory usage for httpd:
perl -pe ‘s/^(\w+)-(-w+) /hex($2) - hex($1) . " "/e’ \

</proc/cat <httpd.pidfile>/maps

I know in mysql there are sort and read buffer memory settings. If not set
high enough, performance can degrade.

Mysql 4.0 will do query caching. Set query_cache_size in the my.cnf.
Problem is that table updates will clear the cache and RT being a
transactional system, the cache may never be used for ticket but may be used
for principal, user, and group lookupsFrom: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Palle
Girgensohn
Sent: Tuesday, January 20, 2004 9:06 PM
To: Mike Frazer; Douglas E. Warner; rt-users@lists.fsck.com
Subject: RE: [rt-users] httpd speed on rt3

I use postgresql. And, I did a check as to how much time was spent by
postgresql handling the queries, and althogh the same question was asked
over a hundred times, which surely is a bug, postgresql spent around 800 ms
doing queries. The request tooks six seconds. And, perl waiting for
postgresql hardly eats CPU, it’s a network connection (or in this case unix
domain socket) waiting for data. Most of the queries where pretty simple,
only a few took more than a couple of milliseconds.

Perl probably goes into a loop handling this same data it gets more than a
hundred times, but it hardly eats cycles while waiting for it to come.

As for MySQL, I have no experience, but isn’t 4.x much more competent in
terms of what SQL it handles, compared to 3.x? This perhaps comes with a
price, a performance penalty? But as I said, I have no experience with
MySQL, been using postgresql since 1995… :slight_smile:

Palle

12

The mytop tool myabe can be of some help too:

http://www.freebsd.org/cgi/url.cgi?ports/databases/mytop/pkg-descrFrom: “AJ” rt@musefoundry.com
To: “‘Palle Girgensohn’” girgen@pingpong.net; “‘Mike Frazer’”
Michael.Frazer@InterCept.Net; “‘Douglas E. Warner’”
dwarner@ctinetworks.com; rt-users@lists.fsck.com
Sent: Wednesday, January 21, 2004 4:31 PM
Subject: RE: [rt-users] httpd speed on rt3

Have you enabled ExtendedStatus in apache and taken a peek at what it is
doing?

Also there is:

<Location /perl-status>
SetHandler perl-script
PerlHandler Apache::Status
PerlSetVar StatusOptionsAll On
Order Deny,Allow
Deny from All
Allow from

That will show you the Perl config, Memory, Loaded Modules, the stack,
etc…

Out of curiosity, as well as CPU, how does the load average and per
process
memory consumption on the box look? Top can give you those, and vmstat
can
at least give you an idea if processes are getting blocked for some
reason.
Also for linux, the following can give you memory usage for httpd:
perl -pe ‘s/^(\w+)-(-w+) /hex($2) - hex($1) . " "/e’ \

</proc/cat <httpd.pidfile>/maps

I know in mysql there are sort and read buffer memory settings. If not
set
high enough, performance can degrade.

Mysql 4.0 will do query caching. Set query_cache_size in the my.cnf.
Problem is that table updates will clear the cache and RT being a
transactional system, the cache may never be used for ticket but may be
used
for principal, user, and group lookups

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Palle
Girgensohn
Sent: Tuesday, January 20, 2004 9:06 PM
To: Mike Frazer; Douglas E. Warner; rt-users@lists.fsck.com
Subject: RE: [rt-users] httpd speed on rt3

I use postgresql. And, I did a check as to how much time was spent by
postgresql handling the queries, and althogh the same question was asked
over a hundred times, which surely is a bug, postgresql spent around 800
ms
doing queries. The request tooks six seconds. And, perl waiting for
postgresql hardly eats CPU, it’s a network connection (or in this case
unix
domain socket) waiting for data. Most of the queries where pretty simple,
only a few took more than a couple of milliseconds.

Perl probably goes into a loop handling this same data it gets more than a
hundred times, but it hardly eats cycles while waiting for it to come.

As for MySQL, I have no experience, but isn’t 4.x much more competent in
terms of what SQL it handles, compared to 3.x? This perhaps comes with a
price, a performance penalty? But as I said, I have no experience with
MySQL, been using postgresql since 1995… :slight_smile:

Palle

Consider that MySQL may not be optimized on the RT3 machine, and what
you
are seeing with Perl is actually just Perl waiting on the records to be
retrieved. There are quite a few joins in the SQL queries, so an
unoptimized database could very well be slowing things down.

In my personal experience, as well, MySQL 3.23.xx always seemed to run
better for me than did 4.0.xx, regardless of what I did to improve the
speed. Maybe it was just me but that was my experience.

-----Original Message-----
From: Palle Girgensohn [mailto:girgen@pingpong.net]
Sent: Saturday, January 17, 2004 11:15 PM
To: Douglas E. Warner; rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

I’m seeing similar performance problems. Pls check out the thread @
rt-devel from Jan 15th.

I run fastcgi, so I see the Perl process eating CPU cycles, since you
run
mod_perl, you see the httpd process hogging, since that’s where perl
runs. It’s definitely perl doing some bad stuff.

I’ve no idea what happens yet, but just wanted to tell you that you’re
not alone, FWIW. :slight_smile:

Palle

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I appologize if this has been covered in the past; please let me know
if
it has.

I’m running a Dual P3/1.26GHz w/ 1G of RAM on Fedora Core 1, with
Apache
1.3.29 and mod_perl 1.29 compiled in and MySQL 4.0.17 RPMs, using RT
3.0.8. Viewing tickets seems especially slow. Most tickets take 8
seconds or longer to load, and one ticket in particular takes about 16
seconds. On my much humbler box running RT2, it takes about half
that
time. During this time, it doesn’t appear to be MySQL that’s using
CPU
or disk, but httpd that’s hogging up 100% of one CPU.

I’d really appreciate it if anybody has any ideas on how to reduce the
load time for tickets, or what I can do to track down why tickets are
taking so long to display.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAB0YDJV36su0A0xIRAgaUAJ4oSNWnu+6/bFqpmBIIacdRsd+UfACgy4gc
FQyHgaosDFe1erCgK1Uj6+s=
=U52p
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

13

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1On Wednesday 21 January 2004 12:53, Jesse Vincent wrote:

Ok. There are two tools that it’s worth using to attempt to debug what
code the performance bottleneck is in: MasonX::Profiler and
Apache::DProf. I’d love to see traces for your longer tickets.

I seem to have Apache::Dprof working, but I’m not getting any output from
MasonX::Profiler. I tried putting the config in both the httpd.conf and in
RT_SiteConfig.pl, but neither seemed to generate output in either my error
log or my RT log (w/ log to file set to ‘debug’). Here’s what I’ve tried:

== httpd.conf ==
ServerName rt3.ctinetworks.com
DocumentRoot /usr/local/rt3/share/html
AddDefaultCharset UTF-8

PerlModule Apache::DProf
PerlModule MasonX::Profiler
PerlSetVar MasonPreamble "my $p = MasonX::Profiler->new($m, $r);"
PerlModule Apache::DBI
PerlRequire /usr/local/rt3/bin/webmux.pl

<Location />
    SetHandler perl-script
    PerlHandler RT::Mason
</Location>

ErrorLog logs/rt3-error_log
CustomLog logs/rt3-access_log combined

== RT_SiteConfig.pl ==
@MasonParameters = (preamble => ‘my $p = MasonX::Profiler->new($m, $r);’); ;
with MasonParameters commented out in httpd.conf
Set($LogToFile , ‘debug’);
Set($LogDir, ‘/usr/local/rt3/var/log’);
Set($LogToFileNamed , “rt.log”); #log to rt.log

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADtiHJV36su0A0xIRAkPwAJ9fxqXkmGgczUADutAbBU1B1wUxRACfQI9h
G+5REt453kwSAP1gei4XNAE=
=+KK/
-----END PGP SIGNATURE-----

14

Perhaps stderr isn’t being logged.

Try starting apache with:
<apache_dir>/bin/httpd -X -f <apache_dir>/conf/httpd.conf 2>errfile

errfile = path and filename of error file
outputfile = path and filename of standard output file

start apache in single process mode and log errors to the stderr. Easier to
trace thenFrom: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Douglas E.
Warner
Sent: Wednesday, January 21, 2004 2:53 PM
To: rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

15

All this talk about performance has got me interested in it again. It
would be nice if RT were a bit snappier.

Searches such as "show all tickets from queue X that are new or open"
take 3-4 seconds on our dual 1.3 GHz 1.5 GB RAM Mac OS X 10.2.8 box
(Perl 5.8.3, PostgreSQL 7.3.4, Apache 1.3.29, mod_perl to match, RT
3.0.8, DBIx::SearchBuilder 0.96, well-vacuumed DB with all recommended
indexes added, and no load at time of test). Our database has a measly
789 tickets, 34 users, 9 queues, 2 groups, and almost never has
simultaneous accesses.

Does this performance sound about right to people? Would switching to
MySQL be worthwhile? (We don’t use postgres for anything else, as it
turns out).

The above search translates into 176 database accesses, including 172
queries, the sum of whose durations is 1.076 seconds, according to the
postgres log. Also according to the PG log, these queries span a
little more than 3 seconds, so there’s more than 2 additional seconds
to account for; I can see httpd/perl using a lot of CPU in ‘top’. The
SQL queries in the log are mysterious to me, but they seem to be doing
a bunch of permissions-type checks.

I realize that RT’s great features necessarily impose significant
overhead, but I’d like to be reassured that nothing is wrong with my
installation.

Thanks for any insight.

-Kevin

16

I have a few timings which might throw some more light on this.

I’m running RT 3.0.7 over postgresql 7.4 on Solaris.

A ticket with 2 small comments takes about 5 seconds to retrieve,
but a ticket with 10 small comments (200 to 500 bytes) and 4 attachments
(50K each)
takes about 18 seconds to retrieve.

My profile of executions shows that there are 106 SELECT statements, most
of which take less than 5 ms. There are 8 permissions checks which take
about 20 to 30 ms each. This gives a total sql time of less than 1 second.

The break down of wall-clock execution time is roughly as follows:

Check permissions; get ticket, custom fields, transaction list - about 3
seconds
Get comments/attachments - 14 seconds.

So it seems fairly clearly that the killer here is the time to process
attachments.

Is it possible to display only the last comment by default, and display all
only
if the user asks for history? Would this be an acceptable compromise?

Mike

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Jesse
Vincent
Sent: Thursday, 22 January 2004 3:26 AM
To: Douglas E. Warner
Cc: rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like
Palle was; there 227
queries for the page load, but there’s 88 transactions
(that includes
comments, correspondance, and actions).

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter
tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t
displayed inline. A
customer asked us to quote a fix for them. We’ve sent them
the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

Jesse

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at
http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.
rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

17

in Ticket/Elements/ShowHistory:

ABOVE:
% while (my $Transaction = $Transactions->Next) {

Put:

% if ($m->request_comp()->title=~/Ticket/Display.html/) {
% my $Transaction = $Transactions->Last;
<& ShowTransaction, Ticket => $Ticket, Transaction => $Transaction,
ShowHeaders => $ShowHeaders, Collapsed => $Collapsed, RowNum => $i,
ShowTitleBarCommands => $ShowTitleBarCommands, %ARGS &>
%}

Moves cursor to the last one and prints it. The hits the while loop but
since you are already at the end goes right over it.

What version of Solaris you run? I originally had the front end of RT on a
fully loaded E250 with Solaris 9 and the db on an 8 processor E3500 with 4.5
gigs ram and it was still slow. I moved the front end to linux and it made a
big difference. Also, there is one param that should be set on Solaris that
will take care of the possible Log::Dispatch errors and allow you to log to
syslog if you want.

A.J.From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Mike Husband
Sent: Wednesday, January 21, 2004 8:46 PM
To: Rt-Users (E-mail)
Subject: RE: [rt-users] httpd speed on rt3

I have a few timings which might throw some more light on this.

I’m running RT 3.0.7 over postgresql 7.4 on Solaris.

A ticket with 2 small comments takes about 5 seconds to retrieve,
but a ticket with 10 small comments (200 to 500 bytes) and 4 attachments
(50K each)
takes about 18 seconds to retrieve.

My profile of executions shows that there are 106 SELECT statements, most
of which take less than 5 ms. There are 8 permissions checks which take
about 20 to 30 ms each. This gives a total sql time of less than 1 second.

The break down of wall-clock execution time is roughly as follows:

Check permissions; get ticket, custom fields, transaction list - about 3
seconds
Get comments/attachments - 14 seconds.

So it seems fairly clearly that the killer here is the time to process
attachments.

Is it possible to display only the last comment by default, and display all
only
if the user asks for history? Would this be an acceptable compromise?

Mike

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Jesse
Vincent
Sent: Thursday, 22 January 2004 3:26 AM
To: Douglas E. Warner
Cc: rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like
Palle was; there 227
queries for the page load, but there’s 88 transactions
(that includes
comments, correspondance, and actions).

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter
tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t
displayed inline. A
customer asked us to quote a fix for them. We’ve sent them
the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

Jesse

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at
http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.
rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

18

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Perhaps stderr isn’t being logged.

Try starting apache with:
<apache_dir>/bin/httpd -X -f <apache_dir>/conf/httpd.conf 2>errfile

You missed the ‘>outputfile’ part :wink:

errfile = path and filename of error file
outputfile = path and filename of standard output file

start apache in single process mode and log errors to the stderr. Easier to
trace then

Still no luck. Any more ideas? I’d love to get you some performance data.

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFAEDC7JV36su0A0xIRAik/AKD9v12NFYt+ui0V2kWhoSI7PxCbRgCgog+q
efrCAYQ8IfHrHW78CibViac=
=Fx+D
-----END PGP SIGNATURE-----

19

Actually Mike reminded me that I should put a correction to what I put
before.

Only one line.

Right under the following line in the html/Ticket/ShowHistory component:
% while (my $Transaction = $Transactions->Next) {

Put:

% $Transaction = $Transactions->Last if
($m->request_comp()->title=~/Ticket/Display.html/);

Display will only show last transaction. History and modify will show all.From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of AJ
Sent: Thursday, January 22, 2004 2:50 AM
To: ‘Mike Husband’; ‘Rt-Users (E-mail)’
Subject: RE: [rt-users] httpd speed on rt3

in Ticket/Elements/ShowHistory:

ABOVE:
% while (my $Transaction = $Transactions->Next) {

Put:

% if ($m->request_comp()->title=~/Ticket/Display.html/) {
% my $Transaction = $Transactions->Last;
<& ShowTransaction, Ticket => $Ticket, Transaction => $Transaction,
ShowHeaders => $ShowHeaders, Collapsed => $Collapsed, RowNum => $i,
ShowTitleBarCommands => $ShowTitleBarCommands, %ARGS &>
%}

Moves cursor to the last one and prints it. The hits the while loop but
since you are already at the end goes right over it.

What version of Solaris you run? I originally had the front end of RT on a
fully loaded E250 with Solaris 9 and the db on an 8 processor E3500 with 4.5
gigs ram and it was still slow. I moved the front end to linux and it made a
big difference. Also, there is one param that should be set on Solaris that
will take care of the possible Log::Dispatch errors and allow you to log to
syslog if you want.

A.J.

From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Mike Husband
Sent: Wednesday, January 21, 2004 8:46 PM
To: Rt-Users (E-mail)
Subject: RE: [rt-users] httpd speed on rt3

I have a few timings which might throw some more light on this.

I’m running RT 3.0.7 over postgresql 7.4 on Solaris.

A ticket with 2 small comments takes about 5 seconds to retrieve,
but a ticket with 10 small comments (200 to 500 bytes) and 4 attachments
(50K each)
takes about 18 seconds to retrieve.

My profile of executions shows that there are 106 SELECT statements, most
of which take less than 5 ms. There are 8 permissions checks which take
about 20 to 30 ms each. This gives a total sql time of less than 1 second.

The break down of wall-clock execution time is roughly as follows:

Check permissions; get ticket, custom fields, transaction list - about 3
seconds
Get comments/attachments - 14 seconds.

So it seems fairly clearly that the killer here is the time to process
attachments.

Is it possible to display only the last comment by default, and display all
only
if the user asks for history? Would this be an acceptable compromise?

Mike

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Jesse
Vincent
Sent: Thursday, 22 January 2004 3:26 AM
To: Douglas E. Warner
Cc: rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like
Palle was; there 227
queries for the page load, but there’s 88 transactions
(that includes
comments, correspondance, and actions).

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter
tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t
displayed inline. A
customer asked us to quote a fix for them. We’ve sent them
the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

Jesse

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at
http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.
rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

20

AJ,

This has sped things up for me.

However, my users want to see this first transaction (typically the body of
an email sent in to create the ticket) and the last one.

What would be the fix for that?

Thanks,

SHFrom: AJ [mailto:rt@musefoundry.com]
Sent: Thursday, January 22, 2004 8:08 PM
To: ‘AJ’; ‘Rt-Users (E-mail)’
Subject: RE: [rt-users] httpd speed on rt3

Actually Mike reminded me that I should put a correction to what I put
before.

Only one line.

Right under the following line in the html/Ticket/ShowHistory component:
% while (my $Transaction = $Transactions->Next) {

Put:

% $Transaction = $Transactions->Last if
($m->request_comp()->title=~/Ticket/Display.html/);

Display will only show last transaction. History and modify will show all.

From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of AJ
Sent: Thursday, January 22, 2004 2:50 AM
To: ‘Mike Husband’; ‘Rt-Users (E-mail)’
Subject: RE: [rt-users] httpd speed on rt3

in Ticket/Elements/ShowHistory:

ABOVE:
% while (my $Transaction = $Transactions->Next) {

Put:

% if ($m->request_comp()->title=~/Ticket/Display.html/) {
% my $Transaction = $Transactions->Last;
<& ShowTransaction, Ticket => $Ticket, Transaction => $Transaction,
ShowHeaders => $ShowHeaders, Collapsed => $Collapsed, RowNum => $i,
ShowTitleBarCommands => $ShowTitleBarCommands, %ARGS &>
%}

Moves cursor to the last one and prints it. The hits the while loop but
since you are already at the end goes right over it.

What version of Solaris you run? I originally had the front end of RT on a
fully loaded E250 with Solaris 9 and the db on an 8 processor E3500 with 4.5
gigs ram and it was still slow. I moved the front end to linux and it made a
big difference. Also, there is one param that should be set on Solaris that
will take care of the possible Log::Dispatch errors and allow you to log to
syslog if you want.

A.J.

From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Mike Husband
Sent: Wednesday, January 21, 2004 8:46 PM
To: Rt-Users (E-mail)
Subject: RE: [rt-users] httpd speed on rt3

I have a few timings which might throw some more light on this.

I’m running RT 3.0.7 over postgresql 7.4 on Solaris.

A ticket with 2 small comments takes about 5 seconds to retrieve,
but a ticket with 10 small comments (200 to 500 bytes) and 4 attachments
(50K each)
takes about 18 seconds to retrieve.

My profile of executions shows that there are 106 SELECT statements, most
of which take less than 5 ms. There are 8 permissions checks which take
about 20 to 30 ms each. This gives a total sql time of less than 1 second.

The break down of wall-clock execution time is roughly as follows:

Check permissions; get ticket, custom fields, transaction list - about 3
seconds
Get comments/attachments - 14 seconds.

So it seems fairly clearly that the killer here is the time to process
attachments.

Is it possible to display only the last comment by default, and display all
only
if the user asks for history? Would this be an acceptable compromise?

Mike

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com]On Behalf Of Jesse
Vincent
Sent: Thursday, 22 January 2004 3:26 AM
To: Douglas E. Warner
Cc: rt-users@lists.fsck.com
Subject: Re: [rt-users] httpd speed on rt3

After fresh restart:
displaying ticket: 10s
replaying queries: 1.5s

Then (not restarted):
displaying ticket: 9s
replaying queries: 0.06s

I’m not seeing the same queries over and over again like
Palle was; there 227
queries for the page load, but there’s 88 transactions
(that includes
comments, correspondance, and actions).

Ok. I think there’s the thing that’s taking the time. how long are the
comments and correspondence? Are there attachments to the ticket? I’m
betting that you don’t see such performance issues on shorter
tickets.

This is a known issue with long tickets with a lot of attachment
content, especially “binary” attachments that aren’t
displayed inline. A
customer asked us to quote a fix for them. We’ve sent them
the quote and
are just waiting for them to decide whether or not to go for it. (It
involves a bunch of changes to RT’s core.) Ruslan sent a draft of a
possible fix to rt-devel as well, which may work for you.

Jesse

  • -Doug

Douglas E. Warner dwarner@ctinetworks.com Network Engineer
CTI/PAdotNET http://ctinetworks.com +1 717 975 9000
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQFADrCVJV36su0A0xIRAsZjAJ9T4bcmFklLUi/tWCW+MuE5Ja5UWACg3jEw
P4Ti1N5piu5t7htri1lrRNk=
=3w72
-----END PGP SIGNATURE-----

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at
http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.
rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm