Performance Issues

RT Users
21

For me, the biggest indicator of how long it will take to display a
ticket is how much correspondence it has, and if any of those include
large attachments such as files. There is a direct correlation
between
number and size with the time to display.

That’s something that should have been wildly improved in RT 3.0.11
or
so. Certainly here, we saw page loads drop from 5+ minutes for long
tickets with multi-megabyte attachments to about 10 seconds for the
longest ticket history pages.

I can’t say that I ever hit a 5+ minute ticket even with RT2.

I’m seeing 20-30 second ticket loads. Most are within 4 or 5 seconds.
There are a lot of sql queries per ticket display page, that’s for
sure!

smime.p7s (2.42 KB)

22

I saw the thread with Jon Masters et. al. last month talking about
performance issues with RT 3.

I’ve got a brand-new installation of RT 3.2.1 running on Fedora
Core 1, with apache 2 & FastCGI, with a mysql backend. We have
fewer than 100 tickets in our database. In general all the tools

It might help for you to talk about

Sorry for the lack of details - i was hoping there were some fixes that
had been identified that I had missed in my perusal of the list archives.

how you’ve configured mysql
to make use of the system’s ram,

First, I don’t think it’s a database problem - see below re
mason_handler CPU time.

I haven’t done anything beyond red hat/fedora’s defaults.

mysql is only using about 10 megs of RAM, FWIW.

I am not running innodb.

mysql 27143 0.0 1.0 33496 10916 ? S Aug06 2:48 /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking

made sure that the system isn’t swapping,

It’s not. It has 1G of ram and does nothing beyond run processes
beyond what’s required for RT. It currently has 256M or so free, plus
256M or so used for file cache.

The machine has 7200rpm scsi disks and is running software RAID1.

that most of the CPU time seems to be in the database or in
perl,

definitely in perl. Mysql has used 3 minutes of CPU in about a month.
my 10 mason_handler fcgi processes have used 2.5-3 mins each in the 24
hours since they were started.

When i was running a single mason_handler fcgi process, I could watch
the system with top and see that mason_handler was eating 100% of one
of the CPUs while I waited (on the order of 5-10 seconds) for the ticket
to display in my browser.

FWIW fastCGI is running under apache’s userid.

what versions of things other than RT you’re running, etc.

(from fedora core 1)
mysql-3.23.58-4
httpd-2.0.49-1.1
perl-5.8.3-16
Linux branford.internet2.edu 2.4.22-1.2197.nptlsmp #1 SMP Thu Jul 1 15:05:23 EDT 2004 i686 i686 i386 GNU/Linux

(built from source)
mod_fastcgi-2.4.2

the long RT “make testdeps” output is at the end of this message.

I just realized that rt was configured to log to a non-existent file
where it did not have permissions to create one, so i don’t have any
log info for you.

I should add for the record that I have been working on Unix systems
since 1991.

Someone else suggested that I should try falling back to RT 3.0.11.
Based on the upgrade instructions it does not look like this is something
I can do with my existing database. Am I correct on this?

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

[root@branford rt-3.2.1]# make testdeps
/usr/bin/perl ./sbin/rt-test-dependencies --verbose --with-mysql
perl:
5.8.3…found
users:
rt group (rt)…found
bin owner (root)…found
libs owner (root)…found
libs group (bin)…found
web owner (apache)…found
web group (apache)…found
MASON dependencies:
Params::Validate 0.02…found
Cache::Cache …found
Exception::Class 1.14…found
HTML::Mason 1.23…found
MLDBM …found
Errno …found
FreezeThaw …found
Digest::MD5 2.27…found
CGI::Cookie 1.20…found
Storable 2.08…found
Apache::Session 1.53…found
XML::RSS …found
MAILGATE dependencies:
HTML::TreeBuilder …found
HTML::FormatText …found
Getopt::Long …found
LWP::UserAgent …found
CLI dependencies:
Getopt::Long 2.24…found
CORE dependencies:
Digest::base …found
Digest::MD5 2.27…found
DBI 1.37…found
Test::Inline …found
Class::ReturnValue 0.40…found
DBIx::SearchBuilder 1.01…found
Text::Template …found
File::Spec 0.8…found
HTML::Entities …found
HTML::Scrubber 0.08…found
Net::Domain …found
Log::Dispatch 2.0…found
Locale::Maketext 1.06…found
Locale::Maketext::Lexicon 0.32…found
Locale::Maketext::Fuzzy …found
MIME::Entity 5.108…found
Mail::Mailer 1.57…found
Net::SMTP …found
Text::Wrapper …found
Time::ParseDate …found
File::Temp …found
Term::ReadKey …found
Text::Autoformat …found
Text::Quoted 1.3…found
Tree::Simple 1.04…found
Scalar::Util …found
Module::Versions::Report …found
DEV dependencies:
Regexp::Common …found
Time::HiRes …found
Test::Inline …found
Apache::Test …MISSING
HTML::Form …found
HTML::TokeParser …found
WWW::Mechanize …found
MYSQL dependencies:
DBD::mysql 2.1018…found

23

mysql is only using about 10 megs of RAM, FWIW.

That’s probably not ideal. Have a look at the sample mysql
configurations and choose which one is appropriate for your setup,
most likely the large.

I am not running innodb.

That’s bad. You really should be running innodb for RT.

It’s not. It has 1G of ram and does nothing beyond run processes
beyond what’s required for RT. It currently has 256M or so free, plus
256M or so used for file cache.

That doesn’t mean its not swapping. You’ve forced mysql to use 10MB
and no more of RAM. If it needs more, it will swap once it reaches
it’s limit, regardless of physical memory.

Michael
Michael S. Liebman m-liebman@northwestern.edu
http://msl521.freeshell.org/
“I have vision and the rest of the world wears bifocals.”
-Paul Newman in “Butch Cassidy & the Sundance Kid”

24

I saw the thread with Jon Masters et. al. last month talking about
performance issues with RT 3.

I’ve got a brand-new installation of RT 3.2.1 running on Fedora
Core 1, with apache 2 & FastCGI, with a mysql backend. We have
fewer than 100 tickets in our database. In general all the tools

It might help for you to talk about

Sorry for the lack of details - i was hoping there were some fixes that
had been identified that I had missed in my perusal of the list archives.

how you’ve configured mysql
to make use of the system’s ram,

First, I don’t think it’s a database problem - see below re
mason_handler CPU time.

I haven’t done anything beyond red hat/fedora’s defaults.

mysql is only using about 10 megs of RAM, FWIW.

I am not running innodb.

Ok. That tells us that 1) you’re running mysql in the default
configuration, which is optimized for a pentium 133 with 64 megs of RAM.
That’s a bad plan. and 2) you missed the note that said that InnoDB is a
hard requirement. You should see massively degraded performance if
you’re running without it. And also, database corruption.

25

mysql is only using about 10 megs of RAM, FWIW.

That’s probably not ideal. Have a look at the sample mysql
configurations and choose which one is appropriate for your setup,
most likely the large.

Where? Is this something I should be looking for on the mysql
site? I couldn’t find it in the RT documentation, the RT source,
or the RT wiki.

I’m surprised I need a “large” configuration for a database that does
not yet have 100 tickets.

It’s not. It has 1G of ram and does nothing beyond run processes
beyond what’s required for RT. It currently has 256M or so free, plus
256M or so used for file cache.

That doesn’t mean its not swapping. You’ve forced mysql to use 10MB
and no more of RAM. If it needs more, it will swap once it reaches
it’s limit, regardless of physical memory.

It might be paging but it’s not swapping. watching with “free” running
in a 1-second loop shows zero swap usage. It has a 33M vsize (vs the 10M
ram usage) so you might have something here - but it should not really
be a performance problem on the level we are talking about if it is not
actually swapping. Since the system file cache is holding everything
in ram anyway, worst case it pages in the files that are already cached.

I’m hardly a database expert, though. I can imagine that mysql
might optimize for memory in doing its queries instead of optimizing
for speed unless told otherwise - but is this really potentially
taking 5-10 seconds to pull up one ticket?

This doesn’t explain why the system spends all this CPU time in
mason_handler.cgi though, and I frankly find it difficult to believe
that more than 10 megs of ram is needed for reasonable DB performance
with a DB of less than 100 tickets.

I will check into InnoDB.

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

26

Ok. That tells us that 1) you’re running mysql in the default
configuration, which is optimized for a pentium 133 with 64 megs of RAM.
That’s a bad plan.

Nice. But is this really the problem here with a db with 100 tickets?

and 2) you missed the note that said that InnoDB is a
hard requirement. You should see massively degraded performance if
you’re running without it. And also, database corruption.

Whoops. I must have modified the my.cnf after installing RT. I probably
did this to turn on skip-networking and copied the skip-innodb from
another server without remembering it was necessary.

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect. Things still take a long time, and mason_handler.fcgi
processes still eat the CPU (visible via top) while I’m waiting for the
web requests to be fulfilled.

thanks for all your help.

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

27

yes. It will make a difference for you. You’ll find example my.cnf
files in
/usr/local/src/mysql-standard-4.0.14-sun-solaris2.8-sparc/support-files/
(or wherever your source build is). Modify and place into /etc/ as
my.cnf and restart the whole shebang. Give it a shot.On Thu, 2004-09-09 at 12:14, Dan Pritts wrote:

On Thu, Sep 09, 2004 at 02:50:26PM -0400, Jesse Vincent wrote:

Ok. That tells us that 1) you’re running mysql in the default
configuration, which is optimized for a pentium 133 with 64 megs of RAM.
That’s a bad plan.

Nice. But is this really the problem here with a db with 100 tickets?

and 2) you missed the note that said that InnoDB is a
hard requirement. You should see massively degraded performance if
you’re running without it. And also, database corruption.

Whoops. I must have modified the my.cnf after installing RT. I probably
did this to turn on skip-networking and copied the skip-innodb from
another server without remembering it was necessary.

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect. Things still take a long time, and mason_handler.fcgi
processes still eat the CPU (visible via top) while I’m waiting for the
web requests to be fulfilled.

thanks for all your help.

danno

dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

The rt-users Archives

Be sure to check out the RT wiki at http://wiki.bestpractical.com
Rick Rezinas
Unix Systems Administrator
Qsent, Inc.

28

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect.

Sure, because by now your tables are in MyISAM instead of InnoDB.
You need to somehow convert them using mysqldump or other techniques.

Thanks,
/Autrijus/

29

That’s probably not ideal. Have a look at the sample mysql
configurations and choose which one is appropriate for your setup,
most likely the large.

Where? Is this something I should be looking for on the mysql
site? I couldn’t find it in the RT documentation, the RT source,
or the RT wiki.

It should have landed somewhere as part of the mysql install. For
fedora it should be in /usr/share/mysql. There should be some
examples ‘my-large.cnf’, ‘my-huge.cnf’, etc. ‘locate’ should find
them if they are somewhere else on your system.

I’m surprised I need a “large” configuration for a database that does
not yet have 100 tickets.

You tune more for the size of your RAM.

This doesn’t explain why the system spends all this CPU time in
mason_handler.cgi though, and I frankly find it difficult to believe
that more than 10 megs of ram is needed for reasonable DB performance
with a DB of less than 100 tickets.

Be sure you have let the system work a while before you start
timing. Each fastcgi process will need to parse and compile
the perl/mason code once the first time it hits it. Subsequent
passes should be faster. You need more than one fcgi but probably
way less than 10.

Les Mikesell
les@futuresource.com

30

It should have landed somewhere as part of the mysql install. For
fedora it should be in /usr/share/mysql. There should be some
examples ‘my-large.cnf’, ‘my-huge.cnf’, etc. ‘locate’ should find
them if they are somewhere else on your system.

yes, found it, thanks. Now i’m using 300 megs of ram for mysql.

But I apparently need to convert my existing tables into innodb format.

I’m surprised I need a “large” configuration for a database that does
not yet have 100 tickets.

You tune more for the size of your RAM.

…but I wouldn’t think i would need much ram for a database that has
100 tickets. Regardless, I’ve got the RAM and i’ve allocated it to mysql
at this point.

Be sure you have let the system work a while before you start
timing. Each fastcgi process will need to parse and compile
the perl/mason code once the first time it hits it. Subsequent
passes should be faster. You need more than one fcgi but probably
way less than 10.

Yes, I noticed when they started up they took maybe 10 CPU seconds each to
get going. But they are persistent, have been running for over a day now.
I am pretty confident this is not the issue.

I didn’t think i really needed that many fcgis but something i read
suggested i should have one for each httpd. At this point it is extremely
unlikely that we are hitting it with more than one or two concurrent
users. I figured i had plenty of RAM so no need to be stingy with them.

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

31

Be sure you have let the system work a while before you start
timing. Each fastcgi process will need to parse and compile
the perl/mason code once the first time it hits it.

Yes, I noticed when they started up they took maybe 10 CPU seconds each to
get going. But they are persistent, have been running for over a day now.
I am pretty confident this is not the issue.

I think they only compile a page when you view it so it is a matter of
having hit all the pages enough times that each fcgi has it compiled.
I have a similar setup with a few hundred tickets on fedora2 with
mysql 4.0.18 and it feels a little slower than an older RT2 setup with
many thousands of tickets, but it only takes a second or 2 to display
a ticket. I’ve turned off the option to log all the email responses -
I’m not sure if that makes a difference.

Les Mikesell
les@futuresource.com

32

I think they only compile a page when you view it so it is a matter of
having hit all the pages enough times that each fcgi has it compiled.

Hmm. Seems reasonable and would explain some of the variability i have
seen - sometimes it’s “slow” and other times it’s “painfully slow”.

slow meaning a couple seconds to update a page, painful in the 5-10
second range.

I have now lowered the number of fcgi’s to 4.

I have a similar setup with a few hundred tickets on fedora2 with
mysql 4.0.18 and it feels a little slower than an older RT2 setup with
many thousands of tickets, but it only takes a second or 2 to display
a ticket. I’ve turned off the option to log all the email responses -
I’m not sure if that makes a difference.

This seems like something valuable for us so i don’t want to disable it.

In other news, I’m now running mysqld with the mysql-large configuration
file, with innodb support enabled. After backing up the database,
I’ve done ALTER TABLE TYPE=INNODB on all the tables in the RT3 database.
can anyone tell me what I can do to confirm this change? The files all
now have .frm extensions.

As before, mysql doesn’t use an appreciable amount of CPU although it
now uses 300 megs of RAM instead of 10.

As before, access is slow or painfully slow (see above) and the
mason_handler processes are eating boatloads of CPU time.

What should I check now?

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

33

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect.

Sure, because by now your tables are in MyISAM instead of InnoDB.
You need to somehow convert them using mysqldump or other techniques.

You should just be able to ‘alter table type=innodb;’

Michael
Michael S. Liebman m-liebman@northwestern.edu
http://msl521.freeshell.org/
“I have vision and the rest of the world wears bifocals.”
-Paul Newman in “Butch Cassidy & the Sundance Kid”

34

As before, access is slow or painfully slow (see above) and the
mason_handler processes are eating boatloads of CPU time.

What should I check now?

You could try an ‘strace -p process_id_number’ on the busy process
to see if it is doing something obviously wrong.

Les Mikesell
les@futuresource.com

35

RT Community:

I’ve hesitated from weighing in on this for a number of reasons, mostly
because the “horrible load times of 5-10 seconds” don’t bother me all that
much, but probably moreso because RT is a free product that we have not
yet found the funds to purchase support for.

But with everyone now claiming they are getting 2s load times on Tickets,
I decided to start timing things with &Debug=1 and on large tickets see
averages around 11 seconds, with peaks as high as 15s. This is for a
Ticket with maybe 30 entries, all text except for a small (25KB) Excel
spreadsheet. An average Ticket in our system runs around 1s (just
created) to 3s (handful of entries).

Like Dan, I am no mysql expert, but have been using the “large system”
template since the beginning and am running InnoDB tables. I’m using
mod_perl1 (compiled as DSO) on Apache 1.3.31. Base system is Linux 2.4,
the hardware is an Intel P4 3.0GHz, 1.5GB DRAM.

We’ve got about 400 tickets, and mysql upon start takes up about 60MB
DRAM.

My holdup seems to be in httpd (CPU spikes hard when pulling up a
“largeish” Ticket), but as I’m running mod_perl I understand this is to be
expected.

Should I look at migrating to FastCGI?

-jdOn Thu, 9 Sep 2004, Les Mikesell wrote:

On Thu, 2004-09-09 at 15:41, Dan Pritts wrote:

As before, access is slow or painfully slow (see above) and the
mason_handler processes are eating boatloads of CPU time.

What should I check now?

You could try an ‘strace -p process_id_number’ on the busy process
to see if it is doing something obviously wrong.

Les Mikesell
les@futuresource.com

The rt-users Archives

Be sure to check out the RT wiki at http://wiki.bestpractical.com

36

You could try an ‘strace -p process_id_number’ on the busy process
to see if it is doing something obviously wrong.

duh. Sometimes i’m a bit dense.

So I’ve watched the mason_handler processes and I’m seeing lots
and lots of two things that probably don’t need to happen
as much as they do.

  1. a boatload of time() calls. In once case, ~4500 within 6 seconds
    (the time it took the process to bring up one ticket diary).

  2. a boatload of stat64 calls looking for various files.
    Over and over again, local directory doesn’t exist.
    In some cases, multiple stats in sequence for the same file.

Appended are examples.

I don’t know enough about perl to know how to profile the code to
see where all these time() syscalls are coming from. Suggestions
welcome.

Also, Matt Mankins suggested I update DBIX-SearchBuilder to the latest
version, which i have done (and restarted the mason_handlers) with no
apparent effect.

danno
dan pritts danno@internet2.edu
systems administrator 734/352-4953 office
internet2 734/834-7224 mobile

stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Callback”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Callback”, {st_mode=S_IFREG|0755, st_size=1906, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Callbacks”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Callbacks”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Tabs”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Tabs”, {st_mode=S_IFREG|0755, st_size=2619, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/CreateTicket”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/CreateTicket”, {st_mode=S_IFREG|0755, st_size=1070, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/SelectNewTicketQueue”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/SelectNewTicketQueue”, {st_mode=S_IFREG|0755, st_size=1756, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/l”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/l”, {st_mode=S_IFREG|0755, st_size=1039, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/SimpleSearch”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/SimpleSearch”, {st_mode=S_IFREG|0755, st_size=1067, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/l”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/l”, {st_mode=S_IFREG|0755, st_size=1039, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Callback”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Callback”, {st_mode=S_IFREG|0755, st_size=1906, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Callbacks”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Callbacks”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/PageLayout”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/PageLayout”, {st_mode=S_IFREG|0755, st_size=2934, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Menu”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Menu”, {st_mode=S_IFREG|0755, st_size=2610, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Menu”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Menu”, {st_mode=S_IFREG|0755, st_size=2610, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Menu”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Menu”, {st_mode=S_IFREG|0755, st_size=2610, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/Menu”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/Menu”, {st_mode=S_IFREG|0755, st_size=2610, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/ListActions”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/ListActions”, {st_mode=S_IFREG|0755, st_size=1329, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/TitleBoxStart”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/TitleBoxStart”, {st_mode=S_IFREG|0755, st_size=2073, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Ticket/Elements/ShowSummary”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Ticket/Elements/ShowSummary”, {st_mode=S_IFREG|0755, st_size=2785, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Elements/TitleBoxStart”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Elements/TitleBoxStart”, {st_mode=S_IFREG|0755, st_size=2073, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/Ticket/Elements/ShowBasics”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/Ticket/Elements/ShowBasics”, {st_mode=S_IFREG|0755, st_size=1887, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/l”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/l”, {st_mode=S_IFREG|0755, st_size=1039, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/l”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/l”, {st_mode=S_IFREG|0755, st_size=1039, …}) = 0
stat64(“/usr/local/pkg/rt-3.2.1/local/html/l”, 0x86200c8) = -1 ENOENT (No such file or directory)
stat64(“/usr/local/pkg/rt-3.2.1/share/html/l”, {st_mode=S_IFREG|0755, st_size=1039, …}) = 0

37

I quote from the “current” draft RT document (2003):
… recommend MySQL (http://www.mysql.com), because RT was primarily
developed on MySQL, so it is the best-tested database for RT. We
currently support version 4.0.9 or newer with support for InnoDB (some
older releases had crippling SQL bugs).

The link to that document should get removed. It’s all been migrated to
the Wiki. The README says explicitly that we only support mysql with
innodb. And RT 3.2.2 tests for this and refuses to run unless you
enable InnoDB.

-jesse
38

Ok. That tells us that 1) you’re running mysql in the default
configuration, which is optimized for a pentium 133 with 64 megs of
RAM.
That’s a bad plan.

Nice. But is this really the problem here with a db with 100 tickets?

The ‘large’ is about the system resources and database complexity, not
necessarily the db size. MySQL does file based operations if the
memory limits configured have been reached. This * will * slow things
down.

and 2) you missed the note that said that InnoDB is a
hard requirement. You should see massively degraded performance if
you’re running without it. And also, database corruption.

Whoops. I must have modified the my.cnf after installing RT. I
probably
did this to turn on skip-networking and copied the skip-innodb from
another server without remembering it was necessary.

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect. Things still take a long time, and
mason_handler.fcgi
processes still eat the CPU (visible via top) while I’m waiting for the
web requests to be fulfilled.

You have to change the tables, too. They are in MyISAM format now. This
is done in MySQL with “ALTER TABLE whatevertable TYPE=innodb;”.
Turning on innodb support in the engine does not automatically convert
Your database.

Regards,
Harald

39

Hello

slow meaning a couple seconds to update a page, painful in the 5-10
second range.
My problem, too. I read the thread and just wanted to confirm that all
suggestions made no improvement on my system, too.

I’d like to note that the display process is not only slow when printing
the ticket data itself but also when just displaying the top of the
frames or other even the new ticket window…

The record:
Problem: 4-5s for /rt/Ticket/Display.html?id=1
with only 1 tiny Ticket in the Database
Versions: 3.0.11 and 3.2.1
Handler: Same results with mod_perl, fcgi (2 processes) and speedy
(with mod_perl and Galeon I see a big load in the browser, too)
Hardware: celeron600,768MB RAM and Athlon2600,1GB RAM
Disk Activity: no buffer or swap activity reported by vmstat 1
CPU Activity: 99% during several seconds in top/vmstat, in mason_handler,
speedy_backend or apache - depending on handler
Base Systems: Debian 3.1 “sarge” and 3.x “sid” (I’m a @debian.org)
MySQL Setup: v4.0.20 for large system, using InnoDB and Query-Cache
Browsers tested: Galeon&Mozilla @ Linux, Internet Explorer @ WinXP

Also here some “ps faxu” lines telling which process uses most CPU
power, all those processes were started in the last 10 minutes!

Sep09 0:36 /usr/sbin/apache
11:00 0:00 _ /usr/sbin/fcgi-p
11:00 0:40 | _ /usr/bin/perl /usr/share/request-tracker3.2/libexec/mason_handler.fcgi

11:00 1:03 | _ /usr/bin/perl /usr/share/request-tracker3.2/libexec/mason_handler.fcgi
11:00 0:12 _ /usr/sbin/apache
11:00 0:01 _ /usr/sbin/apache
11:01 0:01 _ /usr/sbin/apache
11:01 0:24 _ /usr/sbin/apache
11:01 0:12 _ /usr/sbin/apache
11:18 0:12 _ /usr/sbin/apache
10:59 0:00 /bin/sh /usr/bin/mysqld_safe
10:59 0:00 _ /usr/sbin/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.
10:59 0:00 _ logger -p daemon.err -t mysqld_safe -i -t mysqld
11:01 0:00 /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:01 0:53 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:01 0:05 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:17 0:04 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:17 0:04 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:18 0:05 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi
11:25 0:04 _ /usr/bin/speedy_backend /usr/share/request-tracker3.2/libexec/mason_handler.scgi

Some nice infos from the MySQL Database showing a working Query Cache:
| Key_blocks_used | 24 |
| Key_read_requests | 235 |
| Key_reads | 8 |
| Key_write_requests | 149 |
| Key_writes | 53 |

| Qcache_queries_in_cache | 88 |
| Qcache_inserts | 212 |
| Qcache_hits | 1646 |
| Qcache_lowmem_prunes | 0 |
| Qcache_not_cached | 138 |
| Qcache_free_memory | 26081032 |
| Qcache_free_blocks | 9 |
| Qcache_total_blocks | 206 |

My strace-attached-to-process also show a lot of stats for Perl modules
which I find rather strange as I thought that most of the handlers cache
the modules so that apache must even be reloaded when updating them?
(strace output was sent by someone else, no big differences)

bye,

-christian-

Christian Hammers WESTEND GmbH | Internet-Business-Provider
Technik CISCO Systems Partner - Authorized Reseller
Lï¿œtticher Straï¿œe 10 Tel 0241/701333-11
ch@westend.com D-52064 Aachen Fax 0241/911879

40

Am 09.09.2004 um 21:14 schrieb Dan Pritts:

Ok. That tells us that 1) you’re running mysql in the default
configuration, which is optimized for a pentium 133 with 64 megs of
RAM.
That’s a bad plan.

Nice. But is this really the problem here with a db with 100 tickets?

The ‘large’ is about the system resources and database complexity, not
necessarily the db size. MySQL does file based operations if the
memory limits configured have been reached. This * will * slow things
down.

and 2) you missed the note that said that InnoDB is a
hard requirement. You should see massively degraded performance if
you’re running without it. And also, database corruption.

Whoops. I must have modified the my.cnf after installing RT. I
probably
did this to turn on skip-networking and copied the skip-innodb from
another server without remembering it was necessary.

However, i’ve turned it on, and restarted mysql, and there’s no
apparent effect. Things still take a long time, and
mason_handler.fcgi
processes still eat the CPU (visible via top) while I’m waiting for the
web requests to be fulfilled.

You have to change the tables, too. They are in MyISAM format now. This
is done in MySQL with “ALTER TABLE whatevertable TYPE=innodb;”.
Turning on innodb support in the engine does not automatically convert
Your database.

OK, so I didn’t waste a minute thinking about this (I’d have liked to
use postgresql (a real relational database…), but it doesn’t work in a
FreeBSD-jail).

So, I took a look:
(sorry for that - I couldn’t get show table status to only return some
columns - is that possible?)

mysql> show table status ;
| Name | Type | Row_format | Rows | Avg_row_length
| Data_length | Max_data_length | Index_length | Data_free |
Auto_increment | Create_time | Update_time | Check_time
| Create_options | Comment |
| ACL | InnoDB | Dynamic | 17 | 963
| 16384 | NULL | 16384 | 0 |
17 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Attachments | InnoDB | Dynamic | 41 | 1598
| 65536 | NULL | 32768 | 0 |
46 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Attributes | InnoDB | Dynamic | 1 | 16384
| 16384 | NULL | 32768 | 0
| 1 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| CachedGroupMembers | InnoDB | Fixed | 117 | 140
| 16384 | NULL | 16384 | 0 |
131 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| CustomFieldValues | InnoDB | Dynamic | 1 | 16384
| 16384 | NULL | 16384 | 0
| 1 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| CustomFields | InnoDB | Dynamic | 1 | 16384
| 16384 | NULL | 16384 | 0
| 1 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| GroupMembers | InnoDB | Fixed | 39 | 420
| 16384 | NULL | 16384 | 0 |
46 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Groups | InnoDB | Dynamic | 50 | 327
| 16384 | NULL | 32768 | 0 |
59 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Links | InnoDB | Dynamic | 1 | 16384
| 16384 | NULL | 65536 | 0
| 1 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| Principals | InnoDB | Dynamic | 59 | 277
| 16384 | NULL | 16384 | 0 |
59 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Queues | InnoDB | Dynamic | 4 | 4096
| 16384 | NULL | 32768 | 0
| 4 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| ScripActions | InnoDB | Dynamic | 18 | 910
| 16384 | NULL | 0 | 0 |
18 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| ScripConditions | InnoDB | Dynamic | 11 | 1489
| 16384 | NULL | 0 | 0 |
11 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Scrips | InnoDB | Dynamic | 15 | 1092
| 16384 | NULL | 0 | 0 |
15 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Templates | InnoDB | Dynamic | 13 | 1260
| 16384 | NULL | 0 | 0 |
13 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| TicketCustomFieldValues | InnoDB | Dynamic | 1 | 16384
| 16384 | NULL | 16384 | 0
| 1 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| Tickets | InnoDB | Dynamic | 6 | 2730
| 16384 | NULL | 49152 | 0
| 6 | NULL | NULL |
NULL | | InnoDB free: 11264 kB |
| Transactions | InnoDB | Dynamic | 58 | 282
| 16384 | NULL | 32768 | 0 |
58 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| Users | InnoDB | Dynamic | 10 | 1638
| 16384 | NULL | 32768 | 0 |
50 | NULL | NULL | NULL
| | InnoDB free: 11264 kB |
| sessions | MyISAM | Dynamic | 24 | 1112
| 26700 | 4294967295 | 2048 | 0 |
NULL | 2004-07-20 14:09:57 | 2004-09-06 11:44:58 | NULL
| | |
20 rows in set (0.04 sec)

Everything innodb, except sessions - is that normal ?
Does it matter ?

This is rt3.2.1 with mysql-4.20

Another thing: I’m wondering, where the hell one should add the
information about mysql - it’s currently distributed all over the wiki.
Or does it matter ? Should users use the search-function anyway ?

Rainer
~ Rainer Duffner - rainer@ultra-secure.de ~
~ Freising - Munich - Germany ~
~ Unix - Linux - BSD - OpenSource - Security ~
~ http://www.ultra-secure.de/~rainer/pubkey.pgp ~