For quite some time we have been running RT3 to track support issues,
following my recommendation of the software. Although various
configurations have been used, I am currently debugging an instance
running on a Debian testing box with the specification below because the
faster production machine (running Mandrake release 9) cannot be
tinkered with for the moment and performace problems plague both.
Performance is…well horrible. Painfully unpleasant and this is not
with the 40,000 tickets quoted in RT documentation. This is with a
database not exceeding about 50-100MB in size and with only a handfull
of tickets in it. In other words there are almost no tickets.
I have tried various different configurations based around RT3 (various
versions including but not limited to 3.2.0), MySQL 4, and Apache 1 and
2 running every possible configuration of mod_perl 1,2 and FastCGI.
Essentially I have tried every possible choice available.
Query time is so painful that the people here are having to resort to
using regular email and just have RT keep a copy for auditing. I do
unfortunately not have time to delve much in to the bowls of RT (I do
embedded Linux kernel hacking and am not usually a perl person) but have
installed various different profiling options in Apache to at least give
a feel for where this is falling over. I have debug traces of apache and
various other output here and am willing to provide various additional
information and test solutions if I can help to correct this issue.
Clearly the testing machine is not the fastest in the world but it is an
order of magnitude too slow and there is something more fundamental at
fault here. It does not look like MySQL is at fault - it looks like it
is SearchBuilder spewing out way too many searches or something. As I
have read on your mailing list archives and on other online lists,
numerous people report problems with attachments - some of these tickets
have large numbers of attachments but not all, and even then even
listing the queue takes nearly all day.
I am not trying to rant here but there is clearly something very weird
going on which I would like to help you guys to fix. You never know, it
might even be possible to convince me that perl isn’t such a slow
scripting language after all.
— Specification —
Linux kernel 2.6.6 (Not stock, Debian kernel release 2.6.6-1-686)
running on an Intel Celeron 600MHz CPU. This test box is not the fastest
machine in the world but it runs nothing else other than RT and a few
general system daemons. No windowing system cruft. System has been tuned
using sysctls and ulimits adjusted all in vain, IDE DMA is on, etc. etc.
The production machine exhibits the same problems and is much faster.
(So please assume that the base Debian is configured correctly).
To the system has been added MySQL version 4.0.18 (Debian 4.0.18-5).
I have also installed RT 3.2.0 as well as other versions.
I have performed database OPTIMIZE, ANALYZE, etc. etc.
etc. etc. etc. etc.
— dprofpp output —
Storable::nfreeze has 1 unstacked calls in outer
Storable::_freeze has 1 unstacked calls in outer
Exporter::import has -2 unstacked calls in outer
RT::Mason::handler has -1 unstacked calls in outer
CGI::delete has 1 unstacked calls in outer
AutoLoader::AUTOLOAD has -3 unstacked calls in outer
Storable::thaw has 1 unstacked calls in outer
Exporter::Heavy::heavy_export has 2 unstacked calls in outer
Total Elapsed Time = 196.2333 Seconds
User+System Time = 72.21333 Seconds
%Time ExclSec CumulS #Calls sec/call Csec/c Name
40.4 29.24 29.418 4647 0.0063 0.0063
9.20 6.641 71.487 2297 0.0029 0.0311 HTML::Mason::Request::comp
8.09 5.844 5.844 10197 0.0006 0.0006 HTML::Mason::Request::print
6.79 4.902 4.974 52 0.0943 0.0957
6.58 4.753 4.805 41 0.1159 0.1172 RT::Attachment::ContentLength
4.35 3.139 3.178 25 0.1256 0.1271 RT::Attachment::Content
3.52 2.539 2.639 25 0.1016 0.1055 Text::Quoted::extract
2.28 1.644 1.658 41 0.0401 0.0404 RT::Attachment::Headers
1.65 1.195 1.211 53 0.0226 0.0228 RT::Transactions::Next
1.64 1.186 1.182 52 0.0228 0.0227 RT::Record::CreatedAsString
1.52 1.095 1.119 25 0.0438 0.0448 RT::Transaction::IsInbound
1.37 0.986 1.011 42 0.0235 0.0241 RT::Attachments::Next
0.97 0.697 0.695 50 0.0139 0.0139 RT::Transaction::TicketObj
0.87 0.628 0.635 52 0.0121 0.0122 RT::Record::CreatorObj
0.51 0.369 0.367 20 0.0184 0.0184