Performance issues with rt3 - old topic new again

RT Users
1

I have read through pages and pages of list archives concerning performance
tuning for rt3. Yes, this issue has been covered in depth, but I’m still going
to bring the subject up anyway because it’s Christmas and you have to be
charitable. :slight_smile:

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task of
running rt3. (Well, at this point rt3 is just a test install.) I am running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

I am not satisfied with the response times from rt3. Here are some numbers
pulled from my testing:

(Format is: mo day hr min sec year response-time-in-seconds)

Login:

12 13 22 54 31 2003 5.3498
12 13 23 00 09 2003 5.4111
12 13 23 08 41 2003 5.4133
…

Open a typical queue–this one has 30 new/open tickets:

12 13 22 54 38 2003 5.1363
12 13 23 00 16 2003 5.2176
12 13 23 08 48 2003 4.8305
…

Open a ticket with about 15 comments/corresp. in it:

12 13 22 54 57 2003 16.6257
12 13 23 00 35 2003 16.7649
12 13 23 09 06 2003 16.6477
…

16 seconds! That seems excessive. Also, note that the script is pulling the
same ticket each time–the first may be slow, but after that it should be
faster, but it’s not. (That is, if the problem was with MySQL.)

Here are the various optimizations that I have tried:

  1. Use my-large.cnf as the MySQL my.cnf configuration.
  2. Try my-medium.cnf.
  3. Find a happy medium between those two values.
  4. Use modperl.
  5. Use FastCGI.
  6. Use MySQL 3.23
  7. Use MySQL 4.0
  8. Use MySQL 4.0, modperl, my-large.cnf.
  9. Other combinations.

None of these changes affected the response time for opening an existing
ticket. The average always stayed around 16.5 seconds.

Looking at the processor utilization, it would appear that httpd is consuming
the CPU the most. However, it never maxes out. Rather, it will usually
steadily reach 50% during my testing (which simulates one user accessing the
system at a time).

I have also tried rt3 on an Alpha 600Mhz with 512MB of RAM. I get the same
response times. (Well, the average for the Alpha for opening an existing
ticket is actually a bit longer.)

I would say I need a faster CPU, but then shouldn’t httpd/modperl/rt3 be
hitting a higher percentage of CPU usage than 50%?

I am not hitting swap at all.

(The following info is grabbed from the Alpha, which is up right now, but the
configuration is the same for the dual PII.)

Here is my current /etc/my.cnf:

[client]
port = 3306
socket = /tmp/mysql.sock

[mysqld]
port = 3306
socket = /tmp/mysql.sock
skip-locking
set-variable = key_buffer=256M
set-variable = max_allowed_packet=1M
set-variable = table_cache=256
set-variable = sort_buffer=1M
set-variable = record_buffer=1M
set-variable = myisam_sort_buffer_size=64M
set-variable = thread_cache=8

set-variable = thread_concurrency=8
#log-bin
#server-id = 1

[mysqldump]
quick
set-variable = max_allowed_packet=16M

[mysql]
no-auto-rehash

[isamchk]
set-variable = key_buffer=128M
set-variable = sort_buffer=128M
set-variable = read_buffer=2M
set-variable = write_buffer=2M

[myisamchk]
set-variable = key_buffer=128M
set-variable = sort_buffer=128M
set-variable = read_buffer=2M
set-variable = write_buffer=2M

[mysqlhotcopy]
interactive-timeout

Important processes:

last pid: 5347; load averages: 0.05, 0.13, 0.09
up 0+09:05:50 03:41:36
30 processes: 1 running, 28 sleeping, 1 stopped
CPU states: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 100% idle
Mem: 154M Active, 190M Inact, 47M Wired, 408K Cache, 61M Buf, 72M Free
Swap: 1020M Total, 1020M Free

PID USERNAME PRI NICE SIZE RES STATE TIME WCPU CPU COMMAND
493 www 4 0 47736K 40856K accept 59:38 0.00% 0.00% httpd
489 www 4 0 47752K 40840K accept 59:01 0.00% 0.00% httpd
490 www 4 0 47720K 40840K accept 58:35 0.00% 0.00% httpd
491 www 4 0 48168K 41312K accept 12:33 0.00% 0.00% httpd
1846 www 4 0 47992K 41136K accept 12:05 0.00% 0.00% httpd
492 www 4 0 48224K 41368K accept 11:55 0.00% 0.00% httpd
429 root 96 0 38896K 34848K select 0:20 0.00% 0.00% httpd
5343 mysql 96 0 264M 9392K select 0:00 0.00% 0.00% mysqld

Some packages:

p5-HTML-Mason-1.24
mysql-server-3.23.58
apache-1.3.29_1
mod_perl-1.28

2

Hi Seph. In my posting you will note that I did indeed list MySQL 4.0 as
something that I have already tried. Please review the message again if you
have a chance and can offer any other advice.

Thanks for the tip!From: “seph” seph@directionless.org
To: “Dustin Puryear” dpuryear@usa.net
Sent: Sunday, December 14, 2003 11:45 AM
Subject: Re: Performance issues with rt3 - old topic new again…

I haven’t stared to hard at the performance threads, so I’m far from
authoratative, but…

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task
of
running rt3. (Well, at this point rt3 is just a test install.) I am
running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

erm, I think mysql v4 is really recommended over v3. I forget if
that’s performance, or something else, but I’d change that first.

seph

3

Hi Seph. In my posting you will note that I did indeed list MySQL 4.0 as
something that I have already tried. Please review the message again if you
have a chance and can offer any other advice.

Thanks for the tip!From: “seph” seph@directionless.org
To: “Dustin Puryear” dpuryear@usa.net
Sent: Sunday, December 14, 2003 11:45 AM
Subject: Re: Performance issues with rt3 - old topic new again…

I haven’t stared to hard at the performance threads, so I’m far from
authoratative, but…

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task
of
running rt3. (Well, at this point rt3 is just a test install.) I am
running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

erm, I think mysql v4 is really recommended over v3. I forget if
that’s performance, or something else, but I’d change that first.

seph

4

I’m far from an expert here, but I’d guess that Seph is right,
you shouldn’t bother troubleshooting this with anything less than
mysql 4.x, even if that doesn’t fix the problem itself.

This seems like a mysql issue to me, not based on any particular
evidence though. I disagree that subsequent fetches should
be faster necessarily, because: a) with mod_perl, you may hit
a different apache child each request, and that child may
not have anything cached, and b) The ticket info could
change between hits, due to incoming mail or some other user,
so that RT may need to re-fetch each time to be sure.

I know that mysql can use several different backends, e.g.
isam, heap, myisam, innodb, bdb, all on a per-table basis.
This, and certainly indices, can affect you greatly.
Is it possible that in moving the data between mysql
versions, you have somehow dropped some indices?

And just to play devil’s advocate, are there any non-RT
network or apache issues? That is, I assume you can
display a static html page in much less than 5 secs?

bobg
5

It would appear that this is strictly a CPU issue. I moved from a dual PII
333Mhz to a single Athlon 800 with 512MB of RAM and my new numbers are:

(format is: mo day hour min sec year response-time-in-seconds)

index page:
12 14 13 14 50 2003 0.4459
12 14 13 15 40 2003 0.1779
12 14 13 16 01 2003 0.1798
12 14 13 17 09 2003 0.1717

logon page:
12 14 13 14 55 2003 2.3823
12 14 13 15 44 2003 2.4078
12 14 13 16 05 2003 2.1581
12 14 13 17 13 2003 2.2205

open queue:
12 14 13 14 59 2003 2.4465
12 14 13 15 48 2003 2.1757
12 14 13 16 09 2003 2.2386
12 14 13 17 17 2003 2.1054

open existing ticket:
12 14 13 15 08 2003 6.8935
12 14 13 15 57 2003 6.6339
12 14 13 16 18 2003 6.9085
12 14 13 17 26 2003 6.6263

So I went from 16-17 seconds when opening an existing ticket with either a
dual PII 333Mhz or Alpha 600Mhz to 6-7 seconds on an Athlon 800. If I had a
really fast CPU I might be able to get this down to a second or less. What
kinds of numbers are others seeing for even higher speed processors? If I
can click a ticket and get it open in a second or less then I would be
happy.

Are there any options for getting rt3 to run faster on the dual CPU PII I
wonder? I still think that 16-17 seconds is a bit slow, even for that
machine.

(Just for fun, I tried this on a Pentium 200 with 192MB of RAM. It takes
about 90 seconds to open this ticket. Faster than I would have thought…)

6

I have read through pages and pages of list archives concerning performance
tuning for rt3. Yes, this issue has been covered in depth, but I’m still going
to bring the subject up anyway because it’s Christmas and you have to be
charitable. :slight_smile:

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task of
running rt3. (Well, at this point rt3 is just a test install.) I am running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

mod_perl should be compiled static, at least with FreeBSD and Apache 1.3.

Matt Simerson had some info about that:

But most logic and “intelligence” about this procedure has migrated into his PERL-module.

And you should run mysql-4.0.x, if only for the fact that it’s recommended by BestPractical.

Has anyone tried the new postgresql7.4 ?
I’ve heard of dramatic speed increases for big queries.

cheers,
Rainer

7

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the task of
running rt3. (Well, at this point rt3 is just a test install.) I am running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23. 17
queues, 2 users with rights, less than 500 tickets.

Sounds like a similar experience I had (I was the original author of
this thread). Everyone said memory was an issue, but I think it was
CPU. I had a 25-30 second wait, but I only had one P2 333 CPU. Once
going to 677MHz (or thereabouts) with 512 MB of RAM, wait time went to
practically zero.

Just my humble opinion (which seems to be always wrong nowadays).

John

8

I also have one machine, which is a dual P2 499, and one single P3 933
The slower machine works with more users and so on, so it is clear that it’s slower.
Anywhay an old machine has also slower ram, slower bus to access the harddisk than the new one, so a single 933 is twice times faster than a dual 499, when not more in my opinion.
In my case the process that takes most is postgres, not apache.

Samuel-----Original Message-----
From: John Schubert [mailto:jschubert@linearcorp.com]
Sent: Monday,15 December,2003 08:35
To: Rainer Duffner
Cc: rt-users@lists.fsck.com
Subject: Re: [rt-users] Performance issues with rt3 - old topic new again…

On Sun, 2003-12-14 at 20:02, Rainer Duffner wrote:

I have a dual PII 333Mhz system with 512MB of RAM dedicated to the
task of running rt3. (Well, at this point rt3 is just a test
install.) I am running
rt3 with FreeBSD 5.1 with SMP, Apache 1.3, modperl1, and MySQL 3.23.
17 queues, 2 users with rights, less than 500 tickets.

Sounds like a similar experience I had (I was the original author of this thread). Everyone said memory was an issue, but I think it was CPU. I had a 25-30 second wait, but I only had one P2 333 CPU. Once going to 677MHz (or thereabouts) with 512 MB of RAM, wait time went to practically zero.

Just my humble opinion (which seems to be always wrong nowadays).

John

rt-users mailing list
rt-users@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

9

John Schubert jschubert@linearcorp.com said:

Sounds like a similar experience I had (I was the original author of
this thread). Everyone said memory was an issue, but I think it was
CPU. I had a 25-30 second wait, but I only had one P2 333 CPU. Once
going to 677MHz (or thereabouts) with 512 MB of RAM, wait time went to
practically zero.

Just my humble opinion (which seems to be always wrong nowadays).

Well, I think it is. The CPU you dropped in is twice as fast, so you would
expect waits to go to 12-15 secs. You experienced an order of magnitude
speed-up (at least), which the CPU upgrade couldn’t have given you.
However, you doubled RAM as well (in violation of the ceteris paribus
rule ;-)), so that must have been the case.

In general under Linux, if you have 100$ to spend and you have to
choose CPU or RAM, spend it on RAM. For example, I just ordered a new
mobo+CPU+memory setup, and stepped down with the CPU speed a bit to make
sure I could afford 1G of PC400 RAM.

Cees de Groot http://www.tric.nl cg@tric.nl
tric, the new way helpdesk/ticketing software, VoIP/CTI,
web applications, custom development