Rt3.x perfomance degradation (~500%)

Thanks for good suggestions.
After upgrading to mySQL 4.x and using inodb same tests go 1.3 times
slower against rt2 on 3.23.x
After enabling query_cache perfomance go up to 10x faster then rt2 on 3.23.x
Last improvement only for freaquently requests and don’t help to solve
problem with new tickets and attachments.

How do you think Jesse could be code upgraded to improve perfomance
signicantly? It’s very important task for me. May you can give me any
additional suggestions.

Audit code and database and see what queries are killing performance and
whether the issue is in the database and perl. Then, either add db
indices, tune the queries being generated by the perl or alter the
database’s internal tuning parameters.

               Ruslan.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Hello.
Have done snapshot of our RT DB.
Have setup test machine with RT2 and RT3(DB same db generated with
ugrade tools v1.16).
Start apache with rt2 and make ab test on index.html with cookie
sending. Repeat same twice. Use last result(cause of mason caching).
Same with rt3.
Every thing has same setup with almost default configs.
linux 2.4.21+apache 1.3.28+mod_perl 1.28+mySQL 3.23.x+perl 5.8.0+all
latest needed modules from CPAN+RT 3.0.4 <=> RT 2.0.15
!!!Diff at the end and it about 500% overall time DEGRADATION!!!

Questions:
1) Why it is happenings? What to do with it? How to speed up RT3 to
RT2 level?
2) Tried to use Mason cache enlarge to 100MB from default and add
static_code flag through httpd.conf, but no significant changes. May be
it’s wrong to do it in httpd.conf? May be in code? I didn’t find how to
look at Mason runtime config(some pointd on it?).
3) Somebody did or can do mySQL upgrade(3.x → 4.x) perfomance test?
4) Interesting aspect is mySQL vs Postgres?
4) Sometimes while first preparation tests I’s starting top and
almost all the time mySQL used ~100% CPU. May be I could use some
aditionals mySQL options to speed it up for RT3 only.

Latest VHost config:

RT3

Try
Listen xxxx

ServerName xxxx
DocumentRoot /www/rt3/share/html
AddDefaultCharset UTF-8

PerlModule Apache::DBI
PerlRequire /www/rt3/bin/webmux.pl

this section applies to Apache 1 only

<Location />
    SetHandler perl-script
    PerlHandler RT::Mason
</Location>
PerlSetVar MasonCodeCacheMaxSize 100000000 PerlSetVar MasonStaticSource 1

Diff to tests:
RT3 -./ab -n 100 -c 10 -C ‘RT_SID=xxxx’ -v 1 http://xxxx/index.html
RT2 + ./ab -n 100 -c 10 -C ‘AF_SID=xxxx’ -v 1 http://xxxx/index.html
This is ApacheBench, Version 1.3d <$Revision: 1.69 $> apache-1.3
Server Software: Apache/1.3.28
Document Path: /index.html
-Document Length: 11042 bytes
+Document Length: 9830 bytes

Concurrency Level: 10
-Time taken for tests: 194.310 seconds
+Time taken for tests: 35.543 seconds
Complete requests: 100
Failed requests: 0
Broken pipe errors: 0
-Total transferred: 1124400 bytes
-HTML transferred: 1104200 bytes
-Requests per second: 0.51 [#/sec] (mean)
-Time per request: 19431.00 [ms] (mean)
-Time per request: 1943.10 [ms] (mean, across all concurrent requests)
-Transfer rate: 5.79 [Kbytes/sec] received
+Total transferred: 1007374 bytes
+HTML transferred: 992830 bytes
+Requests per second: 2.81 [#/sec] (mean)
+Time per request: 3554.30 [ms] (mean)
+Time per request: 355.43 [ms] (mean, across all concurrent requests)
+Transfer rate: 28.34 [Kbytes/sec] received

Connnection Times (ms)
min mean[+/-sd] median max
-Connect: 0 2349 4294.9 0 17614
-Processing: 5374 12333 5128.7 13179 47345
-Waiting: 0 11760 5557.5 13140 47332
-Total: 5374 14682 4668.0 13691 49184
+Connect: 0 312 284.8 336 776
+Processing: 675 773 416.2 686 2421
+Waiting: 0 458 485.2 341 2421
+Total: 996 1085 237.3 1025 2421

Percentage of the requests served within a certain time (ms)

  • 50% 13691
  • 66% 14505
  • 75% 14882
  • 80% 16853
  • 90% 19636
  • 95% 19636
  • 98% 21758
  • 99% 23790
  • 100% 49184 (last request)
  • 50% 1025
  • 66% 1035
  • 75% 1047
  • 80% 1064
  • 90% 1120
  • 95% 1313
  • 98% 2004
  • 99% 2421
  • 100% 2421 (last request)
   Hello.

Have done snapshot of our RT DB.
Have setup test machine with RT2 and RT3(DB same db generated with
ugrade tools v1.16).
Start apache with rt2 and make ab test on index.html with cookie
sending. Repeat same twice. Use last result(cause of mason caching).
Same with rt3.
Every thing has same setup with almost default configs.
linux 2.4.21+apache 1.3.28+mod_perl 1.28+mySQL 3.23.x+perl 5.8.0+all
latest needed modules from CPAN+RT 3.0.4 <=> RT 2.0.15
!!!Diff at the end and it about 500% overall time DEGRADATION!!!

You need to move to mysql 4.0.13 or newer. This will make a
HUGE difference in performance. I’m betting that you don’t actually
even have innodb tables enabled in your mysql 3.23. Both of those
things are killers.

Questions:

  1. Why it is happenings? What to do with it? How to speed up RT3 to
    RT2 level?
  2. Tried to use Mason cache enlarge to 100MB from default and add
    static_code flag through httpd.conf, but no significant changes. May be
    it’s wrong to do it in httpd.conf? May be in code? I didn’t find how to
    look at Mason runtime config(some pointd on it?).
  3. Somebody did or can do mySQL upgrade(3.x → 4.x) perfomance test?
  4. Interesting aspect is mySQL vs Postgres?
  5. Sometimes while first preparation tests I’s starting top and
    almost all the time mySQL used ~100% CPU. May be I could use some
    aditionals mySQL options to speed it up for RT3 only.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.