RT 3.0 Speed

RT Users
1

Well, after reviewing RT 3.0 extensively, our director is
hesitant because the homepage is slow. So before I say anything,
here is the setup:

Dual P450
512 mb ram
RT 3.0.1
database: Mysql 4.0.12
SearchBuilder 0.80
Mason 1.16
CGI.pm 1.75
apache 1.3.27
perl 5.6.1

And my my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
set-variable = max_allowed_packet=1M
set-variable = table_cache=192
set-variable = key_buffer_size=192M
set-variable = sort_buffer=1M
set-variable = myisam_sort_buffer_size=48M
set-variable = thread_cache=8

set-variable = thread_concurrency=4
log-bin
server-id = 1

innodb_data_home_dir = /var/lib/mysql/innodb
innodb_data_file_path=/ibdata:300M:autoextend
innodb_log_group_home_dir = /var/lib/mysql/innodb/iblogs/
innodb_log_arch_dir = /var/lib/mysql/innodb/iblogs/
set-variable = innodb_log_files_in_group=3
set-variable = innodb_buffer_pool_size=256M
set-variable = innodb_additional_mem_pool_size=20M
set-variable = innodb_log_file_size=57M
set-variable = innodb_log_buffer_size=2M
innodb_flush_log_at_trx_commit=1
set-variable = innodb_lock_wait_timeout=50

One of the current problems is the home page taking so long to load.
We have several managers who just pop up to check the QuickSearch
to see the status of the tickets in every queue and using it would
be painful. This is actually preventing us from migrating fully
to 3.0.

Does anyone see any optimizations to speed up our current setup?
That page is taking on average ~10 seconds to load while the RT 2.0
version takes ~5 seconds.

If no one see anything obvious, then might I suggest having the home
page cached so that displaying the home page doesnt require all
those database hits. We actually wouldn’t mind that the data is not
real time and it’s off by a min or two.

Dean

2

You will see a big benefit from paying $400 for a motherboard/cpu/ram upgrade.
On a single CPU p4/2.4GHz, with 512MB of DDRAM you will have about 3x your
memory bandwidth, and 5x your CPU speed. This will make a BIG difference.
Your current SMP system is a shared bus architecture, and that shared RAM
bus is slow compared to the memory controller on Intel DDRAM motherboards
(i845 and above).

If you want specific recommendations for a particular mobo/cpu/ram email me
directly and I’ll be glad to spec it for you.

Message: 14
Date: Tue, 22 Apr 2003 15:30:07 -0700
From: Dean Kao nouveaux@lightconsulting.com
To: rt-users@lists.fsck.com
Subject: [rt-users] RT 3.0 Speed

Well, after reviewing RT 3.0 extensively, our director is
hesitant because the homepage is slow. So before I say anything,
here is the setup:

Dual P450
512 mb ram
RT 3.0.1
database: Mysql 4.0.12
SearchBuilder 0.80
Mason 1.16
CGI.pm 1.75
apache 1.3.27
perl 5.6.1

And my my.cnf:

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
set-variable = max_allowed_packet=1M
set-variable = table_cache=192
set-variable = key_buffer_size=192M
set-variable = sort_buffer=1M
set-variable = myisam_sort_buffer_size=48M
set-variable = thread_cache=8

set-variable = thread_concurrency=4
log-bin
server-id = 1

innodb_data_home_dir = /var/lib/mysql/innodb
innodb_data_file_path=/ibdata:300M:autoextend
innodb_log_group_home_dir = /var/lib/mysql/innodb/iblogs/
innodb_log_arch_dir = /var/lib/mysql/innodb/iblogs/
set-variable = innodb_log_files_in_group=3
set-variable = innodb_buffer_pool_size=256M
set-variable = innodb_additional_mem_pool_size=20M
set-variable = innodb_log_file_size=57M
set-variable = innodb_log_buffer_size=2M
innodb_flush_log_at_trx_commit=1
set-variable = innodb_lock_wait_timeout=50

One of the current problems is the home page taking so long to load.
We have several managers who just pop up to check the QuickSearch
to see the status of the tickets in every queue and using it would
be painful. This is actually preventing us from migrating fully
to 3.0.

Does anyone see any optimizations to speed up our current setup?
That page is taking on average ~10 seconds to load while the RT 2.0
version takes ~5 seconds.

If no one see anything obvious, then might I suggest having the home
page cached so that displaying the home page doesnt require all
those database hits. We actually wouldn’t mind that the data is not
real time and it’s off by a min or two.

Dean

Nicolae P. Costescu, Ph.D. / Senior Developer
Stronghold Technologies
46040 Center Oak Plaza, Suite 160 / Sterling, Va 20166
Tel: 571-434-1472 / Fax: 571-434-1478

3

Mason 1.16

IIRC, there is some performance boost during 1.16 to 1.19.

perl 5.6.1

Ditto for perl 5.8.0.

One of the current problems is the home page taking so long to load.
We have several managers who just pop up to check the QuickSearch
to see the status of the tickets in every queue and using it would
be painful. This is actually preventing us from migrating fully
to 3.0.

How about putting up a page just for QuickSearch?

Thanks,
/Autrijus/

4

How about putting up a page just for QuickSearch?

http://$RTURL/Elements/Quicksearch

gets you 99% of the way there.

-R

5

Dean Kao wrote:

If no one see anything obvious, then might I suggest having the home
page cached so that displaying the home page doesnt require all
those database hits. We actually wouldn’t mind that the data is not
real time and it’s off by a min or two.

You could run wget in a cron job, writing the frontpage to a static file
every minute or something. This might take a lot of load off your system
and speed up the access for the ties.

Roy-Magne Mo

6

On Tue, Apr 22, 2003 at 11:19:49PM -0400, Nicolae P. Costescu developed
a new theory of relativity and:

You will see a big benefit from paying $400 for a motherboard/cpu/ram upgrade.
On a single CPU p4/2.4GHz, with 512MB of DDRAM you will have about 3x your
memory bandwidth, and 5x your CPU speed. This will make a BIG difference.
Your current SMP system is a shared bus architecture, and that shared RAM
bus is slow compared to the memory controller on Intel DDRAM motherboards
(i845 and above).

If you want specific recommendations for a particular mobo/cpu/ram email me
directly and I’ll be glad to spec it for you.

As much as I’d like to believe that hardware is the main cause of all
this, RT2.0 is 2x faster than RT3.0 for the main page. If it was slower
for a search, that’s fine. Just merely getting to the homepage is the
big killer.

Disabling quicksearch has been thought of but is not the
preferred option because we want that information. This is why I
suggested caching the quicksearch. I will try uprading mason
today to see if there is any improvement. I only remember MySQL 4.0
as the recommended performance booster (which took the login from
60 seconds to 10).

Dean

7

You could run wget in a cron job, writing the frontpage to a static file
every minute or something. This might take a lot of load off your system
and speed up the access for the ties.

I dont think that’ll quiet do it. That wont process the logins
properly and other user information.

Dean

8

On Wed, Apr 23, 2003 at 12:33:00PM +0800, Autrijus Tang developed
a new theory of relativity and:> On Tue, Apr 22, 2003 at 03:30:07PM -0700, Dean Kao wrote:

Mason 1.16

IIRC, there is some performance boost during 1.16 to 1.19.

perl 5.6.1

Ditto for perl 5.8.0.

I upgraded Mason to 1.19 and I’m still seeing 2x in speed from
RT 2.0 to 3.0, and this probably makes sense cause both version
probably benefit from any speed increases.

I’m hesitant on upgrading perl unless someone else has seen
an order of magnitude of speed increase from upgrading to perl
5.8.0.

I think I’m going to look into caching hack for QuickSearch.

Dean

9

Dean Kao wrote:

I think I’m going to look into caching hack for QuickSearch.

Have you considered using the mason caching api ?

Something like this in the autohandler should help :

<%init>
return if $r->uri eq ‘/index.html’
&& !keys %ARGS
&& $m->cache_self(expire_in => ‘5 mins’,key=> ‘/’);

10

Have you considered using the mason caching api ?

Something like this in the autohandler should help :

<%init>
return if $r->uri eq ‘/index.html’
&& !keys %ARGS
&& $m->cache_self(expire_in => ‘5 mins’,key=> ‘/’);

That’s exactly what I did. This is a quick little hack
which has increased the speed of RT homepage significantly.
I’m thinking of investing some time into getting caching
integrated to commonly used functionalities of RT. I’ll
see how far I get into it.

Dean

Quicksearch.diff (1.39 KB)

11

Mason 1.16

IIRC, there is some performance boost during 1.16 to 1.19.

Not that I know of :wink:

1.20 will be faster for some configurations.

perl 5.6.1

Ditto for perl 5.8.0.

Actually, 5.8.0 may be slower than 5.6.1 overall, particularly because of
better Unicode support.

-dave

/=======================
House Absolute Consulting


=======================/

12

Hi,

I have just completed a transfer from 2 to 3 and I am seeing something
very similar:

Single Athlon 1Ghz
512Mb RAM
RT 3.0.1
Apache 1.3.27
MySQL 4.0.12
Linux 2.2.17
perl 5.8.0
mod_perl 1.27
HTML-Mason-1.19
All perl dependencies etc met, (see below for output).

MySQL specific configuration:

datadir=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=128M
innodb_additional_mem_pool_size=32M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

RT3 is just running dog slow, on the same system that runs RT2 at a
decent speed. The only difference I can say other than the difference
between RT2 and RT3 is that RT3 is using InnoDB where as RT2 was not.

I’ve never used InnoDB before, so am ploughing through MySQL manuals and
looking at performance tweaks, however it can take 30 seconds to a minut

  • to load the first screen, and similar for any action that uses the DB.

Any suggestions, pointers or whatever, (aside from RTFM, cos I am doing
… lots) … I would greatly appreciate it. But I can sympathise with
Dan as I am getting lots of grief and its at times taking the box right
over and slowing everything else down that runs on it.

Cheers,

Simon.

== Dependency Output: ==

MASON dependencies:
Params::Validate 0.02…found
Cache::Cache …found
Exception::Class …found
HTML::Mason 1.16…found
MLDBM …found
Errno …found
FreezeThaw …found
Digest::MD5 …found
CGI::Cookie 1.20…found
Storable …found
Apache::Session 1.53…found
MAILGATE dependencies:
HTML::TreeBuilder …found
HTML::FormatText …found
Getopt::Long …found
LWP::UserAgent …found
MODPERL1 dependencies:
CGI …found
Apache::Request …found
Apache::DBI …found
CLI dependencies:
Getopt::Long 2.24…found
CORE dependencies:
Digest::MD5 …found
DBI 1.18…found
Test::Inline …found
Class::ReturnValue 0.40…found
DBIx::SearchBuilder 0.80…found
Text::Template …found
File::Spec 0.8…found
HTML::Entities …found
Net::Domain …found
Log::Dispatch 2.0…found
Locale::Maketext 1.04…found
Locale::Maketext::Lexicon 0.10…found
Locale::Maketext::Fuzzy …found
MIME::Entity 5.108…found
Mail::Mailer 1.57…found
Net::SMTP …found
Text::Wrapper …found
Time::ParseDate …found
File::Temp …found
Term::ReadKey …found
Text::Autoformat …found
Text::Quoted …found
DEV dependencies:
Regexp::Common …found
Time::HiRes …found
Test::Inline …found
WWW::Mechanize …found
MYSQL dependencies:
DBD::mysql 2.1018…found

13

Hi,

I have just completed a transfer from 2 to 3 and I am seeing something
very similar:

Single Athlon 1Ghz
512Mb RAM
RT 3.0.1
Apache 1.3.27
MySQL 4.0.12
Linux 2.2.17
perl 5.8.0
mod_perl 1.27
HTML-Mason-1.19
All perl dependencies etc met, (see below for output).

MySQL specific configuration:

datadir=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=128M
innodb_additional_mem_pool_size=32M

I don’t have the mysql docs in front of me, but these feel low.

innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

RT3 is just running dog slow, on the same system that runs RT2 at a
decent speed. The only difference I can say other than the difference
between RT2 and RT3 is that RT3 is using InnoDB where as RT2 was not.

I’ve never used InnoDB before, so am ploughing through MySQL manuals and
looking at performance tweaks, however it can take 30 seconds to a minut

  • to load the first screen, and similar for any action that uses the DB.

Things that would be useful to know:
How much free memory on the box? Is it swapping?
How many rows in your: Users, Groups, GroupMembers,
CachedGroupMembers, Queues, ACL, Tickets and Transactions
tables?

What load is the box running at?

Are you sure the RT tables are actually innodb? Mysql may
silently ignore the innodb directive if you don't have it
configured right.

What mysql queries appear to be taking a long time?

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

14

Hi Jesse,

I don’t have the mysql docs in front of me, but these feel low.

I tried higher, but the box just didn’t have the available resources.

This server also runs our intranet and Samba file shares for our
internal network, however it managed to run RT2 with no problem, even
using MySQL 4 (MYI, not InnoDB however) So it didn’t like assigning 256M
etc of memory to just mysql. (Samba and intranet have run on this server
no problem for ages and hardly take up any resources).

Things that would be useful to know:
How much free memory on the box? Is it swapping?

;-), with the settings above, yup, its using it all, and its mysql thats
grabbing it … and yes it is swapping … which I accept is a cause for
it being slow, however yet again, it was fine under RT2

How many rows in your: Users, Groups, GroupMembers,
CachedGroupMembers, Queues, ACL, Tickets and Transactions
tables?

Users: 12828
Groups: 170161 } Errr, does this not look too large ?
GroupMembers: 114993 } This data is a fresh import from the
CachedGroupMembers: 387317 } rt2 dumped data using your util.
Queues: 20
ACL: 201
Tickets: 39308
Transactions: 149330

What load is the box running at?

On average, its looking about 1.5 (which is abnormal), but its going up
between 10-40 as it gets busy, and its all mysql.

2:31pm up 8 days, 4:41, 3 users, load average: 2.62, 2.42, 3.81
112 processes: 109 sleeping, 3 running, 0 zombie, 0 stopped
CPU states: 69.8% user, 2.5% system, 0.0% nice, 27.5% idle
Mem: 517176K av, 507700K used, 9476K free, 141460K shrd, 50604K
buff
Swap: 136512K av, 77728K used, 58784K free 116040K
cached

PID USER PRI NI SIZE RSS SHARE STAT LIB %CPU %MEM TIME
COMMAND
14849 mysqld 20 0 97664 95M 2564 R 296 62.2 18.8 1:17
mysqld

Are you sure the RT tables are actually innodb? Mysql may
silently ignore the innodb directive if you don’t have it
configured right.

Yes. I am new to InnoDB, however it seems to be:

-rw-rw---- 1 mysqld daemon 673185792 May 6 14:28 ibdata1
-rw-rw---- 1 mysqld daemon 5242880 May 6 14:28 ib_logfile0
-rw-rw---- 1 mysqld daemon 5242880 May 6 14:28 ib_logfile1

root@cropton:/usr/local/var/mysql# du -sk rt3
484 rt3

root@cropton:/usr/local/var/mysql/rt3# ls -altr
total 488
-rw-rw---- 1 mysqld daemon 9006 May 4 20:56 Queues.frm
-rw-rw---- 1 mysqld daemon 8656 May 4 20:56 Principals.frm
-rw-rw---- 1 mysqld daemon 8838 May 4 20:56 Links.frm
-rw-rw---- 1 mysqld daemon 8920 May 4 20:56 Attachments.frm
-rw-rw---- 1 mysqld daemon 9744 May 4 20:57 Users.frm
-rw-rw---- 1 mysqld daemon 8856 May 4 20:57 Transactions.frm
-rw-rw---- 1 mysqld daemon 9388 May 4 20:57 Tickets.frm
-rw-rw---- 1 mysqld daemon 9134 May 4 20:57 Scrips.frm
-rw-rw---- 1 mysqld daemon 8876 May 4 20:57
ScripConditions.frm
-rw-rw---- 1 mysqld daemon 8696 May 4 20:57 Groups.frm
-rw-rw---- 1 mysqld daemon 8612 May 4 20:57 GroupMembers.frm
-rw-rw---- 1 mysqld daemon 8716 May 4 20:57
CachedGroupMembers.frm
-rw-rw---- 1 mysqld daemon 8812 May 4 20:57 ACL.frm
-rw-rw---- 1 mysqld daemon 8622 May 4 20:57 sessions.frm
-rw-rw---- 1 mysqld daemon 8786 May 4 20:57
TicketCustomFieldValues.frm
-rw-rw---- 1 mysqld daemon 8906 May 4 20:57 Templates.frm
-rw-rw---- 1 mysqld daemon 8820 May 4 20:57 ScripActions.frm
-rw-rw---- 1 mysqld daemon 8868 May 4 20:57 CustomFields.frm
-rw-rw---- 1 mysqld daemon 8824 May 4 20:57
CustomFieldValues.frm
drwx------ 2 mysqld daemon 4096 May 4 20:57 ./
drwx------ 18 mysqld daemon 4096 May 6 13:43 …/
-rw-rw---- 1 mysqld daemon 24576 May 6 14:29 sessions.MYI
-rw-rw---- 1 mysqld daemon 227348 May 6 14:29 sessions.MYD

What mysql queries appear to be taking a long time?

TBH, everything, from logging in, to displaying a ticket … however it should
be noted that those select count(*)'s I did for you above were quick, so I
suspect we’re talking about queries involving joins, but I haven’t got anything
more specific than that.

Cheers,

Simon.

15

datadir=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=128M
innodb_additional_mem_pool_size=32M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

Here’s my configuration for the Win32 binary distribution,
and the performance is rather good:

my %Conf = (
key_buffer_size => ‘256M’,
table_cache => ‘64’,

query_cache_size                    => '4M',
max_allowed_packet                  => '1M',
sort_buffer                         => '1M',
record_buffer                       => '1M',

innodb_buffer_pool_size             => '256M',
innodb_additional_mem_pool_size     => '20M',

);

So it looks like you forgot the first two config, which is
arguably the most critical pair.

/Autrijus/

16

MySQL specific configuration:

datadir=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=128M
innodb_additional_mem_pool_size=32M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

Here’s my config for the Win32 binary distributoni of RT,
and the performance is rather good (on mysql 4.0.1x):

my %Conf = (
key_buffer_size => ‘256M’,
table_cache => ‘64’,

query_cache_size			=> '4M',
max_allowed_packet			=> '1M',
sort_buffer				=> '1M',
record_buffer			=> '1M',

innodb_buffer_pool_size		=> '256M',
innodb_additional_mem_pool_size	=> '20M',

);

The first two are arguably the most important pair; did you
forget to set them?

/Autrijus/

17

The first two are arguably the most important pair; did you
forget to set them?

I must have left them out, they are set:

key_buffer_size=128M
table_cache=64

I have however tweaked further with the box, and now have it running with:

key_buffer_size=256M
table_cache=64
query_cache_size=4M
max_allowed_packet=1M
sort_buffer=1M
record_buffer=1M

innodb_data_file_path=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=256M
innodb_additional_mem_pool_size=64M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

And it hasn’t made a difference :-(.

18

excuse my ignorance, but which config file is this information in? I’d like
to check mine out and see if I can tweak the performance a bit.

thanks,
-ryan-----Original Message-----
From: Simon Woodward [mailto:sw-lists@onyx.net]
Sent: Tuesday, May 06, 2003 7:55 AM
To: Autrijus Tang
Cc: Dean Kao; rt-users@lists.fsck.com
Subject: Re: [rt-users] RT 3.0 Speed

The first two are arguably the most important pair; did you
forget to set them?

I must have left them out, they are set:

key_buffer_size=128M
table_cache=64

I have however tweaked further with the box, and now have it running with:

key_buffer_size=256M
table_cache=64
query_cache_size=4M
max_allowed_packet=1M
sort_buffer=1M
record_buffer=1M

innodb_data_file_path=/usr/local/var/mysql
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M
innodb_buffer_pool_size=256M
innodb_additional_mem_pool_size=64M
innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit=1

And it hasn’t made a difference :-(.

rt-users mailing list
rt-users@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

19

is it /etc/my.cnf?

cause i don’t have ANY of those option variables in there… i guess that
might help. (sorry about the double post)From: Ryan Wheaton [mailto:rwheaton@moguls.com]
Sent: Tuesday, May 06, 2003 8:50 AM
To: rt-users@lists.fsck.com
Subject: RE: [rt-users] RT 3.0 Speed

excuse my ignorance, but which config file is this information in? I’d like
to check mine out and see if I can tweak the performance a bit.

thanks,
-ryan

20

Hi Ryan,

excuse my ignorance, but which config file is this information in? I’d like
to check mine out and see if I can tweak the performance a bit.

Depends how you have it configured … its either in something like
/etc/my.cnf or ~/my.cnf

or, like me, you can set them from the command line (for example):

/usr/local/mysql/bin/safe_mysqld --user=mysqld
–datadir=/usr/local/var/mysql -O key_buffer_size=256M -O table_cache=64
-O query_cache_size=4M -O max_allowed_packet=1M -O sort_buffer=1M -O
record_buffer=1M -O innodb_data_file_path=/usr/local/var/mysql -O
innodb_data_file_path=ibdata1:10M:autoextend:max:2000M -O
innodb_buffer_pool_size=256M -O innodb_additional_mem_pool_size=64M -O
innodb_log_buffer_size=8M -O innodb_flush_log_at_trx_commit=1 &

You can see what they are set to using the mysql command line client, as
a prived user, type:

show variables;

Cheers,

Simon.
Simon Woodward sw-lists@onyx.net
Onyx Internet