Tuning RT4

Hi,

I have a huge(>2M Tickets, ~500 tickets/day) 3.8.1 RT installation and
I’m trying to upgrade to 4.0.1.

I’m facing some performance problems on my staging server(it’s almost
the same spec as my production server).

I’m using on 3.8.1:
Alias /rt/ /opt/rt3/share/html/
<Directory /opt/rt3/share/html/>
Order allow,deny
Allow from all

PerlModule Apache::DBI
PerlRequire /opt/rt3/bin/webmux.pl
PerlSetVar MasonStaticSource true
PerlSetVar MasonBufferPreallocateSize 4096000
PerlSetVar MasonDataCacheDefaults "cache_class=> MemoryCache, cache_depth => 4"
PerlSetVar MasonPreloads ‘/Elements/*’
<Location /rt/>
AddDefaultCharset UTF-8
SetHandler perl-script
PerlHandler RT::Mason

To make 4.0.1 works I’d have to change to:
<Location /rt>
Order allow,deny
Allow from all

        SetHandler modperl
        PerlResponseHandler Plack::Handler::Apache2
        PerlSetVar psgi_app /opt/rt4/sbin/rt-server
    </Location>
    <Perl>
        use Plack::Handler::Apache2;
        Plack::Handler::Apache2->preload("/opt/rt4/sbin/rt-server");
    </Perl>

The question is: How can I assign the Mason variables(StaticSource,
BufferSize, Preloads, …) on RT4?

Thanks,

Wagner Sartori Junior

The question is: How can I assign the Mason variables(StaticSource,
BufferSize, Preloads, …) on RT4?

In RT4 and 3 as well you can use the following option in the config to
set mason’s properties:

Set(@MasonParameters,
preamble => ‘my $p = MasonX::Profiler->new($m, $r);’,
);

Arguments are described in
http://search.cpan.org/dist/HTML-Mason/lib/HTML/Mason/Params.pod, in
RT config you should use “perl name” of the options.

Want to note that:

  • RT uses static source if DevelMode is not enabled
  • Preloading may help, but not everything can be preloaded
  • Buffer preallocation may has less or no effect on RT4

Other options are either dangerouse or useless for admins.

Best regards, Ruslan.

I have a huge(>2M Tickets, ~500 tickets/day) 3.8.1 RT installation and
I’m trying to upgrade to 4.0.1.

I’m facing some performance problems on my staging server(it’s almost
the same spec as my production server).

I’m using on 3.8.1:
Alias /rt/ /opt/rt3/share/html/
<Directory /opt/rt3/share/html/>
Order allow,deny
Allow from all

PerlModule Apache::DBI
PerlRequire /opt/rt3/bin/webmux.pl
PerlSetVar MasonStaticSource true

This should already be set if you’re not in DevelMode

PerlSetVar MasonBufferPreallocateSize 4096000
PerlSetVar MasonDataCacheDefaults "cache_class=> MemoryCache, cache_depth => 4"
PerlSetVar MasonPreloads ‘/Elements/*’

You should be able to set these the same way (since you’re still using
mod_perl) or by using the MasonParameters config option that we’ve
provided for a while

-kevin

hello guys

in this thread there are only recommendations for the old module modperl

why? is the fcgif module not so performant … as the modperl module?

best regards john
View this message in context: http://old.nabble.com/Tuning-RT4-tp32056508p32067302.html

Hello All

Actually we use rt 3.8 in prod environment and the new rt 4.0 in labor
environment

the 3.8 speeds up like an local installed program …very fast …

the 4.0.0 is not so fast as 3.8 but it is acceptable

Neverthless i would like to know why is it that so…?

RT 3.8.9:

Here are my configurations:

rt 3.8.9: http://pastebin.com/2YMK6uzt http://pastebin.com/2YMK6uzt

rt 4.0.1: http://pastebin.com/CFrNE2a3 http://pastebin.com/CFrNE2a3

could someone help me to figure out whats wrong on the rt 4.0.1 config

best regards john s.

View this message in context: http://old.nabble.com/Tuning-RT4-tp32056508p32107474.html

I would love to learn what you find. I am also experiencing slowness with
4.0.1 compared to 3.8.x. Very noticeable - page turnaround time at least
2-3 seconds longer, and sometimes longer than that. Relevant part of my
Apache config:

DocumentRoot "/usr/local/rt4/share/html"

Order allow,deny
Allow from all

               SetHandler modperl
               PerlResponseHandler Plack::Handler::Apache2
               PerlSetVar psgi_app /usr/local/rt4/sbin/rt-server

Added for attempt at tuning

PerlSetVar MasonStaticSource true
PerlSetVar MasonBufferPreallocateSize 4096000
PerlSetVar MasonDataCacheDefaults "cache_class=> MemoryCache, cache_depth
=> 4"
PerlSetVar MasonPreloads '/Elements/'
PerlSetVar MasonPreloads '/Ticket/
'
PerlSetVar MasonPreloads '/Articles/'
PerlSetVar MasonPreloads '/Dashboards/
'
PerlSetVar MasonPreloads '/NoAuth/'
PerlSetVar MasonPreloads '/Tools/
'
PerlSetVar MasonPreloads '/User/'
PerlSetVar MasonPreloads '/Widgets/
'
PerlSetVar MasonPreloads '/Helpers/*'
AddOutputFilterByType DEFLATE text/html text/plain text/xml

           </Location>
           <Perl>
               use Plack::Handler::Apache2;

Plack::Handler::Apache2->preload("/usr/local/rt4/sbin/rt-server");
On Thu, Jul 21, 2011 at 9:35 AM, john s. fireskyer@gmx.de wrote:

Hello All

from my point of view

Actually we use rt 3.8 in prod environment and the new rt 4.0 in labor
environment

the 3.8 speeds up like an local installed program …very fast …

the 4.0.0 is not so fast as 3.8 but it is acceptable

Neverthless i would like to know why is it that so…?

RT 3.8.9:

Here are my configurations:

rt 3.8.9: http://pastebin.com/2YMK6uzt http://pastebin.com/2YMK6uzt

rt 4.0.1: http://pastebin.com/CFrNE2a3 http://pastebin.com/CFrNE2a3

could someone help me to figure out whats wrong on the rt 4.0.1 config

best regards john s.


View this message in context:
http://old.nabble.com/Tuning-RT4-tp32056508p32107474.html
Sent from the Request Tracker - User mailing list archive at Nabble.com.


2011 Training: http://bestpractical.com/services/training.html

I would love to learn what you find. I am also experiencing slowness with 4.0.1 compared to
3.8.x. Very noticeable - page turnaround time at least 2-3 seconds longer, and sometimes
longer than that. Relevant part of my Apache config:

What exactly is slow? Without knowing the part of the stack which is
being slow, it’s hard to fix anything. If you turn on
MasonX::Profiler it may be possible to get a much better sense of the
problem.

You can also see a detailed explanation of the “javascript
minification is slow” problem that occurs for some users on this
ticket:

http://issues.bestpractical.com/Ticket/Display.html?id=17673

You can log in as guest/guest

Added for attempt at tuning

               PerlSetVar MasonStaticSource true
               PerlSetVar MasonBufferPreallocateSize 4096000
               PerlSetVar MasonDataCacheDefaults "cache_class=> MemoryCache, cache_depth =>

4"
PerlSetVar MasonPreloads '/Elements/'
PerlSetVar MasonPreloads '/Ticket/
'
PerlSetVar MasonPreloads '/Articles/'
PerlSetVar MasonPreloads '/Dashboards/
'
PerlSetVar MasonPreloads '/NoAuth/'
PerlSetVar MasonPreloads '/Tools/
'
PerlSetVar MasonPreloads '/User/'
PerlSetVar MasonPreloads '/Widgets/
'
PerlSetVar MasonPreloads ‘/Helpers/*’

Do you find that these help?

-kevin

hello dave

thats the point why i finally decided to use rt 3.8

i have an config with fcgid on my rt 4.0 system

… could someone from the devs say any words to this speeding issue
which we could verify each other ?

and if this the normal state… why i should sacrifice so much more speed
if i use the rt 4.0 version

where is the benefit … ? … for a little prettier user interface? … for
my point of view not really

… but maybe we have overlooked something in the configs … i don’t
know

best regards john

View this message in context: http://old.nabble.com/Tuning-RT4-tp32056508p32132285.html

I can’t speak for the dev team, but I have noticed a distinct trend:

Apache+mod_perl is very slow with RT4.0.
Lighttpd+FCGI is fast with RT4.0.
Standalone PSGI is really fast with RT4.0 (but not very scalable unless
you know what you’re doing).

I haven’t tested Apache with anything other than mod_perl, so I don’t know
if it’s Apache-specific or mod_perl-specific.

Most of the speed problems I’ve seen posted here are with RHEL/CentOS/SL
stock httpd & mod_perl installs, so it’s also possible the problem is
OS-specific.

(And yes, I have a ticket open to try to narrow this down; unfortunately I
haven’t had time to keep following up on it.)

-Adam Thompson
athompso@athompso.net

Hello…

I am a brand new RT user. This is my first post. We are network people and dont know much about apache and web server tuning and so forth. (learning fast).

But, the first trial install of RT that I made using Rt 3.8 using the Ubuntu packages on an Ubuntu 10.04 virtual machine with 1gb ram was super fast.

Having reviewed RT on that platform, I decided to commit to RT, and made an install of RT 4.0.1 on a fresh but identical Ubuntu 10.04 virtual machine, using Apache2 and mod_perl.

The RT 4.0.1 is very sluggish compared to the 3.8 version. My RT installs have only the general queue and 5 or 10 tickets as I am just in the trial stage.

The web UI is Very slow in 4.0.1.

The php.info and other apache status pages, and webmin that I have running on this machine are superfast, so I point to RT as being the culprit.

Can anyone help me troubleshoot this?

Thanks,

Alex

Techy details about my install:

OS: Ubuntu 10.04 vsphere 4.1 esxi virtual machine, 1gb ram, 1cpu, 20gb disk thin provisioned.
Apache: Server Version: Apache/2.2.14 (Ubuntu) PHP/5.3.2-1ubuntu4.9 with Suhosin-Patch mod_python/3.3.1 Python/2.6.5 mod_ssl/2.2.14 OpenSSL/0.9.8k mod_perl/2.0.4 Perl/v5.10.1
PHP: PHP Version 5.3.2-1ubuntu4.9
MySQL: mysql Ver 14.14 Distrib 5.1.41, for debian-linux-gnu (i486) using readline 6.1 - Innodb enabled and set to default
Perl: v5.10.1 (*)
RT: 4.0.1

Contents of RT_SiteConfig

You must restart your webserver after making changes to this file.

Set($rtname, ‘ITworX Limited’);
Set($Organization, ‘rt.itworx.co.ke’);
Set($Timezone, ‘AFRICA/Nairobi’);
Set($WebDomain, ‘rt.itworx.co.ke’);
Set($WebBaseURL, ‘https://rt.itworx.co.ke’);
Set($WebPath, ‘’);
Set($WebPort, 443);
Set($OwnerEmail, "alex@itworx.co.ke");
Set($CorrespondAddress, ‘support@itworx.co.ke’);
Set($CommentAddress, ‘support@itworx.co.ke’);
Set($SendmailArguments, “-oi -t”);

Logging

Set($LogToSyslog, ‘’);
Set($LogToFile, ‘debug’);
Set($LogDir, ‘/opt/rt4/var/log’);
Set($LogToFileNamed, “rt-debug.log”);

Contents of Apache virtual host entry
<VirtualHost *:443>
SSLEngine on
SSLCertificateFile /etc/ssl/certs/cert.pem
ServerAdmin admin@itworx.co.ke
DocumentRoot /opt/rt4/share/html
AddDefaultCharset UTF-8

Order allow,deny Allow from all

SetHandler modperl
SetHandler modperl
PerlResponseHandler Plack::Handler::Apache2
PerlSetVar psgi_app /opt/rt4/sbin/rt-server


use Plack::Handler::Apache2;
Plack::Handler::Apache2->preload("/opt/rt4/sbin/rt-server");

----- Original Message -----
From: “Adam Thompson” athompso@athompso.net
To: “john s.” fireskyer@gmx.de, rt-users@lists.bestpractical.com
Sent: Monday, 25 July, 2011 7:34:27 PM
Subject: Re: [rt-users] Tuning RT4

I can’t speak for the dev team, but I have noticed a distinct trend:

Apache+mod_perl is very slow with RT4.0.
Lighttpd+FCGI is fast with RT4.0.
Standalone PSGI is really fast with RT4.0 (but not very scalable unless
you know what you’re doing).

I haven’t tested Apache with anything other than mod_perl, so I don’t know
if it’s Apache-specific or mod_perl-specific.

Most of the speed problems I’ve seen posted here are with RHEL/CentOS/SL
stock httpd & mod_perl installs, so it’s also possible the problem is
OS-specific.

(And yes, I have a ticket open to try to narrow this down; unfortunately I
haven’t had time to keep following up on it.)

-Adam Thompson
athompso@athompso.net

2011 Training: http://bestpractical.com/services/training.html

This e-mail contains confidential information or information belonging to ITworX Limited and is intended solely for the addressees. The opinions therein, explicit or implied, are solely those of the author and do not necessarily represent those of ITworX Limited as a company. The unauthorized disclosure, use, dissemination or copying (either Whole or partial) of this e-mail, or any information it contains, is prohibited. E-mails are susceptible to alteration and their integrity cannot be guaranteed. ITworX Limited shall not be liable for this e-mail if modified or falsified. If you are not the intended recipient of this e-mail, please delete it immediately from your system and notify the sender of the wrong delivery and the e-mail deletion.

I haven’t tested Apache with anything other than mod_perl, so I don’t know
if it’s Apache-specific or mod_perl-specific.

It’s unlikely to be Apache-specific, as we run multiple RT 4.0.1s
without this issue on Apache.

(And yes, I have a ticket open to try to narrow this down; unfortunately I
haven’t had time to keep following up on it.)

That is the ticket I linked to earlier which explains that a lot of
the slowness complaints are because each apache child needs to compile
JS/CSS on a first request. If you only make 5 requests, that may not
be enough to run through all of the children’s first requests. My
coworker has provided some test harnesses for diagnosing the problems.

If other folks can help run the harness to identify potential
problems, it would be useful to us.

http://issues.bestpractical.com/Ticket/Display.html?id=17673

-kevin

The php.info and other apache status pages, and webmin that I have running on this machine are
superfast, so I point to RT as being the culprit.

php.info and the apache status pages are not actually performance
tests since they just dump information to the page without doing any
performance tweaking.

1G of ram for web + database is a tight fit.

Please see my earlier posts in the thread for a link to a ticket with
a way to get more information about your apache config.

Also, running with MasonX::Profiler (as documented in the config) will
provide information about what is really taking time.

-kevin

Dear Kevin,

Thank you for the input.

I have found my problem. Its a real rookie mistake, so I apologise for wasting your time.

You mentioned that 1gb ram wasnt enough (I thought it would be for my 2 user test platform), so since its virtual, I went over to increase the ram.

Guess what! It was set to 256meg… Ouch!

Sorry about that.

Alex----- Original Message -----
From: “Kevin Falcone” falcone@bestpractical.com
To: rt-users@lists.bestpractical.com
Sent: Monday, 25 July, 2011 8:20:03 PM
Subject: Re: [rt-users] Tuning RT4

On Mon, Jul 25, 2011 at 07:54:54PM +0300, Alex Rhys-Hurn wrote:

The php.info and other apache status pages, and webmin that I have running on this machine are
superfast, so I point to RT as being the culprit.

php.info and the apache status pages are not actually performance
tests since they just dump information to the page without doing any
performance tweaking.

1G of ram for web + database is a tight fit.

Please see my earlier posts in the thread for a link to a ticket with
a way to get more information about your apache config.

Also, running with MasonX::Profiler (as documented in the config) will
provide information about what is really taking time.

-kevin

2011 Training: http://bestpractical.com/services/training.html
This e-mail contains confidential information or information belonging to ITworX Limited and is intended solely for the addressees. The opinions therein, explicit or implied, are solely those of the author and do not necessarily represent those of ITworX Limited as a company. The unauthorized disclosure, use, dissemination or copying (either Whole or partial) of this e-mail, or any information it contains, is prohibited. E-mails are susceptible to alteration and their integrity cannot be guaranteed. ITworX Limited shall not be liable for this e-mail if modified or falsified. If you are not the intended recipient of this e-mail, please delete it immediately from your system and notify the sender of the wrong delivery and the e-mail deletion.

I can’t speak for the dev team, but I have noticed a distinct trend:

Apache+mod_perl is very slow with RT4.0.
Lighttpd+FCGI is fast with RT4.0.
Standalone PSGI is really fast with RT4.0 (but not very scalable unless
you know what you’re doing).

I haven’t tested Apache with anything other than mod_perl, so I don’t know
if it’s Apache-specific or mod_perl-specific.

Most of the speed problems I’ve seen posted here are with RHEL/CentOS/SL
stock httpd& mod_perl installs, so it’s also possible the problem is
OS-specific.

(And yes, I have a ticket open to try to narrow this down; unfortunately I
haven’t had time to keep following up on it.)

Hi, just thought I’d point out my experience, we are currently testing
upgrading RT 3.8.8 to RT 4.0.1.

RT 3.8.8: RHEL5 + Apache + mod_perl + MySQL.

RT 4.0.1: RHEL6 + Apache + mod_perl + MySQL.

~120K tickets, ~13K users, DB ~11GB.

RT 4 is faster than RT3 so far.

Probably the most notable change is setting:

Set($AutocompleteOwners, 1);

Compare the display time “Tickets->New Search”

With AutocompleteOwners: Time to display: 0.442697
Without AutocompleteOwners: Time to display: 17.274538

13K users is quite painful on that page!

Cheers, Jeff.

Set($AutocompleteOwners, 1);

Compare the display time “Tickets->New Search”

With AutocompleteOwners: Time to display: 0.442697
Without AutocompleteOwners: Time to display: 17.274538

13K users is quite painful on that page!

I’ll note that on a system with 25K users, that page takes about .3 to
render for me (without switching to autocomplete). The owner list is
less concerned with User size and more with the distribution of the
OwnTicket ACL to different kinds of users.

-kevin

Set($AutocompleteOwners, 1);

Compare the display time “Tickets->New Search”

With AutocompleteOwners: Time to display: 0.442697
Without AutocompleteOwners: Time to display: 17.274538

13K users is quite painful on that page!

I’ll note that on a system with 25K users, that page takes about .3 to
render for me (without switching to autocomplete). The owner list is
less concerned with User size and more with the distribution of the
OwnTicket ACL to different kinds of users.

This is interesting. Most of the time appears to be spent after the DB
call, I think it’s formatting and sorting the drop down.

Cheers, Jeff.

Hello Jeffry

Due to your really large RT Environment…

I have tested

rt 4.0.0.1

and rt 3.8.9

only with a few user and tickets …

maybe rt 4 can handle large rt instances much better than rt 3.8.8 and by
the othe one is vice versa?

so if this isn’t the case could you post your config ? which has sth to be
with the actual issue? i think i’ll be helpful

best regards john

View this message in context: http://old.nabble.com/Tuning-RT4-tp32056508p32146258.html

I’ll note that on a system with 25K users, that page takes about .3 to
render for me (without switching to autocomplete). The owner list is
less concerned with User size and more with the distribution of the
OwnTicket ACL to different kinds of users.

This is interesting. Most of the time appears to be spent after the DB call,
I think it’s formatting and sorting the drop down.

That’s totally depends on how many users are displayed in the
dropdown. Is it 13k?

Cheers, Jeff.


2011 Training: http://bestpractical.com/services/training.html

Best regards, Ruslan.

Hello Jeffry

Due to your really large RT Environment…

I have tested

rt 4.0.0.1

and rt 3.8.9

only with a few user and tickets …

maybe rt 4 can handle large rt instances much better than rt 3.8.8 and by
the othe one is vice versa?

so if this isn’t the case could you post your config ? which has sth to be
with the actual issue? i think i’ll be helpful

best regards john

Sure, not much different from what’s in the docs:

— start /etc/httpd/conf.d/rt.conf —

Alias /rt "/usr/share/rt/html"
Alias /NoAuth “/usr/share/rt/html/NoAuth”

<Location /rt>
RedirectMatch permanent (.*)/$ $1/index.html
SetHandler modperl
PerlResponseHandler Plack::Handler::Apache2
PerlSetVar psgi_app /usr/sbin/rt-server

— snip authentication settings —

Order deny,allow
Allow from localhost localhost.localdomain
Deny from all

<Location /rt/NoAuth/>
Order deny,allow
Allow from any

use Plack::Handler::Apache2; Plack::Handler::Apache2->preload("/usr/sbin/rt-server");

— end /etc/httpd/conf.d/rt.conf —

And here are the general settings:

— start /etc/rt/RT_SiteConfig.pm —

Set(%GnuPG, Enable => 0 );
Set($UseSQLForACLChecks, 1);
Set($AutocompleteOwners, 1);
Set($ArticleOnTicketCreate, 1);
Set(@Active_MakeClicky, qw(httpurl));
Set($ParseNewMessageForTicketCcs , 1);
Set($StrictLinkACL, 0);
Set($PluginPath,"/usr/share/rt/plugins");
Set($UseSideBySideLayout, 0);
Set($ChartsTimezonesInDB, 1);
Set($UseTransactionBatch, 1);

—end /etc/rt/RT_SiteConfig.pm —

Cheers, Jeff.

Sorry for the delay in responding to this. I have been away and distracted.

I enabled MasonX::Profiler but can’t figure out where the debug info gets
written.

Trying to get to the bottom of what exactly is slow because it is bad enough
that it destroys my workflow.

DaveOn Mon, Jul 25, 2011 at 9:32 AM, Kevin Falcone falcone@bestpractical.comwrote:

On Sat, Jul 23, 2011 at 12:05:37AM -0400, Dave Pascoe wrote:

I would love to learn what you find. I am also experiencing slowness
with 4.0.1 compared to
3.8.x. Very noticeable - page turnaround time at least 2-3 seconds
longer, and sometimes
longer than that. Relevant part of my Apache config:

What exactly is slow? Without knowing the part of the stack which is
being slow, it’s hard to fix anything. If you turn on
MasonX::Profiler it may be possible to get a much better sense of the
problem.

You can also see a detailed explanation of the “javascript
minification is slow” problem that occurs for some users on this
ticket:

http://issues.bestpractical.com/Ticket/Display.html?id=17673

You can log in as guest/guest

Added for attempt at tuning

               PerlSetVar MasonStaticSource true
               PerlSetVar MasonBufferPreallocateSize 4096000
               PerlSetVar MasonDataCacheDefaults "cache_class=>

MemoryCache, cache_depth =>

4"
PerlSetVar MasonPreloads '/Elements/'
PerlSetVar MasonPreloads '/Ticket/
'
PerlSetVar MasonPreloads '/Articles/'
PerlSetVar MasonPreloads '/Dashboards/
'
PerlSetVar MasonPreloads '/NoAuth/'
PerlSetVar MasonPreloads '/Tools/
'
PerlSetVar MasonPreloads '/User/'
PerlSetVar MasonPreloads '/Widgets/
'
PerlSetVar MasonPreloads ‘/Helpers/*’

Do you find that these help?

-kevin


2011 Training: http://bestpractical.com/services/training.html