RT3 tuning - web interface seems slow

We are running RT 3.8.1, Apache 2.2.3 with mysql 5.0.45 on a separate
machine over (at least) 100bT connections.

The web interface to RT3 seems slow to me (I want computers to react
faster than I can, i.e. preferably under 300ms). From clicking on a
ticket in “found N tickets” to seeing the full ticket information can
take up to 7 seconds, even with only 4 or so fairly short attachments.
(the Firefox client is reasonably unloaded and displays other websites
fairly quickly, generally under a second for ones on the LAN).

Our system is not especially busy - not serving lots of concurrent users.
Usually only 1 I suspect.
I have been through some of the tuning guides (mysqltuner.pl,
PerformanceTuning - Request Tracker Wiki,
www.masonhq.com/docs/manual/Params.html) and tweaked some things to
little avail.

DBIx::SearchBuilder.pm is version 1.54
/opt/rt3/etc/RT_SiteConfig.pm has e.g.
Set($DevelMode, ‘0’);
Set($WebFlushDbCacheEveryRequest, ‘0’);
Set($LogToSyslog , ‘info’);
Set($LogToScreen , ‘error’);
Set($LogToFile , undef);

Interrogating the DB over the network seems reasonably fast:
$ time ( echo ‘use rt3 ; select * from Attachments where id = 20367 ; \q’ |
mysql -h rt3db -pXXX )
gives
real 0m0.031s

Using the command-line client over HTTP/SSL is somewhat slower, but still
tolerable:
$ time rt show ticket/2143/attachments
real 0m0.242s

Any suggestions / stupid stuff to check ?

Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager

Anything in apache logs ?-----Original Message-----
From: Andrew Daviel advax@triumf.ca

Date: Mon, 6 Apr 2009 18:49:02
To: rt-users@lists.bestpractical.com
Subject: [rt-users] RT3 tuning - web interface seems slow

We are running RT 3.8.1, Apache 2.2.3 with mysql 5.0.45 on a separate
machine over (at least) 100bT connections.

The web interface to RT3 seems slow to me (I want computers to react
faster than I can, i.e. preferably under 300ms). From clicking on a
ticket in “found N tickets” to seeing the full ticket information can
take up to 7 seconds, even with only 4 or so fairly short attachments.
(the Firefox client is reasonably unloaded and displays other websites
fairly quickly, generally under a second for ones on the LAN).

Our system is not especially busy - not serving lots of concurrent users.
Usually only 1 I suspect.
I have been through some of the tuning guides (mysqltuner.pl,
PerformanceTuning - Request Tracker Wiki,
www.masonhq.com/docs/manual/Params.html) and tweaked some things to
little avail.

DBIx::SearchBuilder.pm is version 1.54
/opt/rt3/etc/RT_SiteConfig.pm has e.g.
Set($DevelMode, ‘0’);
Set($WebFlushDbCacheEveryRequest, ‘0’);
Set($LogToSyslog , ‘info’);
Set($LogToScreen , ‘error’);
Set($LogToFile , undef);

Interrogating the DB over the network seems reasonably fast:
$ time ( echo ‘use rt3 ; select * from Attachments where id = 20367 ; \q’ |
mysql -h rt3db -pXXX )
gives
real 0m0.031s

Using the command-line client over HTTP/SSL is somewhat slower, but still
tolerable:
$ time rt show ticket/2143/attachments
real 0m0.242s

Any suggestions / stupid stuff to check ?

Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Have you enabled pipelining in Firefox?
Also, while there are security implications, if running over SSL
allowing the caching of SSL content can really help performance
(stop refetching CSS & JS for every page).

Cambridge Energy Alliance: Save money. Save the planet.

Have you enabled pipelining in Firefox?

I had not previously heard of it. I just tried; it makes no difference to
the speed of the RT3 page

Also, while there are security implications, if running over SSL
allowing the caching of SSL content can really help performance
(stop refetching CSS & JS for every page).

I just tried turning this on. Again, it makes no difference to the speed,
but I can see that some of the javascript files are being cached; at
least, they are not being asked for every time.

The main ticket pages are served with Pragma:nocache and
Cache-control:no-cache. Also Connection:close which I believe thwarts
pipelining. (according to the “live HTTP headers” plugin)

(for the benefit of others, these tweaks are under the
pseudo-location “about:config” in Firefox3, after a dire warning about
“are you sure you know what you are doing?”)

Chaim Rieger asks “anything in Apache logs?”
Nothing to my (rt3) untrained eye.
I see my ticket view request getting answered e.g.
GET /Ticket/Display.html?id=2085 HTTP/1.1" 200 75857

In the error logs, I see some errors earlier in the day, not associated
with my simple requests, e.g.
FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: \teval {…}
called at /usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/Request.pm line 467

I had not seen these prior to adding some tuning to
/etc/httpd/conf.d/rt3.conf, viz.
PerlSetVar MasonStaticSource true
PerlSetVar MasonBufferPreallocateSize 4096
PerlSetVar MasonPreloads ‘/Elements/*’

the original contents of rt3.conf are
Alias /rt3 “/usr/share/rt3/html”
PerlRequire /usr/sbin/webmux.pl
<Directory “/usr/share/rt3/html”>
AllowOverride All
Options ExecCGI FollowSymLinks
RewriteEngine On
RedirectMatch permanent (.*)/$ $1/index.html
AddDefaultCharset UTF-8
SetHandler perl-script
PerlHandler RT::Mason

I did not do the install myself, but I believe everything is correct. RT3
is working, I just think it’s slow.

What response time are others seeing on the Web interface ?

Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager

Have you enabled pipelining in Firefox?

I had not previously heard of it. I just tried; it makes no difference to
the speed of the RT3 page
Right, I’ve not sen much benefit from this (hadn’t checked for countermanding
headers from RT), but though It’d mention it. I definitely see an improvement
with cacheing, but perhaps you have other, larger bottlenecks.

Cambridge Energy Alliance: Save money. Save the planet.

Andrew,

Could you post the System Information from inside Configuration → Tools
for your RT instance?On Tue, Apr 07, 2009 at 01:36:02PM -0400, Jerrad Pierce wrote:

Have you enabled pipelining in Firefox?

I had not previously heard of it. I just tried; it makes no difference to
the speed of the RT3 page
Right, I’ve not sen much benefit from this (hadn’t checked for countermanding
headers from RT), but though It’d mention it. I definitely see an improvement
with cacheing, but perhaps you have other, larger bottlenecks.


Cambridge Energy Alliance: Save money. Save the planet.


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Andrew,

Could you post the System Information from inside Configuration → Tools
for your RT instance?

See http://andrew2.triumf.ca/helpdesk.config2.notes

(some minor sanitization)

This is running on Scientific Linux 5.2.
(Basically RedHat Enterprise recompiled from public source, like CENTOS)

We had upgraded at one point from I believe 3.0 to 3.8 using the upgrade
scripts.

Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager

Andrew,

Could you post the System Information from inside Configuration → Tools
for your RT instance?

See http://andrew2.triumf.ca/helpdesk.config2.notes

(some minor sanitization)

I’d expect to see Apache::DBI. That’s a small bit of a perf hit right
there.

This is running on Scientific Linux 5.2.
(Basically RedHat Enterprise recompiled from public source, like CENTOS)

We had upgraded at one point from I believe 3.0 to 3.8 using the upgrade
scripts.

You’ll want to carefully doublecheck database indexes. Also, have your
DBAs been maintaining the RT database? Have statistics been recomputed
lately?

I’d expect to see Apache::DBI. That’s a small bit of a perf hit right
there.

rt-test-dependencies says that it’s present, as does the cpan CLI if
asked to install “Apache::DBI”.
I see Apache::Session::Store::DBI v1.02; in the config
$ rt-test-dependencies -v
DBI >= 1.37…found

You’ll want to carefully doublecheck database indexes. Also, have your
DBAs been maintaining the RT database? Have statistics been recomputed
lately?

I believe all install and upgrade recipes were followed properly. Though
we currently have a (presumably) unrelated problem with image attachments
getting corrupted (change to UTF-8 ??).

We have a relatively small staff with no fulltime DBA. I am not sure what
maintenance is recommended; I recently ran “optimize table sessions”
recommended by the tuner which made no difference.

I do not believe any statistics were computed; where is this documented ?
I found RT3BatchStats in the wiki which fails halfway with “Unknown
column ‘Transactions.Ticket’”

Per my original post, raw database queries seem to run quite fast. Unless
I am missing some complex query which the Web interface is making.

regards

Andrew Daviel, TRIUMF, Canada
Tel. +1 (604) 222-7376 (Pacific Time)
Network Security Manager

I’d expect to see Apache::DBI. That’s a small bit of a perf hit right
there.

rt-test-dependencies says that it’s present, as does the cpan CLI if
asked to install “Apache::DBI”.
I see Apache::Session::Store::DBI v1.02; in the config
$ rt-test-dependencies -v
DBI >= 1.37…found

You need to explicitly configure it in your apache config if you’re
using mod_perl

You’ll want to carefully doublecheck database indexes. Also, have your
DBAs been maintaining the RT database? Have statistics been recomputed
lately?

I believe all install and upgrade recipes were followed properly. Though
we currently have a (presumably) unrelated problem with image attachments
getting corrupted (change to UTF-8 ??).

If you’re coming from 3.0, you’ll want to actually compare your database
schema with the current default as described in schema.Oracle.

We have a relatively small staff with no fulltime DBA. I am not sure what
maintenance is recommended; I recently ran “optimize table sessions”
recommended by the tuner which made no difference.

You need to do all tables. Not just sessions. Out of curiosity, how
many tickets and transactions do you have in your RT database.

I do not believe any statistics were computed; where is this documented ?
I found RT3BatchStats in the wiki which fails halfway with “Unknown
column ‘Transactions.Ticket’”

Per my original post, raw database queries seem to run quite fast. Unless
I am missing some complex query which the Web interface is making.

The web interface does a lot of complex queries. Testing one simple
query isn’t likely to give you any useful answer.

The performance numbers you’ve quoted for your RT instance seem fairly
out of whack, but I don’t know which problem might be causing them.

I believe all install and upgrade recipes were followed properly. Though
we currently have a (presumably) unrelated problem with image attachments
getting corrupted (change to UTF-8 ??).

Ah hah. When you followed the upgrade instructions, you probably ran
upgrade-mysql-schema.pl, which doesn’t actually upgrade the mysql schema.
It generates SQL queries to do so. You have to capture those queries to a
file, and then run that file to actually upgrade the schema.

After that, your image attachments should then be fine. Might also solve
your performance issue.

– ============================
Tom Lahti
BIT Statement LLC

(425)251-0833 x 117
http://www.bitstatement.net/
– ============================

The main ticket pages are served with Pragma:nocache and
Cache-control:no-cache. Also Connection:close which I believe thwarts
pipelining. (according to the “live HTTP headers” plugin)
Then your server doesn’t seem to be configured to enable it. Ten years
after HTTP/1.1 was written, many boxes still set KeepAlive off in Apache.

The only things RT does is set a 3-day max-age (and 30 day expires)
on static content, and send no last-modified in lib/RT/Interface/Web.pm

With pipelining and SSL content cache, HTTPfox seems to show a cut
in load time for refreshes from about 15 seconds to 10 seconds.

Cambridge Energy Alliance: Save money. Save the planet.

Hi All,

I am trying to figure out why External Authentication in my production
RT (version 3.6) works, but the test server I set up using (version
3.8.2) is not authenticating against the same ldap server? I followed
the steps in ExternalAuth - Request Tracker Wiki and yet it
is not working! Some relevant info on the test server (version 3.8.2) are:

Loaded perl modules

Perl v5.8.8 under linux
Net::LDAP v0.33;
Net::LDAP::ASN v0.03;
Net::LDAP::Constant v0.04;
Net::LDAP::Filter v0.14;
Net::LDAP::Message v1.08;
Net::LDAP::Util v0.10;

RT Config

Plugins RTx::Calendar, RT::Authen::ExternalAuth

Perl Include Paths (@INC)

/opt/rt3/bin/…/local/lib
/opt/rt3/local/plugins/RTx-Calendar/lib
/opt/rt3/local/plugins/RT-Authen-ExternalAuth/lib

Server is functioning fine and I can use local authentication, but no LDAP!

Any help/advise is greatly appreciated.

Hossein

_____ _____ _____ _ _ _ _ ____ Hossein Rafighi
|_ || _ \ | || | | || _/ || __|TRIUMF, 4004 Wesbrook Mall
| | | |
| ) | | | | | || || |__ Vancouver BC, Canada, V6T 2A3
| | | _ / | | | _/ || _/ || |Voice: (604) 222-1047
| | | | \ \ | | | || | | || | Fax: (604) 222-1074
|| || _|
_| _/ || |||_| Website: http://www.triumf.ca

Hi,

I have an user that has multiple email addresses and would like to send in requests from all of them. From example:

asmith@company.com
asmith@gmail.com

Currently I have AD working fine but don’t want to set up another Auth source. How can I do this?

Thanks

Michael Mai

Please ignore my previous email. I solved my problem.

Hossein

Hossein Rafighi wrote:

Hi All,

I am trying to figure out why External Authentication in my production
RT (version 3.6) works, but the test server I set up using (version
3.8.2) is not authenticating against the same ldap server? I followed
the steps in ExternalAuth - Request Tracker Wiki and yet it
is not working! Some relevant info on the test server (version 3.8.2) are:

Loaded perl modules

Perl v5.8.8 under linux
Net::LDAP v0.33;
Net::LDAP::ASN v0.03;
Net::LDAP::Constant v0.04;
Net::LDAP::Filter v0.14;
Net::LDAP::Message v1.08;
Net::LDAP::Util v0.10;

RT Config

Plugins RTx::Calendar, RT::Authen::ExternalAuth

Perl Include Paths (@INC)

/opt/rt3/bin/…/local/lib
/opt/rt3/local/plugins/RTx-Calendar/lib
/opt/rt3/local/plugins/RT-Authen-ExternalAuth/lib

Server is functioning fine and I can use local authentication, but no LDAP!

Any help/advise is greatly appreciated.

Hossein

_____ _____ _____ _ _ _ _ ____ Hossein Rafighi
|_ || _ \ | || | | || _/ || __|TRIUMF, 4004 Wesbrook Mall
| | | |
| ) | | | | | || || |__ Vancouver BC, Canada, V6T 2A3
| | | _ / | | | _/ || _/ || |Voice: (604) 222-1047
| | | | \ \ | | | || | | || | Fax: (604) 222-1074
|| || _|
_| _/ || |||_| Website: http://www.triumf.ca

Michael Mai wrote:

Hi,

I have an user that has multiple email addresses and would like to send in requests from all of them. From example:

asmith@company.com
asmith@gmail.com

If it is only one person that needs this functionality, simply merge the
two autocreated users (I believe this is included in 3.8.x - if not,
install RT::Extension::MergeUsers from cpan). This way, mail sent from
either account will result in the same requestor assigned.

Currently I have AD working fine but don’t want to set up another Auth source. How can I do this?

Thanks

Michael Mai


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

I’ve configured a DEV instance of RT 3.8.2 to test AD authentication but am getting the following in my rt.log…

[Thu Apr 23 19:37:58 2009] [error]: FAILED LOGIN for jeff from 10.x.x.x (/apps/rt-3.8.2-dev/share/html/autohandler:268)

I do not admin and therefore do not have access to monitor things on the AD side. Is there any way I can further debug the issue via log files, etc. on my RT server?

I know AD is working as I can query it using ldapsearch, however, I’m unsure if I’ve configured my “RT_SiteConfig.pm” correctly based on the working query.

My ldapsearch command uses the following flags…

-D “CN=RT User,OU=Eagle Access,DC=eagleinvsys,DC=com” -w -b “OU=Eagle Access,DC=eagleinvsys,DC=com”

I’m unsure what my “base” should be set to in RT_SiteConfig.pm since I’m using different settings for –D and –b.

Also, does a user have to exist in RT (and have credentials) before the user can be authenticated via AD?

Thanks.

The best way that I have found to do this is to run wireshark or
similar on the box itself and increase the debug output on the web
server.

The wireshark output though will give you all the LDAP details and
will show you what the server’s response is. It will also show you how
you are putting together your bind strings, which is usually what’s
wrong.

Also, take a look at the AD with ldp which is part of the support
tools package. You can use this to make sure that you have your DNs
correct.

Berny2009/4/23 Jeff Lucas jlucas@eagleinvsys.com:

I’ve configured a DEV instance of RT 3.8.2 to test AD authentication but am
getting the following in my rt.log…

[Thu Apr 23 19:37:58 2009] [error]: FAILED LOGIN for jeff from 10.x.x.x
(/apps/rt-3.8.2-dev/share/html/autohandler:268)

I do not admin and therefore do not have access to monitor things on the AD
side. Is there any way I can further debug the issue via log files, etc. on
my RT server?

I know AD is working as I can query it using ldapsearch, however, I’m unsure
if I’ve configured my “RT_SiteConfig.pm” correctly based on the working
query.

My ldapsearch command uses the following flags…

-D “CN=RT User,OU=Eagle Access,DC=eagleinvsys,DC=com” -w -b
“OU=Eagle Access,DC=eagleinvsys,DC=com”

I’m unsure what my “base” should be set to in RT_SiteConfig.pm since I’m
using different settings for –D and –b.

Also, does a user have to exist in RT (and have credentials) before the user
can be authenticated via AD?

Thanks.


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Thu 23 Apr 2009 21:26:12 GMT
Jeff Lucas wrote:

I’ve configured a DEV instance of RT 3.8.2 to test AD authentication but
am getting the following in my rt.log…

[Thu Apr 23 19:37:58 2009] [error]: FAILED LOGIN for jeff from 10.x.x.x
(/apps/rt-3.8.2-dev/share/html/autohandler:268)

You need to turn on debug logging, provide the debug log output, provide
the version ExternalAuth you’re using, and provide your Site_Config

I do not admin and therefore do not have access to monitor things on the
AD side. Is there any way I can further debug the issue via log files,
etc. on my RT server?

I never use the AD side for debugging, it can all be done from the RT
server.

I know AD is working as I can query it using ldapsearch, however, I’m
unsure if I’ve configured my “RT_SiteConfig.pm” correctly based on the
working query.

Which is why you need to provide it.

My ldapsearch command uses the following flags…

-D “CN=RT User,OU=Eagle Access,DC=eagleinvsys,DC=com” -w -b
“OU=Eagle Access,DC=eagleinvsys,DC=com”

I’m unsure what my “base” should be set to in RT_SiteConfig.pm since
I’m using different settings for –D and –b.

This is an LDAP understanding issue. You need to know a little more. The
above search says that your base is “OU=Eagle
Access,DC=eagleinvsys,DC=com” and you do not have anonymous bind enabled
on your AD server suggesting that you need to specify the user “CN=RT
User,OU=Eagle Access,DC=eagleinvsys,DC=com” and the password for that
user inside your external auth config.

Also, does a user have to exist in RT (and have credentials) before the
user can be authenticated via AD?

No. The user is automatically created inside RT when successful AD
authentication occurs.
Kind Regards,

Mike Peachey, IT
Tel: +44 114 281 2655
Fax: +44 114 281 2951
Jennic Ltd, Furnival Street, Sheffield, S1 4QT, UK
Comp Reg No: 3191371 - Registered In England

Has anyone configures RT to authenticate against MSSQL using SHA1 or SHA256?
If so, can someone post some sample configs?

Thanks in advance for the help.

View this message in context: http://requesttracker.8502.n7.nabble.com/External-Authentication-tp54436.html