RT4 slowness

[L_B]

Hi,

I’m testing RT4 and I have slowness problems. I tried these environments:

• RHEL6 64bits/ RedHat httpd-2.2.15 / RedHat perl 5.10 / RedHat
  mod_perl 2.0.4 / RT 4.0
• RHEL6 64bits/ RedHat httpd-2.2.15 / own compiled perl 5.14 / own
  compiled mod_perl 2.0.5 / RT 4.0
• RHEL6 64bits/ RedHat httpd-2.2.15 / own compiled perl 5.14 / own
  compiled mod_perl 2.0.5 / RT 4.1rc1
• RHEL6 64bits/ RedHat httpd-2.2.15 / own compiled perl 5.14 / own
  compiled mod_perl 2.0.6-devel / RT 4.1rc1

All perl dependencies have been installed using CPAN with the make
fixdeps. (in system perl directories in the first case, and my local
perl diretories in the others). My compiled Perl has roughly the same
options as the RedHat perl (ithread included).

All are slow. The machine is quite fast, I don’t think it’s the problem.

I tried with a local postgresql database and a remote one, no change.
Clean new empty database.

My Apache configuration follows the web-deployment.pod documentation,
basically it has:

Order allow,deny
Allow from all

SetHandler perl-script
PerlResponseHandler Plack::Handler::Apache2
PerlSetVar psgi_app /opt/rt4/sbin/rt-server

use Plack::Handler::Apache2; Plack::Handler::Apache2->preload("/opt/rt4-CSAHUT/sbin/rt-server");

Note: the section slows down a lot Apache service
restart but I guess it’s normal if it preloads rt-server

I added this to use my own perl: PerlSwitches
-I/opt/perl-5.14.0/lib/5.14.0/
-I/opt/perl-5.14.0/lib/site_perl/5.14.0/x86_64-linux/
-I/opt/perl-5.14.0/lib/site_perl/5.14.0/

I tried also
/opt/rt4/sbin/rt-server --port 8080
and
/opt/rt4/sbin/rt-server --server Starman --port 8080

Same slowness.

What I mean by slowness is that after sending the GET / request, it
takes 7 seconds before seeing the second request
/NoAuth/css/aileron-squished-30dddb4c81e92207bd3fe516099c0477.css. I
don’t have any network traffic during this 7secs on RT server network
interface, so I guess something is processed. Httpd process is running
at 100% at this time.

I created one ticket, and displaying it takes several seconds.
Browsing in RT is not fast. Even accessing the same pages is not that
fast (maybe a little bit faster than the first access, but not much).

I tried to turn on debugging in Apache and RT, but I see nothing wrong.

Any idea what I can try now ?

Many thanks,
L.B.

[Kevin_Falcone]

What I mean by slowness is that after sending the GET / request, it
takes 7 seconds before seeing the second request
/NoAuth/css/aileron-squished-30dddb4c81e92207bd3fe516099c0477.css. I
don’t have any network traffic during this 7secs on RT server network
interface, so I guess something is processed. Httpd process is running
at 100% at this time.

You should only see this for the first request while things are squished.
If you’re seeing it for subsequent requests on a standalone server,
then something is quite wrong. If you would like to use something
faster than JavaScript::Minifier, you can check out JSMinPath in
RT_Config.pm

-kevin

[Chris_Hall1]

Just for the record I’m having the same issues as L.B. I attempted loading
jsmin, and for a time it appeared that this was going to be a bigtime home
run. In fact, it did fix the issues I was having with speed! However,
jsmin seemed to also break several aspects of the RT interface. Immediately
I noticed the javascript dropdowns were gone… no big deal, switched to the
ballard interface as the default. However, opening a new ticket…
everything was jumbled around… dropdowns were permanently opened, the
minimize links for the different zones wouldn’t work anymore… it was
generally a bad day. So I’ve turned it back off. This does however shed
some light on what may be the cause of… at least… my speed issues.

The search continues…On Tue, May 31, 2011 at 10:34 AM, Kevin Falcone falcone@bestpractical.comwrote:

On Tue, May 31, 2011 at 03:21:17PM +0200, L B wrote:

What I mean by slowness is that after sending the GET / request, it
takes 7 seconds before seeing the second request
/NoAuth/css/aileron-squished-30dddb4c81e92207bd3fe516099c0477.css. I
don’t have any network traffic during this 7secs on RT server network
interface, so I guess something is processed. Httpd process is running
at 100% at this time.

You should only see this for the first request while things are squished.
If you’re seeing it for subsequent requests on a standalone server,
then something is quite wrong. If you would like to use something
faster than JavaScript::Minifier, you can check out JSMinPath in
RT_Config.pm

-kevin

[Kevin_Falcone]

Just for the record I’m having the same issues as L.B. I attempted loading jsmin, and for a
time it appeared that this was going to be a bigtime home run. In fact, it did fix the issues
I was having with speed! However, jsmin seemed to also break several aspects of the RT
interface. Immediately I noticed the javascript dropdowns were gone… no big deal, switched to
the ballard interface as the default. However, opening a new ticket… everything was jumbled
around… dropdowns were permanently opened, the minimize links for the different zones
wouldn’t work anymore… it was generally a bad day. So I’ve turned it back off. This does
however shed some light on what may be the cause of… at least… my speed issues.
The search continues…

RT only builds the squished css/js once per child process. If you’re
actually seeing it built on each request, something is horribly wrong
and isn’t something we’ve seen or been able to replicate.

Also, if jsmin is breaking the UI, that’s the first report I’ve seen
of it. Can you file a bug report with more details?

-kevin> On Tue, May 31, 2011 at 10:34 AM, Kevin Falcone <[1]falcone@bestpractical.com> wrote:

 On Tue, May 31, 2011 at 03:21:17PM +0200, L B wrote:
 > What I mean by slowness is that after sending the GET / request, it
 > takes 7 seconds before seeing the second request
 > /NoAuth/css/aileron-squished-30dddb4c81e92207bd3fe516099c0477.css. I
 > don't have any network traffic during this 7secs on RT server network
 > interface, so I guess something is processed. Httpd process is running
 > at 100% at this time.

 You should only see this for the first request while things are squished.
 If you're seeing it for subsequent requests on a standalone server,
 then something is quite wrong. If you would like to use something
 faster than JavaScript::Minifier, you can check out JSMinPath in
 RT_Config.pm
 -kevin

[Chris_Hall1]

Could you point me in the direction of where to file the bug report?

I’ve just replicated this on another box I have set up as a test server… so
one was Debian, the other Ubuntu. In both cases, I downloaded jsmin from

http://www.crockford.com/javascript/jsmin.c

ran:

gcc -o jsmin jsmin.c

and copied the binary over to /usr/bin, and chmod +x /usr/bin/jsmin

I then added the following line to RT_SiteConfig.pm

Set( $JSMinPath, “/usr/bin/jsmin”);

which resulted in no errors generated, BUT breaks the interface in the
following way: (note the absence of menu at the top, and the always open
dropdowns)

On firefox you get the bonus of getting Links and Basics stacked together as
well

http://i44.photobucket.com/albums/f45/nodaitsu25/jsmin2.pngOn Tue, May 31, 2011 at 12:59 PM, Kevin Falcone falcone@bestpractical.comwrote:

On Tue, May 31, 2011 at 12:56:46PM -0400, Chris Hall wrote:

Just for the record I’m having the same issues as L.B. I attempted
loading jsmin, and for a
time it appeared that this was going to be a bigtime home run. In
fact, it did fix the issues
I was having with speed! However, jsmin seemed to also break several
aspects of the RT
interface. Immediately I noticed the javascript dropdowns were gone…
no big deal, switched to
the ballard interface as the default. However, opening a new ticket…
everything was jumbled
around… dropdowns were permanently opened, the minimize links for
the different zones
wouldn’t work anymore… it was generally a bad day. So I’ve turned
it back off. This does
however shed some light on what may be the cause of… at least… my
speed issues.
The search continues…

RT only builds the squished css/js once per child process. If you’re
actually seeing it built on each request, something is horribly wrong
and isn’t something we’ve seen or been able to replicate.

Also, if jsmin is breaking the UI, that’s the first report I’ve seen
of it. Can you file a bug report with more details?

-kevin

On Tue, May 31, 2011 at 10:34 AM, Kevin Falcone <[1] falcone@bestpractical.com> wrote:

 On Tue, May 31, 2011 at 03:21:17PM +0200, L B wrote:
 > What I mean by slowness is that after sending the GET / request,

it

 > takes 7 seconds before seeing the second request
 > /NoAuth/css/aileron-squished-30dddb4c81e92207bd3fe516099c0477.css.

I

 > don't have any network traffic during this 7secs on RT server

network

 > interface, so I guess something is processed. Httpd process is

running

 > at 100% at this time.

 You should only see this for the first request while things are

squished.

 If you're seeing it for subsequent requests on a standalone server,
 then something is quite wrong. If you would like to use something
 faster than JavaScript::Minifier, you can check out JSMinPath in
 RT_Config.pm
 -kevin

[Kevin_Falcone]

Could you point me in the direction of where to file the bug report?

http://www.bestpractical.com/rt/issues.html has instructions for
filing bug reports.

-kevin

[L_B]

I tried jsmin and I have the same bug as Chris with the javascript
menubar, plus some weird boxes display.

Anyway, browsing is still slow when I go in a ticket from a page to
another (such as “Basics” or “Peoples”). Either there is some js
minified each time I access a page (how can I check that?) or this
problem has nothing to do with js minifiers.

L.B.

[L_B]

This amazing patch fixed the issue, so it has something to do with the minifier:

— lib/RT/Squish/JS.pm.orig 2011-06-01 07:50:27.510862133 +0000
+++ lib/RT/Squish/JS.pm 2011-06-01 07:45:55.096115642 +0000
@@ -110,7 +110,7 @@
$RT::Logger->debug(“can’t load JavaScript::Minifier: $@”);
}
else {

•        $content = JavaScript::Minifier::minify( input => $content );
  

•        #$content = JavaScript::Minifier::minify( input => $content );
     }
  

  }
  return $content;

L.B.

[Robert_Wysocki]

Dnia 2011-05-31, wto o godzinie 12:59 -0400, Kevin Falcone pisze:> On Tue, May 31, 2011 at 12:56:46PM -0400, Chris Hall wrote:

Just for the record I’m having the same issues as L.B. I attempted loading jsmin, and for a
time it appeared that this was going to be a bigtime home run. In fact, it did fix the issues
I was having with speed! However, jsmin seemed to also break several aspects of the RT
interface. Immediately I noticed the javascript dropdowns were gone… no big deal, switched to
the ballard interface as the default. However, opening a new ticket… everything was jumbled
around… dropdowns were permanently opened, the minimize links for the different zones
wouldn’t work anymore… it was generally a bad day. So I’ve turned it back off. This does
however shed some light on what may be the cause of… at least… my speed issues.
The search continues…

RT only builds the squished css/js once per child process. If you’re
actually seeing it built on each request, something is horribly wrong
and isn’t something we’ve seen or been able to replicate.

The same for me - 7 second delay befor loading any page. The obvious fix
for it - after reading this thread, was to remove references to
JavaScript::Minifier in lib/RT/Squish/JS.pm.
Now the delay isn’t greater than 4 seconds and since we’re using RT
almost exclusively on LAN it’s OK for us.

Robert Wysocki
CONTIUM S.A., http://www.contium.pl

[L_B]

This amazing patch fixed the issue, so it has something to do with the minifier:

— lib/RT/Squish/JS.pm.orig 2011-06-01 07:50:27.510862133 +0000
+++ lib/RT/Squish/JS.pm 2011-06-01 07:45:55.096115642 +0000
@@ -110,7 +110,7 @@
$RT::Logger->debug(“can’t load JavaScript::Minifier: $@”);
}
else {

•        $content = JavaScript::Minifier::minify( input => $content );
  

•        #$content = JavaScript::Minifier::minify( input => $content );
    }
  

  }
  return $content;

–
L.B.

But it doesn’t solve the slowness for Display.html page. Displaying a
ticket still need the 7seconds.

L.B.

[Kevin_Falcone]

This amazing patch fixed the issue, so it has something to do with the minifier:

— lib/RT/Squish/JS.pm.orig 2011-06-01 07:50:27.510862133 +0000
+++ lib/RT/Squish/JS.pm 2011-06-01 07:45:55.096115642 +0000
@@ -110,7 +110,7 @@
$RT::Logger->debug(“can’t load JavaScript::Minifier: $@”);
}
else {

•        $content = JavaScript::Minifier::minify( input => $content );
  

•        #$content = JavaScript::Minifier::minify( input => $content );
    }
  

  }
  return $content;

–
L.B.

But it doesn’t solve the slowness for Display.html page. Displaying a
ticket still need the 7seconds.

All this patch will do is solve the once per child slowness of
squishing the JS by reverting to simple concatenation. Children
should be answering multiple requests before being reaped and needing
to resquish.

If you’re seeing this JS related slowness on every page, it points to
other problems.

Please provide details about your installation such as your apache
versions and configuration otherwise it’s quite difficult to
test. We’ve tried replicating in house, but haven’t seen this in any
of our test or production RT4 instances.

If you’re seeing particular slowness on a given page, RT4 contains a
new feature to allow you (as an admin) to see if there are particular
SQL statements causing problems. Because it is debugging, it will
cause other slowness in RT and should not be left on, but StatementLog
will expose queries in the admin UI and allow you to see if something
needs indexing or optimization.

-kevin

[L_B]

Hi Kevin,

But it doesn’t solve the slowness for Display.html page. Displaying a
ticket still need the 7seconds.

All this patch will do is solve the once per child slowness of
squishing the JS by reverting to simple concatenation. Children
should be answering multiple requests before being reaped and needing
to resquish.

If you’re seeing this JS related slowness on every page, it points to
other problems.

I checked some pages and I have these 7 seconds on at least:

• Login page
• Search/Simple.html
• Search/Build.html
• Tools/MyDay.html
• Admin/Users
• Admin/Tools/Configuration.html

Applying the patch above remove the slowness on all the pages except
the Display.html (showing ticket #1 with only one correspondance
inside, no attachments)

Please provide details about your installation such as your apache
versions and configuration otherwise it’s quite difficult to
test. We’ve tried replicating in house, but haven’t seen this in any
of our test or production RT4 instances.

Apache 2.2.15, first email of the thread has all the details.

If you’re seeing particular slowness on a given page, RT4 contains a
new feature to allow you (as an admin) to see if there are particular
SQL statements causing problems. Because it is debugging, it will
cause other slowness in RT and should not be left on, but StatementLog
will expose queries in the admin UI and allow you to see if something
needs indexing or optimization.

We monitored at the database level the SQL statements and we didn’t
find anything wrong.

L.B.

[Kevin_Falcone]

I checked some pages and I have these 7 seconds on at least:

• Login page
• Search/Simple.html
• Search/Build.html
• Tools/MyDay.html
• Admin/Users
• Admin/Tools/Configuration.html

Applying the patch above remove the slowness on all the pages except
the Display.html (showing ticket #1 with only one correspondance
inside, no attachments)

This implies you’re requishing JS on every page, which points to a
misconfiguration.

Please provide details about your installation such as your apache
versions and configuration otherwise it’s quite difficult to
test. We’ve tried replicating in house, but haven’t seen this in any
of our test or production RT4 instances.

Apache 2.2.15, first email of the thread has all the details.

Your first email had differing paths in your preload and psgi_app
lines, so I assumed it was a copy and paste error, so was hoping to
get a clean full copy of your config.

Other things that would be useful are your sanitized RT_SiteConfig or
the System Configuration from the Admin UI.

-kevin

[L_B]

Your first email had differing paths in your preload and psgi_app
lines, so I assumed it was a copy and paste error, so was hoping to
get a clean full copy of your config.

Sorry, copy/paste error. Paths are the same.

Other things that would be useful are your sanitized RT_SiteConfig or
the System Configuration from the Admin UI.

RT_SiteConfig.pm :

Set($rtname, ‘XXXXX’);
Set($WebDomain, ‘10.0.xx.xx’);
Set($WebBaseURL, “http://10.0.xx.xx”);
Set($LogDir, “/opt/rt4/var/log”);

Set($LogToFile , ‘debug’);
Set($LogToScreen , ‘debug’);
Set($LogToSyslog , ‘debug’);

Added when testing a remote database

Set($DatabaseHost , “xxxx”);
Set($DatabasePort , 5432);
Set($DatabaseName , ‘xxxxx’);
Set($DatabaseUser , “xxxxx”);
Set($DatabasePassword , ‘xxxxx’);

Added because debug mode complained

Set($RTAddressRegexp, ‘XXXX@XXX.COM’);
Set(%GnuPG,
Enable => 0,
OutgoingMessagesFormat => “RFC”, # Inline
AllowEncryptDataInDB => 0,

RejectOnMissingPrivateKey => 1,
RejectOnBadData           => 1,

);

1;

Attached a sanitized httpd.conf: it’s the the default redhat
httpd.conf without the comments, and with my configuration at the end
(overriding previous variables)

L.B.

httpd.conf (8.5 KB)

[Kevin_Falcone]

Your first email had differing paths in your preload and psgi_app
lines, so I assumed it was a copy and paste error, so was hoping to
get a clean full copy of your config.

Sorry, copy/paste error. Paths are the same.

Other things that would be useful are your sanitized RT_SiteConfig or
the System Configuration from the Admin UI.

You seem to be loading a lot of caching modules, are any of them enabled in
Include conf.d/*.conf ?

-kevin

[L_B]

Hi Kevin,

You seem to be loading a lot of caching modules, are any of them enabled in
Include conf.d/*.conf ?

Only one (default config):

ls /etc/httpd/conf.d/

README welcome.conf

cat /etc/httpd/conf.d/welcome.conf

This configuration file enables the default “Welcome”

page if there is no default index page present for

the root URL. To disable the Welcome page, comment

out all the lines below.

<LocationMatch “^/+$”>
Options -Indexes
ErrorDocument 403 /error/noindex.html

L.B.