Intermittent slow logins on RT 3.6.3

RT Users
1

Hi everyone,

We’re running RT 3.6.3 on PostgreSQL 8.0.12 and while we’ve always had
some intermittent performance issues, it has gotten especially bad in
the past few weeks. There are a number of actions that are very slow
for us, but particularly bad is the amount of time it takes to log in.
It just took me 136 seconds to log in and display the Home page.
However, if I refresh the Home page while logged in, it “only” takes
about 18 seconds.

Here’s a mini time line to illustrate a typical login:

0. Enter username and password and click submit

1. Immediately afterwards, I see the following in rt.log:
   [Mon Jun 30 21:11:53 2008] [info]: Successful login for allen
from ... (/var/www/localhost/htdocs/rt/autohandler:256)

2. 95 seconds later, the login page disappears and the home page
starts to display

3. At 136 seconds, the home page is complete

I have set logging in RT to ‘debug’ so I can see the timing on all the
queries being executed. The slowest query generated during that last
login attempt took 2.50 seconds. Watching top and iostat on the
Apache+mod_perl running RT shows a load average around 0.01, CPU
utilization between 0 and 5%, and disk I/O at <1%. Watching top and
iostat on the DB server also shows that it is nowhere near “maxed out.”

This problem is intermittent (sometimes it will only take an acceptable
20 seconds to log in); however, it seems to be slow more often than
not. There also seems to be a lot of variance in how slow it gets.
Sometimes it takes 40 seconds, sometimes over two minutes.

It seems like RT is doing something after authentication and before
building the home page that makes logins slow. I have a feeling it
might be database related, but I cannot find any evidence of any slow
queries in rt.log. Could there be any queries that RT executes that
would not show up in rt.log for some reason? Also, I have checked
pg_locks when this is happening but do not see any exclusive locks that
would cause any queries to wait. (Note that I /have/ found slow RT
queries, but not ones associated with logging in.)

Are there any bugs in 3.6.3 that would cause this? Can anyone suggest
anything else I should look at or try?

Thanks much,
-Allen

2

Hi everyone,

Hi,

It seems like RT is doing something after authentication and before
building the home page that makes logins slow. I have a feeling it

RT builds a cache of all allowed queues (that makes a bunch of acl
queries) that will be displayed everywhere in RT (QuickCreate,
QuickSearch, …). This cache is kept in sessions. So if you log in
again with the same browser and computer, it doesn’t need to rebuild the
cache and it’s “fast”. That’s why we have this typical difference
between first and second login.

might be database related, but I cannot find any evidence of any slow
queries in rt.log. Could there be any queries that RT executes that

You would better look in the DB log and activate slow queries login.

Also keep in mind that RT does a lot of small queries, so 0.5s can be
really slow for an RT ACL query because you will have hundred of this if
you have several queues.

Are there any bugs in 3.6.3 that would cause this? Can anyone suggest
anything else I should look at or try?

If you’re problem is related to a high number of queues, you can enhance
a bit by upgrading RT or at least grabing the
share/html/Elements/Quicksearch from 3.6.5 which have a performance
optimisation.
You can also remove the Quickcreate at the bottom of RT at a glance if
you don’t need it (Createticket in the header is often enough) and save
more queries.