Any page showing a user drop down in a ticket is loading slow

Hello,

I just updated from 4.0.10 to 4.2.1 and it seems to be working well
except when I am in a ticket, any page that shows a user drop down, ie
display and people load incredible slowly. I enabled slow query logging
and below are the two queries that appear to be causing the problem. I
appreciate any help in resolving this. 10 second load times is rough.
Please let me know what further information is needed.

Thanks!
//adam

Time: 131217 14:42:38

User@Host: rt_user[rt_user] @ ticket.cs.pitt.edu [136.142.23.83]

Query_time: 9 Lock_time: 0 Rows_sent: 5 Rows_examined: 175403

SELECT DISTINCT main.* FROM Users main CROSS JOIN ACL ACL_3 JOIN
Principals Principals_1 ON ( Principals_1.id = main.id ) JOIN
CachedGroupMembers CachedGroupMembers_2 ON (
CachedGroupMembers_2.MemberId = Principals_1.id ) JOIN
CachedGroupMembers CachedGroupMembers_4 ON (
CachedGroupMembers_4.MemberId = Principals_1.id ) WHERE
((ACL_3.ObjectType = ‘RT::Queue’ AND ACL_3.ObjectId = 3) OR
(ACL_3.ObjectType = ‘RT::System’) OR (ACL_3.ObjectType = ‘RT::Queue’ AND
ACL_3.ObjectId = 3) OR (ACL_3.ObjectType = ‘RT::System’)) AND
(ACL_3.PrincipalId = CachedGroupMembers_4.GroupId) AND
(ACL_3.PrincipalType = ‘Group’) AND (ACL_3.RightName = ‘OwnTicket’ OR
ACL_3.RightName = ‘SuperUser’) AND (CachedGroupMembers_2.Disabled = ‘0’)
AND (CachedGroupMembers_2.GroupId = ‘4’) AND
(CachedGroupMembers_4.Disabled = ‘0’) AND (Principals_1.Disabled = ‘0’)
AND (Principals_1.PrincipalType = ‘User’) AND (Principals_1.id != ‘1’)
ORDER BY main.Name ASC;

Time: 131217 14:42:48

User@Host: rt_user[rt_user] @ ticket.cs.pitt.edu [136.142.23.83]

Query_time: 8 Lock_time: 0 Rows_sent: 5 Rows_examined: 175403

SELECT DISTINCT main.* FROM Users main CROSS JOIN ACL ACL_3 JOIN
Principals Principals_1 ON ( Principals_1.id = main.id ) JOIN
CachedGroupMembers CachedGroupMembers_2 ON (
CachedGroupMembers_2.MemberId = Principals_1.id ) JOIN
CachedGroupMembers CachedGroupMembers_4 ON (
CachedGroupMembers_4.MemberId = Principals_1.id ) WHERE
((ACL_3.ObjectType = ‘RT::Ticket’ AND ACL_3.ObjectId = 35402) OR
(ACL_3.ObjectType = ‘RT::Queue’ AND ACL_3.ObjectId = 3) OR
(ACL_3.ObjectType = ‘RT::System’) OR (ACL_3.ObjectType = ‘RT::Ticket’
AND ACL_3.ObjectId = 35402) OR (ACL_3.ObjectType = ‘RT::Queue’ AND
ACL_3.ObjectId = 3) OR (ACL_3.ObjectType = ‘RT::System’)) AND
(ACL_3.PrincipalId = CachedGroupMembers_4.GroupId) AND
(ACL_3.PrincipalType = ‘Group’) AND (ACL_3.RightName = ‘OwnTicket’ OR
ACL_3.RightName = ‘SuperUser’) AND (CachedGroupMembers_2.Disabled = ‘0’)
AND (CachedGroupMembers_2.GroupId = ‘4’) AND
(CachedGroupMembers_4.Disabled = ‘0’) AND (Principals_1.Disabled = ‘0’)
AND (Principals_1.PrincipalType = ‘User’) AND (Principals_1.id != ‘1’)
ORDER BY main.Name ASC;

Hello,

I just updated from 4.0.10 to 4.2.1 and it seems to be working well except
when I am in a ticket, any page that shows a user drop down, ie display and
people load incredible slowly. I enabled slow query logging and below are
the two queries that appear to be causing the problem. I appreciate any
help in resolving this. 10 second load times is rough. Please let me know
what further information is needed.

Hi Adam,

I’m not sure if this applies but we had this issue and it was an RT ACL
issue. I remember that it was an issue with granting or not-granting
rights to special groups “Everyone” or “Unprivileged” but the details of
which I cannot recall exactly. I do remember there were large ACL joins in
the database whenever we would try to load pages with these drop downs just
like you’ve described in your post.

Landon Stewart :: lstewart@iweb.com
Lead Specialist, Abuse and Security Management
Spécialiste principal, gestion des abus et sécurité
http://iweb.com :: +1 (888) 909-4932

Hi Adam,

I’m not sure if this applies but we had this issue and it was an RT ACL
issue. I remember that it was an issue with granting or not-granting
rights to special groups “Everyone” or “Unprivileged” but the details of
which I cannot recall exactly. I do remember there were large ACL joins in
the database whenever we would try to load pages with these drop downs just
like you’ve described in your post.

Hello Landon,

I removed the rights from Everyone and we had nothing on Unprivileged and
I still see the issue. Thanks for the suggestion though.

Maybe you need to go the other way with it and grant something? Maybe you
need to grant some simple rights for showing users or something to
Privileged. I just checked ours and I have “Privileged” under “General
Rights” allowed to "View custom field values” configured in each queue. In
the Global → Group Rights the "View custom fields” is not granted.

I wish I could remember the details of what solved it for us. I do
remember some things were changed which caused the issue and then had to be
rolled back. I just don’t recall what was changed that caused it.

Landon Stewart :: lstewart@iweb.com
Lead Specialist, Abuse and Security Management
Spécialiste principal, gestion des abus et sécurité
http://iweb.com :: +1 (888) 909-4932

Hi Adam,

I’m not sure if this applies but we had this issue and it was an RT ACL
issue. I remember that it was an issue with granting or not-granting
rights to special groups “Everyone” or “Unprivileged” but the details of
which I cannot recall exactly. I do remember there were large ACL joins in
the database whenever we would try to load pages with these drop downs just
like you’ve described in your post.

Hello Landon,

I removed the rights from Everyone and we had nothing on Unprivileged and
I still see the issue. Thanks for the suggestion though.

Maybe you need to go the other way with it and grant something? Maybe you
need to grant some simple rights for showing users or something to
Privileged. I just checked ours and I have “Privileged” under “General
Rights” allowed to "View custom field values” configured in each queue. In
the Global → Group Rights the "View custom fields” is not granted.

I wish I could remember the details of what solved it for us. I do
remember some things were changed which caused the issue and then had to be
rolled back. I just don’t recall what was changed that caused it.

Landon Stewart :: lstewart@iweb.com
Lead Specialist, Abuse and Security Management
Spécialiste principal, gestion des abus et sécurité
http://iweb.com :: +1 (888) 909-4932

I just updated from 4.0.10 to 4.2.1 and it seems to be working well
except when I am in a ticket, any page that shows a user drop down,
ie display and people load incredible slowly. I enabled slow query
logging and below are the two queries that appear to be causing the
problem. I appreciate any help in resolving this. 10 second load
times is rough. Please let me know what further information is
needed.

The slow query logs are most helpful when you then run EXPLAIN on the
query so you know why the database is having trouble.

Since your list is short (5 rows sent) rather than 500, you’re
unlikely to have the common error someone else was pointing you at
where you grant OwnTicket too widely. RT 4.2 also has a workaround
for this case.

Do you grant your OwnTicket rights out to groups per queue, roles per
queue. Do you assign lots of Ticket level use groups (adminccs/ccs,
etc) which then pick up OwnTicket?

-kevin

I just updated from 4.0.10 to 4.2.1 and it seems to be working well
except when I am in a ticket, any page that shows a user drop down,
ie display and people load incredible slowly. I enabled slow query
logging and below are the two queries that appear to be causing the
problem. I appreciate any help in resolving this. 10 second load
times is rough. Please let me know what further information is
needed.
The slow query logs are most helpful when you then run EXPLAIN on the
query so you know why the database is having trouble.
Here is the EXPLAIN for the two queries.

mysql> EXPLAIN SELECT DISTINCT main.* FROM Users main CROSS JOIN ACL
ACL_3 JOIN Principals Principals_1 ON ( Principals_1.id = main.id )
JOIN CachedGroupMembers CachedGroupMembers_2 ON (
CachedGroupMembers_2.MemberId = Principals_1.id ) JOIN
CachedGroupMembers CachedGroupMembers_4 ON (
CachedGroupMembers_4.MemberId = Principals_1.id ) WHERE
((ACL_3.ObjectType = ‘RT::Queue’ AND ACL_3.ObjectId = 3) OR
(ACL_3.ObjectType = ‘RT::System’) OR (ACL_3.ObjectType = ‘RT::Queue’ AND
ACL_3.ObjectId = 3) OR (ACL_3.ObjectType = ‘RT::System’)) AND
(ACL_3.PrincipalId = CachedGroupMembers_4.GroupId) AND
(ACL_3.PrincipalType = ‘Group’) AND (ACL_3.RightName = ‘OwnTicket’ OR
ACL_3.RightName = ‘SuperUser’) AND (CachedGroupMembers_2.Disabled = ‘0’)
AND (CachedGroupMembers_2.GroupId = ‘4’) AND
(CachedGroupMembers_4.Disabled = ‘0’) AND (Principals_1.Disabled = ‘0’)
AND (Principals_1.PrincipalType = ‘User’) AND (Principals_1.id != ‘1’)
ORDER BY main.Name ASC;
| id | select_type | table | type |
possible_keys | key
| key_len | ref | rows |
Extra |
| 1 | SIMPLE | CachedGroupMembers_2 | range |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 | DisGrouMem
| 10 | NULL | 264 | Using where;
Using index; Using temporary; Using filesort |
| 1 | SIMPLE | main | eq_ref |
PRIMARY | PRIMARY
| 4 | rt4_2_1.CachedGroupMembers_2.MemberId | 1
| |
| 1 | SIMPLE | Principals_1 | eq_ref |
PRIMARY | PRIMARY
| 4 | rt4_2_1.main.id | 1 | Using where;
Distinct |
| 1 | SIMPLE | CachedGroupMembers_4 | ref |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 | CachedGroupMembers3
| 5 | rt4_2_1.CachedGroupMembers_2.MemberId | 1 | Using where;
Distinct |
| 1 | SIMPLE | ACL_3 | range |
ACL1 | ACL1
| 85 | NULL | 13 | Using where;
Using index; Distinct |
5 rows in set (0.01 sec)

mysql> EXPLAIN SELECT DISTINCT main.* FROM Users main CROSS JOIN ACL
ACL_3 JOIN Principals Principals_1 ON ( Principals_1.id = main.id )
JOIN CachedGroupMembers CachedGroupMembers_2 ON (
CachedGroupMembers_2.MemberId = Principals_1.id ) JOIN
CachedGroupMembers CachedGroupMembers_4 ON (
CachedGroupMembers_4.MemberId = Principals_1.id ) WHERE
((ACL_3.ObjectType = ‘RT::Ticket’ AND ACL_3.ObjectId = 35402) OR
(ACL_3.ObjectType = ‘RT::Queue’ AND ACL_3.ObjectId = 3) OR
(ACL_3.ObjectType = ‘RT::System’) OR (ACL_3.ObjectType = ‘RT::Ticket’
AND ACL_3.ObjectId = 35402) OR (ACL_3.ObjectType = ‘RT::Queue’ AND
ACL_3.ObjectId = 3) OR (ACL_3.ObjectType = ‘RT::System’)) AND
(ACL_3.PrincipalId = CachedGroupMembers_4.GroupId) AND
(ACL_3.PrincipalType = ‘Group’) AND (ACL_3.RightName = ‘OwnTicket’ OR
ACL_3.RightName = ‘SuperUser’) AND (CachedGroupMembers_2.Disabled = ‘0’)
AND (CachedGroupMembers_2.GroupId = ‘4’) AND
(CachedGroupMembers_4.Disabled = ‘0’) AND (Principals_1.Disabled = ‘0’)
AND (Principals_1.PrincipalType = ‘User’) AND (Principals_1.id != ‘1’)
ORDER BY main.Name ASC;
| id | select_type | table | type |
possible_keys | key
| key_len | ref | rows |
Extra |
| 1 | SIMPLE | CachedGroupMembers_2 | range |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 | DisGrouMem
| 10 | NULL | 264 | Using where;
Using index; Using temporary; Using filesort |
| 1 | SIMPLE | main | eq_ref |
PRIMARY | PRIMARY
| 4 | rt4_2_1.CachedGroupMembers_2.MemberId | 1
| |
| 1 | SIMPLE | Principals_1 | eq_ref |
PRIMARY | PRIMARY
| 4 | rt4_2_1.main.id | 1 | Using where;
Distinct |
| 1 | SIMPLE | CachedGroupMembers_4 | ref |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 | CachedGroupMembers3
| 5 | rt4_2_1.CachedGroupMembers_2.MemberId | 1 | Using where;
Distinct |
| 1 | SIMPLE | ACL_3 | range |
ACL1 | ACL1
| 85 | NULL | 15 | Using where;
Using index; Distinct |
5 rows in set (0.00 sec)

Since your list is short (5 rows sent) rather than 500, you’re
unlikely to have the common error someone else was pointing you at
where you grant OwnTicket too widely. RT 4.2 also has a workaround
for this case.

Do you grant your OwnTicket rights out to groups per queue, roles per
queue. Do you assign lots of Ticket level use groups (adminccs/ccs,
etc) which then pick up OwnTicket?
We have our users who can own tickets in groups and then assign those
groups to the queue and give them OwnTicket. We have at max 3 groups per
queue. We don’t assign permissions on any role. We also have some
individuals added but those are there so that managers only watch the
tickets but can’t own them.

Thanks!
//adam

The slow query logs are most helpful when you then run EXPLAIN on the
query so you know why the database is having trouble.
Here is the EXPLAIN for the two queries.

mysql> EXPLAIN SELECT DISTINCT main.* FROM Users main CROSS JOIN ACL
ACL_3 JOIN Principals Principals_1 ON ( Principals_1.id = main.id )
JOIN CachedGroupMembers CachedGroupMembers_2 ON (
CachedGroupMembers_2.MemberId = Principals_1.id ) JOIN
CachedGroupMembers CachedGroupMembers_4 ON (
CachedGroupMembers_4.MemberId = Principals_1.id ) WHERE
((ACL_3.ObjectType = ‘RT::Queue’ AND ACL_3.ObjectId = 3) OR
(ACL_3.ObjectType = ‘RT::System’) OR (ACL_3.ObjectType = ‘RT::Queue’
AND ACL_3.ObjectId = 3) OR (ACL_3.ObjectType = ‘RT::System’)) AND
(ACL_3.PrincipalId = CachedGroupMembers_4.GroupId) AND
(ACL_3.PrincipalType = ‘Group’) AND (ACL_3.RightName = ‘OwnTicket’
OR ACL_3.RightName = ‘SuperUser’) AND (CachedGroupMembers_2.Disabled
= ‘0’) AND (CachedGroupMembers_2.GroupId = ‘4’) AND
(CachedGroupMembers_4.Disabled = ‘0’) AND (Principals_1.Disabled =
‘0’) AND (Principals_1.PrincipalType = ‘User’) AND (Principals_1.id
!= ‘1’) ORDER BY main.Name ASC;
±—±------------±---------------------±-------±---------------------------------------------------±--------------------±--------±--------------------------------------±-----±----------------------------------------------------------+
| id | select_type | table | type | possible_keys
| key | key_len | ref
| rows | Extra |
±—±------------±---------------------±-------±---------------------------------------------------±--------------------±--------±--------------------------------------±-----±----------------------------------------------------------+
| 1 | SIMPLE | CachedGroupMembers_2 | range |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 | DisGrouMem
| 10 | NULL | 264 | Using
where; Using index; Using temporary; Using filesort |
| 1 | SIMPLE | main | eq_ref | PRIMARY
| PRIMARY | 4 |
rt4_2_1.CachedGroupMembers_2.MemberId | 1 |
|
| 1 | SIMPLE | Principals_1 | eq_ref | PRIMARY
| PRIMARY | 4 | rt4_2_1.main.id
| 1 | Using where; Distinct |
| 1 | SIMPLE | CachedGroupMembers_4 | ref |
DisGrouMem,CachedGroupMembers3,cachedgroupmembers1 |
CachedGroupMembers3 | 5 |
rt4_2_1.CachedGroupMembers_2.MemberId | 1 | Using where; Distinct
|
| 1 | SIMPLE | ACL_3 | range | ACL1
| ACL1 | 85 | NULL
| 13 | Using where; Using index; Distinct |
±—±------------±---------------------±-------±---------------------------------------------------±--------------------±--------±--------------------------------------±-----±----------------------------------------------------------+
5 rows in set (0.01 sec)

Your query here says that MySQL is looking at a pretty tiny dataset
and running the same query on multiple databases here shows a wildly
different explain and very quick results. This usually points to you
having some enormous tables (although the explain doesn’t indicate
that) or a badly tuned mysql.

Can you download and run http://mysqltuner.pl on your database.

It would also be interesting to see
SELECT TABLE_ROWS, TABLE_NAME
FROM information_schema.tables
WHERE TABLE_SCHEMA = ‘rt4’;
changing your database name where needed.

-kevin

Can you download and run http://mysqltuner.pl on your database.

Fun tip. You can download that with just “wget mysqltuner.pl”. There’s
some redirection magic that makes it work with just that command.

Landon Stewart :: lstewart@iweb.com
Lead Specialist, Abuse and Security Management
Spécialiste principal, gestion des abus et sécurité
http://iweb.com :: +1 (888) 909-4932

Your query here says that MySQL is looking at a pretty tiny dataset
and running the same query on multiple databases here shows a wildly
different explain and very quick results. This usually points to you
having some enormous tables (although the explain doesn’t indicate
that) or a badly tuned mysql.

Can you download and run http://mysqltuner.pl on your database.
I will have to go through this script and consult with a colleague
before running that as our database contains more than just our ticket
system.
It would also be interesting to see
SELECT TABLE_ROWS, TABLE_NAME
FROM information_schema.tables
WHERE TABLE_SCHEMA = ‘rt4’;
changing your database name where needed.
mysql> SELECT TABLE_ROWS, TABLE_NAME
→ FROM information_schema.tables
→ WHERE TABLE_SCHEMA = ‘rt4_2_1’;
| TABLE_ROWS | TABLE_NAME |
| 579 | ACL |
| 0 | Articles |
| 1436268 | Attachments |
| 5708 | Attributes |
| 353565 | CachedGroupMembers |
| 0 | Classes |
| 36 | CustomFieldValues |
| 4 | CustomFields |
| 107281 | GroupMembers |
| 157020 | Groups |
| 1579 | Links |
| 0 | ObjectClasses |
| 14357 | ObjectCustomFieldValues |
| 4 | ObjectCustomFields |
| 45 | ObjectScrips |
| 0 | ObjectTopics |
| 162308 | Principals |
| 16 | Queues |
| 21 | ScripActions |
| 22 | ScripConditions |
| 45 | Scrips |
| 35 | Templates |
| 35232 | Tickets |
| 0 | Topics |
| 578000 | Transactions |
| 10340 | Users |
| 4518 | sessions |
27 rows in set, 1 warning (0.43 sec)

Thanks!
//adam

Your query here says that MySQL is looking at a pretty tiny dataset
and running the same query on multiple databases here shows a wildly
different explain and very quick results. This usually points to you
having some enormous tables (although the explain doesn’t indicate
that) or a badly tuned mysql.

Can you download and run http://mysqltuner.pl on your database.
I will have to go through this script and consult with a colleague
before running that as our database contains more than just our
ticket system.

We’ll see what it returns.
Nothing in your data set is particularly large, which points more and
more to MySQL problems.

-kevin