Search hangs RT

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Kenneth Marshall skrev:> On Tue, Feb 10, 2009 at 02:21:15PM +0100, Anne-Marie Achrenius wrote:

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Thanks for the help, unfortunately the problem doesn’t seem to lie there:

mysql> SELECT COUNT() FROM sessions;
| COUNT(
) |
| 323 |
1 row in set (0.00 sec)

and deleting anything older than 30/1 left 11. Still the same problem,
search keeps hanging.
How do I optimize the table? (I used to work with Oracle db:s around 10
years ago, I’m pretty rusty!)

Depressed,

/Anne-Marie

Kenneth Marshall skrev:

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Thanks for the help, unfortunately the problem doesn’t seem to lie there:

mysql> SELECT COUNT() FROM sessions;
±---------+
| COUNT(
) |
±---------+
| 323 |
±---------+
1 row in set (0.00 sec)

and deleting anything older than 30/1 left 11. Still the same problem,
search keeps hanging.
How do I optimize the table? (I used to work with Oracle db:s around 10
years ago, I’m pretty rusty!)

Depressed,

/Anne-Marie

Anne-Marie,

It is more drastic, but since your RT administrator is out, maybe you
could change to Session logging to a file and not the database. In
3.8 the lines in RT_Config.pm are:

=item C<$WebSessionClass>

C<$WebSessionClass> is the class you wish to use for managing Sessions.
It defaults to use your SQL database, but if you are using MySQL 3.x and
plans to use non-ascii Queue names, uncomment and add this line to
F<RT_SiteConfig.pm> will prevent session corruption.

=cut

Set($WebSessionClass , ‘Apache::Session::File’);

You should be able to add this line to your RT_SiteConfig.pm and then
restart RT to use the file version of session storage. That should
get you working until your admin returns.

Good luck,
Ken

Hi Anne Marie,
to optimize tables use mysql> optimize table sessions;
we had the same issue in the past and turned down the session locking time
in the mysql.pm from your first mail to 1sec, it fixes it for us.

torsten

btw, we also clean each night the session table, which makes rt lot faster.2009/2/10 Anne-Marie Achrenius ama@chalmers.se

Kenneth Marshall skrev:

On Tue, Feb 10, 2009 at 02:21:15PM +0100, Anne-Marie Achrenius wrote:

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our
RT-installation.

I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Thanks for the help, unfortunately the problem doesn’t seem to lie there:

mysql> SELECT COUNT() FROM sessions;
±---------+
| COUNT(
) |
±---------+
| 323 |
±---------+
1 row in set (0.00 sec)

and deleting anything older than 30/1 left 11. Still the same problem,
search keeps hanging.
How do I optimize the table? (I used to work with Oracle db:s around 10
years ago, I’m pretty rusty!)

Depressed,

/Anne-Marie


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

MFG

Torsten Brumm

http://www.torsten-brumm.de

Hello!

Sorry, no change in RT behaviour. Still keeps hanging on searches.

Regards,

/Anne-Marie

Torsten Brumm skrev:

Hello Kenneth,

thanks for your help, but I already tried that. It generates a new type
of problem, every click on a link logs out the user.
I read a thread that seemed to indicate that this could be Web
client-related, unfortunately there were no follow ups on that info.

Searches still hang almost every time, those few times that a search
runs correctly, clicking on one of the resulting tickets cause the same
kind of hang.

Regards,

/Anne-Marie

Kenneth Marshall skrev:> On Tue, Feb 10, 2009 at 04:19:27PM +0100, Anne-Marie Achrenius wrote:

Kenneth Marshall skrev:

On Tue, Feb 10, 2009 at 02:21:15PM +0100, Anne-Marie Achrenius wrote:

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Thanks for the help, unfortunately the problem doesn’t seem to lie there:

mysql> SELECT COUNT() FROM sessions;
±---------+
| COUNT(
) |
±---------+
| 323 |
±---------+
1 row in set (0.00 sec)

and deleting anything older than 30/1 left 11. Still the same problem,
search keeps hanging.
How do I optimize the table? (I used to work with Oracle db:s around 10
years ago, I’m pretty rusty!)

Depressed,

/Anne-Marie

Anne-Marie,

It is more drastic, but since your RT administrator is out, maybe you
could change to Session logging to a file and not the database. In
3.8 the lines in RT_Config.pm are:

=item C<$WebSessionClass>

C<$WebSessionClass> is the class you wish to use for managing Sessions.
It defaults to use your SQL database, but if you are using MySQL 3.x and
plans to use non-ascii Queue names, uncomment and add this line to
F<RT_SiteConfig.pm> will prevent session corruption.

=cut

Set($WebSessionClass , ‘Apache::Session::File’);

You should be able to add this line to your RT_SiteConfig.pm and then
restart RT to use the file version of session storage. That should
get you working until your admin returns.

Good luck,
Ken

That behavior (every click brings you to a new link) is a sessions
issue, as RT can’t find wherever your storing sessions.

If it is doing this and searches are still hanging, it sounds like the
issue is not related to sessions.

~SamOn Wed, Feb 11, 2009 at 7:19 AM, Anne-Marie Achrenius ama@chalmers.se wrote:

Hello Kenneth,

thanks for your help, but I already tried that. It generates a new type
of problem, every click on a link logs out the user.
I read a thread that seemed to indicate that this could be Web
client-related, unfortunately there were no follow ups on that info.

Searches still hang almost every time, those few times that a search
runs correctly, clicking on one of the resulting tickets cause the same
kind of hang.

Regards,

/Anne-Marie

Kenneth Marshall skrev:

On Tue, Feb 10, 2009 at 04:19:27PM +0100, Anne-Marie Achrenius wrote:

Kenneth Marshall skrev:

On Tue, Feb 10, 2009 at 02:21:15PM +0100, Anne-Marie Achrenius wrote:

Hello!

I’m new to the list and I’m in a bit of a pickle, our RT-guru is on
paternity leave and we’ve started having problems with our RT-installation.
I’ve tried searching the list archives for something similar, but
nothing fits exactly:

The problem: Often (not every time, maybe 8 out of 10) when I perform a
Search in RT, the search hangs.
The mysqld process is at 100 % cpu. When reading the log files, this
jumps out:

/var/lib/mysql/blaha.log:

SELECT GET_LOCK(‘Apache-Session-ed5bea237ae21873fecd13ade9cb3f63’, 3600)

and after a few minutes the web-interface returns with “500 Internal
Server Error”

I’ve so far tried to decrease the LOCK time in
/usr/lib/perl5/site_perl/5.8.5/Apache/Session/Lock/MySQL.pm
but that doesn’t seem to help at all.
I’ve enabled logging of slow mysql queries, but so far no log entries

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52

Does anyone have any suggestions or similar problems?
I can’t pinpoint when we started having problems, 2-3 weeks ago, and the
problem seems to slowly escalate.

/Anne-Marie

Maybe your session table needs to be cleaned up. Try deleting any
sessions over a couple of days old and then optimize the table. I
cannot help with the specific commands since we use PostgreSQL for
our RT instance but in SQL I run a command like:

DELETE FROM sessions WHERE lastupdated < ‘2009-1-30’;

or even

TRUNCATE TABLE sessions;

to get them all. You may just have a bloat problem. Try a

SELECT COUNT(*) FROM sessions;

Hope this helps.

Cheers,
Ken

Thanks for the help, unfortunately the problem doesn’t seem to lie there:

mysql> SELECT COUNT() FROM sessions;
±---------+
| COUNT(
) |
±---------+
| 323 |
±---------+
1 row in set (0.00 sec)

and deleting anything older than 30/1 left 11. Still the same problem,
search keeps hanging.
How do I optimize the table? (I used to work with Oracle db:s around 10
years ago, I’m pretty rusty!)

Depressed,

/Anne-Marie

Anne-Marie,

It is more drastic, but since your RT administrator is out, maybe you
could change to Session logging to a file and not the database. In
3.8 the lines in RT_Config.pm are:

=item C<$WebSessionClass>

C<$WebSessionClass> is the class you wish to use for managing Sessions.
It defaults to use your SQL database, but if you are using MySQL 3.x and
plans to use non-ascii Queue names, uncomment and add this line to
F<RT_SiteConfig.pm> will prevent session corruption.

=cut

Set($WebSessionClass , ‘Apache::Session::File’);

You should be able to add this line to your RT_SiteConfig.pm and then
restart RT to use the file version of session storage. That should
get you working until your admin returns.

Good luck,
Ken


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Anne-Marie Achrenius wrote:

We’re running RT 3.7.5
mysql 14.7
apache 2.0.52
Any particular reason you know about that you’re running a
’unsupported’ version of RT?

RT 3.6.7 is the latest stable RT 3.6 release and RT 3.8.2 is the latest
RT 3.8 release. I have had the same problems with sessions being logged
out for every click and my solution was to edit

SetupSessionCookies

I think, to enable transactions.

See the archives for an email from Kenneth Crocker, Subject:
Re: [rt-users] Clicking a link brings me back to login page

Joop