Unchecking "Let this user be granted rights" results in hang

RT Users
1

Afternoon All,
For quite a while we have been having a problem where is you uncheck the
“Let this user be granted rights” option the thread goes into a CPU spin
and never returns.

It happens in all 3.4 (including 3.4.4) and from memory 3.2 as well.

This is an install that started life at 3.0 and has been upgraded
reasonably frequently. If I export the database over to a new machine
it happens there as well.

I just spent most of today upgrading dependancies and working on the
database trying to make the schema as close to the distribution one as
possible but aside from the Instance column in the Groups table being
varchar instead of Integer everything else is either right, or just a
few extra fields in the table.

I’m stumped.

We have a little under 9,000 tickets so it’s not a big database and it
shows the same behaviour with the production system running FreeBSD 4.9,
Mysql 4.0.16, Apache 2.0.50, mod_perl 1.99, Perl 5.8.4 and with my dev
machine running FreeBSD 5.4, Mysql 4.1.9, Apache 2.0.54, mod_fastcgi
2.4.2, perl 5.8.6.

It looks like someone called “murphy@genome” had the same problem in
February with 3.2 which is described in this email;

Any help here would be appreciated.

Carl.

2

Carl Makin wrote:

Afternoon All,
For quite a while we have been having a problem where is you uncheck the
“Let this user be granted rights” option the thread goes into a CPU spin
and never returns.

“CPU spin” means we’ve got some sort of loop going on. Is it hitting the
database? If so, with which queries? Can you run under standalone_httpd
and replicate the issue in the debugger? If so, we might be able to find
the codepath.

signature.asc (189 Bytes)

3

Carl Makin wrote:

Afternoon All,
For quite a while we have been having a problem where is you uncheck the
“Let this user be granted rights” option the thread goes into a CPU spin
and never returns.

“CPU spin” means we’ve got some sort of loop going on. Is it hitting the
database? If so, with which queries? Can you run under standalone_httpd
and replicate the issue in the debugger? If so, we might be able to find
the codepath.

!! I’ve exactly the same issue, under very similar circumstances. It started
occurring under standard Debian Sarge, for both 3.0.12 and 3.4.1, mysql 4.0and
4.1.

Easiest way to reproduce: Toggle “Let this user be granted rights”.
Some other administrative tasks also cause it to hang.

Symptoms: Apache child process is using all cpu, slowly eating all memory.

Associated rt.log entry is:
[Mon Sep 12 18:42:07 2005] [warning]: Deep recursion on subroutine
“RT::ACE::_Value” at /usr/share/perl5/DBIx/SearchBuilder/Record.pm line 425.
(/usr/share/request-tracker3.4/lib/RT.pm:277)

It also happens with the standalone_httpd. I haven’t tried the debugger yet.

The problem does NOT (immediately) occur with a new database. Only the
imported one.

Any pointers (other than starting over:) are appreciated. I’d be happy to
collect more data.
Thanks
-Peter

4

Easiest way to reproduce: Toggle “Let this user be granted rights”.
Some other administrative tasks also cause it to hang.

Symptoms: Apache child process is using all cpu, slowly eating all memory.

Associated rt.log entry is:
[Mon Sep 12 18:42:07 2005] [warning]: Deep recursion on subroutine
“RT::ACE::_Value” at /usr/share/perl5/DBIx/SearchBuilder/Record.pm line 425.
(/usr/share/request-tracker3.4/lib/RT.pm:277)

Ok. Now we’re getting somewhere. In RT::ACE::_Value, add a Carp::cluck;

That will tell us how it’s recurring. then it should be a 30 second
fix.

5

Hi Jesse, Peter,

Jesse Vincent wrote:

Peter wrote…

Easiest way to reproduce: Toggle “Let this user be granted rights”.
Some other administrative tasks also cause it to hang.

Symptoms: Apache child process is using all cpu, slowly eating all memory.

Associated rt.log entry is:
[Mon Sep 12 18:42:07 2005] [warning]: Deep recursion on subroutine
“RT::ACE::_Value” at /usr/share/perl5/DBIx/SearchBuilder/Record.pm line 425.
(/usr/share/request-tracker3.4/lib/RT.pm:277)

I’m not seeing that particular message although everything else is the same.

MySQL was showing heavy traffic but a light load and RT seemed to be
executing the following query in a tight loop;

SELECT * FROM CachedGroupMembers WHERE Disabled = ‘0’ AND GroupId = ‘4’
AND MemberId = ‘1’

This returns an empty set. There are no records in that table with a
MemberId of 1.

When I kill the MySQL thread I get a heap of;

[Wed Sep 14 00:40:21 2005] [err]: ACE 109 couldn’t load its principal
object (/usr/local/rt3/lib/RT/ACE_Overlay.pm:839)

in rt.log.

Ok. Now we’re getting somewhere. In RT::ACE::_Value, add a Carp::cluck;

That will tell us how it’s recurring. then it should be a 30 second
fix.

[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18] FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tDBIx::SearchBuilder::Record::ANON(‘RT::ACE=HASH(0x8cc60a0)’)
called at /usr/local/rt3/lib/RT/ACE_Overlay.pm line 815, referer:
http://newton/Admin/Users/Modify.html?id=70
[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18] FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tRT::ACE::Object(‘RT::ACE=HASH(0x8cc60a0)’) called at
/usr/local/rt3/lib/RT/ACE_Overlay.pm line 869, referer:
http://newton/Admin/Users/Modify.html?id=70
[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18] FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tRT::ACE::_Value(‘RT::ACE=HASH(0x8cc60a0)’, ‘ObjectType’) called at
/usr/local/lib/perl5/site_perl/5.8.6/DBIx/SearchBuilder/Record.pm line
425, referer: http://newton/Admin/Users/Modify.html?id=70

I put the Carp::cluck into the _Value subroutine in ACE_Overlay.pm.

Is there anything more I can give you?

Carl.

6

Carl Makin wrote:

Associated rt.log entry is:
[Mon Sep 12 18:42:07 2005] [warning]: Deep recursion on subroutine
“RT::ACE::_Value” at /usr/share/perl5/DBIx/SearchBuilder/Record.pm
line 425.
(/usr/share/request-tracker3.4/lib/RT.pm:277)

I’m not seeing that particular message although everything else is the
same.

Oops, should have kept looking. I’m seeing that message as well but it
takes a while.

Carl.

7

Hi Carl,

This simple change, courtesy Jesse, solved the “RT hanging” issue for me. I
don’t use fastcgi. Your mileage may vary.

In ACE_Overlay.pm, find

sub Object {
my $self = shift;

my $appliesto_obj;

if ($self->__Value(‘ObjectType’) && $OBJECT_TYPES{$self->__Value(
‘ObjectType’)}
) {
$appliesto_obj = $self->__Value(‘ObjectType’)->new($self->CurrentUser);
unless (ref( $appliesto_obj) eq $self->__Value(‘ObjectType’)) {
return undef;
}
$appliesto_obj->Load( $self->__Value(‘ObjectId’) );
return ($appliesto_obj);
}
else {
$RT::Logger->warning( "$self → Object called for an object "
. “of an unknown type:”
. $self->ObjectType );
return (undef);
}
}

remove:

$RT::Logger->warning( "$self → Object called for an object "
. “of an unknown type:”
. $self->ObjectType );On 9/13/05, Carl Makin carl@xena.ipaustralia.gov.au wrote:

Hi Jesse, Peter,

Jesse Vincent wrote:

Peter wrote…

Easiest way to reproduce: Toggle “Let this user be granted rights”.
Some other administrative tasks also cause it to hang.

Symptoms: Apache child process is using all cpu, slowly eating all
memory.

Associated rt.log entry is:
[Mon Sep 12 18:42:07 2005] [warning]: Deep recursion on subroutine
“RT::ACE::_Value” at /usr/share/perl5/DBIx/SearchBuilder/Record.pm line

(/usr/share/request-tracker3.4/lib/RT.pm:277)

I’m not seeing that particular message although everything else is the
same.

MySQL was showing heavy traffic but a light load and RT seemed to be
executing the following query in a tight loop;

SELECT * FROM CachedGroupMembers WHERE Disabled = ‘0’ AND GroupId = ‘4’
AND MemberId = ‘1’

This returns an empty set. There are no records in that table with a
MemberId of 1.

When I kill the MySQL thread I get a heap of;

[Wed Sep 14 00:40:21 2005] [err]: ACE 109 couldn’t load its principal
object (/usr/local/rt3/lib/RT/ACE_Overlay.pm:839)

in rt.log.

Ok. Now we’re getting somewhere. In RT::ACE::_Value, add a Carp::cluck;

That will tell us how it’s recurring. then it should be a 30 second
fix.

[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18http://10.0.100.18]
FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tDBIx::SearchBuilder::Record::ANON(‘RT::ACE=HASH(0x8cc60a0)’)
called at /usr/local/rt3/lib/RT/ACE_Overlay.pm line 815, referer:
http://newton/Admin/Users/Modify.html?id=70
[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18http://10.0.100.18]
FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tRT::ACE::Object(‘RT::ACE=HASH(0x8cc60a0)’) called at
/usr/local/rt3/lib/RT/ACE_Overlay.pm line 869, referer:
http://newton/Admin/Users/Modify.html?id=70
[Wed Sep 14 11:28:13 2005] [error] [client 10.0.100.18http://10.0.100.18]
FastCGI: server
“/usr/local/rt3/bin/mason_handler.fcgi” stderr:
\tRT::ACE::_Value(‘RT::ACE=HASH(0x8cc60a0)’, ‘ObjectType’) called at
/usr/local/lib/perl5/site_perl/5.8.6/DBIx/SearchBuilder/Record.pm line
425, referer: http://newton/Admin/Users/Modify.html?id=70

I put the Carp::cluck into the _Value subroutine in ACE_Overlay.pm.

Is there anything more I can give you?

Carl.

8

Hi Peter,

Peter Struijk wrote:

This simple change, courtesy Jesse, solved the “RT hanging” issue for
me. I don’t use fastcgi. Your mileage may vary.

In ACE_Overlay.pm, find

sub Object {

remove:

   $RT::Logger->warning( "$self -> Object called for an object "
                         . "of an unknown type:"
                         . $self->ObjectType );

Worked here as well! Woohoo!

Works on both the 3.4.4 FastCGI test system here and the 3.4.2
mod_perl1.99 production system.

Thanks Peter and Jesse!

Carl.

9

Hi Peter,

Peter Struijk wrote:

This simple change, courtesy Jesse, solved the “RT hanging” issue for
me. I don’t use fastcgi. Your mileage may vary.

In ACE_Overlay.pm, find

sub Object {

remove:

   $RT::Logger->warning( "$self -> Object called for an object "
                         . "of an unknown type:"
                         . $self->ObjectType );

Worked here as well! Woohoo!

Works on both the 3.4.4 FastCGI test system here and the 3.4.2
mod_perl1.99 production system.

Thanks Peter and Jesse!

Carl.

The rt-users Archives

Be sure to check out the RT Wiki at http://wiki.bestpractical.com

Buy your copy of our new book, RT Essentials, today!

Download a free sample chapter from http://rtbook.bestpractical.com

Greetings All,

Experienced same issues with disabling users. Applied fix above and
problem has now disappeared. Running with the following env;
RT 3.4.4
FastCGI
Apache/2.0.54
Perl 5.8.4

Thank you for the fix Jesse.

Cheers,
David.