AssetTracker crashes loading asset

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

Sounds exactly like the issue I have, I think something is trying to get
all those records, I tried to trace it but with no luck, I think it may
be related to the back/next links on the ticket display page. It’s
checking each record for something, This is ok with small results but
crashes with large sets. I really wish I could figure this one out, I
get the occasional 500 and users are made aware to keep search results
smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT crash,
and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to get all
those records, I tried to trace it but with no luck, I think it may be
related to the back/next links on the ticket display page. It’s checking each
record for something, This is ok with small results but crashes with large
sets. I really wish I could figure this one out, I get the occasional 500 and
users are made aware to keep search results smaller. In my testing this
affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media. Buy a
copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

The queries would execute fine, the problem at least in my case is it
tries to load the whole result set into memory causing oom-killer to go
on rampage. Apache will peak out, to sort of help this situation I added
a mem limit to the fastcgi handler which will basically die before
things get really ugly. I’m able to see this query in the log, it has no
limit clause so it really is getting everything. While doing a form of
debug/trace on the apache/handler I’m also able to see it trying to load
the full row times search results (table.*) into memory. This is all on
a ticket display from a large search result. So unless you are loading
it in memory as opposed to say a shell client output you won’t run into
memory problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug
report but it appears to be somewhat isolated… people with low tickets
will never run into this problem.

Curtis

John wrote:

not certain it makes much sense…ive never had this problem in rt 3.4.5
with at 1.2.3…On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 12:52:04 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

The queries would execute fine, the problem at least in my case is it tries
to load the whole result set into memory causing oom-killer to go on rampage.
Apache will peak out, to sort of help this situation I added a mem limit to
the fastcgi handler which will basically die before things get really ugly.
I’m able to see this query in the log, it has no limit clause so it really is
getting everything. While doing a form of debug/trace on the apache/handler
I’m also able to see it trying to load the full row times search results
(table.*) into memory. This is all on a ticket display from a large search
result. So unless you are loading it in memory as opposed to say a shell
client output you won’t run into memory problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug report
but it appears to be somewhat isolated… people with low tickets will never
run into this problem.

Curtis

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT crash,
and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to get
all those records, I tried to trace it but with no luck, I think it may be
related to the back/next links on the ticket display page. It’s checking
each record for something, This is ok with small results but crashes with
large sets. I really wish I could figure this one out, I get the
occasional 500 and users are made aware to keep search results smaller. In
my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media. Buy
a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

While my issue is not related to asset tracker, the behavior sounds
identical. It seems to be related to how RT handles search and display,
the only thing I can think of is the menu on the left, It has to
determine the first and last record and also next and prev, I think it’s
doing a full scan to figure it out… pretty intensive operation, It
would be better if it didn’t have to deal with table.* and say only
table.ticketid.

<<First
< Prev
#12345
Next >
Last >>

Sorry for the confusion, could just be a coincidence.

Curtis

John wrote:

hrm…i just pulled in a list of 47,000 RT tickets and was able to load a
single ticket after a time of about 20 seconds…seemed normal.
have you tried Set($LogToSyslog , ‘alert’); as a solution?

this seems isolated to assettracker…i think…On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 13:12:53 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

While my issue is not related to asset tracker, the behavior sounds
identical. It seems to be related to how RT handles search and display, the
only thing I can think of is the menu on the left, It has to determine the
first and last record and also next and prev, I think it’s doing a full scan
to figure it out… pretty intensive operation, It would be better if it
didn’t have to deal with table.* and say only table.ticketid.

<<First
< Prev
#12345
Next >
Last >>

Sorry for the confusion, could just be a coincidence.

Curtis

John wrote:

not certain it makes much sense…ive never had this problem in rt 3.4.5
with at 1.2.3…

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 12:52:04 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

The queries would execute fine, the problem at least in my case is it
tries to load the whole result set into memory causing oom-killer to go on
rampage. Apache will peak out, to sort of help this situation I added a
mem limit to the fastcgi handler which will basically die before things
get really ugly. I’m able to see this query in the log, it has no limit
clause so it really is getting everything. While doing a form of
debug/trace on the apache/handler I’m also able to see it trying to load
the full row times search results (table.*) into memory. This is all on a
ticket display from a large search result. So unless you are loading it in
memory as opposed to say a shell client output you won’t run into memory
problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug
report but it appears to be somewhat isolated… people with low tickets
will never run into this problem.

Curtis

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT crash,
and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to get
all those records, I tried to trace it but with no luck, I think it may
be related to the back/next links on the ticket display page. It’s
checking each record for something, This is ok with small results but
crashes with large sets. I really wish I could figure this one out, I
get the occasional 500 and users are made aware to keep search results
smaller. In my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

That’s around near the limit of what I can pull in. While you are
probably right with it being isolated, 20 seconds does seem a little
suspect. I’m curious how much memory is being used during that call. We
have a lot of attachments which makes each record rather large at times.
If you do end up finding a solution to your problem I’m curious to hear it.

John wrote:

I think i may have found a bit of a solution to the AT problem. we’re
running on 2 quad-core servers as frontends with 8gb memory each. the
fault im seeing in AT is due to a timeout in apache waiting for its
backend processes to respond from huge queries…so:

AddHandler fcgid-script .fcgi

OutputBufferSize 1280000
IdleTimeout 600
ProcessLifeTime 3600
MaxProcessCount 8
DefaultMinClassProcessCount 3
DefaultMaxClassProcessCount 3

IPCConnectTimeout 20 IPCCommTimeout 600

has resolved the AT issue from what im seeing. queries still take a long
time, but thats just because RT is pulling lots of other data from mysql
that im not certain is even pertanent to the asset/ticket at hand. could
the speed of the query be increased by changing maxprocesscount to a
higher value?

im running into issues now with the population of a user rights page that
includes 300-400 users…namely:

RT::Group::Privileged Unimplemented in HTML::Mason::Commands.
(/opt/rt3/share/html/Elements/ShowUserConcise line 52)

any ideas?

Date: Fri, 07 Nov 2008 12:52:04 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

The queries would execute fine, the problem at least in my case is it tries
to load the whole result set into memory causing oom-killer to go on rampage.
Apache will peak out, to sort of help this situation I added a mem limit to
the fastcgi handler which will basically die before things get really ugly.
I’m able to see this query in the log, it has no limit clause so it really is
getting everything. While doing a form of debug/trace on the apache/handler
I’m also able to see it trying to load the full row times search results
(table.*) into memory. This is all on a ticket display from a large search
result. So unless you are loading it in memory as opposed to say a shell
client output you won’t run into memory problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug report
but it appears to be somewhat isolated… people with low tickets will never
run into this problem.

Curtis

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT crash,
and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to get
all those records, I tried to trace it but with no luck, I think it may be
related to the back/next links on the ticket display page. It’s checking
each record for something, This is ok with small results but crashes with
large sets. I really wish I could figure this one out, I get the
occasional 500 and users are made aware to keep search results smaller. In
my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media. Buy
a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

How many assets do you have? Nobody has ever seen this issue before…On Tue, Nov 11, 2008 at 12:57 PM, John nimbius@sdf.lonestar.org wrote:

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

I think i may have found a bit of a solution to the AT problem. we’re
running on 2 quad-core servers as frontends with 8gb memory each. the
fault im seeing in AT is due to a timeout in apache waiting for its
backend processes to respond from huge queries…so:

AddHandler fcgid-script .fcgi

OutputBufferSize 1280000
IdleTimeout 600
ProcessLifeTime 3600
MaxProcessCount 8
DefaultMinClassProcessCount 3
DefaultMaxClassProcessCount 3

IPCConnectTimeout 20 IPCCommTimeout 600

has resolved the AT issue from what im seeing. queries still take a long
time, but thats just because RT is pulling lots of other data from mysql
that im not certain is even pertanent to the asset/ticket at hand. could
the speed of the query be increased by changing maxprocesscount to a
higher value?

im running into issues now with the population of a user rights page that
includes 300-400 users…namely:

RT::Group::Privileged Unimplemented in HTML::Mason::Commands.
(/opt/rt3/share/html/Elements/ShowUserConcise line 52)

any ideas?

Date: Fri, 07 Nov 2008 12:52:04 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

The queries would execute fine, the problem at least in my case is it
tries
to load the whole result set into memory causing oom-killer to go on
rampage.
Apache will peak out, to sort of help this situation I added a mem limit
to
the fastcgi handler which will basically die before things get really
ugly.
I’m able to see this query in the log, it has no limit clause so it
really is
getting everything. While doing a form of debug/trace on the
apache/handler
I’m also able to see it trying to load the full row times search results
(table.*) into memory. This is all on a ticket display from a large
search
result. So unless you are loading it in memory as opposed to say a shell
client output you won’t run into memory problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug
report
but it appears to be somewhat isolated… people with low tickets will
never
run into this problem.

Curtis

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT
crash,

and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to
get

all those records, I tried to trace it but with no luck, I think it may
be

related to the back/next links on the ticket display page. It’s
checking

each record for something, This is ok with small results but crashes
with

large sets. I really wish I could figure this one out, I get the
occasional 500 and users are made aware to keep search results smaller.
In

my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy

a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

9,359On Tue, 11 Nov 2008, Todd Chapman wrote:

Date: Tue, 11 Nov 2008 13:04:14 -0500
From: Todd Chapman todd@chaka.net
To: John nimbius@sdf.lonestar.org
Cc: Curtis Bruneau curtisb@vianet.ca, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset…raw horsepower
solution?

How many assets do you have? Nobody has ever seen this issue before…

On Tue, Nov 11, 2008 at 12:57 PM, John nimbius@sdf.lonestar.org wrote:

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

I think i may have found a bit of a solution to the AT problem. we’re
running on 2 quad-core servers as frontends with 8gb memory each. the
fault im seeing in AT is due to a timeout in apache waiting for its
backend processes to respond from huge queries…so:

AddHandler fcgid-script .fcgi

OutputBufferSize 1280000
IdleTimeout 600
ProcessLifeTime 3600
MaxProcessCount 8
DefaultMinClassProcessCount 3
DefaultMaxClassProcessCount 3

IPCConnectTimeout 20 IPCCommTimeout 600

has resolved the AT issue from what im seeing. queries still take a long
time, but thats just because RT is pulling lots of other data from mysql
that im not certain is even pertanent to the asset/ticket at hand. could
the speed of the query be increased by changing maxprocesscount to a
higher value?

im running into issues now with the population of a user rights page that
includes 300-400 users…namely:

RT::Group::Privileged Unimplemented in HTML::Mason::Commands.
(/opt/rt3/share/html/Elements/ShowUserConcise line 52)

any ideas?

Date: Fri, 07 Nov 2008 12:52:04 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

The queries would execute fine, the problem at least in my case is it
tries
to load the whole result set into memory causing oom-killer to go on
rampage.
Apache will peak out, to sort of help this situation I added a mem limit
to
the fastcgi handler which will basically die before things get really
ugly.
I’m able to see this query in the log, it has no limit clause so it
really is
getting everything. While doing a form of debug/trace on the
apache/handler
I’m also able to see it trying to load the full row times search results
(table.*) into memory. This is all on a ticket display from a large
search
result. So unless you are loading it in memory as opposed to say a shell
client output you won’t run into memory problems.

Best of luck with your situation, I have not been able to get any
confirmation until now with you. I’ve produced somewhat detailed bug
report
but it appears to be somewhat isolated… people with low tickets will
never
run into this problem.

Curtis

John wrote:

curtis:
ive played around re-executing the mysql queries related to the RT
crash,

and to no avail. they execute just fine.

could this perhaps be a syslog issue like in RT? is there a seperate
control for syslogging in AT?

On Fri, 7 Nov 2008, Curtis Bruneau wrote:

Date: Fri, 07 Nov 2008 10:43:48 -0500
From: Curtis Bruneau curtisb@vianet.ca
To: John nimbius@sdf.lonestar.org, rt-users@lists.bestpractical.com
Subject: Re: [rt-users] AssetTracker crashes loading asset

Sounds exactly like the issue I have, I think something is trying to
get

all those records, I tried to trace it but with no luck, I think it may
be

related to the back/next links on the ticket display page. It’s
checking

each record for something, This is ok with small results but crashes
with

large sets. I really wish I could figure this one out, I get the
occasional 500 and users are made aware to keep search results smaller.
In

my testing this affected 3.4.x 3.6.x and 3.8.x

John wrote:

well, just as i thought the RT slowness issue had been resolved,
assettracker
is now dying when loading a ticket out of a large list.

example: clicking “all assets” enumerates a 9000 row list fine,
but clicking on any element in the list will crunch for a while
then die with a 500 error.

smaller lists with say 1000-3000 rows are sluggish when loading
an asset from the list, but succeed.

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy

a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org


http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

nimbius@sdf.lonestar.org
SDF Public Access UNIX System - http://sdf.lonestar.org