Merged Ticket Performance Degredation

RT Users
1

Hi all,

We’re experiencing an issue with RT 3.6.1 wherein loading merged tickets
results in severe performance degredation, e.g. 100 - 300 seconds per
ticket. After 30 seconds, Apache gives up on the FastCGI script and
generates a 500. This problem has been documented and referenced in
several posts, but I have been unable to find a resolution. The
following from Jesse seems to be the most descriptive:

“Yep. it’s not recursion. It’s RT::Transaction::TicketObj which should
be made smarter. Rather than loading a ticket by id, it should be
loading by id and effective id. I’d love a patch.”

(Full thread is available at the Gossamer list archive:
Carbon60: Cloud Consulting - Services and Solutions)

I checked 3.8.1 and the “TODO” is still present in Ticket_Overlay.pm
(Load) so I’m assuming the performance problem is still present in that
release.

I’d love to help by producing a patch but frankly I don’t understand
what needs to be done. Has anyone else found a solution to this
problem?

Following another suggestion, I tried reducing the loglevel such that
the “We found a merged ticket” messages weren’t stored but we’re still
seeing occasional CPU spikes from the FastCGI process (and the
associated delays which lead to HTTP/500s from Apache). Without the
debug messages turned on I can’t be absolutely certain the issue is
loading merged tickets, but it seems very likely given that I can
reproduce the problem by attempting to load a known-merged ticket with
debug logging turned off.

Thanks for any suggestions or feedback.

Regards,

Damon

Damon T. Miller
Director of Application Services
Thinking Phone Networks
damon@thinkingphones.com
617-649-1388 (Office)

2

Hi all,

We’re experiencing an issue with RT 3.6.1 wherein loading merged
tickets
results in severe performance degredation, e.g. 100 - 300 seconds per
ticket. After 30 seconds, Apache gives up on the FastCGI script and
generates a 500. This problem has been documented and referenced in
several posts, but I have been unable to find a resolution.

Thanks for any suggestions or feedback.

adding

@LogToSyslogConf = ( socket => ‘native’ ) unless (@LogToSyslogConf);

to RT_SiteConfig.pm sped up the merged ticket display from 260 seconds
to about 6 seconds for us.

probably still too many log messages being written for the reason
above I suppose.

  • Mark
3

Mark,

Thanks very much for your quick reply. I updated our config file as
suggested but either the ‘native’ socket type isn’t supported in Perl
5.8.5 on Linux or I’m doing something wrong. Here’s what happened when
I started the FastCGI handler from the command-line:

Invalid argument passed to setlogsock; must be ‘stream’, ‘unix’, ‘tcp’,
‘udp’ or ‘inet’ at /usr/lib/perl5/site_perl/5.8.5/Log/Dispatch/Syslog.pm
line 66

To make sure I followed your suggestion correctly, here are the last few
entries in our RT_SiteConfig.pm:

Set($AutoCreateNonExternalUsers, 1);
Set($LogToFileNamed , “rt.log”);
Set($LogToFile , ‘warning’);
Set($UseFriendlyFromLine , 0);

@LogToSyslogConf = ( socket => ‘native’ ) unless (@LogToSyslogConf);

1;

As another potentially-relevant point, I also tried decreasing the
verbosity of the logfile so as to eliminate the “We found a merged
ticket” messages entirely. Does that make the syslog update to
RT_SiteConfig.pm unnecessary? Even after reducing verbosity, I’m still
seeing the “hang” issue upon trying to load a merged ticket through the
UI. Regardless, I just wanted to make sure I wasn’t missing something
between those two changes.

Thanks again,

Damon

Damon T. Miller
Director of Application Services
Thinking Phone Networks
damon@thinkingphones.com
617-649-1388 (Office)

-----Original Message-----
From: Mark Blackman [mailto:mark@exonetric.com]
Sent: Friday, October 17, 2008 11:58 AM
To: Damon Miller
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Merged Ticket Performance Degredation

Hi all,

We’re experiencing an issue with RT 3.6.1 wherein loading merged
tickets
results in severe performance degredation, e.g. 100 - 300 seconds
per

4

You just need to upgrade your Sys::Syslog version.

should be fine after that.

  • MarkOn 17 Oct 2008, at 19:57, Damon Miller wrote:

Mark,

Thanks very much for your quick reply. I updated our config file as
suggested but either the ‘native’ socket type isn’t supported in Perl
5.8.5 on Linux or I’m doing something wrong. Here’s what happened
when
I started the FastCGI handler from the command-line:

Invalid argument passed to setlogsock; must be ‘stream’, ‘unix’,
‘tcp’,
‘udp’ or ‘inet’ at /usr/lib/perl5/site_perl/5.8.5/Log/Dispatch/
Syslog.pm
line 66

To make sure I followed your suggestion correctly, here are the last
few
entries in our RT_SiteConfig.pm:

–

Set($AutoCreateNonExternalUsers, 1);
Set($LogToFileNamed , “rt.log”);
Set($LogToFile , ‘warning’);
Set($UseFriendlyFromLine , 0);

@LogToSyslogConf = ( socket => ‘native’ ) unless (@LogToSyslogConf);

1;

–

As another potentially-relevant point, I also tried decreasing the
verbosity of the logfile so as to eliminate the “We found a merged
ticket” messages entirely. Does that make the syslog update to
RT_SiteConfig.pm unnecessary? Even after reducing verbosity, I’m
still
seeing the “hang” issue upon trying to load a merged ticket through
the
UI. Regardless, I just wanted to make sure I wasn’t missing something
between those two changes.

Thanks again,

Damon

__

Damon T. Miller
Director of Application Services
Thinking Phone Networks
damon@thinkingphones.com
617-649-1388 (Office)

-----Original Message-----
From: Mark Blackman [mailto:mark@exonetric.com]
Sent: Friday, October 17, 2008 11:58 AM
To: Damon Miller
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Merged Ticket Performance Degredation

On 17 Oct 2008, at 16:39, Damon Miller wrote:

Hi all,

We’re experiencing an issue with RT 3.6.1 wherein loading merged
tickets
results in severe performance degredation, e.g. 100 - 300 seconds
per
ticket. After 30 seconds, Apache gives up on the FastCGI script and
generates a 500. This problem has been documented and referenced in
several posts, but I have been unable to find a resolution.

Thanks for any suggestions or feedback.

adding

@LogToSyslogConf = ( socket => ‘native’ ) unless (@LogToSyslogConf);

to RT_SiteConfig.pm sped up the merged ticket display from 260
seconds
to about 6 seconds for us.

probably still too many log messages being written for the reason
above I suppose.

  • Mark