Apache Threads hanging & not gracefully exiting

Hello everyone,

I currently have an instance of RT 4.0.17 running that we’re using for
processing incoming faxes. The faxes come in as attachments, and I have
some customizations to allow our staff to process these attachments (PDFs)
faster.

Everything seems to work great for a while, and the memory usage gradually
increases, and eventually all of the Apache threads (using event mpm) are
stuck in the “W” state (sending reply). If I try to gracefully restart
apache, all of those threads are stuck in the “G” state (gracefully
finishing). Those threads will never exit unless I kill the processes
manually. My guess would be that one of my customizations are causing
this, but does anyone have any tips for how to find out what the problem is?

I tried this:

And all I get is some “No object mapping for field::Name” warnings when
people login (we are using the ExternalAuth module for Active Directory
integration).

I also came across these two things saying to try adding the
PERL_DESTRUCT_LEVEL environment variable to avoid the perl_destruct() call
before exiting, and that could let the threads exit. Is this safe to try
with RT?

Here’s a brief list of what was customized:

  • ExternalAuth module using Active Directory
  • modified Ticket/Elements/ShowAttachments (locally) to show PDF attachment
    preview
    using Callbacks/AddAttachmentsColumn/Elements/RT__Ticket/ColumnMap/Default
    callback to add “Attachments” column to dashboard for quickly viewing
    attachments without going into ticket
  • custom Scrip that uses Filesys::SmbClient to copy attachments to the
    user’s computer when they “Take” the ticket

I would be happy to share any of the custom code, I just want to make sure
nothing is wrong with it before posting it to the Wiki. Any tip on
troubleshooting this would be greatly appreciated!

Thanks,
Nate

(gracefully finishing). Those threads will never exit unless I kill the processes manually. My
guess would be that one of my customizations are causing this, but does anyone have any tips
for how to find out what the problem is?

strace/dtruss?

  • custom Scrip that uses Filesys::SmbClient to copy attachments to the user’s computer when
    they “Take” the ticket

That sounds like the biggest suspect.

-kevin

I switched from mod_perl to mod_fcgid and along with the memory usage
decreasing by about 75%, the problem seems to have disappeared. I’m not
sure if there is a problem with the code and mod_fcgid is just handling it
better, or what the deal is, but everything is working fine now.

Judging by the user reviews of mod_perl (
mod_perl-2.0.13 - Apache mod_perl changes logfile - metacpan.org) it seems like mod_perl should be
the less preferred option, and mod_fastcgi or mod_fcgid should be used if
possible. Is this the general consensus? If so, it might be helpful to
add that recommendation on
Web deployment - RT 5.0.5 Documentation - Best Practical.

Thanks for your suggestions though Kevin, if I do see any further issues
I’ll try using strace.

For anyone else that comes across this, here are some apache mod_perl
documents about debugging mod_perl applications, using strace and other
methods:

-NateOn Wed, Jan 15, 2014 at 11:21 AM, Kevin Falcone falcone@bestpractical.comwrote:

On Tue, Jan 14, 2014 at 01:43:59PM -0500, Nathan Baker wrote:

(gracefully finishing). Those threads will never exit unless I kill
the processes manually. My
guess would be that one of my customizations are causing this, but
does anyone have any tips
for how to find out what the problem is?

strace/dtruss?

  • custom Scrip that uses Filesys::SmbClient to copy attachments to
    the user’s computer when
    they “Take” the ticket

That sounds like the biggest suspect.

-kevin

I switched from mod_perl to mod_fcgid and along with the memory usage decreasing by about 75%,
the problem seems to have disappeared. I’m not sure if there is a problem with the code and
mod_fcgid is just handling it better, or what the deal is, but everything is working fine now.
Judging by the user reviews of mod_perl (mod_perl-2.0.13 - Apache mod_perl changes logfile - metacpan.org) it
seems like mod_perl should be the less preferred option, and mod_fastcgi or mod_fcgid should
be used if possible. Is this the general consensus? If so, it might be helpful to add that
recommendation on Web deployment - RT 5.0.5 Documentation - Best Practical.

I personally never deploy on mod_perl unless forced, and we document a
number of mod_perl specific issues in the web_deployment document. We also
document mod_fastcgi and mod_fcgid before mod_perl in that document
for a reason.

It’s possible we should more strongly discourage mod_perl.

-kevin