I have a peculiar situation which crept up only a few weeks ago and I
have been able to narrow it down to RT + mod_FastCgi.
About 3 weeks ago, I updated to the latest FC2 kernel and a number of
other nondescript packages. Around the same time, I started
experiencing, on a nightly basis around 4:30am, where the web server
would be unresponsive. The server was running, but not handling any
requests. Restarting the web server would bring it back on-line until
the next evening.
I’ll omit all of the debugging here as it would not be relevant, but
suffice it to say that at 4:20am, logrotate would come along and kill
-USR1 httpd after rotating the logs, which was the same time that things
tanked out. Sounds like an Apache problem, right?
Well, I dug further, and recompiled the custom modules that we utilize
(mod_auth_ldap2 and mod_fastcgi) to use the newer version of the web
server. We’re pretty stuck on mod_fastcgi 2.4.0 because 2.4.2 does not
support running FastCgiServer processes within VirtualHosts (who thought
disabling that was a good idea?!).
Long story short, it turns out that, upon startup, if a FastCGI server
is configured to run, most of the httpd processes are running as root
(bad) and the remaining 2 or 3 are running as the apache user (not
including the parent process, of course). In the situations where the
server was unresponsive from 4:20am, all of the processes were running
as root. So, if all of the processes were running as root, the web
server is completely unresponsive.
So, if I disable the FastCgiServer processes (just commenting them out)
and start up the server, everything comes up just fine… If I have the
FastCgiServer processes enabled, then 2 or 3 of forked web server
processes run as apache, the rest run as root…
Is this just us?! Could this be kernel related? I’ve personally been
using RT with mod_fastcgi for ages… it is just plain odd that it has
tanked out on us like this.
I’m thinking of moving back from mod_fastcgi to mod_perl…