RT5 slow performance and intermittent Internal Server Error

OS: RHEL 7.6
Apache2.4.6 with FastCGI mod_fcgid/2.3.9
Oracle Database

I have RT5 running in a custom location and everything works (ish) although RT sometimes takes several minutes to return a page and sometimes I get the dreaded Error 500 Internal Server Error. I’ve read through the docs several times and can’t see what config changes I need to make to improve performance. From the email side of things, everything appears to be running well and cases are being created. It’s simply the web interface that is having the issue.

As per web deployment doc I have disabled mod_speling and mod_cache and have the prefork MPM mod configured.

My SSL virtual host includes:
ScriptAlias /rt /app/rt5/sbin/rt-server.fcgi/
<Location /rt>
Require all granted
Options +ExecCGI
AddHandler fcgid-script fcgi

Does anyone have suggestions where else I could look/investigate.

Unsure if the logs help. I saw similar messages when some files didn’t have the right permissions. As this is intermittent, I’ve ruled out permissions:

Example:
10...* - - [02/Sep/2020:11:08:59 +0100] “GET /rt/Admin/Lifecycles/ HTTP/1.1” 200 38349
10...* - - [02/Sep/2020:11:09:08 +0100] “GET /rt/Admin/Lifecycles/Modify.html?Type=ticket&Name=countermeasures HTTP/1.1” 500 547

[Wed Sep 02 11:15:08.204053 2020] [fcgid:warn] [pid 22862] [client 10.*.*.*:64133] mod_fcgid: error reading data, FastCGI server closed connection, referer: https://*******/rt/Admin/Lifecycles/
[Wed Sep 02 11:15:08.204180 2020] [core:error] [pid 22862] [client 10.*.*.*:64133] End of script output before headers: rt-server.fcgi, referer: https://*******/rt/Admin/Lifecycles/

Is it common for doing some task or visiting some specific pages in RT? If its just random then maybe the servers memory is filling up?

Thanks for your reply. It’s completely random but I am now focussed on the server config itself, as I’ve noticed stopping apache can take a while and result in it timing out and being killed.

I’ve also reviewed atop logs and noticed yesterday that multiple rt-server.fcgi proceses were started that resulted in all swap memory being consumed before the system killed them all off.

I’ll look to tweak the mod_fcgid.conf, and see if other modules are conflicting as I do have php running, but that doesn’t experience any issues and will continue to run whilst I experience issues with RT. I’m currently using PHP for Webmail that I eventually want RT to replace!

A new development is Error 500 on CSS files:
[03/Sep/2020:10:29:10 +0100] “GET /rt/NoAuth/css/elevator-light/squished-07928e9017d9e4f24077f9c5aabcc235.css HTTP/1.1” 500 547

What do you have set for your MaxSpareServers for mpm_prefork.conf?

I don’t have a config set-up for MPM so it’s using defaults. According to Apache docs that would be

MaxSpareServers 10
MinSpareServers 5

But it may be the MaxRequestWorkers directive that needs lowering, as default it is set to 256?

FYI in terms of memory I have:
Total: 3.7GB & 1.8GB Free
Swap: 2GB with 1.7GB free