RT5 high RAM usage

Case

After upgrading from RT4.4.4 to RT5 there seems to be a problem with the RAM usage of the rt-server.fcgi process.

The host server was built from scratch and the database data were exported and then imported to the new machine. Both of the instances are deployed using Apache, but RT 4.4.4 was using mod_fastcgi instead of mod_fcgid that the new machine is using now. The setup was working fine for a month or so but suddenly it started to use way more RAM.

Restarting Apache or the host machine resets the RAM but it goes to 88% after reading 2-3 tickets in the Web UI

During the day Apache crashes and produces server error 500, and then recovers on it’s own until new requests crashes it again.

Technical info below

I tried looking every log possible but i cannot find anything that is related to the high RAM usage.

I enabled debug logging for Apache and RT, still no luck.

I am running MPM_Prefork and i have disabled MPM_Event.

Ubuntu server 20.04
MySQL version 5.7.31

Below is apache configuration :

ServerName rt.example.com

PidFile ${APACHE_PID_FILE}

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 5

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

HostnameLookups Off

ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel debug

IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

Include ports.conf

<Directory />
	Options FollowSymLinks
	AllowOverride None
	Require all denied
</Directory>

<Directory /usr/share>
	AllowOverride None
	Require all granted
</Directory>

<Directory /var/www/>
	Options Indexes FollowSymLinks
	AllowOverride None
	Require all granted
</Directory>

AccessFileName .htaccess

<FilesMatch "^\.ht">
	Require all denied
</FilesMatch>

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

IncludeOptional conf-enabled/*.conf

IncludeOptional sites-enabled/*.conf

ServerSignature Off
ServerTokens Prod

Below is my rt virtual site configuration :

<VirtualHost *:80>
   ServerName rt.example.com
   DocumentRoot "/opt/rt5/share/html"
   Redirect permanent / https://rt.example.com/
</VirtualHost>

<VirtualHost _default_:443>
        ### Optional apache logs for RT
        # Ensure that your log rotation scripts know about these files
        # ErrorLog /opt/rt5/var/log/apache2.error
        # TransferLog /opt/rt5/var/log/apache2.access
        # LogLevel debug

AddDefaultCharset UTF-8

ScriptAlias / /opt/rt5/sbin/rt-server.fcgi/

DocumentRoot "/opt/rt5/share/html"

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

SSLEngine on
SSLCertificateFile /etc/ssl/certs/mycert.pem
SSLCertificateKeyFile /etc/ssl/certs/mycert2.key

SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

<Directory /usr/lib/cgi-bin>
    SSLOptions +StdEnvVars
</Directory>

<Location />
    #Order allow,deny
    #Allow from all
    Require all granted
    Options +ExecCGI
    AddHandler fcgid-script fcgi
</Location>

Below is my MPM_prefork configuration (Also note that i tried several numbers for this, with no changes) :

 <IfModule mpm_prefork_module>
        StartServers            5
        MinSpareServers         5
        MaxSpareServers         10
        MaxRequestWorkers       150
        MaxConnectionsPerChild  0
</IfModule>

Below output of command “apache2 -V”

[Wed Nov 04 18:38:43.988039 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Wed Nov 04 18:38:43.988635 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_USER} is not defined
[Wed Nov 04 18:38:43.988651 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_GROUP} is not defined
[Wed Nov 04 18:38:43.988680 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.110293 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
[Wed Nov 04 18:38:44.123968 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152041 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152079 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2020-08-12T19:46:17
Server's Module Magic Number: 8237482734:90
Server loaded:  APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

Below are screenshots of the RAM and how it is used in case you can observe anything else :

Is it the rt-server process that is overly large or is it the mysql process?

As i understand it from the htop and webmin’s process view it is rt-server. Also the mysql’s process list is empty or has 3 to 4 idle processes.

I observed a similar behaviour. The system fills up the available Ram very fast and then somewhat stablilizes on a high level until stuff stars to crash after some days because out of memory errors.
Just checked, apache occupied ~55% of the available ram right now (at 6GB total).

I have for now prevented crashes by doing a apachectl graceful via cron each night. Got no time until now to look further into this matter and wanted to switch towards nginx first (new company standart).