RT5 high RAM usage

ALone · November 4, 2020, 4:43pm

Case

After upgrading from RT4.4.4 to RT5 there seems to be a problem with the RAM usage of the rt-server.fcgi process.

The host server was built from scratch and the database data were exported and then imported to the new machine. Both of the instances are deployed using Apache, but RT 4.4.4 was using mod_fastcgi instead of mod_fcgid that the new machine is using now. The setup was working fine for a month or so but suddenly it started to use way more RAM.

Restarting Apache or the host machine resets the RAM but it goes to 88% after reading 2-3 tickets in the Web UI

During the day Apache crashes and produces server error 500, and then recovers on it’s own until new requests crashes it again.

Technical info below

I tried looking every log possible but i cannot find anything that is related to the high RAM usage.

I enabled debug logging for Apache and RT, still no luck.

I am running MPM_Prefork and i have disabled MPM_Event.

Ubuntu server 20.04
MySQL version 5.7.31

Below is apache configuration :

ServerName rt.example.com

PidFile ${APACHE_PID_FILE}

Timeout 300

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 5

User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

HostnameLookups Off

ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel debug

IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

Include ports.conf

<Directory />
	Options FollowSymLinks
	AllowOverride None
	Require all denied
</Directory>

<Directory /usr/share>
	AllowOverride None
	Require all granted
</Directory>

<Directory /var/www/>
	Options Indexes FollowSymLinks
	AllowOverride None
	Require all granted
</Directory>

AccessFileName .htaccess

<FilesMatch "^\.ht">
	Require all denied
</FilesMatch>

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

IncludeOptional conf-enabled/*.conf

IncludeOptional sites-enabled/*.conf

ServerSignature Off
ServerTokens Prod

Below is my rt virtual site configuration :

<VirtualHost *:80>
   ServerName rt.example.com
   DocumentRoot "/opt/rt5/share/html"
   Redirect permanent / https://rt.example.com/
</VirtualHost>

<VirtualHost _default_:443>
        ### Optional apache logs for RT
        # Ensure that your log rotation scripts know about these files
        # ErrorLog /opt/rt5/var/log/apache2.error
        # TransferLog /opt/rt5/var/log/apache2.access
        # LogLevel debug

AddDefaultCharset UTF-8

ScriptAlias / /opt/rt5/sbin/rt-server.fcgi/

DocumentRoot "/opt/rt5/share/html"

ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

SSLEngine on
SSLCertificateFile /etc/ssl/certs/mycert.pem
SSLCertificateKeyFile /etc/ssl/certs/mycert2.key

SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1

<FilesMatch "\.(cgi|shtml|phtml|php)$">
    SSLOptions +StdEnvVars
</FilesMatch>

<Directory /usr/lib/cgi-bin>
    SSLOptions +StdEnvVars
</Directory>

<Location />
    #Order allow,deny
    #Allow from all
    Require all granted
    Options +ExecCGI
    AddHandler fcgid-script fcgi
</Location>

Below is my MPM_prefork configuration (Also note that i tried several numbers for this, with no changes) :

 <IfModule mpm_prefork_module>
        StartServers            5
        MinSpareServers         5
        MaxSpareServers         10
        MaxRequestWorkers       150
        MaxConnectionsPerChild  0
</IfModule>

Below output of command “apache2 -V”

[Wed Nov 04 18:38:43.988039 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Wed Nov 04 18:38:43.988635 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_USER} is not defined
[Wed Nov 04 18:38:43.988651 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_GROUP} is not defined
[Wed Nov 04 18:38:43.988680 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.110293 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
[Wed Nov 04 18:38:44.123968 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152041 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152079 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2020-08-12T19:46:17
Server's Module Magic Number: 8237482734:90
Server loaded:  APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/apache2"
 -D SUEXEC_BIN="/usr/lib/apache2/suexec"
 -D DEFAULT_PIDLOG="/var/run/apache2.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="mime.types"
 -D SERVER_CONFIG_FILE="apache2.conf"

Below are screenshots of the RAM and how it is used in case you can observe anything else :

knation · November 4, 2020, 5:35pm

Is it the rt-server process that is overly large or is it the mysql process?

ALone · November 4, 2020, 5:56pm

As i understand it from the htop and webmin’s process view it is rt-server. Also the mysql’s process list is empty or has 3 to 4 idle processes.

Marcant · November 11, 2020, 11:19am

I observed a similar behaviour. The system fills up the available Ram very fast and then somewhat stablilizes on a high level until stuff stars to crash after some days because out of memory errors.
Just checked, apache occupied ~55% of the available ram right now (at 6GB total).

I have for now prevented crashes by doing a apachectl graceful via cron each night. Got no time until now to look further into this matter and wanted to switch towards nginx first (new company standart).

Marcant · December 8, 2020, 10:38am

I wrote a little quick&dirty Bash-script to monitor the issue and restart apache before stuff starts to crash.

#!/bin/bash
MEM=$(free | grep Mem | awk '{printf "%.0f", ($3/$2)*100}')
if [ $MEM -gt 85 ]
    then
        RESULT=$(apachectl configtest 2>&1)

        if [ "$RESULT" == "Syntax OK" ]
             then
                apachectl graceful
             exit 0;
        else
            echo "Probleme mit der httpd Konfiguration vom RT:"
            echo $RESULT
            exit 2;
        fi
fi

This runs now every 5 minutes via cron.

ALone · January 26, 2021, 5:24pm

Following more troubleshooting, i never found a real solution. The workaround i found was to disable the admin menu for everyone except root, and remove the right “do anything and everything” from everyone except root. I think hiding the admin menu will suffice but give it a try. In conclusion there is something going on in the admin menu, if at least one user sees it, my RAM goes up.

ALone · December 19, 2023, 2:50pm

My running RT version is 5.0.3

Following even more troubleshooting, I concluded that the issue was on Apache MPM_prefork configuration. I followed this guide to calculate the correct values needed for my system. As I understand, the real reason was that MPM_prefork was not using the extra RAM that was provided to the server. This is now working reliably for over a year with increased performance. FYI, I did not revert all the previous changes mentioned on the previous posts. There is also an automated script called apache2buddy that can calculate the MPM_prefork values for you, but it did not work reliably for me and the server became even slower, you can find the script here.

Please keep in mind that if you configure these values wrong, the Apache web server will not serve anything. Keep a backup of your current values!!

The file that you need to edit can be found at the path below:
/etc/apache2/mods-available/mpm_prefork.conf

To apply the changes

Disable the MPM_prefork module with the command below:
a2dismod mpm_prefork
Enable the MPM_prefork module again with the command below:
a2enmod mpm_prefork
Check that the files below are identical:
/etc/apache2/mods-available/mpm_prefork.conf
/etc/apache2/mods-enabled/mpm_prefork.conf
Stop and start the Apache server instead of restarting; otherwise, new settings are ignored:
apache2ctl -k stop
apache2ctl -k start