Case
After upgrading from RT4.4.4 to RT5 there seems to be a problem with the RAM usage of the rt-server.fcgi process.
The host server was built from scratch and the database data were exported and then imported to the new machine. Both of the instances are deployed using Apache, but RT 4.4.4 was using mod_fastcgi instead of mod_fcgid that the new machine is using now. The setup was working fine for a month or so but suddenly it started to use way more RAM.
Restarting Apache or the host machine resets the RAM but it goes to 88% after reading 2-3 tickets in the Web UI
During the day Apache crashes and produces server error 500, and then recovers on it’s own until new requests crashes it again.
Technical info below
I tried looking every log possible but i cannot find anything that is related to the high RAM usage.
I enabled debug logging for Apache and RT, still no luck.
I am running MPM_Prefork and i have disabled MPM_Event.
Ubuntu server 20.04
MySQL version 5.7.31
Below is apache configuration :
ServerName rt.example.com
PidFile ${APACHE_PID_FILE}
Timeout 300
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}
HostnameLookups Off
ErrorLog ${APACHE_LOG_DIR}/error.log
LogLevel debug
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf
Include ports.conf
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>
<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>
<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>
AccessFileName .htaccess
<FilesMatch "^\.ht">
Require all denied
</FilesMatch>
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent
IncludeOptional conf-enabled/*.conf
IncludeOptional sites-enabled/*.conf
ServerSignature Off
ServerTokens Prod
Below is my rt virtual site configuration :
<VirtualHost *:80>
ServerName rt.example.com
DocumentRoot "/opt/rt5/share/html"
Redirect permanent / https://rt.example.com/
</VirtualHost>
<VirtualHost _default_:443>
### Optional apache logs for RT
# Ensure that your log rotation scripts know about these files
# ErrorLog /opt/rt5/var/log/apache2.error
# TransferLog /opt/rt5/var/log/apache2.access
# LogLevel debug
AddDefaultCharset UTF-8
ScriptAlias / /opt/rt5/sbin/rt-server.fcgi/
DocumentRoot "/opt/rt5/share/html"
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLCertificateFile /etc/ssl/certs/mycert.pem
SSLCertificateKeyFile /etc/ssl/certs/mycert2.key
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
<Location />
#Order allow,deny
#Allow from all
Require all granted
Options +ExecCGI
AddHandler fcgid-script fcgi
</Location>
Below is my MPM_prefork configuration (Also note that i tried several numbers for this, with no changes) :
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 150
MaxConnectionsPerChild 0
</IfModule>
Below output of command “apache2 -V”
[Wed Nov 04 18:38:43.988039 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_PID_FILE} is not defined
[Wed Nov 04 18:38:43.988635 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_USER} is not defined
[Wed Nov 04 18:38:43.988651 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_GROUP} is not defined
[Wed Nov 04 18:38:43.988680 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.110293 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
[Wed Nov 04 18:38:44.123968 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152041 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
[Wed Nov 04 18:38:44.152079 2020] [core:warn] [pid 26005] AH00111: Config variable ${APACHE_LOG_DIR} is not defined
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2020-08-12T19:46:17
Server's Module Magic Number: 8237482734:90
Server loaded: APR 1.6.5, APR-UTIL 1.6.1
Compiled using: APR 1.6.5, APR-UTIL 1.6.1
Architecture: 64-bit
Server MPM: prefork
threaded: no
forked: yes (variable process count)
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/apache2"
-D SUEXEC_BIN="/usr/lib/apache2/suexec"
-D DEFAULT_PIDLOG="/var/run/apache2.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="mime.types"
-D SERVER_CONFIG_FILE="apache2.conf"
Below are screenshots of the RAM and how it is used in case you can observe anything else :