RT 4.4 FastCGI processes frequently dying

1

We recently upgraded our RT instance from Debian Jessie to Debian Stretch, and with it upgraded from RT 4.2.13 to 4.4.1. We’re using nginx with the rt4-fcgi service as provided by the Debian packages.

Ever since the upgrade, we’ve had occasional reports of users getting a Bad gateway error from nginx when they try to post an update to a ticket, and according to syslog the rt4-fcgi service died and had to be restarted, but there are no obvious error messages from rt4-fcgi itself.

I did a quick strace of the rt4-fcgi process and that revealed that the processes were dying with SIGPIPE when they try to write to their fastcgi socket:

16562 write(5, "\1\6\0\1\0\0\0\0\1\3\0\1\0\10\0\0\0\0\0\0\0\0\0\0", 24) = -1 EPIPE (Broken pipe)
16562 --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=16562, si_uid=33} ---
16562 +++ killed by SIGPIPE +++

Does anyone have any clues as to what might be causing this to happen, or any thoughts on additional troubleshooting steps we can follow to try and track down the issue?

2

I have the same problem, and so far we are unable to determine the exact cause.

Increasing the amount of RAM available on the VM seemed to help a little - it dies less often.

Other than that, we just added a cron job to restart when it dies. Not ideal, but it at least fixes it when it occurs.

# In /etc/cron.d/request-tracker4. Check if RT is running, if not restart:
*/5 * * * *  root /var/scripts/monitor rt4-fcgi


#!/bin/bash

service=$@
/bin/systemctl -q is-active "$service.service"
status=$?
if [ "$status" == 0 ]; then
    #echo "OK"
    EX=0
else
    echo "Restarting service: $service"
    /bin/systemctl start "$service.service"
fi
3

Could this be a nginx timeout? You could try increasing some of the nginx timeout variables:
https://www.scalescale.com/tips/nginx/504-gateway-time-out-using-nginx/

4

Does this happen if you do a large query like id > 0?

5

We’re not setting a fastcgi timeout so this would be set to the default of 60 seconds. However, the users who have reported getting an error when they try to post a ticket update have reported that the error comes up within a second or two, they’re not sitting there for a minute before it times out. If it timed out I’d also expect to see 504 error responses in the nginx logs but there are none.

Looking at the logs though I did notice that all of the 502 errors from /Ticket/Update.html were preceded by a request to /Helpers/PreviewScrips that was recorded with a 499 request status. This is an internal nginx status code that indicates that the client dropped the connection before the request was completed:

[05/Mar/2020:10:50:24 +0000] "POST /Helpers/PreviewScrips HTTP/1.1" 499
[05/Mar/2020:10:50:24 +0000] "POST /Ticket/Update.html HTTP/1.1" 502
[09/Mar/2020:14:28:49 +0000] "POST /Helpers/PreviewScrips HTTP/1.1" 499
[09/Mar/2020:14:28:49 +0000] "POST /Ticket/Update.html HTTP/1.1" 502
[09/Mar/2020:16:12:28 +0000] "POST /Helpers/PreviewScrips HTTP/1.1" 499
[09/Mar/2020:16:12:28 +0000] "POST /Ticket/Update.html HTTP/1.1" 502
[10/Mar/2020:15:39:41 +0000] "POST /Helpers/PreviewScrips HTTP/1.1" 499
[10/Mar/2020:15:39:42 +0000] "POST /Ticket/Update.html HTTP/1.1" 502

This seems consistent with the advice from the FastCGI FAQ which indicates that a SIGPIPE will be sent to a FastCGI application if the client aborts the connection. It looks like RT is not installing a SIGPIPE handler so the FastCGI process is dying whenever a connection is dropped by the client.

6

Are there any errors in the RT logs that coincide with the timing of the 502 errors?

7

No, there are no errors being produced by RT. I believe the 502 errors are being generated by nginx because there are no RT FastCGI processes running at the time to process the request.

Looking at the Javascript in Update.html the aborted requests to /Helpers/PreviewScrips do make some sense. I can see that it triggers an AJAX query for this helper in response to the blur event on the UpdateContent element. Depending on the exact timing of that the user could end up blurring the content update input just before submitting the form, resulting in the AJAX call being aborted by the form submission.

8

Hey,

We’ve just upgraded from 4.4.1 to 4.4.4 and we’re experiencing the same issue on a very regular basis. Other than “restart via cron”, has anyone worked out a better solution?

Cheers,
Andrew

9

Experiencing these too an RT 5.0.1. Managed to “hide” the SIGPIPE errors by running my RT fcgi process like this: /opt/rt5/sbin/rt-server.fcgi --listen /var/tmp/rt-server.sock --nproc 4 With these config RT seems to restart pretty fast after a SIGPIPE error. (User are only experiencing slow UI when this happens, not a user visible error)

Also seeing frequent could not receive data from client: Connection reset by peer in postgres log correlating with the SIGPIPE errors.

10

Interesting.

We’re using the Debian packages with systemd and created the file /etc/systemd/system/rt4-fcgi.service.d/override.conf with the following contents:

[Service]
Restart=always

This, along with running 6 processes, is hiding the issue for us. Looks like we’ll need this once we get to RT5 as well.

11

I couldn’t reproduce this somehow. I’m wondering if it’s because of different versions of some modules.
Could you share your perl module info(i.e. the “Loaded perl modules” section on /Admin/Tools/Configuration.html) so I can investigate?

Thanks!

Regards
sunnavy

12

I’m happy to provide my configuration information directly.

13

We used ‘multiwatch’ with spawn-fcgi to handle this problem.

Regards,
Ken