Sendmailpipe returns EX_TEMPFAIL

Greetings:

We are having an problem with Postfix sendmail returning “code 75”
(EX_TEMPFAIL) on outgoing emails from RT. I have been monitoring the
bug and can find no rhyme or reason to it. Most of the time, emails
leave RT without any trouble. But occasionally they fail
unreproducibly like this:

Aug 12 08:01:16 perelman RT:
rt-4.0.13-29527-1376287275-1791.35212-4-0@suse.de #35212/333905 -
Scrip 4 On Create Notify AdminCcs
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:285)
Aug 12 08:01:16 perelman RT:
rt-4.0.13-29527-1376287275-1791.35212-4-0@suse.de:
/usr/sbin/sendmail -oi -t exited with code 75
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:486)
Aug 12 08:01:16 perelman RT:
rt-4.0.13-29527-1376287275-1791.35212-4-0@suse.de: Could not send
mail with command /usr/sbin/sendmail -oi -t:
rt-4.0.13-29527-1376287275-1791.35212-4-0@suse.de:
/usr/sbin/sendmail -oi -t exited with code 75 at
/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm line 487.
Stack: [/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:487]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:292]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Action/SendEmail.pm:114]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/ScripAction.pm:232]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Scrip.pm:475]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Scrips.pm:188]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Transaction.pm:201]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Record.pm:1504]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Ticket.pm:682]
[/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:1569]
[/usr/share/request-tracker/html/REST/1.0/NoAuth/mail-gateway:61]
(/usr/lib/perl5/vendor_perl/5.10.0/RT/Interface/Email.pm:491)

This is RT 4.0.13 and we are running it on a virtual machine with 8GB
of RAM. Free RAM is around 80%, so the problem is not memory-related.
There is one other mod_perl application running on the same machine,
but adding “PerlOptions +Parent” to the Apache vhost configuration
didn’t help. (Do I need to add it to the vhost configuration of the
other mod_perl app, too?)

In RT_SiteConfig.pm I have:

Set($SendmailCommand, ‘sendmailpipe’);
Set($SendmailArguments, ‘-oi -t’);

I’m running a fairly modern Postfix (v. 2.5.13) on a fairly modern OS
(SLES11-SP2). Here’s apache2ctl -V:

apache2ctl -V

Server version: Apache/2.2.12 (Linux/SUSE)
Server built: Mar 27 2013 18:47:49
Server’s Module Magic Number: 20051115:23
Server loaded: APR 1.3.3, APR-Util 1.3.4
Compiled using: APR 1.3.3, APR-Util 1.3.4
Architecture: 64-bit
Server MPM: Prefork
threaded: no
forked: yes (variable process count)
Server compiled with…
-D APACHE_MPM_DIR=“server/mpm/prefork”
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_SYSVSEM_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=128
-D HTTPD_ROOT=“/srv/www”
-D SUEXEC_BIN=“/usr/sbin/suexec2”
-D DEFAULT_PIDLOG=“/var/run/httpd2.pid”
-D DEFAULT_SCOREBOARD=“logs/apache_runtime_status”
-D DEFAULT_LOCKFILE=“/var/run/accept.lock”
-D DEFAULT_ERRORLOG=“/var/log/apache2/error_log”
-D AP_TYPES_CONFIG_FILE=“/etc/apache2/mime.types”
-D SERVER_CONFIG_FILE=“/etc/apache2/httpd.conf”

What appears to be happening is this: for reasons unknown,
occasionally when RT goes to send an email, the sendmailpipe command
(‘/usr/bin/sendmail -oi -t’) gets executed but nothing is piped in.
This causes sendmail to report:

postfix/sendmail[23352]: fatal: wwwrun(30): No recipient addresses
found in message header

and RT logs the “code 75” error quoted above. In the web UI, when I
look at the transaction “with headers”, it says:

X-RT-Interface: API
MIME-Version: 1.0
X-Mailer: MIME-tools 5.502 (Entity 5.502)
Content-Type: text/plain; charset=“utf-8”
Content-Disposition: inline
Content-Transfer-Encoding: binary
X-RT-Original-Encoding: ascii
RT-Message-ID: rt-4.0.13-29527-1376293428-636.35212-0-0@suse.de
Content-Length: 105

Sending the previous mail has failed. Please contact your admin, they
can find more details in the logs.

Yesterday I added the ‘-v’ switch to the sendmail command line, which
is causing mail delivery reports to be sent to a local user on the
machine. These mail delivery reports reproduce all the headers. I have
not managed to “capture” the bug in this way, yet, however. But as
soon as I do I will post the mail delivery report here.

Does anyone have any other ideas for debugging this issue? Especially
I am interested in how I could confirm or deny that it’s related to
mod_perl “cohabitation” – i.e. two different Perl applications in a
single mod_perl host?

Nathan

Does anyone have any other ideas for debugging this issue? Especially
I am interested in how I could confirm or deny that it’s related to
mod_perl “cohabitation” – i.e. two different Perl applications in a
single mod_perl host?

I noticed that when I don’t have “PerlOptions +Parent” in the apache
config, the Perl library search order on the System Configuration page
is different than when I do. Here’s what it looks like without
PerlOptions +Parent (omitting the line numbers which do not
copy-paste):

Perl library search order

/usr/share/request-tracker/local/lib
/usr/share/request-tracker/local/plugins/RT-Extension-MergeUsers/lib
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/srv/www/perl-lib
/srv/www/vhosts/pdb/
/usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
.
/srv/www

Notice lines 5 and 6 which obviously come from the other mod_perl application.

Now, here’s what it says with PerlOptions +Parent:

Perl library search order

/usr/share/request-tracker/local/lib
/usr/share/request-tracker/local/plugins/RT-Extension-MergeUsers/lib
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
.
/srv/www

So maybe the problem is solved? I did have PerlOptions +Parent in the
apache configuration before, but maybe not correctly? I checked the
other application’s vhosts file and it does have PerlOptions
+Parent.

Nathan

Does anyone have any other ideas for debugging this issue? Especially
I am interested in how I could confirm or deny that it’s related to
mod_perl “cohabitation” – i.e. two different Perl applications in a
single mod_perl host?

I noticed that when I don’t have “PerlOptions +Parent” in the apache
config, the Perl library search order on the System Configuration page
is different than when I do. Here’s what it looks like without
PerlOptions +Parent (omitting the line numbers which do not
copy-paste):

Perl library search order

/usr/share/request-tracker/local/lib
/usr/share/request-tracker/local/plugins/RT-Extension-MergeUsers/lib
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/srv/www/perl-lib
/srv/www/vhosts/pdb/
/usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
.
/srv/www

Notice lines 5 and 6 which obviously come from the other mod_perl application.

Now, here’s what it says with PerlOptions +Parent:

Perl library search order

/usr/share/request-tracker/local/lib
/usr/share/request-tracker/local/plugins/RT-Extension-MergeUsers/lib
/usr/lib/perl5/vendor_perl/5.10.0
/usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/site_perl/5.10.0
/usr/lib/perl5/vendor_perl
/usr/lib/perl5/5.10.0/x86_64-linux-thread-multi
/usr/lib/perl5/5.10.0
.
/srv/www

So maybe the problem is solved? I did have PerlOptions +Parent in the
apache configuration before, but maybe not correctly? I checked the
other application’s vhosts file and it does have PerlOptions
+Parent.

While PerlOptions +Parent is important if you need to run two mod_perl
apps, you should keep your log in place, and possibly dig up one of
the shims from the mailing list archives that has been used to debug
this.

In the past, it was bugs in mod_perl and the handler type, but it’s
nearly impossible to debug from the standard RT debug and mail logs.

-kevin

So maybe the problem is solved? I did have PerlOptions +Parent in the
apache configuration before, but maybe not correctly? I checked the
other application’s vhosts file and it does have PerlOptions
+Parent.

While PerlOptions +Parent is important if you need to run two mod_perl
apps, you should keep your log in place, and possibly dig up one of
the shims from the mailing list archives that has been used to debug
this.

In the past, it was bugs in mod_perl and the handler type, but it’s
nearly impossible to debug from the standard RT debug and mail logs.

Adding on to what Kevin said…

You could also just save yourself some time and switch to a different
deployment option for RT. Running it under mod_fastcgi or mod_fcgid is
simple and would avoid any mod_perl bugs with +Parent. Why fight with
mod_perl?