Help - "Deep Recursion on subroutine..."

Hi All

I’m getting this error:

Deep recursion on subroutine “RT::I18N::SetMIMEEntityToEncoding” at
/opt/rt4/sbin/…/lib/RT/I18N.pm

in my logs, and I’m out of RAM. I’m also getting a lot of bounces:

Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)

I’ve looked for loops in the RT_SiteConfig file (OwnerEmail) but see
nothing.

I’ve reinstalled the machine 4 times siince the problem started, on all
combinations of:

  • RT 3.8.9 (original)
  • RT 4.0.11
  • 512MB - 1GB RAM
  • CentOS 5.x - Centos 6.x

And the problem persists. RT was originally running for about a year
without a single problem or error on a CentOS 5 box with 512MB RAM.
Problems started when I migrated to CentOS 6 (Same RT 3.8.9), and have
persisted ever since.

I’m not sure what else to check, where to look, or how to troubleshoot any
further. Any help would be appreciated.

Regards,

Jason

I’m getting this error:

Deep recursion on subroutine “RT::I18N::SetMIMEEntityToEncoding” at
/opt/rt4/sbin/…/lib/RT/I18N.pm

in my logs, and I’m out of RAM. I’m also getting a lot of bounces:

Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)

I’ve looked for loops in the RT_SiteConfig file (OwnerEmail) but see
nothing.

This implies a very deep mail loop that is getting larger and larger and
hitting recursion limits when processing each successively larger bounce.

RT should be logging at the [crit] level the message IDs. Can you find
those and provide them?

Your RT_SiteConfig.pm would be good to provide too.

Folks can’t help without these details.

Sorry for the oversight, RT_SiteConfig::

================================== Start ==========================

Set($rtname , “IT Support”);
Set($Organization , “obscured.com”);
Set($MinimumPasswordLength , “5”);
Set($Timezone , ‘Africa/Johannesburg’);

Set($HomepageComponents, [qw(
QuickCreate
Quicksearch
MyAdminQueues
MySupportQueues
MyReminders
RefreshHomepage
Dashboards
)]);

Set($DatabaseType , ‘mysql’);
Set($DatabaseHost , ‘localhost’);
Set($DatabaseRTHost , ‘localhost’);
Set($DatabasePort , ‘’);
Set($DatabaseUser , ‘someuser’);
Set($DatabasePassword , ‘reallyreallystrongpassword’);
Set($DatabaseName , ‘rt4’);

#Set($OwnerEmail , ‘IT Support’);
Set($OwnerEmail , ‘jason@obscured.com’);
Set($LoopsToRTOwner , 1);

Set($SendmailArguments , “-oi -t -f support@obscured.com”);

Set($MaxAttachmentSize , 10000000);

#Set($RTAddressRegexp , ‘(^rt@obscured.com$)(.@obscured.co.za)');
Set($RTAddressRegexp , '.
@obscured.co.za’);
Set($CorrespondAddress , ‘no-reply@obscured.co.za’);
Set($CommentAddress , ‘no-reply@obscured.co.za’);

Set($UseFriendlyFromLine , 1);
Set($FriendlyFromLineFormat , “"%s" <%s>”);
Set($UseFriendlyToLine , 1);
Set($FriendlyToLineFormat, “"%s Ticket #%s":;”);

Set($NotifyActor, 0);
Set($RecordOutgoingEmail, 1);

Set($WebPath , “/customerzone”);
Set($WebPort , 443);
Set($WebBaseURL , “https://obscured.com”);
Set($WebURL , $WebBaseURL . $WebPath . “/”);

Set($CanonicalizeRedirectURLs, 1);

Set($MessageBoxWidth , 72);
Set($MessageBoxWrap, “HARD”);

Set($MaxInlineBody, 13456);
Set($DefaultSummaryRows, 10);

Set($OldestTransactionsFirst, ‘1’);
Set($ShowTransactionImages, 1);

Set($DateDayBeforeMonth , 0);
Set($AmbiguousDayInPast , 1);

Set($LogDir, ‘/var/log/rt4’);

1;
====================================== END ==============================

This is a sample of the email error I get (please let me know if you need
more log info):

[Fri Apr 26 20:17:12 2013] [critical]: RT Received mail (<
20130426001825.08C1521490@obscured.com>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Fri Apr 26 20:17:12 2013] [crit]: RT thinks this message may be a bounce
(/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:248)
[Fri Apr 26 20:17:14 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Fri Apr 26 20:17:40 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)

I am unsure how to get more information from the message id "
20130426001825.08C1521490@obscured.com" or whatever it is at that time. I
don’t find an equivalent message in themaillog.

Thanks for the prompt response,

JasonOn Fri, Apr 26, 2013 at 9:12 PM, Thomas Sibley trs@bestpractical.comwrote:

On 04/26/2013 10:01 AM, Jason Doller wrote:

I’m getting this error:

Deep recursion on subroutine “RT::I18N::SetMIMEEntityToEncoding” at
/opt/rt4/sbin/…/lib/RT/I18N.pm

in my logs, and I’m out of RAM. I’m also getting a lot of bounces:

Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)

I’ve looked for loops in the RT_SiteConfig file (OwnerEmail) but see
nothing.

This implies a very deep mail loop that is getting larger and larger and
hitting recursion limits when processing each successively larger bounce.

RT should be logging at the [crit] level the message IDs. Can you find
those and provide them?

Your RT_SiteConfig.pm would be good to provide too.

Folks can’t help without these details.

There are a number of possibilities given the config snippet below.

Jason wrote:

Set($SendmailArguments , “-oi -t -f support@obscured.com”);

Does support@obscured.com go into RT? -f sets the envelope-sender
address which is where bounces go. If it goes into RT, that’s
potentially a mail loop.

Set($MaxAttachmentSize , 10000000);

#Set($RTAddressRegexp , ‘(^rt@obscured.com$)(.@obscured.co.za)');
Set($RTAddressRegexp , '.
@obscured.co.za’);

The commented out $RTAddressRegexp (which I suppose is commented out
because it didn’t work; the regex syntax is incorrect) contains
rt@obscured.com, but the uncommented version of the option doesn’t match
that.

Does rt@obscured.com still go into RT? If so, that could be the source
of a loop if a ticket got a watcher of the rt@ address added. Try this
instead:

'^(rt@obscured\.com|.*@obscured\.co\.za)$'

Escaping the @ isn’t necessary in a single-quoted string, just double
quoted to prevent interpolation.

Additionally, the current $RTAddressRegexp should have the dots (.) in
.co.za escaped with a backslash, though that’s unlikely to cause a loop.

Set($CorrespondAddress , ‘no-reply@obscured.co.za’);
Set($CommentAddress , ‘no-reply@obscured.co.za’);

These values are strange, since normally those addresses are intended to
receive replies! However, they are unlikely to be a mail loop source
since they will match your RTAddressRegexp.

Thomas

I’ve fixed and checked all the potential problems listed, and the “support”
email address may have been the problem.

However, I’ve turned off postfix, and I’m still getting the errors below.
I realise I can probably clear a cache to stop this, but with postfix
turned off, nothing is injecting new mails into the system, I’m concerned
that this problem isn’t going away. I’ve also (temporarily) turned off
loop mail forwarding to owner…

Errors:

[Sat Apr 27 05:21:10 2013] [critical]: RT Received mail (<
20130425182344.6AEFA20EF4@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:21:10 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Sat Apr 27 05:24:27 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:24:28 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:24:42 2013] [critical]: RT Received mail (<
20130426052216.69E6721375@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:24:42 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Sat Apr 27 05:24:43 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:24:43 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:24:56 2013] [critical]: RT Received mail (<
20130425150731.6EF682DEBC@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:24:57 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Sat Apr 27 05:25:44 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:25:44 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:25:57 2013] [critical]: RT Received mail (<
20130426105238.62C7721453@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:25:58 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Sat Apr 27 05:26:19 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:26:19 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:26:28 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:26:29 2013] [warning]: Deep recursion on subroutine
“RT::I18N::SetMIMEEntityToEncoding” at /opt/rt4/sbin/…/lib/RT/I18N.pm line
210. (/opt/rt4/sbin/…/lib/RT/I18N.pm:210)
[Sat Apr 27 05:26:32 2013] [critical]: RT Received mail (<
20130426051826.676422F213@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:26:32 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)
[Sat Apr 27 05:26:43 2013] [critical]: RT Received mail (<
20130426045919.6D40E2107B@ingress01.customermanager.co.za>
) from itself. (/opt/rt4/sbin/…/lib/RT/Interface/Email.pm:1846)
[Sat Apr 27 05:26:43 2013] [error]: Could not record email: Message Bounced
(/opt/rt4/share/html/REST/1.0/NoAuth/mail-gateway:75)

Thanks for the help so far,

JasonOn Sat, Apr 27, 2013 at 1:35 AM, Thomas Sibley trs@bestpractical.comwrote:

There are a number of possibilities given the config snippet below.

Jason wrote:

Set($SendmailArguments , “-oi -t -f support@obscured.com”);

Does support@obscured.com go into RT? -f sets the envelope-sender
address which is where bounces go. If it goes into RT, that’s
potentially a mail loop.

Set($MaxAttachmentSize , 10000000);

#Set($RTAddressRegexp , ‘(^rt@obscured.com$)(.@obscured.co.za)');
Set($RTAddressRegexp , '.
@obscured.co.za’);

The commented out $RTAddressRegexp (which I suppose is commented out
because it didn’t work; the regex syntax is incorrect) contains
rt@obscured.com, but the uncommented version of the option doesn’t match
that.

Does rt@obscured.com still go into RT? If so, that could be the source
of a loop if a ticket got a watcher of the rt@ address added. Try this
instead:

'^(rt@obscured\.com|.*@obscured\.co\.za)$'

Escaping the @ isn’t necessary in a single-quoted string, just double
quoted to prevent interpolation.

Additionally, the current $RTAddressRegexp should have the dots (.) in
.co.za escaped with a backslash, though that’s unlikely to cause a loop.

Set($CorrespondAddress , ‘no-reply@obscured.co.za’);
Set($CommentAddress , ‘no-reply@obscured.co.za’);

These values are strange, since normally those addresses are intended to
receive replies! However, they are unlikely to be a mail loop source
since they will match your RTAddressRegexp.

Thomas

I’ve fixed and checked all the potential problems listed, and the
“support” email address may have been the problem.

However, I’ve turned off postfix, and I’m still getting the errors
below. I realise I can probably clear a cache to stop this, but with
postfix turned off, nothing is injecting new mails into the system, I’m
concerned that this problem isn’t going away. I’ve also (temporarily)
turned off loop mail forwarding to owner…

For those errors to pop up in your logs again, something is submitting
mail to RT.

Look at your access logs for hits to /REST/1.0/NoAuth/mail-gateway.

Postfix was definitely off, and nothing was injecting into the mailgateway,
but it seems that the errors needed a few hours - most of a 24 hour day in
fact - to go away. Unfortunately I wasn’t paying too much attention, as I
assumed that something was stuck in an internal loop :frowning:

I’m still working on the problem, but so far it seems that it may have been
caused by an external mail loop (rt sending to an external user who was
forwarding to another email address that was forwarding to RT). But I’m
not sure. Still busy troubleshooting.

I’ll post success or failure of this testing phase to the list. Have also
migrated from apache to nginx - if the problem seems to be resolved I will
run with apache again for a while to ensure that the problem wasn’t somehow
related to the apache conf (but then, rt was running for a year with the
apache conf so I don’t think that will be the issue).

JasonOn Tue, Apr 30, 2013 at 12:19 AM, Thomas Sibley trs@bestpractical.comwrote:

On 04/26/2013 10:28 PM, Jason Doller wrote:

I’ve fixed and checked all the potential problems listed, and the
“support” email address may have been the problem.

However, I’ve turned off postfix, and I’m still getting the errors
below. I realise I can probably clear a cache to stop this, but with
postfix turned off, nothing is injecting new mails into the system, I’m
concerned that this problem isn’t going away. I’ve also (temporarily)
turned off loop mail forwarding to owner…

For those errors to pop up in your logs again, something is submitting
mail to RT.

Look at your access logs for hits to /REST/1.0/NoAuth/mail-gateway.