Rt-mailgate/fetchmail quietly rejects email for unkown reasons, maybe unapproved X-MDSPF-Result?

1

I am using fetchmail to forward emails into rt-mailgate. We have been recently receiving emails from msnpath.com that are refusing to be accepted through rt-mailgate. No Errors from syslog. Not sure how to find errors for rt-mailgate. The errors I receive from fetchmail look like this:

fetchmail: 2 messages (2 seen) for user@domain.com at mail.domain.com (2465905 octets).

No Errors. It does not do this with all the email, most go through, and then we get like one/two a day that are just quietly skipped. I started to tear into each email, and the only correlation I could find was a rejection from X-SPAM:

X-Spam-Processed: mail.domain.com, Thu, 11 May 2017 12:48:25 -0500
(not processed: message size (1178472) exceeds spam filter configured max size of (204800))
X-MDDKIM-Result: unapproved (mail.domain.com)
X-MDSPF-Result: unapproved (mail.domain.com)

Now, if I forward this email right back into the same mailbox, THAT email does get processed. It’s X_SPAM response states:

X-Spam-Processed: mail.domain.com, Fri, 12 May 2017 12:35:20 -0500
(not processed: message from trusted or authenticated source)

… and nothing else. A good email that also gets processed looks like this:

X-Spam-Processed: mail.domain.com, Fri, 12 May 2017 10:26:30 -0500
X-MDOP-RefID: str=0001.0A020206.5915D423.007F,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0 (_st=1 _vt=0 _iwf=0)
X-MDSPF-Result: neutral (mail.domain.com)

FYI, this is what the fetchmail entry look like:

poll mail.domain.com
protocol pop3
username user@domain.com
password ************************************
mda “/usr/bin/perl /usr/bin/rt-mailgate --url https://localhost:3773 --queue RT_QUEUE --no-verify-ssl --action correspond”
is host_server
no rewrite
no flush
no idle
ssl
sslfingerprint “09:02:4B:F8:C4:DE:C8:9E:D1:BE:D4:98:F2:C5:9A:0D”
sslcertpath /etc/ssl/certs
fetchsizelimit 0

So, my speculation is that rt-mailgate is quietly dropping these unapproved X-MDSPF and X-MDDKIM emails. Am I correct in this? Is there something else I should look for. can I get more debuglike errors from rt-mailgate?

Thanks,
Eric

2

It looks to me like the email doesn’t even get to mailgate.

(not processed: message size (1178472) exceeds spam filter configured max size of (204800))

Looks like your spam filter rejects messages bigger than 204800 when the sender is from an unauthenticated source and so mailgate would never see the message.

I think the solution is to reconfigure the spam filter to allow bigger messages from unapproved sources.

3

The Spam filter is not initiated on mail gate. The email goes thru our spam filter before it dumps into our mail server. I can pull up our email from our private server via GUI. That how I am able to read the header information.

Mailgate is POPing the email from our private server. There is no spam filter between POP and mailgate. Hence above, fetchmail runs:

mda "/usr/bin/perl /usr/bin/rt-mailgate --url https://localhost:3773 --queue RT_QUEUE --no-verify-ssl --action

The point is, I suspect that when mailgate reads headers from an email that state:
X-MDDKIM-Result: unapproved (mail.domain.com)
X-MDSPF-Result: unapproved (mail.domain.com)

mailgate must be quietly skipping them, because the email is still in our GUI. If I forward the email right back onto itself, there are no messages of “unapproved”, hence mailgate takes it.

Here is another example of an unapproved email with a different fail:

X-Spam-Level:
X-Spam-Status: No, score=0.0 required=5.0 tests=HTML_MESSAGE,SPF_PASS,
T_DKIM_INVALID shortcircuit=no autolearn=disabled version=3.4.1
X-Spam-Report:
* -0.0 SPF_PASS SPF: sender matches SPF record
* 0.0 HTML_MESSAGE BODY: HTML included in message
* 0.0 T_DKIM_INVALID DKIM-Signature header exists but is not valid
X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28)
Authentication-Results: mail.domain.com
spf=pass smtp.mailfrom=some_user@msnpath.com;
dkim=pass (good signature) header.d=msnpath.onmicrosoft.com header.b=Du9IFTHZuq;
iprev=pass policy.iprev=104.47.33.46 (PTR mail-bn3nam01on0046.outbound.protection.outlook.com);
iprev=fail policy.iprev=104.47.33.46 reason=“does not match” (HELO NAM01-BN3-obe.outbound.protection.outlook.com);
iprev=fail policy.iprev=104.47.33.46 reason=“does not match” (MAIL some_user@msnpath.com)
Received-SPF: pass (mail.domain.com: domain msnpath.com
designates 104.47.33.46 as permitted sender)
receiver=mail.domain.com; client-ip=104.47.33.46;
mechanism=ip4:104.47.0.0/17; envelope-from="some_user@msnpath.com";
helo=NAM01-BN3-obe.outbound.protection.outlook.com;
Received: from NAM01-BN3-obe.outbound.protection.outlook.com (mail-bn3nam01on0046.outbound.protection.outlook.com [104.47.33.46])
by mail.domain.com with ESMTPS id md50000451319.msg;
Thu, 18 May 2017 09:27:40 -0500
X-MDOP-RefID: str=0001.0A020205.591DAF58.0151,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0 (_st=1 _vt=0 _iwf=0)
X-MDDKIM-Result: unapproved (mail.domain.com)
X-MDSPF-Result: unapproved (mail.domain.com)

4

Understood about the mail making it as far as the inbox so the issue must be after that in the process. But what you say above just seems unlikely. Stock RT anyways just doesn’t do header filtering like that. Which RT extensions do you have installed?

To confirm whether RT will take the mail or not, what you can do is on the server save the full email to a file and then pipe it directly to mailgate:

cat email.txt | /usr/bin/perl /usr/bin/rt-mailgate --url https://localhost:3773 --queue RT_QUEUE --no-verify-ssl --action correspond

You can add --debug to the mailgate command to get debugging from it, and you can add the following to RT_SiteConfig.pm to get debugging from RT itself in to its own file:

Set($LogToFile, 'debug');
Set($LogDir,    '/opt/rt4/var/log'); # or wherever
Set($LogToFileNamed, 'rt.log'); # or whatever

The file will need to be writable by whatever setup you have RT running as, and you’ll need to restart for this to take affect. And you won’t want to leave this on very long in a production setup.

5

Thanks for the reply! Things are become very bizarre…

So I did what you requested, stored a single email inside of a text file, and piped that into the command. BTW, same command and switches that I use for fetchmail:

cat email.txt | /usr/bin/perl /usr/bin/rt-mailgate --url https://localhost:3773 --queue RT_QUEUE --no-verify-ssl --action correspond

The single email worked; loads in to RT just fine, no problems. Then, I delete that email that was just entered into RT, obviously.

Here is where it get’s weird. I wait a minute, for fetchmail to run again, and then it’s as if the flood gates have opened up and there are no filters…and the rest of the rejected email gets POPed, scanned, and loaded into RT as if nothing was ever wrong. But, it doesn’t stay that way, and eventually a couple emails get rejected again and end up staying in the email box.

Any ideas why the floodgates would open up after sending just one email, doesn’t matter which one. And why would the floodgates NOT stay open?

BTW, my snippet of my fetchmail is up above again. Thanks,
-Eric

6

Ok, so now I think I’ve pin pointed the issue a bit more. Like before some email are retrieved by fetchmail, and some are not. If I delete any one of the un-retrieved emails, then the rest of the un-retrieved emails get retrieved and processed by fetchmail.

This is with the latest version of fetchmail 6.3.26

I’ll see what I can find and report back.