Automatted parsing of mails entering an RT queue

Hi all,

we’ve been using RT for almost 15 years now with great success, but our
growing company needs a little more automation now. As we are a hosting
company /carrier, we frequently receive abuse reports and security
advisories (for example, automatted scans for UDP amplifiers by the
German national CERT). These enter our abuse queue.

I would like to parse these mails automatically, and write a parsing
toolkit for each different type of abuse mail (either by sender, or by
specific content signature, or something like that), in order to extract
the affected URIs / IP addresses from the mails and pass them on to an
abuse handling script for further action.

How would I do that? Are there any articles in the RT wiki that might be
a good starting point? Unfortunately, the “automating RT” page is more
about crontool than about the kind of automation I’m looking for.

Thanks a lot,

–ck

Check RT-Extension-ExtractCustomFieldValues> Am 02.03.2017 um 09:35 schrieb Christopher Kunz chrislist@de-punkt.de:

Hi all,

we’ve been using RT for almost 15 years now with great success, but our
growing company needs a little more automation now. As we are a hosting
company /carrier, we frequently receive abuse reports and security
advisories (for example, automatted scans for UDP amplifiers by the
German national CERT). These enter our abuse queue.

I would like to parse these mails automatically, and write a parsing
toolkit for each different type of abuse mail (either by sender, or by
specific content signature, or something like that), in order to extract
the affected URIs / IP addresses from the mails and pass them on to an
abuse handling script for further action.

How would I do that? Are there any articles in the RT wiki that might be
a good starting point? Unfortunately, the “automating RT” page is more
about crontool than about the kind of automation I’m looking for.

Thanks a lot,

–ck

RT 4.4 and RTIR Training Sessions https://bestpractical.com/training

  • Paris - April 24-26, 2017

What version are you using CK?

-mOn Thu, Mar 2, 2017 at 2:35 AM, Christopher Kunz chrislist@de-punkt.de wrote:

Hi all,

we’ve been using RT for almost 15 years now with great success, but our
growing company needs a little more automation now. As we are a hosting
company /carrier, we frequently receive abuse reports and security
advisories (for example, automatted scans for UDP amplifiers by the
German national CERT). These enter our abuse queue.

I would like to parse these mails automatically, and write a parsing
toolkit for each different type of abuse mail (either by sender, or by
specific content signature, or something like that), in order to extract
the affected URIs / IP addresses from the mails and pass them on to an
abuse handling script for further action.

How would I do that? Are there any articles in the RT wiki that might be
a good starting point? Unfortunately, the “automating RT” page is more
about crontool than about the kind of automation I’m looking for.

Thanks a lot,

–ck

RT 4.4 and RTIR Training Sessions https://bestpractical.com/training

  • Paris - April 24-26, 2017

What version are you using CK?

-m

Hi,

we’re using 4.2.8.

Regards,

–ck

Hey CK,

We are also using 4.2.x.

For special parsing of mail, we use RT::Extension::FutureMailgate, which is the mailgate of 4.4.

In our /etc/aliases we set up an action for the email address:

some-user: “|/opt/rt4/bin/rt-mailgate --queue ‘Foo’ --action ‘VerifyAccessRequestRenewal’ --url https://rt.example.com

Here is our VerifyAccessRequestRenewal module…

cat lib/RT/Interface/Email/Action/VerifyAccessRequestRenewal.pm

package RT::Interface::Email::Action::VerifyAccessRequestRenewal;

use strict;
use warnings;

use Role::Basic 'with';
with 'RT::Interface::Email::Role';

=head1 NAME

RT::Interface::Email::Action::VerifyAccessRequestRenewal - Verify Access Request Renewal via the RT mailgateway.

=head1 SYNOPSIS

This plugin, if placed in L<RT_Config/@MailPlugins>, allows the mail
gateway to handle replies from requestors to verify access request renewals.

    | rt-mailgate --action VerifyAccessRequestRenewal --queue General --url http://localhost/

=cut

sub CheckACL {
    my %args = (
        Message     => undef,
        CurrentUser => undef,
        Ticket      => undef,
        Queue       => undef,
        Action      => undef,
        @_,
    );

    if ($args{Action} ne 'VerifyAccessRequestRenewal') {
        return;
    }

    my $mail_error_subject = 'Access Request Renewal NOT Verified';

    my $TicketObj = $args{Ticket};

    if ( ! $TicketObj->Id ) {
        MailError(
            Subject     => $mail_error_subject,
            Explanation => "Could not find a ticket with id ".$TicketObj->Id,
            FAILURE     => 1,
        );
    }

    if ( $TicketObj->Status ne 'activated') {
        MailError(
            Subject     => $mail_error_subject,
            Explanation => 'Ticket is not in a status that needs to have access renewed.',
            FAILURE     => 1,
        );
    }

    my $generated_token = RT::Site::UMN::Duluth::Form::AccessRequest::generate_token(
        ticket_id => $TicketObj->Id,
    );

    my $mime_entity = $args{Message};
    my @parts = $mime_entity->parts;
    PART:
    for my $part (@parts) {
        my $body = $part->bodyhandle;
        if (! defined $body) {
            next PART;
        }
        my $message = $body->as_string;
        if (! defined $message) {
            next PART;
        }
        if (my ($token_from_user) = $message =~ /access_renewal_verification_token=([\da-f]{32})[^\da-f]/i) {
            if ($generated_token eq $token_from_user) {
                # We found the token and it matches the one we generated
                # from the ticket. We're good to continue.
                return 1;
            }
            else {
                # It doesn't look like the user provided the correct token.
                # That is, they didn't respond to the correct email. Do not continue.
                MailError(
                    Subject     => $mail_error_subject,
                    Explanation => 'You did not reply to the most recent email requesting Access Renewal Verification.',
                    FAILURE     => 1,
                );
            }
        }

    }

    # We didn't parse out a token hash. Do not continue.
    MailError(
        Subject     => $mail_error_subject,
        Explanation => 'You did not reply to a email requesting Access Renewal Verification.',
        FAILURE     => 1,
    );
}

sub HandleVerifyAccessRequestRenewal {
    my %args = (
        Message     => undef,
        Ticket      => undef,
        Queue       => undef,
        @_,
    );

    my $TicketObj = $args{Ticket};
    my $mail_error_subject = 'Access Request Renewal NOT Verified';
    my $mail_error_message
        = 'Please contact '.RT->Config->Get('OrganizationTechnicalDepartmentNameShort')
        . ' at '.RT->Config->Get('OrganizationTechnicalDepartmentContact')
        . ' about this error.'
        . "\n\n"
        . 'The RT administrator should check the logs.'
    ;


    if ( ! $TicketObj->Id ) {
        RT->Logger->error('Could not find a ticket with id '.$TicketObj->Id);
        MailError(
            Subject     => $mail_error_subject,
            Explanation => $mail_error_message,
            FAILURE     => 1,
        );
    }

    my $system_user_TicketObj = RT::Ticket->new(RT->SystemUser);
    my ($ok, $msg) = $system_user_TicketObj->Load($TicketObj->Id);
    if (! $ok) {
        RT->Logger->error('Unable to load ticket, '.$TicketObj->Id.": $msg");
        MailError(
            Subject     => $mail_error_subject,
            Explanation => $mail_error_message,
            FAILURE     => 1,
        );
    }

    # For setting the CF value, we'll need to be the system user.
    $TicketObj = $system_user_TicketObj;

    my $cf = $TicketObj->LoadCustomFieldByIdentifier('Renewal Verified At');

    if (! $cf->Id) {
        RT->Logger->error('Unable to load Custom Field "Renewal Verified At"');
        MailError(
            Subject     => $mail_error_subject,
            Explanation => $mail_error_message,
            FAILURE     => 1,
        );
    }

    my $cf_values = $TicketObj->CustomFieldValues($cf->Id);
    if (! $cf_values->HasEntry('now')) {
        ($ok, $msg) = $TicketObj->AddCustomFieldValue(
            Field => $cf->Id,
            Value => 'now',
        );
        if ($ok) {
            # Success! We're done.
            return;
        }
        RT->Logger->error("Unable to update Custom Field 'Renewal Verified At': $msg");
        MailError(
            Subject     => $mail_error_subject,
            Explanation => $mail_error_message,
            FAILURE     => 1,
        );
    }

}

1;

-m

Wow. That pasted like junk. Clearly I have some work to do to figure out this forum. :slight_smile:

If your abuse email follows a specific format, you can potentially build specific parsing libraries or extensions for each type of email. There are examples of extensions that parse specific formats like RT::Extension::ACNS and put values into custom fields.

I took a shot at updating the formatting. I think it’s correct and still has all the code there.