Automatic removal of bounce-generating addresses from ticket

Our RT installation are mostly feed using email, and our users have
not quite learned to limit their request to only one queue. This
typically give us two tickets, one in each queue, with CC set to the
address of the other queue. When someone add correspondence to these
tickets, RT tries to send an email to the queue address of the other
queue, and the RT administrators (for example me), get an email from
RT with the report of a suspected bounce.

I am getting tired of this, so I sat down to write a script to
automatically get rid of these Requestor, CC and AdminCC entries. The
script is below, and it seem to work. The only problem is that it is
dead slow. It uses several seconds per ticket. I see 2-3 seconds
between ‘.’ is printed in debug mode. Am I doing something wrong
here?

#!/site/perl-5.8.6/bin/perl

Author: Petter Reinholdtsen

Date: 2006-01-13

Look at all open tickets, and remove all queue addresses from

requestor, cc and admincc. This reduces the amount of bounce emails

sent to the RT admins.

use warnings;
use strict;

use lib ("/site/rttest/local/lib", “/site/rttest/lib”);

package RT;

use RT::Interface::CLI qw(CleanEnv GetCurrentUser GetMessageContent loc);
use RT::Date;
use RT::Queue;
use RT::Queues;
use RT::Tickets;

my $debug = 1;

Find all queue addresses of enabled queues

my @queueaddrs = ();

$| = 1;
initialize();

my $user = RT::User->new( $RT::SystemUser );

load_queue_addresses();

Loop over all tickets, remove addresses (log changes)

my $tickets = new RT::Tickets($RT::SystemUser);
$tickets->LimitStatus(VALUE => ‘new’);
$tickets->LimitStatus(VALUE => ‘open’);
$tickets->LimitStatus(VALUE => ‘stalled’);

if ($debug) {
my $queue = new RT::Queue($RT::SystemUser);
$queue->Load(“www-drift”);
$tickets->LimitQueue(VALUE => $queue->Id);
}

Sort tickets in to lists for each owner

my $principal = RT::Principal->new( $RT::SystemUser );
my $group = RT::Group->new( $RT::SystemUser );
while (my $ticket = $tickets->Next) {
print “:” if $debug;
# check Requestor, Cc, AdminCc, remove the addresses listed in @queueaddrs
foreach my $type ( “Cc”, “AdminCc”, “Requestor” ) {
print “.” if $debug;
$group->LoadTicketRoleGroup( Type => $type, Ticket => $ticket->Id );
next unless ( $group->id );

    for my $address (@queueaddrs) {
        $user->LoadByEmail( $address );
        $principal->Load( $user->Id );
        if ($group->HasMember( $principal )) {
            my $id = $ticket->Id;
            if ($debug) {
                print "Want to remove $address as $type from ticket $id\n";
            } else {
                $RT::Logger->info("Removed queue address $address ".
                                  "as $type from ticket $id");
                $ticket->DeleteWatcher(Type => $type,
                                       PrincipalId => $principal->Id);
            }
        }
    }
}

}

sub initialize {
CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT
}

sub load_queue_addresses {
my $queues = new RT::Queues($RT::SystemUser);
$queues->LimitToEnabled();
foreach my $queue (@{$queues->ItemsArrayRef()}) {
for my $address ($queue->CorrespondAddress, $queue->CommentAddress) {
next unless $user->LoadByEmail( $address );
push @queueaddrs, $address;
print “Found queue address ‘$address’\n” if $debug;
}
}
}

Our RT installation are mostly feed using email, and our users have
not quite learned to limit their request to only one queue. This
typically give us two tickets, one in each queue, with CC set to the
address of the other queue. When someone add correspondence to these
tickets, RT tries to send an email to the queue address of the other
queue, and the RT administrators (for example me), get an email from
RT with the report of a suspected bounce.
You didn’t configure your RT, see $RTAddressRegexp in config. I don’t
remember if with this option RT addresses wouldbe in Cc, AdminCc…
lists, but I’m sure that notify scrip would skip them.

I am getting tired of this, so I sat down to write a script to
automatically get rid of these Requestor, CC and AdminCC entries. The
script is below, and it seem to work. The only problem is that it is
dead slow. It uses several seconds per ticket. I see 2-3 seconds
between ‘.’ is printed in debug mode. Am I doing something wrong
here?

#!/site/perl-5.8.6/bin/perl

Author: Petter Reinholdtsen

Date: 2006-01-13

Look at all open tickets, and remove all queue addresses from

requestor, cc and admincc. This reduces the amount of bounce emails

sent to the RT admins.

use warnings;
use strict;

use lib ("/site/rttest/local/lib", “/site/rttest/lib”);

package RT;

use RT::Interface::CLI qw(CleanEnv GetCurrentUser GetMessageContent loc);
use RT::Date;
use RT::Queue;
use RT::Queues;
use RT::Tickets;

my $debug = 1;

Find all queue addresses of enabled queues

my @queueaddrs = ();

$| = 1;
initialize();

my $user = RT::User->new( $RT::SystemUser );

load_queue_addresses();

Loop over all tickets, remove addresses (log changes)

my $tickets = new RT::Tickets($RT::SystemUser);
$tickets->LimitStatus(VALUE => ‘new’);
$tickets->LimitStatus(VALUE => ‘open’);
$tickets->LimitStatus(VALUE => ‘stalled’);

if ($debug) {
my $queue = new RT::Queue($RT::SystemUser);
$queue->Load(“www-drift”);
$tickets->LimitQueue(VALUE => $queue->Id);
}

Sort tickets in to lists for each owner

my $principal = RT::Principal->new( $RT::SystemUser );
my $group = RT::Group->new( $RT::SystemUser );
while (my $ticket = $tickets->Next) {
print “:” if $debug;
# check Requestor, Cc, AdminCc, remove the addresses listed in @queueaddrs
foreach my $type ( “Cc”, “AdminCc”, “Requestor” ) {
print “.” if $debug;
$group->LoadTicketRoleGroup( Type => $type, Ticket => $ticket->Id );
next unless ( $group->id );

    for my $address (@queueaddrs) {
        $user->LoadByEmail( $address );
        $principal->Load( $user->Id );
        if ($group->HasMember( $principal )) {
            my $id = $ticket->Id;
            if ($debug) {
                print "Want to remove $address as $type from ticket $id\n";
            } else {
                $RT::Logger->info("Removed queue address $address ".
                                  "as $type from ticket $id");
                $ticket->DeleteWatcher(Type => $type,
                                       PrincipalId => $principal->Id);
            }
        }
    }
}

}

sub initialize {
CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT
}

sub load_queue_addresses {
my $queues = new RT::Queues($RT::SystemUser);
$queues->LimitToEnabled();
foreach my $queue (@{$queues->ItemsArrayRef()}) {
for my $address ($queue->CorrespondAddress, $queue->CommentAddress) {
next unless $user->LoadByEmail( $address );
push @queueaddrs, $address;
print “Found queue address ‘$address’\n” if $debug;
}
}
}


Rt-devel mailing list
Rt-devel@lists.bestpractical.com
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-devel

Best regards, Ruslan.

[Ruslan Zakirov]

You didn’t configure your RT, see $RTAddressRegexp in config. I
don’t remember if with this option RT addresses wouldbe in Cc,
AdminCc… lists, but I’m sure that notify scrip would skip them.

I see your point, but we have a rather complex setup so keeping the
regex in sync with all the mail addresses being forwarded into RT seem
like the wrong solution to me. The people creating the queues using
the web interface are not the people capable of (with access to)
editing the config file. And the problematic addresses are typically
foo@somewhere.uio.no, while the RT instance (and addresses for new RT
queues) have the form bar@rt.uio.no. The regex would be a nice
solution if we didn’t migrate from existing mailing lists and had a
simpler setup, but do not quite work for us.

Any idea why the group operations in the script is so slow?

[Ruslan Zakirov]

You didn’t configure your RT, see $RTAddressRegexp in config. I
don’t remember if with this option RT addresses wouldbe in Cc,
AdminCc… lists, but I’m sure that notify scrip would skip them.

I see your point, but we have a rather complex setup so keeping the
regex in sync with all the mail addresses being forwarded into RT seem
like the wrong solution to me. The people creating the queues using
the web interface are not the people capable of (with access to)
editing the config file. And the problematic addresses are typically
foo@somewhere.uio.no, while the RT instance (and addresses for new RT
queues) have the form bar@rt.uio.no. The regex would be a nice
solution if we didn’t migrate from existing mailing lists and had a
simpler setup, but do not quite work for us.

Any idea why the group operations in the script is so slow?

Yes. Use next algorithm:
for my $address( @addresses ) {
my $tickets = new …;
$tickets->FromSQL( “Watcher.EmailAddress = ‘$address’” );
while( my $ticket = $tickets->Next ) {
$ticket->DeleteWatcher(Email => $address, Type => $_ ) for( qw(Cc
AdminCc Requestor) );
}
}
I think this should speedup things
Best regards, Ruslan.

[Ruslan Zakirov]

I think this should speedup things

It did. The run time went from 1.5 hour to 10 minutes. Thank you for
the suggestion to turn the search strategy upside down. :slight_smile:

Here is the script we now use in production. I’ll keep the copy on
URL:http://www.usit.uio.no/it/rt/modifications/ up to date.

#!/site/perl-5.8.6/bin/perl

Author: Petter Reinholdtsen

Date: 2006-01-13

License: GNU Public License v2 or later

Look at all tickets, and remove all queue addresses from requestor,

cc and admincc. This reduces the amount of bounce emails sent to

the RT admins.

use warnings;
use strict;
use Getopt::Std;

use lib ("/site/rttest/local/lib", “/site/rttest/lib”);

package RT;

use RT::Interface::CLI qw(CleanEnv);
use RT::Queue;
use RT::Queues;
use RT::Tickets;

my %opts;
Getopt::Std::getopts(“dn”, %opts);

my $debug = $opts{‘d’} || 0;
my $dryrun = $opts{‘n’} || 0;

$| = 1;

Find all queue addresses of enabled queues

my @queueaddrs = qw(root@ourhost.uio.no);

my $ticketcount = 0;
my $starttime = time();

CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT

my $user = RT::User->new( $RT::SystemUser );

Merge static list with dynamic list

@queueaddrs = (load_queue_addresses(), @queueaddrs);

Loop over all addresses, remove them from the tickets were they are

registered as watchers

for my $address( sort @queueaddrs ) {
my $tickets = new RT::Tickets($RT::SystemUser);
$tickets->FromSQL( “Watcher.EmailAddress = ‘$address’” );
while( my $ticket = $tickets->Next ) {
$ticketcount++;
my $id = $ticket->Id;
if ($dryrun) {
print “Want to remove $address as watcher from ticket #$id\n”;
} else {
$RT::Logger->info("Removed queue address $address as watcher ".
“from ticket #$id”);
$ticket->DeleteWatcher(Email => $address, Type => $_ )
for( qw(Cc AdminCc Requestor) );
}
}
}

my $duration = time() - $starttime;
$RT::Logger->info(“Processing of $ticketcount tickets took $duration seconds”);

sub load_queue_addresses {
my $queues = new RT::Queues($RT::SystemUser);
$queues->LimitToEnabled();
my @queueaddrs;
foreach my $queue (@{$queues->ItemsArrayRef()}) {
for my $address ($queue->CorrespondAddress, $queue->CommentAddress) {
next unless $user->LoadByEmail( $address );
push @queueaddrs, $address;
print “Found queue address ‘$address’\n” if $debug;
}
}
return @queueaddrs;
}

[Petter Reinholdtsen]

Here is the script we now use in production. I’ll keep the copy on
URL:http://www.usit.uio.no/it/rt/modifications/ up to date.

The script do not work as I expected it to, and I fail to understand
why. It is capable of removing lots of watchers from the tickets, but
not one specific watcher, ‘root@ulrik.uio.no’. When I remove the
problematic watcher using the web interface, I have no problems
getting rid of it. I’ve tried to reduce the script to a small example
demonstrating the problem.

The script searches for all tickets with a watcher using a given email
address, and then loops over all the tickets to remove the watcher
with the given email from the tickets. Works most of the time. But
for one email address, if finds 202 tickets, but is unable to remove
the watcher because it “Could not find that principal”. How is this
possible? I get this output. Notice how it tries to remove the
watcher from the same ticket each time I run the script.

% ./foo
Found 202 tickets with ‘root@ulrik.uio.no’ as watcher
Removing queue address root@ulrik.uio.no as watcher from ticket #141
Failed to remove Cc: Could not find that principal
Failed to remove AdminCc: Could not find that principal
Failed to remove Requestor: Could not find that principal
Processed 1 tickets
% ./foo
Found 202 tickets with ‘root@ulrik.uio.no’ as watcher
Removing queue address root@ulrik.uio.no as watcher from ticket #141
Failed to remove Cc: Could not find that principal
Failed to remove AdminCc: Could not find that principal
Failed to remove Requestor: Could not find that principal
Processed 1 tickets
%

This is the script. I cut the loop short to limit the output of the
example run. Any ideas? There must be something obvious I fail to
see here.

#!/site/perl-5.8.6/bin/perl

Author: Petter Reinholdtsen

Date: 2006-01-19

License: GNU Public License v2 or later

Demonstrate a strange feature in RT, where 'find tickets with

watcher = root@ulrik.uio.no’ return lots of tickets, but 'remove

watcher root@ulrik.uio.no from ticket’ fail because there is no such

principal.

use warnings;
use strict;

use lib ("/site/rttest/local/lib", “/site/rttest/lib”);

package RT;

use RT::Interface::CLI qw(CleanEnv);
use RT::Queue;
use RT::Queues;
use RT::Tickets;

my $debug = 1;

my $ticketcount = 0;

CleanEnv(); # Clean our the environment
RT::LoadConfig(); # Load the RT configuration
RT::Init(); # Initialise RT

my $user = RT::User->new( $RT::SystemUser );

Loop over all addresses, remove them from the tickets were they are

registered as watchers

my $address = "root@ulrik.uio.no";
my $tickets = new RT::Tickets($RT::SystemUser);
$tickets->FromSQL( “Watcher.EmailAddress = ‘$address’” );

print “Found “.$tickets->Count.” tickets with ‘$address’ as watcher\n”;

while( my $ticket = $tickets->Next ) {
$ticketcount++;
my $id = $ticket->Id;
print “Removing queue address $address as watcher from ticket #$id\n”;
for my $type ( qw(Cc AdminCc Requestor) ) {
my ($ok, $msg) =
$ticket->DeleteWatcher(Email => $address, Type => $type );
print " Failed to remove $type: $msg\n"
unless $ok;
}
last;
}
print “Processed $ticketcount tickets\n”;

[Petter Reinholdtsen]

Here is the script we now use in production. I’ll keep the copy on
URL:http://www.usit.uio.no/it/rt/modifications/ up to date.

The script do not work as I expected it to, and I fail to understand
why. It is capable of removing lots of watchers from the tickets, but
not one specific watcher, ‘root@ulrik.uio.no’. When I remove the
problematic watcher using the web interface, I have no problems
getting rid of it. I’ve tried to reduce the script to a small example
demonstrating the problem.

I wonder if there’s something weird with a watcher with that username
and a blank email address.

Jesse

[Jessie Vincent]

I wonder if there’s something weird with a watcher with that
username and a blank email address.

Not that I can see. But I have isolated the problem to LoadByEmail()
failing.

This code do not give the expected result:

my $address = "root@ulrik.uio.no";

$user->LoadByEmail( $address );
print “UI: '”. $user->Id . “’\n”;
print “UN: '”. $user->Name . “’\n”;
print “UE: '”. $user->EmailAddress . “’\n”;

$user->Load( $address );
print “UI: '”. $user->Id . “’\n”;
print “UN: '”. $user->Name . “’\n”;
print “UE: '”. $user->EmailAddress . “’\n”;

I get this output, notice how LoadByEmail() fail even thought the
EmailAddress of the user exist and is set to ‘root@ulrik.uio.no’.

UI: ''
UN: ''
UE: ''
UI: '724’
UN: 'root@ulrik.uio.no’
UE: ‘root@ulrik.uio.no’

Any idea why this could fail?

Ruslan mentioned in private mail that Owners are watchers. The user
in question is not an owner of the problematic ticket, but how do I
configure RT to let Owners behave as AdminCC watchers? I thought
owners were not watchers, as in our installation they do not get any
of the correspondance nor comments.