Stacked email handlers


#1

I’ll still be busy with the schema stuff for another day or two at least
(but I promose, a patch is on its way), so you have time to tell me all
the things that are wrong here before I start writing code (or even a more
detailed spec :slight_smile:

Replication of current RT1 procmail functionality with stackable Perl
plugins inside RT itself. Write plugins to (stacked in this order):

  • Archive message
  • Parsing of header fields, etc.
  • Drop bounces and other messages based on configurable regexs against
    header fields, i.e. abuse.net bounces,
  • Drop duplicates based on checksum of body
  • RT1 `stripmime’ - remove non-text attachments and change to URL
  • (Normal RT processing)
  • Stop processing before autoreply based on queue name
  • (above drop plugin used again to drop undesirable messages before
    autoreply - undesirable From: addresses {mailer-daemon, daemon,
    listserv, majordomo}, Precendence: {junk, list, bulk, noreply,
    bofh}, X-RT: loop-prevention)
  • Per-ticket check for 1 message per timeframe (probably a database
    field for timestamp of last autoreply) -> stop processing before
    autoreply
  • drop messages based on a database table of blacklist addresses
  • Autoreply

Plugin interface:
- Message (probably an object with headers parsed, email
addresses decoded, etc., but need a way for the archive plugin and
possibly others should get the raw text first)
- Queue name
- Ticket#
- Arbitrary data store with per-plugin namespace (for AgentId and
other abuse outsourcing features, etc.)

meow
_ivan


#2

Ok. here’s a first round of commentary on this. I’m rather busy with the new gig,
so this may be somewhat discombobulated…tell me what you think.

-j

I’ll still be busy with the schema stuff for another day or two at least
(but I promose, a patch is on its way), so you have time to tell me all
the things that are wrong here before I start writing code (or even a more
detailed spec :slight_smile:

(For those playing along at home, we’ve already got a stackable-handler system inside RT2. I’m not thrilled with my own answers to the issues I’ve raised. Our best bet may be to take what’s already there to enhance it to do what’s needed for ivan’s new functionality.
)

My big concerns with a stackable-handler system are:
the increased overhead at startup…especially at high-volume sites
the added complexity for debugging.

If we design and build with these things in mind, I think we should be able
to make something really capable.

A number of the things mentioned here are important / easy enough that
they should just be part of the ‘standard’ mail gateway.

Replication of current RT1 procmail functionality with stackable Perl
plugins inside RT itself. Write plugins to (stacked in this order):

  • Archive message

You mean split off a copy into a mailbox? How is it better to do this
inside RT rather than at the MTA level?

  • Parsing of header fields, etc.
  • Drop bounces and other messages based on configurable regexs against
    header fields, i.e. abuse.net bounces,

Take a look at how we’re doing address canonicalization. That may be
the easiest way to do this.

  • Drop duplicates based on checksum of body

Presumably checksum of body and a literal comparison of the requestor

  • RT1 `stripmime’ - remove non-text attachments and change to URL
  • (Normal RT processing)

This is taken care of in the core now.

  • Stop processing before autoreply based on queue name

How is this different than “Don’t send autoreply for some queues”? Which
is easy

  • (above drop plugin used again to drop undesirable messages before
    autoreply - undesirable From: addresses {mailer-daemon, daemon,
    listserv, majordomo}, Precendence: {junk, list, bulk, noreply,
    bofh}, X-RT: loop-prevention)
  • Per-ticket check for 1 message per timeframe (probably a database
    field for timestamp of last autoreply) -> stop processing before
    autoreply
  • drop messages based on a database table of blacklist addresses
  • Autoreply
    Autoreply is an RT::Action that's already in the core. will that work?

Plugin interface:
- Message (probably an object with headers parsed, email
addresses decoded, etc., but need a way for the archive plugin and
possibly others should get the raw text first)
- Queue name
- Ticket#
- Arbitrary data store with per-plugin namespace (for AgentId and
other abuse outsourcing features, etc.)

A lot of this feels like it may make sense to add some functionality to the
existing Scrip system and put in another Scrip check
before committing transactions with wider range of return values. …
to allow better control of the transaction after the check.

Hrm. If we could do real commit-rollback, this would be so much easier.

sigh

The quiccker way to do this may be with a few callbacks in the mail gateway.
Most of what MAPS needs is pretty general :slight_smile:

Jesse


meow
_ivan


Rt-devel mailing list
Rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

jesse reed vincent — root@eruditorum.orgjesse@fsck.com
pgp keyprint: 50 41 9C 03 D0 BC BC C8 2C B9 77 26 6F E1 EB 91
Gur SOV jnagf gb znxr guvf fvt vyyrtny.


#3

As the guy who designed the krufty procmail system that Ivan’s
trying to emulate, I figure I should probably respond a bit…

My big concerns with a stackable-handler system are:
the increased overhead at startup…especially at high-volume sites
the added complexity for debugging.

Yes, but imagine how bad it is when all of that stuff is part
of something that sits in front of rt.  *grin*

Replication of current RT1 procmail functionality with stackable Perl
plugins inside RT itself. Write plugins to (stacked in this order):

  • Archive message

You mean split off a copy into a mailbox? How is it better to do this
inside RT rather than at the MTA level?

This is in our current setup in case of unexpected RT h0rkage,
and for legal reasons; it's probably not necessary for most sites.  
  • Drop duplicates based on checksum of body

Presumably checksum of body and a literal comparison of the requestor

That'd be better, given a smarter program.
  • Stop processing before autoreply based on queue name

How is this different than “Don’t send autoreply for some queues”? Which
is easy

This is for the bounce-catcher and do-not-respond-ever list; no
real need if all that gets built into RT.

J.D. Falk "Laughter is the sound
Product Manager that knowledge makes when it’s born."
Mail Abuse Prevention System LLC – The Cluetrain Manifesto