Patch to enable email threading with RT 3.0.2

RT Users
1

Hello,

here is a patch, which enables RT 3.0.2 to respect and create appropriate
References and In-Reply-To headers for Email-Notifications.

It does so for Email-Notifications on incoming email or web-created
transactions.

For doing both (Email, Web) I decided to put the main subroutine to
RT:Tickets, and because I would like to see this integrated into a future
version of RT, I decided to patch original code and not to provide some
*_Local?

Would integrating it into the code of the next version be possible, please?

Regards,
Dirk.

rt-threading.patch (4.81 KB)

2

This is a good start, but it’s broken in a number of ways. It defaults
to the wronge message-id for in-reply-to and it randomly mangles the
order of the references.

To make this really useful, it would help to have a message-id
permanently attached to replies and comments that are generated by the
web UI as well as ones submitted by email. Jesse, would you accept an
improved threading patch that did that?

-DanOn Tue, May 20, 2003 at 06:51:12PM +0200, Dirk Pape pape-rt@inf.fu-berlin.de wrote:

Hello,

here is a patch, which enables RT 3.0.2 to respect and create appropriate
References and In-Reply-To headers for Email-Notifications.

It does so for Email-Notifications on incoming email or web-created
transactions.

For doing both (Email, Web) I decided to put the main subroutine to
RT:Tickets, and because I would like to see this integrated into a future
version of RT, I decided to patch original code and not to provide some
*_Local?

Would integrating it into the code of the next version be possible, please?

Regards,
Dirk.

3

the trick is that for a given ticket update, you may well have multiple
message ids. the messages sent to owner, requestor, etc are all
different.On Tue, May 20, 2003 at 09:39:17PM -0400, Daniel E. Eisenbud wrote:

This is a good start, but it’s broken in a number of ways. It defaults
to the wronge message-id for in-reply-to and it randomly mangles the
order of the references.

To make this really useful, it would help to have a message-id
permanently attached to replies and comments that are generated by the
web UI as well as ones submitted by email. Jesse, would you accept an
improved threading patch that did that?

-Dan

On Tue, May 20, 2003 at 06:51:12PM +0200, Dirk Pape pape-rt@inf.fu-berlin.de wrote:

Hello,

here is a patch, which enables RT 3.0.2 to respect and create appropriate
References and In-Reply-To headers for Email-Notifications.

It does so for Email-Notifications on incoming email or web-created
transactions.

For doing both (Email, Web) I decided to put the main subroutine to
RT:Tickets, and because I would like to see this integrated into a future
version of RT, I decided to patch original code and not to provide some
*_Local?

Would integrating it into the code of the next version be possible, please?

Regards,
Dirk.

rt-users mailing list
rt-users@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

4

the trick is that for a given ticket update, you may well have multiple
message ids. the messages sent to owner, requestor, etc are all
different.

Oh, shoot. I guess in practice you could get away with giving them all
the same message-id, much of the time, but it would be Wrong.

So it seems to me, then, that it’s not so much that adding a message-id
to web comments and replies is wrong, as that using a single message-id
per comment or reply, whether it comes from email or the web, is wrong.

How many different message-ids can versions of a single comment go out
with?

-Dan

5

Oh, shoot. I guess in practice you could get away with giving them all
the same message-id, much of the time, but it would be Wrong.

So it seems to me, then, that it’s not so much that adding a message-id
to web comments and replies is wrong, as that using a single message-id
per comment or reply, whether it comes from email or the web, is wrong.

How many different message-ids can versions of a single comment go out
with?

How many different scrips can you set up? Also, if we start doing VERP,
that becomes one message per recipient.

-Dan

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

6

Oh, shoot. I guess in practice you could get away with giving them all
the same message-id, much of the time, but it would be Wrong.

So it seems to me, then, that it’s not so much that adding a message-id
to web comments and replies is wrong, as that using a single message-id
per comment or reply, whether it comes from email or the web, is wrong.

How many different message-ids can versions of a single comment go out
with?

How many different scrips can you set up? Also, if we start doing VERP,
that becomes one message per recipient.

But VERPed messages don’t have to have different message-ids. Do
they?!?

That aside, hmm. Hmm hmm hmm. I suppose that RT could keep a list, for
each transaction, of which message-id’s corresponding to which scrips it
used, and then construct a custom references header per-scrip. But
yuck, and anyway, I have a vague feeling that that would need schema
changes (but maybe not?)

Another option that would help mutt with my new threading code, but
probably not many other mailers, would be to just glom on all the
message-ids for a given transaction to the references header. I don’t
know if it would be an issue that it’s a minor information leak about
how many different copies of the message are going out, and anyway it’s
gross.

-Dan

7

That aside, hmm. Hmm hmm hmm. I suppose that RT could keep a list, for
each transaction, of which message-id’s corresponding to which scrips it
used, and then construct a custom references header per-scrip. But
yuck, and anyway, I have a vague feeling that that would need schema
changes (but maybe not?)

Ah, but that’s unnecessary, because RT can store one message-id per
transaction and then append the scrip number to it. So if you have a
bunch of messages that went out via the same scrip, they get threaded
together. And you can make the first reference be the absolute
message-id of the first comment, so the different scrip-specific
subthreads at least get threaded together at the top if you’re receiving
mail on the ticket from more than one scrip. Sound sane? If
accumulating piles of reference headers are an issue, storagewise, each
transaction can just have a pointer to the message-id of its parent, and
RT can track back to construct a references header when it sends out
mail.

-Dan

8

How many different scrips can you set up? Also, if we start doing VERP,
that becomes one message per recipient.

On the note of VERP and bounce detection you might want to have alook at
the sympa mailing list software bounce detection code. I was looking at it
when I hacked in my simple bounce detection.

Stewart

9

That aside, hmm. Hmm hmm hmm. I suppose that RT could keep a list, for
each transaction, of which message-id’s corresponding to which scrips it
used, and then construct a custom references header per-scrip. But
yuck, and anyway, I have a vague feeling that that would need schema
changes (but maybe not?)

Ah, but that’s unnecessary, because RT can store one message-id per
transaction and then append the scrip number to it. So if you have a
bunch of messages that went out via the same scrip, they get threaded
together. And you can make the first reference be the absolute
message-id of the first comment, so the different scrip-specific
subthreads at least get threaded together at the top if you’re receiving
mail on the ticket from more than one scrip. Sound sane? If
accumulating piles of reference headers are an issue, storagewise, each
transaction can just have a pointer to the message-id of its parent, and
RT can track back to construct a references header when it sends out
mail.

I’m going to try to implement this. Additionally, each message should
have as its first reference the “true” message-id of the first
transaction in the ticket, so all the different scrip-specific subtrees
will be threaded together at the top level by smart mailers.

I’m going to punt, for now, on dealing with references to correspondence
that’s not cc’ed to the ticket. It’s not obvious to me whether I’ll
want to go back and do that later, but it shouldn’t be too hard if it
seems warranted. In the case of a ticket being generated by a reply to
a previous thread, the first reference will at least attach it to that
thread.

-Dan

10

I’m going to try to implement this. Additionally, each message should
have as its first reference the “true” message-id of the first
transaction in the ticket, so all the different scrip-specific subtrees
will be threaded together at the top level by smart mailers.

I’m going to punt, for now, on dealing with references to correspondence
that’s not cc’ed to the ticket. It’s not obvious to me whether I’ll
want to go back and do that later, but it shouldn’t be too hard if it
seems warranted. In the case of a ticket being generated by a reply to
a previous thread, the first reference will at least attach it to that
thread.

That works for me. It may be worth considering the case of merged
tickets.

j

-Dan

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

11

I’m going to try to implement this. Additionally, each message should
have as its first reference the “true” message-id of the first
transaction in the ticket, so all the different scrip-specific subtrees
will be threaded together at the top level by smart mailers.

I’m going to punt, for now, on dealing with references to correspondence
that’s not cc’ed to the ticket. It’s not obvious to me whether I’ll
want to go back and do that later, but it shouldn’t be too hard if it
seems warranted. In the case of a ticket being generated by a reply to
a previous thread, the first reference will at least attach it to that
thread.

That works for me. It may be worth considering the case of merged
tickets.

Hmm, yes. Well, I think there’s actually not a lot to do about merged
tickets, because if you have a tree starting with a root of the first
transaction of one ticket and another tree starting with a root of the
first transaction of another ticket, and neither of those tickets refer
to each other, no sane mailer is going to pay attention to some distant
descendant claiming that one is the parent of the other (or else I could
be annoying and send mail to this mailing list that caused everyone’s
mutt to thread all the last month’s messages into one thread, in some
arbitrary order.) If the root message-ids don’t refer to actual
messages that have ever existed (and often they won’t, because users
will only ever see the munged message-ids from RT, and the root will be
the only pure one) you could send a special message whose grandparent
was one root and whose grandparent was the other, and that would be
enough to convince mutt to thread the two trees together. One could
make all new top-level messages do such a thing, but it would have other
mildly annoying threading consequences, and I think it’s better not to.

So unless we get into silliness like making RT send out a special mail
when tickets get merged in an attempt to glue mutt users’ trees
together, there’s nothing useful we can do here, I think. (I might be
tempted to implement such a glue message as a local hack, though, with a
scrip that sends it just to the mutt-users on the ticket. :slight_smile: But
probably not.)

-Dan

12

So unless we get into silliness like making RT send out a special mail
when tickets get merged in an attempt to glue mutt users’ trees
together, there’s nothing useful we can do here, I think. (I might be
tempted to implement such a glue message as a local hack, though, with a
scrip that sends it just to the mutt-users on the ticket. :slight_smile: But
probably not.)

There’s actually a rather clean way to do it for any class of users that
you’re willing to send mail saying that the tickets have been merged,
actually. You just give that message the message-id of the (previously
nonexistent) root of the subsumed tree, and make it refer to the root of
the tree of the ticket that remains. This would just be one message for
anyone who’d get it, because it would need to have a scrip-independent
message-id. Maybe there could be a special scrip that would send this
to all the recipients who were deemed appropriate.

Moreover, this doesn’t stop you from having trees refer to prior
correspondence, eventually. It just means that the prior correspondence
has to be ancestors of the “root” message-id.

OK, it is officially time for me to stop babbling until I’ve written
some code. :slight_smile:

-Dan

13

Hello Dan,

–Am Dienstag, 20. Mai 2003 21:39 Uhr -0400 schrieb “Daniel E. Eisenbud”
eisenbud@cbio.mskcc.org:

This is a good start, but it’s broken in a number of ways. It defaults
to the wronge message-id for in-reply-to and it randomly mangles the
order of the references.

To make this really useful, it would help to have a message-id
permanently attached to replies and comments that are generated by the
web UI as well as ones submitted by email. Jesse, would you accept an
improved threading patch that did that?

yes I saw this, just after (successful :wink: testing and sending the patch. I
have now running a version here, which do

a) correct in-reply-to’s depending on if the message is relayed (a
notification for an incoming message) or if it is originated by RT.

b) preserves the order of references (I did not know that MUAs rely on
this).

Already the original patch adds a reference to the message of the first
transaction of a ticket.

As other pointed out, there are some deficiences, I cannot cope with:

A) there may be arbitrary many notifications to one message, which only see
part of the recipients. To have real threading, message ids of all
notifications must be created prior sending any notification.

B) Messages created in the Web do not have any message id. This is not
really bad, since they are never sent by email (only notifications are
sent), but adding a reference to the first transaction fails, if it is a
web created ticket, and so does some threading on them.

I will later this day send my improved patch, so you might to start working
on the new one, or you may start from scratch.

Regards,
Dirk

14

I will later this day send my improved patch, so you might to start working
on the new one, or you may start from scratch.

Great, I’ll see if I can use your patch as a starting point!

-Dan

15

Hello Dan,

–Am Mittwoch, 21. Mai 2003 15:35 Uhr -0400 schrieb “Daniel E. Eisenbud”
eisenbud@cbio.mskcc.org:

Great, I’ll see if I can use your patch as a starting point!

here it is.

Dirk.

rt_threading.patch (6.07 KB)

16

Hello,

–Am Mittwoch, 21. Mai 2003 21:48 Uhr +0200 schrieb Dirk Pape
pape-rt@inf.fu-berlin.de:

–Am Mittwoch, 21. Mai 2003 15:35 Uhr -0400 schrieb “Daniel E. Eisenbud”
eisenbud@cbio.mskcc.org:

Great, I’ll see if I can use your patch as a starting point!

here it is.

please do not use the patch, since it induces encoding problems in incoming
emails. I dont know yet why it does so, but I will look into this soon.

Dirk.