RT 3.0.3rc2

The only changes here were header parsing and generation fixes.
We backed out the encode_mimewords patch from Dirk, as that was causing
more problems than it was solving and replaced it with a more clever
encode_mimeword that avoids encoding the email address.

We also fixed header-decoding on incoming mail.

We took mitya’s failure cases and dropped them into the test suite, to
make sure it never happens again.

-j

 160    117	  Another shot at the header encoding fix
 161    118	  Mitya's failing processing of html email
 162    119	  Bumping to 3.0.3rc2

http://www.bestpractical.com/rt – Trouble Ticketing. Free.

The only changes here were header parsing and generation fixes.
We backed out the encode_mimewords patch from Dirk, as that was causing
more problems than it was solving and replaced it with a more clever
encode_mimeword that avoids encoding the email address.

We also fixed header-decoding on incoming mail.

We took mitya’s failure cases and dropped them into the test suite, to
make sure it never happens again.

-j

 160    117	  Another shot at the header encoding fix
 161    118	  Mitya's failing processing of html email
 162    119	  Bumping to 3.0.3rc2

Jesse,
thank you very much, many problems were fixes in the past time.

I just tried 3.0.3rc2.

I found a couple of problems:

  1. When I am sending http://mitya.pp.ru/rt-bad-email1.dat, the following
    appears in the maillog:

Jun 19 12:01:01 vh postfix/local[35201]: 1ACFE16E84C: to=info@vh.demos.su, rel
ay=local, delay=1, status=bounced (Command died with status 32: “/usr/local/rt3/
bin/rt-mailgate --queue info --action correspond --url http://vh.demos.su”. Comm
and output: RT server error. The RT server which handled your email did not beh
ave as expected. It said:

System error

<td align="lef
  1. The following e-mail http://mitya.pp.ru/rt-bad-email2.dat looks OK in the
    WebUI, but when RT sends it via e-mail, all non-ascii characters in it’s body
    become ‘???’ (Subject is OK).

Could you please investigate?

Thanks!

error:<
/b> 
Unknown encoding ‘UNKNOWN’ at /usr/local/rt3/
lib/RT/I18N.pm line 275
context: 
  1. When I am sending http://mitya.pp.ru/rt-bad-email1.dat, the following
    appears in the maillog:

That mail has the extremely evil header:

and RT simply does not know how to handle this encoding.

I have committed the patch below into my branch which takes care of
this case by applying the same _GuessCharset routine to MIME words.

As for your second problem, I can’t duplicate it – but maybe you’d
like to check your @EmailOutputEncoding?

Thanks,
/Autrijus/

— //depot/RT/rt/lib/RT/I18N.pm 2003/06/19 08:03:42
+++ //depot/RT/rt/lib/RT/I18N.pm 2003/06/19 08:34:19
@@ -273,7 +273,11 @@
# now we have got a decoded subject, try to convert into
# utf-8 encoding
unless ($charset =~ m/utf-8/i) {

  •   Encode::from_to($enc_str, $charset, "utf8");
    
  •   eval { Encode::from_to($enc_str, $charset, "utf8") };
    
  •   if ($@) {
    
  •   $charset = _GuessCharset( $enc_str );
    
  •   Encode::from_to($enc_str, $charset, "utf8");
    
  •   }
    

    }

    $str .= $prefix . $enc_str . $trailing;
    @@ -296,17 +300,33 @@
    sub _FindOrGuessCharset {
    my $entity = shift;
    my $head = $entity->head;

  • my $charset;

  • my $fallback = ‘iso-8859-1’;

  • return $charset if ($charset = $head->mime_attr(“content-type.charset”));

  • if ($head->mime_attr(“content-type.charset”)) {

  • return $head->mime_attr(“content-type.charset”);

  • }

    my $body = $entity->bodyhandle or return;

  • return ‘utf-8’ if Encode::is_utf8( $body->as_string );
  • return _GuessCharset( $head->as_string . $body->as_string );
    +}
    +# }}}
    +# {{{ _GuessCharset
    +=head2 _GuessCharset STRING
    +use Encode::Guess to try to figure it out the string’s encoding.
    +=cut
    +sub _GuessCharset {

  • my $fallback = ‘iso-8859-1’;

  • my $charset;

    if ( @RT::EmailInputEncodings and eval { require Encode::Guess; 1 } ) {
    Encode::Guess->set_suspects(@RT::EmailInputEncodings);

  • my $decoder = Encode::Guess->guess( $head->as_string . $body->as_string );
  • my $decoder = Encode::Guess->guess( $_[0] );

    if ( ref $decoder ) {
    $charset = $decoder->name;

  1. When I am sending http://mitya.pp.ru/rt-bad-email1.dat, the following
    appears in the maillog:

That mail has the extremely evil header:

From: =?UNKNOWN?Q?=C8=ED=F2=E5=F0=ED=E5=F2_=EC=E0=F0=EA=E5=F2=E8=ED=E3?=

and RT simply does not know how to handle this encoding.

Yes, you are right.
But RT should not crash regardless what it’s input is, obviously…

I have committed the patch below into my branch which takes care of
this case by applying the same _GuessCharset routine to MIME words.

Thanks, I’ll test it.

As for your second problem, I can’t duplicate it – but maybe you’d
like to check your @EmailOutputEncoding?

In my RT_SiteConfig.pm:

Set($EmailOutputEncoding , ‘koi8-r’);

  1. When I am sending http://mitya.pp.ru/rt-bad-email1.dat, the following
    appears in the maillog:

That mail has the extremely evil header:

From: =?UNKNOWN?Q?=C8=ED=F2=E5=F0=ED=E5=F2_=EC=E0=F0=EA=E5=F2=E8=ED=E3?=

and RT simply does not know how to handle this encoding.

Your patch did solve the problem to rt-bad-email1.dat.

Thank you!

The problem with rt-bad-email2.dat remains.

Hey Jesse,

–Am Donnerstag, 19. Juni 2003 3:13 Uhr -0400 schrieb Jesse Vincent
jesse@bestpractical.com:

The only changes here were header parsing and generation fixes.
We backed out the encode_mimewords patch from Dirk, as that was causing
more problems than it was solving and replaced it with a more clever
encode_mimeword that avoids encoding the email address.

that seems to resolve my subject encoding issue and encouraged me to
reapply my threading patch and voila, for now (few testing only) my problem
is gone.

I beg your pardon for having introduced this bug into RT.

But this might be compensated by contruting my (hopefully) soon
rehabilitated threading patch and some more contributions we developped her
to support our workflow (e.g. a slightly changed handling of signatures in
Web-Replies, ability to toggle ticket history display order on the fly or
for the session, “Notify AdminCC only if Ticket is unowned”, public Tickets
in SelfService, some Callbacks for Ticket Actions: “stall until”,
“dispatch” :slight_smile:

BTW: Does a policy exist how to pack contributions for RT3?

Dirk.

Hey Jesse,

–Am Donnerstag, 19. Juni 2003 3:13 Uhr -0400 schrieb Jesse Vincent
jesse@bestpractical.com:

The only changes here were header parsing and generation fixes.
We backed out the encode_mimewords patch from Dirk, as that was causing
more problems than it was solving and replaced it with a more clever
encode_mimeword that avoids encoding the email address.

that seems to resolve my subject encoding issue and encouraged me to
reapply my threading patch and voila, for now (few testing only) my problem
is gone.

I beg your pardon for having introduced this bug into RT.

No worries. We didn’t have the right tests to catch that it would cause
a problem (and really, it’s a deficiency in MIME-Tools’ implementation).
But the bug never made it into a shipping release, so it’s no big deal
:wink:

But this might be compensated by contruting my (hopefully) soon
rehabilitated threading patch and some more contributions we developped her
to support our workflow (e.g. a slightly changed handling of signatures in
Web-Replies, ability to toggle ticket history display order on the fly or
for the session, “Notify AdminCC only if Ticket is unowned”, public Tickets
in SelfService, some Callbacks for Ticket Actions: “stall until”,
“dispatch” :slight_smile:

Neat. I’m sure people will be thrilled to see them. Pack them oen by one
and bounce them to me and I’ll drop them in /contrib

BTW: Does a policy exist how to pack contributions for RT3?

No proper policy exists yet. Off the top of my head:

I’d like to see and
<contribution_name>.README
<contribution_name>.tar.gz

The .tar.gz should contain a structure something like the following

another copy of the readme [required]
a Makefile 
etc/initialdata [optional]
html/
lib/
doc/
po/ (localization)

what else?

Autrijus indicated that he might have something clever for packaging
addons for us soon, too.

Dirk.


rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

http://www.bestpractical.com/rt – Trouble Ticketing. Free.

Jesse,
thank you very much, many problems were fixes in the past time.

I just tried 3.0.3rc2.

I found a couple of problems:

  1. When I am sending http://mitya.pp.ru/rt-bad-email1.dat, the following
    appears in the maillog:

Autrijus patch is in 3.0.3rc3 (out now) and fixes this.

  1. The following e-mail http://mitya.pp.ru/rt-bad-email2.dat looks OK in the
    WebUI, but when RT sends it via e-mail, all non-ascii characters in it’s body
    become ‘???’ (Subject is OK).

It should be fine in attached copies of the message (on
correspondence), but not displayed inline. As this is not a regression
from RT 3.0.2, it will not be fixed for 3.0.3. (3.0.3 is SO MUCH better
about unicode than 3.0.2 was, that I really want to get it out into the
world for folks.

Could you please investigate?

Could you open a bug by sending mail to rt-bugs@fsck.com with examples
and details attached, so we have it recorded?

Thanks.

Jesse

Thanks!


rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

http://www.bestpractical.com/rt – Trouble Ticketing. Free.

  1. The following e-mail http://mitya.pp.ru/rt-bad-email2.dat looks OK in the
    WebUI, but when RT sends it via e-mail, all non-ascii characters in it’s body
    become ‘???’ (Subject is OK).

It should be fine in attached copies of the message (on
correspondence), but not displayed inline. As this is not a regression

Could you please explain this in more detail, please.
I don’t quite understand why it is OK.

Thanks!

  1. The following e-mail http://mitya.pp.ru/rt-bad-email2.dat looks OK in the
    WebUI, but when RT sends it via e-mail, all non-ascii characters in it’s body
    become ‘???’ (Subject is OK).

It should be fine in attached copies of the message (on
correspondence), but not displayed inline. As this is not a regression

Could you please explain this in more detail, please.
I don’t quite understand why it is OK.

So. When the text/html part is attached, it should have the correct
content type. When it’s displayed “inline” as the “Content” of the
message, it’s not automatically converted to plain text in utf8. This
is a bug. But it’s not a bug that will be fixed before 3.0.3.

Thanks!

from RT 3.0.2, it will not be fixed for 3.0.3. (3.0.3 is SO MUCH better
about unicode than 3.0.2 was, that I really want to get it out into the
world for folks.

http://www.bestpractical.com/rt – Trouble Ticketing. Free.

  1. The following e-mail http://mitya.pp.ru/rt-bad-email2.dat looks OK in the
    WebUI, but when RT sends it via e-mail, all non-ascii characters in it’s body
    become ‘???’ (Subject is OK).

It should be fine in attached copies of the message (on
correspondence), but not displayed inline. As this is not a regression

Could you please explain this in more detail, please.
I don’t quite understand why it is OK.

So. When the text/html part is attached, it should have the correct
content type. When it’s displayed “inline” as the “Content” of the
message, it’s not automatically converted to plain text in utf8. This
is a bug. But it’s not a bug that will be fixed before 3.0.3.

Ah, I see, that’s OK.
I’ll remind you about it just after 3.0.3 release, if you don’t mind.
Thanks god release should be out RSN.

Thanks you!

Ah, I see, that’s OK.
I’ll remind you about it just after 3.0.3 release, if you don’t mind.
Thanks god release should be out RSN.

Cool, though feel free to report to the bug tracking system now, so
everyone knows it’s an issue :slight_smile:

Thanks you!

http://www.bestpractical.com/rt – Trouble Ticketing. Free.