Mail notification MIME encoding problems (cont...)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I posted to the list a couple of weeks back now I think about having
problems with multi-byte characters in email subjects (specifically
Japanese)

I’ve found a few problems.
I think I’ve (at least temporarily) fixed one problem.

the first problem was with the line folding of the Subject: header with
a mix of single and multi-byte characters. Ultimately I installed Jcode
from CPAN and added the following lines to the MIMEEncodeString
subroutine in /opt/rt3/libs/RT/Actions/SendEmail.pm

    use Jcode;
    my $jcode = Jcode->new($str);
    my @chunks = $jcode->jfold($max,'\n');

and commented out the following lines:

my @chunks = unpack(“a$max” x int(length($str)/$max

+ ((length($str) % $max) ? 1:0)), $str);

now my mail subjects aren’t getting trashed by line folding.
(for what it’s worth, if i just returned an unfolded encoded Subject: i
didn’T have an problems, so it was obviously an issue where it was
folding mid-2byte character in some cases)

the second problem is a little more complicated and I haven’t been able
to determine exactly why it’s happening. I’ve added debug statements
in the code to track what’s happening but it’s still a little uncertain
as to why it’s happening. Here’s the debug info (it’s kind of long):

[Thu Aug 17 08:07:09 2006] [warning]: Use of uninitialized value in
substitution (s///) at /opt/rt3/lib/RT/Interface/Web.pm line 617.
(/opt/rt3/lib/RT/Interface/Web.pm:617)
[Thu Aug 17 08:07:09 2006] [debug]: About to think about scrips for
transaction #5081 (/opt/rt3/lib/RT/Transaction_Overlay.pm:165)
[Thu Aug 17 08:07:09 2006] [debug]: About to prepare scrips for
transaction #5081 (/opt/rt3/lib/RT/Transaction_Overlay.pm:169)
[Thu Aug 17 08:07:09 2006] [debug]: Found 4 scrips
(/opt/rt3/lib/RT/Scrips_Overlay.pm:363)
[Thu Aug 17 08:07:11 2006] [debug]: sub before: [Comment] testr用な結構
長い件名しなければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:752)
[Thu Aug 17 08:07:11 2006] [debug]: sub after : [Comment] testr用な結構
長い件名しなければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:755)
[Thu Aug 17 08:07:11 2006] [debug]: new subject: [sysrequest #371]
[Comment] testr用な結構長い件名しなければtextならないのでこれは結構長い
件名です。 (/opt/rt3/lib/RT/Action/SendEmail.pm:758)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: field = Subject, enc = utf-8
(/opt/rt3/lib/RT/Action/SendEmail.pm:863)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: value = [sysrequest #371]
[Comment] testr用な結構長い件名しなければtextならないのでこれは結構長い
件名です。 (/opt/rt3/lib/RT/Action/SendEmail.pm:870)
[Thu Aug 17 08:07:11 2006] [debug]: value before utf encoding:
[sysrequest #371] [Comment] testr用な結構長い件名しなければtextならない
のでこれは結構長い件名です。 (/opt/rt3/lib/RT/Action/SendEmail.pm:916)
[Thu Aug 17 08:07:11 2006] [debug]: value = [sysrequest #371] [Comment]
testr?¨???§???¶???°text???§???§???¶??§??
(/opt/rt3/lib/RT/Action/SendEmail.pm:921)
[Thu Aug 17 08:07:11 2006] [debug]: chunks = [sysrequest #371] [Comment]
testr?¨?? ???§???¶??? ???°text??? ???§??? §???¶??§??
(/opt/rt3/lib/RT/Action/SendEmail.pm:936)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: after encoding value =
=?UTF-8?B?W3N5c3JlcXVlc3QgIzM3MV0gW0NvbW1lbnRdIHRlc3Ryw6fClMKow6PCgcKq?=
=?UTF-8?B?w6fCtcKQw6bCp8KLw6nClcK3w6PCgcKEw6TCu8K2w6XCkMKNw6PCgcKXw6M=?=
=?UTF-8?B?woHCqsOjwoHCkcOjwoLCjMOjwoHCsHRleHTDo8KBwqrDo8KCwonDo8KBwqo=?=
=?UTF-8?B?w6PCgcKEw6PCgcKuw6PCgcKnw6PCgcKTw6PCgsKMw6PCgcKvw6fCtcKQw6Y=?=
=?UTF-8?B?wqfCi8OpwpXCt8OjwoHChMOkwrvCtsOlwpDCjcOjwoHCp8OjwoHCmcOjwoA=?=
=?UTF-8?B?woI=?= (/opt/rt3/lib/RT/Action/SendEmail.pm:878)
[Thu Aug 17 08:07:11 2006] [debug]: sub before: testr用な結構長い件名し
なければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:752)
[Thu Aug 17 08:07:11 2006] [debug]: sub after : testr用な結構長い件名し
なければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:755)
[Thu Aug 17 08:07:11 2006] [debug]: new subject: [sysrequest #371] testr
用な結構長い件名しなければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:758)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: field = Subject, enc = utf-8
(/opt/rt3/lib/RT/Action/SendEmail.pm:863)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: value = [sysrequest #371]
testr用な結構長い件名しなければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:870)
[Thu Aug 17 08:07:11 2006] [debug]: value before utf encoding:
[sysrequest #371] testr用な結構長い件名しなければtextならないのでこれは
結構長い件名です。 (/opt/rt3/lib/RT/Action/SendEmail.pm:916)
[Thu Aug 17 08:07:11 2006] [debug]: value = [sysrequest #371] testr用な
結構長い件名しなければtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:921)
[Thu Aug 17 08:07:11 2006] [debug]: chunks = [sysrequest #371] testr用な
結構長い件名しなけ ればtextならないのでこれは結構長い件名です。
(/opt/rt3/lib/RT/Action/SendEmail.pm:936)
[Thu Aug 17 08:07:11 2006] [debug]: SHAE: after encoding value =
=?UTF-8?B?W3N5c3JlcXVlc3QgIzM3MV0gdGVzdHLnlKjjgarntZDmp4vplbfjgYTku7blkI3jgZfjgarjgZE=?=
=?UTF-8?B?44KM44GwdGV4dOOBquOCieOBquOBhOOBruOBp+OBk+OCjOOBr+e1kOani+mVt+OBhOS7tuWQjeOBp+OBmeOAgg==?=
(/opt/rt3/lib/RT/Action/SendEmail.pm:878)
[Thu Aug 17 08:07:19 2006] [debug]: RT::Date used date::parse to make
1970-01-01 -32400 (/opt/rt3/lib/RT/Date.pm:211)

specifically what seems to be happening is, if I add “Subject:
[something] {$Ticket->Subject}” to a template, and that template is used
in the email process then in MIMEEncodeString subroutine, the
Encode::_utf8_off($value) call is mangling the string. oddly, it only
happens if the Subject: header is put into the template.

I’m running RT 3.6.1 on a RH FC 5 machine.

These issues are preventing me from being able to put RT into production
(which I’d really like to do) so any help would be greatly appreciated.

Thanks,

Alan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE5CaxE2gsBSKjZHQRAu+bAKCR2MmvPemfKor7CY93A/g3dTXftwCfb5Y8
DGy9e/7u4bT0uyrz7bnJul8=
=551B
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I posted to the list a couple of weeks back now I think about having
problems with multi-byte characters in email subjects (specifically
Japanese)

Alan, What version of RT?

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I posted to the list a couple of weeks back now I think about having
problems with multi-byte characters in email subjects (specifically
Japanese)

Alan, What version of RT?

(Found it buried after the logs. Probably better to include long logs
last next time).

It’d be great if you could sent us a message or two that exhibit the bug
as gzipped RFC822 messages, so that we can add em to the test suite

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jesse Vincent wrote:

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I posted to the list a couple of weeks back now I think about having
problems with multi-byte characters in email subjects (specifically
Japanese)
Alan, What version of RT?

(Found it buried after the logs. Probably better to include long logs
last next time).

It’d be great if you could sent us a message or two that exhibit the bug
as gzipped RFC822 messages, so that we can add em to the test suite

Jesse,

Sorry about that.

I’ll try and get a few messages zipped up today. Should I email them to
you directly?

Alan
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFE6QIeE2gsBSKjZHQRAo5MAKC6SrAVR9GMNcYvz22dRkXlsC7newCbBpnk
dA4fphje8pDKZ4jC4aQlz2U=
=FZPd
-----END PGP SIGNATURE-----

I posted to the list a couple of weeks back now I think about having
problems with multi-byte characters in email subjects (specifically
Japanese)
Alan, What version of RT?

(Found it buried after the logs. Probably better to include long logs
last next time).

It’d be great if you could sent us a message or two that exhibit the bug
as gzipped RFC822 messages, so that we can add em to the test suite

Jesse,

Sorry about that.

No worries. I was reading too quickly.

I’ll try and get a few messages zipped up today. Should I email them to
you directly?

If you could cc rt-devel and bcc rt-bugs@fsck.com, that’d be great.

Thanks!