Encoding Troubles with RT 3.8.8

RT Users
1

Hi,

I’ve some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

Exemple: Email received in ‘iso-8859-1’, is correctly displayed in
UTF-8 within RT,
when showing ‘Full Header’ we have on top:
content-type: text/plain; charset=“utf-8”

and on bottom:
X-RT-Original-Encoding: ISO-8859-1

If we forward this email to a new address, the forwarded message
include the same headers, however the message body is in “quoted
printable” encoding of a iso-8859-1 message:

==Extract of forwarded email: ======================================
This is a forward of ticket #260

------------=_1280925536-13250-3
Content-Type: message/rfc822
Content-Disposition: attachment
Content-Transfer-Encoding: binary
Content-Description: forwarded message

Content-Type: multipart/mixed; boundary="----------=_1280925536-13250-2"
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)

This is a multi-part message in MIME format…

------------=_1280925536-13250-2
MIME-Version: 1.0
X-RT-Queue: Support
Content-Type: text/plain; charset=“utf-8”
…
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 1116

Bonjour,

Je fais suite =E0 notre entretien t=E9l=E9phonique.
…
==Extract of forwarded email: ======================================

Also, there is a similar problem when using html mail/template,
accentuated chars are converted to iso-8859-1 chars for the
"text/plain" version of the mail with is send with an utf8 headers.
The text/html part contain the html-entity codes so there is no problem.

When not using html templates entities are correctly converted using
utf8 charset.

2

Hi,

I’ve some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

…

Also, there is a similar problem when using html mail/template,
accentuated chars are converted to iso-8859-1 chars for the
“text/plain” version of the mail with is send with an utf8 headers.
The text/html part contain the html-entity codes so there is no problem.

When not using html templates entities are correctly converted using
utf8 charset.

For this part, in _DowngradeFromHTML there is two thing:

  • forced ‘utf8’ mime charset, why not using “EmailOutputEncoding”

  • use of HTML::FormatText which can only generate latin1 text

    my $charset = RT->Config->Get(‘EmailOutputEncoding’);

    my $new_entity = $orig_entity->dup; # this will fail badly if we
    go away from InCore parsing
    $new_entity->head->mime_attr( “Content-Type” => ‘text/plain’ );
    $new_entity->head->mime_attr( “Content-Type.charset” => $charset );

    $orig_entity->head->mime_attr( “Content-Type” => ‘text/html’ );
    $orig_entity->head->mime_attr( “Content-Type.charset” => $charset );
    $orig_entity->make_multipart(‘alternative’, Force => 1);

    require HTML::FormatText;
    require HTML::TreeBuilder;
    require Encode;
    my $tree = HTML::TreeBuilder->new_from_content(
    $new_entity->bodyhandle->as_string
    );
    $new_entity->bodyhandle(MIME::Body::InCore->new(
    (
    Encode::from_to(scalar HTML::FormatText->new(
    leftmargin => 0,
    rightmargin => 78,
    )->format( $tree ),
    ‘iso-8859-1’ => $charset
    )
    )
    ));
    $tree->delete;
    …

3

Hi,

I’ve some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

Exemple: Email received in ‘iso-8859-1’, is correctly displayed in
UTF-8 within RT,
when showing ‘Full Header’ we have on top:
content-type: text/plain; charset=“utf-8”

and on bottom:
X-RT-Original-Encoding: ISO-8859-1

If we forward this email to a new address, the forwarded message
include the same headers, however the message body is in “quoted
printable” encoding of a iso-8859-1 message:

==Extract of forwarded email: ======================================
This is a forward of ticket #260

------------=_1280925536-13250-3
Content-Type: message/rfc822
Content-Disposition: attachment
Content-Transfer-Encoding: binary
Content-Description: forwarded message

Content-Type: multipart/mixed; boundary="----------=_1280925536-13250-2"
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)

This is a multi-part message in MIME format…

------------=_1280925536-13250-2
MIME-Version: 1.0
X-RT-Queue: Support
Content-Type: text/plain; charset=“utf-8”
…
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 1116

Bonjour,

Je fais suite =E0 notre entretien t=E9l=E9phonique.
…
==Extract of forwarded email: ======================================

Also, there is a similar problem when using html mail/template,
accentuated chars are converted to iso-8859-1 chars for the
"text/plain" version of the mail with is send with an utf8 headers.
The text/html part contain the html-entity codes so there is no problem.

When not using html templates entities are correctly converted using
utf8 charset.

For this part, in _DowngradeFromHTML there is two thing:

  • forced ‘utf8’ mime charset, why not using “EmailOutputEncoding”

  • use of HTML::FormatText which can only generate latin1 text

    my $charset = RT->Config->Get(‘EmailOutputEncoding’);

    my $new_entity = $orig_entity->dup; # this will fail badly if we go
    away from InCore parsing
    $new_entity->head->mime_attr( “Content-Type” => ‘text/plain’ );
    $new_entity->head->mime_attr( “Content-Type.charset” => $charset );

    $orig_entity->head->mime_attr( “Content-Type” => ‘text/html’ );
    $orig_entity->head->mime_attr( “Content-Type.charset” => $charset );
    $orig_entity->make_multipart(‘alternative’, Force => 1);

    require HTML::FormatText;
    require HTML::TreeBuilder;
    require Encode;
    my $tree = HTML::TreeBuilder->new_from_content(
    $new_entity->bodyhandle->as_string
    );
    $new_entity->bodyhandle(MIME::Body::InCore->new(
    (
    Encode::encode($charset,
    Encode::decode(“iso-8859-1”,
    scalar HTML::FormatText->new(
    leftmargin => 0,
    rightmargin => 78,
    )->format( $tree )
    )
    )
    )
    );
    $tree->delete;
    …

4

To solve this one, for now i’ve done the following modification, since
we are sending the modified headers,
i changed the RT::Attachment_Overlay::ContentAsMIME to send the
modified content:

— /opt/rt3/lib/RT/Attachment_Overlay.pm 2010-06-03
12:55:03.540297000 +0200
+++ /opt/rt3/local/lib/RT/Attachment_Overlay.pm 2010-08-07
13:57:47.411918000 +0200
@@ -426,7 +426,7 @@

 use MIME::Body;
 $entity->bodyhandle(

  •    MIME::Body::Scalar->new( $self->OriginalContent )

  •    MIME::Body::Scalar->new( $self->Content )

    );

    return $entity;

Any advice ?2010/8/4 benoit plessis plessis.benoit@gmail.com

Hi,

I’ve some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

Exemple: Email received in ‘iso-8859-1’, is correctly displayed in
UTF-8 within RT,
when showing ‘Full Header’ we have on top:
content-type: text/plain; charset=“utf-8”

and on bottom:
X-RT-Original-Encoding: ISO-8859-1

If we forward this email to a new address, the forwarded message
include the same headers, however the message body is in “quoted
printable” encoding of a iso-8859-1 message:

==Extract of forwarded email: ======================================
This is a forward of ticket #260

------------=_1280925536-13250-3
Content-Type: message/rfc822
Content-Disposition: attachment
Content-Transfer-Encoding: binary
Content-Description: forwarded message

Content-Type: multipart/mixed; boundary=“----------=_1280925536-13250-2”
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)

This is a multi-part message in MIME format…

------------=_1280925536-13250-2
MIME-Version: 1.0
X-RT-Queue: Support
Content-Type: text/plain; charset=“utf-8”
…
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 1116

Bonjour,

Je fais suite =E0 notre entretien t=E9l=E9phonique.
…
==Extract of forwarded email: =====================================

5

To solve this one, for now i’ve done the following modification, since
we are sending the modified headers,
i changed the RT::Attachment_Overlay::ContentAsMIME to send the
modified content:

— /opt/rt3/lib/RT/Attachment_Overlay.pm 2010-06-03
12:55:03.540297000 +0200
+++ /opt/rt3/local/lib/RT/Attachment_Overlay.pm 2010-08-07
13:57:47.411918000 +0200
@@ -426,7 +426,7 @@

 use MIME::Body;
 $entity->bodyhandle(

  •    MIME::Body::Scalar->new( $self->OriginalContent )

  •    MIME::Body::Scalar->new( $self->Content )

    );

    return $entity;

Any advice ?

Thanks for digging into this.

I’ve bounced some of your comments into the bug tracker, which is
probably the right place for them at this point.

-kevin> 2010/8/4 benoit plessis plessis.benoit@gmail.com

Hi,

I’ve some problems with iso-8859-1/utf8 encoding of outbound mail with RT.

Exemple: Email received in ‘iso-8859-1’, is correctly displayed in
UTF-8 within RT,
when showing ‘Full Header’ we have on top:
content-type: text/plain; charset=“utf-8”

and on bottom:
X-RT-Original-Encoding: ISO-8859-1

If we forward this email to a new address, the forwarded message
include the same headers, however the message body is in “quoted
printable” encoding of a iso-8859-1 message:

==Extract of forwarded email: ======================================
This is a forward of ticket #260

------------=_1280925536-13250-3
Content-Type: message/rfc822
Content-Disposition: attachment
Content-Transfer-Encoding: binary
Content-Description: forwarded message

Content-Type: multipart/mixed; boundary=“----------=_1280925536-13250-2”
Content-Transfer-Encoding: binary
MIME-Version: 1.0
X-Mailer: MIME-tools 5.427 (Entity 5.427)

This is a multi-part message in MIME format…

------------=_1280925536-13250-2
MIME-Version: 1.0
X-RT-Queue: Support
Content-Type: text/plain; charset=“utf-8”
…
X-RT-Original-Encoding: ISO-8859-1
Content-Length: 1116

Bonjour,

Je fais suite =E0 notre entretien t=E9l=E9phonique.
…
==Extract of forwarded email: =====================================

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com