Email encoding

Samuel_Tardieu · January 11, 2004, 10:36pm

I have a problem with encoding of tickets.

I send mail to RT with “Content-Type: text/plain; charset=iso-8859-1”.
In RT (WWW interface), they show up as improper utf-8 characters.

Moreover, when I send mail through RT, recipients get a mail with
improper accents to, with:

Managed-BY: RT 3.0.8 (Request Tracker — Best Practical Solutions)
X-RT-Original-Encoding: utf-8
Content-Type: text/plain; charset=“iso-8859-1”

Is there a way to get a real conversion iso-8859-1 → utf-8 when a
mail enters RT? Settings in RT_SiteConfig.pm are:

@EmailInputEncodings = qw(iso-8859-1 utf-8 us-ascii) unless (@EmailInputEncodings);
Set($EmailOutputEncoding , ‘iso-8859-1’);

Thanks in advance.

Sam
Samuel Tardieu – sam@rfc1149.net – About me

Ruslan_U_Zakirov · January 12, 2004, 8:12am

Samuel Tardieu wrote:

I have a problem with encoding of tickets.

I send mail to RT with “Content-Type: text/plain; charset=iso-8859-1”.
In RT (WWW interface), they show up as improper utf-8 characters.

Moreover, when I send mail through RT, recipients get a mail with
improper accents to, with:

Managed-BY: RT 3.0.8 (Request Tracker... So much more than a help desk — Best Practical Solutions)
X-RT-Original-Encoding: utf-8
Content-Type: text/plain; charset=“iso-8859-1”

Is there a way to get a real conversion iso-8859-1 → utf-8 when a
mail enters RT? Settings in RT_SiteConfig.pm are:
RT do convert all text parts of emails to UTF-8 and store it in DB.
Internaly RT always try to work with UTF-8.

WebUI always work with UTF-8 encoding, except when you touche download
href, RT use original encoding here.

Outgoing email encoding can be chosen with $EmailOutputEncoding config
option and could be only one. By default it’s UTF, but you’ve changed
this to ‘iso-8859-1’(below).

@EmailInputEncodings = qw(iso-8859-1 utf-8 us-ascii) unless (@EmailInputEncodings);
This option used only if RT fail input encoding detection of email part
then it use Encode::GUESS with your list.
Set($EmailOutputEncoding , ‘iso-8859-1’);
This is used when RT send emails. I use UTF(default), I have to because
of multilang.

Samuel_Tardieu · January 12, 2004, 9:18am

“Ruslan” == Ruslan U Zakirov cubic@acronis.ru writes:

RT do convert all text parts of emails to UTF-8 and store it in DB.

This is the part that is causing me trouble. It looks like the
conversion iso-8859-1 to utf-8 gives improper results.

Internaly RT always try to work with UTF-8.

Yup.

WebUI always work with UTF-8 encoding, except when you touche
download href, RT use original encoding here.

Ok also.

Outgoing email encoding can be chosen with $EmailOutputEncoding
config option and could be only one. By default it’s UTF, but you’ve
changed this to ‘iso-8859-1’(below).

Yes, and this works fine. When I type a comment from the WWW
interface, it gets properly translated into iso-8859-1 in outgoing
emails. The trouble is when iso-8859-1 email enters the system.

@EmailInputEncodings = qw(iso-8859-1 utf-8 us-ascii) unless
(@EmailInputEncodings);
This option used only if RT fail input encoding detection of email
part then it use Encode::GUESS with your list.

It looks like RT cannot recognize that incoming email is in
iso-8859-1. Is there a debugging option I can activate to see what
happens with incoming emails?

Set($EmailOutputEncoding , ‘iso-8859-1’);
This is used when RT send emails. I use UTF(default), I have to
because of multilang.

Yes, this one is fine.

Sam
Samuel Tardieu – sam@rfc1149.net – About me

Jesse_Vincent · January 12, 2004, 9:46am

Can you send me, personally, a message that breaks RT for you? I’ll add
it to the test suite.On Mon, Jan 12, 2004 at 10:18:25AM +0100, Samuel Tardieu wrote:

“Ruslan” == Ruslan U Zakirov cubic@acronis.ru writes:

RT do convert all text parts of emails to UTF-8 and store it in DB.

This is the part that is causing me trouble. It looks like the
conversion iso-8859-1 to utf-8 gives improper results.

Internaly RT always try to work with UTF-8.

Yup.

WebUI always work with UTF-8 encoding, except when you touche
download href, RT use original encoding here.

Ok also.

Outgoing email encoding can be chosen with $EmailOutputEncoding
config option and could be only one. By default it’s UTF, but you’ve
changed this to ‘iso-8859-1’(below).

Yes, and this works fine. When I type a comment from the WWW
interface, it gets properly translated into iso-8859-1 in outgoing
emails. The trouble is when iso-8859-1 email enters the system.

@EmailInputEncodings = qw(iso-8859-1 utf-8 us-ascii) unless
(@EmailInputEncodings);
This option used only if RT fail input encoding detection of email
part then it use Encode::GUESS with your list.

It looks like RT cannot recognize that incoming email is in
iso-8859-1. Is there a debugging option I can activate to see what
happens with incoming emails?

Set($EmailOutputEncoding , ‘iso-8859-1’);
This is used when RT send emails. I use UTF(default), I have to
because of multilang.

Yes, this one is fine.

Sam

Samuel Tardieu – sam@rfc1149.net – About me

rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

Paulo_Matos · January 14, 2004, 5:24pm

Hi!

I've detected some misbehaviour with mail encoding. 
These are the scenarios:

Requestor sends simple email (iso-8859-1) → everything is ok.
RT answers with a simple email → everything is ok.
Requestor sends email with attach (multipart) → everything is ok
RT answers with attach and the conversion isn’t done… see bellow:
— cut here —
…
X-RT-Original-Encoding: utf-8
Content-type: multipart/mixed; boundary=“----------=_1074099935-880-3”
Status: RO
X-Status:
X-Keywords:
X-UID: 46737

This is a multi-part message in MIME format…

------------=_1074099935-880-3
Content-Type: text/plain; charset=“iso-8859-1”

[pm - Wed Jan 14 17:00:51 2004]:

Teste com attach e aÃ§entos Ã¡Ã©iÃ³Ãº… nÃ£o…

…
— end here —

Despite the charset is correct, the mail isn't converted from

utf-8 to iso-8859-1. It happens alaways on an answer with attach…

The only message I get is:

RT: Cannot Encode::Guess; fallback to iso-8859-1 (/usr/lib/rt3/RT/I18N.pm:376)

which I find very odd…

Hope this helps,

Paulo Matos

AJ11 · January 14, 2004, 5:39pm

I actually do not think that is RT’s fault however. Have you tried
attaching the attachment via the gui and then sending it back? That seemed
to work but sending it via email messed up the content type.

I noticed that if you attach via the gui, the headers go in correctly into
the headers and content-type field in the attachments table. However, if
you email it, the attachment goes in as multipart/mixed. I think this is
the behavior of the MTA because it looks like the mailgate just grabs what
it is given. I thought it might be an Exchange thing…From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Paulo Matos
Sent: Wednesday, January 14, 2004 12:25 PM
To: Jesse Vincent
Cc: rt-users@lists.fsck.com; Samuel Tardieu
Subject: Re: [rt-users] Re: Email encoding

Hi!

I've detected some misbehaviour with mail encoding. 
These are the scenarios:

Requestor sends simple email (iso-8859-1) → everything is ok.
RT answers with a simple email → everything is ok.
Requestor sends email with attach (multipart) → everything is ok
RT answers with attach and the conversion isn’t done… see bellow:
— cut here —
…
X-RT-Original-Encoding: utf-8
Content-type: multipart/mixed; boundary=“----------=_1074099935-880-3”
Status: RO
X-Status:
X-Keywords:
X-UID: 46737

This is a multi-part message in MIME format…

------------=_1074099935-880-3
Content-Type: text/plain; charset=“iso-8859-1”

[pm - Wed Jan 14 17:00:51 2004]:

Teste com attach e aÃ§entos Ã¡Ã©iÃ³Ãº… nÃ£o…

…
— end here —

Despite the charset is correct, the mail isn't converted from

utf-8 to iso-8859-1. It happens alaways on an answer with attach…

The only message I get is:

RT: Cannot Encode::Guess; fallback to iso-8859-1
(/usr/lib/rt3/RT/I18N.pm:376)

which I find very odd…

Hope this helps,

Paulo Matos

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Paulo_Matos · January 14, 2004, 6:11pm

aj> I actually do not think that is RT’s fault however. Have you tried
aj> attaching the attachment via the gui and then sending it back? That seemed
aj> to work but sending it via email messed up the content type.

All interaction described before as "RT" is done via web

interface…

aj> I noticed that if you attach via the gui, the headers go in correctly into
aj> the headers and content-type field in the attachments table. However, if
aj> you email it, the attachment goes in as multipart/mixed. I think this is
aj> the behavior of the MTA because it looks like the mailgate just grabs what
aj> it is given. I thought it might be an Exchange thing…

I don't use Exchange as MTA! I haven't yet debug what reaches the

mailgate. But the construction of the message (including attachments) is
done by RT.

RT does it all ok (set boundaries, content-type, charset), except

translating the 1st attachment (the response message itself) from utf-8 to
iso-8859-1.

Paulo Matos

AJ11 · January 14, 2004, 6:49pm

I apologize, my misunderstanding.

How is the weather in Portugal? Spent some time at Compultense in Madrid but
never had the chance to check out anything much beyond that.

Have a prosperous year.-----Original Message-----
From: Paulo Matos [mailto:pjsm@fct.unl.pt]
Sent: Wednesday, January 14, 2004 1:11 PM
To: AJ
Cc: ‘Jesse Vincent’; ‘Samuel Tardieu’; rt-users@lists.fsck.com
Subject: RE: [rt-users] Re: Email encoding

On Wed, 14 Jan 2004, AJ wrote:

aj> I actually do not think that is RT’s fault however. Have you tried
aj> attaching the attachment via the gui and then sending it back? That
seemed
aj> to work but sending it via email messed up the content type.

All interaction described before as "RT" is done via web

interface…

aj> I noticed that if you attach via the gui, the headers go in correctly
into
aj> the headers and content-type field in the attachments table. However,
if
aj> you email it, the attachment goes in as multipart/mixed. I think this
is
aj> the behavior of the MTA because it looks like the mailgate just grabs
what
aj> it is given. I thought it might be an Exchange thing…

I don't use Exchange as MTA! I haven't yet debug what reaches the

mailgate. But the construction of the message (including attachments) is
done by RT.

RT does it all ok (set boundaries, content-type, charset), except

translating the 1st attachment (the response message itself) from utf-8 to
iso-8859-1.

Paulo Matos