Attachment encoding causes internal server error

RT Users
1

Hi everyone!

I’m using rt 3.8.8 with debian linux 5.0 and postgresql 8.3.12.

When sending e-mail with an attachment to RT, i receive following error
message:
“Ticket could not be created due to an internal error”

After some investigation i discovered, that this error occurs only, when
attachment name contains iso-8859-1 letters and mail is sent using
iso-5589-1 encoding. Sending the same file with mail encoding utf8 works
fine.

When error occurs, these messages will be logged into syslog:
Nov 15 22:40:56 rt RT: Converting ‘ISO-8859-1’ to ‘utf-8’ for text/plain

  • Subjectless message
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Converting ‘ascii’ to ‘utf-8’ for text/plain -
    Subjectless message
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Mail from user #23 (some@mail.address)
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #349
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #350
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #351
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #352
    Nov 15 22:40:56 rt RT: Attachment insert failed: ERROR: invalid byte
    sequence for encoding “UTF8”: 0xf6fcf5e4#012HINT: This error can also
    happen if the byte sequence does not match the encoding expected by the
    server, which is controlled by “client_encoding”.
    (/opt/rt3/bin/…/lib/RT/Attachment_Overlay.pm:192)
    Nov 15 22:40:56 rt RT: Attachment insert failed: ERROR: invalid byte
    sequence for encoding “UTF8”: 0xf6fcf5e4#012HINT: This error can also
    happen if the byte sequence does not match the encoding expected by the
    server, which is controlled by “client_encoding”.
    (/opt/rt3/bin/…/lib/RT/Attachment_Overlay.pm:165)
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #353
    Nov 15 22:40:56 rt RT: About to prepare scrips for transaction #353
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x50005f0) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: RT::Scrips=HASH(0x500d580) couldn’t load ticket
    34 (/opt/rt3/bin/…/lib/RT/Scrips_Overlay.pm:291)
    Nov 15 22:40:56 rt RT: Found 5 scrips for TransactionCreate stage with
    applicable type(s) Create
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4fc5008) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66bc0) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4be7000) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66bd8) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x5004930) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x5014810) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66770) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: About to commit scrips for transaction #353
    Nov 15 22:40:56 rt RT: Committing scrip #11 on txn #353 of ticket #
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x501c8a8) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: Tried to load a bogus ticket id: ''
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4ff5840) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: Ticket couldn’t be created:
    (/opt/rt3/bin/…/lib/RT/Ticket_Overlay.pm:679)
    Nov 15 22:40:56 rt RT: Ticket could not be created due to an internal
    error (/opt/rt3/bin/…/lib/RT/Interface/Email.pm:244)

There are also the “invalid byte sequence for encoding “UTF8”:
0xf6fcf5e4” messages in the Postgresql log.

I understand(correct me if i’m wrong) that the cause of the problem is
that RT is trying to insert into UTF8 database something that is not
UTF8. And the reason of that is that for some reason RT does guess’ the
attachments encoding wrong. Allthough the log says “Subjectless
message”, the subject line of the message is not empty.
Does anyone has any idea why is RT guessing the encoding(wrong),
allthough it sees that encoding is ISO-8859-1? Is there any way i can
change this behavior?
I tried to fiddle with the
Set(@EmailInputEncodings, qw(utf-8 iso-8859-1 us-ascii)); option, but so
far no luck.

Searching google did’nt give anything useful so far.

Any help would be much appreciated.

all the best,
Joosep Meus

2

Hi again!

I should probably clarify that problem is related to iso-8859-1 special
characters. Exapmle filename i used for testing was “����.txt”.
Attachments, that do not contain those characters will not cause the
problem.

All the best,
Joosep MEusOn 11/15/2010 11:32 PM, Joosep Meus wrote:

Hi everyone!

I’m using rt 3.8.8 with debian linux 5.0 and postgresql 8.3.12.

When sending e-mail with an attachment to RT, i receive following error
message:
“Ticket could not be created due to an internal error”

After some investigation i discovered, that this error occurs only, when
attachment name contains iso-8859-1 letters and mail is sent using
iso-5589-1 encoding. Sending the same file with mail encoding utf8 works
fine.

When error occurs, these messages will be logged into syslog:
Nov 15 22:40:56 rt RT: Converting ‘ISO-8859-1’ to ‘utf-8’ for text/plain

  • Subjectless message
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Converting ‘ascii’ to ‘utf-8’ for text/plain -
    Subjectless message
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Guessed encoding: ascii
    Nov 15 22:40:56 rt RT: Mail from user #23 (some@mail.address)
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #349
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #350
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #351
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #352
    Nov 15 22:40:56 rt RT: Attachment insert failed: ERROR: invalid byte
    sequence for encoding “UTF8”: 0xf6fcf5e4#012HINT: This error can also
    happen if the byte sequence does not match the encoding expected by the
    server, which is controlled by “client_encoding”.
    (/opt/rt3/bin/…/lib/RT/Attachment_Overlay.pm:192)
    Nov 15 22:40:56 rt RT: Attachment insert failed: ERROR: invalid byte
    sequence for encoding “UTF8”: 0xf6fcf5e4#012HINT: This error can also
    happen if the byte sequence does not match the encoding expected by the
    server, which is controlled by “client_encoding”.
    (/opt/rt3/bin/…/lib/RT/Attachment_Overlay.pm:165)
    Nov 15 22:40:56 rt RT: About to think about scrips for transaction #353
    Nov 15 22:40:56 rt RT: About to prepare scrips for transaction #353
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x50005f0) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: RT::Scrips=HASH(0x500d580) couldn’t load ticket
    34 (/opt/rt3/bin/…/lib/RT/Scrips_Overlay.pm:291)
    Nov 15 22:40:56 rt RT: Found 5 scrips for TransactionCreate stage with
    applicable type(s) Create
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4fc5008) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66bc0) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4be7000) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66bd8) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x5004930) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x5014810) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4f66770) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: About to commit scrips for transaction #353
    Nov 15 22:40:56 rt RT: Committing scrip #11 on txn #353 of ticket #
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x501c8a8) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: HasRight called with no valid object
    (/opt/rt3/bin/…/lib/RT/Principal_Overlay.pm:322)
    Nov 15 22:40:56 rt RT: Tried to load a bogus ticket id: ‘’
    Nov 15 22:40:56 rt RT: RT::Ticket=HASH(0x4ff5840) tried to load a bogus
    ticket: 34
    Nov 15 22:40:56 rt RT: Ticket couldn’t be created:
    (/opt/rt3/bin/…/lib/RT/Ticket_Overlay.pm:679)
    Nov 15 22:40:56 rt RT: Ticket could not be created due to an internal
    error (/opt/rt3/bin/…/lib/RT/Interface/Email.pm:244)

There are also the “invalid byte sequence for encoding “UTF8”:
0xf6fcf5e4” messages in the Postgresql log.

I understand(correct me if i’m wrong) that the cause of the problem is
that RT is trying to insert into UTF8 database something that is not
UTF8. And the reason of that is that for some reason RT does guess’ the
attachments encoding wrong. Allthough the log says “Subjectless
message”, the subject line of the message is not empty.
Does anyone has any idea why is RT guessing the encoding(wrong),
allthough it sees that encoding is ISO-8859-1? Is there any way i can
change this behavior?
I tried to fiddle with the
Set(@EmailInputEncodings, qw(utf-8 iso-8859-1 us-ascii)); option, but so
far no luck.

Searching google did’nt give anything useful so far.

Any help would be much appreciated.

all the best,
Joosep Meus

3

Hi again!

I should probably clarify that problem is related to iso-8859-1
special characters. Exapmle filename i used for testing was
“����.txt”. Attachments, that do not contain those characters will
not cause the problem.

this is a known bug. See the story with some patches here:
http://issues.bestpractical.com/Ticket/Display.html?id=14913&user=guest&pass=guest

4

Hi again!

I should probably clarify that problem is related to iso-8859-1
special characters. Exapmle filename i used for testing was
“����.txt”. Attachments, that do not contain those characters will
not cause the problem.

this is a known bug. See the story with some patches here:
http://issues.bestpractical.com/Ticket/Display.html?id=14913&user=guest&pass=guest

Thanks alot. Patch from the link you provided fixed the problem for me.

All the best,
Joosep Meus