Encoding bug in ticket creation

Hello,

Ticket creation (via the web interface) with a subject containing
accented characters (e.g. “�”) and empty content/body fails with the
following errors logged by RT:

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid byte sequence for encoding “UTF8”: 0xf60a436f HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by “client_encoding”. (/opt/rt3/lib/RT/Attachment_Overlay.pm:153)
Aug 15 09:03:13 web1 RT: About to think about scrips for transaction #795
Aug 15 09:03:13 web1 RT: About to prepare scrips for transaction #795
Aug 15 09:03:13 web1 RT: RT::Ticket=HASH(0x9a6a9f8) tried to load a bogus ticket: 86
Aug 15 09:03:14 web1 RT: RT::Scrips=HASH(0x9a5eab4) couldn’t load ticket 86 (/opt/rt3/lib/RT/Scrips_Overlay.pm:291)
Aug 15 09:03:14 web1 RT: Found 2 scrips for TransactionCreate stage with applicable type(s) Create
Aug 15 09:03:14 web1 RT: RT::Ticket=HASH(0x9a74c48) tried to load a bogus ticket: 86
Aug 15 09:03:14 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:15 web1 RT: RT::Ticket=HASH(0x9a90384) tried to load a bogus ticket: 86
Aug 15 09:03:15 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:15 web1 RT: RT::Ticket=HASH(0x9a6b34c) tried to load a bogus ticket: 86
Aug 15 09:03:15 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:16 web1 RT: About to commit scrips for transaction #795
Aug 15 09:03:16 web1 RT: Committing scrip #11 on txn #795 of ticket #
Aug 15 09:03:16 web1 RT: RT::Ticket=HASH(0x9a8da50) tried to load a bogus ticket: 86
Aug 15 09:03:16 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:17 web1 RT: Tried to load a bogus ticket id: ''
Aug 15 09:03:17 web1 RT: RT::Ticket=HASH(0x9996da0) tried to load a bogus ticket: 86
Aug 15 09:03:17 web1 RT: Ticket couldn’t be created: (/opt/rt3/lib/RT/Ticket_Overlay.pm:667)
Aug 15 09:03:17 web1 RT: WebRT: Ticket could not be created due to an internal error (/opt/rt3/share/html/Elements/Error:82)

It can be worked around by adding some content (accented or not).

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.

Regards,
Daniel N�ri daniel.neri@sigicom.com
Sigicom AB, Stockholm, Sweden

  Hello,

Ticket creation (via the web interface) with a subject containing
accented characters (e.g. “ö”) and empty content/body fails with the
following errors logged by RT:

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid
byte sequence for encoding “UTF8”: 0xf60a436f HINT: This error can
also happen if the byte sequence does not match the encoding
expected by the server, which is controlled by “client_encoding”. (/
opt/rt3/lib/RT/Attachment_Overlay.pm:153)
Aug 15 09:03:13 web1 RT: About to think about scrips for transaction
#795
Aug 15 09:03:13 web1 RT: About to prepare scrips for transaction #795

It can be worked around by adding some content (accented or not).

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.

What version of DBD::Pg? What version of Perl?

Does it happen if you submit the same thing by email or a custom field
contains an ö?

-j

Forgot to CC the list.From: Daniel Néri daniel.neri@sigicom.com
Date: Fri, Aug 15, 2008 at 11:54 AM
Subject: Re: [Rt-devel] Encoding bug in ticket creation
To: Jesse Vincent jesse@bestpractical.com

Hi,

What version of DBD::Pg? What version of Perl?

Currently 2.9.0, but also tested with 1.49. Perl version 5.8.8 (Ubuntu
8.04 LTS).

Does it happen if you submit the same thing by email or a custom field
contains an ö?

No problem by e-mail. Haven’t tested a custom field.

I couldn’t resist trying it on rt3.fsck.com (Sorry!); see ticket
11600. I guess MySQL is less strict, but the encoding seems broken
anyway.

Regards,
Daniel Néri daniel.neri@sigicom.com
Sigicom AB, Stockholm, Sweden | http://www.sigicom.com/

Daniel Néri daniel.neri@sigicom.com
Sigicom AB, Stockholm, Sweden | http://www.sigicom.com/

Hello,

Ticket creation (via the web interface) with a subject containing
accented characters (e.g. “�”) and empty content/body fails with the
following errors logged by RT:

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid byte sequence for encoding “UTF8”: 0xf60a436f HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by “client_encoding”. (/opt/rt3/lib/RT/Attachment_Overlay.pm:153)
Aug 15 09:03:13 web1 RT: About to think about scrips for transaction #795
Aug 15 09:03:13 web1 RT: About to prepare scrips for transaction #795
Aug 15 09:03:13 web1 RT: RT::Ticket=HASH(0x9a6a9f8) tried to load a bogus ticket: 86
Aug 15 09:03:14 web1 RT: RT::Scrips=HASH(0x9a5eab4) couldn’t load ticket 86 (/opt/rt3/lib/RT/Scrips_Overlay.pm:291)
Aug 15 09:03:14 web1 RT: Found 2 scrips for TransactionCreate stage with applicable type(s) Create
Aug 15 09:03:14 web1 RT: RT::Ticket=HASH(0x9a74c48) tried to load a bogus ticket: 86
Aug 15 09:03:14 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:15 web1 RT: RT::Ticket=HASH(0x9a90384) tried to load a bogus ticket: 86
Aug 15 09:03:15 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:15 web1 RT: RT::Ticket=HASH(0x9a6b34c) tried to load a bogus ticket: 86
Aug 15 09:03:15 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:16 web1 RT: About to commit scrips for transaction #795
Aug 15 09:03:16 web1 RT: Committing scrip #11 on txn #795 of ticket #
Aug 15 09:03:16 web1 RT: RT::Ticket=HASH(0x9a8da50) tried to load a bogus ticket: 86
Aug 15 09:03:16 web1 RT: HasRight called with no valid object (/opt/rt3/lib/RT/Principal_Overlay.pm:322)
Aug 15 09:03:17 web1 RT: Tried to load a bogus ticket id: ''
Aug 15 09:03:17 web1 RT: RT::Ticket=HASH(0x9996da0) tried to load a bogus ticket: 86
Aug 15 09:03:17 web1 RT: Ticket couldn’t be created: (/opt/rt3/lib/RT/Ticket_Overlay.pm:667)
Aug 15 09:03:17 web1 RT: WebRT: Ticket could not be created due to an internal error (/opt/rt3/share/html/Elements/Error:82)

It can be worked around by adding some content (accented or not).

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.

Regards,
Daniel N�ri daniel.neri@sigicom.com
Sigicom AB, Stockholm, Sweden

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid
byte sequence for encoding “UTF8”: 0xf60a436f HINT: This error can
also happen if the byte sequence does not match the encoding
expected by the server, which is controlled by “client_encoding”. (/
opt/rt3/lib/RT/Attachment_Overlay.pm:153)

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.

What version of DBD::Pg? What version of Perl?

Does it happen if you submit the same thing by email or a custom field
contains an ᅵ?

FYI, I’m seeing the same error.

Here it’s RT 3.7.21 / one of the RTIR milestone releases. The rest is
Debian stable, thus libdbd-pg-perl at 1.49-2, perl at 5.8.8-7etch3.

In my case it’s triggered by an incoming email, and I see the error in
the postgres logfile as well:

2008-08-26 11:25:07 CEST ERROR: invalid byte sequence for encoding “UTF8”: 0xe92064
2008-08-26 11:25:07 CEST HINT: This error can also happen if the byte sequence does not match the encoding expected by the server, which is controlled by “client_encoding”.

Drilling down on the email (It really pays to keep a copy of each
incoming mail) I found it labeled as:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

There are a few signs of QP being applied, e.g.

---------------------------------------------------------------=20

Now the kicker: the mail has a legal disclaimer appended (NOT
attached!), which containes 8-bit characters. Checking the
encoding shows that they are in latin1 and not utf-8.

Great. In other words, the incoming mail is invalid. (removing the
disclaimer makes RT indeed happy.)

Well, this is suboptimal. As a CERT, we can’t just ignore slightly
broken mails. Any ideas?

/ol
-=- Otmar Lendl – ol@bofh.priv.at -=-

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid
byte sequence for encoding “UTF8”: 0xf60a436f HINT: This error can
also happen if the byte sequence does not match the encoding
expected by the server, which is controlled by “client_encoding”. (/
opt/rt3/lib/RT/Attachment_Overlay.pm:153)

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.

What version of DBD::Pg? What version of Perl?

Does it happen if you submit the same thing by email or a custom
field
contains an ö?

FYI, I’m seeing the same error.

Here it’s RT 3.7.21 / one of the RTIR milestone releases. The rest is
Debian stable, thus libdbd-pg-perl at 1.49-2, perl at 5.8.8-7etch3.

In my case it’s triggered by an incoming email, and I see the error in
the postgres logfile as well:

2008-08-26 11:25:07 CEST ERROR: invalid byte sequence for encoding
"UTF8": 0xe92064
2008-08-26 11:25:07 CEST HINT: This error can also happen if the
byte sequence does not match the encoding expected by the server,
which is controlled by “client_encoding”.

Drilling down on the email (It really pays to keep a copy of each
incoming mail) I found it labeled as:

Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: quoted-printable

There are a few signs of QP being applied, e.g.

---------------------------------------------------------------=20

Now the kicker: the mail has a legal disclaimer appended (NOT
attached!), which containes 8-bit characters. Checking the
encoding shows that they are in latin1 and not utf-8.

Great. In other words, the incoming mail is invalid. (removing the
disclaimer makes RT indeed happy.)

Well, this is suboptimal. As a CERT, we can’t just ignore slightly
broken mails. Any ideas?

There’s a switch to perl’s UTF8 handling to replace broken utf8 in
text with blanks. Alternatively, QP could get forced before write to
the DB on sensitive databases.

The routine you want to be looking at / fixing is lib/RT/Record.pm’s
sub _EncodeLOB.

A patch would be most welcome.

-jesse

Jesse Vincent writes:

Aug 15 09:03:13 web1 RT: Attachment insert failed - ERROR: invalid
byte sequence for encoding “UTF8”: 0xf60a436f HINT: […]
(/opt/rt3/lib/RT/Attachment_Overlay.pm:153)

I’m currently using RT 3.8.1rc5 with PostgreSQL 8.3.
[…]
FYI, I’m seeing the same error.

Here it’s RT 3.7.21 / one of the RTIR milestone releases. The rest is
Debian stable, thus libdbd-pg-perl at 1.49-2, perl at 5.8.8-7etch3.

In my case it’s triggered by an incoming email, and I see the error in
the postgres logfile as well:

2008-08-26 11:25:07 CEST ERROR: invalid byte sequence for encoding
"UTF8": 0xe92064 […]

Drilling down on the email (It really pays to keep a copy of each
incoming mail) I found it labeled as:

Content-Type: text/plain; charset=“utf-8”
[…]
Now the kicker: the mail has a legal disclaimer appended (NOT
attached!), which containes 8-bit characters. Checking the
encoding shows that they are in latin1 and not utf-8.

Great. In other words, the incoming mail is invalid. (removing the
disclaimer makes RT indeed happy.)

Well, this is suboptimal. As a CERT, we can’t just ignore slightly
broken mails. Any ideas?

There’s a switch to perl’s UTF8 handling to replace broken utf8 in
text with blanks. Alternatively, QP could get forced before write to
the DB on sensitive databases.

The routine you want to be looking at / fixing is lib/RT/Record.pm’s
sub _EncodeLOB.

A patch would be most welcome.

Has anybody patched this “bug” in the meanwhile?

One of my colleagues can’t create tickets through the web interface if
his ticket subject contains umlaut characters. He is able to insert
umlaut characters into the subject after he created the ticket though.
My colleague uses Firefox 3 and Opera on Ubuntu 8.10. Both Firefox and
Opera fail.

Here’s part of the error log file:

[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding “UTF8”: 0xfc, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6
[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: RT::Handle=HASH(0xa1ea8c8) couldn’t execute the query ‘INSERT INTO Attachments (ContentType, Parent, Subject, Headers, Creator, MessageId, Created, TransactionId) VALUES (?, ?, ?, ?, ?, ?, ?, ?)’ at /usr/local/share/perl/5.8.8/DBIx/SearchBuilder/Handle.pm line 518, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6
[Fri Nov 14 11:51:02 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: \tDBIx::SearchBuilder::Handle::SimpleQuery(‘RT::Handle=HASH(0xa1ea8c8)’, ‘INSERT INTO Attachments (ContentType, Parent, Subject, Header…’, ‘text/plain’, 0, ‘Support f\xc3\xbcr Synapse 2008’, ‘Content-Type: text/plain; boundary="----------=_1226659862-10…’, 40, ‘’, ‘2008-11-14 10:51:02’, …) called at /usr/local/share/perl/5.8.8/DBIx/SearchBuilder/Handle.pm line 353, referer: https://www.example.com/rt/Ticket/Create.html?Queue=6

0xfc is the Latin-1 code for “�”. \xc3\xbc is the byte representation
of “�” in UTF-8.

I am able to reproduce this error on my colleague’s Ubuntu machine, but
I couldn’t reproduce it on other computers running OpenBSD, CentOS or
Windows.

Occasionally, the error “invalid byte sequence for encoding” is also
triggered by mail messages sent from iPhones. It seems that the
iPhone’s mail software mime-encodes the message subject but not other
mail headers like thread-topic.

I got several messages from iPhones where the subject and the body were
encoded in UTF-8 but where the thread-topic header contained non-encoded
Latin-1 characters. Here’s an example:

Content-Type: text/plain;
format=flowed;
delsp=yes;
charset="utf-8"
Content-Transfer-Encoding: base64
thread-topic: [rt.example.com #1234] Bitte um R�ckruf am Mittwoch
thread-index: AclI3xNq4KwWEYT8QX+ziU8EtI/A4g==
MIME-Version: 1.0 (iPhone Mail 5F136)

The subject in this message contains an “�” but I think that this
failure was triggered by the “�” in the thread-topic header, since the
subject is properly mime-encoded.

[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: DBD::Pg::st execute failed: ERROR: invalid byte sequence for encoding “UTF8”: 0xfc
[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: RT::Handle=HASH(0x9f80578) couldn’t execute the query ‘INSERT INTO Attachments (ContentType, Parent, Subject, Filename, Headers, Creator, MessageId, Created, Content, ContentEncoding, TransactionId) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)’ at /usr/local/share/perl/5.10.0/DBIx/SearchBuilder/Handle.pm line 518
[Mon Nov 17 19:05:42 2008] [error] [client 192.168.0.1] FastCGI: server “/opt/rt3/bin/mason_handler.fcgi” stderr: \tDBIx::SearchBuilder::Handle::SimpleQuery(‘RT::Handle=HASH(0x9f80578)’, ‘INSERT INTO Attachments (ContentType, Parent, Subject, Filena…’, ‘text/plain’, 0, ‘Re: [rt.example.com #1234] Bitte um R\x{c3}\x{bc}ckruf am Mitt…’, undef, ‘X-Amavis-Alert: BAD HEADER Non-encoded 8-bit data (char FC he…’, 1366, ‘6FF0EA71-833A-4E91-8CD9-B2DF7AA22DE8@example.com’, …) called at /usr/local/share/perl/5.10.0/DBIx/SearchBuilder/Handle.pm line 353

We use Request Tracker 3.8.1. Our RT database instance runs under
Debian Lenny and PostgreSQL 8.3.4. I upgraded the RT server yesterday
from Debian Etch to Lenny, which provides Perl 5.10 and libdbd-pg-perl
2.8.7, but that upgrade didn’t change anything.

I’m currently reading Attachment_Overlay.pm but I’m not sure whether the
subject and the head are supposed to be encoded or decoded strings.
What kind of strings do the MIME::Head methods return? And what kind of
strings do the database modules expect?

Regards,
Andreas

avo@trustsec.de (Andreas V�gele) writes:

Has anybody patched this “bug” in the meanwhile?

No, I’ve tried to debug it but my Perl-fu is too weak.

One of my colleagues can’t create tickets through the web interface if
his ticket subject contains umlaut characters.

In my experience, it only breaks when the description field is empty.

Regards,
Daniel N�ri daniel.neri@sigicom.com
Sigicom AB, Stockholm, Sweden