Problems with character encoding on second post (first OK) (Intern)

ronny.pettersen · October 27, 2003, 11:00am

RT : 3.0.6 FastCGI-version
Apache: 2.0.40
Linux : RedHat 9

I have a kindof strange problem occuring when I’m posting a message containing
the norwegian special characters “æøå”.
After the webserver is restartet, the first post always give correct results.
The second post always encodes the characters wrong. “Test æøå” becomes “test Ã¦Ã¸Ã¥”.

Does anyone know what might cause this? See below for the RT debug-log.
Configuration issue, some perl module issue?
(“make testdeps” reports ok on all perl modules)

Regards,
Ronny Pettersen

from debug rt.log

First post with international characters gives correct encoding (message “test æøå” → “test æøå”):

[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.9.95786969530812@test.com #3/68 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:02 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.9.95786969530812@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.8.33597934411323@test.com #3/68 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:02 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:03 2003] [info]: rt-3.0.6-3-68.8.33597934411323@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Second posting with international characters gives wrong encoding (message “test æøå” → “test Ã¦Ã¸Ã¥”)

[Mon Oct 27 09:35:20 2003] [debug]: Guessed encoding: iso-8859-1 (/usr/local/rt3/lib/RT/I18N.pm:351)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘iso-8859-1’ to ‘utf-8’ for text/plain - (/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.5.6143501569494@test.com #3/69 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.5.6143501569494@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.14.9025718099913@test.com #3/69 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.14.9025718099913@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Post without international characters (message “Tesing without int chars”)

Post number 1 and 2 after restart gives the same behaviour

[Mon Oct 27 10:19:44 2003] [debug]: Guessed encoding: ascii (/usr/local/rt3/lib/RT/I18N.pm:351)
[Mon Oct 27 10:19:44 2003] [debug]: Converting ‘ascii’ to ‘utf-8’ for text/plain - (/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.19.8192251337203@test.com #3/75 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 10:19:45 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.19.8192251337203@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.3.56622094384051@test.com #3/75 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 10:19:45 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.3.56622094384051@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Short Perl-module list, just in case it would be of any interest:

HTML::Mason - 1.23
HTML::Parser - 3.31
FCGI - 0.67
DBIx::SearchBuilder - 0.92
Storable - 2.08
Locale::Maketext - 1.06
Locale::Maketext::Lexicon - 0.32
mod_perl - 1.9907

ronny.pettersen · October 29, 2003, 1:11pm

A little update on this…

I wiped out perl, and installed the perl rpm from RedHat and all perl modules again from CPAN.
I’m using a fresh install of RT 3.0.6.
But I still see the same behaviour.
I’ve done some more testing, and found that it only fails when I do 2 successive comments
on a ticket after webserver restart.
If the second comment contains special characters, it will always fail.

This works:
0. Restart webserver

Create ticket#1 - Subject: ÆØÅ - Description: Test æøå
Comment on ticket#1 - Test æøå
Comment on ticket#1 - Test æøå
Comment on ticket#1 - Test æøå
Create ticket#2 - Subject: ÆØÅ - Description: Test æøå
Comment on ticket#1 - Test æøå
…
And it keeps working

This does not work:
0. Restart webserver

Comment on ticket#1 - Test æøå
Comment on ticket#1 - Test æøå
→ FAIL - records comment “Test Ã¦Ã¸Ã¥”
Create ticket#3 - Subject: ÆØÅ - Description: Test æøå
→ FAIL - records comment “Test Ã¦Ã¸Ã¥”
…
And every comment on a ticket will be parsed wrong after this.
Ticket subject is always correct though.

Have anyone else seen this on the FastCGI-version of RT?
Any comments, suggestions are welcome.

Regards,
Ronny Pettersen-----Original Message-----
From: rt-users-admin@lists.fsck.com
[mailto:rt-users-admin@lists.fsck.com]On Behalf Of Pettersen Ronny
(Networks)
Sent: 27. oktober 2003 12:01
To: rt-users@lists.fsck.com
Subject: [rt-users] Problems with character encoding on second post
(first OK) (Intern)

RT : 3.0.6 FastCGI-version
Apache: 2.0.40
Linux : RedHat 9

I have a kindof strange problem occuring when I’m posting a message containing
the norwegian special characters “æøå”.
After the webserver is restartet, the first post always give correct results.
The second post always encodes the characters wrong. “Test æøå” becomes “test Ã¦Ã¸Ã¥”.

Does anyone know what might cause this? See below for the RT debug-log.
Configuration issue, some perl module issue?
(“make testdeps” reports ok on all perl modules)

Regards,
Ronny Pettersen

from debug rt.log

First post with international characters gives correct encoding (message “test æøå” → “test æøå”):

[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.9.95786969530812@test.com #3/68 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:02 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.9.95786969530812@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 09:35:02 2003] [info]: rt-3.0.6-3-68.8.33597934411323@test.com #3/68 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:02 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:03 2003] [info]: rt-3.0.6-3-68.8.33597934411323@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Second posting with international characters gives wrong encoding (message “test æøå” → “test Ã¦Ã¸Ã¥”)

[Mon Oct 27 09:35:20 2003] [debug]: Guessed encoding: iso-8859-1 (/usr/local/rt3/lib/RT/I18N.pm:351)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘iso-8859-1’ to ‘utf-8’ for text/plain - (/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.5.6143501569494@test.com #3/69 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.5.6143501569494@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.14.9025718099913@test.com #3/69 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 09:35:21 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 09:35:21 2003] [info]: rt-3.0.6-3-69.14.9025718099913@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Post without international characters (message “Tesing without int chars”)

Post number 1 and 2 after restart gives the same behaviour

[Mon Oct 27 10:19:44 2003] [debug]: Guessed encoding: ascii (/usr/local/rt3/lib/RT/I18N.pm:351)
[Mon Oct 27 10:19:44 2003] [debug]: Converting ‘ascii’ to ‘utf-8’ for text/plain - (/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.19.8192251337203@test.com #3/75 - Scrip 7 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 10:19:45 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBbQ29tbWVudF0gVGVzdCDm+OU=?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.19.8192251337203@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.3.56622094384051@test.com #3/75 - Scrip 8 (/usr/local/rt3/lib/RT/Action/SendEmail.pm:92)
[Mon Oct 27 10:19:45 2003] [debug]: Converting ‘utf-8’ to ‘iso-8859-1’ for text/plain - =?ISO-8859-1?B?W1JULUFOICMzXSBUZXN0IOb45Q==?=
(/usr/local/rt3/lib/RT/I18N.pm:196)
[Mon Oct 27 10:19:45 2003] [info]: rt-3.0.6-3-75.3.56622094384051@test.com
No recipients found. Not sending.
(/usr/local/rt3/lib/RT/Action/SendEmail.pm:257)

Short Perl-module list, just in case it would be of any interest:

HTML::Mason - 1.23
HTML::Parser - 3.31
FCGI - 0.67
DBIx::SearchBuilder - 0.92
Storable - 2.08
Locale::Maketext - 1.06
Locale::Maketext::Lexicon - 0.32
mod_perl - 1.9907