Hello,
I just upgraded RT from 3.8.0 to 3.8.1 as suggested in the README.
Everything is OK except some charset problems with ticket titles as well as
templates. In fact, characters like “é”, “à” … appear scrambled. However
ticket history, menus and other text is not affected.
Is this a known issue ?
Thanks for your help.
Regards
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19267620.html
I’d like to add that new arriving tickets do not have this problem, only old
ones.
F350 wrote:
Hello,
I just upgraded RT from 3.8.0 to 3.8.1 as suggested in the README.
Everything is OK except some charset problems with ticket titles as well
as templates. In fact, characters like “é”, “à” … appear scrambled.
However ticket history, menus and other text is not affected.Is this a known issue ?
Thanks for your help.Regards
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19267639.html
Hello,
I just upgraded RT from 3.8.0 to 3.8.1 as suggested in the README.
Everything is OK except some charset problems with ticket titles as well as
templates. In fact, characters like “�”, “�” … appear scrambled. However
ticket history, menus and other text is not affected.Is this a known issue ?
You missed reading UPGRADING.mysql …
you didn’t ran “schema.mysql-4.0-4.1.pl”.
I’m running Mysql 5
When applying the command:
perl etc/upgrade/schema.mysql-4.0-4.1.pl rt3 user pass > sql.queries
i get the following message:
DBD::mysql version 4.002 required–this is only version 3.0008 at
etc/upgrade/schema.mysql-4.0-4.1.pl line 7.
BEGIN failed–compilation aborted at etc/upgrade/schema.mysql-4.0-4.1.pl
line 7.
Thanks
Emmanuel Lacour wrote:
Hello,
I just upgraded RT from 3.8.0 to 3.8.1 as suggested in the README.
Everything is OK except some charset problems with ticket titles as well
as
templates. In fact, characters like “é”, “à” … appear scrambled.
However
ticket history, menus and other text is not affected.Is this a known issue ?
You missed reading UPGRADING.mysql …
you didn’t ran “schema.mysql-4.0-4.1.pl”.
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19268595.html
I’m running Mysql 5
When applying the command:
perl etc/upgrade/schema.mysql-4.0-4.1.pl rt3 user pass > sql.queries
i get the following message:
DBD::mysql version 4.002 required–this is only version 3.0008 at
etc/upgrade/schema.mysql-4.0-4.1.pl line 7.
BEGIN failed–compilation aborted at etc/upgrade/schema.mysql-4.0-4.1.pl
line 7.
You have to upgrade your DBD::mysql as requested.
Uses CPAN or distro packages or backports to upgrade.
You can also run this script from a remote machine having the correct
DBD::mysql (you need to modify the script to provide the IP/name of the
mysql server).
Thanks for ur help.
I modified the script and excuted it on a remote machine with a newer
DBD::mysql
The output that i had is the following:
ALTER DATABASE rt3 DEFAULT CHARACTER SET utf8;
ALTER TABLE ACL DEFAULT CHARACTER SET utf8;
ALTER TABLE Attachments DEFAULT CHARACTER SET utf8;
ALTER TABLE Attributes DEFAULT CHARACTER SET utf8;
ALTER TABLE CustomFields DEFAULT CHARACTER SET utf8;
ALTER TABLE CustomFieldValues DEFAULT CHARACTER SET utf8;
ALTER TABLE GroupMembers DEFAULT CHARACTER SET utf8;
ALTER TABLE Groups DEFAULT CHARACTER SET utf8;
ALTER TABLE Links DEFAULT CHARACTER SET utf8;
ALTER TABLE ObjectCustomFields DEFAULT CHARACTER SET utf8;
ALTER TABLE ObjectCustomFieldValues DEFAULT CHARACTER SET utf8;
ALTER TABLE Principals DEFAULT CHARACTER SET utf8;
ALTER TABLE Queues DEFAULT CHARACTER SET utf8;
ALTER TABLE ScripActions DEFAULT CHARACTER SET utf8;
ALTER TABLE ScripConditions DEFAULT CHARACTER SET utf8;
ALTER TABLE Scrips DEFAULT CHARACTER SET utf8;
ALTER TABLE sessions DEFAULT CHARACTER SET utf8;
ALTER TABLE sessions MODIFY id BINARY(32) NOT NULL DEFAULT ‘’;
ALTER TABLE Templates DEFAULT CHARACTER SET utf8;
ALTER TABLE Tickets DEFAULT CHARACTER SET utf8;
ALTER TABLE Tickets MODIFY Status VARBINARY(10) NULL DEFAULT NULL;
ALTER TABLE Tickets MODIFY Status VARCHAR(10) CHARACTER SET ascii NULL
DEFAULT NULL;
ALTER TABLE Transactions DEFAULT CHARACTER SET utf8;
ALTER TABLE Users DEFAULT CHARACTER SET utf8;
ALTER TABLE Users MODIFY WebEncoding VARBINARY(16) NULL DEFAULT NULL;
ALTER TABLE Users MODIFY WebEncoding VARCHAR(16) CHARACTER SET ascii NULL
DEFAULT NULL;
ALTER TABLE Users MODIFY PGPKey BLOB NULL DEFAULT NULL;
ALTER TABLE Users MODIFY Timezone VARBINARY(50) NULL DEFAULT NULL;
ALTER TABLE Users MODIFY Timezone VARCHAR(50) CHARACTER SET ascii NULL
DEFAULT NULL;
ALTER TABLE Users MODIFY EmailAddress VARBINARY(120) NULL DEFAULT NULL;
ALTER TABLE Users MODIFY EmailAddress VARCHAR(120) CHARACTER SET ascii NULL
DEFAULT NULL;
ALTER TABLE Users MODIFY EmailEncoding VARBINARY(16) NULL DEFAULT NULL;
ALTER TABLE Users MODIFY EmailEncoding VARCHAR(16) CHARACTER SET ascii NULL
DEFAULT NULL;
ALTER TABLE Users MODIFY Lang VARBINARY(16) NULL DEFAULT NULL;
ALTER TABLE Users MODIFY Lang VARCHAR(16) CHARACTER SET ascii NULL DEFAULT
NULL;
I then injected these statements into rt3 database: mysql -u root -p rt3
<sql.queries without getting an output.
I checked back RT’s web interface but i still have the same problem: ticket
subjects, custom fields and templates are all scrambled 
Regards
Emmanuel Lacour wrote:
I’m running Mysql 5
When applying the command:
perl etc/upgrade/schema.mysql-4.0-4.1.pl rt3 user pass > sql.queries
i get the following message:
DBD::mysql version 4.002 required–this is only version 3.0008 at
etc/upgrade/schema.mysql-4.0-4.1.pl line 7.
BEGIN failed–compilation aborted at etc/upgrade/schema.mysql-4.0-4.1.pl
line 7.You have to upgrade your DBD::mysql as requested.
Uses CPAN or distro packages or backports to upgrade.
You can also run this script from a remote machine having the correct
DBD::mysql (you need to modify the script to provide the IP/name of the
mysql server).
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19269813.html
I checked back RT’s web interface but i still have the same problem: ticket
subjects, custom fields and templates are all scrambled
Did you cleared the Mason cache (rm -rf ~rt/var/mason_data/*) then
restart apache.
Yes i did but that did not fix the problem.
I’m confused.
Regards
Emmanuel Lacour wrote:
I checked back RT’s web interface but i still have the same problem:
ticket
subjects, custom fields and templates are all scrambledDid you cleared the Mason cache (rm -rf ~rt/var/mason_data/*) then
restart apache.
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19270223.html
Yes i did but that did not fix the problem.
I’m confused.
Me too, maybe I gave wrong ideas, did your RT comes from an upgrade of
3.6.x before 3.8.0 ?
No I installed from scratch 3.8.0.
Somewhere in the UPGRADING.mysql file, they say:
don’t use utf8 as default mysql’s character set, it’s default in some
distributives.
I’m using debian etch but i don’t know if my mysql is using utf-8 by default
or not.
Could that be the problem ?
Regards
Emmanuel Lacour wrote:
Yes i did but that did not fix the problem.
I’m confused.Me too, maybe I gave wrong ideas, did your RT comes from an upgrade of
3.6.x before 3.8.0 ?
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19271060.html
No I installed from scratch 3.8.0.
Somewhere in the UPGRADING.mysql file, they say:
don’t use utf8 as default mysql’s character set, it’s default in some
distributives.I’m using debian etch but i don’t know if my mysql is using utf-8 by default
or not.
Could that be the problem ?
Ok, here is how I understand your problem:
so it was needed to run the mysql4.0-4.1 script, for sure.
what I don’t understand is why it didn’t fix the problem:(
Are you sure you worked on the right DB, right RT instance, etc… ?
Exactly.
I run the show variables like ‘character_set%’; command, here is the output:
| Variable_name | Value |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
I did execute the script again, same problem.
New tickets however are OK and scrambled custom field attributes and
templates can be modified and updated. Is it secure to update everything and
forget about this problem ?
Emmanuel Lacour wrote:
No I installed from scratch 3.8.0.
Somewhere in the UPGRADING.mysql file, they say:
don’t use utf8 as default mysql’s character set, it’s default in some
distributives.I’m using debian etch but i don’t know if my mysql is using utf-8 by
default
or not.
Could that be the problem ?Ok, here is how I understand your problem:
- you installed 3.8.0
- you inserted some datas, mysql on debian etch uses latin1 as default,
so you have a DB with UTF-8 encoded as latin1
(to confirm this, connect with the mysql client to the rt db and
issue: show variables like ‘character_set%’- in 3.8.1, RT forces connexion to mysql to be UTF-8, so you don’t
convert back latin1 anymore and you get garbled charsso it was needed to run the mysql4.0-4.1 script, for sure.
what I don’t understand is why it didn’t fix the problem:(
Are you sure you worked on the right DB, right RT instance, etc… ?
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19271484.html
Exactly.
I run the show variables like ‘character_set%’; command, here is the output:
±-------------------------±---------------------------+
| Variable_name | Value |
±-------------------------±---------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
±-------------------------±---------------------------+I did execute the script again, same problem.
Maybe not a good idea I hope it didn’t did wrong things.
New tickets however are OK and scrambled custom field attributes and
Theoritically you should be in a situation where:
templates can be modified and updated. Is it secure to update everything and
forget about this problem ?
The only risk I see (but I may be missing something) is that corrupted
tickets will continue to display badly, new tickets will be ok, every RT
configuration thinks that may contains non-ascii chars (queue names,
templates, scrips names, users names, group names, …) should be
checked and updated manually if needed to look correctly on the UI.
Thanks a lot for your help and time Emmanuel.
I updated all the tickets, CF and templates that look corrupted.
Next time i’ll make sure I do the updates on a test server 
Thanks again
Emmanuel Lacour wrote:
Exactly.
I run the show variables like ‘character_set%’; command, here is the
output:
±-------------------------±---------------------------+
| Variable_name | Value |
±-------------------------±---------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
±-------------------------±---------------------------+I did execute the script again, same problem.
Maybe not a good idea
New tickets however are OK and scrambled custom field attributes and
Theoritically you should be in a situation where:
- datas inserted before 3.8.1 were converted from latin1 to utf-8 and
should appears right- datas inserted after 3.8.1, before 4.0-4.1 script may be inserted as
utf-8, but tables may be defined as latin1 so may be garbled- datas inserted after 4.0-4.1 scrip should be ok
templates can be modified and updated. Is it secure to update everything
and
forget about this problem ?The only risk I see (but I may be missing something) is that corrupted
tickets will continue to display badly, new tickets will be ok, every RT
configuration thinks that may contains non-ascii chars (queue names,
templates, scrips names, users names, group names, …) should be
checked and updated manually if needed to look correctly on the UI.
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19272127.html
Thanks a lot for your help and time Emmanuel.
I updated all the tickets, CF and templates that look corrupted.
Next time i’ll make sure I do the updates on a test server
You’re welcome
Using a test server or a test db/rt instance is a must have
Hi,
I reported the same problem here (but nobody answered):
http://lists.bestpractical.com/pipermail/rt-users/2008-August/053512.html
I think there is a lack in the mysql upgrade document or in the upgrade
script.
I did it on a test environment and didn’t have time to find out the real
solution,
so I delayed the migration of the real system.
Some more information:
Any suggestion?
Thanks,
BekényOn Tue, Sep 2, 2008 at 5:28 PM, Emmanuel Lacour elacour@easter-eggs.comwrote:
On Tue, Sep 02, 2008 at 08:14:55AM -0700, F350 wrote:
Thanks a lot for your help and time Emmanuel.
I updated all the tickets, CF and templates that look corrupted.
Next time i’ll make sure I do the updates on a test serverYou’re welcome
Using a test server or a test db/rt instance is a must have
Hi,
you must add/change this lines in your my.cnf:
[client]
default-character-set = utf8
[mysqld]
default-character-set=utf8
character_set_server=utf8
collation_server=utf8_general_ci
init_connect=‘SET collation_connection = utf8_general_ci’
init_connect=‘SET CHARACTER_SET utf8’
init_connect=‘SET NAMES utf8’
[mysql]
default-character-set=utf8
Stop/start you mysqld and you have:
mysql> show variables like ‘c%’;
| Variable_name | Value |
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 10 |
14 rows in set (0.00 sec)
After these modifications all works well.
Gian Luca GobbiFrom: rt-users-bounces@lists.bestpractical.com [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of BALINT Bekeny
Sent: Wednesday, September 03, 2008 11:07 AM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Charset issues after upgrade to 3.8.1
Hi,
I reported the same problem here (but nobody answered):
http://lists.bestpractical.com/pipermail/rt-users/2008-August/053512.html
I think there is a lack in the mysql upgrade document or in the upgrade script.
I did it on a test environment and didn’t have time to find out the real solution,
so I delayed the migration of the real system.
Some more information:
Any suggestion?
Thanks,
Bekény
Are u sure it is the way to do ? In the Upgrading.mysql help file, they say
that the default charset should not be utf-8. In my database, all my tables
are set as utf-8 but the default charset of the database is latin1.
Gian Luca Gobbi wrote:
Hi,
you must add/change this lines in your my.cnf:[client]
default-character-set = utf8[mysqld]
default-character-set=utf8
character_set_server=utf8
collation_server=utf8_general_ci
init_connect=‘SET collation_connection = utf8_general_ci’
init_connect=‘SET CHARACTER_SET utf8’
init_connect=‘SET NAMES utf8’[mysql]
default-character-set=utf8Stop/start you mysqld and you have:
mysql> show variables like ‘c%’;
±-------------------------±---------------------------+
| Variable_name | Value |
±-------------------------±---------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 10 |
±-------------------------±---------------------------+
14 rows in set (0.00 sec)After these modifications all works well.
–
Gian Luca Gobbi
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of BALINT
Bekeny
Sent: Wednesday, September 03, 2008 11:07 AM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Charset issues after upgrade to 3.8.1Hi,
I reported the same problem here (but nobody answered):
[rt-users] Upgrade from 3.4.2I think there is a lack in the mysql upgrade document or in the upgrade
script.
I did it on a test environment and didn’t have time to find out the real
solution,
so I delayed the migration of the real system.Some more information:
- After migration, new entries (for example CF-s) seemed good, even if
they have non-ascii characters.- In mysql CLI: show variables like ‘c%’ shows that everything is utf8
(so this seems good)Any suggestion?
Thanks,
Bekény
Thanks a lot for your help and time Emmanuel.
I updated all the tickets, CF and templates that look corrupted.
Next time i’ll make sure I do the updates on a test serverYou’re welcome
Using a test server or a test db/rt instance is a must have
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19308775.html
Here is a sample of my custom fields table schema:
CREATE TABLE CustomFields (
id int(11) NOT NULL auto_increment,
Name varchar(200) default NULL,
Type varchar(200) character set ascii default NULL,
MaxValues int(11) default NULL,
Pattern text,
Repeated smallint(6) NOT NULL default ‘0’,
Description varchar(255) default NULL,
SortOrder int(11) NOT NULL default ‘0’,
LookupType varchar(255) character set ascii NOT NULL,
Creator int(11) NOT NULL default ‘0’,
Created datetime default NULL,
LastUpdatedBy int(11) NOT NULL default ‘0’,
LastUpdated datetime default NULL,
Disabled smallint(6) NOT NULL default ‘0’,
PRIMARY KEY (id)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
Can someone tell me if the schema is correct ? When comparing the old
database version (3.8.0) and the new one (3.8.1) I found no differences in
the schema even after applying the script.
Thanks
F350 wrote:
Are u sure it is the way to do ? In the Upgrading.mysql help file, they
say that the default charset should not be utf-8. In my database, all my
tables are set as utf-8 but the default charset of the database is latin1.Gian Luca Gobbi wrote:
Hi,
you must add/change this lines in your my.cnf:[client]
default-character-set = utf8[mysqld]
default-character-set=utf8
character_set_server=utf8
collation_server=utf8_general_ci
init_connect=‘SET collation_connection = utf8_general_ci’
init_connect=‘SET CHARACTER_SET utf8’
init_connect=‘SET NAMES utf8’[mysql]
default-character-set=utf8Stop/start you mysqld and you have:
mysql> show variables like ‘c%’;
±-------------------------±---------------------------+
| Variable_name | Value |
±-------------------------±---------------------------+
| character_set_client | utf8 |
| character_set_connection | utf8 |
| character_set_database | utf8 |
| character_set_filesystem | binary |
| character_set_results | utf8 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 10 |
±-------------------------±---------------------------+
14 rows in set (0.00 sec)After these modifications all works well.
–
Gian Luca Gobbi
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of BALINT
Bekeny
Sent: Wednesday, September 03, 2008 11:07 AM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Charset issues after upgrade to 3.8.1Hi,
I reported the same problem here (but nobody answered):
[rt-users] Upgrade from 3.4.2I think there is a lack in the mysql upgrade document or in the upgrade
script.
I did it on a test environment and didn’t have time to find out the real
solution,
so I delayed the migration of the real system.Some more information:
- After migration, new entries (for example CF-s) seemed good, even if
they have non-ascii characters.- In mysql CLI: show variables like ‘c%’ shows that everything is
utf8 (so this seems good)Any suggestion?
Thanks,
Bekény
Thanks a lot for your help and time Emmanuel.
I updated all the tickets, CF and templates that look corrupted.
Next time i’ll make sure I do the updates on a test serverYou’re welcome
Using a test server or a test db/rt instance is a must have
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.comDiscover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
View this message in context: http://www.nabble.com/Charset-issues-after-upgrade-to-3.8.1-tp19267620p19308784.html