Big problem with encoding subject

Hi,

I just upgrade to rt 4.0.17 under freebsd.

After the upgrade we got a big problem with encoding subject.

If the subject is detected as utf-8 RT re-encode so a subject like

Essai é

is encode likeSubject: =?UTF-8?B?W21hdGguY25ycy5mciAjMzM3OV0gRXNzYWkgw6k=?=

but since my upgrade the encoding is wrong.

Subject: =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzMxXSBFc3NhaSDvv70=?=

And the problem become very big is with some subjet (always UTF-8) the
scrip don’t work anymore.

If I send a message with

Essai é à ẽ ã

as a subject, RT don’t send any mail to admin.

Any idea ?

Regards.

JAS

NB: I put here all version of all perl package I got on the server

p5-Any-Moose-0.21 Perl extension for/to use Moose or Mouse modules
p5-Apache-DBI-1.12 DBI persistent connection, authentication and authorization
p5-Apache-LogFormat-Compiler-0.13 Perl extension to compile a log format string to perl-code
p5-Apache-Session-1.90 Persistence framework for session data
p5-Authen-NTLM-1.09 An NTLM authentication module
p5-Authen-SASL-2.16 Perl5 module for SASL authentication
p5-BSD-Resource-1.2907 Perl module to access BSD resource limit and priority funct
p5-CGI-Emulate-PSGI-0.15 Perl extension of PSGI adapter for CGI
p5-CGI-PSGI-0.15 Perl extension to enable CGI.pm applications to adapt PSGI
p5-CPAN-Meta-2.132510 The distribution metadata for a CPAN dist
p5-CPAN-Meta-Requirements-2.123 Set of version requirements for a CPAN distribution
p5-CPAN-Meta-YAML-0.008 Read and write a subset of YAML for CPAN Meta files
p5-CSS-Squish-0.10 Compact many CSS files into one big file
p5-Cache-2.04_1 A Perl Cache interface
p5-Cache-Cache-1.06 Cache::Cache – Perl Cache interface
p5-Cache-Simple-TimedExpiry-0.27 A lightweight cache with timed expiration
p5-Carp-1.32 Alternative warn and die for modules
p5-Class-Accessor-0.34 Automated accessor generation
p5-Class-Accessor-Chained-0.01_1 Make chained accessors
p5-Class-Accessor-Lite-0.05 Minimalistic variant of p5-Class-Accessor
p5-Class-Container-0.12 Glues object frameworks together transparently
p5-Class-Data-Inheritable-0.08 Inheritable, overridable class data
p5-Class-Inspector-1.28 Provides information about classes
p5-Class-Load-0.20 A working (require Class::Name) and more
p5-Class-MethodMaker-2.18 Perl module for creating generic methods
p5-Class-ReturnValue-0.55 A return-value object that can be treated as a boolean, arr
p5-Class-Singleton-1.4 Perl module that describes a singular object class
p5-Clone-0.35 Clone - recursively copy Perl datatypes
p5-Compress-Raw-Bzip2-2.062 Low-Level Interface to bzip2 compression library
p5-Compress-Raw-Zlib-2.062 Low-Level Interface to zlib compression library
p5-Convert-ASN1-0.26 Perl5 module to encode and decode ASN.1 data structures
p5-Convert-BinHex-1.123 Perl module to extract data from Macintosh BinHex files
p5-Convert-Color-0.09 Perl extension for color space conversions and named lookup
p5-Crypt-CBC-2.33 Perl5 interface to Cipher Block Chaining with DES and IDEA
p5-Crypt-DES-2.07 Perl5 interface to DES block cipher
p5-Crypt-SSLeay-0.64 Perl5 interface to allow p5-libwww LWP to make https connec
p5-DBD-mysql-4.023 MySQL driver for the Perl5 Database Interface (DBI)
p5-DBI-1.628 The perl5 Database Interface. Required for DBD::* modules
p5-DBIx-DBSchema-0.40 OO-interface to database schemas
p5-DBIx-SearchBuilder-1.65 Perl extension for easy SQL SELECT Statement generation
p5-Data-ICal-0.21 Data::ICal - Generates iCalendar (RFC 2445) calendar files
p5-Data-OptList-0.108 Parse and validate simple name/value option pairs
p5-Date-ICal-2.678 Perl extension for ICalendar date objects
p5-Date-Leapyear-1.72 Perl module to determine if a particular year a leap year
p5-DateTime-1.03 Date and time object
p5-DateTime-Format-Mail-0.30.01 Convert between DateTime and RFC2822/822 formats
p5-DateTime-Format-W3CDTF-0.06 Parse and format W3CDTF datetime strings
p5-DateTime-HiRes-0.01 Create DateTime objects with sub-second current time resolu
p5-DateTime-Locale-0.45 Localization support for DateTime
p5-DateTime-Set-0.32 Datetime sets and set math
p5-DateTime-TimeZone-1.60 Time zone object base class and factory
p5-Devel-GlobalDestruction-0.11_2 Expose PL_dirty, the flag which marks global destruction
p5-Devel-StackTrace-1.30 Stack trace and stack trace frame objects
p5-Devel-StackTrace-AsHTML-0.14 Perl extension to display stack trace in HTML
p5-Devel-Symdump-2.1000 Perl5 module that dumps symbol names or the symbol table
p5-Digest-HMAC-1.03 Perl5 interface to HMAC Message-Digest Algorithms
p5-Digest-SHA1-2.13 Perl interface to the SHA-1 Algorithm
p5-Dist-CheckConflicts-0.02 Declare version conflicts for your dist
p5-Email-Address-1.90.0 RFC 2822 Address Parsing and Creation
p5-Email-Date-Format-1.002 Produce RFC 2822 date strings
p5-Encode-2.55 Provides interfaces between strings and the rest of the sys
p5-Encode-Locale-1.03 Determine the locale encoding
p5-Error-0.17021 Perl module to provide Error/exception support for perl: Er
p5-Exception-Class-1.37 Real exception classes in Perl
p5-ExtUtils-CBuilder-0.2802.12,1 Compile and link C code for Perl modules
p5-ExtUtils-Config-0.007 Wrapper for perl configuration
p5-ExtUtils-Helpers-0.021 Various portability utilities for module builders
p5-ExtUtils-InstallPaths-0.009 Build.PL install path logic made easy
p5-ExtUtils-MakeMaker-6.76 Designed to write a Makefile for an extension module
p5-ExtUtils-XSBuilder-0.28_1 Autogenerating XS-glue Code
p5-File-Listing-6.04 Parse directory listings
p5-File-NFSLock-1.21 File::NFSLock - perl module to do NFS (or not) locking
p5-File-ShareDir-1.03 Locate per-dist and per-module shared files
p5-File-ShareDir-Install-0.05 Install read-only data files from a distribution
p5-File-Slurp-9999.19 Perl5 module for single call read & write file routines
p5-File-Temp-0.23.01 Perl5 module to generate temporary files or directories saf
p5-File-Which-1.09 Portable implementation of `which’ in Perl
p5-Filesys-Notify-Simple-0.12 Perl extension for simple and dumb file system watcher
p5-Font-AFM-1.20 Interface to Adobe font metrics files
p5-FreezeThaw-0.50.01 Module for converting Perl structures to strings and back
p5-GD-2.50 Perl5 interface to Gd Graphics Library version2
p5-GD-Graph-1.48 Graph plotting module for perl5
p5-GD-TextUtil-0.86_4 Text utilities for use with GD drawing package
p5-GSSAPI-0.28 Perl extension providing access to the GSSAPIv2 library
p5-GnuPG-Interface-0.46 Perl module for interfacing with GnuPG
p5-GraphViz-2.14 GraphViz - Perl interface to the GraphViz graphing tool
p5-HTML-Format-2.10_1 Module to format HTML to text or PS
p5-HTML-Mason-1.50 High-performance, dynamic web site authoring system
p5-HTML-Mason-PSGIHandler-0.53 PSGI handler for HTML::Mason
p5-HTML-Parser-3.71 Perl5 module for parsing HTML documents
p5-HTML-Quoted-0.04 Extract structure of quoted HTML mail message
p5-HTML-RewriteAttributes-0.05 Simple yet powerful HTML attribute rewriting
p5-HTML-Scrubber-0.09 Perl extension for scrubbing/sanitizing html
p5-HTML-Tagset-3.20 Some useful data table in parsing HTML
p5-HTML-Tree-5.03 Collection of modules to manipulate HTML syntax trees
p5-HTTP-Body-1.17 HTTP Body Parser
p5-HTTP-CookieJar-0.005 Minimalist HTTP user agent cookie jar
p5-HTTP-Cookies-6.01 HTTP Cookie jars
p5-HTTP-Daemon-6.01 Simple HTTP server class
p5-HTTP-Date-6.02 Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.06_2 Representation of HTTP style messages
p5-HTTP-Negotiate-6.01 Implementation of the HTTP content negotiation algorithm
p5-HTTP-Tiny-0.035 Small, simple, correct HTTP/1.1 client
p5-Hash-MultiValue-0.15 Perl extension to store multiple values per key
p5-Heap-0.80 Perl extensions for keeping data partially sorted
p5-IO-Compress-2.062 Perl5 compression modules (bzip2, deflate, gzip, zlib, zip)
p5-IO-HTML-1.00 Open an HTML file with automatic charset detection
p5-IO-Multiplex-1.13 IO::Multiplex - Manage IO on many file handles
p5-IO-Socket-IP-0.23 Drop-in replacement for IO::Socket::INET supporting IPv4 an
p5-IO-Socket-SSL-1.953 Perl5 interface to SSL sockets
p5-IO-String-1.08 Simplified Perl5 module to handle I/O on in-core strings
p5-IO-Tty-1.10 Flexible I/O Perl5 module that allows manipulation of pseud
p5-IO-stringy-2.110 Perl5 module for using IO handles with non-file objects
p5-IPC-Run-0.92 IPC::Run - Run subprocesses with piping and redirection
p5-IPC-Run3-0.046 Run a subprocess in batch mode
p5-IPC-ShareLite-0.17 Simple interface to access shared memory
p5-JSON-2.59 Perl extension to convert to JSON (JavaScript Object Notati
p5-JSON-PP-2.27202 A JSON::XS compatible pure-Perl module
p5-LWP-MediaTypes-6.02 Guess media type for a file or a URL
p5-LWP-Protocol-https-6.04 Provide https support for LWP::UserAgent
p5-List-MoreUtils-0.33 Provide the stuff missing in List::Util
p5-List-UtilsBy-0.09 Perl extension for higher-order list utility functions
p5-Locale-Maketext-1.23 Framework for software localization and inheritance-based l
p5-Locale-Maketext-Fuzzy-0.11 Locale::Maketext::Fuzzy - Maketext from already interpolate
p5-Locale-Maketext-Lexicon-0.96 Use other catalog formats in Locale::Maketext
p5-Locale-gettext-1.05_3 Message handling functions
p5-Log-Any-0.15 Perl extension to bring loggers and listeners together
p5-Log-Dispatch-2.41 Suite of OO modules for logging messages to multiple output
p5-MIME-Lite-3.02.9 Simple interface for creating (not parsing!) MIME messages
p5-MIME-Tools-5.504,2 Set of perl5 modules for MIME
p5-MIME-Types-2.04 Perl extension for determining MIME types
p5-Mail-Sender-0.8.22 Module for sending mails with attachments through an SMTP s
p5-Mail-Sendmail-0.79 Perl module implementing a simple, platform-independent mai
p5-Mail-Tools-2.12 Perl5 modules for dealing with Internet e-mail messages
p5-Math-BigInt-1.997 Math::BigInt - Arbitrary size integer math package
p5-Math-Round-0.06 Perl extension for rounding numbers
p5-Module-Build-0.4007 Build and install Perl modules
p5-Module-Build-Tiny-0.027 Tiny replacement for Module::Build
p5-Module-Implementation-0.07 Loads one of several alternate underlying implementations f
p5-Module-Runtime-0.013 Perl module for runtime module handling
p5-Module-Versions-Report-1.06 Report versions of all modules in memory
p5-Mouse-1.11,1 Moose minus the antlers
p5-Mozilla-CA-20130114 Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-CIDR-0.17 Perl module to manipulate IPv4/IPv6 netblocks in CIDR notat
p5-Net-Daemon-0.48 Perl5 extension for portable daemons
p5-Net-HTTP-6.06 Low-level HTTP client
p5-Net-SNMP-6.0.1 Object oriented interface to SNMP
p5-Net-SSLeay-1.55 Perl5 interface to SSL
p5-Net-Server-2.007 Configurable base class for writing internet servers in Per
p5-Package-DeprecationManager-0.13 Manage deprecation warnings for your distribution
p5-Package-Stash-0.36 Routines for manipulating stashes
p5-Package-Stash-XS-0.26 Faster and more correct implementation of the Package::Stas
p5-Parallel-Prefork-0.14 A simple prefork server framework
p5-Params-Classify-0.013 Argument type classification
p5-Params-Util-1.07 Utility functions to aid in parameter checking
p5-Params-Validate-1.08 Validate method/function parameters
p5-Parse-CPAN-Meta-1.44.04 Parse META.yml and other similar CPAN metadata files
p5-Parse-RecDescent-1.967.009_1 A recursive descent parsing framework for Perl
p5-Path-Class-0.32 Cross-platform path specification manipulation
p5-PathTools-3.4000 Perl module for portably manipulating file specifications
p5-PerlIO-eol-0.14 PerlIO layer for normalizing line endings
p5-PlRPC-0.2020_1 Perl module for writing RPC servers and clients
p5-Plack-1.0028 Perl extension of PSGI reference implementation and utiliti
p5-Pod-Simple-3.28 Framework for parsing Pod
p5-Proc-Wait3-0.04 Perl extension for wait3 system call
p5-RT-Authen-ExternalAuth-0.17 RT extension to authenticate users against an external sour
p5-RTx-Calendar-0.17 Calendar extension module for the RT ticketing system
p5-Regexp-Common-2013031301 Regexp::Common - Provide commonly requested regular express
p5-Regexp-Common-net-CIDR-0.02 Provide patterns for CIDR blocks
p5-Regexp-IPv6-0.03 Regular expression for IPv6 addresses
p5-Scalar-List-Utils-1.32,1 Perl subroutines that would be nice to have in the perl cor
p5-Scope-Guard-0.20 Lexically scoped resource management
p5-Server-Starter-0.15 Perl extension of superdaemon for hot-deploying server prog
p5-Set-Infinite-0.65 Set::Infinite - Sets of intervals
p5-Socket-2.012 Networking constants and support functions
p5-Starlet-0.19 Perl extension for standalone HTTP/1.0 server
p5-Stream-Buffered-0.02 Perl extension for temporary buffer to save bytes
p5-Sub-Exporter-0.986 Sophisticated exporter for custom-built routines
p5-Sub-Exporter-Progressive-0.001.010 Only use Sub::Exporter if you need it
p5-Sub-Install-0.926 Install subroutines into packages easily
p5-Sub-Uplevel-0.2400 Sub::Uplevel - apparently run a function in a higher stack
p5-Term-ReadKey-2.30 A perl5 module for simple terminal control
p5-Test-Deep-0.110 Test data structures for differences deeply
p5-Test-Exception-0.32 Test::Exception - test functions for exception based code
p5-Test-LongString-0.15 Library to test long strings
p5-Test-Manifest-1.23 Perl module to interact with a t/test_manifest file
p5-Test-NoWarnings-1.04_1 Hide and store warnings while running test scripts
p5-Test-Simple-0.98 Basic utilities for writing tests in perl
p5-Test-Tester-0.10.9 Helps to test various Test::Builder derivatives
p5-Test-Warn-0.24 Perl extension to test methods for warnings
p5-Text-Autoformat-1.669.004 Automatic and manual text wrapping and reformating formatti
p5-Text-Iconv-1.7 Perl interface to iconv() codeset conversion function
p5-Text-Password-Pronounceable-0.30 Perl extension for generating pronounceable passwords
p5-Text-Quoted-2.08 Text::Quoted - Extract the structure of a quoted mail messa
p5-Text-Reform-1.20 Manual text wrapping and reformatting
p5-Text-Template-1.46 Expand template text with embedded Perl
p5-Text-WikiFormat-0.80 Module for translating Wiki formatted text into other forma
p5-Text-Wrapper-1.04 Simple word wrapping routine
p5-Text-vFile-asData-0.08 Parse vFile formatted files into data structures
p5-Tie-IxHash-1.23 Perl module implementing ordered in-memory associative arra
p5-Time-HiRes-1.9726,1 A perl5 module implementing High resolution time, sleep, an
p5-Time-modules-2013.0912 Set of modules for time manipulation
p5-TimeDate-2.30_1,1 Perl5 module containing a better/faster date parser for abs
p5-Tree-DAG_Node-1.17 Super class for representing nodes in a tree
p5-Tree-Simple-1.18 A simple tree object
p5-Try-Tiny-0.18 Minimal try/catch with proper localization of $@
p5-UNIVERSAL-require-0.13 Perl module to require() from a variable
p5-URI-1.60 Perl5 interface to Uniform Resource Identifier (URI) refere
p5-WWW-RobotRules-6.02 Database of robots.txt-derived permissions
p5-Want-0.21 Generalisation of wantarray
p5-XML-Filter-BufferText-1.01 Filter to put all characters() in one event
p5-XML-Handler-YAWriter-0.23 Yet another Perl SAX XML Writer
p5-XML-NamespaceSupport-1.11 Simple generic namespace support class
p5-XML-Parser-2.41_1 Perl extension interface to James Clark’s XML parser, expat
p5-XML-RSS-1.49 Perl extension to manage RDF Site Summary (RSS) files
p5-XML-SAX-0.99 Simple API for XML
p5-XML-SAX-Base-1.08 Base class SAX Drivers and Filters
p5-XML-SAX-Expat-0.50 Simple API for XML
p5-XML-SAX-Writer-0.53 SAX2 XML Writer
p5-XML-Simple-2.20 Trivial API for reading and writing XML (esp config files)
p5-XML-Twig-3.44 Process huge XML documents by chunks via a tree interface
p5-XML-XPath-1.13 Modules for parsing and evaluating XPath statements
p5-capitalization-0.03 Perl module which allows you to use familiar style on metho
p5-libapreq2-2.13_2 Generic Apache2 Request Library
p5-libwww-6.05 Perl5 library for WWW access
p5-libxml-0.08 Collection of Perl5 modules for working with XML
p5-parent-0.225 Establish an ISA relationship with base classes at compile
p5-perl-ldap-0.5700 Client interface to LDAP (includes Net::LDAP)
p5-version-0.99.04 Perl extension for Version Objects
Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
jeu 19 sep 2013 11:52:11 CEST

Le 19/09/2013 à 12:05:02+0200, Albert Shih a écrit

Hi,

I just upgrade to rt 4.0.17 under freebsd.

After the upgrade we got a big problem with encoding subject.

If the subject is detected as utf-8 RT re-encode so a subject like

Essai é

is encode like

Subject: =?UTF-8?B?W21hdGguY25ycy5mciAjMzM3OV0gRXNzYWkgw6k=?=

but since my upgrade the encoding is wrong.

Subject: =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzMxXSBFc3NhaSDvv70=?=

And the problem become very big is with some subjet (always UTF-8) the
scrip don’t work anymore.

If I send a message with

Essai é à ẽ ã

as a subject, RT don’t send any mail to admin.

Any idea ?

I check the log, and found this message

RT: Malformed UTF-8 character (unexpected end of string) in string ne at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538. (/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)
RT: Malformed UTF-8 character (unexpected end of string) in string ne at /usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538. (/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)

not sure it’s related but I guess it’is.

So I installed old version of

p5-Locale-Maketext-Lexicon

but nothing change.

I also try with

p5-Encode

but no chance either.

Can someone tell me which perl package is used by RT to « encode » subject
? So I can check if the problem come from RT 4.0.17 or some new perl packages.

Regards.

JAS
Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
jeu 19 sep 2013 16:42:20 CEST

Hi,

You didn’t say how to reproduce the problem. Should it be email? Should it
be UI?On Thu, Sep 19, 2013 at 6:50 PM, Albert Shih Albert.Shih@obspm.fr wrote:

Le 19/09/2013 à 12:05:02+0200, Albert Shih a écrit

Hi,

I just upgrade to rt 4.0.17 under freebsd.

After the upgrade we got a big problem with encoding subject.

If the subject is detected as utf-8 RT re-encode so a subject like

Essai é

is encode like

Subject: =?UTF-8?B?W21hdGguY25ycy5mciAjMzM3OV0gRXNzYWkgw6k=?=

but since my upgrade the encoding is wrong.

Subject: =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzMxXSBFc3NhaSDvv70=?=

And the problem become very big is with some subjet (always UTF-8) the
scrip don’t work anymore.

If I send a message with

Essai é à ẽ ã

as a subject, RT don’t send any mail to admin.

Any idea ?

I check the log, and found this message

RT: Malformed UTF-8 character (unexpected end of string) in string ne at
/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538.
(/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)
RT: Malformed UTF-8 character (unexpected end of string) in string ne at
/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538.
(/usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)

not sure it’s related but I guess it’is.

So I installed old version of

p5-Locale-Maketext-Lexicon

but nothing change.

I also try with

p5-Encode

but no chance either.

Can someone tell me which perl package is used by RT to « encode » subject
? So I can check if the problem come from RT 4.0.17 or some new perl
packages.

Regards.

JAS

Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
jeu 19 sep 2013 16:42:20 CEST


RT Training in New York, October 8th and 9th:
http://bestpractical.com/training

Best regards, Ruslan.

Le 20/09/2013 à 08:58:35+0400, Ruslan Zakirov a écrit
Hi,

You didn’t say how to reproduce the problem. Should it be email? Should it be
UI?

Only from email.

When I send a email with some utf-8 character inside the subject the
problem appear.

I try to track the problem, so I installe a new server (FreeBSD), dump the
all mysql base and rt config, put inside the new server and the problem
still here.

I disable all plugin and enable debug mode in RT :

Here the log I find (It’s really at the beginning : )

Sep 20 20:53:37 support-new RT: Converting ‘iso-8859-1’ to ‘utf-8’ for text/plain - Essai é, è, ã
Sep 20 20:53:37 support-new RT: Mail from user #22 (Albert.Shih@obspm.fr)
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203642
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203643
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203644
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203645
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203646
Sep 20 20:53:37 support-new RT: About to prepare scrips for transaction #203646
Sep 20 20:53:37 support-new RT: Found 3 scrips for TransactionCreate stage with applicable type(s) Create for txn #203646 on ticket #10767
Sep 20 20:53:37 support-new RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9?=
Sep 20 20:53:37 support-new RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9?=

at this point something is wrong because

echo “W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9” |base64 -d

[Info Obspm #10767] Essai �, �, �

and missing a “\n” at the end

It should be something like :

echo “[Info Obspm #10767] Essai é, è, ã” |base64

W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDDqSwgw6gsIMOjCg==

More stranger, if now I close the ticket from the web UI the mail RT send
to me is correct.

Thank you very much for you help.

> Hi,
>
> I just upgrade to rt 4.0.17 under freebsd.
>
> After the upgrade we got a big problem with encoding subject.
>
> If the subject is detected as utf-8 RT re-encode so a subject like
>
>     Essai é
>
> is encode like
>
> Subject: =?UTF-8?B?W21hdGguY25ycy5mciAjMzM3OV0gRXNzYWkgw6k=?=
>
> but since my upgrade the encoding is wrong.
>
> Subject: =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzMxXSBFc3NhaSDvv70=?=
>
> And the problem become very big is with some subjet (always UTF-8) the
> scrip don't work anymore.
>
> If I send a message with
>
>     Essai é à ẽ ã
>
> as a subject, RT don't send any mail to admin.
>
> Any idea ?

I check the log, and found this message

RT: Malformed UTF-8 character (unexpected end of string) in string ne at /
usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538. (/usr/local
/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)
RT: Malformed UTF-8 character (unexpected end of string) in string ne at /
usr/local/lib/perl5/site_perl/5.16/Locale/Maketext.pm line 538. (/usr/local
/lib/perl5/site_perl/5.16/Locale/Maketext.pm:538)


not sure it's related but I guess it'is.

So I installed old version of

    p5-Locale-Maketext-Lexicon

but nothing change.

I also try with

    p5-Encode

but no chance either.

Can someone tell me which perl package is used by RT to « encode » subject
? So I can check if the problem come from RT 4.0.17 or some new perl
packages.

Bests regards.

JAS
Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
ven 20 sep 2013 22:49:40 CEST

Le 20/09/2013 à 23:01:56+0200, Albert Shih a écrit

Le 20/09/2013 à 08:58:35+0400, Ruslan Zakirov a écrit
Hi,

You didn’t say how to reproduce the problem. Should it be email? Should it be
UI?

Only from email.

When I send a email with some utf-8 character inside the subject the
problem appear.

I try to track the problem, so I installe a new server (FreeBSD), dump the
all mysql base and rt config, put inside the new server and the problem
still here.

I disable all plugin and enable debug mode in RT :

Here the log I find (It’s really at the beginning : )

Sep 20 20:53:37 support-new RT: Converting ‘iso-8859-1’ to ‘utf-8’ for text/plain - Essai é, è, ã
Sep 20 20:53:37 support-new RT: Mail from user #22 (Albert.Shih@obspm.fr)
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203642
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203643
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203644
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203645
Sep 20 20:53:37 support-new RT: About to think about scrips for transaction #203646
Sep 20 20:53:37 support-new RT: About to prepare scrips for transaction #203646
Sep 20 20:53:37 support-new RT: Found 3 scrips for TransactionCreate stage with applicable type(s) Create for txn #203646 on ticket #10767
Sep 20 20:53:37 support-new RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9?=
Sep 20 20:53:37 support-new RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9?=

at this point something is wrong because

echo “W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDvv70sIO+/vSwg77+9” |base64 -d

[Info Obspm #10767] Essai �, �, �

and missing a “\n” at the end

It should be something like :

echo “[Info Obspm #10767] Essai é, è, ã” |base64

W0luZm8gT2JzcG0gIzEwNzY3XSBFc3NhaSDDqSwgw6gsIMOjCg==

More stranger, if now I close the ticket from the web UI the mail RT send
to me is correct.

Any help ?

I’m totally stuck with this problem. The point is not only because I lost
some UTF-8 character, but because with some UTF-8 in the subject some scrip
don’t work, the mail isn’t send by RT to the requestor (for example).

Today I install from empty disk (on vmware) a fresh FreeBSD 9.2-RC4, I
don’t use anything from the old config, I init a empty database
(rt-setup-database --action=init). And I still get the problem.

So the problem don’t come from error in the configuration.

I remark, with fresh install the « autoreply » scrip (who have in the
script «Subject : etc…» is working perfectly, but since the « subject » is
not inside the scrip it’s stop working, the encoding is wrong.

Any help would very very welcome.

For example : where the subject is encoded inside RT and with which perl package ?

Regards.

JAS

Albert SHIH
DIO bâtiment 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
Téléphone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
lun 23 sep 2013 22:25:09 CEST

Le 23/09/2013 � 22:30:21+0200, Albert Shih a �crit

Any help ?

I’m totally stuck with this problem. The point is not only because I lost
some UTF-8 character, but because with some UTF-8 in the subject some scrip
don’t work, the mail isn’t send by RT to the requestor (for example).

Today I install from empty disk (on vmware) a fresh FreeBSD 9.2-RC4, I
don’t use anything from the old config, I init a empty database
(rt-setup-database --action=init). And I still get the problem.

So the problem don’t come from error in the configuration.

I remark, with fresh install the � autoreply ��scrip (who have in the
script �Subject : etc…��is working perfectly, but since the � subject ��is
not inside the scrip it’s stop working, the encoding is wrong.

Any help would very very welcome.

For example : where the subject is encoded inside RT and with which perl package ?

I try to find where the subject is encoded and find this thing little
strange.

I put some RT:Logger inside

Action/SendEmail.pm

in

sub SetHeaderAsEncoding {
my $self = shift;
my ( $field, $enc ) = ( shift, shift );

my $head = $self->TemplateObj->MIMEObj->head;

if ( lc($field) eq 'from' and RT->Config->Get('SMTPFrom') ) {   
    $head->replace( $field, RT->Config->Get('SMTPFrom') ); 
    return;
}

my $value = $head->get( $field );
$RT::Logger->debug( "In Action/Sendmail.pm '$field' and '$value' before ") ;

$value = $self->MIMEEncodeString( $value, $enc );
$head->replace( $field, $value );

$RT::Logger->debug( "In Action/Sendmail.pm '$value' just after ") ;

}

and the thing is in the log I find

Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘Subject’ and '[Support #5] AutoReply: Essai, \xc3\xa9, \xc3\xa0, \xc3\xa3 ’ before
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘=?UTF-8?B?W1N1cHBvcnQgIzVdIEF1dG9SZXBseTogRXNzYWksIMOpLCDDoCwgw6M=?=’ just after
Sep 23 23:05:49 support-9 RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W1N1cHBvcnQgIzVdIEF1dG9SZXBseTogRXNzYWksIMOpLCDDoCwgw6M=?=
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘Subject’ and '[Support #5] Essai, \xef\xbf\xbd, \xef\xbf\xbd, \xef\xbf\xbd ’ before
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘=?UTF-8?B?W1N1cHBvcnQgIzVdIEVzc2FpLCDvv70sIO+/vSwg77+9?=’ just after
Sep 23 23:05:49 support-9 RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W1N1cHBvcnQgIzVdIEVzc2FpLCDvv70sIO+/vSwg77+9?=

the first couple of encoding is correct, but why (normal ? ) RT try to
encode a second time, anyway the second time is wrong.

So or the problem come from RT with two encoding, or he come from some perl
package.

Regards.

JAS

Albert SHIH
DIO b�timent 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
T�l�phone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
lun 23 sep 2013 23:02:21 CEST

Le 23/09/2013 � 23:09:26+0200, Albert Shih a �crit

}

and the thing is in the log I find

Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘Subject’ and '[Support #5] AutoReply: Essai, \xc3\xa9, \xc3\xa0, \xc3\xa3 ’ before
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘=?UTF-8?B?W1N1cHBvcnQgIzVdIEF1dG9SZXBseTogRXNzYWksIMOpLCDDoCwgw6M=?=’ just after
Sep 23 23:05:49 support-9 RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W1N1cHBvcnQgIzVdIEF1dG9SZXBseTogRXNzYWksIMOpLCDDoCwgw6M=?=
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘Subject’ and '[Support #5] Essai, \xef\xbf\xbd, \xef\xbf\xbd, \xef\xbf\xbd ’ before
Sep 23 23:05:49 support-9 RT: In Action/Sendmail.pm ‘=?UTF-8?B?W1N1cHBvcnQgIzVdIEVzc2FpLCDvv70sIO+/vSwg77+9?=’ just after
Sep 23 23:05:49 support-9 RT: Converting ‘utf-8’ to ‘utf-8’ for text/plain - =?UTF-8?B?W1N1cHBvcnQgIzVdIEVzc2FpLCDvv70sIO+/vSwg77+9?=

the first couple of encoding is correct, but why (normal ? ) RT try to
encode a second time, anyway the second time is wrong.

So or the problem come from RT with two encoding, or he come from some perl
package.

YESSS…I find the problem…

The problem come with the new version of

p5-Encode-2.55

with the old version p5-Encode-2.49 it’s working again.

I don’t known where is the real problem, but between p5-Encode-2.55 and RT
they are some incompability.

Regards.

JAS
Albert SHIH
DIO b�timent 15
Observatoire de Paris
5 Place Jules Janssen
92195 Meudon Cedex
France
T�l�phone : +33 1 45 07 76 26/+33 6 86 69 95 71
xmpp: jas@obspm.fr
Heure local/Local time:
lun 23 sep 2013 23:23:26 CEST

YESSS…I find the problem…

The problem come with the new version of

p5-Encode-2.55

We’ve confirmed the bug; it is caused by Encode >= 2.53, and will be
fixed in RT 4.0.18 and 4.2.0. Until 4.0.18 is released, we suggest
downgrading Encode to version 2.52.

  • Alex