Debug info on international searches

RT Developers
1

Hello, Jesse.
Today I debug nasty searches with international searches.

  1. If I drop UTF encode of ARGS in autohandler then searches work.
  2. RT(more Mason, more more HTML::Entities) do wrong escaping of perl
    UTF encoded strings. For eg: it’s do %HHH instead of %HH%HH for Russian
    chars. It could be fixed with own escaping:

$m->interp->set_escape( rtu =>
sub {
return unless defined ${ $[0] };
require Encode;
Encode::utf8_off( ${ $[0] } );
${ $[0] } =~ s/([^a-zA-Z0-9_.-])/uc sprintf("%%%02x",ord($1))/eg;

});

If we don’t fix this then browser intrpret this wrong and sends back
wrong sequence.

  1. This not fix all problems, because we don’t have unescaping of such
    sequences.
    I nasty implement unescaping in autohandler before UTF encoding just for
    test:

%ARGS = map {
# if they’ve passed multiple values, they’ll be an array. if they’ve
# passed just one, a scalar whatever they are, mark them as utf8
my $type = ref($);
(!$type)
? s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg? $ : $_ :
($type eq ‘ARRAY’)
? [ map { ref($) ? $ : s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg?
$_ : $_ } @$_ ] :
($type eq ‘HASH’)
? { map { ref($) ? $ : s/%([0-9A-Fa-f]{2})/chr(hex($1))/eg?
$_ : $_ } %$_ } : $_
} %ARGS;

This fix all FreezeThaw problems.

  1. But searches don’t work still because something escape high bit chars.
    Example of Dumped $TicketsObj->{TicketRestrictions}
    $VAR1 = {
    ‘14’ => {
    ‘FIELD’ => ‘Subject’,
    ‘VALUE’ => “\x{442}\x{435}\x{441}\x{442}”,
    ‘OPERATOR’ => ‘LIKE’,
    ‘DESCRIPTION’ => “Subject LIKE
    \x{442}\x{435}\x{441}\x{442}”
    }
    };

MySQL don’t like it or may be something else. When ARGS was not encoded
everything was fine with search, but it’s not right fix.

Have to go home right now.
Best regards, Ruslan.