RT 2.0.60 / Data migration : Translation and code page issues

THAUVIN_Blaise_Dir_I · January 25, 2003, 1:00am

Hi,

I tried your patch, and it had some strange effects.

On a database loaded from a dumpfile previously converted to UTF8 with iconv
(which is a BAD idea anyway, as discussed in an other thread), it seems to
work.

But on a database loaded from a raw RT2 dumpfile (ie, full of latin1
accented caracters in my case), it crashes with the following message. I
feel it wrongly identifies some latin1 single byte caracters as UTF8 two
bytes caracters.

Loading an RT3 instance with Latin1 strings should not be done, so maybe
this “bug” is acceptable

Blaise

System error

error: panic: utf8_length: unaligned end at
/usr/local/rt3/lib/RT/Tickets_Overlay.pm line 1917.

context:
…
1913: $ea = $ea->{$op};
1914: }
1915: exists $clause{$field} or $clause{$field} = ;
1916: $field =~ s!([‘"])!\$1!g;
1917: $value =~ s!([’"])!\$1!g;
1918: my $data = [ $ea, $type, $field, $op, $value ];
1919:
1920: # here is where we store extra data, say if it’s a keyword or
1921: # something. (I.e. “TYPE SPECIFIC STUFF”)
…
code stack: /usr/local/rt3/lib/RT/Tickets_Overlay.pm:1917
/usr/local/rt3/lib/RT/Tickets_Overlay.pm:1686
/usr/local/rt3/share/html/Elements/Quicksearch:41
/usr/local/rt3/share/html/index.html:38
/usr/local/rt3/share/html/autohandler:160

raw http://hotline.dauphin-affichage.com:81/index.html#raw error

-----Message d’origine-----
Envoye : jeudi 23 janvier 2003 16:57
A : THAUVIN Blaise (Dir. Informatique FRP)
issues…

I was able to understand the crash issue thanks to the patch proposed by
Rob. We have apostrophes in many of our queues
names. This worked fine in RT2, but seems to be forbidden in RT3. From the
Apache log file:

[crit]: _parser( ( ‘Queue’ = ‘Dauphin-Spid’R’ ) AND ( ‘Status’ =
‘open’ ) ) (/usr/local/rt3/lib/RT/
Tickets_Overlay_SQL.pm:123)

This is a bug in the new search code. I will look into this later
today/tonight, but a quick fix might be…

in Tickets_Overlay: _ProcessRestrictions

before my $data = [ $ea, …

stick:

$field =~ s!([‘"])!\$1!g;
$value =~ s!([’"])!\$1!g;

(I’m not awake yet, so that may not even compile.)

I could not see the tickets in the queue unless I remove the status
research criteria. The reason is it is selecting
tickets with status ‘ouvert’ or ‘nouveau’ while the database contains
‘open’ or ‘new’.

This may be a bad interaction between localization and the new search
stuff. The frontend needs to always use the english version when it
passes the data around, it should only use the translated version for
display.

-R

Robert_Spier · January 25, 2003, 4:42am

I tried your patch, and it had some strange effects.

On a database loaded from a dumpfile previously converted to UTF8 with iconv
(which is a BAD idea anyway, as discussed in an other thread), it seems to
work.

Interesting, because my tests this evening show that it probably won’t
work, because it escapes things but doesn’t unescape them. What
database are you using?

But on a database loaded from a raw RT2 dumpfile (ie, full of latin1 accented
caracters in my case), it crashes with the following message. I feel it
wrongly identifies some latin1 single byte caracters as UTF8 two bytes
caracters.
Loading an RT3 instance with Latin1 strings should not be done, so maybe this
“bug” is acceptable

I agree. There’s not much that can be done about this. If we can’t
put data through an arbitrary regex, the data is wrong.

Anyway, here’s a new patchset to help with this quoting issue and a
related search issue.

Jesse, please confirm receipt.

-R

— Tickets_Overlay_SQL.pm 2003-01-15 20:22:20-08 1.3
+++ Tickets_Overlay_SQL.pm 2003-01-24 20:34:15-08
@@ -170,6 +170,10 @@
substr($val,-1,1) = “”;
}

```
 # Unescape escaped characters
```
```
 $key =~ s!\\(.)!$1!g;
```

 $val =~ s!\\(.)!$1!g;
 #    print "$ea Key=[$key] op=[$op]  val=[$val]\n";

 my $class;

@@ -289,7 +293,8 @@

set SB’s dirty flag

$self->{‘must_redo_search’} = 1;

$self->{‘RecalcTicketLimits’} = 0;
$self;
}

— Tickets_Overlay.pm 2003-01-15 22:43:38-08 1.6
+++ Tickets_Overlay.pm 2003-01-24 20:32:17-08
@@ -1907,6 +1907,10 @@
}
exists $clause{$field} or $clause{$field} = ;

```
 # Escape Quotes
```
```
 $field =~ s!(['"])!\\$1!g;
```

 $value =~ s!(['"])!\\$1!g;
 my $data = [ $ea, $type, $field, $op, $value ];

 # here is where we store extra data, say if it's a keyword or

@@ -1921,7 +1925,6 @@

 $self->FromSQL($sql);

$self->{‘RecalcTicketLimits’} = 0;
}

THAUVIN_Blaise_Dir_I · January 25, 2003, 11:25pm

Hi,

I applied the patch to 2.1.61.
Queues with apostrophes in their names are now working well.
Thats all I could test for now as I can’t yet upload utf8 compliant tickets.