RT 2.0.60 / Data migration : Translation and code page issues

Good morning Jesse, Rob,

I have good and bad news for you both. I start with good :

The upload script worked fine. No warning or error. All my data seems to be
in.

I was able to understand the crash issue thanks to the patch proposed by
Rob. We have apostrophes in many of our queues names. This worked fine in
RT2, but seems to be forbidden in RT3. From the Apache log file:
[crit]: _parser( ( ‘Queue’ = ‘Dauphin-Spid’R’ ) AND ( ‘Status’ = ‘open’
) ) (/usr/local/rt3/lib/RT/Tickets_Overlay_SQL.pm:123)

And now the bad news :

Performance : dead slow, especially compared to the RT2 instance running
aside on the same server. Server CPU usage is not high.

Everything else is related to UTF8 and translations:

  • I could not see the tickets in the queue unless I remove the status
    research criteria. The reason is it is selecting tickets with status
    ’ouvert’ or ‘nouveau’ while the database contains ‘open’ or ‘new’.
    Translating the status in the database (can’t be done during import as
    target language is unknown and client UI dependent) would solve my problem,
    but would not in a multi-language environnement where users with different
    languages can share tickets. It would probably be best to show the
    translated description of each status on the UI, but still work on IDs or
    english name in the database. I guess this is a major modification… in
    the meantime, we could probably remove all translations on such keywords.
    Maybe this happens with some other field I don’t think of yet.

  • Only the login page shows accented caracters correctly. Once loged, every
    accented caracter is schown as two caracters. The explorer (IE6) says its
    using UTF8. Forcing it to ISO shwos 4 caracters for one. I remember
    something about double UTF8 translation in the mailing list… I guess I am
    hitting this bug.

  • UTF8 translation within the database. I did not take care of configuring
    anything related to UTF8 when I compiled or configured MySQL. Maybe I should
    have?

  • UTF8 conversion during upload : data is uploaded “as is”, which means
    accented caracters from the RT2 instance are stored as 8 bit caracters in
    the RT3 database. Data typed in from the RT3 UI is stored in UTF8 format.
    They show OK on the UI afterwards. Obviously, imported accents are not shown
    correctly.
    Based on my experience with an Oracle database, I would say we need to
    create the RT3 database stating its native format as UTF8 and then specify
    which codepage each client is using when connecting so the translation is
    done “on the fly”. In that case, there would be a translation done when
    importing (client connected ISO to database running UTF-8), but no
    translation afterwards. From a performance point of view, this sounds
    better. Is it feasible with MySQL?
    This implies foreign users specify to the export or import tool the code
    page they used when running RT2 so the translation is done properly.

That’s all for now.

Blaise

-----Message d’origine-----
Envoyé : jeudi 23 janvier 2003 00:47À : Jesse Vincent
Cc : THAUVIN Blaise (Dir. Informatique FRP)
Objet : Re: [fwd] Failure report : RT 2.0.60 / Data migration (from:
bthauvin@clearchannel.fr)

Stick

$RT::Logger->crit("_parser( $string ) ")

towards the beginning of _parser in Tickets_Overlay_SQL.

That should get me the information that will help trace it back. I’ll
probably also need more information, but this will help get an idea of
what’s going wrong and where.

-R

At Wed, 22 Jan 2003 18:20:28 -0500,
Jesse Vincent wrote:

Any idea?

----- Forwarded message from "THAUVIN Blaise (Dir. Informatique FRP)"
bthauvin@clearchannel.fr -----

From: “THAUVIN Blaise (Dir. Informatique FRP)” bthauvin@clearchannel.fr
To: ‘Jesse Vincent’ jesse@bestpractical.com
Subject: Failure report : RT 2.0.60 / Data migration
Date: Wed, 22 Jan 2003 23:58:21 +0100

Last thing, RT crashes with the imported data, while it works fine with a
freshly initialized database. I get the login window, but can’t get any
further. When I log, I get a page saying :


error: Error near ->R<- expecting a PAREN

context: …
123: $current = PAREN if _match($re_paren,$val);
124:
125: unless ($current && $want & $current) {
126: # Error
127: die "Error near ->$val<- expecting a ", $tokens[((log $want)/(log
2))], “\n”;
128: }
129:
130: # State Machine:
131:

code stack: /usr/local/rt3/lib/RT/Tickets_Overlay_SQL.pm:127
/usr/local/rt3/lib/RT/Tickets_Overlay_SQL.pm:259
/usr/local/rt3/lib/RT/Tickets_Overlay.pm:1929
/usr/local/rt3/lib/RT/Tickets_Overlay.pm:1686
/usr/local/rt3/share/html/Elements/Quicksearch:41
/usr/local/rt3/share/html/index.html:38
/usr/local/rt3/share/html/autohandler:160

There is nothing in the Apache log and I must say I don’t know where to
start investigate for that one.

Blaise

-----Message d’origine-----
De : Jesse Vincent [mailto:jesse@bestpractical.com]
Envoyé : mercredi 22 janvier 2003 23:54
À : THAUVIN Blaise (Dir. Informatique FRP)
Objet : Re: [rt-devel] Failure report : RT 2.0.60 / Data migration

The new one I sent you should cope with the case where there are links
to missing tickets. please try with that and the bogus dependency in
place. Thanks!

----- End forwarded message -----


»|« http://www.bestpractical.com/rt – Trouble Ticketing. Free.

I was able to understand the crash issue thanks to the patch proposed by Rob. We have apostrophes in many of our queues
names. This worked fine in RT2, but seems to be forbidden in RT3. From the Apache log file:

[crit]: _parser( ( ‘Queue’ = ‘Dauphin-Spid’R’ ) AND ( ‘Status’ = ‘open’ ) ) (/usr/local/rt3/lib/RT/
Tickets_Overlay_SQL.pm:123)

This is a bug in the new search code. I will look into this later
today/tonight, but a quick fix might be…

in Tickets_Overlay: _ProcessRestrictions

before my $data = [ $ea, …

stick:

$field =~ s!([’"])!\$1!g;
$value =~ s!([’"])!\$1!g;

(I’m not awake yet, so that may not even compile.)

  • I could not see the tickets in the queue unless I remove the status research criteria. The reason is it is selecting
    tickets with status ‘ouvert’ or ‘nouveau’ while the database contains ‘open’ or ‘new’.

This may be a bad interaction between localization and the new search
stuff. The frontend needs to always use the english version when it
passes the data around, it should only use the translated version for
display.

-R