RT2-to-RT3 migration lost ticket data and status change issues

Hello all,

I am using the RT2-to-RT3 migration tool (latest) to migrate a RT2
instance (Mysql 3.23, DB is 1+ GB in size) from 2.0.14 to RT 3.6.7
(Mysql 5). I have copied the original RT2 install, and the new RT3
from their respective machines to a go between system. I’ve fixed the
paths and configurations so that both installs still can connect to
their respective databases.

This go between machine is running perl 5.8.8 with all the modules and
dependancies for a mysql based RT 3.6.7 instance. RT2 is running on a
Mac OS X panther server with Perl 5 (I think 004) and RT3 is running
on a Leopard Server also running Perl 5.8.8.

I had to update the RT2-to-dumpfile script in the migration package to
change the status as well as the priority of the tickets it exports.
In addition I had to add to RT3 the additional status files used by
the client. After that I was able to successfully run both the rt-2.0-
to-dumpfile script and the dumpfile to rt-3.0 script. All the basic
data appears to have come across intact including queues, acl’s, users
and ticket metadata.

First Problem:
While the ticket metadata has been exported, the actual transactions,
as well as ticket contents, emails, attachments etc have not been
exported. I have verified that they exist in the original database.

Second Problem:
Within RT3 I have edited the etc/RT_SiteConfig.pm to include the
additional status lines here:
@ActiveStatus = qw(new open waiting monitoring ongoing stalled verify
EC) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected dead deferred deleted) unless
@InactiveStatus;

From the default below:
@ActiveStatus = qw(new open stalled) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected deleted) unless @InactiveStatus;

The catch I have discovered is that when the old tickets were imported
about 90% of the tickets which were resolved now have the status of
new (specifically “new (unchanged)” ). These tickets instead should be
listed
as resolved.

Has anyone seen or experienced either of these behaviors before and
could provide any advice?

Any help would be appreciated,

  • Brian

When I upgraded several years ago, I had to run rt2-to-dumpfile BEFORE I
installed RT3 at all. Something about the version of SearchBuilder (I
think) that RT# used broke the data export of RT2. At one point I was
very good with the procedure for that upgrade, but my memory seems to
have been lost to time and projects past.

I hope this may help you out some. Searching for part of my email on
http://www.gossamer-threads.com/lists/rt/ may bring some of those posts
back up for you.

Brian Friday wrote:

Hello all,

I am using the RT2-to-RT3 migration tool (latest) to migrate a RT2
instance (Mysql 3.23, DB is 1+ GB in size) from 2.0.14 to RT 3.6.7
(Mysql 5). I have copied the original RT2 install, and the new RT3
from their respective machines to a go between system. I’ve fixed the
paths and configurations so that both installs still can connect to
their respective databases.

This go between machine is running perl 5.8.8 with all the modules and
dependancies for a mysql based RT 3.6.7 instance. RT2 is running on a
Mac OS X panther server with Perl 5 (I think 004) and RT3 is running
on a Leopard Server also running Perl 5.8.8.

I had to update the RT2-to-dumpfile script in the migration package to
change the status as well as the priority of the tickets it exports.
In addition I had to add to RT3 the additional status files used by
the client. After that I was able to successfully run both the rt-2.0-
to-dumpfile script and the dumpfile to rt-3.0 script. All the basic
data appears to have come across intact including queues, acl’s, users
and ticket metadata.

First Problem:

While the ticket metadata has been exported, the actual transactions,
as well as ticket contents, emails, attachments etc have not been
exported. I have verified that they exist in the original database.

Second Problem:

Within RT3 I have edited the etc/RT_SiteConfig.pm to include the
additional status lines here:
@ActiveStatus = qw(new open waiting monitoring ongoing stalled verify
EC) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected dead deferred deleted) unless
@InactiveStatus;

From the default below:
@ActiveStatus = qw(new open stalled) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected deleted) unless @InactiveStatus;

The catch I have discovered is that when the old tickets were imported
about 90% of the tickets which were resolved now have the status of
new (specifically “new (unchanged)” ). These tickets instead should be
listed
as resolved.


Has anyone seen or experienced either of these behaviors before and
could provide any advice?

Any help would be appreciated,

  • Brian

http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

Thanks Drew,

Your tips led me to the library issues. RT2 can not be exported
without the same perl modules used for RT2. Once I installed the
correct libraries everything exported correctly.

My import however still has one snag, or rather 51,000 or so snags. I
have 51,000 tickets with the status of “new” 99% of which did not
originally have that status when they were exported and did not have
that status in the dump file.

Anyone have any ideas on this issue that still remains?

  • BrianOn Dec 15, 2008, at 7:19 AM, Drew Barnes wrote:

When I upgraded several years ago, I had to run rt2-to-dumpfile
BEFORE I installed RT3 at all. Something about the version of
SearchBuilder (I think) that RT# used broke the data export of RT2.
At one point I was very good with the procedure for that upgrade,
but my memory seems to have been lost to time and projects past.

I hope this may help you out some. Searching for part of my email
on http://www.gossamer-threads.com/lists/rt/ may bring some of those
posts back up for you.

Brian Friday wrote:

Hello all,

I am using the RT2-to-RT3 migration tool (latest) to migrate a RT2
instance (Mysql 3.23, DB is 1+ GB in size) from 2.0.14 to RT 3.6.7
(Mysql 5). I have copied the original RT2 install, and the new RT3
from their respective machines to a go between system. I’ve fixed
the paths and configurations so that both installs still can
connect to their respective databases.

This go between machine is running perl 5.8.8 with all the modules
and dependancies for a mysql based RT 3.6.7 instance. RT2 is
running on a Mac OS X panther server with Perl 5 (I think 004) and
RT3 is running on a Leopard Server also running Perl 5.8.8.

I had to update the RT2-to-dumpfile script in the migration package
to change the status as well as the priority of the tickets it
exports. In addition I had to add to RT3 the additional status
files used by the client. After that I was able to successfully
run both the rt-2.0- to-dumpfile script and the dumpfile to rt-3.0
script. All the basic data appears to have come across intact
including queues, acl’s, users and ticket metadata.

First Problem:

While the ticket metadata has been exported, the actual
transactions, as well as ticket contents, emails, attachments etc
have not been exported. I have verified that they exist in the
original database.

Second Problem:

Within RT3 I have edited the etc/RT_SiteConfig.pm to include the
additional status lines here:
@ActiveStatus = qw(new open waiting monitoring ongoing stalled
verify EC) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected dead deferred deleted)
unless @InactiveStatus;

From the default below:
@ActiveStatus = qw(new open stalled) unless @ActiveStatus;
@InactiveStatus = qw(resolved rejected deleted) unless
@InactiveStatus;

The catch I have discovered is that when the old tickets were
imported about 90% of the tickets which were resolved now have the
status of new (specifically “new (unchanged)” ). These tickets
instead should be listed
as resolved.


Has anyone seen or experienced either of these behaviors before
and could provide any advice?

Any help would be appreciated,

  • Brian

http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly
Media. Buy a copy at http://rtbook.bestpractical.com


Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

Thanks Drew,

Your tips led me to the library issues. RT2 can not be exported
without the same perl modules used for RT2. Once I installed the
correct libraries everything exported correctly.

My import however still has one snag, or rather 51,000 or so snags. I
have 51,000 tickets with the status of “new” 99% of which did not
originally have that status when they were exported and did not have
that status in the dump file.

You didn’t say what version of RT-Extension-RT2toRT3 you’re
using, but I’ve done numerous rt2 to rt3 conversions using
the cpan versions without seeing this

If Status isn’t making it into the dumpfiles generated by rt-2.0-to-
dumpfile
then they will default to ‘new’ when being created by in rt3

-kevin

Thanks Drew,

Your tips led me to the library issues. RT2 can not be exported
without the same perl modules used for RT2. Once I installed the
correct libraries everything exported correctly.

My import however still has one snag, or rather 51,000 or so snags. I
have 51,000 tickets with the status of “new” 99% of which did not
originally have that status when they were exported and did not have
that status in the dump file.

You didn’t say what version of RT-Extension-RT2toRT3 you’re
using, but I’ve done numerous rt2 to rt3 conversions using
the cpan versions without seeing this

If Status isn’t making it into the dumpfiles generated by rt-2.0-to-
dumpfile
then they will default to ‘new’ when being created by in rt3

-kevin

Actually I did, latest, though I guess I could be more clear and said
SVN version as well.
The status exists in the dumped tickets but that status is not
maintained in the dumpfile to rt3
push rather all resolved tickets are pushed back to new a status we
did not have in RT2.

  • Brian

Thanks Drew,

Your tips led me to the library issues. RT2 can not be exported
without the same perl modules used for RT2. Once I installed the
correct libraries everything exported correctly.

My import however still has one snag, or rather 51,000 or so
snags. I
have 51,000 tickets with the status of “new” 99% of which did not
originally have that status when they were exported and did not have
that status in the dump file.

You didn’t say what version of RT-Extension-RT2toRT3 you’re
using, but I’ve done numerous rt2 to rt3 conversions using
the cpan versions without seeing this

If Status isn’t making it into the dumpfiles generated by rt-2.0-to-
dumpfile
then they will default to ‘new’ when being created by in rt3

-kevin

Actually I did, latest, though I guess I could be more clear and
said SVN version as well.

I don’t trust ‘latest’ as a version number since people occasionally
download the
incredibly old version on downloads.bestpractical.com rather than the
CPAN
version, and I fixed quite a bit for the CPAN version.

The status exists in the dumped tickets but that status is not
maintained in the dumpfile to rt3
push rather all resolved tickets are pushed back to new a status we
did not have in RT2.

The only change made to Status during the dumpfile -> rt3 script is
that if the status
is dead it is changed to deleted.

Have you verified the output by using Storable to reinflate one of the
stored ticket
files? If so, please send along the content (feel free to strip the
subject and anonymise
the other data) seeing the hash that ends up in the file would help me
make
suggestions, since the Status field is a rather straightforward copy
across the
versions.

-kevin

Actually I did, latest, though I guess I could be more clear and
said SVN version as well.

I don’t trust ‘latest’ as a version number since people occasionally
download the
incredibly old version on downloads.bestpractical.com rather than the
CPAN
version, and I fixed quite a bit for the CPAN version.

As I found when I went searching for this originally.

The status exists in the dumped tickets but that status is not
maintained in the dumpfile to rt3
push rather all resolved tickets are pushed back to new a status we
did not have in RT2.

The only change made to Status during the dumpfile -> rt3 script is
that if the status
is dead it is changed to deleted.

Status is definitely not dead

Have you verified the output by using Storable to reinflate one of the
stored ticket files?

Without sample code I’d hazard a guess you’ve lost me regarding what
or how you wish me to use storable.

If so, please send along the content (feel free to strip the
subject and anonymise
the other data) seeing the hash that ends up in the file would help me
make
suggestions, since the Status field is a rather straightforward copy
across the
versions.

I had thought so as well but in RT2 we are using status variables
which do not exist in RT3 and had to be
added. The comment section of that portion of the SiteConfig indicates
the existing status’s can not be removed.

I’ve sent a off list message containing a sample ticket santized but
which shows up as resolved in our current database, resolved in the
ticket dumpfile but new in the RT3 instance. I have also attached
screenshots of the ticket in the new RT instance showing that the last
transaction was to resolve the ticket but yet it was imported as new.

  • Brian

The status exists in the dumped tickets but that status is not
maintained in the dumpfile to rt3
push rather all resolved tickets are pushed back to new a status we
did not have in RT2.

The only change made to Status during the dumpfile -> rt3 script is
that if the status
is dead it is changed to deleted.

Status is definitely not dead

I assume select Status from Tickets where id = t-58818; returns
resolved?

Have you verified the output by using Storable to reinflate one of
the
stored ticket files?

Without sample code I’d hazard a guess you’ve lost me regarding what
or how you wish me to use storable.

Something like
perl -MStorable -MData::Dumper -le 'print Dumper retrieve(“t-111111”)'
should retrieve the contents.

If so, please send along the content (feel free to strip the
subject and anonymise
the other data) seeing the hash that ends up in the file would help
me
make
suggestions, since the Status field is a rather straightforward copy
across the
versions.

I had thought so as well but in RT2 we are using status variables
which do not exist in RT3 and had to be
added. The comment section of that portion of the SiteConfig indicates
the existing status’s can not be removed.

You can’t remove them, there are hooks if you want to rename the rt2
statuses to different ones in rt3 during the import.

I’ve sent a off list message containing a sample ticket santized but
which shows up as resolved in our current database, resolved in the
ticket dumpfile but new in the RT3 instance. I have also attached
screenshots of the ticket in the new RT instance showing that the last
transaction was to resolve the ticket but yet it was imported as new.

I teased apart the ticket file and it doesn’t actually have a Status
field,
which explains why it is showing up as new in rt3.

You should verify that FIELD_MAPPINGS in rt-2.0-to-dumpfile
for RT::Ticket contains Status and that your rt2 DB has a
Status column that matches up. The newer versions of
rt-2.0-to-dumpfile use a hardcoded rt2 schema to avoid
needing a really old version of SearchBuilder to function.

-kevin

I assume select Status from Tickets where id = t-58818; returns
resolved?

Correct

Current System:
mysql> select Status from Tickets where id = 58818;
| Status |
| resolved |

New System:
mysql> select Status from Tickets where id = 58818;
| Status |
| new |
1 row in set (0.00 sec)

Have you verified the output by using Storable to reinflate one of
the
stored ticket files?

Without sample code I’d hazard a guess you’ve lost me regarding what
or how you wish me to use storable.

Something like
perl -MStorable -MData::Dumper -le 'print Dumper retrieve(“t-111111”)'
should retrieve the contents.

bash-3.2$ perl -MStorable -MData::Dumper -le ‘print Dumper
retrieve(“t-58818”)’
$VAR1 = undef;

I had thought so as well but in RT2 we are using status variables
which do not exist in RT3 and had to be
added. The comment section of that portion of the SiteConfig
indicates
the existing status’s can not be removed.

You can’t remove them, there are hooks if you want to rename the rt2
statuses to different ones in rt3 during the import.

Yah we want to use our old status names where possible. Would be nice in
future versions for the status values to be in a lookup table or
something less
static…

I teased apart the ticket file and it doesn’t actually have a Status
field,
which explains why it is showing up as new in rt3.

You should verify that FIELD_MAPPINGS in rt-2.0-to-dumpfile
for RT::Ticket contains Status and that your rt2 DB has a
Status column that matches up. The newer versions of
rt-2.0-to-dumpfile use a hardcoded rt2 schema to avoid
needing a really old version of SearchBuilder to function.

Hmm… weird I am showing resolved as the status as I look at the
ticket contents but then I am not familiar enough with how the ticket
is constructed in the dumpfile. Status is in the FIELD_MAPPINGS of the
rt-2.0-to-dumpfile and everything seems fine.

If you have indeed fixed the old version of search builder issue I can
tell you that the fix does not work. I was only able to export the
ticket metadata with the current search builder and was unable to
export any of the transactions or attachment data. To get what I
thought was the full data I had to use DBIx::SearchBuilder 0.99 only
then did the data extract out (minus the status it seems).

  • Brian

I assume select Status from Tickets where id = t-58818; returns
resolved?

Correct

Ok, so it isn’t a mapping issue from “old custom status” to “new custom
status” which is what I was worried about as first.

I teased apart the ticket file and it doesn’t actually have a Status
field,
which explains why it is showing up as new in rt3.

Hmm… weird I am showing resolved as the status as I look at the
ticket contents but then I am not familiar enough with how the
ticket is constructed in the dumpfile. Status is in the
FIELD_MAPPINGS of the rt-2.0-to-dumpfile and everything seems fine.

You’re probably seeing the transaction setting it from new to resolved

You should search for nstore in rt-2.0-to-dumpfile, you’ll find a call
to it around line 447
You want to add something like
use Data::Dumper; print STDERR Dumper($tick_ds);
above that so you can get a log of what is actually
going to disk.

After that you’ll need to work backwords through the code
to see where the Status isn’t being found/set and try
to trace the code that builds up $tick_ds

-kevin

If you have indeed fixed the old version of search builder issue I
can tell you that the fix does not work. I was only able to export
the ticket metadata with the current search builder and was unable
to export any of the transactions or attachment data. To get what I
thought was the full data I had to use DBIx::SearchBuilder 0.99 only
then did the data extract out (minus the status it seems).

I used a 1.3x or a 1.4x version of SearchBuilder, haven’t tried with
one of the 1.5xs
I usually end up having the rt2 install and db copied to the new host
and working from
that for the export, although it varies from situation to situation.
I don’t have access to
the hosts where the last export was done to check DBIx::SearchBuilder
versions.

-kevin