More rt2 -> rt3 conversion woes

Hello All,

So I’ve got another mucked up rt2 -> rt3 conversion issue. I’ve searched
and read what I believe would be all the posts in regards to this, and
I’ve tried some of the things that they’ve told.

Here’s the tale. We’ve got an rt instance with almost 200k users, and a
little over 300k tickets. I’ve got some new hardware to put the latest
version of rt on. So i’ve got the new system setup and ready to go for rt3.
No biggy.

I’ve made a copy of the original rt database and put the copy on a staging
system while I test out the rt2 -> rt3 export/import script, as to not mess with
production instances. I had DBIx::SearchBuilder-1.39 installed (the
only reason I mention this, was that this module was the focal point of
other questions regarding the import). I was able to do an export, but
like others, I was not able to import. There was a lot of user (and later to
be found… a lot of other) information that was missing.

I downgraded the DBIx::SearchBuilder down to 0.61, and started an export.
It just churned, churned, and chuned some more before the perl process
took all memory and exited out. Quirky, I thought. I figured that this
must have to do with the amount of users that we have, since that’s the
first stage of the export process. I re-upgraded to 1.39 and ran an
export, to at least get the user data out. I then massaged that data
with another script to populate the rest of the missing fields from the
database.

It also exported the queues, tickets and so forth, but all that information was
bunk as well. However, the users were ok! I copied the metadata and tickets
over and imported the users using SearchBuilder-1.39 (I believe… this process
has been painful.) Things went well until it had to import everything else,
which it couldn’t once I got past the user import. Gah!

I decided to go back to the staging host and nuke the users, or at least the
users without the Privileged bit set. I downgraded SearchBuilder back to 0.61,
and restarted the export. This time it started cranking through everything.
It didn’t have any memory issues, and I was able to successfully (From what
I gathered) export everything else. I did comparisons of the values I was
able to export using 0.61 versus 1.39, and the 0.61 export had a bit more
information. So I assumed the export went ok.

I took this exported data and copied it back to the new rt instance. I was
still using SearchBuilder-1.39, and when I do the import, here is the issue
that I run into:

rt2# ./dumpfile-to-rt-3.0 test Importing users .... user stuff ... .uCreating user stegen . Importing groups rrrCouldn't load principal "" to grant them AdminGroups globally $VAR1 = { 'PrincipalType' => 'Group', 'PrincipalId' => '7', 'Name' => 'AdminGroups' }; [Wed Mar 22 23:21:48 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) [Wed Mar 22 23:21:48 2006] [crit]: Can't load a principal for id (/usr/local/rt /lib/RT/ACE_Overlay.pm:907) rrr[Wed Mar 22 23:21:48 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906 ) [Wed Mar 22 23:21:48 2006] [crit]: Can't load a principal for id (/usr/local/rt /lib/RT/ACE_Overlay.pm:907) rrrrCouldn't load principal "" to grant them AdminKeywords globally $VAR1 = { 'PrincipalType' => 'Group', 'PrincipalId' => '7', 'Name' => 'AdminKeywords' };

gah… good times. For giggles and grins, I decided to downgrade SearchBuilder
down to 0.61, to see what happens.

Here’s what happens:

rt2# ./dumpfile-to-rt-3.0 test
Importing users
uCreating user ronni
.uCreating user root
.[Wed Mar 22 23:27:09 2006] [crit]: Can’t locate object method “ApplyLimits” via
package “RT::Handle” at /usr/local/rt/lib/RT/Principal_Overlay.pm line 417.
(/usr/local/rt/lib/RT.pm:276)
</another output snippet>

ApplyLimits being a method that does not exist in SearchBuilder-0.61.

So I’ve been sitting here trying to get some sort of import to work. I’m
wondering if anyone else may have any insight as to what is going on. I’m
sure there’s some information that I’m neglecting to include…

Anyways… any help would be appreciated.

thanks!
stegen

Hello All,

So I’ve got another mucked up rt2 -> rt3 conversion issue. I’ve searched
and read what I believe would be all the posts in regards to this, and
I’ve tried some of the things that they’ve told.

Here’s the tale. We’ve got an rt instance with almost 200k users, and a
little over 300k tickets. I’ve got some new hardware to put the latest
version of rt on. So i’ve got the new system setup and ready to go for rt3.
No biggy.

Brutal, I went through this with a 800,000 ticket system - your pain
is heartfelt!

I downgraded the DBIx::SearchBuilder down to 0.61, and started an export.
It just churned, churned, and chuned some more before the perl process
took all memory and exited out. Quirky, I thought. I figured that this
must have to do with the amount of users that we have, since that’s the
first stage of the export process. I re-upgraded to 1.39 and ran an
export, to at least get the user data out. I then massaged that data
with another script to populate the rest of the missing fields from the
database.

You need to prune the Users table - I suspect your system received a
large number of spam emails at one point, or it truly has a large
number of distinct Users.

At any rate, check this post:

http://lists.bestpractical.com/pipermail/rt-users/2004-September/025971.html

I basically pruned off all users older than 3 months in order to get
the import to work. It was smooth sailing after that point.

Hope this helps!

Hi Brian,

Thanks for the advice. I went ahead and trimmed the Users table quite a bit,
and re ran the export. It ran successfully, so I copied the export to the
new rt system and restarted the import. I’ve done a dumpdb and
initialize-database to have a fresh start on rt3. Here’s what went down.

rt2# ./dumpfile-to-rt-3.0 test Importing users ... email addresses .uCreating user stegen ... after email addresses . Importing groups ggrrrr[Fri Mar 24 00:16:14 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) [Fri Mar 24 00:16:14 2006] [crit]: Can't load a principal for id (/usr/local/rt/lib/RT/ACE_Overlay.pm:907) rr[Fri Mar 24 00:16:14 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) [Fri Mar 24 00:16:14 2006] [crit]: Can't load a principal for id (/usr/local/rt/lib/RT/ACE_Overlay.pm:907) rrrrrrr[Fri Mar 24 00:16:14 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) [Fri Mar 24 00:16:14 2006] [crit]: Can't load a principal for id (/usr/local/rt/lib/RT/ACE_Overlay.pm:907) rr[Fri Mar 24 00:16:14 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) [Fri Mar 24 00:16:14 2006] [crit]: Can't load a principal for id (/usr/local/rt/lib/RT/ACE_Overlay.pm:907) rrrrrrr[Fri Mar 24 00:16:14 2006] [crit]: 1 (/usr/local/rt/lib/RT/ACE_Overlay.pm:906) .....

This didn’t look good to me; however, I let the import finish, and decided
to check out one of the tickets. The login, worked, the queues were there,
and tickets existed with subjects in the main home screen. When I click on one
of the ticket links, I would see the following error

error: RT::Queue::TicketCustomFields Unimplemented in HTML::Mason::Commands. (/usr/local/rt/local/html/Ticket/Elements/ShowSummary line 56)
context:

492: else {
493: my ( $package, $filename, $line );
494: ( $package, $filename, $line ) = caller;
495:
496: die “$AUTOLOAD Unimplemented in $package. ($filename line $line) \n”;
497: }
498:
499: }
500:

code stack: /usr/local/lib/perl5/site_perl/5.8.7/DBIx/SearchBuilder/Record.pm:496
/usr/local/rt/local/html/Ticket/Elements/ShowSummary:56
/usr/local/rt/local/html/Ticket/Display.html:57
/usr/local/rt/share/html/autohandler:215
raw error

I was skeptical to even see what I saw after seeing so many errors, but it’s
a drag that the ticket content wasn’t there. Have you ever seen this error
before? It’s stumping me…

Thanks for the help!
stegen

Brian Kerr [kerrboy@gmail.com] wrote:> On 3/22/06, Stegen Smith stegen.smith@websidestory.com wrote:

Hello All,

So I’ve got another mucked up rt2 -> rt3 conversion issue. I’ve searched
and read what I believe would be all the posts in regards to this, and
I’ve tried some of the things that they’ve told.

Here’s the tale. We’ve got an rt instance with almost 200k users, and a
little over 300k tickets. I’ve got some new hardware to put the latest
version of rt on. So i’ve got the new system setup and ready to go for rt3.
No biggy.

Brutal, I went through this with a 800,000 ticket system - your pain
is heartfelt!

I downgraded the DBIx::SearchBuilder down to 0.61, and started an export.
It just churned, churned, and chuned some more before the perl process
took all memory and exited out. Quirky, I thought. I figured that this
must have to do with the amount of users that we have, since that’s the
first stage of the export process. I re-upgraded to 1.39 and ran an
export, to at least get the user data out. I then massaged that data
with another script to populate the rest of the missing fields from the
database.

You need to prune the Users table - I suspect your system received a
large number of spam emails at one point, or it truly has a large
number of distinct Users.

At any rate, check this post:

http://lists.bestpractical.com/pipermail/rt-users/2004-September/025971.html

I basically pruned off all users older than 3 months in order to get
the import to work. It was smooth sailing after that point.

Hope this helps!