First attempt at importing

Tried importing a 8000 ticket RT1 database yesterday using rt-1-3-90.

I’m running Postgres on another box and found that bin/initacls.Pg is
missing the -h option:

*** initacls.Pg-orig Tue Jun 5 18:14:50 2001
initacls.Pg Tue Jun 5 18:15:27 2001
*** 14,18 ****
export PATH

echo “Enter the postgres administrator’s database password to create a new user for rt”
! psql -d $DATABASENAME -f $DATABASEACLS -U $DATABASEADMIN

— 14,18 ----
export PATH

echo “Enter the postgres administrator’s database password to create a new user for rt”
! psql -h $HOSTNAME -d $DATABASENAME -f $DATABASEACLS -U $DATABASEADMIN

My RT1 database has tickets imported from Req – import-1.0-to-2.0
didn’t like the “import” transaction type. I changed it to treat
"import" like “create”. I’m not sure if it’s correct, but it seems to
work.

*** 502,507 ****
— 502,511 ----
$load_content = 1;
$trans_args{‘Type’} = “Create”;
}

  • elsif ( $row->{type} eq ‘import’ ) {
  •   $load_content = 1;
    
  •   $trans_args{'Type'} = "Create";
    
  • }
    elsif ( $row->{type} eq ‘status’ ) {
    $trans_args{‘Type’} = “Status”;
    $trans_args{‘Field’} = “Status”;

There were a number of warnings in the log that I haven’t had a chance
to track down yet. These two appeared once near the start:

Odd number of elements in hash assignment at /usr/local/rt/lib/RT/User.pm line 538.
Use of uninitialized value at /usr/local/lib/perl5/site_perl/5.005/DBIx/SearchBuilder/Record/Cachable.pm line 188.

The following errors frequently occur while importing tickets:

Use of uninitialized value at /usr/local/rt/lib/RT/Ticket.pm line 397, chunk 521.
Use of uninitialized value at /usr/local/rt/lib/RT/Ticket.pm line 409, chunk 521.
RT::Ticket=HASH(0x881b9c8) ->Create called with an unknown datatype for Owner: . Defaulting to Nobody.

and:

RT::Ticket=HASH(0x87d3f6c) user sharris(4) was proposed as a ticket owner but has no rights to own tickets in this queue

It appears that user acls were not imported correctly. The tickets in
the database look OK but I can’t access them from the web ui. Sounds
like this may be fixed already, I’ll try again with the current
snapshot this afternoon.

– Bob

These fixes have been applied to RT currently in cvs. they’ll be in 1.3.95.
I look forward to hearing how the import goes with 1.3.95.

    -jOn Wed, Jun 06, 2001 at 11:45:34AM -0700, Bob Finch wrote:

Tried importing a 8000 ticket RT1 database yesterday using rt-1-3-90.

I’m running Postgres on another box and found that bin/initacls.Pg is
missing the -h option:

*** initacls.Pg-orig Tue Jun 5 18:14:50 2001
initacls.Pg Tue Jun 5 18:15:27 2001


*** 14,18 ****
export PATH

echo “Enter the postgres administrator’s database password to create a new user for rt”
! psql -d $DATABASENAME -f $DATABASEACLS -U $DATABASEADMIN

— 14,18 ----
export PATH

echo “Enter the postgres administrator’s database password to create a new user for rt”
! psql -h $HOSTNAME -d $DATABASENAME -f $DATABASEACLS -U $DATABASEADMIN

My RT1 database has tickets imported from Req – import-1.0-to-2.0
didn’t like the “import” transaction type. I changed it to treat
"import" like “create”. I’m not sure if it’s correct, but it seems to
work.


*** 502,507 ****
— 502,511 ----
$load_content = 1;
$trans_args{‘Type’} = “Create”;
}

  • elsif ( $row->{type} eq 'import' ) {
    
  •     $load_content = 1;
    
  •     $trans_args{'Type'} = "Create";
    
  • } 
    elsif ( $row->{type} eq 'status' ) {
        $trans_args{'Type'} = "Status";
        $trans_args{'Field'} = "Status";
    

There were a number of warnings in the log that I haven’t had a chance
to track down yet. These two appeared once near the start:

Odd number of elements in hash assignment at /usr/local/rt/lib/RT/User.pm line 538.
Use of uninitialized value at /usr/local/lib/perl5/site_perl/5.005/DBIx/SearchBuilder/Record/Cachable.pm line 188.

The following errors frequently occur while importing tickets:

Use of uninitialized value at /usr/local/rt/lib/RT/Ticket.pm line 397, chunk 521.
Use of uninitialized value at /usr/local/rt/lib/RT/Ticket.pm line 409, chunk 521.
RT::Ticket=HASH(0x881b9c8) ->Create called with an unknown datatype for Owner: . Defaulting to Nobody.

and:

RT::Ticket=HASH(0x87d3f6c) user sharris(4) was proposed as a ticket owner but has no rights to own tickets in this queue

It appears that user acls were not imported correctly. The tickets in
the database look OK but I can’t access them from the web ui. Sounds
like this may be fixed already, I’ll try again with the current
snapshot this afternoon.

– Bob


Rt-devel mailing list
Rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

jesse reed vincent – root@eruditorum.orgjesse@fsck.com
70EBAC90: 2A07 FC22 7DB4 42C1 9D71 0108 41A3 3FB3 70EB AC90

autoconf is your friend until it mysteriously stops working, at which
point it is a snarling wolverine attached to your genitals by its teeth
(that said, it’s better than most of the alternatives) – Nathan Mehl

“Jesse” == Jesse jesse@fsck.com writes:

Jesse> I look forward to hearing how the import goes with 1.3.95.

So far, importing with 1.3.95 has fixed all the problems I ran into
earlier.

My latest import hasn’t finished yet, it’s got about 500 (out of 7575)
tickets to go. It’s been running about 8 hours now – at the
beginning it was doing 1 or 2 tickets per second, now it is averaging
about 2 tickets per minute, so I’d guess I’ve got a few more hours to
go…

The setup here is probably the worst possible, though. RT2 is on one
machine, RT1 is on a second, and the Postgres RT2 database is on a
third.

A couple things I’ve noticed so far…

Merged tickets don’t look right. Oops, I just looked at the import
script and it looks like it doesn’t fix up merged records untill the
end. Guess I’ll have to wait.

The transactions appear out of order on some tickets. I think this
only happens with tickets that were originally imported from Req.
Maybe “order by trans_date” should be added to the select in
MigrateTransactions?

– Bob

“Jesse” == Jesse jesse@fsck.com writes:

Jesse> I look forward to hearing how the import goes with 1.3.95.

So far, importing with 1.3.95 has fixed all the problems I ran into
earlier.

Cool. FWIW, 1.3.97 fixes a few bugs with ‘set owner’ transactions, I believe.

My latest import hasn’t finished yet, it’s got about 500 (out of 7575)
tickets to go. It’s been running about 8 hours now – at the
beginning it was doing 1 or 2 tickets per second, now it is averaging
about 2 tickets per minute, so I’d guess I’ve got a few more hours to
go…

There are definitely some more performance tweaks that would be useful.
I suspect that it’s mainly losing due to the lack of some index or other on
requestor.

The transactions appear out of order on some tickets. I think this
only happens with tickets that were originally imported from Req.
Maybe “order by trans_date” should be added to the select in
MigrateTransactions?

Hmmm. That shouldn’t be necessary with a standard RT install. But give that
a shot on your local instance. if it looks like it does the right thing,
I’ll roll it into the release.

– Bob

jesse reed vincent – root@eruditorum.orgjesse@fsck.com
70EBAC90: 2A07 FC22 7DB4 42C1 9D71 0108 41A3 3FB3 70EB AC90

“Mary had a crypto key / She kept it in escrow
And everything that Mary said / The Feds were sure to know” – Sam Simpson

“Jesse” == Jesse jesse@fsck.com writes:

> The transactions appear out of order on some tickets.  I think this
> only happens with tickets that were originally imported from Req.
> Maybe "order by trans_date" should be added to the select in
> MigrateTransactions?

Jesse> Hmmm. That shouldn't be necessary with a standard RT
Jesse> install. But give that a shot on your local instance. if it
Jesse> looks like it does the right thing, I'll roll it into the
Jesse> release.

OK. I’ll try that today. The problem is just with Req records that
were imported into RT1. When I do an unordered select in the RT1
database for a serial number that was imported from Req, they appear
in reverse cronological order – the ones inserted by RT1 are OK.

The import finally finished, but only one ticket was merged. Looks
like MyMerges is only picking up the first one it’s passed. I’m
going to try it again with the following patch:

*** import-1.0-to-2.0 Wed Jun 6 13:15:03 2001
— import-1.0-to-2.0-test Fri Jun 8 10:54:35 2001
*** 797,803 ****

{{{ sub MigrateMerges

sub MigrateMerges {
! my @merges = shift;

  foreach my $row (@merges) {
print "Merging ".$row->{'ticket'} ." into ". $row->{'mergeinto'} ."\n";

— 797,803 ----

{{{ sub MigrateMerges

sub MigrateMerges {
! my @merges = @_;

  foreach my $row (@merges) {
print "Merging ".$row->{'ticket'} ." into ". $row->{'mergeinto'} ."\n";

– Bob

“Jesse” == Jesse jesse@fsck.com writes:

> The transactions appear out of order on some tickets.  I think this
> only happens with tickets that were originally imported from Req.
> Maybe "order by trans_date" should be added to the select in
> MigrateTransactions?

Jesse> Hmmm. That shouldn't be necessary with a standard RT
Jesse> install. But give that a shot on your local instance. if it
Jesse> looks like it does the right thing, I'll roll it into the
Jesse> release.

OK. I’ll try that today. The problem is just with Req records that
were imported into RT1. When I do an unordered select in the RT1
database for a serial number that was imported from Req, they appear
in reverse cronological order – the ones inserted by RT1 are OK.

Weird. Incidentally, which of the three req importers did you use?

The import finally finished, but only one ticket was merged. Looks
like MyMerges is only picking up the first one it’s passed. I’m
going to try it again with the following patch:

Uh. yeah. sheepish grin That might be good :slight_smile:

*** import-1.0-to-2.0 Wed Jun 6 13:15:03 2001
— import-1.0-to-2.0-test Fri Jun 8 10:54:35 2001


*** 797,803 ****

{{{ sub MigrateMerges

sub MigrateMerges {
! my @merges = shift;

  foreach my $row (@merges) {
  print "Merging ".$row->{'ticket'} ." into ". $row->{'mergeinto'} ."\n";

— 797,803 ----

{{{ sub MigrateMerges

sub MigrateMerges {
! my @merges = @_;

  foreach my $row (@merges) {
  print "Merging ".$row->{'ticket'} ." into ". $row->{'mergeinto'} ."\n";

– Bob

jesse reed vincent – root@eruditorum.orgjesse@fsck.com
70EBAC90: 2A07 FC22 7DB4 42C1 9D71 0108 41A3 3FB3 70EB AC90

Transporters are so ungodly. if god had wanted us to travel great distances
instantaneously, he would have given us an internal
materialisation/dematerialisation control.
– Shoshe Cole

“Jesse” == Jesse jesse@fsck.com writes:

Jesse> Weird. Incidentally, which of the three req importers did
Jesse> you use?

yar2r-1.13

Just finished importing w/ 1.3.97. Adding an index for Users on
EmailAddress shaved about an hour off the total time.

Merging looks good this time. The handful of merged tickets I looked
at were all OK. I’ll take a closer look over the weekend…

I had to do:

select setval(‘tickets_id_seq’, (select max(id) from tickets));

in psql before I could add new tickets. I guess since import
explicitly sets ticket numbers, the sequence number never gets
incremented…

– Bob

“Jesse” == Jesse jesse@fsck.com writes:

Jesse> Weird. Incidentally, which of the three req importers did
Jesse> you use?

yar2r-1.13

Just finished importing w/ 1.3.97. Adding an index for Users on
EmailAddress shaved about an hour off the total time.

Urk. the pg schema was out of sync with the base schema. sorry about that.

Merging looks good this time. The handful of merged tickets I looked
at were all OK. I’ll take a closer look over the weekend…

I had to do:

select setval(‘tickets_id_seq’, (select max(id) from tickets));

in psql before I could add new tickets. I guess since import
explicitly sets ticket numbers, the sequence number never gets
incremented…

nod I will add a note to this effect to the install instructions.

– Bob


Rt-devel mailing list
Rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

jesse reed vincent – root@eruditorum.orgjesse@fsck.com
70EBAC90: 2A07 FC22 7DB4 42C1 9D71 0108 41A3 3FB3 70EB AC90

I think co-ordinating 1000 prima donnas living all over the world will be as
easy as herding cats…" – Andy Tanenbaum on the linux development model, 1992