Bulk Load Performance Problem

I’ve written a bulk-load script for loading user info in RT(3) and have a
strange performance issue.

My tests have been loading 6,000 users into an empty RT mysql database
using the RT API. The first 500 users take 4 minutes. From then on
each batch of 500 users created takes progressively longer - the
last 500 users takes 1 hr, 13 minutes to load!

I am using the RT API’s transactions with commits every 500 creations.

One odd thing is that when I run the program a second time to load the
next 6000 users (with 6000 already in the database), the speed is comparable
to the first run - in other words the speed is the same regardless of how
many users are already in the database.

Any suggestions as to why this process would get slower and slower as
it progresses?

Thanks,
Steve

I’ve written a bulk-load script for loading user info in RT(3) and have a
strange performance issue.

My tests have been loading 6,000 users into an empty RT mysql database
using the RT API. The first 500 users take 4 minutes. From then on
each batch of 500 users created takes progressively longer - the
last 500 users takes 1 hr, 13 minutes to load!

I am using the RT API’s transactions with commits every 500 creations.

One odd thing is that when I run the program a second time to load the
next 6000 users (with 6000 already in the database), the speed is comparable
to the first run - in other words the speed is the same regardless of how
many users are already in the database.

Any suggestions as to why this process would get slower and slower as
it progresses?

That sounds like a memory leak might be causing you to eat into swap.
First up, are you running 3.0.8 or something earlier? Does RT’s memory
use keep increasing throughout a program run?

Thanks,
Steve


rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

At 12:02 PM 1/27/2004 -0500, Jesse Vincent wrote:

That sounds like a memory leak might be causing you to eat into swap.
First up, are you running 3.0.8 or something earlier? Does RT’s memory
use keep increasing throughout a program run?

Thanks - I’m running 3.0.6. I’m not sure how to monitor RT’s memory
usage.

As an early attempt to see if this was some kind of memory leak
I tried disconnecting & reconnecting to RT after each transaction,
but this had no effect.

Steve>On Tue, Jan 27, 2004 at 10:12:23AM -0500, Stephen Turner wrote:

I’ve written a bulk-load script for loading user info in RT(3) and have a
strange performance issue.