Rt2-to-rt3-1.23 out of memory

We’re planning to migrate from RT2 to RT3…

We have roughly 500,000 tickets, the DB is about 5GB.

When I tried to run rt-2.0-to-dumpfile it ran out of memory. The Linux
machine in question has 4GB and 4GB of swap.

Any ideas on how to proceed? Is there any way to do this incrementally?

Rob

Robert Loomans, Programmer/Analyst robertl@apnic.net
Asia Pacific Network Information Centre (APNIC) Tel: +61-7-3858-3100
http://www.apnic.net Fax: +61-7-3858-3199

smime.p7s (3.67 KB)

When I tried to run rt-2.0-to-dumpfile it ran out of memory. The Linux
machine in question has 4GB and 4GB of swap.

A common failing is understanding that unix like systems are not
originally single-user boxes. That means that just because your box
has X amount of resource it does not mean that any single process can
use that much resource. In particular, check the memory resource
limits of your process that is running the export.

Anyhow, did your export get any tickets out or did it die while
exporting the other info like users list and configs?

I’ve found that it helps to purge the user list of owners who do not
own any tickets (be sure not to delete the root user, or any privileged
user). You can do this with a left join on the tickets table on the
owner id. I do this after purging dead tickets (we get a boatload of
spam to our abuse@ queue).

A common failing is understanding that unix like systems are not
originally single-user boxes. That means that just because your box has
X amount of resource it does not mean that any single process can use
that much resource. In particular, check the memory resource limits of
your process that is running the export.

No such luck. The machine is a test box… Aside from the usual
suspects (sshd, syslog, etc), the only things running on the machine
were mysql and the script. There are no ulimits set up for users.

I’m not sure if the kernel is using the Linux default 3G/1G
userspace/kernel split or 4G/4G, but the process had topped 2.5G when I
went home… When I got home and checked, the process had aborted with
the message “Out of memory!”

Anyhow, did your export get any tickets out or did it die while
exporting the other info like users list and configs?

Nope… looks like it was still loading stuff for the metadata file.

I’ve found that it helps to purge the user list of owners who do not own
any tickets (be sure not to delete the root user, or any privileged
user). You can do this with a left join on the tickets table on the
owner id. I do this after purging dead tickets (we get a boatload of
spam to our abuse@ queue).

That’s my next step before I start modifying the script… I’ve written
a script that looks for dead tickets, removes them and prunes any orphan
transactions, attachments, users, keyword objects, links, etc, etc.
Unfortunately, we have a huge number of perfectly legitimate users…

Rob

Robert Loomans, Programmer/Analyst robertl@apnic.net
Asia Pacific Network Information Centre (APNIC) Tel: +61-7-3858-3100
http://www.apnic.net Fax: +61-7-3858-3199

smime.p7s (3.67 KB)

`On Mon, May 31, 2004 at 11:59:01AM +1000, Robert Loomans wrote:

I’m not sure if the kernel is using the Linux default 3G/1G
userspace/kernel split or 4G/4G, but the process had topped 2.5G when I
went home… When I got home and checked, the process had aborted with
the message “Out of memory!”

I can easily believe we’re not freeing a cache or that there’s some
other very legitimate (as in definitely broken) memory leak involved.
The migration tool was really intended as one-off tool that would be
sufficient for most everybody. I’m sorry it’s biting you. It may well be
worth prodding at the process to see what specifically is taking up so
much memory and whether that gives you any clues for where to focus your
optimization efforts.

Jesse

Can you give an example of the query to removing orphan’d users? I think I
might be having this issue … we have 265,000+ users in the table and way
less tickets :)----- Original Message -----
From: “Vivek Khera” vivek@khera.org
To: rt-devel@lists.bestpractical.com
Sent: Friday, May 28, 2004 7:36 AM
Subject: Re: [Rt-devel] rt2-to-rt3-1.23 out of memory

On May 28, 2004, at 3:16 AM, Robert Loomans wrote:

When I tried to run rt-2.0-to-dumpfile it ran out of memory. The Linux
machine in question has 4GB and 4GB of swap.

A common failing is understanding that unix like systems are not
originally single-user boxes. That means that just because your box
has X amount of resource it does not mean that any single process can
use that much resource. In particular, check the memory resource
limits of your process that is running the export.

Anyhow, did your export get any tickets out or did it die while
exporting the other info like users list and configs?

I’ve found that it helps to purge the user list of owners who do not
own any tickets (be sure not to delete the root user, or any privileged
user). You can do this with a left join on the tickets table on the
owner id. I do this after purging dead tickets (we get a boatload of
spam to our abuse@ queue).


Rt-devel mailing list
Rt-devel@lists.bestpractical.com
The rt-devel Archives

Can you give an example of the query to removing orphan’d users? I
think I
might be having this issue … we have 265,000+ users in the table and
way
less tickets :slight_smile:

Well, if you assume that users are only useful if they own tickets, do
a left-join on tickets with users on the owner field, then search for
users with ticket id IS NULL. I believe I posted the query to this
list (or the users list) a few months ago when I did the migration.

Vivek Khera, Ph.D.
+1-301-869-4449 x806

smime.p7s (2.42 KB)

Can you give an example of the query to removing orphan’d users? I
think I
might be having this issue … we have 265,000+ users in the table and
way
less tickets :slight_smile:

here. i found it for ya:

http://marc.theaimsgroup.com/?l=rt-users&m=108983329911797&w=2

come to think of it, I should have “ANDed” those delete clauses rather
than “ORed” them as separate queries in case some user owned a ticket
but no attachments or vice versa… or perhaps it is ok. RT had no
errors importing it so I’m assuming it was ok.

I think I was also safe doing that since I had foreign keys defined in
my RT2 db to cascade any such deletes.

Vivek Khera, Ph.D.
+1-301-869-4449 x806

smime.p7s (2.42 KB)

Anyone have a script that will do this. Our DB is mysql and thus doesn’t
support sub-queries.

Sure hope removing orphaned users (probably over 200,000 of them) will help
in the migration process.----- Original Message -----
From: “Vivek Khera” vivek@khera.org
To: rt-devel@lists.bestpractical.com
Sent: Wednesday, October 06, 2004 1:56 PM
Subject: Re: [Rt-devel] rt2-to-rt3-1.23 out of memory


Rt-devel mailing list
Rt-devel@lists.bestpractical.com
The rt-devel Archives