Rt-shredder choking on gigantic ticket

RT Users
1

Heya guys!

We’re running rt4 on an Ubuntu 12.04 system. We’re in the process of moving to a new server, and as part of that process I’m trying to us rt-shredder to cull out old resolved/rejected/deleted tickets. I’ve run into an issue where rt-shredder is encountering a ticket with too many dependencies. No problem, I think, and I tweak the DependenciesLimit upwards. Still doesn’t work. So I tweak it up a bit more, and then a bit more, and then a bit more… and finally, with the DependenciesLimit up at 50000, rt-shredder will actually take a crack at it.

Looking at the sqldump file as it comes out, it appears to all be one single “deleted” ticket with a huge mail loop.

However, rt-shredder just churns and churns, gradually consuming all of the memory on the system. I actually went ahead and added 26GB of Swap just to see if it would complete, but eventually it used that up too, and killed the rt-shredder proc in the middle of last night.

The rt-shredder sqldump file from this 10 hour attempt to remove that single problem ticket is 5.4GB.

So, I guess my question is whether anyone has any better suggestion for how to get rid of this ticket?

Thanks,

  •      Luke

Copyright in this message and any attachments remains with us. It is confidential and may be legally privileged. If this message is not intended for you it must not be read, copied or used by you or disclosed to anyone else. Please advise the sender immediately if you have received this message in error. Although this message and any attachments are believed to be free of any virus or other defect that might affect any computer system into which it is received and opened it is the responsibility of the recipient to ensure that it is virus free and no responsibility is accepted by Encore Tickets Limited for any loss or damage in any way arising from its use.

2

So, I guess my question is whether anyone has any better suggestion for how to
get rid of this ticket?

Shred all the transactions, one at a time (in a programmed loop) and
then come back for the ticket itself.

-kevin

3

Hi @Kevin_Falcone and @Luke_Benfey
I can see this post is quite old, but would ask if you can give an example of how to delete a specific number of transactions for a specific ticket.

As far as I have been able to see the Shredder option for transactions only works with files names or file sizes.

We are using RT 4.0.17 and mysql and have a massive ticket caused by an email loop.

Rob

4

Hey all,

I realise this is an old thread, but for posterity’s sake, i’m noting down my experience in case it helps others. We’re running RT 5.0.2 but this should work on 4.X installs.

Problem: An automated process (in-house) that interacts with RT via the API got stuck on a ticket and generated ~8000 transactions and ~1.1 million attachments for that ticket. The ticket no longer loaded in the web interface, even with a 300 second timeout in apache.

Solution: I pulled the transactions attached to the ticket from the database directly, and then wrote a wrapper around rt-shredder that shreds each transaction individually, using the Object plugin. This also cleans up the attachments which are linked to each transaction. I chose not to shred the ticket entirely as it’s attached to our billing system.

example of the shredder command:

rt-shredder --force --plugin 'Objects=Transaction,123456' --sqldump sql/bigticket-123456-tr-123456.sql

SQL queries to find transactions for the ticket:

select id from Transactions where ObjectId = XXX and ObjectType = 'RT::Ticket' and Type = 'EmailRecord';
select id from Transactions where ObjectId = XXX and ObjectType = 'RT::Ticket' and Type = 'Correspond';

excerpt of the python script I used to run rt-shredder (chopped around a bit to remove stuff not relevant to others - test before running on your own environment! no warranty implied or provided!)

import os
import re
import os.path
import datetime
import subprocess

from timeit import default_timer as timer
from time import sleep

basepath = '/some/path/bigticket'
logsdir = os.path.join(basepath, 'shred-logs')


with open('transactions.txt', 'r') as infile:
        transactionlist = infile.readlines()

for tr in transactionlist:
        tr = tr.strip()
        shredcmd = ['/some/path/sbin/rt-shredder', '--force', '--plugin', 'Objects=Transaction,{}'.format(tr), '--sqldump', 'sql/bigticket-123456-tr-{}.sql'.format(tr)]


        today = datetime.datetime.now()


        start = timer()

        proc = subprocess.run(shredcmd, cwd=outpath, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        cmdout = proc.stdout + proc.stderr

        duration = timer() - start
        durstring = "{:.2f}".format(duration)

        outlog = '{}/shred-bigticket-{}.log'.format(logsdir, tr)
        with open(outlog, 'wb+') as l:
                l.write(cmdout)

        logtime = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        print('{} - finished shredding operation in {} seconds. process timeout: {}. log written to {}'.format(logtime, durstring, timedout, outlog))

Cheers, Anthony

1 Like