Something is seriously wrong with my RT deployment. I first noticed the
problem with Ubuntu Hardy 8.04’s RT 3.6 packages; yesterday I upgraded
to 3.8.1rc5 without noticeable improvement. I also tried upgrading the
hardware to a 2.4GHz Celeron with 2Gb of RAM; this didn’t seem to help.
I’m using the postgres backend and there are currently less than 200
tickets in my RT instance (this was a pilot deployment).
This (draft) bash script to rip timesheet data out of RT falls down and
dies:
FIELDS=Ticket,Created,Creator,TimeTaken
rt ls -i "LastUpdated >= '$(date -d "1 month ago")'" |
sed 's|$|/history/type/Correspond|' |
rt show -l -f "$FIELDS" - |
egrep -v '^(#|$|--|RT/.* 200 Ok)' | # Remove lines that aren't data
cut -d: -f 2- | # remove field names
tr -d , | # kludge; comma is the delimiter
paste -sd "${FIELDS//[!,]},\n"
It doesn’t seem to always fail at the same point; right now it’s failing
before it even gets to “rt show”:
$ rt ls -i "LastUpdated >= '$(date -I -d "1 week ago")'"
++ date -I -d '1 week ago'
+ rt ls -i 'LastUpdated >= '\''2008-08-14'\'''
rt: Server error: Internal Server Error (500)
I strongly suspect this is because apache’s perl interface (mod_perl2?)
is severing the connection because it took too long.
Possibly related, there are tens of thousands of lines in syslog of the
form:
Aug 19 16:08:54 oyster RT: We found a merged ticket.3/66
Aug 19 16:09:10 oyster last message repeated 376 times
Aug 19 16:09:11 oyster RT: We found a merged ticket.4/65
Aug 19 16:09:38 oyster last message repeated 686 times
These syslog messages seem to be appear when doing “rt show
ticket/$n/history”, but not “rt ls -i $query”.
The offending process seems to be CPU-bound for some time after rt(1)
fails, for example:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
29445 www-data 20 0 63304 56m 4084 R 96.6 2.8 29:27.53
/usr/bin/perl /usr/share/request-tracker3.8/libexec/mason_handler.fcgi