Rt: Server error: Internal Server Error (500) for rt ls -i "LastUpdated >= '$(date -d "1 month ago")'"

Something is seriously wrong with my RT deployment. I first noticed the
problem with Ubuntu Hardy 8.04’s RT 3.6 packages; yesterday I upgraded
to 3.8.1rc5 without noticeable improvement. I also tried upgrading the
hardware to a 2.4GHz Celeron with 2Gb of RAM; this didn’t seem to help.

I’m using the postgres backend and there are currently less than 200
tickets in my RT instance (this was a pilot deployment).

This (draft) bash script to rip timesheet data out of RT falls down and

rt ls -i "LastUpdated >= '$(date -d "1 month ago")'" |
sed 's|$|/history/type/Correspond|' |
rt show -l -f "$FIELDS" - |
egrep -v '^(#|$|--|RT/.* 200 Ok)' | # Remove lines that aren't data
cut -d: -f 2- |                     # remove field names
tr -d , |                           # kludge; comma is the delimiter
paste -sd "${FIELDS//[!,]},\n"

It doesn’t seem to always fail at the same point; right now it’s failing
before it even gets to “rt show”:

$ rt ls -i "LastUpdated >= '$(date -I -d "1 week ago")'"
++ date -I -d '1 week ago'
+ rt ls -i 'LastUpdated >= '\''2008-08-14'\'''
rt: Server error: Internal Server Error (500)

I strongly suspect this is because apache’s perl interface (mod_perl2?)
is severing the connection because it took too long.

Possibly related, there are tens of thousands of lines in syslog of the

Aug 19 16:08:54 oyster RT: We found a merged ticket.3/66
Aug 19 16:09:10 oyster last message repeated 376 times
Aug 19 16:09:11 oyster RT: We found a merged ticket.4/65
Aug 19 16:09:38 oyster last message repeated 686 times

These syslog messages seem to be appear when doing “rt show
ticket/$n/history”, but not “rt ls -i $query”.

The offending process seems to be CPU-bound for some time after rt(1)
fails, for example:

29445 www-data  20   0 63304  56m 4084 R 96.6  2.8  29:27.53
/usr/bin/perl /usr/share/request-tracker3.8/libexec/mason_handler.fcgi