RT + Docker + Performance

1

We have been using RT for a number of years as part of a large (300+ student) engineering design course. One of the things that we use RT for is submitting students’ assignments and their resource materials for midterms and final exams.

Last year - using a Docker image based on Phusion Passenger and an untuned Postgres database - we had a catastrophic failure. Our students were attempting to upload very large attachments (contemporary students don’t seem to understand that files have a size and that photographs can be large) and it was timing out and failing and they were retrying … the net result was that they were filling up our database at something like 17GB per minute.

In our post-mortem we realized that (e.g.) the sessions table was growing faster than everything else and that that the attachments table wasn’t far behind. And that our Postgres installation wasn’t tuned properly at all. However we also saw that the server process also seemed to be spiked at-or-near 100% and that it looked like we weren’t getting multiple threads of execution.

To make matters “worse” we are running everything behind an Apache proxy that is using Shibboleth (SAML) for authentication and are using some custom code to auto-create users based on Shibboleth-set environment variables.

I get the impression from the Dockerfiles that I found that using Phusion may be overkill. I also get the impression that our use case may not be the best match for RT. But we really want to expose our students to a Ticketing system and we’ve had pretty good success with RT so far.

So my question is what the best combination of Docker images, servers, FastCGI, etc., to use given that we are trying to provide a class of 300 impatient and not terribly technically astute students with the ability to upload a rather silly amount of attachments without bringing the server down? This is running on a virtual server so we can do silly things with CPUs and memory, but I’m betting my basic system isn’t the best.

Thanks for any thoughts / help / support / suggestions / paths to pursue.

Jason

2

Have you looked at putting the attachments in the filesystem rather than in the database? And maybe limiting attachment size so that they are forced to load smaller pictures?

3

Do you have a cron job that clears the sessions table out as well?

  perldoc /opt/rt4/sbin/rt-clean-sessions
4

My understanding is that even if you have externalize attachments active, the initial upload is still into the database and you need to run a script through cron to cause the attachments to be exported to the filesystem.

And then if you’re running Postgres you need to run a vacuum all

So we are externalizing, but before that script can kick in the students have tossed both 15GB or so at us.

5

We do run rt-clean-sessions but the minimum time interval is (I believe) 1 hour and that was enough time for them to smash the database (as if I understand correctly the session contains any uploaded attachments, and if the students get frustrated through a timeout and try again then there will be multiple copies of the uploaded files in separate sessions).

6

Alternative idea then: get the students to email their ticket submissions in. That way they won’t be frustrated by timeouts, etc.