I’m not sure if I’m more baffled or vexed, but our RT instance has developed an interesting new quirk: I can’t upload any files larger than 40 kilobytes (the upload dies when the temporary file reaches exactly 40,960 bytes) to our production server.
It’s not any of the obvious-to-me stuff:
- RT’s MaxAttachmentSize is well over 40K (128MB, or specifically 134217728 bytes)
- There’s no LimitRequestBody directive in Apache (and just to test I set that to 134217728 bytes too)
- We’re using Apache & mod_perl2 so FcgidMaxRequestLen shouldn’t enter into it (mod_fcgid isn’t even enabled)
- The disk it’s using for scratch space isn’t full (I’ve got 2GB and I’m using 32KB of it)
- There’s no limit on the Apache user that would prevent it from creating a larger file
- RT’s log (cranked up to debug) doesn’t give any errors
- Apache’s error log is silent, the access log shows a HTTP/500
Vexingly I CAN upload the large files on our development server, which gave me some hope: I eyeballed the RT System Configuration pages and nothing jumped out at me besides “The dev server is on RT 5.0.5.” - I went ahead and upgraded Production because that was supposed to happen today anyway, but nope, still dying at 40,960 bytes.
I also diffed the Apache configurations, just a few server name lines different there.
Sooooo I’m stumped - perhaps one of you fine folks can suggest a stupidly obvious thing I might be missing?
Sounds like you’ve already done most of the debugging I’d do! The only thing I can suggest to try is if your server is running SELinux in enforcing mode try (temporarily) disabling it and then see if uploads work. If it does, then it’s time to peer through the audit log to work out what you might need to add to the SELinux policies to stop this and turn the security protection back on. If you’re not running SELinux or that doesn’t work… well I’d be head scratching as much as you at that point.
Oooh, just had a thought: are you using MySQL/mariaDB? If so, check what the max_allowed_packet is set to there. I remember having to tweak that on one of our servers years ago to handle large data blocks being fed into the database.
Nope, Postgres backend (I probably should have mentioned that), and a FreeBSD box so no SELinux screwiness to worry about (and no oddly set resource limits in login.conf - I double checked to make sure Apache’s user is still in the Default class).
It is Most Vexing, and the fact that it’s exactly 40K/40,960 bytes would seem to indicate this is something configured somewhere, but I’ll be damned if I can figure out where! (And what’s really bugging me is I can’t break our dev environment’s instance or see the difference between the two. I know there has to be one, but it’s eluding me.)
My next step might have to be spinning up RT in standalone mode to isolate that part of the system from the Apache/mod_perl2 bits, but if that fixes it I still have no idea where the breakage is coming from.
For anyone else that encounters this particular vexation and has checked all the RT and Apache things:
Do you have a meddlesome IPSEC tunnel between you and the RT server?
If so, Has someone dropped a security policy they shouldn’t have?
Turns out our problem was unrelated to the RT upgrade but rather a change to our VPN backhaul made a couple of days before: An IPSEC policy association for traffic from the datacenter to the office sites got dropped, so traffic to the RT server was being passed properly but traffic from the RT server could get into a broken state (some of it properly encrypting, some of it just getting encapsulated and sent over the tunnel “naked”).
This should have broken a bunch of stuff - and probably did - but RT was the one that made noise first.
1 Like