If you don’t have $MaxAttachmentSize set in RT_SiteConfig.pm and neither $TruncateLongAttachments nor $DropLongAttachments is set to anything other than undef, then there shouldn’t be a size constraint in RT.
What web server are you using?
We are using Nginx and we had to set the client_max_body_size on the location block of our web server configuration.
We set ours to 50M:
client_max_body_size 50M;
If you are using Apache there are plenty of posts troubleshooting similar issues. I have one in my notes.
These are the links from my notes: