Hi all -
I am trying to upgrade a fairly large RT installation from RT 4.4.4 to the latest RT 5.0.7. This will be on a new server. The old server was running MariaDB 10.5.22 and the new server is running 11.6.2 (though I have tried this using the same version as well).
When running the “make upgrade-database” command, it always crashes the MariaDB database during the “Processing 4.5.2” step. MariaDB crashes with a core dump and an “unknown” error:
2024-12-03 12:06:15 4 [Warning] InnoDB: 1048576 bytes should have been written at 2501902336 from (unknown file), but got only 458752. Retrying.
2024-12-03 12:06:16 4 [Warning] InnoDB: Retry attempts for writing partial data failed.
2024-12-03 12:06:16 4 [ERROR] [FATAL] InnoDB: Unknown error Temp file write failure
241203 12:06:16 [ERROR] mysqld got signal 6 ;
The query that is causing this issue is:
ALTER TABLE Attachments CONVERT TO CHARACTER SET utf8mb4
From what I can tell, the system is not running out of resources when it crashes with plenty of RAM and disk space available.
Any help would be greatly appreciated as this is a crucial server and I don’t want to be stuck at RT 4.4.4 forever or start with a brand new RT instance. We have over 12 years of history in this RT install. Any pointers would be appreciated.
1 Like
What do you have your max_allowed_packet set to in the MariaDB server.cnf and client.cnf files? We have 64M here. If you don’t have an entry, or only have a small one try adding it to both files, restarting the database server and see if that lets the upgrade complete.
1 Like
Note that for this particular bit of the upgrade “lots of disk space” means “more than the size of the Attachments table”. Your database partition should be less than 50% full when you start this to ensure success.
Going from anything to utf8mb4 means copying the table(s) even if nothing changes in the encoding.
Thank you for the quick replies
max_allowed_packet was set to 32M. I’ve upped it to 64M. I had also upped the tmp_table_size value to 8GB which caused the error to happen still but without crashing MariaDB. I’m now just getting these errors:
mariadbd[286016]: 2024-12-05 9:27:16 4 [Warning] InnoDB: Retry attempts for writing partial data failed.
mariadbd[286016]: 2024-12-05 9:27:16 4 [ERROR] mariadbd: Index for table 'Attachments' is corrupt; try to repair it
But there is no more core dump. Increasing the max_allowed_packet to 64 did not change this unfortunately. I tried upping it all the way to 512M but same result. I played with some other variables such as “innodb_buffer_pool_size”, but so far no success.
I timed the make upgrade-database command and it fails at roughly the same spot no matter what settings I use. On the last attempt, I also got warnings like this
mariadbd[286309]: 2024-12-05 9:42:01 0 [Warning] InnoDB: Total InnoDB FTS size 32262124 for the table `rt5`.`AttachmentsIndex` exceeds the innodb_ft_cache_size 8000000
mariadbd[286309]: 2024-12-05 9:42:09 0 [Warning] InnoDB: Total InnoDB FTS size 155196463 for the table `rt5`.`AttachmentsIndex` exceeds the innodb_ft_cache_size 8000000
I upped the value of innodb_ft_cache_size by a lot, but that does not help. It still fails after about the same amount of time with:
[Warning] InnoDB: 1048576 bytes should have been written at 1076887552 from (unknown file), but got only 458752. Retrying.
[Warning] InnoDB: Retry attempts for writing partial data failed.
[ERROR] mariadbd: Index for table 'Attachments' is corrupt; try to repair it
Please note that I drop and restore the database before each attempt.
The file system should have enough space:
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_rt-var_lib_mysql 71G 26G 46G 36% /var/lib/mysql
Any other suggestions would be very welcome!
Jus to update you all here on the progress in case someone else has the same issue. I manually ran the “ALTER TABLE” command and it failed during the ‘Enabling keys’ phase. After some googling, I increased the tempdir even more and also adjusted
sort_buffer_size = 128M
key_buffer_size = 512M
innodb_buffer_pool_size = 8G
These values may be overkill for production but it actually finished the ALTER TABLE query. I’ll update here if I find out anything else in case someone stumbles onto the same issues.
1 Like