Apologies in advance since I’m a newbie. A few weeks after I learned of our RT server’s existence, it crashed. After a lot of detective work I was able to determine that the RT server was only running the app and the database is housed on another server. It is running Postgre and the RT version of the database is 3.8.8. Therefore my recovery strategy is to rebuild the front-end server from scratch and do a new installation of the current RT version except that I configured the external database server and then I performed a “make upgrade-database.” There were some warnings:
[9547] [Mon Aug 7 17:39:37 2017] [warning]: You have 15 users(s) with a non-empty value for column ‘ExternalAuthId’. Core RT does not use this column, so perhaps an extension or local modification makes use of it. Please migrate these user values to a custom field or an attribute because this upgrade will drop these columns. at ./etc/upgrade/4.3.0/content line 67. (./etc/upgrade/4.3.0/content:67)
make: *** [upgrade-database] Error 1
[9547] [Mon Aug 7 17:39:37 2017] [warning]: You have 15 users(s) with a non-empty value for column ‘ExternalAuthId’. Core RT does not use this column, so perhaps an extension or local modification makes use of it. Please migrate these user values to a custom field or an attribute because this upgrade will drop these columns. at ./etc/upgrade/4.3.0/content line 67. (./etc/upgrade/4.3.0/content:67)
make: *** [upgrade-database] Error 1
These do not seem unexpected because we were previously using LDAP authentication. However I do not think the warnings stopped the database upgrade. I believe it said it was successful. (Sorry I cannot remember for sure. I have been at this a long time and it is all starting to run together.)
At this point loading the RT login page takes 4-5 minutes. I am unable to login with the default password. In Apache’s error_log I see the following.
[Tue Aug 08 15:19:03 2017] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Tue Aug 08 15:19:03 2017] [notice] Digest: generating secret for digest authentication …
[Tue Aug 08 15:19:03 2017] [notice] Digest: done
[Tue Aug 08 15:19:03 2017] [notice] Apache/2.2.15 (Unix) DAV/2 mod_fcgid/2.3.9 mod_ssl/2.2.15 OpenSSL/1.0.1e-fips mod_perl/2.0.4 Perl/v5.10.1 configured – resuming normal operations
[4442] [Tue Aug 8 19:19:31 2017] [warning]: DBD::Pg::st execute failed: ERROR: relation “customroles” does not exist at /usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 586. (/usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm:586)
[4442] [Tue Aug 8 19:19:31 2017] [warning]: RT::Handle=HASH(0x22b8850) couldn’t execute the query 'SELECT main.* FROM CustomRoles main WHERE (main.Disabled = ‘0’) ’ at /usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 599.
DBIx::SearchBuilder::Handle::SimpleQuery(RT::Handle=HASH(0x22b8850), “SELECT main.* FROM CustomRoles main WHERE (main.Disabled = '”…) called at /usr/local/share/perl5/DBIx/SearchBuilder.pm line 241
DBIx::SearchBuilder::_DoSearch(RT::CustomRoles=HASH(0x54a3818)) called at /opt/rt4/sbin/…/lib/RT/SearchBuilder.pm line 982
RT::SearchBuilder::_DoSearch(RT::CustomRoles=HASH(0x54a3818)) called at /usr/local/share/perl5/DBIx/SearchBuilder.pm line 513
DBIx::SearchBuilder::Next(RT::CustomRoles=HASH(0x54a3818)) called at /opt/rt4/sbin/…/lib/RT/CustomRoles.pm line 104
RT::CustomRoles::RegisterRoles(“RT::CustomRoles”) called at /opt/rt4/sbin/…/lib/RT.pm line 206
RT::Init(“RT”, “Heavy”, 1) called at /opt/rt4/sbin/rt-server.fcgi line 126 (/usr/local/share/perl5/Carp.pm:170)
[4444] [Tue Aug 8 19:19:34 2017] [warning]: DBD::Pg::st execute failed: ERROR: relation “customroles” does not exist at /usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 586. (/usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm:586)
[4444] [Tue Aug 8 19:19:34 2017] [warning]: RT::Handle=HASH(0x1e3d850) couldn’t execute the query 'SELECT main.* FROM CustomRoles main WHERE (main.Disabled = ‘0’) ’ at /usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 599.
DBIx::SearchBuilder::Handle::SimpleQuery(RT::Handle=HASH(0x1e3d850), “SELECT main.* FROM CustomRoles main WHERE (main.Disabled = '”…) called at /usr/local/share/perl5/DBIx/SearchBuilder.pm line 241
DBIx::SearchBuilder::_DoSearch(RT::CustomRoles=HASH(0x5028888)) called at /opt/rt4/sbin/…/lib/RT/SearchBuilder.pm line 982
RT::SearchBuilder::_DoSearch(RT::CustomRoles=HASH(0x5028888)) called at /usr/local/share/perl5/DBIx/SearchBuilder.pm line 513
DBIx::SearchBuilder::Next(RT::CustomRoles=HASH(0x5028888)) called at /opt/rt4/sbin/…/lib/RT/CustomRoles.pm line 104
RT::CustomRoles::RegisterRoles(“RT::CustomRoles”) called at /opt/rt4/sbin/…/lib/RT.pm line 206
RT::Init(“RT”, “Heavy”, 1) called at /opt/rt4/sbin/rt-server.fcgi line 126 (/usr/local/share/perl5/Carp.pm:170)
That would seem to indicate that something is missing from the database structure. As a test I tried the same SELECT query from the psql client and it indeed failed. I am wondering if perhaps the database upgrade failed after all. If so can I re-run “make upgrade-database” without causing any harm?