Post upgrade 3.8.5 to 4.0.12 Global->RT At A Glance error

Hello,

I just completed a migration from MySQL to Oracle as well as an upgrade from 3.8.5 to 4.0.12. So far everything is running well however I have run across one error that occurs when, as super user, I click on the Global–>RT At A Glance link. I receive an error in the WebUI followed by the resulting log file error message:

[Sun May 5 02:14:09 2013] [error]: Can’t call method “Content” on an undefined value at /rt4/share/html/Admin/Global/MyRT.html line 69, line 445.

Stack:
[/rt4/share/html/Admin/Global/MyRT.html:69]
[/rt4/share/html/Admin/autohandler:49]
[/rt4/sbin/…/lib/RT/Interface/Web.pm:634]
[/rt4/sbin/…/lib/RT/Interface/Web.pm:335]
[/rt4/share/html/autohandler:53] (/rt4/sbin/…/lib/RT/Interface/Web/Handler.pm:208)

This is running on OEL 6.4 x64, Oracle 11gR2 DB, httpd v.2.2.15 and perl v.5.10.1. Any help on where I can begin troubleshooting this would be wonderful.

Cheers,

~Pete

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

I just completed a migration from MySQL to Oracle as well as an
upgrade from 3.8.5 to 4.0.12. So far everything is running well
however I have run across one error that occurs when, as super user,
I click on the Global–>RT At A Glance link. I receive an error in
the WebUI followed by the resulting log file error message:

[Sun May 5 02:14:09 2013] [error]: Can’t call method “Content” on an
undefined value at /rt4/share/html/Admin/Global/MyRT.html line 69,
line 445.

Stack: [/rt4/share/html/Admin/Global/MyRT.html:69]
[/rt4/share/html/Admin/autohandler:49]
[/rt4/sbin/…/lib/RT/Interface/Web.pm:634]
[/rt4/sbin/…/lib/RT/Interface/Web.pm:335]
[/rt4/share/html/autohandler:53]
(/rt4/sbin/…/lib/RT/Interface/Web/Handler.pm:208)

I can’t reproduce this problem and looking at the code don’t see how it
would happen. Do you have any local customizations?

Can you send us the output of running this SQL query?

select id, Name, Description, ObjectType, ObjectId from Attributes where
ObjectType=‘RT::System’;

Could you also navigate to Configuration → Tools → System Config in
RT, save that page as HTML, and send the file? Make sure any sensitive
data is redacted first.

Thanks for the information, notes below.

Do you have any local customizations?

None that I’m aware of. Only plugins I’ve currently go in place are:
RT::Authen::ExternalAuth RT::Extension::ActivityReports

Can you send us the output of running this SQL query? select
id, Name, Description, ObjectType, ObjectId from Attributes
where ObjectType=‘RT::System’;

Could you also navigate to Configuration → Tools → System
Config in RT, save that page as HTML, and send the file? Make
sure any sensitive data is redacted first.

Both the Select query and the system config HTML page and files have
been included in the attached ZIP file.

Your system config page and SQL results reveal that you have duplicate
copies (3 total of each record) of certain global data. This suggests
your migration from MySQL to Oracle was flawed. How did you migrate,
exactly?

Your database may contain duplicate internal records in other areas too,
which are likely to lead to more problems (and potentially very subtle
ones). What do these queries return?

select id, Name, Created, LastUpdated from Users where Name in
(‘RT_System’, ‘Nobody’, ‘root’);

select id, Domain, Type, Created, LastUpdated from Groups where Type in
(‘Everyone’, ‘Privileged’, ‘Unprivileged’);

I’m unsure of how to reply to this and have it also update the forum
thread so I’ve omitted the rt-users@lists.bestpractical.com address
when sending the e-mail.

You reply by sending to rt-users@lists.bestpractical.com. That sends to
the mailing list, and I’ll read it since I’m subscribed.

Do you have any local customizations?

None that I’m aware of. Only plugins I’ve currently go in place are: RT::Authen::ExternalAuth RT::Extension::ActivityReports

Can you send us the output of running this SQL query? select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType=‘RT::System’;

Could you also navigate to Configuration → Tools → System Config in RT, save that page as HTML, and send the file? Make sure any sensitive data is redacted first.

Both the Select query and the system config HTML page and files have been included in the attached ZIP file.

I’m unsure of how to reply to this and have it also update the forum thread so I’ve omitted the rt-users@lists.bestpractical.com address when sending the e-mail.

Thank you, Thomas.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

System Configuration.zip (224 KB)

Your system config page and SQL results reveal that you have duplicate
copies (3 total of each record) of certain global data. This suggests
your migration from MySQL to Oracle was flawed. How did you migrate,
exactly?

As follows:

  1. Created new OEL 6.4 x64 server running Apache/Mysqld from distrib packages. Additionally installed a newer version of Perl and fought with character set issues between old and new server configurations.
  2. mysqldump’d data from old server and imported into an empty rt4 mysql database on the new server
  3. Upgraded mysql RT3 data to RT4 using make upgrade-database (4.0.12).
  4. Prepped Oracle schema using RT 4.0.12 configuration and then truncated all tables from new schema as well as disabled constraints as prep for import.
  5. Ran a perl script obtained from a user from this thread: http://requesttracker.8502.n7.nabble.com/mysql-to-oracle-migration-td34166.html. Script pretty much copies records from their mysql tables into their respective Oracle tables
  6. Re-created the Oracle sequences after import.
  7. Re-enabled Oracle table constraints.

I could forward you a PDF of the actual upgrade steps however it’s not for public digestion since it’s replete with confidential data that’d be a PIA to scrub.

What do these queries return?

select id, Name, Created, LastUpdated from Users where Name in (‘RT_System’, ‘Nobody’, ‘root’);

ID NAME,CREATED,LASTUPDATED
10 Nobody,6-Sep-05,6-Sep-05
1 RT_System,6-Sep-05,6-Sep-05
12 root,6-Sep-05, 4-May-13

select id, Domain, Type, Created, LastUpdated from Groups where Type in (‘Everyone’, ‘Privileged’, ‘Unprivileged’);

ID DOMAIN, TYPE,CREATED,LASTUPDATED
3 SystemInternal,Everyone,
4 SystemInternal,Privileged,
5 SystemInternal,Unprivileged,

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

This might narrow down the issue…I just noticed another thread titled “RT at a Glance error after upgrade from 4.011 to 4.0.12” that also states the issue at hand.

Just to be sure, I repeated the import on our RTDEV server using v4.0.11 of RT (using MySQL only, not Oracle) and confirmed that the Global RT At A Glance error does ~not~ occur.

I will proceed with the migration to Oracle using 4.0.11 only and update this thread with the results.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

Just performed the RT migration from MySQL to Oracle using just 4.0.11 and re-tested the Global RT At A Glance error without issue.

I will now repeat the 4.0.11 to 4.0.12 upgrade using just Oracle and update with results.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

The 4.0.11 to 4.0.12 Oracle upgrade completed and this RT At A Glance issue still did not reappear. This was all done on our Dev system, however. The Prod OS/config is a clone of the so I’m at a loss as to what the difference would be that’s causing this to occur.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

The 4.0.11 to 4.0.12 Oracle upgrade completed and this RT At A Glance
issue still did not reappear. This was all done on our Dev system,
however. The Prod OS/config is a clone of the so I’m at a loss as to
what the difference would be that’s causing this to occur.

That result is quite strange, but suggests a database inconsistency.
Have you run rt-validator?
http://bestpractical.com/rt/docs/latest/rt-validator

What are the output of the 3 SQL queries I sent you earlier on this
newly upgraded dev instance?

As of yet, I haven’t tracked down the problem locally. By my count,
there are 3 reports of similar issues on rt-users. At least two of
three migrated database types, so my suspicions are there right now but
I have no proof/explanation. I hope to have more time to debug this soon.

Hi Thomas,

Have you run rt-validator?

Log file attached. All references are likely due to the deletion of old user records directly from mysql which we now know is not recommended.

What are the output of the 3 SQL queries I sent you earlier on this newly upgraded dev instance?

All from the upgrade Oracle RT DEV instance:

select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType=‘RT::System’;

137 Search - My Tickets [_1] highest priority tickets I own RT::System 1
138 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
139 HomepageSettings HomepageSettings RT::System 1
140 Search - My Tickets [_1] highest priority tickets I own RT::System 1
141 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
142 HomepageSettings HomepageSettings RT::System 1
153 Search - My Tickets [_1] highest priority tickets I own RT::System 1
154 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
155 HomepageSettings HomepageSettings RT::System 1
300 Search - Bookmarked Tickets Bookmarked Tickets RT::System 1
318 BrandedSubjectTag Queue id => subject tag map RT::System 1
3021 SavedSearch 10 highest priority tickets I own RT::System 1
3842 QueueCacheNeedsUpdate 0 RT::System 1

select id, Name, Created, LastUpdated from Users where Name in (‘RT_System’, ‘Nobody’, ‘root’);

10 Nobody 06-SEP-05 06-SEP-05
1 RT_System 06-SEP-05 06-SEP-05
12 root 06-SEP-05 15-MAY-13

select id, Domain, Type, Created, LastUpdated from Groups where Type in (‘Everyone’, ‘Privileged’, ‘Unprivileged’);

3 SystemInternal Everyone
4 SystemInternal Privileged
5 SystemInternal Unprivileged

As a note, the duplicate “Search - Unowned Tickets” duplicates are likely a remnant of trying to fix a link issue in the previous 3.8.5 instance. Relevant thread here: Carbon60: Managed Cloud Services It’s a side issue that was never satisfactorily resolved and would be preferable to clean out the duplicates.

Here’s the same SQL queries output from the original MySQL 3.8.5 database:

mysql> select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType=‘RT::System’;
| id | Name | Description | ObjectType | ObjectId |
| 137 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 138 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 139 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 140 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 141 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 142 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 153 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 154 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 155 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 300 | Search - Bookmarked Tickets | Bookmarked Tickets | RT::System | 1 |
| 318 | BrandedSubjectTag | Queue id => subject tag map | RT::System | 1 |
| 3021 | SavedSearch | 10 highest priority tickets I own | RT::System | 1 |
12 rows in set (0.00 sec)

mysql> select id, Name, Created, LastUpdated from Users where Name in (‘RT_System’, ‘Nobody’, ‘root’);
| id | Name | Created | LastUpdated |
| 10 | Nobody | 2005-09-06 20:48:58 | 2005-09-06 20:48:58 |
| 12 | root | 2005-09-06 20:48:58 | 2013-04-09 23:32:14 |
| 1 | RT_System | 2005-09-06 20:48:58 | 2005-09-06 20:48:58 |
3 rows in set (0.00 sec)

mysql> select id, Domain, Type from Groups where Type in (‘Everyone’,‘Privileged’,‘Unprivileged’);
| id | Domain | Type |
| 3 | SystemInternal | Everyone |
| 4 | SystemInternal | Privileged |
| 5 | SystemInternal | Unprivileged |
3 rows in set (0.04 sec)

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

rt-validator log of shame.txt (1.06 MB)

Hi Thomas,

The guilt of leaving you with a rt-validator log file full of warnings didn’t sit well with me. I went ahead and re-added the removed user records. Once done the rt-validator --check output was empty.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

Looks like the mail filter stopped this one from going through due to the attached log file. Re-forwarding minus log file.

Also of note, yesterday evening I brought our RTPROD server offline for a reboot. When the OS was back online I backed up the existing RTPROD installation directory and re-ran the configuration to re-create the install directory. I then copied over our RT_SiteConfig file and custom log and started the web service back up. I re-tested the Global At A Glance setting and found that the issue was no longer appearing.

So…nothing done on the database end (other than re-instating those user records). For this particular problem it seems that me laying down a new install base somehow corrected the original issue.

~Pete_JibeFrom: Pete Beebe
Sent: Thursday, May 16, 2013 10:55 AM
To: ‘rt-users@lists.bestpractical.com’
Subject: RE: [rt-users] Post upgrade 3.8.5 to 4.0.12 Global->RT At A Glance error

Hi Thomas,

Have you run rt-validator?

Log file attached. All references are likely due to the deletion of old user records directly from mysql which we now know is not recommended.

What are the output of the 3 SQL queries I sent you earlier on this newly upgraded dev instance?

All from the upgrade Oracle RT DEV instance:

select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType=‘RT::System’;

137 Search - My Tickets [_1] highest priority tickets I own RT::System 1
138 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
139 HomepageSettings HomepageSettings RT::System 1
140 Search - My Tickets [_1] highest priority tickets I own RT::System 1
141 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
142 HomepageSettings HomepageSettings RT::System 1
153 Search - My Tickets [_1] highest priority tickets I own RT::System 1
154 Search - Unowned Tickets [_1] newest unowned tickets RT::System 1
155 HomepageSettings HomepageSettings RT::System 1
300 Search - Bookmarked Tickets Bookmarked Tickets RT::System 1
318 BrandedSubjectTag Queue id => subject tag map RT::System 1
3021 SavedSearch 10 highest priority tickets I own RT::System 1
3842 QueueCacheNeedsUpdate 0 RT::System 1

select id, Name, Created, LastUpdated from Users where Name in (‘RT_System’, ‘Nobody’, ‘root’);

10 Nobody 06-SEP-05 06-SEP-05
1 RT_System 06-SEP-05 06-SEP-05
12 root 06-SEP-05 15-MAY-13

select id, Domain, Type, Created, LastUpdated from Groups where Type in (‘Everyone’, ‘Privileged’, ‘Unprivileged’);

3 SystemInternal Everyone
4 SystemInternal Privileged
5 SystemInternal Unprivileged

As a note, the duplicate “Search - Unowned Tickets” duplicates are likely a remnant of trying to fix a link issue in the previous 3.8.5 instance. Relevant thread here: Carbon60: Managed Cloud Services It’s a side issue that was never satisfactorily resolved and would be preferable to clean out the duplicates.

Here’s the same SQL queries output from the original MySQL 3.8.5 database:

mysql> select id, Name, Description, ObjectType, ObjectId from Attributes where ObjectType=‘RT::System’;
| id | Name | Description | ObjectType | ObjectId |
| 137 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 138 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 139 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 140 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 141 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 142 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 153 | Search - My Tickets | [_1] highest priority tickets I own | RT::System | 1 |
| 154 | Search - Unowned Tickets | [_1] newest unowned tickets | RT::System | 1 |
| 155 | HomepageSettings | HomepageSettings | RT::System | 1 |
| 300 | Search - Bookmarked Tickets | Bookmarked Tickets | RT::System | 1 |
| 318 | BrandedSubjectTag | Queue id => subject tag map | RT::System | 1 |
| 3021 | SavedSearch | 10 highest priority tickets I own | RT::System | 1 |
12 rows in set (0.00 sec)

mysql> select id, Name, Created, LastUpdated from Users where Name in (‘RT_System’, ‘Nobody’, ‘root’);
| id | Name | Created | LastUpdated |
| 10 | Nobody | 2005-09-06 20:48:58 | 2005-09-06 20:48:58 |
| 12 | root | 2005-09-06 20:48:58 | 2013-04-09 23:32:14 |
| 1 | RT_System | 2005-09-06 20:48:58 | 2005-09-06 20:48:58 |
3 rows in set (0.00 sec)

mysql> select id, Domain, Type from Groups where Type in (‘Everyone’,‘Privileged’,‘Unprivileged’);
| id | Domain | Type |
| 3 | SystemInternal | Everyone |
| 4 | SystemInternal | Privileged |
| 5 | SystemInternal | Unprivileged |
3 rows in set (0.04 sec)

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

Hi Thomas,

Looks like the mail filter stopped this one from going through due to
the attached log file. Re-forwarding minus log file.

It was briefly stopped, yes, but I let it through manually. There’s a
size limit on posts at which point they require manual moderation.

Also of note, yesterday evening I brought our RTPROD server offline
for a reboot. When the OS was back online I backed up the existing
RTPROD installation directory and re-ran the configuration to
re-create the install directory. I then copied over our
RT_SiteConfig file and custom log and started the web service back
up. I re-tested the Global At A Glance setting and found that the
issue was no longer appearing.

So…nothing done on the database end (other than re-instating those
user records). For this particular problem it seems that me laying
down a new install base somehow corrected the original issue.

Out of curiosity, how did you reinstate the user records? The error was
still occurring after you’d reinstated them, but before you reinstalled,
right?

I’m surprised that a new install would correct the issue, but it is
possible. Basically a component was returning an unexpected empty
value. Can you run diff -ruN against the two install trees (the one
you backed up and the one currently in use)? Alternatively, sent two
tarballs over and I can do it.

Thanks for continuing to dig at this and all the info you’ve provided.

Thomas

Hi Thomas,

Out of curiosity, how did you reinstate the user records?

On the original RT 3.8.5 server I had a couple of mysql dumps available with the previously deleted records still intact. On the original server I copied one of the mysql backups to the RTDEV server, imported it into an empty MySQL rt database and then ran the upgrade to 4.0.12 followed by the migration to the Oracle schema. I then extracted the specific deleted records from the RTDEV Oracle schema and inserted them into the RTPROD User table followed by re-generating the User table sequence. If I was more savvy I probably could have just reconstructed the insert queries manually but the above method worked out well enough, just took a bit more time.

The error was still occurring after you’d reinstated them, but before you reinstalled, right?

This is correct. I re-instated the deleted user records into RTPROD and the Global RT At A Glance error was still occurring. It was only when I rebooted the RTPROD server and re-ran the base directory installation that the error cleared. What I didn’t do which I should have done was re-test after the RTPROD reboot and before running the new base directory installation.

Can you run diff -ruN against the two install trees (the one you backed up and the one currently in use)?

Zip file is 190KB in size. I will be sending it separate from this reply. I cut out the session_data diff output to trim down the original log file (4MB). If you prefer that this additional diff info be included I can simply send you the zipped 4MB log file.

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

E-mailing 190KB Diff.zip file for review (minus session_data diff output in order to trim down log file).

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.

diff.zip (190 KB)

Can you run diff -ruN against the two install trees (the one
you backed up and the one currently in use)?

Zip file is 190KB in size. I will be sending it separate from this
reply. I cut out the session_data diff output to trim down the
original log file (4MB). If you prefer that this additional diff
info be included I can simply send you the zipped 4MB log file.

The session data isn’t necessary (and shouldn’t be published publicly
either; sessions contain private data).

I filtered out irrelevant files (the new RT-Authen-ExternalAuth install,
Mason cache files, documentation files (man and pod)) from the diff you
sent, and it left no significant changes.

Frustratingly, I still can’t reproduce the error locally on a handful of
long-lived RT instances all of which have been upgraded through 4.0.12.

Thomas

Frustratingly, I still can’t reproduce the error locally on a handful of long-lived RT instances all of which have been upgraded through 4.0.12.

Hi Thomas,

Thank you for spending the time looking into this. Seeing as how my last step of laying down a new install base corrected the matter on our RTPROD server I’m going to shelve this and call it good. I wish I could’ve narrowed it down to a specific cause and resolution however due to the number of higher priority projects screaming for attention I won’t be able to put much more time into investigating this further.

If for some reason you need/require additional information out of what I’ve provided so far, please let me know and I’d be happy to oblige when possible.

Cheers,

~Pete_Jibe

Disclaimer: This electronic message may contain information that is Confidential or legally privileged. It is intended only for the use of the individual(s) and entity named in the message. If you are not an intended recipient of this message, please notify the sender immediately and delete the material from your computer. Do not deliver, distribute or copy this message and do not disclose its contents or take any action in reliance on the information it contains.