Perl script issue after upgrading from 4.4.3 to 5.0.6

Use case:
Utilize the REST 2.0 API to query each log and write it to a file where I pull it into Splunk and create reports and dashboards.

Changes:
We migrated from RHEL6 something to RHEL8.9 and did a fresh install of RT5 and imported the database.

Now when running the perl script I get this output:
Querying custom fields
Plan on querying 34 tickets
Open our log file
Grabbing data for ticket #146523
Not a HASH reference at /opt/dump_rt_tickets/rt_queue1_incremental_splunk.pl line 99.

I have no idea how to write perl. This was written by someone at BP years ago (and I’ve very grateful for their help!). If anyone can help, I appreciate it. ChatGPT was not helpful :confused:

FYI, Line 99 is
my %cfs = %{$ticketObj->{‘CustomFields’}};

#!/usr/bin/perl
use strict;
use Encode;
use warnings;

use HTTP::Request::Common qw(POST);
use LWP::UserAgent;
use JSON qw/decode_json encode_json/;

=head1

We start by performing a Tickets query using ticket SQL, in this example
we are just searching the queue 'General'. Next we need to grab our
custom field ID's based off of their name so that we know which value
correlates to which custom field.

Then we can loop through our tickets and get the desired values from each ticket.

We then output the data to a file.

=cut

my $ua = LWP::UserAgent->new;
$ua->timeout(15);



# CHANGE THESE FOUR LINES, note keep the '&page=' portion at the end of the query.
my $url  = 'https://rt.redacted.com/REST/2.0/';
$ua->default_header( Authorization => 'token redacted' );
# Add a list of Customfields here
my @cf_names = ('Determination', 'Method', 'Analyst Name', 'Feedback');
my $query = "tickets?query=Queue = 'Redacted' AND LastUpdated >= '5 minutes ago'&page=";
#my $query ="tickets?query=LastUpdated > '2024-04-24T15:46:04Z' AND Queue = 'Redacted'&page=";


my %customfields;
print "Querying custom fields\n";
foreach my $cf_name (@cf_names) {
    my $content = [{field => 'Name', value => $cf_name}];
    my $cf_query_result = decode_json($ua->post( $url . 'customfields', Content => encode_json($content) )->decoded_content);

    if ( ref $cf_query_result ne 'HASH' || not $cf_query_result->{'count'} ) {
         print "[ERROR] No data was received for custom field: " . $cf_name . ", this could be due to RT rights.\n";
         next;
    }

    # Store our ID and name
    $customfields{$cf_name} = $cf_query_result->{'items'}[0]{id};
}

# Caching our queues by ID is more efficient than checking for every ticket
my $queues = decode_json($ua->get($url . 'queues/all' )->decoded_content);
my %queues;
for (@{$queues->{'items'}}) {
    my $queueObj = $ua->get( $_->{'_url'} )->decoded_content;
    $queueObj = decode_json($queueObj) if $queueObj;

    $queues{$queueObj->{'id'}} = $queueObj->{'Name'};
}

# This will most likely need to be changed to avoid timeout, an easy solution would be to split the
# query by 'created' dates so that one days worth of tickets are pulled at a time.
my $tickets_query = decode_json($ua->get($url . $query )->decoded_content);
print "Plan on querying " . $tickets_query->{'total'} . " tickets \n";

print "Open our log file\n";
my $filename = '/opt/dump_rt_tickets/redacted_incremental_dump.txt';
open(my $fh, '>', $filename) or die "Could not open file '$filename' $!";

my $count = 1;
my $page = decode_json($ua->get($url . $query . $count )->decoded_content);
my $final_output;
while ( $page->{'count'} ) {
    my $data_for_file;
    foreach my $ticket (sort {$a->{'id'} <=> $b->{'id'}} @{$page->{items}}) {
        my $ret;
        my $ticketObj = $ua->get($url . "ticket/" . $ticket->{'id'} )->decoded_content;
        $ticketObj = decode_json($ticketObj) if $ticketObj;

        if ( ref $ticketObj ne 'HASH' ) {
            print "[ERROR] No data was received for ticket: " . $ticket->{'id'} . "\n";
            next;
        }
        print "Grabbing data for ticket #" . $ticket->{'id'} . "\n";

        for (qw/id Subject Status Queue Owner Requestor Creator Created Resolved LastUpdated/) {
            if ( $_ eq 'Queue' ) {
                $ret .= "\"$_\":\"".$queues{$ticketObj->{$_}->{'id'}}."\"\n";
            } elsif ( ref $ticketObj->{$_} eq 'ARRAY' ) {
                $ret .= "\"$_\":".join(", ", map {'"'.($_->{'id'}||'-').'"'} @{$ticketObj->{$_}})."\n";
            } elsif ( ref $ticketObj->{$_} eq 'HASH' ) {
                $ret .= "\"$_\":\"".($ticketObj->{$_}->{'id'}||'-')."\"\n";
            }else {
                $ret .= "\"$_\":\"".($ticketObj->{$_}||'-')."\"\n";
            }
        }

        my %cfs   = %{$ticketObj->{'CustomFields'}};
        foreach my $cf (@cf_names) {
            my @cf_values = @{$cfs{$customfields{$cf}}} ? map{"\"$_\""} @{$cfs{$customfields{$cf}}} : ('"-"');
            $ret .= "\"$cf\":".join(", ", @cf_values)."\n";
        }
        $final_output .= $ret;
    }
    $count++;
    $page = decode_json($ua->get($url . $query . $count )->decoded_content)
}
print $fh encode('UTF-8', "$final_output");
close $fh;```

I’d probably start by sticking some debugging in just before line 99 to dump out what the contents of $ticketObj is. It sounds like it doesn’t have any custom fields in it.

ChatGPT helped me with that bit :wink:
Here is a snippet of the output when I add DataDumper and tell the script to print $ticketObj just before line 99

                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/56',
                               'ref' => 'customfield',
                               'name' => 'Determination',
                               'type' => 'customfield',
                               'id' => '56'
                             },

It appears it sees and identifies the custom fields… Any suggestion on where to go next? Please keep in mind I have little-to-no perl experience.

I did run the perl script on the old server with the same debug options and compared the output on the new server and setup:

RT 4.0.3 output

                    {
                               'ref' => 'customfield',
                               'id' => '49',
                               'type' => 'customfield',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/49'
                             },

RT 5.0.6 output for the same ticket, same custom field

                    {
                               'type' => 'customfield',
                               'ref' => 'customfield',
                               'id' => '49',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/49',
                               'name' => 'Feedback'
                             },

It appears RT5 has added the “name” value to the REST output? Just spitballing here, I really have no idea what I’m talking about.

Thanks again.

I can’t tell from the snippet of the Data::Dumper output, but does the $TicketObj have a key called CustomFields? That key is what the code is looking for on line 99.

Sorry, not it does not.

Here is the full, redacted output from the RT4.0.3 instance.

Grabbing data for ticket #146571
Content of ticketObj: $VAR1 = {
          'TimeEstimated' => 0,
          'Status' => 'new',
          '_hyperlinks' => [
                             {
                               'ref' => 'self',
                               'type' => 'ticket',
                               'id' => 146571,
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                             },
                             {
                               'ref' => 'customfield',
                               'id' => '56',
                               'type' => 'customfield',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/56'
                             },
                             {
                               'ref' => 'customfield',
                               'id' => '58',
                               'type' => 'customfield',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/58'
                             },
                             {
                               'ref' => 'customfield',
                               'id' => '50',
                               'type' => 'customfield',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/50'
                             },
                             {
                               'ref' => 'customfield',
                               'id' => '49',
                               'type' => 'customfield',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/49'
                             },
                             {
                               'ref' => 'history',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/history'
                             },
                             {
                               'ref' => 'correspond',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/correspond'
                             },
                             {
                               'ref' => 'comment',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/comment'
                             },
                             {
                               'to' => 'Investigation Underway',
                               'from' => '*',
                               'ref' => 'lifecycle',
                               'label' => 'Launch Investigation',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                             },
                             {
                               'to' => 'rejected',
                               'from' => '*',
                               'ref' => 'lifecycle',
                               'label' => 'Reject',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                             },
                             {
                               'to' => 'deleted',
                               'from' => '*',
                               'ref' => 'lifecycle',
                               'label' => 'Delete',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                             }
                           ],
          'Queue' => {
                       'type' => 'queue',
                       'id' => '9',
                       '_url' => 'https://rt.redacted.com/REST/2.0/queue/9'
                     },
          'AdminCc' => [],
          'InitialPriority' => 0,
          'Type' => 'ticket',
          'Started' => '1970-01-01T00:00:00Z',
          'Starts' => '1970-01-01T00:00:00Z',
          'TimeWorked' => 0,
          'id' => 146571,
          'LastUpdated' => '2024-05-09T17:20:05Z',
          'CustomFields' => {
                              '50' => [],
                              '56' => [],
                              '49' => [],
                              '58' => []
                            },
          'Requestor' => [
                           {
                             'type' => 'user',
                             'id' => 'redacted@redacted.com',
                             '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com'
                           }
                         ],
          'Cc' => [],
          'Subject' => 'Fwd: Last chance to recover funds from Visa Mastercard Settlement',
          'FinalPriority' => 0,
          'TimeLeft' => 0,
          'Creator' => {
                         'type' => 'user',
                         'id' => 'redacted@redacted.com',
                         '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com'
                       },
          'Owner' => {
                       'type' => 'user',
                       'id' => 'Nobody',
                       '_url' => 'https://rt.redacted.com/REST/2.0/user/Nobody'
                     },
          'LastUpdatedBy' => {
                               'type' => 'user',
                               'id' => 'redacted@redacted.com',
                               '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com'
                             },
          'EffectiveId' => {
                             'type' => 'ticket',
                             'id' => '146571',
                             '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                           },
          'Resolved' => '1970-01-01T00:00:00Z',
          'Created' => '2024-05-09T17:20:03Z',
          'Priority' => 0,
          'Due' => '1970-01-01T00:00:00Z'
        };

And here is the redacted output for the same ticket but running RT 5.0.6

Grabbing data for ticket #146571
Content of ticketObj: $VAR1 = {
          'Told' => '2024-05-14T16:46:10Z',
          'TimeLeft' => '0',
          'Type' => 'ticket',
          'Subject' => 'Fwd: Last chance to recover funds from Visa Mastercard Settlement',
          'Creator' => {
                         'id' => 'redacted@redacted.com',
                         'type' => 'user',
                         '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com'
                       },
          'Owner' => {
                       '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com',
                       'type' => 'user',
                       'id' => 'redacted@redacted.com'
                     },
          'Due' => '1970-01-01T00:00:00Z',
          'InitialPriority' => '0',
          'TimeEstimated' => '0',
          'Requestor' => [
                           {
                             '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com',
                             'id' => 'redacted@redacted.com',
                             'type' => 'user'
                           }
                         ],
          'Resolved' => '2024-05-14T16:46:10Z',
          'AdminCc' => [],
          'CustomFields' => [
                              {
                                'type' => 'customfield',
                                'id' => '49',
                                '_url' => 'https://rt.redacted.com/REST/2.0/customfield/49',
                                'values' => [],
                                'name' => 'Feedback'
                              },
                              {
                                '_url' => 'https://rt.redacted.com/REST/2.0/customfield/56',
                                'name' => 'Determination',
                                'values' => [
                                              'Non-malicious'
                                            ],
                                'type' => 'customfield',
                                'id' => '56'
                              },
                              {
                                'id' => '58',
                                'type' => 'customfield',
                                'values' => [],
                                'name' => 'Method',
                                '_url' => 'https://rt.redacted.com/REST/2.0/customfield/58'
                              },
                              {
                                'name' => 'Analyst Name',
                                'values' => [],
                                '_url' => 'https://rt.redacted.com/REST/2.0/customfield/50',
                                'id' => '50',
                                'type' => 'customfield'
                              }
                            ],
          'EffectiveId' => {
                             '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571',
                             'id' => '146571',
                             'type' => 'ticket'
                           },
          '_hyperlinks' => [
                             {
                               'id' => 146571,
                               'ref' => 'self',
                               'type' => 'ticket',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571'
                             },
                             {
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/56',
                               'name' => 'Determination',
                               'ref' => 'customfield',
                               'type' => 'customfield',
                               'id' => '56'
                             },
                             {
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/58',
                               'name' => 'Method',
                               'type' => 'customfield',
                               'ref' => 'customfield',
                               'id' => '58'
                             },
                             {
                               'type' => 'customfield',
                               'ref' => 'customfield',
                               'id' => '50',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/50',
                               'name' => 'Analyst Name'
                             },
                             {
                               'type' => 'customfield',
                               'ref' => 'customfield',
                               'id' => '49',
                               '_url' => 'https://rt.redacted.com/REST/2.0/customfield/49',
                               'name' => 'Feedback'
                             },
                             {
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/history',
                               'ref' => 'history'
                             },
                             {
                               'ref' => 'correspond',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/correspond'
                             },
                             {
                               'ref' => 'comment',
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571/comment'
                             },
                             {
                               '_url' => 'https://rt.redacted.com/REST/2.0/ticket/146571',
                               'from' => '*',
                               'to' => 'Investigation Underway',
                               'ref' => 'lifecycle',
                               'label' => 'Launch Investigation'
                             }
                           ],
          'Cc' => [],
          'TimeWorked' => '0',
          'id' => 146571,
          'LastUpdatedBy' => {
                               '_url' => 'https://rt.redacted.com/REST/2.0/user/redacted@redacted.com',
                               'id' => 'redacted@redacted.com',
                               'type' => 'user'
                             },
          'Starts' => '1970-01-01T00:00:00Z',
          'Status' => 'Resolved',
          'Created' => '2024-05-09T17:20:03Z',
          'Queue' => {
                       '_url' => 'https://rt.redacted.com/REST/2.0/queue/9',
                       'id' => '9',
                       'type' => 'queue'
                     },
          'FinalPriority' => '0',
          'Started' => '1970-01-01T00:00:00Z',
          'Priority' => '0',
          'LastUpdated' => '2024-05-14T16:46:12Z'
        };

There is actually a lot different between these two results. I didn’t expect that…

OK, looks like in 4.0.3 the CustomFields key was pointing to a hash with just custom field IDs as subkeys. In 5.0.6 the CustomFields isn’t a hash any more - it’s now an ordered array of hashes, where each hash is the full details of the custom fields. So that explains the error on line 99 - it’s now getting an array reference rather than a hash reference. But then the whole foreach loop after it is now wrong too because its try to process a hash it doesn’t have.

I suspect the 5.0.6 REST2.0 ticket pull may now contain all the information you need, so the earlier REST2.0 calls to get custom field IDs also aren’t needed. So I’m afraid this looks like a script rewrite.

Now I’m writing this “blind” without your system to test it on, but you could make a safe backup copy of this script and then (and only then!) change the lines:

        my %cfs   = %{$ticketObj->{'CustomFields'}};
        foreach my $cf (@cf_names) {
            my @cf_values = @{$cfs{$customfields{$cf}}} ? map{"\"$_\""} @{$cfs{$customfields{$cf}}} : ('"-"');
            $ret .= "\"$cf\":".join(", ", @cf_values)."\n";
        }

to be something like:

my @cfs = @{$ticketObj->{'CustomFields'}};
foreach my $thisCF (@cfs) {
   foreach my $wantedCFName (@cf_names) {
      if($thisCF->{'name'} eq $wantedCFName) {
         my @cf_values = @{$thisCF->{'values'}} ? map{"\"$_\""} @{$thisCF->{'values'}} : ('"-"');
         $ret .= '"'.$thisCF->{'name'}.'":'.join(", ", @cf_values)."\n";
      }
   }
}

No promises though!

That did it! Thank you so much!