Slow Ticket History (from rt-users)

Hi chaps,

Some of you may not have seen the discussion about slow display of tickets with large numbers of transactions, on rt-users.

I’ve found a fairly significant performance issue in the code, which causes it to run O(N^2) or possibly worse, with respect to the number of transactions in the ticket. My patch converts this to O(N) (I think!).

It’s certainly a lot faster than it was, on my RT setup. Here’s my post to rt-users - I’d appreciate comments on my changes in case I’ve accidentally done something which will toast my data, but I don’t think I have, and I’ve been testing this all day.On 6 Sep 2010, at 5:54 pm, rt-users-request@lists.bestpractical.com wrote:

So far we’ve tried installing RT on different hardware, both 32 and 64bit versions of linux. RT is still very slow for long tickets. All the time is
taken up by the perl/apache process maxing out a core of CPU.

We’ve even gone as far as trying to profile the code. We came up with this graph of where the time was going:

<TIMING.png>
We then tried to go further into those functions but can’t find a single smoking gun call that is taking all the time.

For example in a ticket that takes 22s to render approx 5 secs goes on these 2 lines:

File: Ticket/Elements/ShowHistory line: 100-103 version 3.8.8

my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;

grep { ($->TransactionId == $Transaction->Id ) && ($trans_content->{$->Id} = $_) } @attachment_content;

From what I can tell, the real problem here is repeated scanning of both the @attachments and @attachment_content arrays, which makes the execution speed of the ShowHistory element O(N^2) with respect to the number of transactions; painful, to say the least.

  1. The %$trans_content hash can be made up front in a single pass, for all the attachments in the ticket, turning this into an O(N) operation, rather than O(N^2)
  2. The $trans_content variable is only used in one place; it’s passed to ShowTransaction, where it is then passed on to ShowTransactionAttachments.
  3. In there, there are some errors which cause some autovivification of hash members which needn’t happen.
  4. You can do much the same up-front calculation with $trans_attachments as well, so you don’t have to keep grepping through it

I’ve now made those changes, and on a reasonably large ticket (216 transactions) it reduced the ticket rendering time on my system from 2.5 minutes to 34 seconds, which is a pretty good improvement, I think.

On a more extreme ticket, with 417 transactions, the 3.8.8 release code takes at least 20 minutes to render the ticket (I gave up waiting and stopped the page load after that long), so in fact it’s considerably worse order execution time than I thought. My patched code takes 2.2 minutes - still not brilliant but hey, this ticket is now renderable, which it was not before.

Just for some background, the hardware I’m running on:

RT database: 2 CPU virtual machine, 8GB RAM, MySQL, running on vSphere 4.1 on a 2.0 GHz E5504 Nehalem system
RT web server: 2 CPU virtual machine, 2GB RAM, on the same type of physical hardware as the database server

As far as I can tell, I have not semantically altered the code, but others may want to test more thoroughly. I have not yet put this on my production web server - I cloned my web server VM and made my changes to the clone (God, I love VMs for this sort of thing!)

Regards,

Tim

Here is my patch to Ticket/Elements/ShowHistory:

— /opt/rt3/share/html/Ticket/Elements/ShowHistory 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowHistory 2010-09-07 11:59:30.000000000 +0100
@@ -84,6 +84,10 @@
<%perl>
my @attachments = @{$Attachments->ItemsArrayRef()};
my @attachment_content = @{$AttachmentContent->ItemsArrayRef()};
+my $trans_content = {};
+map { $trans_content->{$->TransactionId}->{$->Id} = $_ } @attachment_content;
+my $trans_attachments = {};
+map { push (@{$trans_attachments->{$->TransactionId}}, $) } @attachments;

while ( my $Transaction = $Transactions->Next ) {
my $skip = 0;
@@ -97,12 +101,6 @@

$i++;
  • my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;
  • my $trans_content = {};
  • grep { ($->TransactionId == $Transaction->Id ) && ($trans_content->{$->Id} = $_) } @attachment_content;
    my $IsLastTransaction = 0;
    if ( $OldestFirst ) {
    $IsLastTransaction = $Transactions->IsLast;
    @@ -118,7 +116,7 @@
    Transaction => $Transaction,
    ShowHeaders => $ShowHeaders,
    RowNum => $i,
  •          Attachments          => \@trans_attachments,
    
  •          Attachments          => $trans_attachments->{$Transaction->id},
            AttachmentContent    => $trans_content,
            LastTransaction      => $IsLastTransaction
    

);

and here’s the patch to ShowTransactionAttachments (both files need to be patched):

— /opt/rt3/share/html/Ticket/Elements/ShowTransactionAttachments 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowTransactionAttachments 2010-09-07 11:04:53.000000000 +0100
@@ -189,8 +189,10 @@
{

        my $content;
  •        if ( $AttachmentContent->{ $message->id } ) {
    
  •            $content = $AttachmentContent->{ $message->id }->Content;
    
  •   my ($transaction_id, $message_id) = ($Transaction->id, $message->id);
    
  •        if ( exists($AttachmentContent->{ $transaction_id }) &&
    
  •    exists($AttachmentContent->{ $transaction_id }->{$message_id}) ) {
    
  •            $content = $AttachmentContent->{ $transaction_id }->{ $message_id }->Content;
          }
          else {
              $content = $message->Content;
    

The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

Hi Tim,

You might want to send your patch to the RT devel list. I would be
interested to see what others, including the BP people have to say about it.

~JasonOn 09/07/2010 07:24 AM, Tim Cutts wrote:

On 6 Sep 2010, at 5:54 pm, rt-users-request@lists.bestpractical.com wrote:

So far we’ve tried installing RT on different hardware, both 32 and 64bit versions of linux. RT is still very slow for long tickets. All the time is taken up by the perl/apache process maxing out a core of CPU.

We’ve even gone as far as trying to profile the code. We came up with this graph of where the time was going:

<TIMING.png>
We then tried to go further into those functions but can’t find a single smoking gun call that is taking all the time.

For example in a ticket that takes 22s to render approx 5 secs goes on these 2 lines:

File: Ticket/Elements/ShowHistory line: 100-103 version 3.8.8

my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;

grep { ($->TransactionId == $Transaction->Id )&& ($trans_content->{$->Id} = $_) } @attachment_content;

From what I can tell, the real problem here is repeated scanning of both the @attachments and @attachment_content arrays, which makes the execution speed of the ShowHistory element O(N^2) with respect to the number of transactions; painful, to say the least.

  1. The %$trans_content hash can be made up front in a single pass, for all the attachments in the ticket, turning this into an O(N) operation, rather than O(N^2)
  2. The $trans_content variable is only used in one place; it’s passed to ShowTransaction, where it is then passed on to ShowTransactionAttachments.
  3. In there, there are some errors which cause some autovivification of hash members which needn’t happen.
  4. You can do much the same up-front calculation with $trans_attachments as well, so you don’t have to keep grepping through it

I’ve now made those changes, and on a reasonably large ticket (216 transactions) it reduced the ticket rendering time on my system from 2.5 minutes to 34 seconds, which is a pretty good improvement, I think.

On a more extreme ticket, with 417 transactions, the 3.8.8 release code takes over 20 minutes to render the ticket (I gave up waiting), so in fact it’s considerably worse order execution time than I thought. My patched code takes 2.2 minutes - still not brilliant but hey, this ticket is now renderable, which it was not before.

Just for some background, the hardware I’m running on:

RT database: 2 CPU virtual machine, 8GB RAM, MySQL, running on vSphere 4.1 on a 2.0 GHz E5504 Nehalem system
RT web server: 2 CPU virtual machine, 2GB RAM, on the same type of physical hardware as the database server

As far as I can tell, I have not semantically altered the code, but others may want to test more thoroughly. I have not yet put this on my production web server - I cloned my web server VM and made my changes to the clone (God, I love VMs for this sort of thing!)

Regards,

Tim

Here are my patches to Ticket/Elements/ShowHistory:

— /opt/rt3/share/html/Ticket/Elements/ShowHistory 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowHistory 2010-09-07 11:59:30.000000000 +0100
@@ -84,6 +84,10 @@
<%perl>
my @attachments = @{$Attachments->ItemsArrayRef()};
my @attachment_content = @{$AttachmentContent->ItemsArrayRef()};
+my $trans_content = {};
+map { $trans_content->{$->TransactionId}->{$->Id} = $_ } @attachment_content;
+my $trans_attachments = {};
+map { push (@{$trans_attachments->{$->TransactionId}}, $) } @attachments;

while ( my $Transaction = $Transactions->Next ) {
my $skip = 0;
@@ -97,12 +101,6 @@

  $i++;
  • my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;
  • my $trans_content = {};
  • grep { ($->TransactionId == $Transaction->Id )&& ($trans_content->{$->Id} = $_) } @attachment_content;
  • my $IsLastTransaction = 0;
    if ( $OldestFirst ) {
        $IsLastTransaction = $Transactions->IsLast;
    

@@ -118,7 +116,7 @@
Transaction => $Transaction,
ShowHeaders => $ShowHeaders,
RowNum => $i,

  •          Attachments          =>  \@trans_attachments,
    
  •          Attachments          =>  $trans_attachments->{$Transaction->id},
              AttachmentContent    =>  $trans_content,
              LastTransaction      =>  $IsLastTransaction
    
    );

and here’s the patch to ShowTransactionAttachments (both files need to be patched):

— /opt/rt3/share/html/Ticket/Elements/ShowTransactionAttachments 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowTransactionAttachments 2010-09-07 11:04:53.000000000 +0100
@@ -189,8 +189,10 @@
{

          my $content;
  •        if ( $AttachmentContent->{ $message->id } ) {
    
  •            $content = $AttachmentContent->{ $message->id }->Content;
    
  • my ($transaction_id, $message_id) = ($Transaction->id, $message->id);
    
  •        if ( exists($AttachmentContent->{ $transaction_id })&&
    
  •  exists($AttachmentContent->{ $transaction_id }->{$message_id}) ) {
    
  •            $content = $AttachmentContent->{ $transaction_id }->{ $message_id }->Content;
            }
            else {
                $content = $message->Content;
    

smime.p7s (3.97 KB)

We gave those patches a try. Shaved ~4s from our 20s, which isn’t bad so we might look at rolling them into Live.

Obviously if you’re getting 20min loads then something must be much more fundamentally wrong with your setup I’d say. We’ve never had anything even remotely that long.

Thanks for the patch!

Justin

Justin Hayes
OpenBet Support Manager
justin.hayes@openbet.comOn 7 Sep 2010, at 12:24, Tim Cutts wrote:

On 6 Sep 2010, at 5:54 pm, rt-users-request@lists.bestpractical.com wrote:

So far we’ve tried installing RT on different hardware, both 32 and 64bit versions of linux. RT is still very slow for long tickets. All the time is taken up by the perl/apache process maxing out a core of CPU.

We’ve even gone as far as trying to profile the code. We came up with this graph of where the time was going:

<TIMING.png>
We then tried to go further into those functions but can’t find a single smoking gun call that is taking all the time.

For example in a ticket that takes 22s to render approx 5 secs goes on these 2 lines:

File: Ticket/Elements/ShowHistory line: 100-103 version 3.8.8

my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;

grep { ($->TransactionId == $Transaction->Id ) && ($trans_content->{$->Id} = $_) } @attachment_content;

From what I can tell, the real problem here is repeated scanning of both the @attachments and @attachment_content arrays, which makes the execution speed of the ShowHistory element O(N^2) with respect to the number of transactions; painful, to say the least.

  1. The %$trans_content hash can be made up front in a single pass, for all the attachments in the ticket, turning this into an O(N) operation, rather than O(N^2)
  2. The $trans_content variable is only used in one place; it’s passed to ShowTransaction, where it is then passed on to ShowTransactionAttachments.
  3. In there, there are some errors which cause some autovivification of hash members which needn’t happen.
  4. You can do much the same up-front calculation with $trans_attachments as well, so you don’t have to keep grepping through it

I’ve now made those changes, and on a reasonably large ticket (216 transactions) it reduced the ticket rendering time on my system from 2.5 minutes to 34 seconds, which is a pretty good improvement, I think.

On a more extreme ticket, with 417 transactions, the 3.8.8 release code takes over 20 minutes to render the ticket (I gave up waiting), so in fact it’s considerably worse order execution time than I thought. My patched code takes 2.2 minutes - still not brilliant but hey, this ticket is now renderable, which it was not before.

Just for some background, the hardware I’m running on:

RT database: 2 CPU virtual machine, 8GB RAM, MySQL, running on vSphere 4.1 on a 2.0 GHz E5504 Nehalem system
RT web server: 2 CPU virtual machine, 2GB RAM, on the same type of physical hardware as the database server

As far as I can tell, I have not semantically altered the code, but others may want to test more thoroughly. I have not yet put this on my production web server - I cloned my web server VM and made my changes to the clone (God, I love VMs for this sort of thing!)

Regards,

Tim

Here are my patches to Ticket/Elements/ShowHistory:

— /opt/rt3/share/html/Ticket/Elements/ShowHistory 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowHistory 2010-09-07 11:59:30.000000000 +0100
@@ -84,6 +84,10 @@
<%perl>
my @attachments = @{$Attachments->ItemsArrayRef()};
my @attachment_content = @{$AttachmentContent->ItemsArrayRef()};
+my $trans_content = {};
+map { $trans_content->{$->TransactionId}->{$->Id} = $_ } @attachment_content;
+my $trans_attachments = {};
+map { push (@{$trans_attachments->{$->TransactionId}}, $) } @attachments;

while ( my $Transaction = $Transactions->Next ) {
my $skip = 0;
@@ -97,12 +101,6 @@

$i++;
  • my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;
  • my $trans_content = {};
  • grep { ($->TransactionId == $Transaction->Id ) && ($trans_content->{$->Id} = $_) } @attachment_content;
  • my $IsLastTransaction = 0;
    if ( $OldestFirst ) {
    $IsLastTransaction = $Transactions->IsLast;
    @@ -118,7 +116,7 @@
    Transaction => $Transaction,
    ShowHeaders => $ShowHeaders,
    RowNum => $i,
  •          Attachments          => \@trans_attachments,
    
  •          Attachments          => $trans_attachments->{$Transaction->id},
            AttachmentContent    => $trans_content,
            LastTransaction      => $IsLastTransaction
    

);

and here’s the patch to ShowTransactionAttachments (both files need to be patched):

— /opt/rt3/share/html/Ticket/Elements/ShowTransactionAttachments 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowTransactionAttachments 2010-09-07 11:04:53.000000000 +0100
@@ -189,8 +189,10 @@
{

        my $content;
  •        if ( $AttachmentContent->{ $message->id } ) {
    
  •            $content = $AttachmentContent->{ $message->id }->Content;
    
  • my ($transaction_id, $message_id) = ($Transaction->id, $message->id);
    
  •        if ( exists($AttachmentContent->{ $transaction_id }) &&
    
  •  exists($AttachmentContent->{ $transaction_id }->{$message_id}) ) {
    
  •            $content = $AttachmentContent->{ $transaction_id }->{ $message_id }->Content;
          }
          else {
              $content = $message->Content;
    


The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

RT Training in Washington DC, USA on Oct 25 & 26 2010
Last one this year – Learn how to get the most out of RT!

Tim,

Thanks for these patches. I’m…very surprised at the numbers you’re seeing, but appreciate that you have something that can help. I’ll look at getting this rolled into core.On Tue, Sep 07, 2010 at 12:24:35PM +0100, Tim Cutts wrote:

On 6 Sep 2010, at 5:54 pm, rt-users-request@lists.bestpractical.com wrote:

So far we’ve tried installing RT on different hardware, both 32 and 64bit versions of linux. RT is still very slow for long tickets. All the time is taken up by the perl/apache process maxing out a core of CPU.

We’ve even gone as far as trying to profile the code. We came up with this graph of where the time was going:

<TIMING.png>
We then tried to go further into those functions but can’t find a single smoking gun call that is taking all the time.

For example in a ticket that takes 22s to render approx 5 secs goes on these 2 lines:

File: Ticket/Elements/ShowHistory line: 100-103 version 3.8.8

my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;

grep { ($_->TransactionId == $Transaction->Id ) && ($trans_content->{$_->Id} = $_)  } @attachment_content;

From what I can tell, the real problem here is repeated scanning of both the @attachments and @attachment_content arrays, which makes the execution speed of the ShowHistory element O(N^2) with respect to the number of transactions; painful, to say the least.

  1. The %$trans_content hash can be made up front in a single pass, for all the attachments in the ticket, turning this into an O(N) operation, rather than O(N^2)
  2. The $trans_content variable is only used in one place; it’s passed to ShowTransaction, where it is then passed on to ShowTransactionAttachments.
  3. In there, there are some errors which cause some autovivification of hash members which needn’t happen.
  4. You can do much the same up-front calculation with $trans_attachments as well, so you don’t have to keep grepping through it

I’ve now made those changes, and on a reasonably large ticket (216 transactions) it reduced the ticket rendering time on my system from 2.5 minutes to 34 seconds, which is a pretty good improvement, I think.

On a more extreme ticket, with 417 transactions, the 3.8.8 release code takes over 20 minutes to render the ticket (I gave up waiting), so in fact it’s considerably worse order execution time than I thought. My patched code takes 2.2 minutes - still not brilliant but hey, this ticket is now renderable, which it was not before.

Just for some background, the hardware I’m running on:

RT database: 2 CPU virtual machine, 8GB RAM, MySQL, running on vSphere 4.1 on a 2.0 GHz E5504 Nehalem system
RT web server: 2 CPU virtual machine, 2GB RAM, on the same type of physical hardware as the database server

As far as I can tell, I have not semantically altered the code, but others may want to test more thoroughly. I have not yet put this on my production web server - I cloned my web server VM and made my changes to the clone (God, I love VMs for this sort of thing!)

Regards,

Tim

Here are my patches to Ticket/Elements/ShowHistory:

— /opt/rt3/share/html/Ticket/Elements/ShowHistory 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowHistory 2010-09-07 11:59:30.000000000 +0100
@@ -84,6 +84,10 @@
<%perl>
my @attachments = @{$Attachments->ItemsArrayRef()};
my @attachment_content = @{$AttachmentContent->ItemsArrayRef()};
+my $trans_content = {};
+map { $trans_content->{$->TransactionId}->{$->Id} = $_ } @attachment_content;
+my $trans_attachments = {};
+map { push (@{$trans_attachments->{$->TransactionId}}, $) } @attachments;

while ( my $Transaction = $Transactions->Next ) {
my $skip = 0;
@@ -97,12 +101,6 @@

 $i++;
  • my @trans_attachments = grep { $_->TransactionId == $Transaction->Id } @attachments;
  • my $trans_content = {};
  • grep { ($->TransactionId == $Transaction->Id ) && ($trans_content->{$->Id} = $_) } @attachment_content;
  • my $IsLastTransaction = 0;
    if ( $OldestFirst ) {
    $IsLastTransaction = $Transactions->IsLast;
    @@ -118,7 +116,7 @@
    Transaction => $Transaction,
    ShowHeaders => $ShowHeaders,
    RowNum => $i,
  •          Attachments          => \@trans_attachments,
    
  •          Attachments          => $trans_attachments->{$Transaction->id},
             AttachmentContent    => $trans_content,
             LastTransaction      => $IsLastTransaction
    
    );

and here’s the patch to ShowTransactionAttachments (both files need to be patched):

— /opt/rt3/share/html/Ticket/Elements/ShowTransactionAttachments 2010-05-14 13:58:15.000000000 +0100
+++ /opt/rt3/local/html/Ticket/Elements/ShowTransactionAttachments 2010-09-07 11:04:53.000000000 +0100
@@ -189,8 +189,10 @@
{

         my $content;
  •        if ( $AttachmentContent->{ $message->id } ) {
    
  •            $content = $AttachmentContent->{ $message->id }->Content;
    
  • my ($transaction_id, $message_id) = ($Transaction->id, $message->id);
    
  •        if ( exists($AttachmentContent->{ $transaction_id }) &&
    
  •  exists($AttachmentContent->{ $transaction_id }->{$message_id}) ) {
    
  •            $content = $AttachmentContent->{ $transaction_id }->{ $message_id }->Content;
           }
           else {
               $content = $message->Content;
    


The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.

RT Training in Washington DC, USA on Oct 25 & 26 2010
Last one this year – Learn how to get the most out of RT!

We gave those patches a try. Shaved ~4s from our 20s, which isn’t bad so we might look at rolling them into Live.

Justin,

Have you tried running ticket loads under Devel::NYTProf with the standalone server to try to identify what’s up here for you?

We gave those patches a try. Shaved ~4s from our 20s, which isn’t bad so we might look at rolling them into Live.

Justin,

Have you tried running ticket loads under Devel::NYTProf with the standalone server to try to identify what’s up here for you?

RT Training in Washington DC, USA on Oct 25 & 26 2010
Last one this year – Learn how to get the most out of RT!

NYTProf can also be used with Apache easily enough, short how to should
anyone want to try it.

1: install Devel::NYTProf on the web server

2: edit httpd.conf

comment out:

#PerlModule Apache::DProf

add:

MaxClients 1 PerlPassEnv NYTPROF PerlModule Devel::NYTProf::Apache

3: enable nytprof in service config file (/etc/sysconfig/httpd on Red
Hat systems)

Add:

OPTIONS=‘-D NYTPROF’

4: restart httpd

5: do a few things that lag in the gui

6: stop httpd, you need to stop it for nytprof to post process the
results. Remember to comment out the OPTIONS line from step 3 when you
want to disable profiling.

7: cd /tmp

8: convert the nytprof output to html, the file names are
nytprof.<parent_pid>.out.<child_pid>

e.g.

$ nytprofhtml -f nytprof.5766.out.5771

9: view output

e.g.

$ links nytprof/index.html

Cheers, Jeff.

Thanks for these patches. I’m…very surprised at the numbers you’re seeing, but appreciate that you have something that can help. I’ll look at getting this rolled into core.

The numbers are because it turns out I’d miscounted the number of transactions on the ticket by more than a factor of 10 - there were actually more than 5500 transactions on the ticket I was testing (the transaction count I initially quoted was actually just on one of the tickets that had subsequently been merged into the monster ticket).

Regards,

Tim

I’ll have a look at that Jesse, thanks.

I think part of the problem is not really knowing how fast it should actually be. Perhaps my install is working as per normal and I just think it slow. Certainly after adding that patch of Tim’s and running faster hardware it’s a lot perkier.

Ruslan also suggested that the ticket history has had a rewrite in 3.10. Any idea how far off that is?

Thanks,

Justin

Justin Hayes
OpenBet Support Manager
justin.hayes@openbet.comOn 10 Sep 2010, at 03:52, Jesse Vincent wrote:

On Thu, Sep 09, 2010 at 08:15:37AM +0100, Justin Hayes wrote:

We gave those patches a try. Shaved ~4s from our 20s, which isn’t bad so we might look at rolling them into Live.

Justin,

Have you tried running ticket loads under Devel::NYTProf with the standalone server to try to identify what’s up here for you?

From what I can tell, the real problem here is repeated scanning of both the @attachments and @attachment_content arrays, which makes the execution speed of the ShowHistory element O(N^2) with respect to the number of transactions; painful, to say the least.

That still shouldn’t have been slow, but you’re 100% right that it
was inefficient. I’ve taken a slightly cleaned up version of the patch
into 3.9 - along with a bunch of other ticket history perf work I’ve
been doing over the past few weeks, I think you’ll find it to be a
pleasant change.

Thank you!

-jesse