Memory 'leak' in Ticket/Display.html

Hi,

When a large search is done and the user clicks the link to display
the ticket then apache starts using a lot more memory.
This is due to a memory ‘leak’ in share/html/Ticket/Display.html:


lines 200-206:

if (defined $session{‘tickets’} and ($ARGS{‘Query’} or
$session{‘CurrentSearchHash’}->{‘Query’})) {
my $item_map = $session{‘tickets’}->ItemMap;
$link_rel{first} = “Ticket/Display.html?id=” . $item_map->{first}
if $item_map->{$TicketObj->Id}{prev};
$link_rel{prev} = “Ticket/Display.html?id=” .
$item_map->{$TicketObj->Id}{prev} if $item_map->{$TicketObj->Id}{prev};
$link_rel{next} = “Ticket/Display.html?id=” .
$item_map->{$TicketObj->Id}{next} if $item_map->{$TicketObj->Id}{next};
$link_rel{last} = “Ticket/Display.html?id=” . $item_map->{last}
if $item_map->{$TicketObj->Id}{next};
}

This code will execute the query in $session{‘tickets’} and store the
results of the search in $session{‘tickets’}->…
Meaning: the session object now contains the query + the results of
the search. Which obviously results in a lot higher memory usage when
the search is large.

(When I run a search which returns 13000 tickets apache jumps from
40/60 mb of memory to 200mb of memory.
Since it is in the session it also means every later request from the
same session will result in another process using 200mb of memory.)

I modified the code a bit and added Dumper($session{tickets});
I then did a search for 2 tickets. (truncated Dumper output at the end).

A (truncated) diff from the dumper output:

(- is before, + is after)

  •             'must_redo_search' => 1,
    
  •             'must_redo_search' => 0,
    
  •             'items_array' => [],
    
  •             'dbix_sb_unique_cache' => {
    
  •                                         '37737' => 1,
    
  •                                         '39447' => 1
    
  •                                       },
    
  •             'item_map' => {
    
  •                             '37737' => {
    
  •                                        'next' => '39447',
    
  •                                        'prev' => 0,
    
  •                                        'defined' => 1
    
  •                                      },
    
  •                             'first' => '37737',
    
  •                             '39447' => {
    
  •                                          'prev' => '37737',
    
  •                                          'defined' => 1
    
  •                                        },
    
  •                             'last' => '39447'
    
  •                           },
    
  •             'items' => [
    
  •                          bless( {
    
  •                                   '_Class' => 'RT::Ticket',
    
  •                                   'original_user' => undef,
    

‘_SB_Record_Primary_RecordCache_key’ => ‘id=37737’,

  •                                   'user' => $VAR1->{'user'},
    
  •                                   'table' => 'Tickets',
    
  •                                   'values' => {
    
  •                                                  # [snipped]
    
  •                                               },
    
  •                                   'fetched' => {
    
  •                                                  # [snipped]
    
  •                                                }
    
  •                                 }, 'RT::Ticket' ),
    
  •                          bless( {
    
  •                                   '_Class' => 'RT::Ticket',
    
  •                                   'original_user' => undef,
    

‘_SB_Record_Primary_RecordCache_key’ => ‘id=39447’,

  •                                   'user' => $VAR1->{'user'},
    
  •                                   'table' => 'Tickets',
    
  •                                   'values' => {
    
  •                                                  # [snipped]
    
  •                                               },
    
  •                                   'fetched' => {
    
  •                                                  # [snipped]
    
  •                                 }, 'RT::Ticket' )
    
  •                        ]
    

This means that after the display it contains extra data in items,
item_map and dbix_sb_unique_cache.
Clearing items is simple and can be done by adding:
$session{‘tickets’}->PrepForSerialization();

in the code (which is what Search/Results.html calls).

But that still leaves item_map and dbix_sb_unique_cache in the session.
Any advice on clearing those?

Best regards,

Bram

(Truncated) Dumper output:

[Thu Jul 23 12:49:59 2009] [info]: BEFORE
(/opt/rt3/local/html/Ticket/Display.html:223)
[Thu Jul 23 12:49:59 2009] [info]: $VAR1 = bless( {
’_open_parens’ => {},
‘alias_count’ => 0,
‘table’ => ‘Tickets’,
‘where_clause’ => ‘’,
‘order_by’ => [
{
‘FIELD’ => ‘id’,
‘ORDER’ => ‘ASC’
}
],
‘RecalcTicketLimits’ => 0,
’_sql_trattachalias’ => undef,
’_sql_object_cfv_alias’ => undef,
‘is_limited’ => 1,
‘order’ => ‘’,
’_sql_watcher_join_users_alias’ => undef,
‘user’ => bless( {
# [ snipped ]
}, ‘RT::CurrentUser’ ),
‘restriction_index’ => 1,
’_sql_query’ => ‘id = 37737 OR id = 39447’,
’_sql_transalias’ => undef,
‘DBIxHandle’ => bless( {
‘dsn’ =>
‘dbi:Pg:dbname=ayuda;host=db’,
‘StatementLog’ => [],
‘DisconnectHandleOnDestroy’ => undef
}, ‘RT::Handle’ ),
‘limit_clause’ => ‘’,
‘restrictions’ => {
‘main.Status’ => [
{
‘value’ =>
’‘deleted’’,
‘field’ =>
‘main.Status’,
‘op’ => ‘!=’
}
],
‘ticketsql’ => [
{
‘value’ =>
’‘37737’’,
‘field’ => ‘main.id’,
‘op’ => ‘=’
},
‘OR’,
{
‘value’ =>
’‘39447’’,
‘field’ => ‘main.id’,
‘op’ => ‘=’
}
],
‘main.EffectiveId’ => [
{

‘value’ => ‘main.id’,

‘field’ => ‘main.EffectiveId’,
‘op’ => ‘=’
}
],
‘main.Type’ => [
{
‘value’ =>
’‘ticket’’,
‘field’ =>
‘main.Type’,
‘op’ => ‘=’
}
]
},
‘primary_key’ => ‘id’,
‘count_all’ => ‘2’,
‘must_redo_search’ => 1,
‘looking_at_effective_id’ => 0,
‘itemscount’ => 0,
‘show_rows’ => 0,
’_sql_cf_alias’ => undef,
‘left_joins’ => {},
‘aliases’ => [],
‘subclauses’ => {
‘generic_restrictions’ =>
’(main.Status != ‘deleted’) AND (main.id = ‘37737’ OR main.id =
‘39447’) AND (main.Type = ‘ticket’) AND (main.EffectiveId =
main.id)’
},
‘first_row’ => 0,
‘looking_at_type’ => 0,
’_sql_looking_at’ => {
‘id’ => 1
}
}, ‘RT::Tickets’ );
(/opt/rt3/local/html/Ticket/Display.html:224)

[Thu Jul 23 12:50:00 2009] [info]: AFTER
(/opt/rt3/local/html/Ticket/Display.html:233)
[Thu Jul 23 12:50:00 2009] [info]: $VAR1 = bless( {
’_open_parens’ => {},
‘table’ => ‘Tickets’,
’_sql_trattachalias’ => undef,
’_sql_object_cfv_alias’ => undef,
‘is_limited’ => 1,
‘order’ => ‘’,
’_sql_watcher_join_users_alias’ => undef,
‘user’ => bless( {
# [ snipped ]
}, ‘RT::CurrentUser’ ),
’_sql_query’ => ‘id = 37737 OR id = 39447’,
‘restrictions’ => {
‘main.Status’ => [
{
‘value’ =>
’‘deleted’’,
‘field’ =>
‘main.Status’,
‘op’ => ‘!=’
}
],
‘ticketsql’ => [
{
‘value’ =>
’‘37737’’,
‘field’ => ‘main.id’,
‘op’ => ‘=’
},
‘OR’,
{
‘value’ =>
’‘39447’’,
‘field’ => ‘main.id’,
‘op’ => ‘=’
}
],
‘main.EffectiveId’ => [
{

‘value’ => ‘main.id’,

‘field’ => ‘main.EffectiveId’,
‘op’ => ‘=’
}
],
‘main.Type’ => [
{
‘value’ =>
’‘ticket’’,
‘field’ =>
‘main.Type’,
‘op’ => ‘=’
}
]
},
‘primary_key’ => ‘id’,
‘count_all’ => ‘2’,
‘items_array’ => [],
‘must_redo_search’ => 0,
‘looking_at_effective_id’ => 0,
‘subclauses’ => {
‘generic_restrictions’ =>
’(main.Status != ‘deleted’) AND (main.id = ‘37737’ OR main.id =
‘39447’) AND (main.EffectiveId = main.id) AND (main.Type =
‘ticket’)’
},
’_sql_looking_at’ => {
‘id’ => 1
},
‘alias_count’ => 0,
‘dbix_sb_unique_cache’ => {
‘37737’ => 1,
‘39447’ => 1
},
‘order_by’ => [
{
‘FIELD’ => ‘id’,
‘ORDER’ => ‘ASC’
}
],
‘where_clause’ => ‘’,
‘RecalcTicketLimits’ => 0,
’_sql_transalias’ => undef,
‘restriction_index’ => 1,
‘item_map’ => {
‘37737’ => {
‘next’ => ‘39447’,
‘prev’ => 0,
‘defined’ => 1
},
‘first’ => ‘37737’,
‘39447’ => {
‘prev’ => ‘37737’,
‘defined’ => 1
},
‘last’ => ‘39447’
},
‘limit_clause’ => ‘’,
‘DBIxHandle’ => bless( {
‘dsn’ =>
‘dbi:Pg:dbname=ayuda;host=db’,
‘StatementLog’ => [],
‘DisconnectHandleOnDestroy’ => undef
}, ‘RT::Handle’ ),
‘itemscount’ => 0,
’_sql_cf_alias’ => undef,
‘show_rows’ => 0,
‘left_joins’ => {},
‘aliases’ => [],
‘first_row’ => 0,
‘looking_at_type’ => 0,
‘items’ => [
bless( {
’_Class’ => ‘RT::Ticket’,
‘original_user’ => undef,

‘_SB_Record_Primary_RecordCache_key’ => ‘id=37737’,
‘user’ => $VAR1->{‘user’},
‘table’ => ‘Tickets’,
‘values’ => {
# [snipped]
},
‘fetched’ => {
# [snipped]
}
}, ‘RT::Ticket’ ),
bless( {
’_Class’ => ‘RT::Ticket’,
‘original_user’ => undef,

‘_SB_Record_Primary_RecordCache_key’ => ‘id=39447’,
‘user’ => $VAR1->{‘user’},
‘table’ => ‘Tickets’,
‘values’ => {
# [snipped]
},
‘fetched’ => {
# [snipped]
}
}, ‘RT::Ticket’ )
]
}, ‘RT::Tickets’ );
(/opt/rt3/local/html/Ticket/Display.html:234)

Quoting Bram rtdevel@lists.wizbit.be:

Hi,

When a large search is done and the user clicks the link to display
the ticket then apache starts using a lot more memory.
This is due to a memory ‘leak’ in share/html/Ticket/Display.html:

http://github.com/bestpractical/rt/blob/90546414c8887e344f995da5f20f53a58939bbd8/share/html/Ticket/Display.html
lines 200-206:

This also happens in


lines 69-73.

There PrepForSerialization is called but that still leaves items,
item_map and dbix_sb_unique_cache to contain data.

When both are completly cleared there still is a 40 mb increase in
memory usage on a display page (after a search which returned 13000
tickets) because it builds the entire ItemMap.

This looks like overkill to me because it only 4 values are used…
(that is: first, last, next and prev). Perhaps ItemMap can be improved
so that it only looks up the first, last, next and prev value relative
to the ticket that is being viewed?

Best regards,

Bram

I also have this issue but have worked around it by keeping the searches
small. We have a lot of tickets and large searches will basically have
apache grow until it runs out of memory after you click display. I have
reported it before but wasn’t able to detail it like this, hopefully
there is a workable solution for it.

Bram wrote: