Upgrade to 3.8.1 Out of Memory Errors

I upgraded from 3.8.0 to 3.8.1 and added graphviz support in the wee
hours of last night. All seemed fine. I was able to graph ticket
relationships and I worked until about lunch before I encountered a
problem where each webpage is blank. I am definitely connecting to the
web server, but nothing is displayed. I have “Out of memory! Callback
called exit.” Errors in my apache2 logs. My system is Debian running in
a vm with about 700mb of ram available and I have about 2000 tickets.

I found the following information (below) which seems to pertain to my
situation. My question is (and no offense to Sebastien as I really
appreciate his posting this solution), but is this the appropriate
action to take? Do I simply need more RAM or is this an error with the
schema? I am reticent to go making changes that will come back to haunt
me later.

Thus far in an attempt to resolve the problem I have only restarted
apache2 and the server. Neither made any difference.

Thank you,

Joshua

Re: Problem after an upgrade to 3.8.1 Remove Highlighting [In reply to]
Hello David,

The problem looks like the session table is being corrupted.
I modified the schema using: ALTER TABLE sessions CHANGE a_session
a_session LONGBLOB;
Now everything seems to be working fine.
By the way, I am also using a virtual machine. 256Mo were enough for
3.8.0. I increased the available memory to 512Mo but this did not
solved the problem.
I hope this trick will solve your problem.
Regards

Sebastien

Sebastien,

I had this issue as well - luckily I was running in a Virtual Machine,
so I
upped the allocated memory from 1Gig to 2Gig and the problem went
away. It
looks like RT is a bit of a memory pig (in the nicest possible way!)
How

The increase in blob size should be transparent. Smaller blobs can
still be contained in the larger type and they are functionally
equivalent. If this is the problem, this is the only answer unless
you feel like doing heavy RT coding to reduce the size of a session.
This is definitely the easiest way out.

Cheers,
KenOn Thu, Aug 21, 2008 at 01:56:13PM -0500, Joshua Hunter wrote:

I upgraded from 3.8.0 to 3.8.1 and added graphviz support in the wee
hours of last night. All seemed fine. I was able to graph ticket
relationships and I worked until about lunch before I encountered a
problem where each webpage is blank. I am definitely connecting to the
web server, but nothing is displayed. I have “Out of memory! Callback
called exit.” Errors in my apache2 logs. My system is Debian running in
a vm with about 700mb of ram available and I have about 2000 tickets.

I found the following information (below) which seems to pertain to my
situation. My question is (and no offense to Sebastien as I really
appreciate his posting this solution), but is this the appropriate
action to take? Do I simply need more RAM or is this an error with the
schema? I am reticent to go making changes that will come back to haunt
me later.

Thus far in an attempt to resolve the problem I have only restarted
apache2 and the server. Neither made any difference.

Thank you,

Joshua

Re: Problem after an upgrade to 3.8.1 Remove Highlighting [In reply to]
Hello David,

The problem looks like the session table is being corrupted.
I modified the schema using: ALTER TABLE sessions CHANGE a_session
a_session LONGBLOB;
Now everything seems to be working fine.
By the way, I am also using a virtual machine. 256Mo were enough for
3.8.0. I increased the available memory to 512Mo but this did not
solved the problem.
I hope this trick will solve your problem.
Regards

Sebastien

On Thu, Aug 21, 2008 at 4:36 AM, David Hobley <david.hobley[at]mionegroup.com> wrote:

Sebastien,

I had this issue as well - luckily I was running in a Virtual Machine,
so I
upped the allocated memory from 1Gig to 2Gig and the problem went
away. It
looks like RT is a bit of a memory pig (in the nicest possible way!)
How
much memory do you have in your machine?

Cheers,
David

----- Original Message -----
From: “Sebastien Termeau” <st[at]mobilepeople.com>
To: rt-users[at]lists.bestpractical.com
Sent: Wednesday, 20 August, 2008 11:22:13 PM GMT +10:00 Brisbane
Subject: [rt-users] Problem after an upgrade to 3.8.1

Dear RT-Users,

I have a really recent RT 3.6.7 installation (around 200 tickets)
running on CentOS 5.2.
I was able to upgrade to 3.8.0 without problems.
The problems appear when I tried to upgrade either from 3.6.7 to 3.8.1
or from 3.8.0 to 3.8.1. A fresh 3.8.1 installation works fine.
After the upgrade, when I try to access the web interface I only
receive a blank page.
In my apache error log file, the following message is present:
Out of memory!
Callback called exit.
I got the same error when I tried to use “rt-clean-sessions”.
Using ‘perl -d’, I tracked down the error to
RT::Interface::Web::Session::ClearByUser line 246
eval { tie %session, $class, $id, $attrs };
→ $class=Apache::Session::MySQL
→ $id=3d97a1c2c09740acfee006d0221d1d12
→ attrs is an empty hash

I tried to remove all the existing sessions in the database : delete
from sessions;
And it was working fine for 5 minutes. Then the same error rise again.

Did I missed something in the upgrade process?
Do you have a suggestion?

Regards


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales[at]bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com


Cheers,
David Hobley

IT Manager
Creators of Miessence, MiVitality and MiEnviron

Phone: +61 (7) 5582 7020
Fax: +61 (7) 5539 6719
USA Fax 1800 840 0827
Email : david.hobley[at]mionegroup.com
Website: www.mionegroup.com


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

I also ran across this.

I added local mods back into the local tree after trying out 3.8.0. It
worked fine across apache restarts.

When I upgraded to 3.8.1 I got the “Out of memory! Callback called
exit.” error every time I tried to load RT in my browser. Figuring that
one of my local mods had something in it causing this, I wiped the
entire RT tree, reinstalled and then promptly got too busy to try to add
them back in.

I hope to have some time to play and see if I can replicate this.

Drew

Joshua Hunter wrote:

I upgraded from 3.8.0 to 3.8.1 and added graphviz support in the wee
hours of last night. All seemed fine. I was able to graph ticket
relationships and I worked until about lunch before I encountered a
problem where each webpage is blank. I am definitely connecting to the
web server, but nothing is displayed. I have “Out of memory! Callback
called exit.” Errors in my apache2 logs. My system is Debian running in
a vm with about 700mb of ram available and I have about 2000 tickets.

I found the following information (below) which seems to pertain to my
situation. My question is (and no offense to Sebastien as I really
appreciate his posting this solution), but is this the appropriate
action to take? Do I simply need more RAM or is this an error with the
schema? I am reticent to go making changes that will come back to haunt
me later.

Thus far in an attempt to resolve the problem I have only restarted
apache2 and the server. Neither made any difference.

Thank you,

Joshua

Re: Problem after an upgrade to 3.8.1 Remove Highlighting [In reply to]
Hello David,

The problem looks like the session table is being corrupted.
I modified the schema using: ALTER TABLE sessions CHANGE a_session
a_session LONGBLOB;
Now everything seems to be working fine.
By the way, I am also using a virtual machine. 256Mo were enough for
3.8.0. I increased the available memory to 512Mo but this did not
solved the problem.
I hope this trick will solve your problem.
Regards

Sebastien

Sebastien,

I had this issue as well - luckily I was running in a Virtual Machine,

so I

upped the allocated memory from 1Gig to 2Gig and the problem went

away. It

looks like RT is a bit of a memory pig (in the nicest possible way!)

How

much memory do you have in your machine?

Cheers,
David

----- Original Message -----
From: “Sebastien Termeau” <st[at]mobilepeople.com>
To: rt-users[at]lists.bestpractical.com
Sent: Wednesday, 20 August, 2008 11:22:13 PM GMT +10:00 Brisbane
Subject: [rt-users] Problem after an upgrade to 3.8.1

Dear RT-Users,

I have a really recent RT 3.6.7 installation (around 200 tickets)
running on CentOS 5.2.
I was able to upgrade to 3.8.0 without problems.
The problems appear when I tried to upgrade either from 3.6.7 to 3.8.1
or from 3.8.0 to 3.8.1. A fresh 3.8.1 installation works fine.
After the upgrade, when I try to access the web interface I only
receive a blank page.
In my apache error log file, the following message is present:
Out of memory!
Callback called exit.
I got the same error when I tried to use “rt-clean-sessions”.
Using ‘perl -d’, I tracked down the error to
RT::Interface::Web::Session::ClearByUser line 246
eval { tie %session, $class, $id, $attrs };
→ $class=Apache::Session::MySQL
→ $id=3d97a1c2c09740acfee006d0221d1d12
→ attrs is an empty hash

I tried to remove all the existing sessions in the database : delete
from sessions;
And it was working fine for 5 minutes. Then the same error rise again.

Did I missed something in the upgrade process?
Do you have a suggestion?

Regards


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales[at]bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com


Cheers,
David Hobley

IT Manager
Creators of Miessence, MiVitality and MiEnviron

Phone: +61 (7) 5582 7020
Fax: +61 (7) 5539 6719
USA Fax 1800 840 0827
Email : david.hobley[at]mionegroup.com
Website: www.mionegroup.com


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

Earlier in this thread, it was mentioned in passing that a clean install of 3.8.1 is fine, implying this problem only affects upgrades. Is this so? If I can afford to do a clean install rather than an upgrade from 380, do I then not have to make this change to LONGBLOB type? Does 381 already use LONGBLOB or is there something else occuring (or not occuring) during the update process?

Is LONGBLOB’ing the a_session field going to be the recommended method, or maybe even become part of the official update process when moving say to 3.8.2?

I’ve not yet noticed this problem on our system, but ours is very very small. We have less than a dozen people using the web interface. All our ticket creators only know how to email our helpdesk@ address and we don’t tell them about the actual website, keeping the website itself strictly for internal support staff. And I’m guessing this issue is related more to the number of sessions on the website, than the total number of tickets in the RT database.

For those of you who have run into this issue, any idea how close you used to be to maximum memory usage before updating to 381? Was it just “I was close at 380 and 381 just tipped me over” scenario? Or was your system only using up say 50% memory and going to 381 more than doubled your memory footprint?

Thank you,
PH

The increase in blob size should be transparent. Smaller blobs can
still be contained in the larger type and they are functionally
equivalent. If this is the problem, this is the only answer unless
you feel like doing heavy RT coding to reduce the size of a session.
This is definitely the easiest way out.

Cheers,
Ken

I upgraded from 3.8.0 to 3.8.1 and added graphviz support in the wee
hours of last night. All seemed fine. I was able to graph ticket
relationships and I worked until about lunch before I encountered a
problem where each webpage is blank. I am definitely connecting to the
web server, but nothing is displayed. I have “Out of memory! Callback
called exit.” Errors in my apache2 logs. My system is Debian running in
a vm with about 700mb of ram available and I have about 2000 tickets.

I found the following information (below) which seems to pertain to my
situation. My question is (and no offense to Sebastien as I really
appreciate his posting this solution), but is this the appropriate
action to take? Do I simply need more RAM or is this an error with the
schema? I am reticent to go making changes that will come back to haunt
me later.

Thus far in an attempt to resolve the problem I have only restarted
apache2 and the server. Neither made any difference.

Thank you,

Joshua

Re: Problem after an upgrade to 3.8.1 Remove Highlighting [In reply to]
Hello David,

The problem looks like the session table is being corrupted.
I modified the schema using: ALTER TABLE sessions CHANGE a_session
a_session LONGBLOB;
Now everything seems to be working fine.
By the way, I am also using a virtual machine. 256Mo were enough for
3.8.0. I increased the available memory to 512Mo but this did not
solved the problem.
I hope this trick will solve your problem.
Regards

Sebastien

Sebastien,

I had this issue as well - luckily I was running in a Virtual Machine,
so I
upped the allocated memory from 1Gig to 2Gig and the problem went
away. It
looks like RT is a bit of a memory pig (in the nicest possible way!)
How
much memory do you have in your machine?

Cheers,
David

----- Original Message -----
From: “Sebastien Termeau” <st[at]mobilepeople.com>
To: rt-users[at]lists.bestpractical.com
Sent: Wednesday, 20 August, 2008 11:22:13 PM GMT +10:00 Brisbane
Subject: [rt-users] Problem after an upgrade to 3.8.1

Dear RT-Users,

I have a really recent RT 3.6.7 installation (around 200 tickets)
running on CentOS 5.2.
I was able to upgrade to 3.8.0 without problems.
The problems appear when I tried to upgrade either from 3.6.7 to 3.8.1
or from 3.8.0 to 3.8.1. A fresh 3.8.1 installation works fine.
After the upgrade, when I try to access the web interface I only
receive a blank page.
In my apache error log file, the following message is present:
Out of memory!
Callback called exit.
I got the same error when I tried to use “rt-clean-sessions”.
Using ‘perl -d’, I tracked down the error to
RT::Interface::Web::Session::ClearByUser line 246
eval { tie %session, $class, $id, $attrs };
→ $class=Apache::Session::MySQL
→ $id=3d97a1c2c09740acfee006d0221d1d12
→ attrs is an empty hash

I tried to remove all the existing sessions in the database : delete
from sessions;
And it was working fine for 5 minutes. Then the same error rise again.

Did I missed something in the upgrade process?
Do you have a suggestion?

Regards


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales[at]bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com


Cheers,
David Hobley

IT Manager
Creators of Miessence, MiVitality and MiEnviron

Phone: +61 (7) 5582 7020
Fax: +61 (7) 5539 6719
USA Fax 1800 840 0827
Email : david.hobley[at]mionegroup.com
Website: www.mionegroup.com


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Paul Hirose : pthirose@ucdavis.edu : Sysadm Motto: rm -fr /MyLife
1034 Academic Surge : Programmer/Analyst : Backup Motto : rm -fr /
One Shields Avenue : Voice (530) 752-7181 : Robot, n.: Univ. Admin
Davis, CA 95616-8770 : Fax (530) 752-4465 : rec.pets.cat.anecdotes

Paul,

For my part I can tell you that it was in fact an upgrade and not a
clean install.

I’m not sure about what kind of system resources I was using prior as
I’d never had any qualms with the performance. I am the only user of
the web interface and only receive 5-10 tickets or so per day.

Clearing the sessions table immediately restores my access … for a
little while.

Joshua Hunter

Message: 5Date: Thu, 21 Aug 2008 12:49:42 -0700 (PDT)
From: Paul Hirose pthirose@ucdavis.edu
Subject: Re: [rt-users] Upgrade to 3.8.1 Out of Memory Errors
To: rt-users@lists.bestpractical.com
Message-ID: Pine.GSO.4.58.0808211241560.17404@vici.ucdavis.edu
Content-Type: TEXT/PLAIN; charset=US-ASCII

Earlier in this thread, it was mentioned in passing that a clean install
of 3.8.1 is fine, implying this problem only affects upgrades. Is this
so? If I can afford to do a clean install rather than an upgrade from
380, do I then not have to make this change to LONGBLOB type? Does 381
already use LONGBLOB or is there something else occuring (or not
occuring) during the update process?

Is LONGBLOB’ing the a_session field going to be the recommended method,
or maybe even become part of the official update process when moving say
to 3.8.2?

I’ve not yet noticed this problem on our system, but ours is very very
small. We have less than a dozen people using the web interface. All
our ticket creators only know how to email our helpdesk@ address and we
don’t tell them about the actual website, keeping the website itself
strictly for internal support staff. And I’m guessing this issue is
related more to the number of sessions on the website, than the total
number of tickets in the RT database.

For those of you who have run into this issue, any idea how close you
used to be to maximum memory usage before updating to 381? Was it just
“I was close at 380 and 381 just tipped me over” scenario? Or was your
system only using up say 50% memory and going to 381 more than doubled
your memory footprint?

Thank you,
PH

Joshua,

I missed a step in the upgrade process.
I figured this out while looking at another thread dealing with
attachment corruption.
The upgrade process is:

  • Upgrade to 3.8.0
  • perl etc/upgrade/schema.mysql-4.0-4.1.pl db user passwd > sql.queries
    mysql -u root -p rt3 < sql.queries
  • Upgrade from 3.8.0 to 3.8.1

If you look into the generated sql.queries file, a_session is changed
to LONGBLOB.
I am using Mysql5, that is why I discarded this upgrade step.
I hope this will solve your problems too.
Regards

SebastienOn Thu, Aug 21, 2008 at 8:56 PM, Joshua Hunter jhunter@telwestservices.com wrote:

I upgraded from 3.8.0 to 3.8.1 and added graphviz support in the wee
hours of last night. All seemed fine. I was able to graph ticket
relationships and I worked until about lunch before I encountered a
problem where each webpage is blank. I am definitely connecting to the
web server, but nothing is displayed. I have “Out of memory! Callback
called exit.” Errors in my apache2 logs. My system is Debian running in
a vm with about 700mb of ram available and I have about 2000 tickets.

I found the following information (below) which seems to pertain to my
situation. My question is (and no offense to Sebastien as I really
appreciate his posting this solution), but is this the appropriate
action to take? Do I simply need more RAM or is this an error with the
schema? I am reticent to go making changes that will come back to haunt
me later.

Thus far in an attempt to resolve the problem I have only restarted
apache2 and the server. Neither made any difference.

Thank you,

Joshua

Re: Problem after an upgrade to 3.8.1 Remove Highlighting [In reply to]
Hello David,

The problem looks like the session table is being corrupted.
I modified the schema using: ALTER TABLE sessions CHANGE a_session
a_session LONGBLOB;
Now everything seems to be working fine.
By the way, I am also using a virtual machine. 256Mo were enough for
3.8.0. I increased the available memory to 512Mo but this did not
solved the problem.
I hope this trick will solve your problem.
Regards

Sebastien

On Thu, Aug 21, 2008 at 4:36 AM, David Hobley <david.hobley[at]mionegroup.com> wrote:

Sebastien,

I had this issue as well - luckily I was running in a Virtual Machine,
so I
upped the allocated memory from 1Gig to 2Gig and the problem went
away. It
looks like RT is a bit of a memory pig (in the nicest possible way!)
How
much memory do you have in your machine?

Cheers,
David

----- Original Message -----
From: “Sebastien Termeau” <st[at]mobilepeople.com>
To: rt-users[at]lists.bestpractical.com
Sent: Wednesday, 20 August, 2008 11:22:13 PM GMT +10:00 Brisbane
Subject: [rt-users] Problem after an upgrade to 3.8.1

Dear RT-Users,

I have a really recent RT 3.6.7 installation (around 200 tickets)
running on CentOS 5.2.
I was able to upgrade to 3.8.0 without problems.
The problems appear when I tried to upgrade either from 3.6.7 to 3.8.1
or from 3.8.0 to 3.8.1. A fresh 3.8.1 installation works fine.
After the upgrade, when I try to access the web interface I only
receive a blank page.
In my apache error log file, the following message is present:
Out of memory!
Callback called exit.
I got the same error when I tried to use “rt-clean-sessions”.
Using ‘perl -d’, I tracked down the error to
RT::Interface::Web::Session::ClearByUser line 246
eval { tie %session, $class, $id, $attrs };
→ $class=Apache::Session::MySQL
→ $id=3d97a1c2c09740acfee006d0221d1d12
→ attrs is an empty hash

I tried to remove all the existing sessions in the database : delete
from sessions;
And it was working fine for 5 minutes. Then the same error rise again.

Did I missed something in the upgrade process?
Do you have a suggestion?

Regards


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales[at]bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com


Cheers,
David Hobley

IT Manager
Creators of Miessence, MiVitality and MiEnviron

Phone: +61 (7) 5582 7020
Fax: +61 (7) 5539 6719
USA Fax 1800 840 0827
Email : david.hobley[at]mionegroup.com
Website: www.mionegroup.com


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com