"Out of memory" error in logs when restarting apache

Whenever I use mod_perl since my 2.0.15 to 3.2.2 upgrade (which involved
a 3.0.x upgrade in the middle), I get errors like this in my
/var/log/apache_error_log file and find that apache isn’t running.

[Thu Jan 5 08:00:01 2006] [notice] SIGUSR1 received. Doing graceful restart
[Thu Jan 5 08:00:24 2006] [error] (2)No such file or directory: FastCGI:
doesn’t exist and can’t be created: No such file or directory
[Thu Jan 5 08:00:24 2006] [error] (2)No such file or directory: FastCGI:
can’t create dynamic directory “/usr/local/logs/fastcgi/dynamic”: doesn’t
exist and ca
n’t be created: No such file or directory
[Thu Jan 5 08:00:24 2006] [notice] FastCGI: process manager initialized (pid
50744)
[Thu Jan 5 08:00:27 2006] [notice] Apache/1.3.34 (Unix) mod_perl/1.29
mod_fastcgi/2.4.2 PHP/4.4.1 configured – resuming normal operations
[Thu Jan 5 08:00:27 2006] [notice] Accept mutex: flock (Default: flock)
[Thu Jan 5 09:00:00 2006] [notice] SIGUSR1 received. Doing graceful restart
Out of memory during request for 24 bytes, total sbrk() is 18515968 bytes!
Callback called exit.
END failed–call queue aborted at /dev/null line 9.
Callback called exit at /dev/null line 9.
END failed–call queue aborted at /dev/null line 12.
Callback called exit at /dev/null line 12.
END failed–call queue aborted at /dev/null line 57.
Callback called exit at /dev/null line 57.
END failed–call queue aborted at /dev/null line 86.
Callback called exit at /dev/null line 86.

 I worked around this my switching to FastCGI for over a year.  RT was

very slow (15 seconds between pages and I only have about 6500 tickets)
whether I used mod_perl or FastCGI.

 I recently tried mod_perl again (I upgraded to RT 3.4.4, as well as

upgrading FreeBSD, MySQL, Perl, etc. so I figured it was worth another
try) and have experienced the error listed above.

 Can anyone shed some light on this?  At least point me in the right

direction?

                                                   Thanks in advance,
                                                   Jaime

Network Administrator
Cairo-Durham Central School District
http://cns.cairodurham.org

Jaime Kikpole wrote:

Whenever I use mod_perl since my 2.0.15 to 3.2.2 upgrade (which involved

a 3.0.x upgrade in the middle), I get errors like this in my
/var/log/apache_error_log file and find that apache isn’t running.

Can you start Apache single-threaded, with ’ -X’? Perhaps by adding a stanza
like this to the start script? I scribbled this in for Linux, not
FreeBSD, but you
get the general idea.

start-debug() {
echo $"Starting $prog in DEBUG mode: "
check13 || exit 1
LANG=$HTTPD_LANG daemon $httpd -X $OPTIONS && echo -n "Exiting
… "
}

After getting a single-threaded Apache, I switch to another (much
larger) window
and attach strace to it. On FreeBSD, you can use ktrace instead.

(start Apache in debug mode)
(get the PID of the httpd)
ktrace -f /tmp/apache.kd -p $PID

Then you can try to reproduce the error you saw in your logs, using whatever
combination of page-loads or logins that it required.

Stop the tracing with ktrace -C after you see the bug.

Read the data with kdump. Find out what was happening while the 18MB
of data were being allocated (big sbrk()). Watch the file i/o, since
that is how
the app communicates with the database. Is that memory being malloc’ed
to hold
some large part of your database? Look for the last SQL query before the
problem.

Caveats: The problem you mention may require a multi-process Apache. Using
a single-process Apache to debug your problem could make the bug impossible
to reproduce.

Once again, use

ktrace -C

to stop the tracing to that file so that your disk does not fill up with
dump data. :slight_smile:

That last part is pretty important.

Do let us know what you find.

Greg