FastCGI vs/or FastCGId? System memory "leak"?

RT Users
1

Hi,

There are many nice advantages to use mod_FastCGI over mod_Perl. Our RT
instances has gotten much faster in response time when we switched from
mod_Perl to mod_FastCGI.

As mentioned earlier on this list we are slowly loosing system memory on our
RT application server, and at some point we need to restart Apache
(running v2.0.52) when memory and swap has been “eaten away”.

Currently we’re using mod_FastCGI v2.4.2 from http://www.fastcgi.com/, but
the source code hasn’t been maintained since 2003.

In httpd.conf we have configured:
FastCgiServer /site/rt-3.4.2/bin/mason_handler.fcgi -idle-timeout 300 -processes 10

(still on 3.4.2, since newer RT versions in 3.4 broke our e-mail administration
code modifications, which our *nix users fanaticaly demands as opposed to
RT’s Mason view.)

Typically after a long run:

ps aux | grep mason

nobody 8204 0.5 3.1 310572 122160 ? S 2005 287:57 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8208 0.5 3.1 303116 120788 ? S 2005 284:10 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8211 0.5 4.3 320840 169716 ? S 2005 296:36 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8242 0.5 3.2 326480 124112 ? R 2005 284:07 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8352 0.5 3.4 298260 133776 ? S 2005 279:41 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8467 0.5 3.6 330692 139136 ? S 2005 290:36 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8576 0.5 3.0 343052 116868 ? S 2005 282:37 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8579 0.5 3.9 635708 153624 ? S 2005 303:06 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8582 0.5 3.2 315260 124844 ? S 2005 280:31 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8585 0.5 3.2 293456 126316 ? S 2005 284:20 /site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi

It’s expected that a running Perl process normaly do not free up memory,
but e.g. PID 8204 does not time out at all. Is it because it’s not idle at
all, and too busy working with requests? Should we increase number of
processes? They consume a lot of memory, and I think it just increases
more and more.

top says:

13:44:27 up 204 days, 23:12, 2 users, load average: 0.45, 0.50, 0.65
99 processes: 98 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.0% 0.0% 0.2% 0.0% 0.0% 0.0% 99.7%
cpu00 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu01 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu02 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu03 0.0% 0.0% 0.9% 0.0% 0.0% 0.0% 99.0%
Mem: 3857224k av, 3321624k used, 535600k free, 0k shrd, 36708k
buff
2736968k actv, 338356k in_d, 15600k in_c
Swap: 4192944k av, 2149104k used, 2043840k free 1772712k
cached

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will have
a closer look at this source. And, the source code looks newer and
maintained. :wink:

Nice little article reference:
http://www.groovie.org/articles/2004/12/18/fast-cgi-with-html-mason

Tomi

Tomas A. P. Olaj, email: tomas.olaj@usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group

2

I’m far from a fastCGI expert, but my understanding is that you don’t
want to run too many fcgi processes - look back through the mailing list
archive for a performance discussion which explains why. I believe it
had to do with how much caching each process did, independent of the
other. If i recall the rule of thumb is only as many fcgis as you
expected simultaneous web hits to the RT server. based on that in
my small environment i ended up with two.

In my experience it is generally safe to kill the fcgi processes
although i suppose there might be race conditions in them that i’m not aware
of - i run a very small RT installation compared to what i imagine
the size of yours must be, if you’re using that much CPU time.

if your environment allows it, you might try running a cron job to kill
the fcgi jobs every night in the middle of the night.

You definitely don’t need to restart apache to kill the fcgis. You may
even be able to kill fcgis individually and new ones will be spawned, which
might reduce the possibility of downtime for you if you have a demanding
environment.On Fri, Jan 13, 2006 at 01:52:43PM +0100, Tomas Olaj wrote:

Hi,

There are many nice advantages to use mod_FastCGI over mod_Perl. Our RT
instances has gotten much faster in response time when we switched from
mod_Perl to mod_FastCGI.

As mentioned earlier on this list we are slowly loosing system memory on our
RT application server, and at some point we need to restart Apache
(running v2.0.52) when memory and swap has been “eaten away”.

Currently we’re using mod_FastCGI v2.4.2 from http://www.fastcgi.com/, but
the source code hasn’t been maintained since 2003.

In httpd.conf we have configured:
FastCgiServer /site/rt-3.4.2/bin/mason_handler.fcgi -idle-timeout 300
-processes 10

(still on 3.4.2, since newer RT versions in 3.4 broke our e-mail
administration code modifications, which our *nix users fanaticaly demands
as opposed to RT’s Mason view.)

Typically after a long run:

ps aux | grep mason

nobody 8204 0.5 3.1 310572 122160 ? S 2005 287:57
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8208 0.5 3.1 303116 120788 ? S 2005 284:10
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8211 0.5 4.3 320840 169716 ? S 2005 296:36
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8242 0.5 3.2 326480 124112 ? R 2005 284:07
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8352 0.5 3.4 298260 133776 ? S 2005 279:41
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8467 0.5 3.6 330692 139136 ? S 2005 290:36
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8576 0.5 3.0 343052 116868 ? S 2005 282:37
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8579 0.5 3.9 635708 153624 ? S 2005 303:06
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8582 0.5 3.2 315260 124844 ? S 2005 280:31
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi
nobody 8585 0.5 3.2 293456 126316 ? S 2005 284:20
/site/perl-5.8.6/bin/perl /site/rt-3.4.2/bin/mason_handler.fcgi

It’s expected that a running Perl process normaly do not free up memory,
but e.g. PID 8204 does not time out at all. Is it because it’s not idle at
all, and too busy working with requests? Should we increase number of
processes? They consume a lot of memory, and I think it just increases
more and more.

top says:

13:44:27 up 204 days, 23:12, 2 users, load average: 0.45, 0.50, 0.65
99 processes: 98 sleeping, 1 running, 0 zombie, 0 stopped
CPU states: cpu user nice system irq softirq iowait idle
total 0.0% 0.0% 0.2% 0.0% 0.0% 0.0% 99.7%
cpu00 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu01 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu02 0.0% 0.0% 0.0% 0.0% 0.0% 0.0% 100.0%
cpu03 0.0% 0.0% 0.9% 0.0% 0.0% 0.0% 99.0%
Mem: 3857224k av, 3321624k used, 535600k free, 0k shrd, 36708k
buff
2736968k actv, 338356k in_d, 15600k in_c
Swap: 4192944k av, 2149104k used, 2043840k free 1772712k
cached

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will have
a closer look at this source. And, the source code looks newer and
maintained. :wink:

Nice little article reference:
http://www.groovie.org/articles/2004/12/18/fast-cgi-with-html-mason

Tomi

–

Tomas A. P. Olaj, email: tomas.olaj@usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group

The rt-users Archives

Be sure to check out the RT Wiki at http://wiki.bestpractical.com

Download a free sample chapter of RT Essentials from O’Reilly Media at
http://rtbook.bestpractical.com

WE’RE COMING TO YOUR TOWN SOON - RT Training in Amsterdam, Boston and
San Francisco - Find out more at
http://bestpractical.com/services/training.html

danno
dan pritts - systems administrator - internet2
734/352-4953 office 734/834-7224 mobile

3

Hi,
[…]

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I
will have a closer look at this source. And, the source code looks
newer and maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

Harald Wagener
Technischer Leiter

Foote Cone & Belding
FCB Wilkens
An der Alster 42
20099 Hamburg
Germany

T: +49 (0)40 2881 1252
F: +49 (0)40 2881 1217
hwagener@hamburg.fcb.com
http://www.footeconebelding.de

4

Harald Wagener wrote:

Am 13.01.2006 um 13:52 schrieb Tomas Olaj:

Hi,

[…]

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will
have a closer look at this source. And, the source code looks newer
and maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

I tried it, and it seemed to work great, until a few users noticed a
place where they could not create a ticket. When I switched it back to
cgi, or fastcgi, it worked again. So much for using that…

Meanwhile, I can the fastcgi processes slowly taking up more and more ram.

Justin Zygmont
System Administrator
Cityfone Telecommunications Inc

direct 604.629.8841
cell 778.239.7215
email justin@cityfone.net

5

On the marvelous Wed, 1 Feb 2006, Justin Zygmont wrote kindly to me …

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will have
a closer look at this source. And, the source code looks newer and
maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

I tried it, and it seemed to work great, until a few users noticed a place
where they could not create a ticket. When I switched it back to cgi, or
fastcgi, it worked again. So much for using that…

Meanwhile, I can the fastcgi processes slowly taking up more and more ram.

I really haven’t got time to test FCGId. Do You say that this software
is not recommendable? Could there be other alternatives than FastCGI?
FastCGI works, but it consumes memory, and I worry because no one
maintains the source code.

Tomas A. P. Olaj, email: tomas.olaj@usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group

6

On the marvelous Wed, 1 Feb 2006, Justin Zygmont wrote kindly to me …

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will have
a closer look at this source. And, the source code looks newer and
maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

I tried it, and it seemed to work great, until a few users noticed a place
where they could not create a ticket. When I switched it back to cgi, or
fastcgi, it worked again. So much for using that…

Meanwhile, I can the fastcgi processes slowly taking up more and more ram.

I really haven’t got time to test FCGId. Do You say that this software
is not recommendable? Could there be other alternatives than FastCGI?
FastCGI works, but it consumes memory, and I worry because no one
maintains the source code.

Tomas A. P. Olaj, email: tomas.olaj@usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

I gave up on FastCGI when I saw what was involved getting it to work
with apache2 under red Hat Enterprose linux. The endless HTML rubbish that
FastCGI will dump into the syslog when it throws an exception
completely defeats the purpose of system logs with one line informative
messages. The punishment for FastCGI’s lack of consideration for my system
was a corresponding lack of consideration of it. I decided to use mod_perl.
At first, the situation with Apache 2.2.0 and mod_perl2 weren’t much better.
The calls to the Apache compatibility library begin with ap_null_handler,
and descended from there to abysmal depths…Apparently, Apache 2.x jettisoned
the apache compatibility library at some point (perhaps Apache 2.1). I would
have had to revert to an earlier Apache 2. I ended up compiling mod_perl1
version 1.26 with apache 1.13.34, since this combination seems to be
the one recommended (ever so slightly more than FastCGI on the bestpractical site),
and mod_perl1 seems to be considered more stable than mod_perl2. This choice
resulted in my first working system, though it could have been influenced by
“learning experiences” I had during the process. This was on a Debian sarge system.
I avoided apt-get and chose to compile the necessary software.

FL

7

On the marvelous Wed, 1 Feb 2006, Harald Wagener wrote kindly to me …

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I will have a
closer look at this source. And, the source code looks newer and
maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

Jesse, what do BP use in their labs? Have BP testet Apache1|2 and
mod_FastCGI|mod_FCGId? But You do recommend fcgi contra mod_perl?

cheers,
Tomas

Tomas A. P. Olaj, email: tomas.olaj@usit.uio.no, web: folk.uio.no/tomaso
University of Oslo / USIT (Center for Information Technology Services)
System- and Application Management / Applications Management Group

8

Tomas Olaj wrote:

On the marvelous Wed, 1 Feb 2006, Justin Zygmont wrote kindly to me …

Jesse mentioned at the Amsterdam class that mod_FCGId is another
alternative (http://fastcgi.coremail.cn/) to mod_FastCGI, and I
will have a closer look at this source. And, the source code looks
newer and maintained. :wink:

Hello,
did anybody use fcgid successfully to replace fascgi? I would love an
rt-specific configuration sample.

Regards,
Harald

I tried it, and it seemed to work great, until a few users noticed a
place where they could not create a ticket. When I switched it back
to cgi, or fastcgi, it worked again. So much for using that…

Meanwhile, I can the fastcgi processes slowly taking up more and more
ram.

I really haven’t got time to test FCGId. Do You say that this software
is not recommendable? Could there be other alternatives than FastCGI?
FastCGI works, but it consumes memory, and I worry because no one
maintains the source code.

It’s really easy to test, I think it would work as a drop in replacement
for fastcgi, but when I tried it, I noticed there was 1 situation where
I used to be able to create a ticket that would not work anymore. When
I switched back to fastcgi, it worked again. so I know fcgid can’t be
100%, it’s too bad, I don’t know why something as useful as fastcgi
wouldn’t be maintained anymore. I’m not sure where the problem was, I
was hoping for feedback from other people about it too, i’d have to
recommend to stay with fastcgi, the memory leak doesnt seem huge, it
just needs a restart once in a while. I can’t wait for RHEL5 with
modperl 2 :slight_smile: I sure hope it will use it.

Justin Zygmont
System Administrator
Cityfone Telecommunications Inc

direct 604.629.8841
cell 778.239.7215
email justin@cityfone.net

9

I switched back to fastcgi, it worked again. so I know fcgid can’t be
100%, it’s too bad, I don’t know why something as useful as fastcgi
wouldn’t be maintained anymore.

When something works there’s not much need to be maintained. I
haven’t had any problems that I’d blame on fastcgi after installing
according to:
http://wiki.bestpractical.com/index.cgi?RHEL4InstallGuide

The one thing that might be different between mod_perl and
fastcgi is that apache has a directive for how many pages
to let a child process serve before killing it and starting
a new one that work with mod_perl processes. You can use this
to balance between speed and memory leaks by forcing the
perl processes to restart regularly. I don’t think fastcgi
has a similar option, although if the program exited on its
own it should restart a new one. Is there any RT option to
say that if it is consuming some amount of memory and has served
some number of pages that it should die at the end of a request?

Les Mikesell
lesmikesell@gmail.com

10

Les Mikesell wrote:> On Mon, 2006-02-06 at 11:52, Justin Zygmont wrote:

I switched back to fastcgi, it worked again. so I know fcgid can’t be
100%, it’s too bad, I don’t know why something as useful as fastcgi
wouldn’t be maintained anymore.

When something works there’s not much need to be maintained. I
haven’t had any problems that I’d blame on fastcgi after installing
according to:
Request Tracker Wiki

The one thing that might be different between mod_perl and
fastcgi is that apache has a directive for how many pages
to let a child process serve before killing it and starting
a new one that work with mod_perl processes. You can use this
to balance between speed and memory leaks by forcing the
perl processes to restart regularly. I don’t think fastcgi
has a similar option, although if the program exited on its
own it should restart a new one. Is there any RT option to
say that if it is consuming some amount of memory and has served
some number of pages that it should die at the end of a request?

If it has a memory leak, there’s a reason to maintain the code, it hasnt
been changed since 2003, which is why some of us are thinking it may no
longer be maintained.

Is that MaxRequestsPerChild ? such a setting for fastcgi could provide
a way to control the problem. Just watch the memory usage for your fcgi
processes, as long as you dont restart apache, you’ll se them slowly get
bigger and bigger.

Justin Zygmont
System Administrator
Cityfone Telecommunications Inc

direct 604.629.8841
cell 778.239.7215
email justin@cityfone.net

11

When something works there’s not much need to be maintained. I
haven’t had any problems that I’d blame on fastcgi after installing
according to:
Request Tracker Wiki

The one thing that might be different between mod_perl and
fastcgi is that apache has a directive for how many pages
to let a child process serve before killing it and starting
a new one that work with mod_perl processes. You can use this
to balance between speed and memory leaks by forcing the
perl processes to restart regularly. I don’t think fastcgi
has a similar option, although if the program exited on its
own it should restart a new one. Is there any RT option to
say that if it is consuming some amount of memory and has served
some number of pages that it should die at the end of a request?

If it has a memory leak, there’s a reason to maintain the code, it hasnt
been changed since 2003, which is why some of us are thinking it may no
longer be maintained.

The memory leak probably isn’t in fastcgi itself. It is just a
framework to start as many instances of some perl program as
you tell it and keep them running.

Is that MaxRequestsPerChild ? such a setting for fastcgi could provide
a way to control the problem. Just watch the memory usage for your fcgi
processes, as long as you dont restart apache, you’ll se them slowly get
bigger and bigger.

Yes, apache uses MaxRequestsPerChild to kill the process regardless
of the language extension and I don’t think there is an equivalent
for fastcgi. However, note that if you kill one of the running
mason_handler.fcgi processes, it will start a new one, so if that
program could be modified to exit gracefully at appropriate times
things would take care of themselves.

Les Mikesell
les@futuresource.com

12

Jesse, what do BP use in their labs? Have BP testet Apache1|2 and
mod_FastCGI|mod_FCGId? But You do recommend fcgi contra mod_perl?

we use mod_fastcgi most of the time. We test mod_perl1, mod_perl2 and
mod_fastcgi here. I’m hearing good things about mod_fcgid, but it’s
still fairly new for me.

Jesse

13

Yes, apache uses MaxRequestsPerChild to kill the process regardless
of the language extension and I don’t think there is an equivalent
for fastcgi. However, note that if you kill one of the running
mason_handler.fcgi processes, it will start a new one, so if that
program could be modified to exit gracefully at appropriate times
things would take care of themselves.

Or just have a shell script run out of cron and kill them occasionally.

danno
dan pritts - systems administrator - internet2
734/352-4953 office 734/834-7224 mobile