RT3 is spawning 2 additional processes every time something is done

I have been running RT3 for several months with no problems whatever. The
hard drive recently went bad on the server and I had to restore everything
from tar files. Not everything copied back automatically so my current
problem could be nothing more than a missing file somewhere.

When I start mysqld, I have 4 /usr/libexec/mysqld --defaults-… processes
running

Every time I do something (log in, click a link, refresh the view), 2 more
mysqld processes are started. This continues until something around 14
processes are running at which time the RT web connection dies. Restarting
mysqld will restore operation again (briefly).

I’m sure that this is going to be a simple fix but unfortunately I am not a
DBA and I have very limited RT experience. As near as I can see,
everything else is working great. RT3 worked perfectly before the disk
problem. The dependencies test for mysql passes. Any help would be hugely
appreciated. Thanks in advance!

Depending on your OS…

For linux each ‘thread’ shows up in the ps output as mysqld process. So for
instance, if you have 20 connections to the db, youll see 20 mysqld
processes. Having a number of mysqld processes isn’t a huge issue unless
you have system resource problems. For example, I manage a website on the
side that gets in the neighborhood of 300,000 page views a month. On any
given day that RedHat box shows at least 60 or so mysqld processes.

For Solaris and I believe AIX and HP (don’t quote me on the last two) you’ll
just see the memory size of the mysqld process get larger but only one
mysqld process.

It could also be a threading issue. For instance, on the later versions of
RedHat, Perl is compiled with threading enabled. I also believe that the
mysql/client binary distributions are compiled with threading and
thread_safe enabled. In this case you would want to make sure that the
webserver is compiled with threading enabled. If using mod_perl, it would
give you a warning about that when compiling as a DSO (external module)
against apxs on apache.

Additionally, mysql has several threading params. First there is
thread_concurrency. Even though I haven’t seen this on the web, I have
several development manuals for mysql 4.x and they all say that this only
applies to Solaris… but wouldn’t hurt to set even if you don’t have it.

There is also thread_cache and thread_stack_size.

For mysql itself you could also try modifying the connection params in the
my.cnf.
max_connections to something respective of how many connections you may
expect
wait_timeout the default is something huge, like 300 seconds or something.
Lower it to 60 seconds or something like that.
interactive_timeout this is also high, you can also try reducing this.

The timeout params will at least clean up connections made to the db as
opposed to holding them.

One more thing to do when you notice bad things happening is:
Either log in as root and : show processlist
Or
./mysqladmin -u -p processlist

Both of these will give you a list of the running processes and what they
are doing.

The mysql config is probably the easiest way to fix the issue. You want to
try and stay away from recompiling apps, especially if you are on a box that
uses RPMs and such.

In the end, the issue is not RT specifically. RT is simply identifying some
problem that has always existed, but not seen until activity gets high.

A.J.-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Frank Saxton
Sent: Thursday, January 08, 2004 1:44 AM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

I have been running RT3 for several months with no problems whatever. The
hard drive recently went bad on the server and I had to restore everything
from tar files. Not everything copied back automatically so my current
problem could be nothing more than a missing file somewhere.

When I start mysqld, I have 4 /usr/libexec/mysqld --defaults-… processes
running

Every time I do something (log in, click a link, refresh the view), 2 more
mysqld processes are started. This continues until something around 14
processes are running at which time the RT web connection dies. Restarting
mysqld will restore operation again (briefly).

I’m sure that this is going to be a simple fix but unfortunately I am not a
DBA and I have very limited RT experience. As near as I can see,
everything else is working great. RT3 worked perfectly before the disk
problem. The dependencies test for mysql passes. Any help would be hugely
appreciated. Thanks in advance!

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

Thanks very much for your thoughtful and detailed reply A.J. But I don’t
think my particular problem is quite that complex.

I am running RT-3.0.1 on Redhat Linux 7.3. I am the only RT user and I
only create a tiny number of tickets a day. And as I mentioned, RT ran
just fine before the hard drive dropped dead. I think I have managed to
restore all of the binaries, lib files, config files, etc., but I suspect
that one or more may still be missing. If there is a way to turn on
tracing or verbose logging, I may be able to get a clue what’s spawning
these additional threads.

The number of additional processes being created isn’t a problem per
se. I’m just noticing that RT quits working after the mysqld PID count
gets up to 14 or so. I never really paid any attention to it before but I
don’t recall seeing 2 new processes being created every time RT refreshed
or any time I clicked any link on the RT web GUI. If I set RT to refresh
every 2 minutes, RT will quit after 5-6 minutes just because each refresh
will spawn 2 new processes.

What does the following command return when things have run amuck?

mysql -e ‘show processlist’

-=| Ben

Ben,

Currently running ‘ok’… I found a couple of missing binaries and now that
I’ve restored them from the tarball, I seem to be holding at 12 mysqld
processes. Don’t know what will happen if I open a second RT GUI though.

I think A.J. may have had an excellent suggestion about wait_timeout
though. It’s set at 28800!!! I tried putting wait_timeout = 60 in the
[mysqld] field but apparently I am using the wrong syntax or I am putting
it in the wrong place. A 5 second crash course on my.cnf would really help!

[root@rt rt3]# /etc/init.d/mysqld status
mysqld (pid 26778 26777 26716 26715 26665 26664 26634 26633 26632 26630) is
runn
ing…
[root@rt rt3]# mysql -e ‘show processlist’
| Id | User | Host | db | Command | Time | State | Info |
| 1 | rt_user | localhost | rt3 | Sleep | 420 | | NULL |
| 2 | rt_user | localhost | rt3 | Sleep | 419 | | NULL |
| 3 | rt_user | localhost | rt3 | Sleep | 277 | | NULL |
| 4 | rt_user | localhost | rt3 | Sleep | 277 | | NULL |
| 7 | rt_user | localhost | rt3 | Sleep | 134 | | NULL |
| 8 | rt_user | localhost | rt3 | Sleep | 134 | | NULL |
| 9 | root | localhost | NULL | Query | 0 | NULL | show processlist |
[root@rt rt3]#

Each httpd process will open a MySQL connection … though MySQL should have
no problem going over 14 with the default config. In any case to set your
default timeout (I suggest this for an application I wrote, too) use the
following in /etc/my.cnf:

[mysqld]
set-variable=wait_timeout=60

and restart mysql.

60 might be a little low, though.

Are you the only one accessing RT? I’m wondering why different httpd’s are
processing your request. Are you running Apache 1.x or 2.x?

It’s also funky that you have 3 sets of database connections, each between
134 and 143 seconds apart. Perhaps apache’s still got some issues? Can you
set the following in your apache config (and restart) and then go to
http://your-server/server-status and paste the results (or summarize… a
paste of the table might be ugly)?

ExtendedStatus On

<Location /server-status>
SetHandler server-status
Order deny,allow
Deny from all
Allow from your.ip.addr.ess

-=| BenFrom: "Frank Saxton" <frankie@easyrider.com>

To: rt-users@lists.bestpractical.com
Sent: Thursday, January 08, 2004 4:48 PM
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

Ben,

Currently running ‘ok’… I found a couple of missing binaries and now
that
I’ve restored them from the tarball, I seem to be holding at 12 mysqld
processes. Don’t know what will happen if I open a second RT GUI though.

I think A.J. may have had an excellent suggestion about wait_timeout
though. It’s set at 28800!!! I tried putting wait_timeout = 60 in the
[mysqld] field but apparently I am using the wrong syntax or I am putting
it in the wrong place. A 5 second crash course on my.cnf would really
help!

[root@rt rt3]# /etc/init.d/mysqld status
mysqld (pid 26778 26777 26716 26715 26665 26664 26634 26633 26632 26630)
is
runn
ing…
[root@rt rt3]# mysql -e ‘show processlist’

| Id | User | Host | db | Command | Time | State | Info
|

| 1 | rt_user | localhost | rt3 | Sleep | 420 | | NULL
|
| 2 | rt_user | localhost | rt3 | Sleep | 419 | | NULL
|
| 3 | rt_user | localhost | rt3 | Sleep | 277 | | NULL
|
| 4 | rt_user | localhost | rt3 | Sleep | 277 | | NULL
|
| 7 | rt_user | localhost | rt3 | Sleep | 134 | | NULL
|
| 8 | rt_user | localhost | rt3 | Sleep | 134 | | NULL
|
| 9 | root | localhost | NULL | Query | 0 | NULL | show
processlist |

[root@rt rt3]#

Thanks ben!

I am running apache-1.3.27-4

Set the wait timeout to 300. If that’s still too short, I can increase it.

And yes, I am the only one running RT and I have just one session logged
in. I have refresh set to very 2 minutes.

Apache appears to be running very well. I have a couple of other
applications that use it and I’m not seeing any problems (that’s not to say
that there might not be possible configuration setting improvements)

Server-status follows:

Server Version: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_python/2.7.8
Python/1.5.2 mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26 mod_throttle/3.1.2
Server Built: Dec 10 2003 05:05:00

Current Time: Thursday, 08-Jan-2004 17:33:01 PST
Restart Time: Thursday, 08-Jan-2004 17:25:21 PST
Parent Server Generation: 0
Server uptime: 7 minutes 40 seconds
Total accesses: 15 - Total Traffic: 67 kB
CPU Usage: u2.88 s.18 cu0 cs0 - .665% CPU load
.0326 requests/sec - 149 B/second - 4573 B/request
1 requests currently being processed, 7 idle servers
_______W…

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 10525 0/2/2 _ 0.50 458 489 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
1-0 10526 0/2/2 _ 0.50 141 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
2-0 10527 0/2/2 _ 0.11 458 0 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1
3-0 10528 0/2/2 _ 0.85 8 1566 0.0 0.02 0.02 216.99.212.113
rt.vigilancemonitoring.com GET /rt/Ticket/Display.html?id=8 HTTP/1.1
4-0 10529 0/2/2 _ 0.50 7 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
5-0 10530 0/2/2 _ 0.49 4 397 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
6-0 10531 0/2/2 _ 0.11 268 112 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
7-0 10532 0/1/1 W 0.00 233 0 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1

cache type: DBM, maximum size: unlimited
current sessions: 1, current size: 152 bytes
average session size: 152 bytes

Apache/1.3.27 Server at rt.vigilancemonitoring.com Port 443

Actually, perhaps someone out there can help, but I remember SOMEWHERE I
read that having a short refresh can cause bad things to happen…

Okay, here it is:
http://lists.fsck.com/pipermail/rt-devel/2001-March/000953.html

http://lists.fsck.com/pipermail/rt-devel/2001-March/001030.html is later but
same thread

Its an old one from someone with a similar problem of apache and spawning
two processes. It might help.

Have you tried
log=/mysqld.log in the my.cnf. Perhaps that will give you more info.

Also, if you are using a binary dist of redhat, there is a ‘support-files’
directory in there with sample my.cnf files for small, medium, large, and
huge dbs…

Grab one, copy it to wherever it needs to be and rename it to my.cnf and see
what happens. :slight_smile:

Back up your existing one.

Last but not least set MaxClients in the httpd.conf to something low and set
MaxRequestsPerChild to 200 or so…

Don’t give apache enough time to run up mysql-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Frank Saxton
Sent: Thursday, January 08, 2004 8:37 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

Thanks ben!

I am running apache-1.3.27-4

Set the wait timeout to 300. If that’s still too short, I can increase it.

And yes, I am the only one running RT and I have just one session logged
in. I have refresh set to very 2 minutes.

Apache appears to be running very well. I have a couple of other
applications that use it and I’m not seeing any problems (that’s not to say
that there might not be possible configuration setting improvements)

Server-status follows:

Server Version: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_python/2.7.8
Python/1.5.2 mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26 mod_throttle/3.1.2
Server Built: Dec 10 2003 05:05:00

Current Time: Thursday, 08-Jan-2004 17:33:01 PST
Restart Time: Thursday, 08-Jan-2004 17:25:21 PST
Parent Server Generation: 0
Server uptime: 7 minutes 40 seconds
Total accesses: 15 - Total Traffic: 67 kB
CPU Usage: u2.88 s.18 cu0 cs0 - .665% CPU load
.0326 requests/sec - 149 B/second - 4573 B/request
1 requests currently being processed, 7 idle servers
_______W…

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 10525 0/2/2 _ 0.50 458 489 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
1-0 10526 0/2/2 _ 0.50 141 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
2-0 10527 0/2/2 _ 0.11 458 0 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1
3-0 10528 0/2/2 _ 0.85 8 1566 0.0 0.02 0.02 216.99.212.113
rt.vigilancemonitoring.com GET /rt/Ticket/Display.html?id=8 HTTP/1.1
4-0 10529 0/2/2 _ 0.50 7 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
5-0 10530 0/2/2 _ 0.49 4 397 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
6-0 10531 0/2/2 _ 0.11 268 112 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
7-0 10532 0/1/1 W 0.00 233 0 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1

cache type: DBM, maximum size: unlimited
current sessions: 1, current size: 152 bytes
average session size: 152 bytes

Apache/1.3.27 Server at rt.vigilancemonitoring.com Port 443

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

A.J.,

Yes, the problem the Author describes in that thread is exactly the problem
I am having (2 mysqld threads being created for every “click”). It wasn’t
clear to me in the discussion that any solution or possible solutions were
proposed.

I do not have any sample my.cnf files on my system so I am flying by the
seat of my pants with some assistance from Ben :slight_smile:

Mysqld logging has already been implemented but nothing interesting is
being logged. Perhaps there is a way to set a higher debug level of logging?

httpd MaxClients is set to 150

httpd MaxRequestsPerChild was set to 1000. Lowered it to 200 at your
suggestion. I don;t know that I am getting anywhere near either of those
numbers.

A.J.,

Yes, the problem the Author describes in that thread is exactly the problem
I am having (2 mysqld threads being created for every “click”). It wasn’t
clear to me in the discussion that any solution or possible solutions were
proposed.

This sounds like your Apache::DBI isn’t recycling connections correctly.
It may be worth disabling Apache::DBI for testing. Not sure if that will
help or hurt you.
2 processes for each click makes perfect sense in that case as for each
click your browser is downloading the page you asked for and RT’s CSS
page.

Request Tracker — Best Practical Solutions – Trouble Ticketing. Free.

I presume you mean commenting out as so:

Perl handler for RT

Alias /rt/ “/opt/rt3/share/html/”
Alias /rt “/opt/rt3/share/html/”
AddDefaultCharset UTF-8

PerlModule Apache::DBI

 PerlRequire /opt/rt3/bin/webmux.pl

<Location /rt>
SetHandler perl-script
PerlHandler RT::Mason

Tried that and httpd died after restarting

At 10:09 PM 1/8/2004 -0500, you wrote:>On Thu, Jan 08, 2004 at 07:06:49PM -0800, Frank Saxton wrote:

A.J.,

Yes, the problem the Author describes in that thread is exactly the
problem
I am having (2 mysqld threads being created for every “click”). It wasn’t
clear to me in the discussion that any solution or possible solutions were
proposed.

This sounds like your Apache::DBI isn’t recycling connections correctly.
It may be worth disabling Apache::DBI for testing. Not sure if that will
help or hurt you.
2 processes for each click makes perfect sense in that case as for each
click your browser is downloading the page you asked for and RT’s CSS
page.


Request Tracker — Best Practical Solutions – Trouble Ticketing. Free.

Each httpd process will open a MySQL connection … though MySQL should
have no problem going over 14 with the default config. In any case to
set your default timeout (I suggest this for an application I wrote, too)
use the following in /etc/my.cnf:

Maybe I’m looking at the wrong thing, and perhaps this behaviour you talk
of is mostly for RT3 (We’re still on RT2), but I only ever see one rt_user
session logged in to MySQL, regardless of how busy my Apache is.

“mysqladmin -v proc” is what I do to verify this. When the system is idle,
I only see:

| 6806 | rt_user | localhost | rt2 | Sleep | 0 | |
|

But the number of processes is never greater than one for the rt_user.
Perhaps I’m confusing the mysql threads and processes with unix processes
(there being 13 mysqld processes running)…I’m not much into databases.

If there’s something definitely wrong with what I’m seeing, that the
mysqladmin -v proc should report many rt_user queries when the system is
"busy" then I’d like to hear about it so I can figure out what’s not tuned
properly.

Thanks,
Scott

Nothing unusual in your apache status. Same type of thing I see. I do
notice that my RT requests ‘round robin’ between 2 sets of apache processes
like yours do. I’m not sure why it does that (perhaps it’s by design, but I
thought HTTP keepalive would keep you hitting the same servers). You should
have 7 or 8 MySQL processes in ‘sleep’ for that apache status you sent.
The MySQL timeout should help, as would the apache config changes others
have emailed about.
When you say ‘the RT web connection dies’ (when you hit 14 mysql procs),
what does that mean, exactly? What message do you get and/or what happens?
Perhaps we’re barking up the wrong tree…

-=| BenFrom: "Frank Saxton" <frankie@easyrider.com>

To: rt-users@lists.bestpractical.com
Sent: Thursday, January 08, 2004 8:36 PM
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

Thanks ben!

I am running apache-1.3.27-4

Set the wait timeout to 300. If that’s still too short, I can increase
it.

And yes, I am the only one running RT and I have just one session logged
in. I have refresh set to very 2 minutes.

Apache appears to be running very well. I have a couple of other
applications that use it and I’m not seeing any problems (that’s not to
say
that there might not be possible configuration setting improvements)

Server-status follows:

Server Version: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_python/2.7.8
Python/1.5.2 mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26 mod_throttle/3.1.2
Server Built: Dec 10 2003 05:05:00


Current Time: Thursday, 08-Jan-2004 17:33:01 PST
Restart Time: Thursday, 08-Jan-2004 17:25:21 PST
Parent Server Generation: 0
Server uptime: 7 minutes 40 seconds
Total accesses: 15 - Total Traffic: 67 kB
CPU Usage: u2.88 s.18 cu0 cs0 - .665% CPU load
.0326 requests/sec - 149 B/second - 4573 B/request
1 requests currently being processed, 7 idle servers
_______W…

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 10525 0/2/2 _ 0.50 458 489 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
1-0 10526 0/2/2 _ 0.50 141 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
2-0 10527 0/2/2 _ 0.11 458 0 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1
3-0 10528 0/2/2 _ 0.85 8 1566 0.0 0.02 0.02 216.99.212.113
rt.vigilancemonitoring.com GET /rt/Ticket/Display.html?id=8 HTTP/1.1
4-0 10529 0/2/2 _ 0.50 7 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
5-0 10530 0/2/2 _ 0.49 4 397 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
6-0 10531 0/2/2 _ 0.11 268 112 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
7-0 10532 0/1/1 W 0.00 233 0 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1

cache type: DBM, maximum size: unlimited
current sessions: 1, current size: 152 bytes
average session size: 152 bytes


Apache/1.3.27 Server at rt.vigilancemonitoring.com Port 443


rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

If you want to check out what mod_perl is doing in more detail:
Make sure Apache::Status, Devel::Symdump, and B::Size perl modules are
installed and then put this in your httpd.conf:

PerlModule Devel::Symdump
PerlModule B::TerseSize
PerlModule Apache::Status
<Location /perl-status>
SetHandler perl-script
PerlHandler Apache::Status
PerlSetVar StatusOptionsAll On

I question whether the issue is actually the number of processes
(mnysqld/httpd) or perhaps something lower level.

What type of system are you running this on? CPU, Memory

Have you looked in /var/log/messages (or it could be /var/log/syslog) and
see if there are any actual system errors.

Also, do a top on the box and watch it. Is it swapping a lot, using all
available memory/shared memory?

When RT crashes try logging into the db from the command line. With the
rt_user and not the root user. If it really is a mysqld problem, it
shouldn’t let you log in.

I have attached two sample my.cnf files from the mysql distribution

[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Ben GoodwinSent: Friday, January 09, 2004 8:22 AM
To: rt-users@lists.bestpractical.com; Frank Saxton
Subject: Re: [rt-users] RT3 is spawning 2 additional processes every
timesomething is done

Nothing unusual in your apache status. Same type of thing I see. I do
notice that my RT requests ‘round robin’ between 2 sets of apache processes
like yours do. I’m not sure why it does that (perhaps it’s by design, but I
thought HTTP keepalive would keep you hitting the same servers). You should
have 7 or 8 MySQL processes in ‘sleep’ for that apache status you sent.
The MySQL timeout should help, as would the apache config changes others
have emailed about.
When you say ‘the RT web connection dies’ (when you hit 14 mysql procs),
what does that mean, exactly? What message do you get and/or what happens?
Perhaps we’re barking up the wrong tree…

-=| Ben

From: “Frank Saxton” frankie@easyrider.com
To: rt-users@lists.bestpractical.com
Sent: Thursday, January 08, 2004 8:36 PM
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

Thanks ben!

I am running apache-1.3.27-4

Set the wait timeout to 300. If that’s still too short, I can increase
it.

And yes, I am the only one running RT and I have just one session logged
in. I have refresh set to very 2 minutes.

Apache appears to be running very well. I have a couple of other
applications that use it and I’m not seeing any problems (that’s not to
say
that there might not be possible configuration setting improvements)

Server-status follows:

Server Version: Apache/1.3.27 (Unix) (Red-Hat/Linux) mod_python/2.7.8
Python/1.5.2 mod_ssl/2.8.12 OpenSSL/0.9.6b DAV/1.0.3 PHP/4.1.2
mod_perl/1.26 mod_throttle/3.1.2
Server Built: Dec 10 2003 05:05:00


Current Time: Thursday, 08-Jan-2004 17:33:01 PST
Restart Time: Thursday, 08-Jan-2004 17:25:21 PST
Parent Server Generation: 0
Server uptime: 7 minutes 40 seconds
Total accesses: 15 - Total Traffic: 67 kB
CPU Usage: u2.88 s.18 cu0 cs0 - .665% CPU load
.0326 requests/sec - 149 B/second - 4573 B/request
1 requests currently being processed, 7 idle servers
_______W…

Srv PID Acc M CPU SS Req Conn Child Slot Client VHost Request
0-0 10525 0/2/2 _ 0.50 458 489 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
1-0 10526 0/2/2 _ 0.50 141 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
2-0 10527 0/2/2 _ 0.11 458 0 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1
3-0 10528 0/2/2 _ 0.85 8 1566 0.0 0.02 0.02 216.99.212.113
rt.vigilancemonitoring.com GET /rt/Ticket/Display.html?id=8 HTTP/1.1
4-0 10529 0/2/2 _ 0.50 7 21 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
5-0 10530 0/2/2 _ 0.49 4 397 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/ HTTP/1.1
6-0 10531 0/2/2 _ 0.11 268 112 0.0 0.01 0.01 216.99.212.113
rt.vigilancemonitoring.com GET /rt/NoAuth/webrt.css HTTP/1.1
7-0 10532 0/1/1 W 0.00 233 0 0.0 0.00 0.00 216.99.212.113
rt.vigilancemonitoring.com GET /server-status HTTP/1.1

cache type: DBM, maximum size: unlimited
current sessions: 1, current size: 152 bytes
average session size: 152 bytes


Apache/1.3.27 Server at rt.vigilancemonitoring.com Port 443


rt-users mailing list
rt-users@lists.bestpractical.com
The rt-users Archives

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

my-small.cnf.sample (2.36 KB)

my-medium.cnf.sample (4.72 KB)

Ben,

By “dying”, I mean that I lose my web connection to the RT GUI. I get the
standard Apache “server problem, see your Administrator” page. After
implementing the suggestions made here, mysqld seems to be holding at 8
connections and no further failures have been seen. I now have 2 RT
sessions going and mysqld is still at 8 connections. This is good :slight_smile:

That is good. So for everyone who may have been watching this and for people
who may have the same problem:
What changes did you make exactly?-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Frank Saxton
Sent: Friday, January 09, 2004 2:46 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

Ben,

By “dying”, I mean that I lose my web connection to the RT GUI. I get the
standard Apache “server problem, see your Administrator” page. After
implementing the suggestions made here, mysqld seems to be holding at 8
connections and no further failures have been seen. I now have 2 RT
sessions going and mysqld is still at 8 connections. This is good :slight_smile:

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm

I’m not certain what actually fixed it, but reducing the time_wait value
from 8 hours to 5 minutes certainly helped.

This is a dual 2.5 GHz Pentium Linux server with 1 Gig of RAM (soon to be
upgraded to 2 Gig). Rarely see much swapping and the load average runs
about 1.0 most of the time.

Not able to view your my.cnf example which came across as an exe file. I
seem to be running fairly well now (Thanks especially to Ben!!! Thank you,
thank you, thank you!) but I’m sure mysqld could stand some tuning.

This has been my first experience with this forum and you guys have been
awesome!

Well, with that kind of hardware, it’s DEFINTELY is not a system issue. :slight_smile:

The site I mentioned before is a single 1.7 ghz Celeron running rehat 7.3
and mysql 3.23.54 and I have mysql set to 50 max connections (50 processes)
and it runs fine (although load average can get pretty high) with persistent
connections enabled in php.

You can do a mysqladmin -u root -p variables and see what the settings are.

Since setting the wait_timeout lower helped look at max_connections,
max_delayed_threads, open_files_limit (if using MyISAM tables which is RT
default should be set to 0), table_cache, and thread_stack (should be set to
65536) as well.

Also try:
mysqladmin -u -p status
pay attention to ‘Open tables’ and make sure the table_cache value is higher
than the open_tables value.

Here are they mysql settings I have. My timeouts are low because the system
cant handle too much so you don’t necessarily have to set your interactive
timeout that low. In fact, with a 2-3 minute refresh, you probably
shouldn’t. If anyone else has any suggestions for improving these values I
would be very happy.

One last thing, drop by your local Borders or however you typically get
books and pick up “Mysql, Second Edition” by Paul DuBios. It is by far the
best Mysql overall reference I have found, covering everything from sql and
query tuning to mysql performance tuning. Even though it talks a lot about
4.x it covers 3.x very well.

Have a great week and prosperous new year

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
set-variable = key_buffer=12M
set-variable = max_allowed_packet=4M
set-variable = table_cache=256
set-variable = sort_buffer=8M
set-variable = record_buffer=8M
set-variable = myisam_sort_buffer_size=12M
set-variable = thread_cache=20
set-variable = max_connections=50
set-variable = wait_timeout=60
set-variable = interactive_timeout=60
[mysql.server]
user=mysql
basedir=/var/lib

[safe_mysqld]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

[mysql]
#no-auto-rehash

Remove the next comment character if you are not familiar with SQL

#safe-updates

[isamchk]
set-variable = key_buffer=12M
set-variable = sort_buffer=12M
set-variable = read_buffer=12M
set-variable = write_buffer=2M

[myisamchk]
set-variable = key_buffer=12M
set-variable = sort_buffer=12M
set-variable = read_buffer=12M
set-variable = write_buffer=2M-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Frank Saxton
Sent: Friday, January 09, 2004 9:26 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] RT3 is spawning 2 additional processes every time
something is done

I’m not certain what actually fixed it, but reducing the time_wait value
from 8 hours to 5 minutes certainly helped.

This is a dual 2.5 GHz Pentium Linux server with 1 Gig of RAM (soon to be
upgraded to 2 Gig). Rarely see much swapping and the load average runs
about 1.0 most of the time.

Not able to view your my.cnf example which came across as an exe file. I
seem to be running fairly well now (Thanks especially to Ben!!! Thank you,
thank you, thank you!) but I’m sure mysqld could stand some tuning.

This has been my first experience with this forum and you guys have been
awesome!

rt-users mailing list
rt-users@lists.bestpractical.com
http://lists.bestpractical.com/mailman/listinfo/rt-users

Have you read the FAQ? The RT FAQ Manager lives at http://fsck.com/rtfm