CPU Wait Maxed Out

[Eriks_Goodwin-Pfiste]

Of late, our RT server (CentOS 5.2 running RT ) has started maxing out “%wa” as monitored in top (avg: 90-99%). I can’t seem to find anything on point with regard to this issue and RT. Since this is a dedicated server, I assume it is related to RT somehow, but can’t seem to figure it out. Can someone point me in the right direction?

Thanks!
–Eriks

[William_Graboyes]

Hi Eriks,

Do you know what process is taking up the cpu?

I know that wa is a process that is waiting for i/o. But I think knowing
which process or processes are causing these problems may help to point in
the right direction.

Thanks,
BillOn Mon, May 31, 2010 at 22:52, Eriks Goodwin-Pfister eriks@centritech.netwrote:

Of late, our RT server (CentOS 5.2 running RT ) has started maxing out
“%wa” as monitored in top (avg: 90-99%). I can’t seem to find anything on
point with regard to this issue and RT. Since this is a dedicated server, I
assume it is related to RT somehow, but can’t seem to figure it out. Can
someone point me in the right direction?

Thanks!
–Eriks

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Eriks_Goodwin-Pfiste]

That’s the kicker, nothing looks “abnormal” in the top listing or processes to me. Perhaps you see something I don’t?

I have included:

1. results of “ps -auxww”
2. results of “top -b -i -n 20 >> ./top_procs”
3. odd entries in "mailq "

Thanks in advance for any more advice…

–Eriks

[root@support ~]# ps -auxww
Warning: bad syntax, perhaps a bogus ‘-’? See /usr/share/doc/procps-3.2.7/FAQ
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 2072 512 ? Ss May29 0:01 init [3]
root 2 0.0 0.0 0 0 ? S< May29 0:00 [migration/0]
root 3 0.0 0.0 0 0 ? SN May29 0:00 [ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S< May29 0:00 [watchdog/0]
root 5 0.0 0.0 0 0 ? S< May29 0:00 [events/0]
root 6 0.0 0.0 0 0 ? S< May29 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S< May29 0:00 [kthread]
root 10 0.0 0.0 0 0 ? S< May29 0:16 [kblockd/0]
root 11 0.0 0.0 0 0 ? S< May29 0:00 [kacpid]
root 93 0.0 0.0 0 0 ? S< May29 0:00 [cqueue/0]
root 96 0.0 0.0 0 0 ? S< May29 0:00 [khubd]
root 98 0.0 0.0 0 0 ? S< May29 0:00 [kseriod]
root 159 0.0 0.0 0 0 ? S May29 0:00 [khungtaskd]
root 162 0.1 0.0 0 0 ? S< May29 6:52 [kswapd0]
root 163 0.0 0.0 0 0 ? S< May29 0:00 [aio/0]
root 315 0.0 0.0 0 0 ? S< May29 0:00 [kpsmoused]
root 337 0.0 0.0 0 0 ? S< May29 0:00 [ata/0]
root 338 0.0 0.0 0 0 ? S< May29 0:00 [ata_aux]
root 341 0.0 0.0 0 0 ? S< May29 0:00 [scsi_eh_0]
root 342 0.0 0.0 0 0 ? S< May29 0:00 [scsi_eh_1]
root 347 0.0 0.0 0 0 ? S< May29 0:00 [kstriped]
root 356 0.0 0.0 0 0 ? S< May29 0:00 [ksnapd]
root 367 0.0 0.0 0 0 ? D< May29 0:38 [kjournald]
root 392 0.0 0.0 0 0 ? S< May29 0:00 [kauditd]
root 425 0.0 0.0 2280 364 ? S<s May29 0:00 /sbin/udevd -d
apache 630 4.9 4.7 213088 46192 ? D 07:45 3:16 /usr/sbin/httpd
root 1139 0.0 0.0 0 0 ? S< May29 0:00 [kmpathd/0]
root 1140 0.0 0.0 0 0 ? S< May29 0:00 [kmpath_handlerd]
root 1162 0.0 0.0 0 0 ? S< May29 0:00 [kjournald]
root 1682 0.0 0.0 12520 596 ? S<sl May29 0:00 auditd
root 1684 0.0 0.0 13100 600 ? S<sl May29 0:00 /sbin/audispd
root 1714 0.0 0.0 1728 492 ? Ss May29 0:04 syslogd -m 0
root 1717 0.0 0.0 1680 312 ? Ss May29 0:00 klogd -x
root 1753 0.0 0.0 0 0 ? S< May29 0:29 [kondemand/0]
rpc 1788 0.0 0.0 1816 376 ? Ss May29 0:00 portmap
root 1818 0.0 0.0 0 0 ? S< May29 0:00 [rpciod/0]
rpcuser 1824 0.0 0.0 1868 480 ? Ss May29 0:00 rpc.statd
root 1856 0.0 0.0 5820 228 ? Ss May29 0:00 rpc.idmapd
dbus 1879 0.0 0.0 2748 764 ? Ss May29 0:00 dbus-daemon --system
root 1892 0.0 0.0 2172 396 ? Ss May29 0:00 /usr/sbin/hcid
root 1898 0.0 0.0 1748 372 ? Ss May29 0:00 /usr/sbin/sdpd
apache 1901 3.4 6.7 204920 65568 ? S 08:16 1:12 /usr/sbin/httpd
apache 1902 1.1 3.0 137472 29992 ? S 08:16 0:23 /usr/sbin/httpd
apache 1903 2.1 2.2 208788 22224 ? S 08:16 0:45 /usr/sbin/httpd
root 1919 0.0 0.0 0 0 ? S< May29 0:00 [krfcommd]
root 1941 0.0 0.0 12868 548 ? Ssl May29 0:00 pcscd
root 1955 0.0 0.0 1676 376 ? Ss May29 0:00 /usr/sbin/acpid
68 1972 0.0 0.1 5628 992 ? Ss May29 0:00 hald
root 1973 0.0 0.0 3164 496 ? S May29 0:00 hald-runner
68 1980 0.0 0.0 2020 472 ? S May29 0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 1988 0.0 0.0 1976 356 ? S May29 0:04 hald-addon-storage: polling /dev/hdc
root 2008 0.0 0.0 1916 348 ? Ss May29 0:00 /usr/bin/hidd --server
root 2032 0.0 0.0 27260 720 ? Ssl May29 0:00 automount
root 2056 0.0 0.0 7080 672 ? Ss May29 0:00 /usr/sbin/sshd
root 2069 0.0 0.0 10128 676 ? Ss May29 0:00 cupsd
root 2109 0.0 0.0 4536 504 ? S May29 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql --socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log --pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql 2159 0.0 0.3 149024 3716 ? Sl May29 0:18 /usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-external-locking --socket=/var/lib/mysql/mysql.sock
root 2196 0.0 0.0 9232 948 ? Ss May29 0:07 sendmail: accepting connections
smmsp 2204 0.0 0.0 8148 688 ? Ss May29 0:00 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 2218 0.0 0.0 1908 244 ? Ss May29 0:00 gpm -m /dev/input/mice -t exps2
root 2286 0.0 0.0 5292 552 ? Ss May29 0:00 crond
xfs 2309 0.0 0.0 3168 524 ? Ss May29 0:00 xfs -droppriv -daemon
root 2334 0.0 0.0 2268 336 ? Ss May29 0:00 /usr/sbin/atd
avahi 2360 0.0 0.0 2600 560 ? Ss May29 0:00 avahi-daemon: running [support.local]
avahi 2361 0.0 0.0 2600 116 ? Ss May29 0:00 avahi-daemon: chroot helper
root 2861 0.0 0.0 3516 348 ? S May29 0:00 /usr/sbin/smartd -q never
root 2868 0.0 0.0 1664 316 tty1 Ss+ May29 0:00 /sbin/mingetty tty1
root 2869 0.0 0.0 1664 316 tty2 Ss+ May29 0:00 /sbin/mingetty tty2
root 2872 0.0 0.0 1664 316 tty3 Ss+ May29 0:00 /sbin/mingetty tty3
root 2875 0.0 0.0 1664 316 tty4 Ss+ May29 0:00 /sbin/mingetty tty4
root 2877 0.0 0.0 1664 316 tty5 Ss+ May29 0:00 /sbin/mingetty tty5
root 2878 0.0 0.0 1664 316 tty6 Ss+ May29 0:00 /sbin/mingetty tty6
root 2926 0.0 0.2 26108 2768 ? SN May29 0:02 /usr/bin/python -tt /usr/sbin/yum-updatesd
root 2928 0.0 0.0 2564 628 ? SN May29 0:00 /usr/libexec/gam_server
root 3012 0.0 0.2 9512 2636 ? S 08:45 0:00 sendmail: ./o51CGaSM001908 from queue
root 3023 0.0 0.2 9932 2844 ? Ss 08:45 0:00 sshd: root@pts/0
root 3026 0.0 0.1 4660 1436 pts/0 Ss 08:45 0:00 -bash
root 3093 0.0 0.2 9432 2196 ? S 08:48 0:00 sendmail: ./o51CmDqe003090 from queue
mail 3094 0.1 0.9 10808 8880 ? Ss 08:48 0:00 /usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk --action correspond --url http://support.unimatrix.net
mail 3095 0.2 0.9 10808 8844 ? Ss 08:48 0:00 /usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk --action correspond --url http://support.unimatrix.net
root 3097 0.0 0.2 9440 2212 ? S 08:48 0:00 sendmail: ./o51CmICW003096 from queue
mail 3098 0.2 0.9 10808 8844 ? Ss 08:48 0:00 /usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk --action correspond --url http://support.unimatrix.net
root 3105 0.0 0.0 0 0 ? S 08:49 0:00 [pdflush]
root 3107 0.0 0.2 9432 2136 ? S 08:49 0:00 sendmail: ./o51CnAWd003106 from queue
mail 3108 0.3 0.9 10808 8844 ? Ss 08:49 0:00 /usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk --action correspond --url http://support.unimatrix.net
root 3111 0.0 0.0 0 0 ? S 08:50 0:00 [pdflush]
root 3113 0.2 0.4 9568 4332 ? DN 08:50 0:00 /usr/bin/python -tt /usr/libexec/yum-updatesd-helper --check --dbus
root 3114 0.0 0.0 4260 944 pts/0 R+ 08:51 0:00 ps -auxww
root 22634 0.0 0.0 4868 484 ? S 02:12 0:00 /usr/sbin/nss_pcache off /etc/httpd/alias
root 22636 0.0 1.0 118420 9916 ? Ssl 02:12 0:04 /usr/sbin/httpd
apache 22641 4.7 7.3 211988 71428 ? D 02:12 19:01 /usr/sbin/httpd
apache 22642 4.7 10.4 227944 101056 ? D 02:12 19:08 /usr/sbin/httpd
apache 22643 4.9 7.0 212152 68752 ? D 02:12 19:38 /usr/sbin/httpd
apache 22644 4.5 7.8 207620 75932 ? S 02:12 18:10 /usr/sbin/httpd
apache 22645 4.3 7.7 218668 74904 ? D 02:12 17:15 /usr/sbin/httpd
apache 22646 4.7 3.7 214824 36704 ? S 02:12 19:03 /usr/sbin/httpd
apache 22647 4.9 7.9 210512 76936 ? S 02:12 19:43 /usr/sbin/httpd
apache 22648 4.3 10.0 215600 97432 ? S 02:12 17:29 /usr/sbin/httpd
apache 26071 7.8 11.4 216356 111292 ? S 04:45 19:17 /usr/sbin/httpd

top - 08:48:04 up 2 days, 21:03, 1 user, load average: 6.04, 3.78, 2.64
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.5%us, 1.1%sy, 0.2%ni, 33.1%id, 45.0%wa, 0.3%hi, 0.7%si, 0.0%st
Mem: 970084k total, 960168k used, 9916k free, 2844k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30420k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 49m 3076 D 1.2 5.2 1:09.12 httpd
22642 apache 16 0 222m 59m 3116 D 1.2 6.3 19:04.86 httpd
22644 apache 18 0 204m 74m 3116 D 1.2 7.9 18:08.90 httpd
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.52 kswapd0
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
1988 root 15 0 1976 356 340 R 0.0 0.0 0:04.13 hald-addon-stor
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3081 apache 17 0 8176 3060 2092 D 0.0 0.3 0:00.05 sendmail
3082 root 15 0 2200 896 696 R 0.0 0.1 0:00.00 top

top - 08:48:07 up 2 days, 21:04, 1 user, load average: 6.28, 3.86, 2.67
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 29.9%us, 1.9%sy, 0.0%ni, 0.0%id, 66.4%wa, 0.3%hi, 1.6%si, 0.0%st
Mem: 970084k total, 960012k used, 10072k free, 2848k buffers
Swap: 1966072k total, 983092k used, 982980k free, 30928k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 74m 3116 R 30.6 7.9 18:09.99 httpd
1901 apache 17 0 201m 50m 3076 D 2.2 5.3 1:09.20 httpd
3084 root 16 0 9276 2300 1564 D 1.4 0.2 0:00.05 sendmail
22642 apache 16 0 222m 61m 3116 D 0.6 6.5 19:04.88 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.53 kswapd0
3082 root 15 0 2208 992 784 R 0.3 0.1 0:00.01 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
top - 08:48:10 up 2 days, 21:04, 1 user, load average: 6.66, 3.98, 2.71
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 23.7%us, 2.3%sy, 0.0%ni, 0.0%id, 72.3%wa, 0.3%hi, 1.3%si, 0.0%st
Mem: 970084k total, 959132k used, 10952k free, 2872k buffers
Swap: 1966072k total, 983076k used, 982996k free, 31244k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 75m 3116 D 20.6 8.0 18:10.61 httpd
1901 apache 17 0 201m 52m 3076 D 2.7 5.5 1:09.28 httpd
22642 apache 16 0 222m 63m 3116 D 0.7 6.7 19:04.90 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2348 996 784 R 0.0 0.1 0:00.01 top
3084 root 16 0 9276 2364 1616 D 0.0 0.2 0:00.05 sendmail

top - 08:48:13 up 2 days, 21:04, 1 user, load average: 6.66, 3.98, 2.71
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.6%us, 2.3%sy, 0.0%ni, 0.0%id, 88.7%wa, 1.0%hi, 1.3%si, 0.0%st
Mem: 970084k total, 955480k used, 14604k free, 2920k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31000k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 53m 3076 D 2.7 5.7 1:09.36 httpd
22642 apache 16 0 222m 65m 3116 D 1.0 6.9 19:04.93 httpd
3085 apache 17 0 8176 3080 2116 D 0.7 0.3 0:00.05 sendmail
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.02 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
22647 apache 18 0 207m 76m 3124 D 0.0 8.1 19:43.79 httpd

top - 08:48:16 up 2 days, 21:04, 1 user, load average: 6.52, 4.00, 2.73
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.6%us, 5.0%sy, 0.0%ni, 0.0%id, 71.1%wa, 1.0%hi, 1.3%si, 0.0%st
Mem: 970084k total, 949180k used, 20904k free, 3044k buffers
Swap: 1966072k total, 982932k used, 983140k free, 29604k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3094 mail 19 0 7352 5544 1704 R 6.3 0.6 0:00.19 rt-mailgate
1901 apache 17 0 201m 56m 3076 D 6.0 5.9 1:09.54 httpd
22642 apache 16 0 222m 70m 3116 D 3.3 7.5 19:05.03 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.03 top
3016 mail 16 0 10808 8664 1940 R 0.0 0.9 0:00.30 rt-mailgate

top - 08:48:19 up 2 days, 21:04, 1 user, load average: 6.48, 4.03, 2.74
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.7%us, 3.0%sy, 0.0%ni, 0.0%id, 74.0%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 956980k used, 13104k free, 3084k buffers
Swap: 1966072k total, 983140k used, 982932k free, 30380k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 57m 3076 D 2.7 6.1 1:09.62 httpd
22642 apache 16 0 222m 73m 3116 D 2.3 7.8 19:05.10 httpd
162 root 11 -5 0 0 0 D 0.3 0.0 6:51.59 kswapd0
22641 apache 18 0 208m 10m 3116 D 0.3 1.1 19:00.45 httpd
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.03 top
3096 root 15 0 9300 2512 1712 R 0.0 0.3 0:00.00 sendmail
22645 apache 16 0 211m 10m 3124 R 0.0 1.1 17:13.98 httpd

top - 08:48:22 up 2 days, 21:04, 1 user, load average: 6.48, 4.03, 2.74
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 1.1%sy, 0.0%ni, 0.0%id, 93.8%wa, 0.5%hi, 1.6%si, 0.0%st
Mem: 970084k total, 960256k used, 9828k free, 3092k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30920k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3096 root 16 0 9300 2512 1712 D 2.3 0.3 0:00.07 sendmail
22642 apache 16 0 222m 75m 3116 D 1.7 8.0 19:05.15 httpd
1901 apache 17 0 201m 58m 3076 D 1.0 6.2 1:09.65 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.60 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.04 top
22641 apache 18 0 208m 11m 3116 D 0.3 1.2 19:00.46 httpd
22645 apache 16 0 211m 10m 3124 D 0.3 1.1 17:13.99 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush

top - 08:48:26 up 2 days, 21:04, 1 user, load average: 6.68, 4.11, 2.78
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 1.3%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi, 0.7%si, 0.0%st
Mem: 970084k total, 960216k used, 9868k free, 3092k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30996k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 59m 3076 D 2.4 6.3 1:09.74 httpd
22642 apache 16 0 222m 77m 3116 D 1.6 8.2 19:05.21 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.61 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.05 top
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.00 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1760 928 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 11m 3116 R 0.0 1.2 19:00.46 httpd

top - 08:48:29 up 2 days, 21:04, 1 user, load average: 6.68, 4.11, 2.78
Tasks: 99 total, 1 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 1.0%sy, 0.0%ni, 0.0%id, 93.7%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 960192k used, 9892k free, 3092k buffers
Swap: 1966072k total, 983160k used, 982912k free, 30960k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 60m 3076 D 2.7 6.4 1:09.82 httpd
22642 apache 16 0 222m 78m 3116 D 1.0 8.3 19:05.24 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.63 kswapd0
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.01 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.05 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1764 932 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 12m 3116 D 0.0 1.3 19:00.46 httpd

top - 08:48:32 up 2 days, 21:04, 1 user, load average: 6.87, 4.19, 2.81
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.3%us, 2.7%sy, 0.0%ni, 0.0%id, 83.0%wa, 0.3%hi, 1.7%si, 0.0%st
Mem: 970084k total, 960100k used, 9984k free, 3028k buffers
Swap: 1966072k total, 983072k used, 983000k free, 31044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3098 mail 18 0 7352 5504 1656 D 6.7 0.6 0:00.20 rt-mailgate
1901 apache 18 0 201m 61m 3076 D 4.3 6.5 1:09.95 httpd
22642 apache 16 0 222m 80m 3116 D 1.7 8.5 19:05.29 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.65 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.06 top
22641 apache 18 0 208m 12m 3116 D 0.3 1.3 19:00.47 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 11m 3124 D 0.0 1.2 17:14.01 httpd

top - 08:48:35 up 2 days, 21:04, 1 user, load average: 7.12, 4.29, 2.85
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.9%us, 1.7%sy, 0.0%ni, 0.0%id, 86.8%wa, 0.3%hi, 1.3%si, 0.0%st
Mem: 970084k total, 960128k used, 9956k free, 3028k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30988k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 4.0 6.6 1:10.07 httpd
22642 apache 16 0 222m 81m 3116 D 1.3 8.6 19:05.33 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.66 kswapd0
22641 apache 18 0 208m 12m 3116 D 0.3 1.4 19:00.48 httpd
22645 apache 16 0 211m 12m 3124 D 0.3 1.3 17:14.02 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.06 top
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd

top - 08:48:38 up 2 days, 21:04, 1 user, load average: 7.12, 4.29, 2.85
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.3%us, 1.3%sy, 0.0%ni, 0.0%id, 90.0%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 959856k used, 10228k free, 3024k buffers
Swap: 1966072k total, 985244k used, 980828k free, 31032k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 6.3 6.6 1:10.26 httpd
22642 apache 16 0 222m 82m 3116 D 1.7 8.7 19:05.38 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.68 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.07 top
22641 apache 18 0 208m 13m 3116 D 0.3 1.4 19:00.49 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd
22645 apache 16 0 211m 12m 3124 D 0.0 1.3 17:14.02 httpd

top - 08:48:41 up 2 days, 21:04, 1 user, load average: 7.27, 4.37, 2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.3%us, 1.3%sy, 0.0%ni, 0.0%id, 87.0%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 959516k used, 10568k free, 3016k buffers
Swap: 1966072k total, 983232k used, 982840k free, 31028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 9.0 6.6 1:10.53 httpd
22642 apache 17 0 222m 83m 3116 D 1.7 8.8 19:05.43 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.69 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.08 top
22643 apache 16 0 207m 12m 3116 D 0.3 1.4 19:36.91 httpd
22645 apache 16 0 211m 13m 3124 D 0.3 1.4 17:14.03 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 18 0 208m 13m 3116 D 0.0 1.4 19:00.49 httpd

top - 08:48:44 up 2 days, 21:04, 1 user, load average: 7.27, 4.37, 2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.0%us, 1.3%sy, 0.0%ni, 0.0%id, 83.4%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 959932k used, 10152k free, 3012k buffers
Swap: 1966072k total, 983124k used, 982948k free, 31024k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 13.2 6.7 1:10.93 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 8.9 19:05.46 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.70 kswapd0
22641 apache 16 0 208m 14m 3116 D 0.3 1.5 19:00.50 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.08 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.91 httpd

top - 08:48:47 up 2 days, 21:04, 1 user, load average: 7.33, 4.43, 2.91
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 41.9%us, 1.7%sy, 0.0%ni, 0.0%id, 54.5%wa, 0.7%hi, 1.3%si, 0.0%st
Mem: 970084k total, 960020k used, 10064k free, 2992k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31048k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 41.9 6.7 1:12.19 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 9.0 19:05.49 httpd
22643 apache 16 0 207m 13m 3116 D 0.7 1.4 19:36.93 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.09 top
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.70 kswapd0
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 16 0 208m 14m 3116 D 0.0 1.5 19:00.50 httpd
22645 apache 16 0 211m 14m 3124 D 0.0 1.5 17:14.04 httpd

top - 08:48:50 up 2 days, 21:04, 1 user, load average: 7.46, 4.50, 2.94
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 1.7%sy, 0.0%ni, 0.0%id, 93.1%wa, 0.3%hi, 1.7%si, 0.0%st
Mem: 970084k total, 960184k used, 9900k free, 2992k buffers
Swap: 1966072k total, 984032k used, 982040k free, 31180k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 2.0 6.8 1:12.25 httpd
22642 apache 17 0 222m 85m 3116 R 1.0 9.0 19:05.52 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.72 kswapd0
22645 apache 16 0 211m 14m 3124 D 0.7 1.6 17:14.06 httpd
22641 apache 16 0 208m 15m 3116 D 0.3 1.6 19:00.51 httpd
367 root 11 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.09 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.93 httpd

top - 08:48:53 up 2 days, 21:04, 1 user, load average: 7.46, 4.50, 2.94
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.3%us, 1.7%sy, 0.0%ni, 0.0%id, 91.7%wa, 0.7%hi, 0.7%si, 0.0%st
Mem: 970084k total, 960156k used, 9928k free, 2992k buffers
Swap: 1966072k total, 983068k used, 983004k free, 31464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 3.5 6.9 1:12.37 httpd
22642 apache 17 0 222m 85m 3116 D 1.4 9.1 19:05.57 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.73 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.10 top
22641 apache 16 0 208m 15m 3116 D 0.3 1.7 19:00.52 httpd
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.94 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 15m 3124 D 0.0 1.6 17:14.06 httpd

top - 08:48:56 up 2 days, 21:04, 1 user, load average: 7.50, 4.56, 2.97
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.2%us, 0.9%sy, 0.0%ni, 0.0%id, 96.5%wa, 0.6%hi, 0.9%si, 0.0%st
Mem: 970084k total, 960380k used, 9704k free, 2936k buffers
Swap: 1966072k total, 983044k used, 983028k free, 31392k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.0 9.1 19:05.60 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.74 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.11 top
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.95 httpd
22645 apache 16 0 211m 15m 3124 D 0.3 1.6 17:14.07 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3099 apache 17 0 8172 3052 2092 D 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.7 19:00.52 httpd

top - 08:49:00 up 2 days, 21:04, 1 user, load average: 7.62, 4.63, 3.00
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 0.0%id, 95.3%wa, 0.3%hi, 1.7%si, 0.0%st
Mem: 970084k total, 959064k used, 11020k free, 2936k buffers
Swap: 1966072k total, 983532k used, 982540k free, 31148k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.4 9.2 19:05.65 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.6 19:36.96 httpd
22645 apache 16 0 211m 16m 3124 D 0.3 1.7 17:14.08 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.11 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3099 apache 17 0 8172 3076 2116 R 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.8 19:00.52 httpd

top - 08:49:03 up 2 days, 21:04, 1 user, load average: 7.62, 4.63, 3.00
Tasks: 103 total, 2 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 1.7%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi, 1.0%si, 0.0%st
Mem: 970084k total, 959560k used, 10524k free, 2928k buffers
Swap: 1966072k total, 983440k used, 982632k free, 30496k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 87m 3116 D 2.0 9.2 19:05.71 httpd
3102 root 16 0 9276 2300 1564 D 1.3 0.2 0:00.04 sendmail
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.77 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.12 top
22641 apache 16 0 208m 17m 3116 D 0.3 1.8 19:00.53 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.7 19:36.97 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
22645 apache 16 0 211m 16m 3124 R 0.0 1.7 17:14.08 httpd

[root@support ~]# mailq
/var/spool/mqueue (6 requests)
-----Q-ID----- --Size-- -----Q-Time----- ------------Sender/Recipient-----------
o51Cxkw2003214 8112 Tue Jun 1 08:59 jay@centritech.net
7BIT (Deferred: prog mailer (/usr/sbin/smrsh) exited with EX_TEMPF)
“|/etc/smrsh/rt-mailgate --queue 'ccs:
o51CuKTF003192 633211 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate --queue 'umx:
o51CGaSM001908 1170766 Tue Jun 1 08:16 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate --queue 'umx:
o51CuIdl003186 1355798 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate --queue 'umx:
o51Ckond003072 3612709 Tue Jun 1 08:46 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate --queue 'umx:
o51CutnV003202 3618441 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client helpdesk’ --ac)
"|/etc/smrsh/rt-mailgate --queue 'umx:
Total requests: 6
[root@support ~]#

[Eriks_Goodwin-Pfiste]

[root@support ~]# vmstat 5
procs -----------memory---------- —swap-- -----io---- --system-- -----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 2 983748 48248 3472 14964 556 259 605 448 1 59 20 2 33 45 0
1 2 983572 17100 2824 22092 2367 501 2367 2045 1276 866 45 9 0 47 0
0 2 983036 18208 2936 22112 1085 73 1101 611 1359 375 27 5 0 68 0
0 2 983036 11340 2944 22488 1306 0 1306 265 1199 442 22 2 0 77 0
0 3 983748 11296 2596 22656 790 650 793 861 1252 373 39 6 0 55 0
0 3 983340 11144 2640 21908 887 234 914 487 1346 445 13 4 0 83 0
1 2 983036 11208 2536 22268 659 49 659 249 1142 284 46 2 0 52 0
0 3 983616 11428 2236 22188 911 581 970 1010 1392 512 19 4 0 76 0
0 2 983836 12452 2236 22548 640 739 640 962 1224 335 19 3 0 78 0
0 2 983136 11516 2184 22596 824 428 824 566 1241 357 5 2 0 92 0
3 1 983036 11216 1928 21544 802 482 802 508 1218 278 18 2 0 80 0
0 4 984124 12052 1944 22168 630 1196 630 1413 1283 270 23 3 0 74 0
1 1 983828 14684 2080 21628 506 548 546 842 1347 354 22 4 0 74 0
1 2 983228 11896 2096 22312 973 440 973 718 1204 378 25 3 0 72 0
0 2 983036 15360 2136 22060 430 294 430 586 1177 252 39 3 0 59 0
0 2 983036 13252 2212 22516 628 0 628 190 1271 372 39 3 0 57 0
0 1 983036 23836 2296 21768 597 17 597 622 1216 331 53 4 0 43 0
0 2 983036 11968 2360 23068 765 0 814 167 1215 376 42 3 0 55 0From: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:00:49 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
You have run out of memory.
What does vmstat 5 show?

Eriks Goodwin-Pfister wrote:

That’s the kicker, nothing looks “abnormal” in the top listing or
processes to me. Perhaps you see something I don’t?

I have included:

1. results of “ps -auxww”
2. results of “top -b -i -n 20 >> ./top_procs”
3. odd entries in “mailq”

Thanks in advance for any more advice…

–Eriks

======================================

[root@support ~]# ps -auxww
Warning: bad syntax, perhaps a bogus ‘-’? See
/usr/share/doc/procps-3.2.7/FAQ USER PID %CPU %MEM VSZ RSS TTY STAT
START TIME COMMAND
root 1 0.0 0.0 2072 512 ? Ss May29 0:01 init
[3] root 2 0.0 0.0 0 0 ? S< May29 0:00
[migration/0] root 3 0.0 0.0 0 0 ? SN May29 0:00
[ksoftirqd/0] root 4 0.0 0.0 0 0 ? S< May29 0:00
[watchdog/0] root 5 0.0 0.0 0 0 ? S< May29 0:00
[events/0] root 6 0.0 0.0 0 0 ? S< May29 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S< May29 0:00 [kthread]
root 10 0.0 0.0 0 0 ? S< May29 0:16
[kblockd/0] root 11 0.0 0.0 0 0 ? S< May29 0:00 [kacpid]
root 93 0.0 0.0 0 0 ? S< May29 0:00
[cqueue/0] root 96 0.0 0.0 0 0 ? S< May29 0:00 [khubd]
root 98 0.0 0.0 0 0 ? S< May29 0:00 [kseriod]
root 159 0.0 0.0 0 0 ? S May29 0:00
[khungtaskd] root 162 0.1 0.0 0 0 ? S< May29 6:52 [kswapd0]
root 163 0.0 0.0 0 0 ? S< May29 0:00 [aio/0]
root 315 0.0 0.0 0 0 ? S< May29 0:00
[kpsmoused] root 337 0.0 0.0 0 0 ? S< May29 0:00 [ata/0]
root 338 0.0 0.0 0 0 ? S< May29 0:00 [ata_aux]
root 341 0.0 0.0 0 0 ? S< May29 0:00
[scsi_eh_0] root 342 0.0 0.0 0 0 ? S< May29 0:00
[scsi_eh_1] root 347 0.0 0.0 0 0 ? S< May29 0:00
[kstriped] root 356 0.0 0.0 0 0 ? S< May29 0:00 [ksnapd]
root 367 0.0 0.0 0 0 ? D< May29 0:38
[kjournald] root 392 0.0 0.0 0 0 ? S< May29 0:00 [kauditd]
root 425 0.0 0.0 2280 364 ? S<s May29 0:00
/sbin/udevd -d
apache 630 4.9 4.7 213088 46192 ? D 07:45 3:16
/usr/sbin/httpd root 1139 0.0 0.0 0 0 ? S< May29 0:00
[kmpathd/0] root 1140 0.0 0.0 0 0 ? S< May29 0:00
[kmpath_handlerd] root 1162 0.0 0.0 0 0 ? S< May29 0:00
[kjournald] root 1682 0.0 0.0 12520 596 ? S<sl May29 0:00 auditd
root 1684 0.0 0.0 13100 600 ? S<sl May29 0:00
/sbin/audispd root 1714 0.0 0.0 1728 492 ? Ss May29 0:04
syslogd -m 0
root 1717 0.0 0.0 1680 312 ? Ss May29 0:00 klogd -x
root 1753 0.0 0.0 0 0 ? S< May29 0:29
[kondemand/0] rpc 1788 0.0 0.0 1816 376 ? Ss May29 0:00 portmap
root 1818 0.0 0.0 0 0 ? S< May29 0:00
[rpciod/0] rpcuser 1824 0.0 0.0 1868 480 ? Ss May29 0:00 rpc.statd
root 1856 0.0 0.0 5820 228 ? Ss May29 0:00
rpc.idmapd dbus 1879 0.0 0.0 2748 764 ? Ss May29 0:00
dbus-daemon --system
root 1892 0.0 0.0 2172 396 ? Ss May29 0:00
/usr/sbin/hcid root 1898 0.0 0.0 1748 372 ? Ss May29 0:00
/usr/sbin/sdpd apache 1901 3.4 6.7 204920 65568 ? S 08:16 1:12
/usr/sbin/httpd apache 1902 1.1 3.0 137472 29992 ? S 08:16 0:23
/usr/sbin/httpd apache 1903 2.1 2.2 208788 22224 ? S 08:16 0:45
/usr/sbin/httpd root 1919 0.0 0.0 0 0 ? S< May29 0:00
[krfcommd] root 1941 0.0 0.0 12868 548 ? Ssl May29 0:00 pcscd
root 1955 0.0 0.0 1676 376 ? Ss May29 0:00
/usr/sbin/acpid 68 1972 0.0 0.1 5628 992 ? Ss May29 0:00 hald
root 1973 0.0 0.0 3164 496 ? S May29 0:00
hald-runner 68 1980 0.0 0.0 2020 472 ? S May29 0:00
hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 1988 0.0 0.0 1976 356 ? S May29 0:04
hald-addon-storage: polling /dev/hdc
root 2008 0.0 0.0 1916 348 ? Ss May29 0:00
/usr/bin/hidd --server
root 2032 0.0 0.0 27260 720 ? Ssl May29 0:00 automount
root 2056 0.0 0.0 7080 672 ? Ss May29 0:00
/usr/sbin/sshd root 2069 0.0 0.0 10128 676 ? Ss May29 0:00 cupsd
root 2109 0.0 0.0 4536 504 ? S May29 0:00
/bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql
–socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log
–pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql 2159 0.0 0.3 149024 3716 ? Sl May29 0:18
/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql
–user=mysql --pid-file=/var/run/mysqld/mysqld.pid
–skip-external-locking --socket=/var/lib/mysql/mysql.sock
root 2196 0.0 0.0 9232 948 ? Ss May29 0:07
sendmail: accepting connections
smmsp 2204 0.0 0.0 8148 688 ? Ss May29 0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 2218 0.0 0.0 1908 244 ? Ss May29 0:00 gpm
-m /dev/input/mice -t exps2
root 2286 0.0 0.0 5292 552 ? Ss May29 0:00 crond
xfs 2309 0.0 0.0 3168 524 ? Ss May29 0:00 xfs
-droppriv -daemon
root 2334 0.0 0.0 2268 336 ? Ss May29 0:00
/usr/sbin/atd avahi 2360 0.0 0.0 2600 560 ? Ss May29 0:00
avahi-daemon: running [support.local]
avahi 2361 0.0 0.0 2600 116 ? Ss May29 0:00
avahi-daemon: chroot helper
root 2861 0.0 0.0 3516 348 ? S May29 0:00
/usr/sbin/smartd -q never
root 2868 0.0 0.0 1664 316 tty1 Ss+ May29 0:00
/sbin/mingetty tty1
root 2869 0.0 0.0 1664 316 tty2 Ss+ May29 0:00
/sbin/mingetty tty2
root 2872 0.0 0.0 1664 316 tty3 Ss+ May29 0:00
/sbin/mingetty tty3
root 2875 0.0 0.0 1664 316 tty4 Ss+ May29 0:00
/sbin/mingetty tty4
root 2877 0.0 0.0 1664 316 tty5 Ss+ May29 0:00
/sbin/mingetty tty5
root 2878 0.0 0.0 1664 316 tty6 Ss+ May29 0:00
/sbin/mingetty tty6
root 2926 0.0 0.2 26108 2768 ? SN May29 0:02
/usr/bin/python -tt /usr/sbin/yum-updatesd
root 2928 0.0 0.0 2564 628 ? SN May29 0:00
/usr/libexec/gam_server root 3012 0.0 0.2 9512 2636 ? S 08:45 0:00
sendmail: ./o51CGaSM001908 from queue
root 3023 0.0 0.2 9932 2844 ? Ss 08:45 0:00 sshd:
root@pts/0 root 3026 0.0 0.1 4660 1436 pts/0 Ss 08:45 0:00 -bash
root 3093 0.0 0.2 9432 2196 ? S 08:48 0:00
sendmail: ./o51CmDqe003090 from queue
mail 3094 0.1 0.9 10808 8880 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
mail 3095 0.2 0.9 10808 8844 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3097 0.0 0.2 9440 2212 ? S 08:48 0:00
sendmail: ./o51CmICW003096 from queue
mail 3098 0.2 0.9 10808 8844 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3105 0.0 0.0 0 0 ? S 08:49 0:00 [pdflush]
root 3107 0.0 0.2 9432 2136 ? S 08:49 0:00
sendmail: ./o51CnAWd003106 from queue
mail 3108 0.3 0.9 10808 8844 ? Ss 08:49 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3111 0.0 0.0 0 0 ? S 08:50 0:00 [pdflush]
root 3113 0.2 0.4 9568 4332 ? DN 08:50 0:00
/usr/bin/python -tt /usr/libexec/yum-updatesd-helper --check --dbus
root 3114 0.0 0.0 4260 944 pts/0 R+ 08:51 0:00 ps -auxww
root 22634 0.0 0.0 4868 484 ? S 02:12 0:00
/usr/sbin/nss_pcache off /etc/httpd/alias
root 22636 0.0 1.0 118420 9916 ? Ssl 02:12 0:04
/usr/sbin/httpd apache 22641 4.7 7.3 211988 71428 ? D 02:12 19:01
/usr/sbin/httpd apache 22642 4.7 10.4 227944 101056 ? D 02:12 19:08
/usr/sbin/httpd apache 22643 4.9 7.0 212152 68752 ? D 02:12 19:38
/usr/sbin/httpd apache 22644 4.5 7.8 207620 75932 ? S 02:12 18:10
/usr/sbin/httpd apache 22645 4.3 7.7 218668 74904 ? D 02:12 17:15
/usr/sbin/httpd apache 22646 4.7 3.7 214824 36704 ? S 02:12 19:03
/usr/sbin/httpd apache 22647 4.9 7.9 210512 76936 ? S 02:12 19:43
/usr/sbin/httpd apache 22648 4.3 10.0 215600 97432 ? S 02:12 17:29
/usr/sbin/httpd apache 26071 7.8 11.4 216356 111292 ? S 04:45 19:17
/usr/sbin/httpd

======================================

top - 08:48:04 up 2 days, 21:03, 1 user, load average: 6.04, 3.78,
2.64
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.5%us, 1.1%sy, 0.2%ni, 33.1%id, 45.0%wa, 0.3%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960168k used, 9916k free, 2844k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30420k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 49m 3076 D 1.2 5.2 1:09.12 httpd
22642 apache 16 0 222m 59m 3116 D 1.2 6.3 19:04.86 httpd
22644 apache 18 0 204m 74m 3116 D 1.2 7.9 18:08.90 httpd
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.52 kswapd0
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
1988 root 15 0 1976 356 340 R 0.0 0.0 0:04.13
hald-addon-stor
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3081 apache 17 0 8176 3060 2092 D 0.0 0.3 0:00.05 sendmail
3082 root 15 0 2200 896 696 R 0.0 0.1 0:00.00 top

top - 08:48:07 up 2 days, 21:04, 1 user, load average: 6.28, 3.86,
2.67
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 29.9%us, 1.9%sy, 0.0%ni, 0.0%id, 66.4%wa, 0.3%hi,
1.6%si, 0.0%st
Mem: 970084k total, 960012k used, 10072k free, 2848k buffers
Swap: 1966072k total, 983092k used, 982980k free, 30928k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 74m 3116 R 30.6 7.9 18:09.99 httpd
1901 apache 17 0 201m 50m 3076 D 2.2 5.3 1:09.20 httpd
3084 root 16 0 9276 2300 1564 D 1.4 0.2 0:00.05 sendmail
22642 apache 16 0 222m 61m 3116 D 0.6 6.5 19:04.88 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.53 kswapd0
3082 root 15 0 2208 992 784 R 0.3 0.1 0:00.01 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
top - 08:48:10 up 2 days, 21:04, 1 user, load average: 6.66, 3.98,
2.71
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 23.7%us, 2.3%sy, 0.0%ni, 0.0%id, 72.3%wa, 0.3%hi,
1.3%si, 0.0%st
Mem: 970084k total, 959132k used, 10952k free, 2872k buffers
Swap: 1966072k total, 983076k used, 982996k free, 31244k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 75m 3116 D 20.6 8.0 18:10.61 httpd
1901 apache 17 0 201m 52m 3076 D 2.7 5.5 1:09.28 httpd
22642 apache 16 0 222m 63m 3116 D 0.7 6.7 19:04.90 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2348 996 784 R 0.0 0.1 0:00.01 top
3084 root 16 0 9276 2364 1616 D 0.0 0.2 0:00.05 sendmail

top - 08:48:13 up 2 days, 21:04, 1 user, load average: 6.66, 3.98,
2.71
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.6%us, 2.3%sy, 0.0%ni, 0.0%id, 88.7%wa, 1.0%hi,
1.3%si, 0.0%st
Mem: 970084k total, 955480k used, 14604k free, 2920k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31000k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 53m 3076 D 2.7 5.7 1:09.36 httpd
22642 apache 16 0 222m 65m 3116 D 1.0 6.9 19:04.93 httpd
3085 apache 17 0 8176 3080 2116 D 0.7 0.3 0:00.05 sendmail
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.02 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
22647 apache 18 0 207m 76m 3124 D 0.0 8.1 19:43.79 httpd

top - 08:48:16 up 2 days, 21:04, 1 user, load average: 6.52, 4.00,
2.73
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.6%us, 5.0%sy, 0.0%ni, 0.0%id, 71.1%wa, 1.0%hi,
1.3%si, 0.0%st
Mem: 970084k total, 949180k used, 20904k free, 3044k buffers
Swap: 1966072k total, 982932k used, 983140k free, 29604k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3094 mail 19 0 7352 5544 1704 R 6.3 0.6 0:00.19 rt-mailgate
1901 apache 17 0 201m 56m 3076 D 6.0 5.9 1:09.54 httpd
22642 apache 16 0 222m 70m 3116 D 3.3 7.5 19:05.03 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.03 top
3016 mail 16 0 10808 8664 1940 R 0.0 0.9 0:00.30 rt-mailgate

top - 08:48:19 up 2 days, 21:04, 1 user, load average: 6.48, 4.03,
2.74
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.7%us, 3.0%sy, 0.0%ni, 0.0%id, 74.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 956980k used, 13104k free, 3084k buffers
Swap: 1966072k total, 983140k used, 982932k free, 30380k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 57m 3076 D 2.7 6.1 1:09.62 httpd
22642 apache 16 0 222m 73m 3116 D 2.3 7.8 19:05.10 httpd
162 root 11 -5 0 0 0 D 0.3 0.0 6:51.59 kswapd0
22641 apache 18 0 208m 10m 3116 D 0.3 1.1 19:00.45 httpd
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.03 top
3096 root 15 0 9300 2512 1712 R 0.0 0.3 0:00.00 sendmail
22645 apache 16 0 211m 10m 3124 R 0.0 1.1 17:13.98 httpd

top - 08:48:22 up 2 days, 21:04, 1 user, load average: 6.48, 4.03,
2.74
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 1.1%sy, 0.0%ni, 0.0%id, 93.8%wa, 0.5%hi,
1.6%si, 0.0%st
Mem: 970084k total, 960256k used, 9828k free, 3092k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30920k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3096 root 16 0 9300 2512 1712 D 2.3 0.3 0:00.07 sendmail
22642 apache 16 0 222m 75m 3116 D 1.7 8.0 19:05.15 httpd
1901 apache 17 0 201m 58m 3076 D 1.0 6.2 1:09.65 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.60 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.04 top
22641 apache 18 0 208m 11m 3116 D 0.3 1.2 19:00.46 httpd
22645 apache 16 0 211m 10m 3124 D 0.3 1.1 17:13.99 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush

top - 08:48:26 up 2 days, 21:04, 1 user, load average: 6.68, 4.11,
2.78
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 1.3%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960216k used, 9868k free, 3092k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30996k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 59m 3076 D 2.4 6.3 1:09.74 httpd
22642 apache 16 0 222m 77m 3116 D 1.6 8.2 19:05.21 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.61 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.05 top
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.00 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1760 928 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 11m 3116 R 0.0 1.2 19:00.46 httpd

top - 08:48:29 up 2 days, 21:04, 1 user, load average: 6.68, 4.11,
2.78
Tasks: 99 total, 1 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 1.0%sy, 0.0%ni, 0.0%id, 93.7%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 960192k used, 9892k free, 3092k buffers
Swap: 1966072k total, 983160k used, 982912k free, 30960k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 60m 3076 D 2.7 6.4 1:09.82 httpd
22642 apache 16 0 222m 78m 3116 D 1.0 8.3 19:05.24 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.63 kswapd0
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.01 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.05 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1764 932 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 12m 3116 D 0.0 1.3 19:00.46 httpd

top - 08:48:32 up 2 days, 21:04, 1 user, load average: 6.87, 4.19,
2.81
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.3%us, 2.7%sy, 0.0%ni, 0.0%id, 83.0%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 960100k used, 9984k free, 3028k buffers
Swap: 1966072k total, 983072k used, 983000k free, 31044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3098 mail 18 0 7352 5504 1656 D 6.7 0.6 0:00.20 rt-mailgate
1901 apache 18 0 201m 61m 3076 D 4.3 6.5 1:09.95 httpd
22642 apache 16 0 222m 80m 3116 D 1.7 8.5 19:05.29 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.65 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.06 top
22641 apache 18 0 208m 12m 3116 D 0.3 1.3 19:00.47 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 11m 3124 D 0.0 1.2 17:14.01 httpd

top - 08:48:35 up 2 days, 21:04, 1 user, load average: 7.12, 4.29,
2.85
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.9%us, 1.7%sy, 0.0%ni, 0.0%id, 86.8%wa, 0.3%hi,
1.3%si, 0.0%st
Mem: 970084k total, 960128k used, 9956k free, 3028k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30988k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 4.0 6.6 1:10.07 httpd
22642 apache 16 0 222m 81m 3116 D 1.3 8.6 19:05.33 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.66 kswapd0
22641 apache 18 0 208m 12m 3116 D 0.3 1.4 19:00.48 httpd
22645 apache 16 0 211m 12m 3124 D 0.3 1.3 17:14.02 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.06 top
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd

top - 08:48:38 up 2 days, 21:04, 1 user, load average: 7.12, 4.29,
2.85
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.3%us, 1.3%sy, 0.0%ni, 0.0%id, 90.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959856k used, 10228k free, 3024k buffers
Swap: 1966072k total, 985244k used, 980828k free, 31032k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 6.3 6.6 1:10.26 httpd
22642 apache 16 0 222m 82m 3116 D 1.7 8.7 19:05.38 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.68 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.07 top
22641 apache 18 0 208m 13m 3116 D 0.3 1.4 19:00.49 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd
22645 apache 16 0 211m 12m 3124 D 0.0 1.3 17:14.02 httpd

top - 08:48:41 up 2 days, 21:04, 1 user, load average: 7.27, 4.37,
2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.3%us, 1.3%sy, 0.0%ni, 0.0%id, 87.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959516k used, 10568k free, 3016k buffers
Swap: 1966072k total, 983232k used, 982840k free, 31028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 9.0 6.6 1:10.53 httpd
22642 apache 17 0 222m 83m 3116 D 1.7 8.8 19:05.43 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.69 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.08 top
22643 apache 16 0 207m 12m 3116 D 0.3 1.4 19:36.91 httpd
22645 apache 16 0 211m 13m 3124 D 0.3 1.4 17:14.03 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 18 0 208m 13m 3116 D 0.0 1.4 19:00.49 httpd

top - 08:48:44 up 2 days, 21:04, 1 user, load average: 7.27, 4.37,
2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.0%us, 1.3%sy, 0.0%ni, 0.0%id, 83.4%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959932k used, 10152k free, 3012k buffers
Swap: 1966072k total, 983124k used, 982948k free, 31024k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 13.2 6.7 1:10.93 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 8.9 19:05.46 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.70 kswapd0
22641 apache 16 0 208m 14m 3116 D 0.3 1.5 19:00.50 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.08 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.91 httpd

top - 08:48:47 up 2 days, 21:04, 1 user, load average: 7.33, 4.43,
2.91
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 41.9%us, 1.7%sy, 0.0%ni, 0.0%id, 54.5%wa, 0.7%hi,
1.3%si, 0.0%st
Mem: 970084k total, 960020k used, 10064k free, 2992k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31048k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 41.9 6.7 1:12.19 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 9.0 19:05.49 httpd
22643 apache 16 0 207m 13m 3116 D 0.7 1.4 19:36.93 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.09 top
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.70 kswapd0
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 16 0 208m 14m 3116 D 0.0 1.5 19:00.50 httpd
22645 apache 16 0 211m 14m 3124 D 0.0 1.5 17:14.04 httpd

top - 08:48:50 up 2 days, 21:04, 1 user, load average: 7.46, 4.50,
2.94
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 1.7%sy, 0.0%ni, 0.0%id, 93.1%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 960184k used, 9900k free, 2992k buffers
Swap: 1966072k total, 984032k used, 982040k free, 31180k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 2.0 6.8 1:12.25 httpd
22642 apache 17 0 222m 85m 3116 R 1.0 9.0 19:05.52 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.72 kswapd0
22645 apache 16 0 211m 14m 3124 D 0.7 1.6 17:14.06 httpd
22641 apache 16 0 208m 15m 3116 D 0.3 1.6 19:00.51 httpd
367 root 11 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.09 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.93 httpd

top - 08:48:53 up 2 days, 21:04, 1 user, load average: 7.46, 4.50,
2.94
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.3%us, 1.7%sy, 0.0%ni, 0.0%id, 91.7%wa, 0.7%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960156k used, 9928k free, 2992k buffers
Swap: 1966072k total, 983068k used, 983004k free, 31464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 3.5 6.9 1:12.37 httpd
22642 apache 17 0 222m 85m 3116 D 1.4 9.1 19:05.57 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.73 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.10 top
22641 apache 16 0 208m 15m 3116 D 0.3 1.7 19:00.52 httpd
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.94 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 15m 3124 D 0.0 1.6 17:14.06 httpd

top - 08:48:56 up 2 days, 21:04, 1 user, load average: 7.50, 4.56,
2.97
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.2%us, 0.9%sy, 0.0%ni, 0.0%id, 96.5%wa, 0.6%hi,
0.9%si, 0.0%st
Mem: 970084k total, 960380k used, 9704k free, 2936k buffers
Swap: 1966072k total, 983044k used, 983028k free, 31392k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.0 9.1 19:05.60 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.74 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.11 top
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.95 httpd
22645 apache 16 0 211m 15m 3124 D 0.3 1.6 17:14.07 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00
pdflush
3099 apache 17 0 8172 3052 2092 D 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.7 19:00.52 httpd

top - 08:49:00 up 2 days, 21:04, 1 user, load average: 7.62, 4.63,
3.00
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 0.0%id, 95.3%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 959064k used, 11020k free, 2936k buffers
Swap: 1966072k total, 983532k used, 982540k free, 31148k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.4 9.2 19:05.65 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.6 19:36.96 httpd
22645 apache 16 0 211m 16m 3124 D 0.3 1.7 17:14.08 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.11 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3099 apache 17 0 8172 3076 2116 R 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.8 19:00.52 httpd

top - 08:49:03 up 2 days, 21:04, 1 user, load average: 7.62, 4.63,
3.00
Tasks: 103 total, 2 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 1.7%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959560k used, 10524k free, 2928k buffers
Swap: 1966072k total, 983440k used, 982632k free, 30496k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 87m 3116 D 2.0 9.2 19:05.71 httpd
3102 root 16 0 9276 2300 1564 D 1.3 0.2 0:00.04 sendmail
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.77 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.12 top
22641 apache 16 0 208m 17m 3116 D 0.3 1.8 19:00.53 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.7 19:36.97 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
22645 apache 16 0 211m 16m 3124 R 0.0 1.7 17:14.08 httpd

==========================================

[root@support ~]# mailq
/var/spool/mqueue (6 requests)
-----Q-ID----- --Size-- -----Q-Time-----
------------Sender/Recipient----------- o51Cxkw2003214 8112 Tue Jun 1
08:59 jay@centritech.net
7BIT (Deferred: prog mailer (/usr/sbin/smrsh) exited with
EX_TEMPF)
“|/etc/smrsh/rt-mailgate
–queue 'ccs:
o51CuKTF003192 633211 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CGaSM001908 1170766 Tue Jun 1 08:16 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CuIdl003186 1355798 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51Ckond003072 3612709 Tue Jun 1 08:46 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CutnV003202 3618441 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
"|/etc/smrsh/rt-mailgate
–queue 'umx:
Total requests: 6
[root@support ~]#

Cheers

Richard Skelton
Richard.Skelton@infineon.com Infineon Technologies UK Ltd
Infineon House
Great Western Court
Hunts Ground Road
Stoke Gifford
Bristol BS34 8HP
Tel +44(0)117 9528808

[Eriks_Goodwin-Pfiste]

But then my question becomes, “why did this happen all of a sudden?” We have not changed user loads, usage levels, etc. This is a dedicated box being used by about 5 active users and 20 intermittent (once a week) users. Is 1 Gb of memory not enought for RT with such a light loading?

And does anyone have any idea what those odd mailq entries are? I have to go in and delete several hundred of them every day or two.

Thanks,
EriksFrom: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:07:58 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
Yes as I thought you are short on real memory.
Look at the swap.
Here is a busy 16 core Linux system of mine with load of free memory:-
procs -----------memory---------- —swap-- -----io---- --system–
----cpu---- r b swpd free buff cache si so bi bo in cs us sy
id wa
9 0 204 13177160 157388 25908864 0 0 0 46 3774 4411
52 1 47 0
7 0 204 12952512 157388 25915572 0 0 0 48 3405 3409
52 1 46 0
10 0 204 12847832 157388 25918668 0 0 0 23 3043 3351
51 2 47 0
9 0 204 12678600 157388 25921764 0 0 0 39 1776 2432
50 5 45 0
9 0 204 12779600 157388 25924860 0 0 0 26 2480 6489
49 6 46 0
6 0 204 13102136 157388 25910412 0 0 0 106 3861 17726
38 7 55 0
5 0 204 13522232 157388 25903188 0 0 0 43 3290 23044
27 7 66 0
5 0 204 13375776 157388 25882548 0 0 0 46 1946 1996
21 5 75 0
5 0 204 13189392 157388 25883580 0 0 0 22 1161 905
20 5 75 0
5 0 204 12992112 157388 25884612 0 0 0 21 1194 1002
21 5 75 0

Eriks Goodwin-Pfister wrote:

[root@support ~]# vmstat 5
procs -----------memory---------- —swap-- -----io---- --system–
-----cpu------
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 2 983748 48248 3472 14964 556 259 605 448 1 59 20 2 33 45 0
1 2 983572 17100 2824 22092 2367 501 2367 2045 1276 866 45 9 0 47 0
0 2 983036 18208 2936 22112 1085 73 1101 611 1359 375 27 5 0 68 0
0 2 983036 11340 2944 22488 1306 0 1306 265 1199 442 22 2 0 77 0
0 3 983748 11296 2596 22656 790 650 793 861 1252 373 39 6 0 55 0
0 3 983340 11144 2640 21908 887 234 914 487 1346 445 13 4 0 83 0
1 2 983036 11208 2536 22268 659 49 659 249 1142 284 46 2 0 52 0
0 3 983616 11428 2236 22188 911 581 970 1010 1392 512 19 4 0 76 0
0 2 983836 12452 2236 22548 640 739 640 962 1224 335 19 3 0 78 0
0 2 983136 11516 2184 22596 824 428 824 566 1241 357 5 2 0 92 0
3 1 983036 11216 1928 21544 802 482 802 508 1218 278 18 2 0 80 0
0 4 984124 12052 1944 22168 630 1196 630 1413 1283 270 23 3 0 74 0
1 1 983828 14684 2080 21628 506 548 546 842 1347 354 22 4 0 74 0
1 2 983228 11896 2096 22312 973 440 973 718 1204 378 25 3 0 72 0
0 2 983036 15360 2136 22060 430 294 430 586 1177 252 39 3 0 59 0
0 2 983036 13252 2212 22516 628 0 628 190 1271 372 39 3 0 57 0
0 1 983036 23836 2296 21768 597 17 597 622 1216 331 53 4 0 43 0
0 2 983036 11968 2360 23068 765 0 814 167 1215 376 42 3 0 55 0

----- Original Message -----
From: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:00:49 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
You have run out of memory.
What does vmstat 5 show?

Eriks Goodwin-Pfister wrote:

That’s the kicker, nothing looks “abnormal” in the top listing or
processes to me. Perhaps you see something I don’t?

I have included:

1. results of “ps -auxww”
2. results of “top -b -i -n 20 >> ./top_procs”
3. odd entries in “mailq”

Thanks in advance for any more advice…

–Eriks

======================================

[root@support ~]# ps -auxww
Warning: bad syntax, perhaps a bogus ‘-’? See
/usr/share/doc/procps-3.2.7/FAQ USER PID %CPU %MEM VSZ RSS TTY STAT
START TIME COMMAND
root 1 0.0 0.0 2072 512 ? Ss May29 0:01 init
[3] root 2 0.0 0.0 0 0 ? S< May29 0:00
[migration/0] root 3 0.0 0.0 0 0 ? SN May29 0:00
[ksoftirqd/0] root 4 0.0 0.0 0 0 ? S< May29 0:00
[watchdog/0] root 5 0.0 0.0 0 0 ? S< May29 0:00
[events/0] root 6 0.0 0.0 0 0 ? S< May29 0:00 [khelper]
root 7 0.0 0.0 0 0 ? S< May29 0:00 [kthread]
root 10 0.0 0.0 0 0 ? S< May29 0:16
[kblockd/0] root 11 0.0 0.0 0 0 ? S< May29 0:00 [kacpid]
root 93 0.0 0.0 0 0 ? S< May29 0:00
[cqueue/0] root 96 0.0 0.0 0 0 ? S< May29 0:00 [khubd]
root 98 0.0 0.0 0 0 ? S< May29 0:00 [kseriod]
root 159 0.0 0.0 0 0 ? S May29 0:00
[khungtaskd] root 162 0.1 0.0 0 0 ? S< May29 6:52 [kswapd0]
root 163 0.0 0.0 0 0 ? S< May29 0:00 [aio/0]
root 315 0.0 0.0 0 0 ? S< May29 0:00
[kpsmoused] root 337 0.0 0.0 0 0 ? S< May29 0:00 [ata/0]
root 338 0.0 0.0 0 0 ? S< May29 0:00 [ata_aux]
root 341 0.0 0.0 0 0 ? S< May29 0:00
[scsi_eh_0] root 342 0.0 0.0 0 0 ? S< May29 0:00
[scsi_eh_1] root 347 0.0 0.0 0 0 ? S< May29 0:00
[kstriped] root 356 0.0 0.0 0 0 ? S< May29 0:00 [ksnapd]
root 367 0.0 0.0 0 0 ? D< May29 0:38
[kjournald] root 392 0.0 0.0 0 0 ? S< May29 0:00 [kauditd]
root 425 0.0 0.0 2280 364 ? S<s May29 0:00
/sbin/udevd -d
apache 630 4.9 4.7 213088 46192 ? D 07:45 3:16
/usr/sbin/httpd root 1139 0.0 0.0 0 0 ? S< May29 0:00
[kmpathd/0] root 1140 0.0 0.0 0 0 ? S< May29 0:00
[kmpath_handlerd] root 1162 0.0 0.0 0 0 ? S< May29 0:00
[kjournald] root 1682 0.0 0.0 12520 596 ? S<sl May29 0:00 auditd
root 1684 0.0 0.0 13100 600 ? S<sl May29 0:00
/sbin/audispd root 1714 0.0 0.0 1728 492 ? Ss May29 0:04
syslogd -m 0
root 1717 0.0 0.0 1680 312 ? Ss May29 0:00 klogd -x
root 1753 0.0 0.0 0 0 ? S< May29 0:29
[kondemand/0] rpc 1788 0.0 0.0 1816 376 ? Ss May29 0:00 portmap
root 1818 0.0 0.0 0 0 ? S< May29 0:00
[rpciod/0] rpcuser 1824 0.0 0.0 1868 480 ? Ss May29 0:00 rpc.statd
root 1856 0.0 0.0 5820 228 ? Ss May29 0:00
rpc.idmapd dbus 1879 0.0 0.0 2748 764 ? Ss May29 0:00
dbus-daemon --system
root 1892 0.0 0.0 2172 396 ? Ss May29 0:00
/usr/sbin/hcid root 1898 0.0 0.0 1748 372 ? Ss May29 0:00
/usr/sbin/sdpd apache 1901 3.4 6.7 204920 65568 ? S 08:16 1:12
/usr/sbin/httpd apache 1902 1.1 3.0 137472 29992 ? S 08:16 0:23
/usr/sbin/httpd apache 1903 2.1 2.2 208788 22224 ? S 08:16 0:45
/usr/sbin/httpd root 1919 0.0 0.0 0 0 ? S< May29 0:00
[krfcommd] root 1941 0.0 0.0 12868 548 ? Ssl May29 0:00 pcscd
root 1955 0.0 0.0 1676 376 ? Ss May29 0:00
/usr/sbin/acpid 68 1972 0.0 0.1 5628 992 ? Ss May29 0:00 hald
root 1973 0.0 0.0 3164 496 ? S May29 0:00
hald-runner 68 1980 0.0 0.0 2020 472 ? S May29 0:00
hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
root 1988 0.0 0.0 1976 356 ? S May29 0:04
hald-addon-storage: polling /dev/hdc
root 2008 0.0 0.0 1916 348 ? Ss May29 0:00
/usr/bin/hidd --server
root 2032 0.0 0.0 27260 720 ? Ssl May29 0:00 automount
root 2056 0.0 0.0 7080 672 ? Ss May29 0:00
/usr/sbin/sshd root 2069 0.0 0.0 10128 676 ? Ss May29 0:00 cupsd
root 2109 0.0 0.0 4536 504 ? S May29 0:00
/bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql
–socket=/var/lib/mysql/mysql.sock --log-error=/var/log/mysqld.log
–pid-file=/var/run/mysqld/mysqld.pid --user=mysql
mysql 2159 0.0 0.3 149024 3716 ? Sl May29 0:18
/usr/libexec/mysqld --basedir=/usr --datadir=/var/lib/mysql
–user=mysql --pid-file=/var/run/mysqld/mysqld.pid
–skip-external-locking --socket=/var/lib/mysql/mysql.sock
root 2196 0.0 0.0 9232 948 ? Ss May29 0:07
sendmail: accepting connections
smmsp 2204 0.0 0.0 8148 688 ? Ss May29 0:00
sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue
root 2218 0.0 0.0 1908 244 ? Ss May29 0:00 gpm
-m /dev/input/mice -t exps2
root 2286 0.0 0.0 5292 552 ? Ss May29 0:00 crond
xfs 2309 0.0 0.0 3168 524 ? Ss May29 0:00 xfs
-droppriv -daemon
root 2334 0.0 0.0 2268 336 ? Ss May29 0:00
/usr/sbin/atd avahi 2360 0.0 0.0 2600 560 ? Ss May29 0:00
avahi-daemon: running [support.local]
avahi 2361 0.0 0.0 2600 116 ? Ss May29 0:00
avahi-daemon: chroot helper
root 2861 0.0 0.0 3516 348 ? S May29 0:00
/usr/sbin/smartd -q never
root 2868 0.0 0.0 1664 316 tty1 Ss+ May29 0:00
/sbin/mingetty tty1
root 2869 0.0 0.0 1664 316 tty2 Ss+ May29 0:00
/sbin/mingetty tty2
root 2872 0.0 0.0 1664 316 tty3 Ss+ May29 0:00
/sbin/mingetty tty3
root 2875 0.0 0.0 1664 316 tty4 Ss+ May29 0:00
/sbin/mingetty tty4
root 2877 0.0 0.0 1664 316 tty5 Ss+ May29 0:00
/sbin/mingetty tty5
root 2878 0.0 0.0 1664 316 tty6 Ss+ May29 0:00
/sbin/mingetty tty6
root 2926 0.0 0.2 26108 2768 ? SN May29 0:02
/usr/bin/python -tt /usr/sbin/yum-updatesd
root 2928 0.0 0.0 2564 628 ? SN May29 0:00
/usr/libexec/gam_server root 3012 0.0 0.2 9512 2636 ? S 08:45 0:00
sendmail: ./o51CGaSM001908 from queue
root 3023 0.0 0.2 9932 2844 ? Ss 08:45 0:00 sshd:
root@pts/0 root 3026 0.0 0.1 4660 1436 pts/0 Ss 08:45 0:00 -bash
root 3093 0.0 0.2 9432 2196 ? S 08:48 0:00
sendmail: ./o51CmDqe003090 from queue
mail 3094 0.1 0.9 10808 8880 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
mail 3095 0.2 0.9 10808 8844 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3097 0.0 0.2 9440 2212 ? S 08:48 0:00
sendmail: ./o51CmICW003096 from queue
mail 3098 0.2 0.9 10808 8844 ? Ss 08:48 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3105 0.0 0.0 0 0 ? S 08:49 0:00 [pdflush]
root 3107 0.0 0.2 9432 2136 ? S 08:49 0:00
sendmail: ./o51CnAWd003106 from queue
mail 3108 0.3 0.9 10808 8844 ? Ss 08:49 0:00
/usr/bin/perl -w /etc/smrsh/rt-mailgate --queue umx: client helpdesk
–action correspond --url http://support.unimatrix.net
root 3111 0.0 0.0 0 0 ? S 08:50 0:00 [pdflush]
root 3113 0.2 0.4 9568 4332 ? DN 08:50 0:00
/usr/bin/python -tt /usr/libexec/yum-updatesd-helper --check --dbus
root 3114 0.0 0.0 4260 944 pts/0 R+ 08:51 0:00 ps -auxww
root 22634 0.0 0.0 4868 484 ? S 02:12 0:00
/usr/sbin/nss_pcache off /etc/httpd/alias
root 22636 0.0 1.0 118420 9916 ? Ssl 02:12 0:04
/usr/sbin/httpd apache 22641 4.7 7.3 211988 71428 ? D 02:12 19:01
/usr/sbin/httpd apache 22642 4.7 10.4 227944 101056 ? D 02:12 19:08
/usr/sbin/httpd apache 22643 4.9 7.0 212152 68752 ? D 02:12 19:38
/usr/sbin/httpd apache 22644 4.5 7.8 207620 75932 ? S 02:12 18:10
/usr/sbin/httpd apache 22645 4.3 7.7 218668 74904 ? D 02:12 17:15
/usr/sbin/httpd apache 22646 4.7 3.7 214824 36704 ? S 02:12 19:03
/usr/sbin/httpd apache 22647 4.9 7.9 210512 76936 ? S 02:12 19:43
/usr/sbin/httpd apache 22648 4.3 10.0 215600 97432 ? S 02:12 17:29
/usr/sbin/httpd apache 26071 7.8 11.4 216356 111292 ? S 04:45 19:17
/usr/sbin/httpd

======================================

top - 08:48:04 up 2 days, 21:03, 1 user, load average: 6.04, 3.78,
2.64
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 19.5%us, 1.1%sy, 0.2%ni, 33.1%id, 45.0%wa, 0.3%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960168k used, 9916k free, 2844k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30420k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 49m 3076 D 1.2 5.2 1:09.12 httpd
22642 apache 16 0 222m 59m 3116 D 1.2 6.3 19:04.86 httpd
22644 apache 18 0 204m 74m 3116 D 1.2 7.9 18:08.90 httpd
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.52 kswapd0
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
1988 root 15 0 1976 356 340 R 0.0 0.0 0:04.13
hald-addon-stor
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3081 apache 17 0 8176 3060 2092 D 0.0 0.3 0:00.05 sendmail
3082 root 15 0 2200 896 696 R 0.0 0.1 0:00.00 top

top - 08:48:07 up 2 days, 21:04, 1 user, load average: 6.28, 3.86,
2.67
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 29.9%us, 1.9%sy, 0.0%ni, 0.0%id, 66.4%wa, 0.3%hi,
1.6%si, 0.0%st
Mem: 970084k total, 960012k used, 10072k free, 2848k buffers
Swap: 1966072k total, 983092k used, 982980k free, 30928k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 74m 3116 R 30.6 7.9 18:09.99 httpd
1901 apache 17 0 201m 50m 3076 D 2.2 5.3 1:09.20 httpd
3084 root 16 0 9276 2300 1564 D 1.4 0.2 0:00.05 sendmail
22642 apache 16 0 222m 61m 3116 D 0.6 6.5 19:04.88 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.53 kswapd0
3082 root 15 0 2208 992 784 R 0.3 0.1 0:00.01 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
top - 08:48:10 up 2 days, 21:04, 1 user, load average: 6.66, 3.98,
2.71
Tasks: 102 total, 2 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 23.7%us, 2.3%sy, 0.0%ni, 0.0%id, 72.3%wa, 0.3%hi,
1.3%si, 0.0%st
Mem: 970084k total, 959132k used, 10952k free, 2872k buffers
Swap: 1966072k total, 983076k used, 982996k free, 31244k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22644 apache 18 0 204m 75m 3116 D 20.6 8.0 18:10.61 httpd
1901 apache 17 0 201m 52m 3076 D 2.7 5.5 1:09.28 httpd
22642 apache 16 0 222m 63m 3116 D 0.7 6.7 19:04.90 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
3010 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3078 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2348 996 784 R 0.0 0.1 0:00.01 top
3084 root 16 0 9276 2364 1616 D 0.0 0.2 0:00.05 sendmail

top - 08:48:13 up 2 days, 21:04, 1 user, load average: 6.66, 3.98,
2.71
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 6.6%us, 2.3%sy, 0.0%ni, 0.0%id, 88.7%wa, 1.0%hi,
1.3%si, 0.0%st
Mem: 970084k total, 955480k used, 14604k free, 2920k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31000k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 53m 3076 D 2.7 5.7 1:09.36 httpd
22642 apache 16 0 222m 65m 3116 D 1.0 6.9 19:04.93 httpd
3085 apache 17 0 8176 3080 2116 D 0.7 0.3 0:00.05 sendmail
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.02 top
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.12 kjournald
22647 apache 18 0 207m 76m 3124 D 0.0 8.1 19:43.79 httpd

top - 08:48:16 up 2 days, 21:04, 1 user, load average: 6.52, 4.00,
2.73
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.6%us, 5.0%sy, 0.0%ni, 0.0%id, 71.1%wa, 1.0%hi,
1.3%si, 0.0%st
Mem: 970084k total, 949180k used, 20904k free, 3044k buffers
Swap: 1966072k total, 982932k used, 983140k free, 29604k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3094 mail 19 0 7352 5544 1704 R 6.3 0.6 0:00.19 rt-mailgate
1901 apache 17 0 201m 56m 3076 D 6.0 5.9 1:09.54 httpd
22642 apache 16 0 222m 70m 3116 D 3.3 7.5 19:05.03 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.03 top
3016 mail 16 0 10808 8664 1940 R 0.0 0.9 0:00.30 rt-mailgate

top - 08:48:19 up 2 days, 21:04, 1 user, load average: 6.48, 4.03,
2.74
Tasks: 99 total, 3 running, 96 sleeping, 0 stopped, 0 zombie
Cpu(s): 21.7%us, 3.0%sy, 0.0%ni, 0.0%id, 74.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 956980k used, 13104k free, 3084k buffers
Swap: 1966072k total, 983140k used, 982932k free, 30380k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 57m 3076 D 2.7 6.1 1:09.62 httpd
22642 apache 16 0 222m 73m 3116 D 2.3 7.8 19:05.10 httpd
162 root 11 -5 0 0 0 D 0.3 0.0 6:51.59 kswapd0
22641 apache 18 0 208m 10m 3116 D 0.3 1.1 19:00.45 httpd
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.03 top
3096 root 15 0 9300 2512 1712 R 0.0 0.3 0:00.00 sendmail
22645 apache 16 0 211m 10m 3124 R 0.0 1.1 17:13.98 httpd

top - 08:48:22 up 2 days, 21:04, 1 user, load average: 6.48, 4.03,
2.74
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.0%us, 1.1%sy, 0.0%ni, 0.0%id, 93.8%wa, 0.5%hi,
1.6%si, 0.0%st
Mem: 970084k total, 960256k used, 9828k free, 3092k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30920k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3096 root 16 0 9300 2512 1712 D 2.3 0.3 0:00.07 sendmail
22642 apache 16 0 222m 75m 3116 D 1.7 8.0 19:05.15 httpd
1901 apache 17 0 201m 58m 3076 D 1.0 6.2 1:09.65 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.60 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.04 top
22641 apache 18 0 208m 11m 3116 D 0.3 1.2 19:00.46 httpd
22645 apache 16 0 211m 10m 3124 D 0.3 1.1 17:13.99 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush

top - 08:48:26 up 2 days, 21:04, 1 user, load average: 6.68, 4.11,
2.78
Tasks: 99 total, 2 running, 97 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.3%us, 1.3%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960216k used, 9868k free, 3092k buffers
Swap: 1966072k total, 983072k used, 983000k free, 30996k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 59m 3076 D 2.4 6.3 1:09.74 httpd
22642 apache 16 0 222m 77m 3116 D 1.6 8.2 19:05.21 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.61 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.05 top
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.00 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1760 928 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 11m 3116 R 0.0 1.2 19:00.46 httpd

top - 08:48:29 up 2 days, 21:04, 1 user, load average: 6.68, 4.11,
2.78
Tasks: 99 total, 1 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 4.0%us, 1.0%sy, 0.0%ni, 0.0%id, 93.7%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 960192k used, 9892k free, 3092k buffers
Swap: 1966072k total, 983160k used, 982912k free, 30960k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 17 0 201m 60m 3076 D 2.7 6.4 1:09.82 httpd
22642 apache 16 0 222m 78m 3116 D 1.0 8.3 19:05.24 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.63 kswapd0
22645 apache 16 0 211m 11m 3124 D 0.3 1.2 17:14.01 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.05 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3097 root 18 0 9300 1764 932 D 0.0 0.2 0:00.00 sendmail
22641 apache 18 0 208m 12m 3116 D 0.0 1.3 19:00.46 httpd

top - 08:48:32 up 2 days, 21:04, 1 user, load average: 6.87, 4.19,
2.81
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 12.3%us, 2.7%sy, 0.0%ni, 0.0%id, 83.0%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 960100k used, 9984k free, 3028k buffers
Swap: 1966072k total, 983072k used, 983000k free, 31044k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3098 mail 18 0 7352 5504 1656 D 6.7 0.6 0:00.20 rt-mailgate
1901 apache 18 0 201m 61m 3076 D 4.3 6.5 1:09.95 httpd
22642 apache 16 0 222m 80m 3116 D 1.7 8.5 19:05.29 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.65 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.06 top
22641 apache 18 0 208m 12m 3116 D 0.3 1.3 19:00.47 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 11m 3124 D 0.0 1.2 17:14.01 httpd

top - 08:48:35 up 2 days, 21:04, 1 user, load average: 7.12, 4.29,
2.85
Tasks: 100 total, 2 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 9.9%us, 1.7%sy, 0.0%ni, 0.0%id, 86.8%wa, 0.3%hi,
1.3%si, 0.0%st
Mem: 970084k total, 960128k used, 9956k free, 3028k buffers
Swap: 1966072k total, 983036k used, 983036k free, 30988k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 4.0 6.6 1:10.07 httpd
22642 apache 16 0 222m 81m 3116 D 1.3 8.6 19:05.33 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.66 kswapd0
22641 apache 18 0 208m 12m 3116 D 0.3 1.4 19:00.48 httpd
22645 apache 16 0 211m 12m 3124 D 0.3 1.3 17:14.02 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.06 top
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd

top - 08:48:38 up 2 days, 21:04, 1 user, load average: 7.12, 4.29,
2.85
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.3%us, 1.3%sy, 0.0%ni, 0.0%id, 90.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959856k used, 10228k free, 3024k buffers
Swap: 1966072k total, 985244k used, 980828k free, 31032k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 6.3 6.6 1:10.26 httpd
22642 apache 16 0 222m 82m 3116 D 1.7 8.7 19:05.38 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.68 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.07 top
22641 apache 18 0 208m 13m 3116 D 0.3 1.4 19:00.49 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 12m 3116 D 0.0 1.3 19:36.90 httpd
22645 apache 16 0 211m 12m 3124 D 0.0 1.3 17:14.02 httpd

top - 08:48:41 up 2 days, 21:04, 1 user, load average: 7.27, 4.37,
2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 10.3%us, 1.3%sy, 0.0%ni, 0.0%id, 87.0%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959516k used, 10568k free, 3016k buffers
Swap: 1966072k total, 983232k used, 982840k free, 31028k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 62m 3076 D 9.0 6.6 1:10.53 httpd
22642 apache 17 0 222m 83m 3116 D 1.7 8.8 19:05.43 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.69 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.08 top
22643 apache 16 0 207m 12m 3116 D 0.3 1.4 19:36.91 httpd
22645 apache 16 0 211m 13m 3124 D 0.3 1.4 17:14.03 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 18 0 208m 13m 3116 D 0.0 1.4 19:00.49 httpd

top - 08:48:44 up 2 days, 21:04, 1 user, load average: 7.27, 4.37,
2.88
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 14.0%us, 1.3%sy, 0.0%ni, 0.0%id, 83.4%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959932k used, 10152k free, 3012k buffers
Swap: 1966072k total, 983124k used, 982948k free, 31024k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 13.2 6.7 1:10.93 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 8.9 19:05.46 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.70 kswapd0
22641 apache 16 0 208m 14m 3116 D 0.3 1.5 19:00.50 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.08 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.91 httpd

top - 08:48:47 up 2 days, 21:04, 1 user, load average: 7.33, 4.43,
2.91
Tasks: 100 total, 1 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 41.9%us, 1.7%sy, 0.0%ni, 0.0%id, 54.5%wa, 0.7%hi,
1.3%si, 0.0%st
Mem: 970084k total, 960020k used, 10064k free, 2992k buffers
Swap: 1966072k total, 983036k used, 983036k free, 31048k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 63m 3076 D 41.9 6.7 1:12.19 httpd
22642 apache 17 0 222m 84m 3116 D 1.0 9.0 19:05.49 httpd
22643 apache 16 0 207m 13m 3116 D 0.7 1.4 19:36.93 httpd
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.09 top
162 root 10 -5 0 0 0 D 0.0 0.0 6:51.70 kswapd0
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22641 apache 16 0 208m 14m 3116 D 0.0 1.5 19:00.50 httpd
22645 apache 16 0 211m 14m 3124 D 0.0 1.5 17:14.04 httpd

top - 08:48:50 up 2 days, 21:04, 1 user, load average: 7.46, 4.50,
2.94
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.2%us, 1.7%sy, 0.0%ni, 0.0%id, 93.1%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 960184k used, 9900k free, 2992k buffers
Swap: 1966072k total, 984032k used, 982040k free, 31180k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 2.0 6.8 1:12.25 httpd
22642 apache 17 0 222m 85m 3116 R 1.0 9.0 19:05.52 httpd
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.72 kswapd0
22645 apache 16 0 211m 14m 3124 D 0.7 1.6 17:14.06 httpd
22641 apache 16 0 208m 15m 3116 D 0.3 1.6 19:00.51 httpd
367 root 11 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.09 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22643 apache 16 0 207m 13m 3116 D 0.0 1.4 19:36.93 httpd

top - 08:48:53 up 2 days, 21:04, 1 user, load average: 7.46, 4.50,
2.94
Tasks: 101 total, 1 running, 100 sleeping, 0 stopped, 0 zombie
Cpu(s): 5.3%us, 1.7%sy, 0.0%ni, 0.0%id, 91.7%wa, 0.7%hi,
0.7%si, 0.0%st
Mem: 970084k total, 960156k used, 9928k free, 2992k buffers
Swap: 1966072k total, 983068k used, 983004k free, 31464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1901 apache 18 0 201m 64m 3076 D 3.5 6.9 1:12.37 httpd
22642 apache 17 0 222m 85m 3116 D 1.4 9.1 19:05.57 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.73 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.10 top
22641 apache 16 0 208m 15m 3116 D 0.3 1.7 19:00.52 httpd
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.94 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
22645 apache 16 0 211m 15m 3124 D 0.0 1.6 17:14.06 httpd

top - 08:48:56 up 2 days, 21:04, 1 user, load average: 7.50, 4.56,
2.97
Tasks: 101 total, 3 running, 98 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.2%us, 0.9%sy, 0.0%ni, 0.0%id, 96.5%wa, 0.6%hi,
0.9%si, 0.0%st
Mem: 970084k total, 960380k used, 9704k free, 2936k buffers
Swap: 1966072k total, 983044k used, 983028k free, 31392k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.0 9.1 19:05.60 httpd
162 root 10 -5 0 0 0 D 0.3 0.0 6:51.74 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.11 top
22643 apache 16 0 207m 14m 3116 D 0.3 1.5 19:36.95 httpd
22645 apache 16 0 211m 15m 3124 D 0.3 1.6 17:14.07 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
3080 root 15 0 0 0 0 R 0.0 0.0 0:00.00 pdflush
3086 root 15 0 0 0 0 R 0.0 0.0 0:00.00
pdflush
3099 apache 17 0 8172 3052 2092 D 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.7 19:00.52 httpd

top - 08:49:00 up 2 days, 21:04, 1 user, load average: 7.62, 4.63,
3.00
Tasks: 101 total, 2 running, 99 sleeping, 0 stopped, 0 zombie
Cpu(s): 1.7%us, 1.0%sy, 0.0%ni, 0.0%id, 95.3%wa, 0.3%hi,
1.7%si, 0.0%st
Mem: 970084k total, 959064k used, 11020k free, 2936k buffers
Swap: 1966072k total, 983532k used, 982540k free, 31148k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 86m 3116 D 1.4 9.2 19:05.65 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.6 19:36.96 httpd
22645 apache 16 0 211m 16m 3124 D 0.3 1.7 17:14.08 httpd
3080 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3082 root 15 0 2316 996 784 R 0.0 0.1 0:00.11 top
3086 root 15 0 0 0 0 D 0.0 0.0 0:00.00 pdflush
3099 apache 17 0 8172 3076 2116 R 0.0 0.3 0:00.04 sendmail
22641 apache 16 0 208m 16m 3116 D 0.0 1.8 19:00.52 httpd

top - 08:49:03 up 2 days, 21:04, 1 user, load average: 7.62, 4.63,
3.00
Tasks: 103 total, 2 running, 101 sleeping, 0 stopped, 0 zombie
Cpu(s): 3.7%us, 1.7%sy, 0.0%ni, 0.0%id, 93.3%wa, 0.3%hi,
1.0%si, 0.0%st
Mem: 970084k total, 959560k used, 10524k free, 2928k buffers
Swap: 1966072k total, 983440k used, 982632k free, 30496k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22642 apache 17 0 222m 87m 3116 D 2.0 9.2 19:05.71 httpd
3102 root 16 0 9276 2300 1564 D 1.3 0.2 0:00.04 sendmail
162 root 10 -5 0 0 0 D 0.7 0.0 6:51.77 kswapd0
3082 root 15 0 2316 996 784 R 0.3 0.1 0:00.12 top
22641 apache 16 0 208m 17m 3116 D 0.3 1.8 19:00.53 httpd
22643 apache 16 0 207m 15m 3116 D 0.3 1.7 19:36.97 httpd
367 root 10 -5 0 0 0 D 0.0 0.0 0:38.13 kjournald
22645 apache 16 0 211m 16m 3124 R 0.0 1.7 17:14.08 httpd

==========================================

[root@support ~]# mailq
/var/spool/mqueue (6 requests)
-----Q-ID----- --Size-- -----Q-Time-----
------------Sender/Recipient----------- o51Cxkw2003214 8112 Tue Jun 1
08:59 jay@centritech.net
7BIT (Deferred: prog mailer (/usr/sbin/smrsh) exited with
EX_TEMPF)
“|/etc/smrsh/rt-mailgate
–queue 'ccs:
o51CuKTF003192 633211 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CGaSM001908 1170766 Tue Jun 1 08:16 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CuIdl003186 1355798 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51Ckond003072 3612709 Tue Jun 1 08:46 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
“|/etc/smrsh/rt-mailgate
–queue 'umx:
o51CutnV003202 3618441 Tue Jun 1 08:56 <>
7BIT (”|/etc/smrsh/rt-mailgate --queue ‘umx: client
helpdesk’ --ac)
"|/etc/smrsh/rt-mailgate
–queue 'umx:
Total requests: 6
[root@support ~]#

–

Cheers

Richard Skelton
Richard.Skelton@infineon.com Infineon Technologies UK Ltd
Infineon House
Great Western Court
Hunts Ground Road
Stoke Gifford
Bristol BS34 8HP
Tel +44(0)117 9528808

Cheers

Richard Skelton
Richard.Skelton@infineon.com Infineon Technologies UK Ltd
Infineon House
Great Western Court
Hunts Ground Road
Stoke Gifford
Bristol BS34 8HP
Tel +44(0)117 9528808

[Kenneth_Marshall]

But then my question becomes, “why did this happen all of a sudden?” We have not changed user loads, usage levels, etc. This is a dedicated box being used by about 5 active users and 20 intermittent (once a week) users. Is 1 Gb of memory not enought for RT with such a light loading?

And does anyone have any idea what those odd mailq entries are? I have to go in and delete several hundred of them every day or two.

Thanks,
Eriks

Hi Eriks,

Obviously, something did change or you would not be having a
problem. The harder part is to discover what changed and how
it is affecting your system. Those odd mailq entries if they
are being fed to RT could bump the load on your system. Do
you have any kind of filtering pre-RT to prevent bogus mail
from hitting the RT server?

Regards,
Ken

----- Original Message -----
From: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:07:58 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
Yes as I thought you are short on real memory.
Look at the swap.
Here is a busy 16 core Linux system of mine with load of free memory:-
procs -----------memory---------- —swap-- -----io---- --system–
----cpu---- r b swpd free buff cache si so bi bo in cs us sy
id wa

…lots of repetitive information deleted…

[Eriks_Goodwin-Pfiste]

I deleted the mailq, redirected the address for the odd mailq entries to an alternate mailbox, etc.

As for memory, I see from sorting “top” by memory usage that apache is the memory hog… Any idea what would cause this on an RT box?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2665 apache 15 0 207m 81m 3692 S 0.0 8.6 1:29.41 httpd
2387 apache 15 0 206m 78m 3692 S 0.0 8.3 1:44.80 httpd
2668 apache 17 0 206m 75m 3896 S 0.0 8.0 1:32.49 httpd
2496 apache 15 0 205m 70m 3880 S 0.0 7.5 1:12.07 httpd
2669 apache 15 0 204m 67m 3892 S 0.0 7.1 1:10.97 httpd
2391 apache 15 0 206m 65m 3636 S 0.0 6.9 1:07.24 httpd
2385 apache 15 0 210m 62m 3308 S 0.0 6.6 1:29.17 httpd
2401 apache 17 0 208m 57m 3644 S 0.0 6.1 1:18.16 httpd
2388 apache 15 0 204m 56m 3568 S 0.0 6.0 1:01.46 httpd
2389 apache 15 0 207m 56m 3824 S 0.0 5.9 1:10.61 httpd
2390 apache 15 0 208m 54m 3592 S 0.0 5.8 1:33.90 httpd
2493 apache 17 0 203m 50m 3844 S 0.0 5.4 1:02.30 httpd
2392 apache 15 0 207m 50m 3232 S 0.0 5.3 1:10.83 httpd
2386 apache 15 0 207m 47m 3592 S 0.0 5.0 1:24.97 httpd
2494 apache 15 0 205m 46m 3648 S 0.0 4.9 1:10.75 httpd
2492 apache 15 0 204m 43m 3812 S 0.0 4.6 1:11.58 httpd
2275 root 15 0 115m 15m 3116 S 0.0 1.6 0:04.33 httpd
2159 mysql 18 0 138m 8704 2792 S 0.0 0.9 0:01.67 mysqldFrom: “Kenneth Marshall” ktm@rice.edu
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Cc: “rt-users” rt-users@lists.bestpractical.com
Sent: Tuesday, June 1, 2010 9:55:06 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

But then my question becomes, “why did this happen all of a sudden?”
We have not changed user loads, usage levels, etc. This is a dedicated
box being used by about 5 active users and 20 intermittent (once a
week) users. Is 1 Gb of memory not enought for RT with such a light
loading?

And does anyone have any idea what those odd mailq entries are? I have
to go in and delete several hundred of them every day or two.

Thanks,
Eriks

Hi Eriks,

Obviously, something did change or you would not be having a
problem. The harder part is to discover what changed and how
it is affecting your system. Those odd mailq entries if they
are being fed to RT could bump the load on your system. Do
you have any kind of filtering pre-RT to prevent bogus mail
from hitting the RT server?

Regards,
Ken

----- Original Message -----
From: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:07:58 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
Yes as I thought you are short on real memory.
Look at the swap.
Here is a busy 16 core Linux system of mine with load of free memory:-
procs -----------memory---------- —swap-- -----io---- --system–
----cpu---- r b swpd free buff cache si so bi bo in cs us sy
id wa

…lots of repetitive information deleted…

[Kenneth_Marshall]

In versions prior to RT 3.8.8, certain searchs will result in the
use of a large amount of memory in the webserver process. You can
use a “graceful” restart of the apache server to reduce the size
of the process. Maybe an hourly cron job would do the trick?

Regards,
KenOn Tue, Jun 01, 2010 at 10:57:41AM -0400, Eriks Goodwin-Pfister wrote:

I deleted the mailq, redirected the address for the odd mailq entries to an alternate mailbox, etc.

As for memory, I see from sorting “top” by memory usage that apache is the memory hog… Any idea what would cause this on an RT box?

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2665 apache 15 0 207m 81m 3692 S 0.0 8.6 1:29.41 httpd
2387 apache 15 0 206m 78m 3692 S 0.0 8.3 1:44.80 httpd
2668 apache 17 0 206m 75m 3896 S 0.0 8.0 1:32.49 httpd
2496 apache 15 0 205m 70m 3880 S 0.0 7.5 1:12.07 httpd
2669 apache 15 0 204m 67m 3892 S 0.0 7.1 1:10.97 httpd
2391 apache 15 0 206m 65m 3636 S 0.0 6.9 1:07.24 httpd
2385 apache 15 0 210m 62m 3308 S 0.0 6.6 1:29.17 httpd
2401 apache 17 0 208m 57m 3644 S 0.0 6.1 1:18.16 httpd
2388 apache 15 0 204m 56m 3568 S 0.0 6.0 1:01.46 httpd
2389 apache 15 0 207m 56m 3824 S 0.0 5.9 1:10.61 httpd
2390 apache 15 0 208m 54m 3592 S 0.0 5.8 1:33.90 httpd
2493 apache 17 0 203m 50m 3844 S 0.0 5.4 1:02.30 httpd
2392 apache 15 0 207m 50m 3232 S 0.0 5.3 1:10.83 httpd
2386 apache 15 0 207m 47m 3592 S 0.0 5.0 1:24.97 httpd
2494 apache 15 0 205m 46m 3648 S 0.0 4.9 1:10.75 httpd
2492 apache 15 0 204m 43m 3812 S 0.0 4.6 1:11.58 httpd
2275 root 15 0 115m 15m 3116 S 0.0 1.6 0:04.33 httpd
2159 mysql 18 0 138m 8704 2792 S 0.0 0.9 0:01.67 mysqld

----- Original Message -----
From: “Kenneth Marshall” ktm@rice.edu
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Cc: “rt-users” rt-users@lists.bestpractical.com
Sent: Tuesday, June 1, 2010 9:55:06 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

On Tue, Jun 01, 2010 at 09:48:24AM -0400, Eriks Goodwin-Pfister wrote:

But then my question becomes, “why did this happen all of a sudden?”
We have not changed user loads, usage levels, etc. This is a dedicated
box being used by about 5 active users and 20 intermittent (once a
week) users. Is 1 Gb of memory not enought for RT with such a light
loading?

And does anyone have any idea what those odd mailq entries are? I have
to go in and delete several hundred of them every day or two.

Thanks,
Eriks

Hi Eriks,

Obviously, something did change or you would not be having a
problem. The harder part is to discover what changed and how
it is affecting your system. Those odd mailq entries if they
are being fed to RT could bump the load on your system. Do
you have any kind of filtering pre-RT to prevent bogus mail
from hitting the RT server?

Regards,
Ken

----- Original Message -----
From: “Richard Skelton” Richard.Skelton@infineon.com
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Sent: Tuesday, June 1, 2010 9:07:58 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

Hi Eriks,
Yes as I thought you are short on real memory.
Look at the swap.
Here is a busy 16 core Linux system of mine with load of free memory:-
procs -----------memory---------- —swap-- -----io---- --system–
----cpu---- r b swpd free buff cache si so bi bo in cs us sy
id wa

…lots of repetitive information deleted…

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

[Eriks_Goodwin-Pfiste]

I upgraded to 3.8.8 and all seems to be fine now.

I think that there were/are some user searches that were being used that were causing the overload.From: “Kenneth Marshall” ktm@rice.edu
To: “Eriks Goodwin-Pfister” eriks@centritech.net
Cc: “rt-users” rt-users@lists.bestpractical.com
Sent: Tuesday, June 1, 2010 11:12:36 AM
Subject: Re: [rt-users] CPU Wait Maxed Out

In versions prior to RT 3.8.8, certain searchs will result in the
use of a large amount of memory in the webserver process. You can
use a “graceful” restart of the apache server to reduce the size
of the process. Maybe an hourly cron job would do the trick?

Regards,
Ken