I had my RT server crash yesterday (or, more accurately, decide to not
accept new e-mail).
Our server handles a high amount of traffic and had been running for
almost a year before this issue. After about 30 minutes of debugging,
I discovered that the /tmp directory was completely full of File::Temp
files. It appears that RT creates a temporary file/directory each time
an e-mail arrives. These files don’t appear to be properly cleaned up
by the system.
In investigating, I found a line in lib/RT/Interface/Email.pm:
last if ( $fh, $temp_file ) = eval { File::Temp::tempfile(undef, UNLINK
=> 0) };
UNLINK => 0 specifically tells File::Temp to NOT remove a file when the
object is destroyed… this looks like some of the NFS logic, so maybe
that was intentional, but I believe this is where the file is created
but not removed.
Once I stopped apache, cleaned up the /tmp directory, and restarted,
the system ran fine again.
Hopefully you guys can track this down… everyone else, check your
/tmp directory!
Cheers,
Chad
I believe this issue was fixed in 3.0.11. I had the same problem in 3.0.10 but
it magically disappeared when I upgraded 
-Jeff::On Thursday 24 June 2004 09:15 am, Chad Scott chad@idworld.net wrote:
::I had my RT server crash yesterday (or, more accurately, decide to not
::accept new e-mail).
::
::Our server handles a high amount of traffic and had been running for
::almost a year before this issue. After about 30 minutes of debugging,
::I discovered that the /tmp directory was completely full of File::Temp
::files. It appears that RT creates a temporary file/directory each time
::an e-mail arrives. These files don’t appear to be properly cleaned up
::by the system.
::
::In investigating, I found a line in lib/RT/Interface/Email.pm:
::
::last if ( $fh, $temp_file ) = eval { File::Temp::tempfile(undef, UNLINK
::=> 0) };
::
::UNLINK => 0 specifically tells File::Temp to NOT remove a file when the
::object is destroyed… this looks like some of the NFS logic, so maybe
::that was intentional, but I believe this is where the file is created
::but not removed.
::
::Once I stopped apache, cleaned up the /tmp directory, and restarted,
::the system ran fine again.
::
::Hopefully you guys can track this down… everyone else, check your
::/tmp directory!
::
::Cheers,
::Chad
::
::_______________________________________________
::The rt-users Archives
::
::RT Developer and Administrator training is coming to LA, DC and Frankfurt
:: this spring and summer. http://bestpractical.com/services/training.html
::
::Sign up early, as class space is limited.
I see the same issue in 3.0.11, unfortunately.On Jun 24, 2004, at 6:25 AM, jeff rak wrote:
I believe this issue was fixed in 3.0.11. I had the same problem in
3.0.10 but
it magically disappeared when I upgraded
I see the same issue in 3.0.11, unfortunately.
What new tempfiles is 3.0.11 creating for you? What’s in them? What
actions in RT generate tempfiles? Are you using mod_perl or fastcgi?
What database are you using?
Jesse> On Jun 24, 2004, at 6:25 AM, jeff rak wrote:
I believe this issue was fixed in 3.0.11. I had the same problem in
3.0.10 but
it magically disappeared when I upgraded
The rt-users Archives
RT Developer and Administrator training is coming to LA, DC and Frankfurt
this spring and summer.
http://bestpractical.com/services/training.html
Sign up early, as class space is limited.
Here’s an example of a directory listing for my system. I’m running RT
3.0.11, with perl 5.8.3 on Apache/1.3.29 (Unix) mod_perl/1.29
mod_ssl/2.8.16 OpenSSL/0.9.7c. This is all on a FreeBSD 4.9-RELEASE-p1
system.
Below is a snippet of directory listing. Notice some of the files are
dated yesterday (PDT), which is when I cleared the directory out due to
the crash. RT was upgraded to 3.0.11 far before that.
ls -la /tmp
-rw------- 1 nobody wheel 1567 Jun 24 11:45 0G1BZmuRrx
-rw------- 1 nobody wheel 1899 Jun 24 14:36 0MZYb1QcSy
-rw------- 1 nobody wheel 3716 Jun 23 20:13 0pE2uebslm
-rw------- 1 nobody wheel 2463 Jun 24 03:58 0qfC9F2jgo
-rw------- 1 nobody wheel 1595 Jun 24 09:36 1X7KbKs1M7
-rw------- 1 nobody wheel 32691 Jun 24 02:21 1jJWWlMTHZ
-rw------- 1 nobody wheel 4197 Jun 24 02:27 22O3IRL5v2
-rw------- 1 nobody wheel 2129 Jun 24 13:57 25mY7wdI5h
-rw------- 1 nobody wheel 1886 Jun 23 20:39 2OF1k9vqdr
drwx------ 2 nobody wheel 512 Jun 23 15:57 3lbooSmrSW
-rw------- 1 nobody wheel 1908 Jun 23 16:10 4Qutt7K4rY
-rw------- 1 nobody wheel 1588 Jun 24 10:45 4T0EZngfsO
-rw------- 1 nobody wheel 4190 Jun 24 06:08 4UEe1w3TeD
-rw------- 1 nobody wheel 4628 Jun 23 17:41 4WWs9Duk90
-rw------- 1 nobody wheel 1662 Jun 24 06:26 4fxrUrmWpP
-rw------- 1 nobody wheel 3054 Jun 23 16:14 4iRs73VArR
-rw------- 1 nobody wheel 4613 Jun 24 12:18 5ALTfZTd7g
-rw------- 1 nobody wheel 2103 Jun 24 11:42 5AVRiu5JuA
-rw------- 1 nobody wheel 5725 Jun 24 11:09 672E2jqhe8
-rw------- 1 nobody wheel 4166 Jun 24 07:37 6AumAxA6w5
-rw------- 1 nobody wheel 4931 Jun 23 18:48 6GSsaP5BRs
-rw------- 1 nobody wheel 773279 Jun 24 10:23 6GXabF4Hck
-rw------- 1 nobody wheel 5282 Jun 24 12:32 6eHKdLJxEr
-rw------- 1 nobody wheel 3605 Jun 24 09:35 80ex3HrtRx
-rw------- 1 nobody wheel 2577 Jun 24 03:43 8NT3XbWzAG
-rw------- 1 nobody wheel 2300 Jun 24 11:35 8se864YEEgOn Jun 24, 2004, at 3:30 PM, Chad Scott wrote:
I see the same issue in 3.0.11, unfortunately.
On Jun 24, 2004, at 6:25 AM, jeff rak wrote:
I believe this issue was fixed in 3.0.11. I had the same problem in
3.0.10 but
it magically disappeared when I upgraded
The rt-users Archives
RT Developer and Administrator training is coming to LA, DC and
Frankfurt this spring and summer.
http://bestpractical.com/services/training.html
Sign up early, as class space is limited.
The files contain the RFC822 data for a message, as if it were about to
be written to an mbox. I’ll send you one privately if you really want
to see one.
I don’t know what actions in RT generate temp files… it appears to be
a message (any message) coming into the system.
I’m using mod_perl, static within Apache. The database is mySQL.
I’ll do anything you’d like to help you nail it down.
Thanks for taking a look, Jesse.
Thanks,
ChadOn Jun 24, 2004, at 3:36 PM, Jesse Vincent wrote:
On Thu, Jun 24, 2004 at 03:30:39PM -0700, Chad Scott wrote:
I see the same issue in 3.0.11, unfortunately.
What new tempfiles is 3.0.11 creating for you? What’s in them? What
actions in RT generate tempfiles? Are you using mod_perl or fastcgi?
What database are you using?
Jesse
On Jun 24, 2004, at 6:25 AM, jeff rak wrote:
I believe this issue was fixed in 3.0.11. I had the same problem in
3.0.10 but
it magically disappeared when I upgraded
The rt-users Archives
RT Developer and Administrator training is coming to LA, DC and
Frankfurt
this spring and summer.
http://bestpractical.com/services/training.html
Sign up early, as class space is limited.
–
I’m using mod_perl, static within Apache. The database is mySQL.
I’ll do anything you’d like to help you nail it down.
Just to be sure, you’ve stopped and started apache since upgrading RT to
3.0.11 and the web ui now tells you it’s running 3.0.11?
Yeah.
Part of my “push” is to stop, load, and restart. The UI says 3.0.11.On Jun 24, 2004, at 3:51 PM, Jesse Vincent wrote:
I’m using mod_perl, static within Apache. The database is mySQL.
I’ll do anything you’d like to help you nail it down.
Just to be sure, you’ve stopped and started apache since upgrading RT
to
3.0.11 and the web ui now tells you it’s running 3.0.11?
Next question. If you shutdown apache, do the tempfiles go away?On Thu, Jun 24, 2004 at 03:54:42PM -0700, Chad Scott wrote:
Yeah.
Part of my “push” is to stop, load, and restart. The UI says 3.0.11.
On Jun 24, 2004, at 3:51 PM, Jesse Vincent wrote:
I’m using mod_perl, static within Apache. The database is mySQL.
I’ll do anything you’d like to help you nail it down.
Just to be sure, you’ve stopped and started apache since upgrading RT
to
3.0.11 and the web ui now tells you it’s running 3.0.11?
Unfortunately, no. :(On Jun 24, 2004, at 4:09 PM, Jesse Vincent wrote:
Next question. If you shutdown apache, do the tempfiles go away?
Unfortunately, no. 
Chad,
How’s your perl? If I point you in the right direction to triage
this one can you poke around?
JesseOn Jun 24, 2004, at 4:09 PM, Jesse Vincent wrote:
Next question. If you shutdown apache, do the tempfiles go away?
I’m not a master hacker like yourself, but I hold my own.
Point away and I’ll do what I can.
Ok. so the first thing we want to do is figure out which tempfile
is hanging around. Have a look in lib/RT/EmailParser.pm and
lib/RT/Interface/Email.pm (particularly sub Gateway). Each time RT
creates or asks to create a tempfile, log the name of that file and
where in the code it’s creating it. Once we know which tempfile isn’t
being cleaned, then we can see how to whack it.
J> On Jun 25, 2004, at 9:33 AM, Jesse Vincent wrote:
On Thu, Jun 24, 2004 at 04:11:50PM -0700, Chad Scott wrote:
Unfortunately, no.
Chad,
How’s your perl? If I point you in the right direction to triage
this one can you poke around?
Jesse
On Jun 24, 2004, at 4:09 PM, Jesse Vincent wrote:
Next question. If you shutdown apache, do the tempfiles go away?
–