Emails with attachments keep inserting into RT

RT Users
1

I’m having a problem with large mail attachment, but it is different than
what I’ve read in the mailing list archives. The problem occurs when the
attachments are about 250 Kbytes and bigger and they keep getting inserted
into RT untill I manually delete them from the mail queue.

What happens is that the ticket does get created or appended to with the
attachment, but the email stays in the mail queue.

Thanks for any help or light you can shed on this.

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# ls -la 11.

-rw------- 1 cyrus mail 631472 Jan 28 13:27 11.

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# cat 11. |
/usr/bin/rt-mailgate --timeout 999 --debug --queue General --action
correspond --url http://tix.xyz.com http://tix.netcal.com/

Connecting to .xyz Domain Names | Join Generation XYZ at
/usr/bin/rt-mailgate line 99, <> line 1.

An Error Occurred

500 Internal Server Error

This is /usr/bin/rt-mailgate exiting because of an undefined server error at
/usr/bin/rt-mailgate line 147, <> line 1.

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts#

RT’s log from running the above…

Jan 28 13:31:03 localhost RT: Converting ‘us-ascii’ to ‘utf-8’ for
text/plain -

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

Jan 28 13:31:04 localhost RT: About to think about scrips for transaction
#4743

Jan 28 13:31:04 localhost RT: About to think about scrips for transaction
#4744

Jan 28 13:31:04 localhost RT: About to think about scrips for transaction
#4745

Jan 28 13:31:04 localhost RT: About to think about scrips for transaction
#4746

Jan 28 13:31:05 localhost RT: About to think about scrips for transaction
#4747

Jan 28 13:31:05 localhost RT: About to prepare scrips for transaction #4747

Jan 28 13:31:05 localhost RT: Found 3 scrips

Jan 28 13:31:06 localhost RT: About to commit scrips for transaction #4747

Jan 28 13:31:06 localhost RT: About to think about scrips for transaction
#4748

No big packet errors in mysql log…

grep -i ‘packet big’ mysql.log | wc –l

0

Version info:

libdbd-mysql-perl 2.9006-1

mysql-server-4.1 4.1.11-3

request-tracker3.4 3.4.4-1

2

We had the same problem! I thought it was a problem with Oracle, but
you are running MySQL. It timed out via email, and took a very long
time in the web UI as well. After a week of troubleshooting with the
DBAs and others, we couldn’t find where the problem was. Doing a
straight insert into the attachments table worked fine. And the same
thing worked fine on the MySQL instance in Dev. We ended up having to
enforce attachment limit sizes around 1MB. But this didn’t help all of
the time, either, because the attachment limit works on individual
attachments, not the sum of all of those. So if someone attached 6 500K
objects, we’d have the problem since they were trying to put a total of
3MB through.

I thought it may be the mail server, but not since the web UI also takes
forever. Could you try testing that portion, with something like a 2MB
or larger attachment and see if it takes a long time to return? It
appears something in RT is telling the mail server to give up and try
again later, even though it eventually processes the ticket information.

[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Brandon
HallSent: Saturday, January 28, 2006 2:09 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] emails with attachments keep inserting into
RT

I'm having a problem with large mail attachment, but it is

different than what I’ve read in the mailing list archives. The problem
occurs when the attachments are about 250 Kbytes and bigger and they
keep getting inserted into RT untill I manually delete them from the
mail queue.

What happens is that the ticket does get created or appended to

with the attachment, but the email stays in the mail queue.

Thanks for any help or light you can shed on this.

		 

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# ls

-la 11.

-rw-------  1 cyrus mail 631472 Jan 28 13:27 11. 

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# cat

  1. | /usr/bin/rt-mailgate --timeout 999 --debug --queue General
    –action correspond --url http://tix.xyz.com http://tix.netcal.com/

    Connecting to .xyz Domain Names | Join Generation XYZ
    http://tix.xyz.com/REST/1.0/NoAuth/mail-gateway at
    /usr/bin/rt-mailgate line 99, <> line 1.

    An Error Occurred

    500 Internal Server Error

    This is /usr/bin/rt-mailgate exiting because of an undefined
    server error at /usr/bin/rt-mailgate line 147, <> line 1.

    localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts#

    RT’s log from running the above…

    Jan 28 13:31:03 localhost RT: Converting ‘us-ascii’ to ‘utf-8’
    for text/plain -

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:03 localhost RT: Guessed encoding: ascii

    Jan 28 13:31:04 localhost RT: About to think about scrips for
    transaction #4743

    Jan 28 13:31:04 localhost RT: About to think about scrips for
    transaction #4744

    Jan 28 13:31:04 localhost RT: About to think about scrips for
    transaction #4745

    Jan 28 13:31:04 localhost RT: About to think about scrips for
    transaction #4746

    Jan 28 13:31:05 localhost RT: About to think about scrips for
    transaction #4747

    Jan 28 13:31:05 localhost RT: About to prepare scrips for
    transaction #4747

    Jan 28 13:31:05 localhost RT: Found 3 scrips

    Jan 28 13:31:06 localhost RT: About to commit scrips for
    transaction #4747

    Jan 28 13:31:06 localhost RT: About to think about scrips for
    transaction #4748

    No big packet errors in mysql log…

    grep -i ‘packet big’ mysql.log | wc -l

    0

    Version info:

    libdbd-mysql-perl 2.9006-1

    mysql-server-4.1 4.1.11-3

    request-tracker3.4 3.4.4-1

3

We had the same problem! I thought it was a problem with Oracle, but
you are running MySQL. It timed out via email, and took a very long
time in the web UI as well. After a week of troubleshooting with the
DBAs and others, we couldn’t find where the problem was. Doing a
straight insert into the attachments table worked fine. And the same
thing worked fine on the MySQL instance in Dev. We ended up having to
enforce attachment limit sizes around 1MB. But this didn’t help all of
the time, either, because the attachment limit works on individual
attachments, not the sum of all of those. So if someone attached 6 500K
objects, we’d have the problem since they were trying to put a total of
3MB through.

I thought it may be the mail server, but not since the web UI also takes
forever. Could you try testing that portion, with something like a 2MB
or larger attachment and see if it takes a long time to return? It
appears something in RT is telling the mail server to give up and try
again later, even though it eventually processes the ticket information.

Hmm. I think its comforting knowing I’m not the only one :slight_smile:

I think the below error is really the root of the problem. It appears
mail gate proxies the connection to the web interface which has a
problem somehow, somewhere.

I’m also using fastcgi. I wonder if this problem would occur with
mod_perl? I originally tried mod_perl with RT, but it was not working
out. I don’t recall the specific reason, but it was either crashing or
just plain FUBAR’ed.

Like you said, while the server is processing an attachment, a mason
process causes a very high CPU spike for approximately 60 seconds. I
imagine this is some how related.

libcgi-fast-perl 5.8.7-7
libapache-mod-fastcgi 2.4.2-7

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# cat 11. |
/usr/bin/rt-mailgate --timeout 999 --debug --queue General --action
correspond --url http://tix.xyz.com

Connecting to http://tix.xyz.com/REST/1.0/NoAuth/mail-gateway at
/usr/bin/rt-mailgate line 99, <> line 1.

An Error Occurred

500 Internal Server Error
This is /usr/bin/rt-mailgate exiting because of an undefined server
error at /usr/bin/rt-mailgate line 147, <> line 1.
localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts#

4

Like you said, while the server is processing an attachment, a mason
process causes a very high CPU spike for approximately 60 seconds. I
imagine this is some how related.

Just for kicks, turn up the --timeout on the fastcgi script too and see
if it runs to completion?

5

Like you said, while the server is processing an attachment, a mason
process causes a very high CPU spike for approximately 60 seconds. I
imagine this is some how related.

Just for kicks, turn up the --timeout on the fastcgi script too and see
if it runs to completion?

libcgi-fast-perl 5.8.7-7
libapache-mod-fastcgi 2.4.2-7

localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts# cat 11. |
FastCgiServer /usr/share/request-tracker3.4/libexec/mason_handler.fcgi -idle-timeout 300 -processes 3 /usr/bin/rt-mailgate --timeout 999 --debug --queue General --action
correspond --url http://tix.xyz.com

Connecting to http://tix.xyz.com/REST/1.0/NoAuth/mail-gateway at
/usr/bin/rt-mailgate line 99, <> line 1.

An Error Occurred

500 Internal Server Error
This is /usr/bin/rt-mailgate exiting because of an undefined server
error at /usr/bin/rt-mailgate line 147, <> line 1.
localhost:/var/spool/cyrus/mail/b/user/user_user_com/Drafts#

Excellent! That did it!

FastCgiServer /usr/share/request-tracker3.4/libexec/mason_handler.fcgi
-idle-timeout 300 -processes 3

6

Yes, but seems like you are just treating the symptom and not the
disease. That is, why is it taking so long for the process to complete
that the timeout needs to be set to 5 minutes?

7

Yes, but seems like you are just treating the symptom and not the
disease. That is, why is it taking so long for the process to complete
that the timeout needs to be set to 5 minutes?

There’s a pessimal failure mode in MIME-Tools. I believe you may be
running into:

http://svn.bestpractical.com/cgi-bin/index.cgi/bps/revision/?rev=3892