Continuous 500 errors

I recently upgraded RT from 3.6.3 to 3.8.1 which went fairly smooth
however after making the db changes suggested from running:

perl etc/upgrade/schema.mysql-4.0-4.1.pl db user pass > sql.queries

RT works but I get completely random 500 errors in the RT web interface.
I’ve set the rt log to debug mode but nothing abnormal shows there. I
get this over and over in my apache error log though:

[Fri Oct 10 14:30:56 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server “/opt/rt3/bin/mason_handler.fcgi”
[Fri Oct 10 14:30:56 2008] [warn] FastCGI: server
"/opt/rt3/bin/mason_handler.fcgi" (pid 2326) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 10 14:30:56 2008] [warn] FastCGI: server
"/opt/rt3/bin/mason_handler.fcgi" restarted (pid 2590)

I’ve found a lot of other issues involving 500 errors but nothing
matching my issue exactly. What could be causing 0 headers to be sent to
mason_handler.fcgi ? And why does it only happen randomly?

-steven

Running out of memory perhaps? I’ll occationally get these if my search
results are too big and you try to display a ticket (I get some odd bug
where it tries to get every result of a search into memory on a single
ticket display). It’s really hard to say what though…

SIGSEGV 11 Core Invalid memory reference

Curtis

scollins wrote:

It happens instantly during general interface navigation and it’s
totally random. Not necessarily searching or long queries. And it did
not happen prior to updating the mysql db tables to support the new
3.8.1 install.

-stevenOn Fri, 2008-10-10 at 15:22 -0400, Curtis Bruneau wrote:

Running out of memory perhaps? I’ll occationally get these if my search
results are too big and you try to display a ticket (I get some odd bug
where it tries to get every result of a search into memory on a single
ticket display). It’s really hard to say what though…

SIGSEGV 11 Core Invalid memory reference

Curtis

scollins wrote:

I recently upgraded RT from 3.6.3 to 3.8.1 which went fairly smooth
however after making the db changes suggested from running:

perl etc/upgrade/schema.mysql-4.0-4.1.pl db user pass > sql.queries

RT works but I get completely random 500 errors in the RT web interface.
I’ve set the rt log to debug mode but nothing abnormal shows there. I
get this over and over in my apache error log though:

[Fri Oct 10 14:30:56 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server “/opt/rt3/bin/mason_handler.fcgi”
[Fri Oct 10 14:30:56 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 2326) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 10 14:30:56 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 2590)

I’ve found a lot of other issues involving 500 errors but nothing
matching my issue exactly. What could be causing 0 headers to be sent to
mason_handler.fcgi ? And why does it only happen randomly?

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

This is getting very frustrating. I’ve completely re-installed a fresh
RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the old
perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki. All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 17 11:57:45 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm file
but that does not seem to have yielded any usefull information regarding
the segfaults. I really need some help with this. RT is completely
unusable for me at this point.

-steven

Install Scalar::util from cpan and not from RPM. RedHat’s RPM install
is broken.

scollins wrote:

This is getting very frustrating. I’ve completely re-installed a fresh
RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the old
perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki. All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 17 11:57:45 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm file
but that does not seem to have yielded any usefull information regarding
the segfaults. I really need some help with this. RT is completely
unusable for me at this point.

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

The install wiki doesn’t say to install that via rpm and I don’t. I
double checked and it is in fact installed via cpan. I force
re-installed to be sure:

Scalar::Util is up to date (1.19).

500 errors still happening. Any other ideas?

-stevenOn Fri, 2008-10-17 at 12:33 -0400, Drew Barnes wrote:

Install Scalar::util from cpan and not from RPM. RedHat’s RPM install
is broken.

scollins wrote:

This is getting very frustrating. I’ve completely re-installed a fresh
RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the old
perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki. All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 17 11:57:45 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm file
but that does not seem to have yielded any usefull information regarding
the segfaults. I really need some help with this. RT is completely
unusable for me at this point.

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Try searching list archives for segmentation faults:

Could also be Text::Quoted needs re-install. I also see some mysql
possibilities in there.

Good luck.

scollins wrote:

This is getting very frustrating. I’ve completely re-installed a fresh
RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the old
perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki. All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 17 11:57:45 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm file
but that does not seem to have yielded any usefull information regarding
the segfaults. I really need some help with this. RT is completely
unusable for me at this point.

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Drew Barnes
Applications Analyst
Network Resources Department
Raymond Walters College
University of Cincinnati

Yes, I had already tried re-installing Text::Quoted as I had been
searching for an answer in the user list long before posting to it.
Seems there are a lot of similar issues but nothing exactly like mine.
It’s either my system or the RT version I’m using (3.8.1) and it’s not
easy to eliminate either of these on a production system just for
trouble shooting :frowning:

-stevenOn Fri, 2008-10-17 at 14:56 -0400, Drew Barnes wrote:

Try searching list archives for segmentation faults:
Carbon60: Managed Cloud Services

Could also be Text::Quoted needs re-install. I also see some mysql
possibilities in there.

Good luck.

scollins wrote:

This is getting very frustrating. I’ve completely re-installed a fresh
RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the old
perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki. All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Fri Oct 17 11:57:45 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm file
but that does not seem to have yielded any usefull information regarding
the segfaults. I really need some help with this. RT is completely
unusable for me at this point.

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Yes, I had already tried re-installing Text::Quoted as I had been
searching for an answer in the user list long before posting to it. Seems
there are a lot of similar issues but nothing exactly like mine. It’s
either my system or the RT version I’m using (3.8.1) and it’s not easy to
eliminate either of these on a production system just for trouble shooting
:frowning:

-steven

I’m also having this issue with 3.8.1. I usually notice it when calling RT
with rt-mailgate. My procmail logs have the random error “500 Internal
Server Error”

I just noticed it today that some emails have been getting lost!
_BB

Are you upgraded from earlier versions? Are you on mysql? If both
answers are ‘yes’ then you didn’t read UPGRADING.mysql. I’m 75% sure
it’s the reason.On Fri, Dec 12, 2008 at 12:01 AM, Brendon Baumgartner brendon@brendon.com wrote:

On Fri, October 17, 2008 11:25 am, scollins wrote:

Yes, I had already tried re-installing Text::Quoted as I had been
searching for an answer in the user list long before posting to it. Seems
there are a lot of similar issues but nothing exactly like mine. It’s
either my system or the RT version I’m using (3.8.1) and it’s not easy to
eliminate either of these on a production system just for trouble shooting
:frowning:

-steven

I’m also having this issue with 3.8.1. I usually notice it when calling RT
with rt-mailgate. My procmail logs have the random error “500 Internal
Server Error”

I just noticed it today that some emails have been getting lost!

_BB

On Fri, 2008-10-17 at 14:56 -0400, Drew Barnes wrote:

Try searching list archives for segmentation faults:
Carbon60: Managed Cloud Services
earch_forum=forum_3;search_string=segmentation%20fault;search_type=AND&
sb=post_time

Could also be Text::Quoted needs re-install. I also see some mysql
possibilities in there.

Good luck.

scollins wrote:

This is getting very frustrating. I’ve completely re-installed a
fresh RT and database following the instructions at
RHEL4InstallGuide - Request Tracker Wiki exactly. The only
exception is I’m using Module::Build 0.30 from cpan instead of the
old perl-Module-Build-0.2612-1.rhel4.noarch.rpm listed in the wiki.
All
other perl modules and rpms are consistant with the install guide.

I am still seeing 500 errors randomly while navigating the RT
insterface:

[Fri Oct 17 11:57:43 2008] [error] [client myIP] FastCGI: incomplete
headers (0 bytes) received from server
“/opt/rt3/bin/mason_handler.fcgi”, referer:
http://rt.mydomain.com/index.html
[Fri Oct 17 11:57:43 2008] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 18996) terminated due to
uncaught signal ‘11’ (Segmentation fault) [Fri Oct 17 11:57:45 2008]
[warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 19036)

I also set Set($LogStackTraces , ‘debug’); in the RT_SiteConfig.pm
file but that does not seem to have yielded any usefull information
regarding the segfaults. I really need some help with this. RT is
completely unusable for me at this point.

-steven


The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Best regards, Ruslan.

Yes. I upgraded from 3.6.x and am on mysql. I did read UPGRADING.mysql a
few weeks ago when I did the upgrade and was very careful.

I just went through and compared a vanilla schema to mine. There were
some differences, but nothing I thought was major. I also updated one
perl package that I read might be related to this error.

Thanks. Fingers crossed!

Ruslan Zakirov wrote:

Hi-

The problem persists after checking my schema and a few perl modules.
I’m using the mailgate with httpS and I get intermittent segfaults from
mason/apache and the promail logs show 500 errors. I found another
thread that talked about mailgate causing segfaults and noauth disabled.
I’m not quote sure what that meant, but I don’t think thats exactly my
issue.

I’ll be digging further. Any .02 would be helpful.

_BB

Ruslan Zakirov wrote: