RT suddenly no longer works (blank titleless page for everything, no errors in apache)

Hello,

We have had a successfully operating RT installing for many months now.
This weekend, we upgraded perl and now RT no longer works at all. The
upgrade of perl was rolled back (it was just a minor update, so it
shouldn’t have caused a major problem, in any case). The problem
persists. We have tried reinstalling RT, enabling debugging and not
only does RT no longer work, but there are no errors in the Apache logs
and the debugging log I set up in /opt/rt3/var/log remains empty. None
of the configs changed prior or during the outage, except to add in
debugging options, which have had no visible effect.

Because our RT system is behind HTTPS, and also load-balanced using PEN,
it is hard to test with, e.g. telnet or bin/rt. Note, of course, that
it worked successfully using this configuration before.

We are using RT 3.6.5 on CentOS 5 (x86_64) running inside two Xen
virtual machines, themselves running on CentOS (x86_64). Perl version
is 5.8.8 (build 10), mod_perl is version 2.0.2 (build 6.3). All CPAN
dependencies are okay, verified with sbin/rt-test-dependencies.

Here are the relevant config files:
/opt/rt3/etc/RT_SiteConfig.pm:

Set( $rtname, ‘XXXXX’);
Set( $Organization, “XXXXXX”);
Set( $OWnerEmail , ‘XXXXXX’);
Set( $WebBaseURL, “https://XXXXXXX”);
Set( $Timezone, “US/Eastern”);
Set( $WebExternalAuth, 1);
Set($DatabaseHost , ‘XXXXX’);
Set($DatabaseRTHost , ‘XXXXX’);
Set($LogToFileName, “rt.log”);
Set($LogToFile, “debug”);
1;

/etc/httpd/conf/httpd.conf (relevant section)

ServerName XXXXX DocumentRoot /opt/rt3/share/html AddDefaultCharset UTF-8

PerlRequire /opt/rt3/bin/webmux.pl

SetHandler perl-script PerlHandler RT::Mason

SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM:+LOW

SSLCertificateFile XXXXXX
SSLCertificateKeyFile XXXXXX
SSLCertificateChainFile XXXXXX

    ErrorLog /var/log/httpd/error_log
    TransferLog /var/log/httpd/access_log

RedirectMatch permanent (.*)/$ https://XXXXXX$1/index.html

/etc/httpd/conf.d/perl.conf

LoadModule perl_module modules/mod_perl.so

and a large number of commented out things [we haven’t otherwise

modified this file]

So, at this point, we need some advice on the direction to take, and
what things to try. I don’t know enough about RT internals to properly
debug this situation. I’ve read some of the documentation, but again,
since we didn’t change any config before this broke, it shouldn’t make a
difference.

Thanks in advance.

  • Joel

Hello,

We have had a successfully operating RT installing for many months now.
This weekend, we upgraded perl and now RT no longer works at all. The
upgrade of perl was rolled back (it was just a minor update, so it
shouldn’t have caused a major problem, in any case). The problem
persists.

It looks like RedHat replaced Scalar::Util with a broken
version…again. Try reinstalling it?

We have tried reinstalling RT, enabling debugging and not

Jesse Vincent wrote:

Hello,

We have had a successfully operating RT installing for many months now.
This weekend, we upgraded perl and now RT no longer works at all. The
upgrade of perl was rolled back (it was just a minor update, so it
shouldn’t have caused a major problem, in any case). The problem
persists.

It looks like RedHat replaced Scalar::Util with a broken
version…again. Try reinstalling it?

I installed (via CPAN) Scalar::Util and it installed fine, but the
problem persists. There is no perl-Scalar-Util package installed (or
any permutation thereof). Any other packages I should check?

perl -MCPAN -e ‘install “G/GB/GBARR/Scalar-List-Utils-1.18.tar.gz”’

Then, restart apache.

I also did an upgrade install of RT somewhere in there, which may or may
not have been necessary. We’re working perfectly now.

We are using RT 3.6.5 on CentOS 5 (x86_64) running inside two Xen
virtual machines, themselves running on CentOS (x86_64). Perl version
is 5.8.8 (build 10), mod_perl is version 2.0.2 (build 6.3). All CPAN
dependencies are okay, verified with sbin/rt-test-dependencies.

We are using the same RT with RHEL4 (x86_64) with no virtualization in
between and now have the same problem.

Seems that the infrastructure staff had updated the perl.rpm a few
weeks ago and we did not notice at first because apache has been up
for many months now (all config changes always applied “gracefully”).

Today we had to restart apache “properly” for the first time in months
and now see the same thing - no error messages, no logs any more,
empty pages and RT offers the main URL as “httpd/unix-directory”, the
bookmarked pages are all returned empty (with a 200 result code but
zero length in the log files).

We are desperately in need of ideas now!

Michael

Chris McCafferty wrote:

perl -MCPAN -e ‘install “G/GB/GBARR/Scalar-List-Utils-1.18.tar.gz”’

Then, restart apache.

I also did an upgrade install of RT somewhere in there, which may or may
not have been necessary. We’re working perfectly now.


The rt-users Archives

SAVE THOUSANDS OF DOLLARS ON RT SUPPORT:

If you sign up for a new RT support contract before December 31, we’ll take
up to 20 percent off the price. This sale won’t last long, so get in touch today.
Email us at sales@bestpractical.com or call us at +1 617 812 0745.

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

That fixed it! Thank you very much.

  • Joel