Apache core dump when RT4 is activated

When I include my rt4.conf into Apache, I get apache dumping core. Not sure what is the issue, but here is what I get from gdb:

(gdb) bt full
#0 0x00000008116eca10 in ?? ()
No symbol table info available.
#1 0x0000000806144fdd in ENGINE_remove () from /usr/local/lib/libcrypto.so.9
No symbol table info available.
#2 0x0000000806145485 in ENGINE_up_ref () from /usr/local/lib/libcrypto.so.9
No symbol table info available.
#3 0x00000008061449bb in ENGINE_cleanup () from /usr/local/lib/libcrypto.so.9
No symbol table info available.
#4 0x000000080615784f in sk_pop_free () from /usr/local/lib/libcrypto.so.9
No symbol table info available.
#5 0x000000080614499c in ENGINE_cleanup () from /usr/local/lib/libcrypto.so.9
No symbol table info available.
#6 0x0000000805a982e8 in ?? () from /usr/local/libexec/apache24/mod_ssl.so
No symbol table info available.
#7 0x000000080159ad1e in apr_pool_destroy () from /usr/local/lib/libapr-1.so.0
No symbol table info available.
#8 0x000000080159ad05 in apr_pool_destroy () from /usr/local/lib/libapr-1.so.0
No symbol table info available.
#9 0x0000000000434999 in destroy_and_exit_process ()
No symbol table info available.
#10 0x0000000000434959 in main ()
No symbol table info available.
(gdb)

You’ll need to include a bit more information:

  • What version of RT, specifically?
  • What version of Apache? Looks like 2.4, but what version of 2.4?
  • Which deployment mechanism (mod_fcgi, mod_fastcgi, mod_perl, etc)?

Try removing sections of rt4.conf until you determine which section causes Apache to dump core. It looks like this crash is from openssl, so I expect you’ll find the problem lies in how you’re configuring TLS for the vhost.

This is rt-4.4.3

Apache/2.4.38

I am using the standard mod_perl deployment mechanism.

Looking at the rt4.conf, I don’t see anything that is removable.

<VirtualHost *:80>
  ServerName rt.mydomain.com
  ServerAdmin odhiambo@gmail.com
  DocumentRoot "/opt/rt4/share/html"
  # optional apache logs for RT
  ErrorLog /opt/rt4/var/log/apache2.error
  TransferLog /opt/rt4/var/log/apache2.access
  # LogLevel debug
  <Location />
    Require all granted
    SetHandler modperl
    PerlResponseHandler Plack::Handler::Apache2
    PerlSetVar psgi_app /opt/rt4/sbin/rt-server
  </Location>
  <Perl>
    use Plack::Handler::Apache2;
    Plack::Handler::Apache2->preload("/opt/rt4/sbin/rt-server");
  </Perl>
</VirtualHost>

[edited for formatting – @chmrr]

In general, mod_perl is not suggested deployment strategy – it leads to weird bugs like this, since the perl process is embedded into Apache. My suspicion is that this is caused by differing versions of openssl compiled into Apache from what is compiled into Perl, which is leading to the segfault.

Try removing the <Perl></Perl> sections. If it still crashes, we can rule out interactions with loading the OpenSSL libraries from Perl.

With the … block removed, the crash ceases!

Now, I need to find a way to track down the OpenSSL issue on my FreeBSD box.

root@waridi:/usr/local/etc/apache24/Includes # ldd /usr/local/libexec/apache24/mod_perl.so

/usr/local/libexec/apache24/mod_perl.so:

libperl.so.5.28 => /usr/local/lib/perl5/5.28/mach/CORE/libperl.so.5.28 (0x801400000)

libthr.so.3 => /lib/libthr.so.3 (0x80191a000)

libm.so.5 => /lib/libm.so.5 (0x801b42000)

libcrypt.so.5 => /lib/libcrypt.so.5 (0x801d71000)

libutil.so.9 => /lib/libutil.so.9 (0x801f90000)

libc.so.7 => /lib/libc.so.7 (0x800823000)

root@waridi:/usr/local/etc/apache24/Includes # ldd /usr/local/sbin/httpd

/usr/local/sbin/httpd:

libpcre.so.1 => /usr/local/lib/libpcre.so.1 (0x8008b5000)

libaprutil-1.so.0 => /usr/local/lib/libaprutil-1.so.0 (0x800b53000)

libdb-6.2.so => /usr/local/lib/libdb-6.2.so (0x800d7f000)

libgdbm.so.6 => /usr/local/lib/libgdbm.so.6 (0x801141000)

libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x80134f000)

libapr-1.so.0 => /usr/local/lib/libapr-1.so.0 (0x801579000)

libcrypt.so.5 => /lib/libcrypt.so.5 (0x8017b2000)

libthr.so.3 => /lib/libthr.so.3 (0x8019d1000)

libc.so.7 => /lib/libc.so.7 (0x801bf9000)

libintl.so.8 => /usr/local/lib/libintl.so.8 (0x801fb1000)

root@waridi:/usr/local/etc/apache24/Includes #

I note the stacktrace you showed is against /usr/local/lib/libcrypto.so.9 and not the /lib/libcrypt.so.5 that your Apache and perl are apparently linked against. Check for the version that Net::SSLeay is bound against, using ldd on the auto/Net/SSLeay/SSLeay.so file that should be somewhere.

However, my suggestion to you is to switch to mod_fcgi or mod_fastcgi – I suspect you will find them easier to use than finding and resolving this incompatibility. The tight coupling of Apache to Perl doesn’t yield any performance gains, only headaches – we’re not using mod_perl for any of its advanced hooks into Apache, for authentication or the like, merely for the content handling.

I don’t believe there’s anything further RT-specific to discuss here. Good luck tracking down the segfault.

For reference, this is similar to the bug fixed in Defer loading of Net:SSLeay, to prevent segfaults under mod_perl · bestpractical/rt-authen-externalauth@8c71de0 · GitHub

Hi Alex,

Seeing as these were done in 2014, is it that I have some problem in my RT_SiteConfig.pm?

I don’t see why I should be affected when running 4.4.3 unless my RT_SiteConfig is the issue, or these mods were never

integrated into the main source.

What am I missing?