Eeek! RT is unable to connect to database

RT Users
1

I applied some of the recommendations from mysqltuner.pl (since backed out),
including the users able to connect (to the test database) without
authentication
(not reversed), and now RT cannot connect to the database:

[Tue Feb 10 13:29:12 2009] [error] [client 129.44.190.12] Connect
Failed Can’t connect to local MySQL server through socket
‘/var/lib/mysql/mysql.sock’ (13)
at /opt/rt3/bin/…/lib/RT.pm line 204

The socket is there, I can connect with mysql -u rt_user -p, and with
a test perl script via DBI.
I’m at a loss, and have found nothing but these obvious bits in the
list archives, any thoughts?

Cambridge Energy Alliance: Save money. Save the planet.

2

What is RT trying to do, that is should fail when this debug script works?

use EZDBI;
use Data::Dumper;

Connect ‘mysql:rt’, ‘rt_user’, ‘PASSWORD’;
my $r = Select ‘* From Users LIMIT 2’;
print Dumper $_ while( $_ = $r->({}) );

END
$VAR1 = {
‘WebEncoding’ => undef,
‘AuthSystem’ => undef,
‘PGPKey’ => undef,
‘Password’ => ‘NO-PASSWORD’,
‘MobilePhone’ => undef,
‘WorkPhone’ => undef,
‘id’ => ‘1’,
‘LastUpdated’ => ‘2008-08-02 14:59:35’,
‘PagerPhone’ => undef,
‘ExternalContactInfoId’ => undef,
‘ContactInfoSystem’ => undef,
‘Creator’ => ‘1’,
‘LastUpdatedBy’ => ‘1’,
‘HomePhone’ => undef,
‘Address1’ => undef,
‘Created’ => ‘2008-08-02 14:59:35’,
‘ExternalAuthId’ => undef,
‘Comments’ => ‘Do not delete or modify this user. It is
integral to RT’s internal database structures’,
‘NickName’ => undef,
‘Address2’ => undef,
‘Timezone’ => undef,
‘FreeformContactInfo’ => undef,
‘RealName’ => ‘The RT System itself’,
‘City’ => undef,
‘EmailAddress’ => undef,
‘EmailEncoding’ => undef,
‘State’ => undef,
‘Signature’ => undef,
‘Zip’ => undef,
‘Organization’ => undef,
‘Lang’ => undef,
‘Gecos’ => undef,
‘Country’ => undef,
‘Name’ => ‘RT_System’
};
$VAR1 = {
‘WebEncoding’ => undef,
‘AuthSystem’ => undef,
‘PGPKey’ => undef,
‘Password’ => ‘NO-PASSWORD’,
‘MobilePhone’ => undef,
‘WorkPhone’ => undef,
‘id’ => ‘6’,
‘LastUpdated’ => ‘2008-08-02 14:59:35’,
‘PagerPhone’ => undef,
‘ExternalContactInfoId’ => undef,
‘ContactInfoSystem’ => undef,
‘Creator’ => ‘1’,
‘LastUpdatedBy’ => ‘1’,
‘HomePhone’ => undef,
‘Address1’ => undef,
‘Created’ => ‘2008-08-02 14:59:35’,
‘ExternalAuthId’ => undef,
‘Comments’ => ‘Do not delete or modify this user. It is
integral to RT’s internal data structures’,
‘NickName’ => undef,
‘Address2’ => undef,
‘Timezone’ => undef,
‘FreeformContactInfo’ => undef,
‘RealName’ => ‘Nobody in particular’,
‘City’ => undef,
‘EmailAddress’ => ‘’,
‘EmailEncoding’ => undef,
‘State’ => undef,
‘Signature’ => undef,
‘Zip’ => undef,
‘Organization’ => undef,
‘Lang’ => undef,
‘Gecos’ => undef,
‘Country’ => undef,
‘Name’ => ‘Nobody’
};

Cambridge Energy Alliance: Save money. Save the planet.

3

This second debug script also works?! and yields similar output.

use lib qw(/opt/rt3/lib/ /opt/rt3/local/lib);
use RT;
use Data::Dumper;

RT::LoadConfig();
RT->Init();

$r = $RT::Handle->SimpleQuery(’ Select * From Users LIMIT 2’);
print Dumper $_ while( $_ = $r->fetchrow_hashref );

4

Setting the log level to debug provides literally no information.
I’m stumped, and really need to get this system back ASAP.
any help would be greatly appreciated.

Cambridge Energy Alliance: Save money. Save the planet.

5

Is mysql setup to listen to TCP connections via localhost and/or it’s
‘real’ IP address?

James Moseley

         Jerrad Pierce                                                 
         <jpierce@cambridg                                             
         eenergyalliance.o                                          To 
         rg>                       RT Users                            
         Sent by:                  <RT-Users@lists.bestpractical.com>  
         rt-users-bounces@                                          cc 
         lists.bestpractic                                             
         al.com                                                Subject 
                                   [rt-users] Eeek! RT is unable to    
                                   connect to database                 
         02/10/2009 12:30                                              
         PM

I applied some of the recommendations from mysqltuner.pl (since backed
out),
including the users able to connect (to the test database) without
authentication
(not reversed), and now RT cannot connect to the database:

[Tue Feb 10 13:29:12 2009] [error] [client 129.44.190.12] Connect
Failed Can’t connect to local MySQL server through socket
‘/var/lib/mysql/mysql.sock’ (13)
at /opt/rt3/bin/…/lib/RT.pm line 204

The socket is there, I can connect with mysql -u rt_user -p, and with
a test perl script via DBI.
I’m at a loss, and have found nothing but these obvious bits in the
list archives, any thoughts?

6

Is mysql setup to listen to TCP connections via localhost and/or it’s
‘real’ IP address?
No. This configuration has always had skip-networking, without a problem.

Cambridge Energy Alliance: Save money. Save the planet.

7

Just ran perl sbin/rt-setup-database --action acl --prompt-for-dba-password,
and no change to the db or situation…

8

Sorry, didn’t realize RT had the capability to use local Unix sockets…
Does the owner of the httpd process have access to the mysql socket?
Assuming nothing changed within the RT directory, it might be helpful for
you to post the mysql you used to tune the database and the syntax used to
back out of the changes.

Hopefully you have a current mysql backup of the RT database. :wink:

James Moseley

         Jerrad Pierce                                                 
         <jpierce@cambridg                                             
         eenergyalliance.o                                          To 
         rg>                       jmoseley@corp.xanadoo.com           
                                                                    cc 
         02/10/2009 01:08          RT Users                            
         PM                        <RT-Users@lists.bestpractical.com>, 
                                   rt-users-bounces@lists.bestpractica 
                                   l.com                               
                                                               Subject 
                                   Re: [rt-users] Eeek! RT is unable   
                                   to connect to databaseOn Tue, Feb 10, 2009 at 14:06,  <jmoseley@corp.xanadoo.com> wrote:

Is mysql setup to listen to TCP connections via localhost and/or it’s
‘real’ IP address?
No. This configuration has always had skip-networking, without a problem.

9

Assuming nothing changed within the RT directory, it might be helpful for
you to post the mysql you used to tune the database and the syntax used to
back out of the changes.
No SQL was used, just variables via my.cnf
The problem persists whether or not they are set.

Hopefully you have a current mysql backup of the RT database. :wink:
Relatively, as of about 3 this morning. But the database itself is there
and fine. As previously indicated. I can connect to it manually, and
programmatically. RT just seems to have decided to throw a hissy fit.

10

check the apache error_log
Yes, that is where the quoted error was from.

and enable and check the mysql full logs. the
apache logs may give you and idea of what’s going on and if you’re being
rejected by mysql you’ll see that in the full log
Not entirely sure what you mean. There’s nothing in the error log
other than start/stop.
And log= begets nothing since the connection never forms and no
queries are issued?

11

Report the results of a diff between the old version of my.cnf and the one
after your changes.

James Moseley

         Jerrad Pierce                                                 
         <jpierce@cambridg                                             
         eenergyalliance.o                                          To 
         rg>                       jmoseley@corp.xanadoo.com           
                                                                    cc 
         02/10/2009 01:17          RT Users                            
         PM                        <RT-Users@lists.bestpractical.com>, 
                                   rt-users-bounces@lists.bestpractica 
                                   l.com                               
                                                               Subject 
                                   Re: [rt-users] Eeek! RT is unable   
                                   to connect to database

Assuming nothing changed within the RT directory, it might be helpful for
you to post the mysql you used to tune the database and the syntax used
to
back out of the changes.
No SQL was used, just variables via my.cnf
The problem persists whether or not they are set.

Hopefully you have a current mysql backup of the RT database. :wink:
Relatively, as of about 3 this morning. But the database itself is there
and fine. As previously indicated. I can connect to it manually, and
programmatically. RT just seems to have decided to throw a hissy fit.

12

Report the results of a diff between the old version of my.cnf and the one
after your changes.
As previously reported, the changes have already been undone. The diff
is /dev/null

They were nothing ground-breaking:
— /etc/my.cnf 2008-10-06 17:08:18.000000000 -0400
+++ /etc/my.new 2009-02-10 14:34:40.000000000 -0500
@@ -5,19 +5,26 @@

clients (those using the mysqlclient10 compatibility package).

#old_passwords=1
skip-networking
+skip-isam
+skip-bdb

#my-large.cnf
key_buffer = 256M
max_allowed_packet = 4M
table_cache = 256
-sort_buffer_size = 1M
+sort_buffer_size = 2M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
myisam_sort_buffer_size = 64M
thread_cache = 8
query_cache_size= 16M
thread_concurrency = 4
+#mysqltuner.pl{

  • tmp_table_size = 32M
  • max_heap_table_size = 32M
    +#}

+#XXX skip-innodb

You can set …_buffer_pool_size up to 50 - 80 %

of RAM but beware of setting memory usage too high

innodb_buffer_pool_size = 256M
@@ -33,7 +40,6 @@
innodb_additional_mem_pool_size=2M
innodb_file_io_threads=4
innodb_lock_wait_timeout=50
-sort_buffer=2M
innodb_additional_mem_pool_size = 20M

Cambridge Energy Alliance: Save money. Save the planet.

13

check the apache error_log and enable and check the mysql full logs.
the apache logs may give you and idea of what’s going on and if you’re
being rejected by mysql you’ll see that in the full log

Jerrad Pierce wrote:

14

OK, and your certain my.cnf is setup so that this is the correct socket:
/var/lib/mysql/mysql.sock?

And the httpd process (assuming Apache here) has proper permissions to
mysql.sock?

James Moseley

         Jerrad Pierce                                                 
         <jpierce@cambridg                                             
         eenergyalliance.o                                          To 
         rg>                       jmoseley@corp.xanadoo.com           
                                                                    cc 
         02/10/2009 01:42          RT Users                            
         PM                        <RT-Users@lists.bestpractical.com>, 
                                   rt-users-bounces@lists.bestpractica 
                                   l.com                               
                                                               Subject 
                                   Re: [rt-users] Eeek! RT is unable   
                                   to connect to database

Report the results of a diff between the old version of my.cnf and the
one
after your changes.
As previously reported, the changes have already been undone. The diff
is /dev/null

15

/usr/libexec/mysqld --verbose --help | grep socket
socket /var/lib/mysql/mysql.sock

4 srwxrwxrwx  1 mysql mysql        0 Feb 10 14:59 mysql.sock=

Doh! WTF? Somehow SELinux context on the socket got borked.

Trying to fix it now…
Cambridge Energy Alliance: Save money. Save the planet.

16

Okay, I eventually got RT working again. Something really screwed up SELinux.
I had to add new relaxed permissions for apache to access mysql’s socket,
and then restorecon -R /opt/rt3/var so RT could access its logs.

Cambridge Energy Alliance: Save money. Save the planet.