RT loses MySQL connection

I just installed a new instance of RT (4.2.1). I’ve been using RT for
quite a long time now, through a lot of different versions, but this is a
new issue for me.

Everything starts up fine, but after about 10 minutes or so, it loses
connection to MySQL. It can only be recovered by resarting the web server.
It’s not limited to any particular functionality - I can be doing anything
and the next click will serve up the issue.

It’s always the same error in the logs, no matter what I was doing in the
web interface when it occurred:

[6876] [Thu Nov 21 23:12:06 2013] [warning]: DBI
connect(‘dbname=rt4;host=localhost;port=3306’,‘rt_user’,…) failed: Can’t
connect to local MySQL server through socket ‘’ (2) at
/usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 105.
(/usr/local/share/perl5/Carp.pm:168)

The socket is at /var/lib/mysql/mysql.sock.

Below is a really long stack trace from the web interface that may or may
not be useful. I’ve tried setting MYSQL_UNIX_PORT in RT_SiteConfig.pm,
moving it to /tmp, etc… but it just seems to transition from working fine
to broken for no apparent reason.

Thanks!
Jay

Error trace

Connect Failed Can’t connect to local MySQL server through socket ‘’ (2)
at /var/opt/rt-4.2.1/sbin/…/lib/RT.pm line 210.
at /usr/local/share/perl5/Carp.pm line 166

  1. in Carp::croak at /usr/local/share/perl5/Carp.pm line 166
 163:     shortmess_heavy(@_);
 164: }
 165: *  166: sub croak   { die shortmess @_ }
  • 167: sub confess { die longmess @_ }
    168: sub carp { warn shortmess @_ }
    169: sub cluck { warn longmess @_ }

Show function arguments
2. in DBIx::SearchBuilder::Handle::Connect at
/usr/local/share/perl5/DBIx/SearchBuilder/Handle.pm line 105

 102:     # Only connect if we're not connected to this source already
 103:     return undef if $self->dbh && $self->dbh->ping &&

$new_dsn eq $old_dsn;
104: * 105: my $handle = DBI->connect(

  • 106: $new_dsn, $args{‘User’}, $args{‘Password’}
    107: ) or croak “Connect Failed $DBI::errstr\n”;
    108:

Show function arguments
3. in RT::Handle::Connect at /var/opt/rt-4.2.1/sbin/…/lib/RT/Handle.pm
line 124

 121:     }
 122:
 123:     $self->SUPER::Connect(*  124:         User =>

RT->Config->Get(‘DatabaseUser’),

  • 125: Password => RT->Config->Get(‘DatabasePassword’),
    126: DisconnectHandleOnDestroy => 1,
    127: %args,

Show function arguments
4. in RT::ConnectToDatabase at /var/opt/rt-4.2.1/sbin/…/lib/RT.pm line
210

 207: sub ConnectToDatabase {
 208:     require RT::Handle;
 209:     $Handle = RT::Handle->new unless $Handle;*  210:

$Handle->Connect;

  • 211: return $Handle;
    212: }
    213:
  1. in RT::Interface::Web::Handler::ANON at
    /var/opt/rt-4.2.1/sbin/…/lib/RT/Interface/Web/Handler.pm line 278
 275:             return $self->_psgi_response_cb( $res->finalize ) if $res;
 276:         }
 277: *  278:         RT::ConnectToDatabase() unless RT->InstallMode;
  • 279:
    280: my $req = Plack::Request->new($env);
    281:

Show function arguments
6. in Plack::Middleware::Static::call at
/usr/local/share/perl5/Plack/Middleware/Static.pm line 18

  15:         return $res;
  16:     }
  17: *   18:     return $self->app->($env);
  • 19: }
    20:
    21: sub _handle_static {

Show function arguments
7. in Plack::Component::ANON at
/usr/local/share/perl5/Plack/Component.pm line 50

  47: sub to_app {
  48:     my $self = shift;
  49:     $self->prepare_app;*   50:     return sub { $self->call(@_) };
  • 51: }
    52:
    53:

Show function arguments
8. in Plack::Middleware::Static::call at
/usr/local/share/perl5/Plack/Middleware/Static.pm line 18

  15:         return $res;
  16:     }
  17: *   18:     return $self->app->($env);
  • 19: }
    20:
    21: sub _handle_static {

Show function arguments
9. in Plack::Component::ANON at
/usr/local/share/perl5/Plack/Component.pm line 50

  47: sub to_app {
  48:     my $self = shift;
  49:     $self->prepare_app;*   50:     return sub { $self->call(@_) };
  • 51: }
    52:
    53:

Show function arguments
10. in Plack::Middleware::Lint::call at
/usr/local/share/perl5/Plack/Middleware/Lint.pm line 24

  21:     my $env = shift;
  22:
  23:     $self->validate_env($env);*   24:     my $res =

$self->app->($env);

  • 25: return $self->validate_res($res);
    26: }
    27:

Show function arguments
11. in Plack::Middleware::StackTrace::ANON at
/usr/local/share/perl5/Plack/Component.pm line 50

  47: sub to_app {
  48:     my $self = shift;
  49:     $self->prepare_app;*   50:     return sub { $self->call(@_) };
  • 51: }
    52:
    53:
  1. in (eval) at /usr/local/share/perl5/Try/Tiny.pm line 78
  75:     $@ = $prev_error;
  76:
  77:     # evaluate the try block in the correct context*   78:

if ( $wantarray ) {

  • 79: @ret = $try->();
    80: } elsif ( defined $wantarray ) {
    81: $ret[0] = $try->();
  1. in Plack::Middleware::StackTrace::call at
    /usr/local/share/perl5/Try/Tiny.pm line 74
  71:
  72:   # failed will be true if the eval dies, because 1 will not

be returned
73: # from the eval body* 74: my $failed = not eval {

  • 75: $@ = $prev_error;
    76:
    77: # evaluate the try block in the correct context

Show function arguments
14. in Plack::Component::ANON at
/usr/local/share/perl5/Plack/Component.pm line 50

  47: sub to_app {
  48:     my $self = shift;
  49:     $self->prepare_app;*   50:     return sub { $self->call(@_) };
  • 51: }
    52:
    53:

Show function arguments
15. in Plack::Middleware::AccessLog::call at
/usr/local/share/perl5/Plack/Middleware/AccessLog.pm line 24

  21:     my $self = shift;
  22:     my($env) = @_;
  23: *   24:     my $res = $self->app->($env);
  • 25:
    26: if ( ref($res) && ref($res) eq ‘ARRAY’ ) {
    27: my $content_length = Plack::Util::content_length($res->[2]);

Show function arguments
16. in Plack::Component::ANON at
/usr/local/share/perl5/Plack/Component.pm line 50

  47: sub to_app {
  48:     my $self = shift;
  49:     $self->prepare_app;*   50:     return sub { $self->call(@_) };
  • 51: }
    52:
    53:

Show function arguments
17. in (eval) at /usr/local/share/perl5/Plack/Util.pm line 142

 139: sub run_app($$) {
 140:     my($app, $env) = @_;
 141: *  142:     return eval { $app->($env) } || do {
  • 143: my $body = “Internal Server Error”;
    144: $env->{‘psgi.errors’}->print($@);
    145: [ 500, [ ‘Content-Type’ => ‘text/plain’,
    ‘Content-Length’ => length($body) ], [ $body ] ];
  1. in Plack::Util::run_app at /usr/local/share/perl5/Plack/Util.pm line
    142
 139: sub run_app($$) {
 140:     my($app, $env) = @_;
 141: *  142:     return eval { $app->($env) } || do {
  • 143: my $body = “Internal Server Error”;
    144: $env->{‘psgi.errors’}->print($@);
    145: [ 500, [ ‘Content-Type’ => ‘text/plain’,
    ‘Content-Length’ => length($body) ], [ $body ] ];

Show function arguments
19. in Plack::Handler::FCGI::run at
/usr/local/share/perl5/Plack/Handler/FCGI.pm line 134

 131:             $env->{HTTP_AUTHORIZATION} = $HTTP_AUTHORIZATION;
 132:         }
 133: *  134:         my $res = Plack::Util::run_app $app, $env;
  • 135:
    136: if (ref $res eq ‘ARRAY’) {
    137: $self->_handle_response($res);

Show function arguments
20. in Plack::Loader::run at /usr/local/share/perl5/Plack/Loader.pm line
84

  81:
  82: sub run {
  83:     my($self, $server, $builder) = @_;*   84:

$server->run($self->{app});

  • 85: }
    86:
    87: 1;

Show function arguments
21. in Plack::runner::run at /usr/local/share/perl5/Plack/Runner.pm line
277

 274:     $loader->preload_app($app);
 275:
 276:     my $server = $self->load_server($loader);*  277:

$loader->run($server);

  • 278: }
    279:
    280: 1;

Show function arguments
22. at /usr/local/bin/plackup line 10

   7:
   8: my $runner = Plack::Runner->new;
   9: $runner->parse_options(@ARGV);*   10: $runner->run;
  • 11:
    12: END
    13:

I just installed a new instance of RT (4.2.1). I’ve been using RT for
quite a long time now, through a lot of different versions, but this
is a new issue for me.

Is there anything of note in the mysql logs?

  • Alex

No, no entries beyond the startup messages. I thought maybe there would be
some connection errors (a flush-hosts situation or something), but nothing.On Thu, Nov 21, 2013 at 6:02 PM, Alex Vandiver alexmv@bestpractical.comwrote:

On Thu, 2013-11-21 at 15:20 -0800, Jay Christopherson wrote:

I just installed a new instance of RT (4.2.1). I’ve been using RT for
quite a long time now, through a lot of different versions, but this
is a new issue for me.

Is there anything of note in the mysql logs?

  • Alex

No, no entries beyond the startup messages. I thought maybe there
would be some connection errors (a flush-hosts situation or
something), but nothing.

That’s odd. Can you show your my.cnf, webserver configuration, and
RT_SiteConfig.pm ? Is this under SELinux, or anything else that might
muck about with sockets?

  • Alex

No, no entries beyond the startup messages. I thought maybe there would be
some connection errors (a flush-hosts situation or something), but nothing.

I just installed a new instance of RT (4.2.1). I’ve been using RT for
quite a long time now, through a lot of different versions, but this
is a new issue for me.

Is there anything of note in the mysql logs?

  • Alex

Hi Jay,

It might be a long shot, but do you have a connection timeout set for your
MySQL DB? Try disabling that. I was bit by that once and was astounded to
find out that the DB just dropped a valid connection like that. It seems
more useful in a broken web app type of way to keep from leaking connections
but normal apps do not expect to lose a good connection. :slight_smile:

Regards,
Ken

Ken, thanks for the suggestion/reminder - I cribbed a my.cnf file from
another database I setup. I had forgotten that I set a short wait_timeout
(300), for just the reason you suggested. I reset it to be a little more
sane about an hour ago and so far, things look ok.On Fri, Nov 22, 2013 at 5:37 AM, ktm@rice.edu ktm@rice.edu wrote:

On Thu, Nov 21, 2013 at 06:05:23PM -0800, Jay Christopherson wrote:

No, no entries beyond the startup messages. I thought maybe there would
be
some connection errors (a flush-hosts situation or something), but
nothing.

On Thu, Nov 21, 2013 at 6:02 PM, Alex Vandiver <alexmv@bestpractical.com wrote:

On Thu, 2013-11-21 at 15:20 -0800, Jay Christopherson wrote:

I just installed a new instance of RT (4.2.1). I’ve been using RT
for
quite a long time now, through a lot of different versions, but this
is a new issue for me.

Is there anything of note in the mysql logs?

  • Alex

Hi Jay,

It might be a long shot, but do you have a connection timeout set for your
MySQL DB? Try disabling that. I was bit by that once and was astounded to
find out that the DB just dropped a valid connection like that. It seems
more useful in a broken web app type of way to keep from leaking
connections
but normal apps do not expect to lose a good connection. :slight_smile:

Regards,
Ken

Interestingly, adjusting the wait_timeout to the default (8 hours) worked
great - for 8 hours. After that 8 hours, RT once again lost all
connectivity to the database. Running a “show processlist” on MySQL showed
that RT had no connections to the DB, even though RT was still running. A
restart fixed it - for now.

I am not sure how RT does connection handling, but it seems like it should
attempt to reconnect?On Fri, Nov 22, 2013 at 8:32 AM, Jay Christopherson jc.listmail@gmail.comwrote:

Ken, thanks for the suggestion/reminder - I cribbed a my.cnf file from
another database I setup. I had forgotten that I set a short wait_timeout
(300), for just the reason you suggested. I reset it to be a little more
sane about an hour ago and so far, things look ok.

On Fri, Nov 22, 2013 at 5:37 AM, ktm@rice.edu ktm@rice.edu wrote:

On Thu, Nov 21, 2013 at 06:05:23PM -0800, Jay Christopherson wrote:

No, no entries beyond the startup messages. I thought maybe there
would be
some connection errors (a flush-hosts situation or something), but
nothing.

On Thu, Nov 21, 2013 at 6:02 PM, Alex Vandiver < alexmv@bestpractical.com>wrote:

On Thu, 2013-11-21 at 15:20 -0800, Jay Christopherson wrote:

I just installed a new instance of RT (4.2.1). I’ve been using RT
for
quite a long time now, through a lot of different versions, but this
is a new issue for me.

Is there anything of note in the mysql logs?

  • Alex

Hi Jay,

It might be a long shot, but do you have a connection timeout set for your
MySQL DB? Try disabling that. I was bit by that once and was astounded to
find out that the DB just dropped a valid connection like that. It seems
more useful in a broken web app type of way to keep from leaking
connections
but normal apps do not expect to lose a good connection. :slight_smile:

Regards,
Ken

Interestingly, adjusting the wait_timeout to the default (8 hours) worked
great - for 8 hours. After that 8 hours, RT once again lost all
connectivity to the database. Running a “show processlist” on MySQL showed
that RT had no connections to the DB, even though RT was still running. A
restart fixed it - for now.

I am not sure how RT does connection handling, but it seems like it should
attempt to reconnect?

Hi Jay,

Again, a properly managed system does not require and certainly does not expect
to have a connection ripped out from under it. Turn off your timeout and you
will be fine. RT will connect just fine, as you have seen. It should not need
to “reconnect” to a properly configured DB once it has a valid connection.

Regards,
Ken

Hey Ken-

Yep, I get that. I don’t believe you can disable the wait_timeout entirely

  • by default, it’s set to 8 hours, which is what I currently have. RT
    works great - for 8 hours. Just to be sane, I replaced my.cnf with the
    default my.cnf that gets built with a vanilla install of MySQL. Same issue.

Clearly though, I have something going on that is not-normal because if
this were a real bug, lots of people would be reporting it and it’s never
been an issue for me before. I’m going to start over with a clean build
and try again.

  • JayOn Mon, Nov 25, 2013 at 5:53 AM, ktm@rice.edu ktm@rice.edu wrote:

On Sat, Nov 23, 2013 at 08:35:06AM -0800, Jay Christopherson wrote:

Interestingly, adjusting the wait_timeout to the default (8 hours) worked
great - for 8 hours. After that 8 hours, RT once again lost all
connectivity to the database. Running a “show processlist” on MySQL
showed
that RT had no connections to the DB, even though RT was still running.
A
restart fixed it - for now.

I am not sure how RT does connection handling, but it seems like it
should
attempt to reconnect?

Hi Jay,

Again, a properly managed system does not require and certainly does not
expect
to have a connection ripped out from under it. Turn off your timeout and
you
will be fine. RT will connect just fine, as you have seen. It should not
need
to “reconnect” to a properly configured DB once it has a valid connection.

Regards,
Ken