Webmux connecting to the database; causing mod_perl startup to fail

Hello,

I’m investigating Debian bug report
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=595054
concerning the complete failure of startup of Apache when webmux.pl
is being PerlRequire’d and the database server is unavailable.

I’ve been able to reproduce this myself with both MySQL and PostgreSQL.

In this particular instance, the reason the database server is not
running is because the user is using parallel bootup script, but the
same would apply if, for example, the database server was on a separate
system from the web server. However it’s worth noting that parallel
booting will be the default for the next version of Debian so it’s going
to become a common problem.

From a look at the code in webmux.pl (3.8.8) I don’t see a simple
solution to this problem; the database connection is established by
RT::Init which is needed before launching into the Mason handler.

The parts of the Init call which use the database appear to be:

  • InitSystemObjects()
  • InitPlugins(); (possibly plugins talk to the database in init…)

The explicit use of the database handle is to call CheckCompatibility.
This looks like a “nice to have” function rather than an essential -
it is after all also called by rt-setup-database.

Maybe it would be possible to split the Init into two stages and only
do the database connection in the second stage, which could be
run by the Mason handler the first time a request comes in.

I couldn’t find an open bug about this, which is a bit surprising
because it seems relatively likely to happen in all sort of scenarios.

Do people agree that this is a problem that should be fixed? Any obvious
(minimally-invasive, given that this is something that should be fixed
in a frozen Debian release) methods?

Cheers,
Dominic.

Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford

signature.asc (189 Bytes)

Maybe it would be possible to split the Init into two stages and only
do the database connection in the second stage, which could be
run by the Mason handler the first time a request comes in.

That’s going to become impossible as we continue to move RT’s configuration into the database. I consider having your database around before application startup to be a fairly serious requirement. Parallel startup scripts don’t have a way to declare dependencies? How do you deal with, say, starting up apps that are mounted on NFS filesystems?

That’s going to become impossible as we continue to move RT’s configuration into the database. I consider having your database around before application startup to be a fairly serious requirement. Parallel startup scripts don’t have a way to declare dependencies? How do you deal with, say, starting up apps that are mounted on NFS filesystems?

They do, but the problem is that I’m not in a position to mandate
the dependencies declared by Apache, and it’s not just RT that fails
to start up but Apache in total, which may be used in other ways on the
same system.

There are network filesystem dependencies already included in the
parallel startup stuff, I believe. But as pointed out already, there
are some requirements you can’t satisfy as easily: lack of a database
server on a remote host being the obvious one, so even if there was
sufficient information in the dependency information of the init
scripts, that doesn’t solve the underlying problem.

I should have pointed out the short discussion on debian-devel on this
matter, which was fairly strongly of the opinion that RT shouldn’t
break Apache in this way, and was also fairly negative about trying
to persuade the Apache maintainers to add some dependencies so Apache
will always start before the database servers:

http://lists.debian.org/debian-devel/2010/08/msg00786.html
and continuing:
http://lists.debian.org/debian-devel/2010/09/msg00011.html

If you think it’s impossible to fix this in RT, then probably the
only thing we can do is deprecate support for mod_perl, and suggest
that people use FastCGI instead; because FastCGI doesn’t turn fatal
errors in the application into a fatal error to start the web server
(certainly the Apache modules both seem to have a retry-with-backoff
approach to the problem). Or maybe there’s a mod_perl trick to do a
similar thing which I’m not aware of.

Cheers,
Dominic.

Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford

signature.asc (189 Bytes)

I suspect there’s a way to make RT under modperl fail more gracefully. I’d support such a change.“Dominic Hargreaves” dominic.hargreaves@oucs.ox.ac.uk wrote:

On Sat, Sep 11, 2010 at 11:38:32AM -0400, Jesse Vincent wrote:

That’s going to become impossible as we continue to move RT’s configuration into the database. I consider having your database around before application startup to be a fairly serious requirement. Parallel startup scripts don’t have a way to declare dependencies? How do you deal with, say, starting up apps that are mounted on NFS filesystems?

They do, but the problem is that I’m not in a position to mandate
the dependencies declared by Apache, and it’s not just RT that fails
to start up but Apache in total, which may be used in other ways on the
same system.

There are network filesystem dependencies already included in the
parallel startup stuff, I believe. But as pointed out already, there
are some requirements you can’t satisfy as easily: lack of a database
server on a remote host being the obvious one, so even if there was
sufficient information in the dependency information of the init
scripts, that doesn’t solve the underlying problem.

I should have pointed out the short discussion on debian-devel on this
matter, which was fairly strongly of the opinion that RT shouldn’t
break Apache in this way, and was also fairly negative about trying
to persuade the Apache maintainers to add some dependencies so Apache
will always start before the database servers:

Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv
and continuing:
Re: [request-tracker-maintainers] Bug#595054: request-tracker3.8: Race condition between RT3.8+apache2 and MySQL when booting by insserv

If you think it’s impossible to fix this in RT, then probably the
only thing we can do is deprecate support for mod_perl, and suggest
that people use FastCGI instead; because FastCGI doesn’t turn fatal
errors in the application into a fatal error to start the web server
(certainly the Apache modules both seem to have a retry-with-backoff
approach to the problem). Or maybe there’s a mod_perl trick to do a
similar thing which I’m not aware of.

Cheers,
Dominic.


Dominic Hargreaves, Systems Development and Support Team
Computing Services, University of Oxford