Spawn-fcgi and rt-server.fcgi fail

Hi guys!

I’ve searched the googles and found a similar issue but the result
didn’t work. When I try to run spawn-fcgi it fails with exit code 255.
If I try to run rt-server.fcgi with the --port option, it says that the
port is in use no matter the port I choose. The similar issue I found
on the mailing list turned out to be a permissions problem but all my
permissions appear to be correct on /opt/rt4/var/mason_data. The
spawn-fcgi command I used was “spawn-fcgi -u nginx -g nginx -a 127.0.0.2
-p 9090 – /opt/rt4/sbin/rt-server.fcgi”. If I run rt-server it works
just fine. The log file set to debug gives little useful information,
specifically it only complains about the AddressRegex that I am not
ready to set yet. Any ideas?

Thank you very much.

Alex

The usual questions - what os, version of RT? Suggestions that worked for me this week - see if SELinux is blocking, make sure owner on the mason-data folder and contents is set to your web server owner’s account. Test with a browser on the localhost, check your firewall iptables.

Mike-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Alex Hanselka
Sent: Wednesday, October 17, 2012 2:57 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] spawn-fcgi and rt-server.fcgi fail

Hi guys!

I’ve searched the googles and found a similar issue but the result didn’t work. When I try to run spawn-fcgi it fails with exit code 255.
If I try to run rt-server.fcgi with the --port option, it says that the port is in use no matter the port I choose. The similar issue I found on the mailing list turned out to be a permissions problem but all my permissions appear to be correct on /opt/rt4/var/mason_data. The spawn-fcgi command I used was “spawn-fcgi -u nginx -g nginx -a 127.0.0.2 -p 9090 – /opt/rt4/sbin/rt-server.fcgi”. If I run rt-server it works just fine. The log file set to debug gives little useful information, specifically it only complains about the AddressRegex that I am not ready to set yet. Any ideas?

Thank you very much.

Alex

Final RT training for 2012 in Atlanta, GA - October 23 & 24
Training — Best Practical Solutions

We’re hiring! http://bestpractical.com/jobs

Hi,

Ah I should have included this in the first place! I am using
Scientific Linux 6.3 (RHEL repack thing) and the latest RT, 4.0.7.
SELinux is disabled in this case. The owner and group of mason-data is
set to my webserver user. rt-server runs fine but rt-server.fcgi
refuses to start regardless of port. Is there anywhere other than
mason-data where I should check permissions? Also, I should add that my
webserver user is in the rt group as suggested in the security doc.

AlexOn 2012-10-17 15:10, Mike James wrote:

The usual questions - what os, version of RT? Suggestions that
worked for me this week - see if SELinux is blocking, make sure owner
on the mason-data folder and contents is set to your web server
owner’s account. Test with a browser on the localhost, check your
firewall iptables.

Mike

-----Original Message-----
From: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Alex
Hanselka
Sent: Wednesday, October 17, 2012 2:57 PM
To: rt-users@lists.bestpractical.com
Subject: [rt-users] spawn-fcgi and rt-server.fcgi fail

Hi guys!

I’ve searched the googles and found a similar issue but the result
didn’t work. When I try to run spawn-fcgi it fails with exit code
255.
If I try to run rt-server.fcgi with the --port option, it says that
the port is in use no matter the port I choose. The similar issue I
found on the mailing list turned out to be a permissions problem but
all my permissions appear to be correct on /opt/rt4/var/mason_data.
The spawn-fcgi command I used was “spawn-fcgi -u nginx -g nginx -a
127.0.0.2 -p 9090 – /opt/rt4/sbin/rt-server.fcgi”. If I run
rt-server it works just fine. The log file set to debug gives little
useful information, specifically it only complains about the
AddressRegex that I am not ready to set yet. Any ideas?

Thank you very much.

Alex


Final RT training for 2012 in Atlanta, GA - October 23 & 24
Training — Best Practical Solutions

We’re hiring! http://bestpractical.com/jobs

Ah I should have included this in the first place! I am using
Scientific Linux 6.3 (RHEL repack thing) and the latest RT, 4.0.7.
SELinux is disabled in this case. The owner and group of mason-data is
set to my webserver user. rt-server runs fine but rt-server.fcgi
refuses to start regardless of port. Is there anywhere other than
mason-data where I should check permissions? Also, I should add that my
webserver user is in the rt group as suggested in the security doc.

You might be able to get some enlightenment through strace’ing the
process, but your failure mode doesn’t ring any bells to me. What was
your configure line, out of curiosity?

  • Alex

You might be able to get some enlightenment through strace’ing the
process, but your failure mode doesn’t ring any bells to me. What
was
your configure line, out of curiosity?

  • Alex

Thank you for your response!

Upon strace-ing the file, it seems to have a lot of issues with “(No
such file or directory)” errors. I assume this is because it searches
for the files and the first places it tries are wrong? I also see a
bunch of “Inappropriate ioctl for device.” I can provide a pastebin of
the strace if necessary.

As for my configure line, it was very simple: “./configure
–with-web-user=nginx --with-web-group=nginx --with-db-type=Pg”… Did I
miss something?

Alex

Upon strace-ing the file, it seems to have a lot of issues with "(No

such file or directory)" errors. I assume this is because it searches
for the files and the first places it tries are wrong?

Correct, those are expected as perl searches through @INC looking for
library files.

I also see a bunch of “Inappropriate ioctl for device.” I can provide
a pastebin of the strace if necessary.

The ioctl error is somewhat more surprising. Seeing a pastebin of the
strace might be useful.

As for my configure line, it was very simple: “./configure
–with-web-user=nginx --with-web-group=nginx --with-db-type=Pg”… Did I
miss something?

Nope. I just wanted to see the user and group you’d passed to
configure.

Re-reading your original mail, I’d expect the spawn-fcgi incant to work,
but the failure case of rt-server.fcgi not working with --port is a
known bug, and one which is resolved in the 4.2/refactor-rt-server
branch. I’ve included below a backport of the relevant commit; I’d be
curious to hear if it solves your problem or gives a more coherent error
message.

  • Alex

------------------------------------ 8< ------------------------------->From 0745a7c578ffb50ff17124334934fefdb0c61a2d Mon Sep 17 00:00:00 2001
From: Alex Vandiver alexmv@bestpractical.com
Date: Mon, 20 Jun 2011 21:58:30 -0400
Subject: [PATCH] Refactor rt-server.in into RT::PlackRunner

This properly detects --port, --listen, and --socket options, and falls
back to $WebPort only when none are specified. It also provides better
handling for fastcgi, detecting when STDIN is a socket, and bailing with
a correct error message if it has no other options to listen on.

diff --git a/lib/RT/PlackRunner.pm b/lib/RT/PlackRunner.pm
new file mode 100644
index 0000000…8a7ea3d
— /dev/null
+++ b/lib/RT/PlackRunner.pm
@@ -0,0 +1,139 @@
+use warnings;
+use strict;
+package RT::PlackRunner;
+use base ‘Plack::Runner’;
+sub parse_options {

  • my $self = shift;
  • my @args = @_;
  • handle “rt-server 8888” for back-compat, but complain about it

  • if (@args && $args[0] =~ m/^\d+$/) {
  •    warn "Deprecated: please run $0 --port $ARGV[0] instead\n";
    
  •    unshift @args, '--port';
    
  • }
  • $self->SUPER::parse_options(@args);
  • $self->{app} ||= $self->app;
  • $self->{server} ||= $self->loader->guess;
  • my %args = @{$self->{options}};
  • if ($self->{server} eq “FCGI”) {
  •    # We deal with the possible failure modes of this in ->run
    
  • } elsif ($args{port}) {
  •    $self->{explicit_port} = 1;
    
  •    my $old_app = $self->{app};
    
  •    $self->{app} = sub {
    
  •        my $env = shift;
    
  •        $env->{'rt.explicit_port'} = $args{port};
    
  •        $old_app->($env, @_);
    
  •    };
    
  • } else {
  •    $self->set_options(port => (RT->Config->Get('WebPort') || '8080'));
    
  • }
    +}
    +# Override to not default to port 5000
    +sub mangle_host_port_socket {
  • my($self, $host, $port, $socket, @listen) = @_;
  • for my $listen (reverse @listen) {
  •    if ($listen =~ /:\d+$/) {
    
  •        ($host, $port) = split /:/, $listen, 2;
    
  •        $host = undef if $host eq '';
    
  •    } else {
    
  •        $socket ||= $listen;
    
  •    }
    
  • }
  • unless (@listen) {
  •    if ($socket) {
    
  •        @listen = ($socket);
    
  •    } elsif ($port) {
    
  •        @listen = ($host ? "$host:$port" : ":$port");
    
  •    }
    
  • }
  • return host => $host, port => $port, listen => @listen, socket => $socket;
    +}
    +sub prepare_devel {
  • my($self, $app) = @_;
  • Don’t install the Lint, StackTrace, and AccessLog middleware

  • push @{$self->{options}}, server_ready => sub {
  •    my($args) = @_;
    
  •    my $name  = $args->{server_software} || ref($args);
    
  •    my $host  = $args->{host}  || RT->Config->Get('WebDomain');
    
  •    my $proto = $args->{proto} || 'http';
    
  •    print STDERR "$name: Accepting connections at $proto://$host:$args->{port}/\n";
    
  • };
  • $app;
    +}
    +sub app {
  • require RT::Interface::Web::Handler;
  • my $app = RT::Interface::Web::Handler->PSGIApp;
  • if ($ENV{RT_TESTING}) {
  •    my $screen_logger = $RT::Logger->remove('screen');
    
  •    require Log::Dispatch::Perl;
    
  •    $RT::Logger->add(
    
  •        Log::Dispatch::Perl->new(
    
  •            name      => 'rttest',
    
  •            min_level => $screen_logger->min_level,
    
  •            action    => {
    
  •                error    => 'warn',
    
  •                critical => 'warn'
    
  •            }
    
  •        )
    
  •    );
    
  •    require Plack::Middleware::Test::StashWarnings;
    
  •    $app = Plack::Middleware::Test::StashWarnings->wrap($app);
    
  • }
  • return $app;
    +}
    +sub run {
  • my $self = shift;
  • my %args = @{$self->{options}};
  • Plack::Handler::FCGI has its own catch for this, but doesn’t

  • notice that listen is an empty list, and we can also provide a

  • better error message.

  • if ($self->{server} eq “FCGI” and not -S STDIN and not @{$args{listen}}) {
  •    print STDERR "STDIN is not a socket, and no --listen, --socket, or --port provided\n";
    
  •    exit 1;
    
  • }
  • eval { $self->SUPER::run(@_) };
  • my $err = $@;
  • exit 0 unless $err;
  • if ( $err =~ /listen/ ) {
  •    print STDERR <<EOF;
    

+WARNING: RT couldn’t start up a web server on port $args{port}.
+This is often the case if the port is already in use or you’re running @{[$0]}
+as someone other than your system’s “root” user. You may also specify a
+temporary port with: $0 --port
+EOF

  •    if ($self->{explicit_port}) {
    
  •        print STDERR
    
  •            "Please check your system configuration or choose another port\n\n";
    
  •    }
    
  •    exit 1;
    
  • } else {
  •    die
    
  •        "Something went wrong while trying to run RT's standalone web server:\n\t"
    
  •            . $err;
    
  • }
    +}
    +1;
    diff --git a/sbin/rt-server b/sbin/rt-server
    index f84f6c1…fccd895 100755
    — a/sbin/rt-server
    +++ b/sbin/rt-server
    @@ -99,7 +99,7 @@ my ($integrity, $state, $msg) = RT::Handle->CheckIntegrity;

unless ( $integrity ) {
print STDERR <<EOF;
RT couldn’t connect to the database where tickets are stored.
If this is a new installation of RT, you should visit the URL below
to configure RT and initialize your database.
@@ -141,127 +141,20 @@ if ($RT::Handle) {
undef $RT::Handle;
}

-require RT::Interface::Web::Handler;
-my $app = RT::Interface::Web::Handler->PSGIApp;
-if ($ENV{RT_TESTING}) {

  • my $screen_logger = $RT::Logger->remove(‘screen’);
  • require Log::Dispatch::Perl;
  • $RT::Logger->add(
  •    Log::Dispatch::Perl->new(
    
  •        name      => 'rttest',
    
  •        min_level => $screen_logger->min_level,
    
  •        action    => {
    
  •            error    => 'warn',
    
  •            critical => 'warn'
    
  •        }
    
  •    )
    
  • );
  • require Plack::Middleware::Test::StashWarnings;
  • $app = Plack::Middleware::Test::StashWarnings->wrap($app);
    -}
    +require RT::PlackRunner;

when used as a psgi file

if (caller) {

  • return $app;
    -}
    -# load appropriate server
    -require Plack::Runner;
    -my $is_fastcgi = $0 =~ m/fcgi$/;
    -my $r = Plack::Runner->new( $0 =~ /standalone/ ? ( server => ‘Standalone’ ) :
  •                        $is_fastcgi        ? ( server => 'FCGI' )
    
  •                                           : (),
    
  •                        env => 'deployment' );
    

-# figure out the port
-my $port;
-# handle “rt-server 8888” for back-compat, but complain about it
-if ($ARGV[0] && $ARGV[0] =~ m/^\d+$/) {

  • warn “Deprecated: please run $0 --port $ARGV[0] instead\n”;
  • unshift @ARGV, ‘–port’;
    -}
    -my @args = @ARGV;
    -use List::MoreUtils ‘last_index’;
    -my $last_index = last_index { $_ eq ‘–port’ } @args;
    -my $explicit_port;
    -if ( $last_index != -1 && $args[$last_index+1] =~ /^\d+$/ ) {
  • $explicit_port = $args[$last_index+1];
  • $port = $explicit_port;
  • inform the rest of the system what port we manually chose

  • my $old_app = $app;
  • $app = sub {
  •    my $env = shift;
    
  •    $env->{'rt.explicit_port'} = $port;
    
  •    $old_app->($env, @_);
    
  • };
    -}
    -else {
  • default to the configured WebPort and inform Plack::Runner

  • $port = RT->Config->Get(‘WebPort’) || ‘8080’;
  • push @args, ‘–port’, $port;
    -}
    -push @args, ‘–server’, ‘Standalone’ if RT->InstallMode;
    -push @args, ‘–server’, ‘Starlet’ unless $r->{server} || grep { m/–server/ } @args;
    -$r->parse_options(@args);
    -delete $r->{options} if $is_fastcgi; ### mangle_host_port_socket ruins everything
    -unless ($r->{env} eq ‘development’) {
  • push @{$r->{options}}, server_ready => sub {
  •    my($args) = @_;
    
  •    my $name  = $args->{server_software} || ref($args); # $args is $server
    
  •    my $host  = $args->{host} || 0;
    
  •    my $proto = $args->{proto} || 'http';
    
  •    print STDERR "$name: Accepting connections at $proto://$host:$args->{port}/\n";
    
  • };
    -}
    -eval { $r->run($app) };
    -if (my $err = $@) {
  • handle_startup_error($err);
  • return RT::PlackRunner->app;
    }

-exit 0;
-sub handle_startup_error {

  • my $err = shift;
  • if ( $err =~ /listen/ ) {
  •    handle_bind_error();
    
  • } else {
  •    die
    
  •        "Something went wrong while trying to run RT's standalone web server:\n\t"
    
  •        . $err;
    
  • }
    -}

+my $r = RT::PlackRunner->new( RT->InstallMode ? ( server => ‘Standalone’ ) :

  •                          $0 =~ /standalone/ ? ( server => 'Standalone' ) :
    
  •                          $0 =~ /fcgi$/      ? ( server => 'FCGI',    env => "deployment" )
    
  •                                             : ( server => 'Starlet', env => "deployment" ) );
    

+$r->parse_options(@ARGV);
+$r->run;

-sub handle_bind_error {

  • print STDERR <<EOF;
    -WARNING: RT couldn’t start up a web server on port @{[$port]}.
    -This is often the case if the port is already in use or you’re running @{[$0]}
    -as someone other than your system’s “root” user. You may also specify a
    -temporary port with: $0 --port
    -EOF
  • if ($explicit_port) {
  •    print STDERR
    
  •        "Please check your system configuration or choose another port\n\n";
    
  • }
    -}

END

1.7.11.3

I also see a bunch of “Inappropriate ioctl for device.” I can
provide
a pastebin of the strace if necessary.

The ioctl error is somewhat more surprising. Seeing a pastebin of
the
strace might be useful.

You can find that at Sign in · GitLab

Re-reading your original mail, I’d expect the spawn-fcgi incant to
work,
but the failure case of rt-server.fcgi not working with --port is a
known bug, and one which is resolved in the 4.2/refactor-rt-server
branch. I’ve included below a backport of the relevant commit; I’d
be
curious to hear if it solves your problem or gives a more coherent
error
message.

  • Alex

Didn’t seem to do anything. When I do the spawn-fcgi incant, it now
says “spawn-fcgi: child spawned successfully: PID: 27689” but nothing
stays running. 27689 doesn’t exist.

You can find that at Sign in · GitLab

Was this from before or after the patch? I don’t see any of the
expected calls to bind or listen there, which is the problem the patch
was meant to address. Can you strace the spawn-fcgi version?

Didn’t seem to do anything. When I do the spawn-fcgi incant, it now
says “spawn-fcgi: child spawned successfully: PID: 27689” but nothing
stays running. 27689 doesn’t exist.

How does /opt/rt4/sbin/rt-server.fcgi --port 9090 fare with the patch?

  • Alex

You can find that at Sign in · GitLab

Was this from before or after the patch? I don’t see any of the
expected calls to bind or listen there, which is the problem the
patch
was meant to address. Can you strace the spawn-fcgi version?

Sorry, this was BEFORE the patch. After the patch is
Sign in · GitLab and Sign in · GitLab

to verify it worked, patch output:

patch -p1 < ./patch.patch

patching file lib/RT/PlackRunner.pm
patching file sbin/rt-server
Hunk #2 succeeded at 141 with fuzz 2.

Didn’t seem to do anything. When I do the spawn-fcgi incant, it now
says “spawn-fcgi: child spawned successfully: PID: 27689” but
nothing
stays running. 27689 doesn’t exist.

How does /opt/rt4/sbin/rt-server.fcgi --port 9090 fare with the
patch?

  • Alex

Same error after the patch.

Alex

You can find that at Sign in · GitLab

Was this from before or after the patch? I don’t see any of the
expected calls to bind or listen there, which is the problem the
patch
was meant to address. Can you strace the spawn-fcgi version?

Sorry, this was BEFORE the patch. After the patch is
Sign in · GitLab and
Sign in · GitLab

to verify it worked, patch output:

patch -p1 < ./patch.patch

patching file lib/RT/PlackRunner.pm
patching file sbin/rt-server
Hunk #2 succeeded at 141 with fuzz 2.

Didn’t seem to do anything. When I do the spawn-fcgi incant, it
now
says “spawn-fcgi: child spawned successfully: PID: 27689” but
nothing
stays running. 27689 doesn’t exist.

How does /opt/rt4/sbin/rt-server.fcgi --port 9090 fare with the
patch?

  • Alex

Same error after the patch.

I lied! I misapplied that patch! I feel stupid and now it is recorded
for all time in the archives. In any case, the rt-server.fcgi works now
by itself, but spawn-fcgi still fails to spawn a persistent process.

I lied! I misapplied that patch! I feel stupid and now it is recorded
for all time in the archives. In any case, the rt-server.fcgi works now
by itself, but spawn-fcgi still fails to spawn a persistent process.

Glad to hear that half-worked. Can you re-run the spawn-fcgi incant
with strace -ff?

  • Alex

I lied! I misapplied that patch! I feel stupid and now it is
recorded
for all time in the archives. In any case, the rt-server.fcgi works
now
by itself, but spawn-fcgi still fails to spawn a persistent process.

Glad to hear that half-worked. Can you re-run the spawn-fcgi incant
with strace -ff?

  • Alex

I can! Sign in · GitLab and
Sign in · GitLab.

I can! Sign in · GitLab and
Sign in · GitLab.

spawn-fcgi is redirecting stdout and stderr to /dev/null, which is
hiding the helpful error message we’re printing
about /opt/rt4/var/log/rt.log not being writable. Check the permissions
on that file and directory.

  • Alex

I can! Sign in · GitLab and
Sign in · GitLab.

spawn-fcgi is redirecting stdout and stderr to /dev/null, which is
hiding the helpful error message we’re printing
about /opt/rt4/var/log/rt.log not being writable. Check the
permissions
on that file and directory.

  • Alex

well, that was it! What a long process. Thank you very very much!

Alex

I also see a bunch of “Inappropriate ioctl for device.” I can
provide
a pastebin of the strace if necessary.

The ioctl error is somewhat more surprising. Seeing a pastebin of
the
strace might be useful.

You can find that at Sign in · GitLab

Re-reading your original mail, I’d expect the spawn-fcgi incant to
work,
but the failure case of rt-server.fcgi not working with --port is a
known bug, and one which is resolved in the 4.2/refactor-rt-server
branch. I’ve included below a backport of the relevant commit;
I’d be
curious to hear if it solves your problem or gives a more coherent
error
message.

  • Alex

Didn’t seem to do anything. When I do the spawn-fcgi incant, it now
says “spawn-fcgi: child spawned successfully: PID: 27689” but
nothing stays running. 27689 doesn’t exist.

Hi Alex,

Are you certain you are not running afoul of selinux? It certainly
caused “mysterious” behavior.

Cheers,
Ken