Production systems down

1

Hello

I am desperate here my production system is down and I am not getting error messages.

Can’t login as root
System has been running flawlessly for several years I have not made changes in awhile.

I am even willing to pay someone a reasonable amount to help me trouble shoot the problem.

I get

Server error!

The server encountered an internal error and was unable to complete your request.

Error message:
Premature end of script headers: rt-server.fcgi

If you think this is a server error, please contact the webmaster.

Error 500

Mon Mar 11 11:48:11 2019
Apache/2.2.29 (Linux/SUSE)

Messages in rt.log are

username: root , service: Connect_LDAP (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:439)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: LDAP Search === Base: DC=copesan,DC=local == Filter: (&(&(ObjectCategory=User)(ObjectClass=Person))(sAMAccountName=root)) == Attrs: physicalDeliveryOfficeName,l,cn,st,mail,sAMAccountName,co,streetAddress,postalCode,telephoneNumber,sAMAccountName,physicalDeliveryOfficeName,sAMAccountName (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:469)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Password validation required for service - Executing… (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth.pm:304)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Trying external auth service: Connect_LDAP (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:153)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: LDAP Search === Filter: (&(sAMAccountName=root)(&(ObjectCategory=User)(ObjectClass=Person))) == Attrs: dn (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:186)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Found LDAP DN: (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:220)
[4246] [Mon Mar 11 16:48:11 2019] [info]: Connect_LDAP AUTH FAILED root (can’t bind: LDAP_INVALID_CREDENTIALS 49 ) (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:227)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: LDAP password validation result: 0 (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth.pm:483)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Password Validation Check Result: 0 (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth.pm:308)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Attempting to use external auth service: Connect_LDAP_WilKil (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth.pm:213)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: Calling UserExists with $username (root) and $service (Connect_LDAP_WilKil) (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth.pm:254)
[4246] [Mon Mar 11 16:48:11 2019] [debug]: UserExists params:
username: root , service: Connect_LDAP_WilKil (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:439)

Apache log
[Mon Mar 11 11:37:39 2019] [notice] Apache/2.2.29 (Linux/SUSE) mod_ssl/2.2.29 OpenSSL/1.0.1j mod_fcgid/2.3.6 configured – resuming normal operations
[4137] [Mon Mar 11 16:37:59 2019] [info]: RT::Authen::ExternalAuth::LDAP::GetAuth External Auth OK ( Connect_LDAP ): bbaker (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:301)
[4144] [Mon Mar 11 16:41:02 2019] [info]: Connect_LDAP AUTH FAILED root (can’t bind: LDAP_INVALID_CREDENTIALS 49 ) (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:227)
[4246] [Mon Mar 11 16:48:11 2019] [info]: Connect_LDAP AUTH FAILED root (can’t bind: LDAP_INVALID_CREDENTIALS 49 ) (/opt/rt4/local/plugins/RT-Authen-ExternalAuth/lib/RT/Authen/ExternalAuth/LDAP.pm:227)I

Thanks

2

Forgot to say the version 4.2.3 on Linux Suse

3 
 Premature end of script headers: rt-server.fcgi

Usually this error is seen when Linux file rights are not correct, can you confirm that your web server can access RT’s files? it may also be worth checking that SELInux is disabled.

4

You may not have made any changes to the RT server, but those LDAP errors may indicate that you, or someone else, has changed the account credentials on the LDAP server that you use to authenticate people.

5

Thanks for the response

There is no reason for the rights to have changed. I there a command I can use to verify access to the files. There are not any messages in the Apache logs to indicate file access errors.

Thanks

6

Thanks for the response

going down through the log you will see a successful ldap hand off the other messages are from trying different domain trees in my AD. This also would not explain why root can’t login.

Thanks

7

Oh by the way I am the only it no one else has the rights or ability to change rights.

Thanks for the suggestions but keep them coming.

8

OK, you say you’ve not made any changes, but what about automated updates, etc? Have updated OS packages been loaded that might have (for example) restarted the web server, upgraded the database server or turned on SELinux?

Another thought: if you know when it was last working, go back in your various logs and see what happened around then.

9

I have rebooted the server all directories and files apear to have riths to allow the webserver to access. I have rebooted the server. I can access the database and all tables. All Request tracker automated process that I have built are ruining and working – some of these process create tickets and respond to tickets and close tickets.

All automated process to retrieve emails and create tickets are working I can see new tickets being created in the database. This means that RT-mailgate is working.

All that seems to be the issue is the Web interface.

You talk about SELinux being turned on I do not know what that means.

Thanks

10

I verified that no updates have been applied sense 2017-03-28. that is according to the zypp history log.

11

Umm, well there’s been a heck of a lot of security patches issued by Suse since March 2017, so once you’ve got the RT fastCGI stuff working again, you might want to get some patching done!

SELinux is an addition to the Linux/Unix security/permissions model. I’ve not used Suse, but on my Debian box:

/usr/sbin/sestatus

will tell you whether it is disabled, enforcing, etc.

12

Hi

Your logs would seem to indicate your failing to bind with LDAP as the user root. Does root exist in your LDAP directory. You should be able to see the SE Linux status with a command like “sestatus”