RT LDAP failed after new root cert installed(solved)

We have a RT v4.4.2 running on CentOS 7.3. It was working fine until we installed a new root certificate CA. The new root ca works fine with other services. For example, we also have a Gitlab server which also use LDAP/TLS to authenticate users and that server works fine. So I know our SSL certs is working.

On this RT, the apache works fine with the new host certificates signed by the new CA. The login page load up with correct certificate. But user can’t login any more. Here is our config file and debug log, please help to troubleshoot this issue.

03_ldap.pm:
Plugin( “RT::Authen::ExternalAuth” );
Set( $WebFallbackToRTLogin, 1 );
Set( $ExternalAuthPriority, [‘My_LDAP’] );
Set( $ExternalInfoPriority, [‘My_LDAP’] );
Set( $ExternalSettings, {
My_LDAP => {
‘type’ => ‘ldap’,
‘server’ => ‘ldap.hq.mydomain.com’,
‘tls’ => {
verify => ‘require’,
cafile => ‘/etc/pki/ca-trust/source/anchors/root-ca.pem’,
},
‘base’ => ‘ou=employees,ou=people,dc=hq,dc=mydomain,dc=com’,
‘group’ => ‘cn=users,ou=group,dc=hq,dc=mydomain,dc=com’,
‘group_attr’ => ‘member’,
‘filter’ => ‘(objectClass=inetOrgPerson)’,
‘attr_match_list’ => [‘Name’],
‘attr_map’ => {
‘Name’ => ‘uid’,
‘EmailAddress’ => ‘mail’,
‘RealName’ => ‘cn’,
‘WorkPhone’ => ‘telephoneNumber’,
‘MobilePhone’ => ‘mobile’
},
},
});

Plugin( "RT::LDAPImport" );
# Connection Details
Set( $LDAPHost, 'ldap-01.hq.mydomain.com' );
Set( $LDapOptions, [ port    => 389,
                 scheme  => 'ldap', # Hopefully TLS support comes soon
                 raw     => qr/(\;binary)/,
                 version => 3,
                 verify  => 'required',
                 cafile  => '/etc/pki/ca-trust/source/anchors/root-ca.pem' 
]);
Set( $LDAPUser, 'uid=RTBind,ou=binds,dc=hq,dc=mydomain,dc=com' );
Set( $LDAPPassword, '8B6f9023mdrtggggsd' );

# Import Users
Set( $LDAPBase, 'ou=employees,ou=people,dc=hq,dc=mydomain,dc=com' );
Set( $LDAPFilter, '(objectClass=inetOrgPerson)' );
Set( $LDAPMapping, { 'Name'              => 'uid',
                 'EmailAddress'      => 'mail',
                 'RealName'          => 'cn',
                 'WorkPhone'         => 'telephoneNumber',
                 'MobilePhone'       => 'mobile'
});
Set( $LDAPCreatePrivileged, 1 );
# Import Groups
Set( $LDAPGroupBase, 'ou=group,dc=hq,dc=mydomain,dc=com' );
Set( $LDAPGroupFilter, '(objectClass=groupOfNames)' );
Set( $LDAPGroupMapping, { 'Name'              => 'cn',
                      'Member_Attr'       => 'member',
                      'Member_Attr_Value' => 'dn'
});

Debug log:
[4426] [Mon Apr 8 18:05:44 2019] [debug]: Using internal Perl HTML → text conversion (/usr/sbin/…/lib/RT/Interface/Email.pm:1475)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: The RTAddressRegexp option is not set in the config. Not setting this option results in additional SQL queries to check whether each address belongs to RT or not. It is especially important to set this option if RT receives emails on addresses that are not in the database or config. (/usr/sbin/…/lib/RT/Config.pm:577)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: Attempting to use external auth service: My_LDAP (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:288)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: SSO Failed and no user to test with. Nexting (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:316)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: Autohandler called ExternalAuth. Response: (0, No User) (/usr/share/html/Elements/DoAuth:58)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: Attempting to use external auth service: My_LDAP (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:288)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: SSO Failed and no user to test with. Nexting (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:316)
[4426] [Mon Apr 8 18:05:44 2019] [debug]: Autohandler called ExternalAuth. Response: (0, No User) (/usr/share/html/Elements/DoAuth:58)
[4426] [Mon Apr 8 18:05:47 2019] [debug]: Attempting to use external auth service: My_LDAP (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:288)
[4426] [Mon Apr 8 18:05:47 2019] [debug]: Calling UserExists with $username (gao) and $service (My_LDAP) (/usr/sbin/…/lib/RT/Authen/ExternalAuth.pm:329)
[4426] [Mon Apr 8 18:05:47 2019] [debug]: UserExists params:
username: gao , service: My_LDAP (/usr/sbin/…/lib/RT/Authen/ExternalAuth/LDAP.pm:486)
[4426] [Mon Apr 8 18:05:47 2019] [critical]: RT::Authen::ExternalAuth::LDAP::_GetBoundLdapObj Can’t bind: LDAP_OPERATIONS_ERROR 1 (/usr/sbin/…/lib/RT/Authen/ExternalAuth/LDAP.pm:678)
[4426] [Mon Apr 8 18:05:47 2019] [debug]: Autohandler called ExternalAuth. Response: (0, No User) (/usr/share/html/Elements/DoAuth:58)
[4426] [Mon Apr 8 18:05:47 2019] [error]: FAILED LOGIN for gao from 10.36.9.18 (/usr/sbin/…/lib/RT/Interface/Web.pm:826)
[4457] [Mon Apr 8 18:06:02 2019] [warning]: RT::Authen::ExternalAuth has been cored since RT 4.4, please check the upgrade document for more details (/usr/sbin/…/lib/RT.pm:754)
[4457] [Mon Apr 8 18:06:02 2019] [warning]: RT::Authen::ExternalAuth has been cored since RT 4.4, please check the upgrade document for more details (/usr/sbin/…/lib/RT.pm:754)
[4457] [Mon Apr 8 18:06:02 2019] [debug]: Using internal Perl HTML → text conversion (/usr/sbin/…/lib/RT/Interface/Email.pm:1475)
[4457] [Mon Apr 8 18:06:02 2019] [debug]: The RTAddressRegexp option is not set in the config. Not setting this option results in additional SQL queries to check whether each address belongs to RT or not. It is especially important to set this option if RT receives emails on addresses that are not in the database or config. (/usr/sbin/…/lib/RT/Config.pm:577)

Is RT store/cache the root cert somewhere else?

It doesn’t look like the web server SSL/TLS cert to me - that error is saying that your RT instance can’t bind to the LDAP server. If you’ve not changed anything on the LDAP server at the same time, have you accidentally (or otherwise) turned on SELinux on the RT server? That can sometime cause issues with code being run by the web server not being able to start outboard network connections (such as LDAP lookups).

I already tried disable the Selinux and its still trouble.

We roll out new certificates on all servers, including all web server and the LDAP. Like I mentioned, all other servers works fine except this RT server. I see that the RT works as both server (for the web server part, i.e, apache) and client (The LDAP part, it query the LDAP via TLS). The issue here is when RT try to connect LDAP is failed on TLS negotiation, which indicate something went wrong with the certificates. since it is the only thing changed here.

In the RT_SiteConfig.pm you posted originally, have you tried replacing:

‘tls’ => {
verify => ‘require’,
cafile => ‘/etc/pki/ca-trust/source/anchors/root-ca.pem’,
},

with just

'tls' => 1,

and then clearing Mason cache & restarting your web server? If that works (its doing TLS but not validating the certs) then I’d be checking that /etc/pki/ca-trust/source/anchors/root-ca.pem is actually the root CA cert for your new LDAP server certificate.

Hi GreenJimll,

Thank you so much. I changes to 'tls' => 1, and restarted httpd and it works now! Although I still don’t understand what cause the issue with the old configuration.

What 'tls' => 1, is doing is allowing the transport layer security protocol to run, but not checking the certificate that is returned. What you had before was a stronger security - you were asking for TLS protocol use and checking that the certificate that the LDAP server returned was valid based on the given root Certificate Authority (CA) certificate.

I’m guessing that your new LDAP server certificate comes with a different root CA path, so you need to tell RT what that new root CA is (ie replace /etc/pki/ca-trust/source/anchors/root-ca.pem with whatever root CA came with your new LDAP server cert).

Yes we just make new root CA (because we need the root CA cover more sub domains) , intermediate and host certificates.

So when I test with our LDAP server, I can run this command from the RT server. So I think the LDAP auth works but somehow RT didn’t send correct rootCA cert. I see you mentioned mason_cache and I haven’t figured out how to clear it.

[jgao@support-01 ~]$ openssl s_client -connect ldap:636 -CAfile /etc/pki/ca-trust/source/anchors/DQT.pem 
CONNECTED(00000003)
depth=2 C = CA, ST = British Columbia, L = Mycity, O = mydomain, OU = IT, CN = DQT Root CA: 2019-03-20
verify return:1
depth=1 C = CA, ST = British Columbia, L = Mycity, O = mydomain, OU = IT, CN = DQT Intermediate CA: 2019-03-20
verify return:1
depth=0 C = CA, ST = British Columbia, L = Mycity, O = mydomain, OU = IT, CN = ldap.hq.mydomain.com
verify return:1
---
Certificate chain
 0 s:/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=ldap.hq.mydomain.com
   i:/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=DQT Intermediate CA: 2019-03-20
 1 s:/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=DQT Intermediate CA: 2019-03-20
   i:/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=DQT Root CA: 2019-03-20
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIE1zCCA7+gAwIBAgIQQG1mHJJm1W6WeAAoVGUdKzANBgkqhkiG9w0BAQsFADCB
iTELMAkGA1UEBhMCQ0ExGTAXBgNVBAgMEEJyaXRpc2ggQ29sdW1iaWExEjAQBgNV
BAcMCVZhbmNvdXZlcjEUMBIGA1UECgwLM3ZHZW9tYXRpY3MxCzAJBgNVBAsMAklU
MSgwJgYDVQQDDB8zdkcgSW50ZXJtZWRpYXRlIENBOiAyMDE5LTAzLTIwMB4XDTE5
MDQwMzAwNDIxMloXDTIxMDMxOTAwNDIxMlowgYUxCzAJBgNVBAYTAkNBMRkwFwYD
VQQIDBBCcml0aXNoIENvbHVtYmlhMRIwEAYDVQQHDAlWYW5jb3V2ZXIxFDASBgNV
BAoMCzN2R2VvbWF0aWNzMQswCQYDVQQLDAJJVDEkMCIGA1UEAwwbbGRhcC0wMS52
YW4uM3ZnZW9tYXRpY3MuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAsbu53rFkgOGgibzV09fmVpsstwn88zyeA7TapQ86AR3BvWmVEncmK8nnKMkQ
sIHtxwODR6WDsXJhTf8hw8tnUUb1kFUPahzGR3HZrfBTKSq5nV/Yejb+5/eEYIDK
19tCyHJa5YDHeewOPNllaeLIku4pptwSIsPrTbQ/oms42HHXWIW/NHjnQhEJt5Gd
ydnRVDZTuCbAKxGnGNeeT1f4dknz/2pISm3PIHi1rOmdfBFPlSM3GWduV+lmyD7k
eHOSgCry1VuX3nbLCDqmnaFpR1/Duv+E3ygHd6MAhBc0c+oe6QIgYi3G6UnrVt0G
z5tMeyaby9g3GujobI5Ohb8KqQIDAQABo4IBOzCCATcwCQYDVR0TBAIwADAOBgNV
HQ8BAf8EBAMCBaAwIAYDVR0lAQH/BBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0G
A1UdDgQWBBQkBOGqAZc75PHckUBXDPLtY3oKUjAfBgNVHSMEGDAWgBTDc3bKnWLB
BQcwAoY5aHR0cDovL2NlcnQudmFuLjN2Z2VvbWF0aWNzLmNvbS8zdmdfaW50ZXJt
ZWRpYXRlLmNlcnQucGVtMCYGA1UdEQQfMB2CG2xkYXAtMDEudmFuLjN2Z2VvbWF0
aWNzLmNvbTANBgkqhkiG9w0BAQsFAAOCAQEAnq6tjBeaBmYw6AVpIz33AidOr/rJ
5exT39WWb/gCSB8dsx8bX+wd4X1ijxLoN4j7B509njznLGLzw7JRMsSB6PkXUZAm
UTaZikzDK71gwZRZTHs0vjkZjAesUAX1wDI2MYSBOl8WmFb0OriKxyQ5Yei0sPgk
9aUWhHz892BectAk+rDgZP8lG0fR7OyxUXWrlkFos4tD6jx3wRLla9q5txOrn8mk
iWPuEDUGbAdUp1FUiUiaNgbOOkwfJeQfaagdh1eBphJBfphjzR8t3amYWNGpR9zq
YtXYzML4Qc20aUKQEsdqaWf762H1ZFaS0auTOFGGBHSoHfVrKQvrHexgEQ==
-----END CERTIFICATE-----
subject=/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=ldap.hq.mydomain.com
issuer=/C=CA/ST=British Columbia/L=Mycity/O=mydomain/OU=IT/CN=DQT Intermediate CA: 2019-03-20
---
No client certificate CA names sent
Server Temp Key: ECDH, prime256v1, 256 bits
---
SSL handshake has read 3218 bytes and written 373 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 5C4520D9218063F62C2CB98FF7F9249EC13235C59B0EDFBF91DF3AFD5C8A7A0A
    Session-ID-ctx: 
    Master-Key: 2E22222222E98B5FFCC1AD099F5048F1BB2A66DA0D8405738349A11111110A67B979BBFEB4F026E862DCCDB641B3CC9B
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1555439154
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---