Hi,
My colleague Mike Johnson originally sent this request. I’d like to follow
up on our investigation.
We have figured out the why, but we still don’t understand what is causing
this issue. Details below.
While debugging, we captured packets and received error code 52e from the
AD server (LDAP equivalent of error 49 RT was throwing). This indicates
that the login username is correct, but the password is incorrect. This is
what threw us off since it didn’t make sense. The config user/pass is
correct and the account would work 90% of the time and we would get this
error seemingly at random.
So we decided to edit the RT source code where the LDAP bind happens
(/opt/rt4/lib/RT/Authen/ExternalAuth/LDAP.pm:634) and added lines to log
the $ldap_user and $ldap_pass variables to see what RT was trying to bind
with.
On successful binds, the username/password combination was correct, as
specified in our config. We waited for the LDAP bind error to happen and
when it did, we found this in the log:
ldap_user: ldap_user_account, ldap_pass:Password not printed. (literal
Password not Printed). This is whats causing the bind to fail. For some
reason, RT is sending the masked password string (used in the web ui when
looking at system configs), which of course isn’t the right password for
the bind account.
As a temporary bandaid solution, we’ve hardcoded the LDAP bind credentials
in the source code. Since doing so, we haven’t seen the error happen again.
So at this point, this seems like a bug in RT? Has anyone else experienced
something similar?
Thanks,
Michel Daoust, BSc.
Software Developer / Systems Analyst, Information Technology
Northern Ontario School of Medicine
935 Ramsey Lake Road
Sudbury, ON. P3E 2C6
Phone: 705-662-7193
Email: midaoust@nosm.caOn Wed, Nov 23, 2016 at 3:21 PM, Mike Johnson mike.johnson@nosm.ca wrote:
Hi,
It happened again today. Our AD admin didn’t see anything unusual in the
logs. I’m getting him to see if successful bind attempts show up anywhere,
and if so… if RT is actually successful and the error message is just not
appropriate(ie something else behind the scenes is going on and it’s just
reported as a failed bind).
Anyone have any thoughts on this?
We have multiple other LDAP authenticated, and Windows authenticated
systems on campus using this AD service(different usernames) and we haven’t
had reports of any of these failing.
The things that have changed from what it was working:
- OS: CentOS 7.2.15.11
- perl 5.16.3
- RT version 4.4.1
I can’t recall the previous OS version or perl version, but it was at
least on Redhat 4 or 5, and RT was 3.8.X using ExternalAuth extension(on
3.8 it wasn’t rolled into baseline yet).
Any thoughts are appreciated!
Mike.
On Tue, Nov 22, 2016 at 4:40 PM, Kenneth Marshall ktm@rice.edu wrote:
On Tue, Nov 22, 2016 at 04:13:46PM -0500, Mike Johnson wrote:
We just went live with RT 4.4.1 and it seems that LDAP authentication is
failing.
It has now died 2 days in a row, at approximately the same time.
The RT log is showing the following error:
2819] [Mon Nov 21 21:10:28 2016] [critical]:
RT::Authen::ExternalAuth::LDAP::_GetBoundLdapObj Can’t bind:
LDAP_INVALID_CREDENTIALS 49
(/opt/rt4/sbin/…/lib/RT/Authen/ExternalAuth/LDAP.pm:678)
This seems like a generic LDAP error, and it’s not pointing us to a
specific issue.
The user that we are binding with is a user that was in-use on our RT
3.8.X
environment that hadn’t had an issue in years (3?).
Restarting apache resolves the immediate issue, but clearly there is
something else going on that we should be able to fix permanently.
Anyone
have any ideas on where to look?
This didn’t come up in our testing, but I don’t believe we had the
volume
of credential testing that we do in prod.
Any help would be great!
P.S. The LDAP server is a Microsoft Active Directory server. This same
server was being used for ExternalAuth extension in 3.8.
Mike.
Hi Mike,
You probably will need to check your AD logs as well. We have seen issues
with AD authentication when an account is locked out by a bad password
login attempt. Since it is about the same time of day, maybe something
else is trying to login with those credentials and causing it to lock.
Regards,
Ken
–
Mike Johnson
Datatel Programmer/Analyst
Northern Ontario School of Medicine
955 Oliver Road
Thunder Bay, ON P7B 5E1
Phone: (807) 766-7331
Email: mike.johnson@nosm.ca