Intermittent login / sessioning problem

Bug Report Summary
Behavior Expected: Web interface should allow logins for known active
users
Behavior Observed: Periodically the web interface will deny logins
for any valid user, returning a “Your username or
password is incorrect” message. Once a sequence
of steps is taken (described below) the problem
disappears. The problem recurs periodically.
Versions: RT-2.0.14 from source, Debian Linux (kernel SMP
2.4.19), Apache 1.3.26, MySQL 3.23.52

Discussion

I’m experiencing a frustrating problem with rt-2.0.14 where all logins
via the web interface (using any valid user/password) result in a
"Your username or password is incorrect" message.

This first happened after the initial installation. I even dropped and
reloaded the database, stopped and restarted apache, scoured the mailing
lists, etc. Eventually some combination of clearing out the sessiondata
directory and restarting the web server did the trick and I could log in.

Since then the problem periodically occurs again. When it happens NOONE
can log into the system until the problem is fixed. Through combing the
mailing list for suggestions I’ve experimented a bit with isolating the
sequence of operations which will fix the problem and I’ve gotten it
down to this (which I’ve saved as a script I run whenever the problem
arises):

------------------------- cut ------------------------------
#!/bin/sh
RT_PATH=’/path/to/rt’
/etc/init.d/apache stop &&
/bin/rm ${RT_PATH}/WebRT/sessiondata/* &&
/usr/bin/perl -i.orig -pne ‘s/METHOD=POST/METHOD=GET/’ ${RT_PATH}/WebRT/html/Elements/Login &&
/etc/init.d/apache start &&
/usr/bin/perl -MLWP::Simple -e ‘get(“http://www.domain.com/?user=someuser&pass=somepass”);’ &&
perl -i.orig -pne ‘s/METHOD=GET/METHOD=POST/’ ${RT_PATH}/WebRT/html/Elements/Login &&
rm ${RT_PATH}/WebRT/html/Elements/Login.orig
------------------------- cut ------------------------------

Basically, shut down the web server, clear out sessions, change the
Login form submission method from POST to GET, start the web server, log
in a user, set the Login form submission method back to POST, and clean
up. Once I run this script everyone can log in happily until the next
time the problem arises. Under very light RT usage the problem arises
about 1 time per day, but frequency rises as usage rises.

The sequence of steps in the script always seems to work, but it may not
be a minimal set of operations. I do know that the POST->GET swap is
absolutely necessary to break the logjam (so to say), but once the jam
is broken I can set GET back to POST without any trouble. We don’t feel
comfortable sending passwords via GET – they appear in browser location
bars and in Apache logfiles.

Suspicions:

  • letting a user session expire by leaving it logged in for hours
    and then trying to perform an operation (forcing re-authentication)
    may be a trigger for this behavior.

This denied login behavior happens under Mozilla 1.0 & 1.1, Netscape
4.72, IE5.5, and Opera6.

Note that barring this intermittent login problem our RT installation
appears to function perfectly.

Here are some details about my installation:

This is a Debian Linux system.

I am running RT 2.0.14, built from source tarball:

/devel/tools/tgz$ md5sum rt-2.0.14.tar.gz
59179c054c0e05336f9b92fe3b031270 rt-2.0.14.tar.gz

% uname -a
Linux dev 2.4.19 #4 SMP Tue Sep 17 19:58:10 CDT 2002 i686 unknown unknown GNU/Linux

Apache 1.3.26 vhost configuration for this host:

<VirtualHost ###.###.###.###:80>
DocumentRoot /path/to/rt/WebRT/html
ServerName www.domain.com
PerlModule Apache::DBI
ErrorLog logs/error_support
CustomLog logs/access_support combined
PerlRequire /path/to/rt/bin/webmux.pl

SetHandler perl-script
PerlHandler RT::Mason

% perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 0) configuration:
Platform:
osname=linux, osvers=2.4.19, archname=i386-linux-thread-multi
uname=‘linux cyberhq 2.4.19 #1 smp sun aug 4 11:30:45 pdt 2002 i686 unknown unknown gnulinux ‘
config_args=’-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dcccdlflags=-fPIC -Darchname=i386-linux -Dprefix=/usr -Dprivlib=/usr/share/perl/5.8.0 -Darchlib=/usr/lib/perl/5.8.0 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.8.0 -Dsitearch=/usr/local/lib/perl/5.8.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Uusesfio -Uusenm -Duseshrplib -Dlibperl=libperl.so.5.8.0 -Dd_dosuid -des’
hint=recommended, useposix=true, d_sigaction=define
usethreads=define use5005threads=undef useithreads=define usemultiplicity=define
useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
use64bitint=undef use64bitall=undef uselongdouble=undef
usemymalloc=n, bincompat5005=undef
Compiler:
cc=‘cc’, ccflags =’-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64’,
optimize=’-O3’,
cppflags=’-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing -I/usr/local/include’
ccversion=’’, gccversion=‘2.95.4 20011002 (Debian prerelease)’, gccosandvers=’‘
intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=12
ivtype=‘long’, ivsize=4, nvtype=‘double’, nvsize=8, Off_t=‘off_t’, lseeksize=8
alignbytes=4, prototype=define
Linker and Libraries:
ld=‘cc’, ldflags =’ -L/usr/local/lib’
libpth=/usr/local/lib /lib /usr/lib
libs=-lgdbm -ldb -ldl -lm -lpthread -lc -lcrypt
perllibs=-ldl -lm -lpthread -lc -lcrypt
libc=/lib/libc-2.2.5.so, so=so, useshrplib=true, libperl=libperl.so.5.8.0
gnulibc_version=‘2.2.5’
Dynamic Linking:
dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags=’-rdynamic’
cccdlflags=’-fPIC’, lddlflags=’-shared -L/usr/local/lib’

Characteristics of this binary (from libperl):
Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL_IMPLICIT_CONTEXT
Built under linux
Compiled at Sep 14 2002 17:36:21
@INC:
/etc/perl
/usr/local/lib/perl/5.8.0
/usr/local/share/perl/5.8.0
/usr/lib/perl5
/usr/share/perl5
/usr/lib/perl/5.8.0
/usr/share/perl/5.8.0
/usr/local/lib/site_perl
.

Package versions of interest (via dkpg):

ii apache 1.3.26-1.1 Versatile, high-performance HTTP server
ii apache-common 1.3.26-1.1 Support files for all Apache webservers
ii apache-dev 1.3.26-1.1 Apache webserver development kit
rc apache-ssl 1.3.26.1+1.48- Versatile, high-performance HTTP server with
ii libapache-dbi- 0.89-1 Connect apache server to database via perl’s
ii libapache-mod- 1.27-2 Integration of perl with the Apache web serv
ii libapache-requ 1.0-0.1 Generic Apache Request Library
ii libapache-sess 1.54-1 Perl modules for keeping persistent user dat
ii libappconfig-p 1.52-5 Perl module for configuration file and comma
ii libapt-pkg-per 0.1.6 Perl interface to libapt-pkg
ii libcgi-perl 2.76-21 modules for perl5, for use in writing CGI sc
ii libcgi-pm-perl 2.81-6 a Perl5 CGI Library
ii libcrypt-sslea 0.23-2.1 Support for https protocol in LWP
ii libdbd-mysql-p 1.2219-6 mySQL database interface for Perl
ii libdbd-pg-perl 1.13-4 a PostgreSQL interface for Perl 5 using DBI.
ii libdbi-perl 1.28-4 The Perl5 Database Interface by Tim Bunce
ii libdbix-search 0.61-1 Perl extension for easy SQL SELECT Statement
ii libdevel-symdu 2.03-1 Perl module for inspecting perl’s symbol tab
ii liberror-perl 0.15-1 Exception module for Perl
ii libfreezethaw- 0.43-1 converting Perl structures to strings and ba
ii libhtml-mason- 1.13-2 HTML::Mason Perl module
ii libio-stringy- 2.108-1 Perl5 modules for IO from scalars and arrays
ii libipc-shareli 0.08-5 Perl module that provides a simple interface
ii libmailtools-p 1.49-1 Manipulate email in perl programs
ii libmd5-perl 2.02-3.1 backwards-compatible wrapper for Digest::MD5
ii libmime-base64 2.12-4.1 MIME/Base64 decoding for Perl
ii libmime-perl 5.411-2 Perl5 modules for MIME-compliant messages (M
ii libmldbm-perl 2.00-9 MLDBM perl module
ii libmysqlclient 3.23.52-2 mysql database client library
ii libneon23 0.23.2-1 An HTTP and WebDAV client library
rc libnet-perl 1.09.01-1 Implementation of Internet protocols for Per
ii libnet-snpp-pe 1.11-2.1 Perl module for accessing SNPP servers
ii libparams-vali 0.24-2 Validate parameters to Perl method/function
ii libpcre3 3.4-1.1 Philip Hazel’s Perl Compatible Regular Expre
ii libperl5.6 5.6.1-7 Shared Perl library.
ii libperl5.8 5.8.0-13 Shared Perl library.
ii libpgperl 7.2.2-2 Perl modules for PostgreSQL.
ii libtext-templa 1.42-2 Text::Template perl module
ii libtie-ixhash- 1.21-2 ordered associative arrays for Perl
ii libtimedate-pe 1.11-5 Time and date functions for perl.
ii liburi-perl 1.18-1 Manipulates and accesses URI strings
ii libwww-perl 5.65-0.1 WWW client/server library for Perl
ii mysql-client 3.23.52-2 mysql database client binaries
ii mysql-common 3.23.52-2 mysql database common files (e.g. /etc/mysql
ii mysql-server 3.23.52-2 mysql database server binaries
ii perl 5.8.0-13 Larry Wall’s Practical Extraction and Report
ii perl-base 5.8.0-13 The Pathologically Eclectic Rubbish Lister.
ii perl-doc 5.6.1-7 Perl documentation.
ii perl-modules 5.8.0-13 Core Perl modules.
ii perl-suid 5.8.0-13 Runs setuid Perl scripts.

Any help with this matter is greatly appreciated.

Rick
http://www.rickbradley.com MUPRN: 702 (66F/66F)
| Yeah, I was pretty
random email haiku | well torched by like midnight (that
| was well afterwards).

Bug Report Summary

Behavior Expected: Web interface should allow logins for known active
users
Behavior Observed: Periodically the web interface will deny logins
for any valid user, returning a “Your username or
password is incorrect” message. Once a sequence
of steps is taken (described below) the problem
disappears. The problem recurs periodically.
Versions: RT-2.0.14 from source, Debian Linux (kernel SMP
2.4.19), Apache 1.3.26, MySQL 3.23.52

Additional information

I can reliably trigger this behavior in the following manner (note that
this is almost certainly not how we reach this no-login state in normal
usage):

  • log in as a valid user and do not log out
  • stop the Apache web server
  • delete the session files in WebRT/sessiondata
  • start the Apache web server
  • on the still-open logged in web page click a link

User is forcibly logged out and no users can log in again until the
"fix" script I included in the initial report is run.

There is some other means of triggering this problem through normal
usage. Note that I disabled the sessiondata cleanup cron job after a
couple of days of dealing with this problem so that’s not triggering
this lockout behavior.

By modifying the login error messages in WebRT/data/obj/standard/autohandler
I have been able to determine that the “Your username or password is
incorrect” is coming from line 62 of that file. This is where the
session’s password is checked, but my guess is the session data as
retrieved is invalid and should be invalidated before checking. Not
being familiar with the code it would take me a while to identify a fix.

Rick
http://www.rickbradley.com MUPRN: 333 (64F/64F)
| think it got hit by
random email haiku | lightning or something." So, I
| think that one’s no go.

Bug Report Summary

Behavior Expected: Web interface should allow logins for known active
users
Behavior Observed: Periodically the web interface will deny logins
for any valid user, returning a “Your username or
password is incorrect” message. Once a sequence
of steps is taken (described below) the problem
disappears. The problem recurs periodically.
Versions: RT-2.0.14 from source, Debian Linux (kernel SMP
2.4.19), Apache 1.3.26, MySQL 3.23.52

We were able to track down what appears to be the origin of this
problem. Via the modperl list [0] (mostly off-list) we received some
assistance which suggested that crypt() is indeed not reentrant, but
that there is a common reentrant crypt() which some Perl’s support.
Following up further we found that we appear to have a reentrant crypt()
that appears to be recognized by Perl but are still having the problem.

Additional research turned up a problem in Linux’s glibc implementation
of crypt() which already has a workaround implemented for bleeding edge
Perls. [1]

With respect to RT I consider this issue resolved.

[0] http://marc.theaimsgroup.com/?l=apache-modperl&m=103367209622204&w=2
[1] http://marc.theaimsgroup.com/?l=perl5-porters&m=103012185631309&w=2

Rick
http://www.rickbradley.com MUPRN: 753 (59F/57F)
| good little girl much
random email haiku | less a sinner like you!). No
| packing as of yet.