RT Upgrade failing from 3.6.6 to 3.8.6

RT Users
1

Ok I’ve been fighting this forever and its really starting to get to me.

I’m trying to upgrade RT and I’m at the step where I’m supposed to
upgrade the database to a certain point before applying a schema.

The command to run is:

/opt/rt3/sbin/rt-setup-database --prompt-for-dba-password --action
upgrade

It then attempts to connect as rt_user. I give it the password, proceed
through the prompts and then get:

DBI connect(‘dbname=rt3;host=localhost’,‘rt_user’,…) failed: Access
denied for user ‘rt_user’@‘localhost’ (using password: YES) at
/usr/lib/perl5/site_perl/5.8.8/DBIx/SearchBuilder/Handle.pm line 106

Connect Failed Access denied for user ‘rt_user’@‘localhost’ (using
password: YES)

at /opt/rt3/sbin/…/lib/RT.pm line 204

So basically, it appears that my password is incorrect, HOWEVER, I can
connect to mysql using the exact user and password. Its almost like the
perl script is NOT taking the prompts.

Hope someone can help.

-Josh

2

Hi Tom,

Yes I tried connecting to mysql directly from localhost and that worked:

[jbarron@help01 ~]$ mysql -u rt_user@localhost -p
Enter password:
ERROR 1045 (28000): Access denied for user
‘rt_user@localhos’@‘localhost’ (using password: YES)
[jbarron@help01 ~]$ mysql -u rt_user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 138
Server version: 5.0.77 Source distribution

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> exit;
Bye
[jbarron@help01 ~]$

The first connection failed when I specified localhost but connecting
directly and specifying no host worked fine. Just to verify it was
connecting to localhost I deliberately typed the password wrong and it
showed me using rt_user@localhost

-JoshFrom: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Monday, November 16, 2009 5:25 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6

Connect Failed Access denied for user ‘rt_user’@‘localhost’ (using
password: YES)

So basically, it appears that my password is incorrect, HOWEVER, I can

connect to mysql using the exact user and password. Its almost like
the
perl script is NOT taking the prompts.

Connecting at localhost?

In MySQL, the user ‘rt_user’@‘something-else’ is not the same user as
‘rt_user’@‘localhost’.

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

3

Connect Failed Access denied for user ‘rt_user’@‘localhost’ (using
password: YES)

So basically, it appears that my password is incorrect, HOWEVER, I can
connect to mysql using the exact user and password. Its almost like the
perl script is NOT taking the prompts.

Connecting at localhost?

In MySQL, the user ‘rt_user’@‘something-else’ is not the same user as
‘rt_user’@‘localhost’.

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

4

Try> Yes I tried connecting to mysql directly from localhost and that
worked:

[jbarron@help01 ~]$ mysql -u rt_user@localhost -p
Enter password:
ERROR 1045 (28000): Access denied for user
‘rt_user@localhos’@‘localhost’ (using password: YES)
[jbarron@help01 ~]$ mysql -u rt_user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 138
Server version: 5.0.77 Source distribution

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> exit;
Bye
[jbarron@help01 ~]$

On *nix, mysql programs read startup options from the following in order:

/etc/my.cnf
SYSCONFDIR/my.cnf
$MYSQL_HOME/my.cnf
The file specified with --defaults-extra-file, if any
~/.my.cnf

If any of these exist, and there is a [mysql] or [client] section that
contains a “host=…” line, then “mysql -u rt_user -p” will connect to
that host, not localhost.

To force a localhost connection, do:

mysql -h localhost -u rt_user -p

What I’m getting at is: are you sure your MySQL instance for RT is on
localhost?

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

5

Hi Tom,

There is only one my.cnf and it doesn’t contain those sections with that
line. From what I can determine mysql is set to listen to any address.
When I forced a localhost connection, I was able to connect as both root
and rt_user.

-JoshFrom: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Monday, November 16, 2009 5:38 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6

Try> Yes I tried connecting to mysql directly from localhost and that
worked:

[jbarron@help01 ~]$ mysql -u rt_user@localhost -p
Enter password:
ERROR 1045 (28000): Access denied for user
‘rt_user@localhos’@‘localhost’ (using password: YES)
[jbarron@help01 ~]$ mysql -u rt_user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 138
Server version: 5.0.77 Source distribution

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> exit;
Bye
[jbarron@help01 ~]$

On *nix, mysql programs read startup options from the following in
order:

/etc/my.cnf
SYSCONFDIR/my.cnf
$MYSQL_HOME/my.cnf
The file specified with --defaults-extra-file, if any
~/.my.cnf

If any of these exist, and there is a [mysql] or [client] section that
contains a “host=…” line, then “mysql -u rt_user -p” will connect to
that host, not localhost.

To force a localhost connection, do:

mysql -h localhost -u rt_user -p

What I’m getting at is: are you sure your MySQL instance for RT is on
localhost?

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

6

Hi Tom,

There is only one my.cnf and it doesn’t contain those sections with that
line. From what I can determine mysql is set to listen to any address.
When I forced a localhost connection, I was able to connect as both root
and rt_user.

-Josh

Huh. So you did:

mysql -h localhost -u rt_user -p

and you’re sure there’s no ~/.my.cnf (note leading dot, its a hidden file).

That is strange. I’m not sure how the schema upgrade script makes its
connection to mysql, if its using the mysql client program then it
should work exactly the same.

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

7

I was able to finally get the script to take. Now it seems I have a
permissions issue that I can’t get around:

[Tue Nov 17 16:24:02 2009] [error] [client 216.222.31.30] Could not
create ‘/opt/rt3/var/mason_data/obj/.__obj_create_marker’: Permission
denied\nStack:\n
[/usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/Interp.pm:222]\n
[/usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/Interp.pm:169]\n
[/usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/Interp.pm:155]\n
[/usr/lib/perl5/vendor_perl/5.8.8/Class/Container.pm:329]\n
[/usr/lib/perl5/vendor_perl/5.8.8/Class/Container.pm:53]\n
[/usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/ApacheHandler.pm:633]\n
[/opt/rt3/bin/…/lib/RT/Interface/Web/Handler.pm:161]\n
[/opt/rt3/bin/…/lib/RT/Interface/Web/Handler.pm:141]\n
[/opt/rt3/bin/webmux.pl:160]\n
[/usr/lib/perl5/vendor_perl/5.8.8/HTML/Mason/ApacheHandler.pm:0]\n

The directory in question has the following permissions on it:
[root@help01 rt3]# ls -l /opt/rt3/var/
total 24
drwxrwxrwx 2 apache apache 4096 Oct 30 15:51 log
drwxrwxrwx 5 apache apache 4096 Nov 17 13:33 mason_data
drwxrwxrwx 2 apache apache 4096 Oct 30 15:51 session_data

It doesn’t look like a permissions issue but I can’t figure out why it
is being denied. There is no .htaccess in there.

-JoshFrom: rt-users-bounces@lists.bestpractical.com
[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Barron,
Josh
Sent: Tuesday, November 17, 2009 9:13 AM
To: Tom Lahti
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6

Hi Tom,

There is only one my.cnf and it doesn’t contain those sections with that
line. From what I can determine mysql is set to listen to any address.
When I forced a localhost connection, I was able to connect as both root
and rt_user.

-Josh

From: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Monday, November 16, 2009 5:38 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6

Try> Yes I tried connecting to mysql directly from localhost and that
worked:

[jbarron@help01 ~]$ mysql -u rt_user@localhost -p
Enter password:
ERROR 1045 (28000): Access denied for user
‘rt_user@localhos’@‘localhost’ (using password: YES)
[jbarron@help01 ~]$ mysql -u rt_user -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 138
Server version: 5.0.77 Source distribution

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql> exit;
Bye
[jbarron@help01 ~]$

On *nix, mysql programs read startup options from the following in
order:

/etc/my.cnf
SYSCONFDIR/my.cnf
$MYSQL_HOME/my.cnf
The file specified with --defaults-extra-file, if any
~/.my.cnf

If any of these exist, and there is a [mysql] or [client] section that
contains a “host=…” line, then “mysql -u rt_user -p” will connect to
that host, not localhost.

To force a localhost connection, do:

mysql -h localhost -u rt_user -p

What I’m getting at is: are you sure your MySQL instance for RT is on
localhost?

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/
The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

8

[Tue Nov 17 16:24:02 2009] [error] [client 216.222.31.30] Could not
create ‘/opt/rt3/var/mason_data/obj/.__obj_create_marker’: Permission
denied\nStack:\n

You need to look specifically at the permissions on
/opt/rt3/var/mason_data/obj and verify that the user the web server runs
as can write to that directory. The higher level directories are
irrelevant.

Also, if you are using POSIX ACLs you might need to

getfacl /opt/rt3/var/mason_data/obj

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

9

Thanks Tom,

Looks like the permissions are set correctly:
[root@help01 rt3]# ls -l /opt/rt3/var/mason_data/
total 24
drwxrwxrwx 2 apache apache 4096 Oct 30 15:51 cache
drwxrwxrwx 2 apache apache 4096 Oct 30 15:51 etc
drwxrwxrwx 2 apache apache 4096 Nov 17 13:33 obj

[root@help01 rt3]# getfacl /opt/rt3/var/mason_data/obj
getfacl: Removing leading ‘/’ from absolute path names

file: opt/rt3/var/mason_data/obj

owner: apache

group: apache

user::rwx
group::rwx
other::rwxFrom: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Tuesday, November 17, 2009 2:47 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6 UPDATE:
mason_data directory

[Tue Nov 17 16:24:02 2009] [error] [client 216.222.31.30] Could not
create ‘/opt/rt3/var/mason_data/obj/.__obj_create_marker’: Permission
denied\nStack:\n

You need to look specifically at the permissions on
/opt/rt3/var/mason_data/obj and verify that the user the web server runs

as can write to that directory. The higher level directories are
irrelevant.

Also, if you are using POSIX ACLs you might need to

getfacl /opt/rt3/var/mason_data/obj

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

10

[Tue Nov 17 16:24:02 2009] [error] [client 216.222.31.30] Could not
create ‘/opt/rt3/var/mason_data/obj/.__obj_create_marker’: Permission

Does the file .__obj_create_marker already exist in that location,
with some un-overwritable permissions perhaps?

Or perhaps its in use?

lsof | grep marker

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

11

That file does not exist in the new RT directory, nor is any file by
that name in use anywhere from what I can see.

I’m really baffled by this.

-JoshFrom: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Tuesday, November 17, 2009 3:52 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6 UPDATE:
mason_data directory

[Tue Nov 17 16:24:02 2009] [error] [client 216.222.31.30] Could not
create ‘/opt/rt3/var/mason_data/obj/.__obj_create_marker’: Permission

Does the file .__obj_create_marker already exist in that location,
with some un-overwritable permissions perhaps?

Or perhaps its in use?

lsof | grep marker

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

12

-----Original Message-----
From: Barron, Josh

That file does not exist in the new RT directory, nor is any file by
that name in use anywhere from what I can see.

I’m really baffled by this.

SELinux isn’t turned on by any chance is it?

getenforce
ausearch -m avc -ts today

Stuart

13

Looks like it is on:

[root@help01 jbarron]# /usr/sbin/getenforce
Enforcing

[root@help01 jbarron]# /sbin/ausearch -m avc -ts today
From: Stuart Browne [mailto:stuart.browne@ausregistry.com.au]
Sent: Tuesday, November 17, 2009 6:04 PM
To: Barron, Josh; Tom Lahti
Cc: rt-users@lists.bestpractical.com
Subject: RE: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6
UPDATE:mason_data directory

-----Original Message-----
From: Barron, Josh

That file does not exist in the new RT directory, nor is any file by
that name in use anywhere from what I can see.

I’m really baffled by this.

SELinux isn’t turned on by any chance is it?

getenforce
ausearch -m avc -ts today

Stuart

14

I wasn’t able to get this working after trying a good portion of the
evening.
I finally ended up setting up a new virtual server and just migrating
the database over to it.

-Josh

[mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Barron,
JoshSent: Tuesday, November 17, 2009 6:14 PM
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to3.8.6
UPDATE:mason_data directory

Looks like it is on:

[root@help01 jbarron]# /usr/sbin/getenforce
Enforcing

[root@help01 jbarron]# /sbin/ausearch -m avc -ts today

From: Stuart Browne [mailto:stuart.browne@ausregistry.com.au]
Sent: Tuesday, November 17, 2009 6:04 PM
To: Barron, Josh; Tom Lahti
Cc: rt-users@lists.bestpractical.com
Subject: RE: [rt-users] RT Upgrade failing from 3.6.6 to 3.8.6
UPDATE:mason_data directory

-----Original Message-----
From: Barron, Josh

That file does not exist in the new RT directory, nor is any file by
that name in use anywhere from what I can see.

I’m really baffled by this.

SELinux isn’t turned on by any chance is it?

getenforce
ausearch -m avc -ts today

Stuart
http://lists.bestpractical.com/cgi-bin/mailman/listinfo/rt-users

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

15

Barron, Josh wrote:

I wasn’t able to get this working after trying a good portion of the
evening.
I finally ended up setting up a new virtual server and just migrating
the database over to it.

Did you ever try turning SELinux off? Just curious, I’ve never used it
so I’m not sure what the impact would be.

[root@help01 jbarron]# /usr/sbin/getenforce
Enforcing

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/

16

I did try and it was giving me other issues after doing that. Since the
server was old and running on limited resources, we made the final
decision to migrate clean anyways instead of throwing more time
resources at the old server.

Thanks for your help though!

-JoshFrom: Tom Lahti [mailto:toml@bitstatement.net]
Sent: Wednesday, November 18, 2009 1:18 PM
To: Barron, Josh
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] RT Upgrade failing from 3.6.6 to3.8.6
UPDATE:mason_data directory

Barron, Josh wrote:

I wasn’t able to get this working after trying a good portion of the
evening.
I finally ended up setting up a new virtual server and just migrating
the database over to it.

Did you ever try turning SELinux off? Just curious, I’ve never used it
so I’m not sure what the impact would be.

[root@help01 jbarron]# /usr/sbin/getenforce
Enforcing

Tom Lahti, SCMDBA, LPIC-1
BIT LLC
(425)251-0833 x 117
http://www.bitstatement.net/