Bug in regression tests?

RT Developers
1

I’ve just been setting up a test machine to run the regression tests on, and
I’m getting these results:

…
ok 33 - Opened the mailgate -
ok 34 - user exists and was created by ticket comment submission
ok 35 - Opened the mailgate -
[Tue Jun 17 08:14:07 2003] [crit]: read-open …/…/…/html/NoAuth/images/spacer.gif: No such file or directory at /usr/share/perl5/MIME/Body.pm line 417.
(/tmp/rt-test-28500/inst/lib/RT.pm:242)
1…35

and then it returns 0, indicating success. It looks like it’s supposed to be a fail though, and I’m sure there’s more than 35 tests, right?

Also, any ideas why this would be failing? (where’s test 35?)

jaq@spacepants.org http://spacepants.org/jaq.gpg

2

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

From untarring, what steps did you take to try to run the test suite?
(and that failure is in the embedded tests in bin/rt-mailgate.in.
If the test had failed, rather than having the harness die, it would
tell you where it failed.On Tue, Jun 17, 2003 at 06:16:35PM +1000, Jamie Wilkinson wrote:

I’ve just been setting up a test machine to run the regression tests on, and
I’m getting these results:

…
ok 33 - Opened the mailgate -
ok 34 - user exists and was created by ticket comment submission
ok 35 - Opened the mailgate -
[Tue Jun 17 08:14:07 2003] [crit]: read-open …/…/…/html/NoAuth/images/spacer.gif: No such file or directory at /usr/share/perl5/MIME/Body.pm line 417.
(/tmp/rt-test-28500/inst/lib/RT.pm:242)
1…35

and then it returns 0, indicating success. It looks like it’s supposed to be a fail though, and I’m sure there’s more than 35 tests, right?

Also, any ideas why this would be failing? (where’s test 35?)

–
jaq@spacepants.org http://spacepants.org/jaq.gpg

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

3

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

Found it.

ok 34 - user exists and was created by ticket comment submission
ok 35 - Opened the mailgate -
[Tue Jun 17 08:14:07 2003] [crit]: read-open …/…/…/html/NoAuth/images/spacer.gif: No such file or directory at /usr/share/perl5/MIME/Body.pm line 417.
(/tmp/rt-test-28500/inst/lib/RT.pm:242)

It turns out that the test for the binary attachment is trying to attach
spacer.gif, but the relative path given there doesn’t exist and isn’t where
spacer.gif lives in my little test harness setup.

I changed the string “…/…/…/html” to @MASON_HTML_PATH@ on lines 230 and 249
in bin/rt-mailgate.in and the test passes again. I dunno if that’s going to
break your regression setup, as it’ll mean needing a configure and install
before running the regression tests – I can’t remember if you’d said that’s
the way it happens or not.

jaq@spacepants.org http://spacepants.org/jaq.gpg

4

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

Found it.

ok 34 - user exists and was created by ticket comment submission
ok 35 - Opened the mailgate -
[Tue Jun 17 08:14:07 2003] [crit]: read-open …/…/…/html/NoAuth/images/spacer.gif: No such file or directory at /usr/share/perl5/MIME/Body.pm line 417.
(/tmp/rt-test-28500/inst/lib/RT.pm:242)

It turns out that the test for the binary attachment is trying to attach
spacer.gif, but the relative path given there doesn’t exist and isn’t where
spacer.gif lives in my little test harness setup.

your little test harness setup?
Something other than:

./configure; make regression

?

I changed the string “…/…/…/html” to @MASON_HTML_PATH@ on lines 230 and 249
in bin/rt-mailgate.in and the test passes again. I dunno if that’s going to
break your regression setup, as it’ll mean needing a configure and install
before running the regression tests – I can’t remember if you’d said that’s
the way it happens or not.

–
jaq@spacepants.org http://spacepants.org/jaq.gpg

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

5

This one time, at band camp, Jesse Vincent wrote:

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

Found it.

ok 34 - user exists and was created by ticket comment submission
ok 35 - Opened the mailgate -
[Tue Jun 17 08:14:07 2003] [crit]: read-open …/…/…/html/NoAuth/images/spacer.gif: No such file or directory at /usr/share/perl5/MIME/Body.pm line 417.
(/tmp/rt-test-28500/inst/lib/RT.pm:242)

It turns out that the test for the binary attachment is trying to attach
spacer.gif, but the relative path given there doesn’t exist and isn’t where
spacer.gif lives in my little test harness setup.

your little test harness setup?
Something other than:

./configure; make regression

Yes, it does a little bit more, because I can’t rely on using the current
apache and postgres installations for regression testing, I set up a
temporary copy of both, and configure rt for them, install it into a
temporary directory, then run the regression.

This is a single aegis test program that runs the RT regressions, but
self contained so that it doesn’t impact anything else on the system. It
should be simple enough to modify for using with mysql, and other
combinations of mod_perl and speedy cgi (I’m currently using FastCGI).

Once I’ve debugged it to the point that I’m happy with it, and I can trust
it for rolling out new versions of my package onto my production machine,
I’ll send you a copy for perusal.

jaq@spacepants.org http://spacepants.org/jaq.gpg

6

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

I’m now getting up to 154 with no errors, and now no log messages indicating
a failure either:

…
ok 154 - Has three attachments, presumably a text-plain, a text-html and a
multipart alternative
1…154

So it seems to be getting to all the tests up to and through
04send_email.pl.

Are there any more that should be run, or are there some 750 tests that
don’t get run because of my configuration?

jaq@spacepants.org http://spacepants.org/jaq.gpg

7

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

I’m now getting up to 154 with no errors, and now no log messages indicating
a failure either:

…
ok 154 - Has three attachments, presumably a text-plain, a text-html and a
multipart alternative
1…154

So it seems to be getting to all the tests up to and through
04send_email.pl.

Are there any more that should be run, or are there some 750 tests that
don’t get run because of my configuration?

Can you try the vanilla regression test suite the way I’ve described it,
please?

–
jaq@spacepants.org http://spacepants.org/jaq.gpg

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

8

This one time, at band camp, Jesse Vincent wrote:

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

I’m now getting up to 154 with no errors, and now no log messages indicating
a failure either:

…
ok 154 - Has three attachments, presumably a text-plain, a text-html and a
multipart alternative
1…154

So it seems to be getting to all the tests up to and through
04send_email.pl.

Are there any more that should be run, or are there some 750 tests that
don’t get run because of my configuration?

“my conifiguration” == “–with-postgres and --with-fastcgi”, not the
self-contained regression area.

Can you try the vanilla regression test suite the way I’ve described it,
please?

The problems I have with the vanilla regression test suite is that it does
not just work out of the box, which is why I’m trying to get to a point
where I can automate the entire set up and clean up of the regression.

That’s fine, but to fix anything, I need you to start with the very
basic step of attempting to run the test suite as it’s designed to be
run, not through a self-designed harness.

So, you say there are 900 tests. I am only seeing 154 of them. Is it
possible that some tests only get run with mysql and mod_perl1, for example,
or should I be expecting to see 900 tests get run regardless of how it’s set
up?

The tests are not conditional. You should see all of them.

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

9

This one time, at band camp, Jamie Wilkinson wrote:

This one time, at band camp, Jesse Vincent wrote:

That does indeed look like a bug in the test suite or your setup. there
are about 900 tests :wink:

I’m now getting up to 154 with no errors, and now no log messages indicating
a failure either:

Fixed my other problem now. It was due to the way testify-pods creates the
tests, and the way my test harness was setting things up… long story cut
short is that now all of the parts of Makefile.in that I care about now
support split build/source directories.

So, the full regression suite runs nicely now and I discover that 2 tests
fail and one test seems to fail but doesn’t actually report a failure.

First, in lib/RT/CurrentUser.pm:

not ok 431 - Localized TEST_STRING into Frenc

Failed test (/home/jaq/src/rt.3.0.C038/lib/RT/CurrentUser.pm at line 321)

I modified the test a little, so that line 321 reads

ok ($cu->loc(‘Before’) eq “Avant”, "Localized TEST_STRING into French: " . $cu->loc(‘Before’));

and it reports that $cu->loc(‘Before’) returns the string ‘Before’. I’ve
just installed the fr locales on my system and the test still fails. What
would I need for this to pass?

Second, I get this error message from a test that doesn’t report failure:

ok 449 - Created the scrip action
[Thu Jun 19 10:06:57 2003] [error]: Ticket creation failed for 20 Can’t
call method “Id” on an undefined value at template line 7.
(/tmp/rt-test-32176/inst/lib/RT/Action/CreateTickets.pm:358)
[Thu Jun 19 10:06:57 2003] [error]: Ticket creation failed for 20 Can’t call
method “Id” on an undefined value at template line 2.
(/tmp/rt-test-32176/inst/lib/RT/Action/CreateTickets.pm:358)
ok 450
ok 451
…

Looks like the ActionObj part of the CreateTickets test is breaking somehow,
but no idea why.

Finally, the recent encoding test is breaking:

ok 644 - The only part is text/plain
not ok 645 - The =?KOI8-R?B?W2V4YW1wbGUuY29tICMyN10g1MXT1CDUxdPU?= is
encoded correctly

Failed test ((eval 899) at line 10)

ok 646 - found ticket 27
…

…
1…649

Looks like you failed 2 tests of 649.

Still not 900 though… I’m using all of the tests out of bin and lib now, I’m
pretty sure of that.

jaq@spacepants.org http://spacepants.org/jaq.gpg

10

This one time, at band camp, Jamie Wilkinson wrote:

So, the full regression suite runs nicely now and I discover that 2 tests
fail and one test seems to fail but doesn’t actually report a failure.

I should have mentioned that I’m using 3.0.3rc2 now.

jaq@spacepants.org http://spacepants.org/jaq.gpg

11

This one time, at band camp, Jamie Wilkinson wrote:

I should have mentioned that I’m using 3.0.3rc2 now.

And those fixes to WhoHaveRight seem to have done the trick! Sub-second
page loads when Reply/Resolve/Comment are clicked on!

Thanks heaps Jesse. Have a good birthday.

jaq@spacepants.org http://spacepants.org/jaq.gpg

12

This one time, at band camp, Jamie Wilkinson wrote:

I should have mentioned that I’m using 3.0.3rc2 now.

And those fixes to WhoHaveRight seem to have done the trick! Sub-second
page loads when Reply/Resolve/Comment are clicked on!

Thanks heaps Jesse. Have a good birthday.

That’s very good news. Thanks :slight_smile:

–
jaq@spacepants.org http://spacepants.org/jaq.gpg

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

13

First, in lib/RT/CurrentUser.pm:

not ok 431 - Localized TEST_STRING into Frenc

Failed test (/home/jaq/src/rt.3.0.C038/lib/RT/CurrentUser.pm at line 321)

I modified the test a little, so that line 321 reads

ok ($cu->loc(‘Before’) eq “Avant”, "Localized TEST_STRING into French: " . $cu->loc(‘Before’));

and it reports that $cu->loc(‘Before’) returns the string ‘Before’. I’ve
just installed the fr locales on my system and the test still fails. What
would I need for this to pass?

That implies to me that you’ve not got the lexicon files somewhere that
the test harness can load them.

Second, I get this error message from a test that doesn’t report failure:

ok 449 - Created the scrip action
[Thu Jun 19 10:06:57 2003] [error]: Ticket creation failed for 20 Can’t
call method “Id” on an undefined value at template line 7.
(/tmp/rt-test-32176/inst/lib/RT/Action/CreateTickets.pm:358)
[Thu Jun 19 10:06:57 2003] [error]: Ticket creation failed for 20 Can’t call
method “Id” on an undefined value at template line 2.
(/tmp/rt-test-32176/inst/lib/RT/Action/CreateTickets.pm:358)

It’s not actually a test failure. It’s a buggy example that needs to get
cleaned up.

ok 450
ok 451
…

Looks like the ActionObj part of the CreateTickets test is breaking somehow,
but no idea why.

Finally, the recent encoding test is breaking:

ok 644 - The only part is text/plain
not ok 645 - The =?KOI8-R?B?W2V4YW1wbGUuY29tICMyN10g1MXT1CDUxdPU?= is
encoded correctly

Failed test ((eval 899) at line 10)

I’d be curious if that’s fixed as of RC3. There appears to be minor
oddness with the encoding of that KOI8 string being non-deterministic.
really, we should encode and then decode and check it. A patch would be
appreciated.

ok 646 - found ticket 27
…

…
1…649

Looks like you failed 2 tests of 649.

Still not 900 though… I’m using all of the tests out of bin and lib now, I’m
pretty sure of that.

Perhaps you’re failing to run the web tests in 03web.pl

–
jaq@spacepants.org http://spacepants.org/jaq.gpg

rt-devel mailing list
rt-devel@lists.fsck.com
http://lists.fsck.com/mailman/listinfo/rt-devel

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.

14

This one time, at band camp, Jesse Vincent wrote:

That implies to me that you’ve not got the lexicon files somewhere that
the test harness can load them.

Yep, that was it, thanks.

It’s not actually a test failure. It’s a buggy example that needs to get
cleaned up.

Ok, cool.

I’d be curious if that’s fixed as of RC3. There appears to be minor
oddness with the encoding of that KOI8 string being non-deterministic.
really, we should encode and then decode and check it. A patch would be
appreciated.

Using RC4 now, and the KOI8 test is still failing.

Perhaps you’re failing to run the web tests in 03web.pl

I’m pretty sure that 03web.pl is being used, I’ll double-check.

jaq@spacepants.org http://spacepants.org/jaq.gpg

15

I’d be curious if that’s fixed as of RC3. There appears to be minor
oddness with the encoding of that KOI8 string being non-deterministic.
really, we should encode and then decode and check it. A patch would be
appreciated.

Using RC4 now, and the KOI8 test is still failing.

Yeah. I’m pretty sure it’s a spurious test failure. The right fix is to
test the decoded output

Request Tracker... So much more than a help desk — Best Practical Solutions – Trouble Ticketing. Free.