Ticket Transactions get lost on the way to WebUI

Hello everyone!

Here’s my current setup:

RT 3.6.1

Perl v5.8.8 under linux

Apache v1.27;

DBIx::SearchBuilder v1.48;

HTML::Mason v1.35;

mod_perl v1.29;

Mysql 5

Approx 20 queues, 50 000 tickets

First of all, I’ve failed to find any solution in the lists, so I post
this question.

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html.

First thing I did, according to the Lists was reversing the history
display order. Guess what? The history kept displaying in that case only
till this very transaction!

So, for example, if there are transactions 1,2,3,4,5 defined for a
ticket, I could see transactions 1 and 2 in direct history order, and
transactions 5 and 4 in the reverse one!

Given that, I tried simply to remove transaction 3 using the Shredder.
In some cases that helped, in others not. No 100% solution here.

I looked directly into the mysql database and found out that all the
transactions that are not displayed are stored and are stored correctly,
as their respective attachments. They just are not displayed.

Then, I tried to consecutively remove all the page customizations I
store in /opt/rt3/local/html folder to see which one causes trouble.
None of them do. Even the neat one from the wiki:
HideTransactions - Request Tracker Wiki does not seem to
cause the issue))

Then, I tried to look into the ShowHistory module from
/opt/rt3/share/html/Ticket/ to see how this works.

I transferred the following lines that choose the ticket’s transactions
to a separate script:

my $Transactions = new RT::Transactions($session{‘CurrentUser’});

if ($Tickets) {

while (my $t = $Tickets->Next) {

$Transactions->LimitToTicket($t->id);

        }

} else {

$Transactions = $Ticket->Transactions;

}

I had my script return the Count of transactions for the ticket loaded:
it is the same as the mysql query result.

So, I do not know where to go from here.

Would really appreciate any help or directions.

Thank you very much!

Sincerely,

Kostya

Konstantin,

Is anything strange showing up in your apache error log?On Mon, Nov 19, 2007 at 04:57:15PM +0300, Konstantin Naryzhniy wrote:

Hello everyone!

Here’s my current setup:

RT 3.6.1

Perl v5.8.8 under linux

Apache v1.27;

DBIx::SearchBuilder v1.48;

HTML::Mason v1.35;

mod_perl v1.29;

Mysql 5

Approx 20 queues, 50 000 tickets

First of all, I’ve failed to find any solution in the lists, so I post
this question.

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html.

First thing I did, according to the Lists was reversing the history
display order. Guess what? The history kept displaying in that case only
till this very transaction!

So, for example, if there are transactions 1,2,3,4,5 defined for a
ticket, I could see transactions 1 and 2 in direct history order, and
transactions 5 and 4 in the reverse one!

Given that, I tried simply to remove transaction 3 using the Shredder.
In some cases that helped, in others not. No 100% solution here.

I looked directly into the mysql database and found out that all the
transactions that are not displayed are stored and are stored correctly,
as their respective attachments. They just are not displayed.

Then, I tried to consecutively remove all the page customizations I
store in /opt/rt3/local/html folder to see which one causes trouble.
None of them do. Even the neat one from the wiki:
HideTransactions - Request Tracker Wiki does not seem to
cause the issue))

Then, I tried to look into the ShowHistory module from
/opt/rt3/share/html/Ticket/ to see how this works.

I transferred the following lines that choose the ticket’s transactions
to a separate script:

my $Transactions = new RT::Transactions($session{‘CurrentUser’});

if ($Tickets) {

while (my $t = $Tickets->Next) {

$Transactions->LimitToTicket($t->id);

        }

} else {

$Transactions = $Ticket->Transactions;

}

I had my script return the Count of transactions for the ticket loaded:
it is the same as the mysql query result.

So, I do not know where to go from here.

Would really appreciate any help or directions.

Thank you very much!

Sincerely,

Kostya


The rt-users Archives

SAVE THOUSANDS OF DOLLARS ON RT SUPPORT:

If you sign up for a new RT support contract before December 31, we’ll take
up to 20 percent off the price. This sale won’t last long, so get in touch today.
Email us at sales@bestpractical.com or call us at +1 617 812 0745.

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html.

My guess is that you’re running into one of the very rare, but
catastrophic bugs in Text::Quoted. I alluded to this in my previous
email. You should see weird segfaults in the apache error log if this
is the case. Try 1) bringing it current or 2) removing it entirely.

Best,
Jesse

Hi Jesse, Konstantin,

same problem here, but slightly different.

Setup: 2 Webservers, 1 DB Server. I thought both webservers are configured
the same way. RT 3.6.5 with Apache 2, FastCGI and MySQL 5.0.41.

www1 is displaying everything correct, www2 has the same problems as
described by Konstantin, but not on every Ticket, only from Time to Time.

Got today a error Report on this from my Users with two differents tickets
in different Queues.

Both Servers have Text::Quoted 2.02. This takes me to a point to compare all
perl modules of both servers. This point me to a small difference between
both servers. The working one was the first server we setup, the second one
came some months later. The working server has modules installed which are
mostly many version older than the new one.

Only the Text::Quoted Module i installed after i found a threat from Ruslan.
So i think, not only the Text::Quoted Module alone can be the source of the
Problem.

Here is the difference between the servers:

Working Server Not Working Server

AutoLoader v5.60; AutoLoader v5.63;
Calendar::Simple v1.17;
CGI v3.05; CGI v3.29;
CGI::Cookie v1.24; CGI::Cookie v1.28;
CGI::Fast v1.05; CGI::Fast v1.07;
Cwd v2.19; Cwd v3.25;
Digest::MD5 v2.33; Digest::MD5 v2.36;
Encode v2.01; Encode v2.23;
Encode::Alias v2.00; Encode::Alias v2.07;
Encode::Byte v2.00; Encode::Byte v2.03;
Encode::Config v2.00; Encode::Config v2.04;
Encode::Encoding v2.00; Encode::Encoding v2.05;
Encode::Guess v2.00; Encode::Guess v2.02;
Encode::Unicode v2.00; Encode::Unicode v2.05;
File::Spec v0.87; File::Spec v3.25;
File::Temp v0.14; File::Temp v0.18;
I18N::LangTags v0.33; I18N::LangTags v0.35;
I18N::LangTags::List
v0.35;
IO v1.21; IO v1.23;
IO::File v1.10; IO::File v1.14;
IO::Handle v1.24; IO::Handle v1.27;
IO::Seekable v1.09; IO::Seekable v1.1;
List::Util v1.14; List::Util v1.19;
Locale::Maketext v1.09; Locale::Maketext v1.10;
MIME::Decoder::Base64
v5.420;
MIME::Decoder::Binary
v5.420;

MIME::Decoder::QuotedPrint v5.420;
RT::Interface::REST v1.00;
Scalar::Util v1.14; Scalar::Util v1.19;
Storable v2.13; Storable v2.16;
Sys::Hostname v1.11;
Sys::Syslog v0.08; Sys::Syslog v0.18;
Text::Tabs v98.112801; Text::Tabs v2007.1117;
Time::HiRes v1.55; Time::HiRes v1.9707;
Time::Local v1.1; Time::Local v1.17;

I will check now the Apache Error Log for some segfaults.

Torsten2007/11/19, Jesse Vincent jesse@bestpractical.com:

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html .

My guess is that you’re running into one of the very rare, but
catastrophic bugs in Text::Quoted. I alluded to this in my previous
email. You should see weird segfaults in the apache error log if this
is the case. Try 1) bringing it current or 2) removing it entirely.

Best,
Jesse


The rt-users Archives

SAVE THOUSANDS OF DOLLARS ON RT SUPPORT:

If you sign up for a new RT support contract before December 31, we’ll
take
up to 20 percent off the price. This sale won’t last long, so get in touch
today.
Email us at sales@bestpractical.com or call us at +1 617 812 0745.

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

MFG

Torsten Brumm

http://www.torsten-brumm.de

OK, i found some segfaults at the logs if i open the Ticket:

[Mon Nov 19 17:34:06 2007] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” (pid 3763) terminated due to uncaught
signal ‘11’ (Segmentation fault)
[Mon Nov 19 17:34:06 2007] [warn] FastCGI: server
“/opt/rt3/bin/mason_handler.fcgi” restarted (pid 3899)

Jesse, what do you think is the best way? Remove it totally or replace by
the newest???

Torsten2007/11/19, Jesse Vincent jesse@bestpractical.com:

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html.

My guess is that you’re running into one of the very rare, but
catastrophic bugs in Text::Quoted. I alluded to this in my previous
email. You should see weird segfaults in the apache error log if this
is the case. Try 1) bringing it current or 2) removing it entirely.

Best,
Jesse


The rt-users Archives

SAVE THOUSANDS OF DOLLARS ON RT SUPPORT:

If you sign up for a new RT support contract before December 31, we’ll
take
up to 20 percent off the price. This sale won’t last long, so get in touch
today.
Email us at sales@bestpractical.com or call us at +1 617 812 0745.

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

MFG

Torsten Brumm

http://www.torsten-brumm.de

OK, now it becomes very strange…!!!

I’ve upgraded to latest:

root@bruchtal-www4:~/Text-Quoted-2.03# perl Makefile.PL
Checking if your kit is complete…
Looks good
Writing Makefile for Text::Quoted
root@bruchtal-www4:~/Text-Quoted-2.03# make
cp Quoted.pm blib/lib/Text/Quoted.pm
Manifying blib/man3/Text::Quoted.3pm
root@bruchtal-www4:~/Text-Quoted-2.03# make test
PERL_DL_NONLAZY=1 /usr/bin/perl “-MExtUtils::Command::MM” “-e”
“test_harness(0, ‘inc’, ‘blib/lib’, ‘blib/arch’)” t/*.t
t/1…ok
t/2…ok
t/3…ok
t/4…ok
t/5…ok
t/6…ok
All tests successful.
Files=6, Tests=14, 1 wallclock secs ( 0.52 cusr + 0.08 csys = 0.60 CPU)
root@bruchtal-www4:~/Text-Quoted-2.03# make install
Installing /usr/lib/perl5/site_perl/5.8.5/Text/Quoted.pm
Installing /usr/share/man/man3/Text::Quoted.3pm
Writing
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/Text/Quoted/.packlist
Appending installation info to
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod
root@bruchtal-www4:~/Text-Quoted-2.03# cd /usr/lib/perl5/site_perl/5.8.5/
root@bruchtal-www4:/usr/lib/perl5/site_perl/5.8.5# ll

I removed the mason cache (to make sure everything is clean) restarted
apache and…

the module is not even loaded:

Sys::Syslog v0.18;
Text::Template v1.44;
Text::Wrapper v1.01;
Time::HiRes v1.9707;
Time::JulianDay v2003.1125;
Time::Local v1.17;
Time::ParseDate v2006.0814;
Time::Timezone v2006.0814;

So, remove and upgrade the Text::Quoted can’t be the solution?!?2007/11/19, Jesse Vincent jesse@bestpractical.com:

Recently weird things started to happen: random tickets in WebUI were
not displayed further than a certain transaction in the middle.

This happened both in Display.html and History.html.

My guess is that you’re running into one of the very rare, but
catastrophic bugs in Text::Quoted. I alluded to this in my previous
email. You should see weird segfaults in the apache error log if this
is the case. Try 1) bringing it current or 2) removing it entirely.

Best,
Jesse


The rt-users Archives

SAVE THOUSANDS OF DOLLARS ON RT SUPPORT:

If you sign up for a new RT support contract before December 31, we’ll
take
up to 20 percent off the price. This sale won’t last long, so get in touch
today.
Email us at sales@bestpractical.com or call us at +1 617 812 0745.

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

MFG

Torsten Brumm

http://www.torsten-brumm.de

Jesse, Torsten - thanks for digging into it.

The only Segmentation error I got in apache error log goes like this:

/usr/local/apache/logs/error_log:[Tue Nov 20 01:31:45 2007] [notice]
child pid 29555 exit signal Segmentation fault (11)

I got about 20 of these for last 3 days, but I guess that’s not the
case.

I’ve updated Text::Quoted to 2.03, but unlike Torsten’s setup, I got the
new version displayed correctly at the RT’s System Configuration page.

So, I guess, my next step would be to remove it? I’ve no faintest idea
how to
=)))))

As for the randomness of this bug (less than 0,1% tickets affected): may
it be the issue with encodings of the messages? As far as I see, all the
transactions that get “stuck” are the “Correspondence added” ones - both
created by external users and from RT’s WebUI.

Again, thanks for digging into this, guys!

KostyaFrom: Torsten Brumm [mailto:torsten.brumm@googlemail.com]
Sent: Monday, November 19, 2007 8:42 PM
To: Jesse Vincent
Cc: Konstantin Naryzhniy; rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Ticket Transactions get lost on the way to WebUI

OK, now it becomes very strange…!!!

I’ve upgraded to latest:

root@bruchtal-www4:~/Text-Quoted-2.03# perl Makefile.PL
Checking if your kit is complete…
Looks good
Writing Makefile for Text::Quoted
root@bruchtal-www4:~/Text-Quoted-2.03# make
cp Quoted.pm blib/lib/Text/Quoted.pm
Manifying blib/man3/Text::Quoted.3pm
root@bruchtal-www4:~/Text-Quoted-2.03# make test
PERL_DL_NONLAZY=1 /usr/bin/perl “-MExtUtils::Command::MM” “-e”
“test_harness(0, ‘inc’, ‘blib/lib’, ‘blib/arch’)” t/*.t
t/1…ok
t/2…ok
t/3…ok
t/4…ok
t/5…ok
t/6…ok
All tests successful.
Files=6, Tests=14, 1 wallclock secs ( 0.52 cusr + 0.08 csys = 0.60
CPU)
root@bruchtal-www4:~/Text-Quoted-2.03# make install
Installing /usr/lib/perl5/site_perl/5.8.5/Text/Quoted.pm
Installing /usr/share/man/man3/Text::Quoted.3pm
Writing
/usr/lib/perl5/site_perl/5.8.5/i386-linux-thread-multi/auto/Text/Quoted/
.packlist
Appending installation info to
/usr/lib/perl5/5.8.5/i386-linux-thread-multi/perllocal.pod
root@bruchtal-www4:~/Text-Quoted-2.03# cd
/usr/lib/perl5/site_perl/5.8.5/
root@bruchtal-www4:/usr/lib/perl5/site_perl/5.8.5# ll

I removed the mason cache (to make sure everything is clean) restarted
apache and…

the module is not even loaded:

Sys::Syslog v0.18;

Text::Template v1.44;

Text::Wrapper v1.01;

Time::HiRes v1.9707;

Time::JulianDay v2003.1125;

Time::Local v1.17;

Time::ParseDate v2006.0814;

Time::Timezone v2006.0814;

So, remove and upgrade the Text::Quoted can’t be the solution?!?

Jesse, Torsten - thanks for digging into it.

The only Segmentation error I got in apache error log goes like this:

/usr/local/apache/logs/error_log:[Tue Nov 20 01:31:45 2007] [notice]
child pid 29555 exit signal Segmentation fault (11)

I got about 20 of these for last 3 days, but I guess that’s not the
case.

I’ve updated Text::Quoted to 2.03, but unlike Torsten’s setup, I got the
new version displayed correctly at the RT’s System Configuration page.

So, I guess, my next step would be to remove it? I’ve no faintest idea
how to
=)))))

You need to find the Quoted.pm file inside a “Text/” directory somewhere
in your perl library path and remove it.

-j

You need to find the Quoted.pm file inside a “Text/” directory
somewhere in >your perl library path and remove it.

Hello Jesse,
Did that - apache won’t start with the error:

[Tue Nov 20 11:40:33 2007] [error] Can’t locate Text/Quoted.pm in @INC
(@INC …) at /work/opt/rt3/lib/RT/Interface/Web/Handler.pm line
56.\nBEGIN failed–compilation aborted at
/work/opt/rt3/lib/RT/Interface/Web/Handler.pm line 56.\nCompilation
failed in require at /work/opt/rt3/bin/webmux.pl line 87.\nBEGIN
failed–compilation aborted at /work/opt/rt3/bin/webmux.pl line
87.\nCompilation failed in require at (eval 8) line 1.\n
Syntax error on line 13 of /usr/local/apache/conf/rt.conf:

I reinstalled Text::Quoted from CPAN - same issue. Probably I’ve missed
something here?

Thanks again,
Kostya

Hello Jesse,
Did that - apache won’t start with the error:

What version of RT? 3.6.3 or so and newer should have it as an optional
dependency.

No, sir, I’m still on 3.6.1From: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: Tuesday, November 20, 2007 8:12 PM
To: Konstantin Naryzhniy
Cc: Jesse Vincent; Torsten Brumm; rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Ticket Transactions get lost on the way to WebUI

Hello Jesse,
Did that - apache won’t start with the error:

What version of RT? 3.6.3 or so and newer should have it as an optional
dependency.

[Tue Nov 20 11:40:33 2007] [error] Can’t locate Text/Quoted.pm in @INC
(@INC …) at /work/opt/rt3/lib/RT/Interface/Web/Handler.pm line
56.\nBEGIN failed–compilation aborted at
/work/opt/rt3/lib/RT/Interface/Web/Handler.pm line 56.\nCompilation
failed in require at /work/opt/rt3/bin/webmux.pl line 87.\nBEGIN
failed–compilation aborted at /work/opt/rt3/bin/webmux.pl line
87.\nCompilation failed in require at (eval 8) line 1.\n
Syntax error on line 13 of /usr/local/apache/conf/rt.conf:

I reinstalled Text::Quoted from CPAN - same issue. Probably I’ve
missed
something here?

Thanks again,
Kostya

No, sir, I’m still on 3.6.1

Ok. Coming up to more recent is recommended.

Hello Jesse!
You were right: upgrading from 3.6.1 to 3.6.5 and removing the Quoted.pm
module seems to have solved the issue of the ‘lost transactions’

Thank you and Torsten very much for the investigation and the solution!
Hope this thread will help anyone who encounters this rare bug.

Sincerely,
KostyaFrom: Jesse Vincent [mailto:jesse@bestpractical.com]
Sent: Tuesday, November 20, 2007 8:12 PM
To: Konstantin Naryzhniy
Cc: Jesse Vincent; Torsten Brumm; rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Ticket Transactions get lost on the way to WebUI

Hello Jesse,
Did that - apache won’t start with the error:

What version of RT? 3.6.3 or so and newer should have it as an optional
dependency.

[Tue Nov 20 11:40:33 2007] [error] Can’t locate Text/Quoted.pm in @INC
(@INC …) at /work/opt/rt3/lib/RT/Interface/Web/Handler.pm line
56.\nBEGIN failed–compilation aborted at
/work/opt/rt3/lib/RT/Interface/Web/Handler.pm line 56.\nCompilation
failed in require at /work/opt/rt3/bin/webmux.pl line 87.\nBEGIN
failed–compilation aborted at /work/opt/rt3/bin/webmux.pl line
87.\nCompilation failed in require at (eval 8) line 1.\n
Syntax error on line 13 of /usr/local/apache/conf/rt.conf:

I reinstalled Text::Quoted from CPAN - same issue. Probably I’ve
missed
something here?

Thanks again,
Kostya