Full text indexer issue with img html tag

Peter_Viskup · March 30, 2016, 3:41pm

Hello all,
we are experiencing issue with full text indexer which seems to be
related to HTML email with tag “”
(28398 characters long) which seems to cause rt-fulltext-indexer fail
to update full text index.

We do see following debug messages:
Found attachment #3085
[834] [Wed Mar 30 12:33:32 2016] [warning]: NOTICE: word is too long
to be indexed
DETAIL: Words longer than 2047 characters are ignored.
(/opt/rt4/sbin/rt-fulltext-indexer:322)
[834] [Wed Mar 30 12:33:32 2016] [warning]: NOTICE: word is too long
to be indexed
DETAIL: Words longer than 2047 characters are ignored.
(/opt/rt4/sbin/rt-fulltext-indexer:322)
Processed attachment #3085

Anyone with another/better solution than removal of the affected ticket?

Peter Viskup

chmrr · March 31, 2016, 4:32am

we are experiencing issue with full text indexer

What version of RT? That’s a symptom of RT 4.0.9 or earlier; gradi4.0.10
contains a fix that makes RT simply skip that attachment.

Alex

chmrr · March 31, 2016, 4:49am

What version of RT? That’s a symptom of RT 4.0.9 or earlier; 4.0.10
contains a fix that makes RT simply skip that attachment.

Actually, my mistake – it was fixed in 4.0.3:

github.com/bestpractical/rt

In Postgres, simply skip attachments whose tsvectors are too large

committed 04:41AM - 20 Sep 11 UTC

alexmv

+29 -3

While PostgreSQL's tsvector format has a number of limitations [1], one of them …is particularly important: * The length of a tsvector (lexemes + positions) must be less than 1 megabyte This causes an error which causes the statement to fail, and thus rt-fulltext-indexer stops there. Detect that particular failure case, and allow indexing to proceed, simply skipping that attachment. The rest of the limitations, and the reasons they do not need to be dealt with similarly, are: * The length of each lexeme must be less than 2K bytes This produces a warning from Postgres, but the statement completes. * The number of lexemes must be less than 2^64 This limit is hit far after the 1 megabyte limitation above, and thus essentially can never be triggered. * Position values in tsvector must be greater than 0 and no more than 16,383 If a lexeme is found later than position 16,383, it is simply capped at being position 16,383. This causes information loss, but as RT does not present the position of the match in any form, this does not matter. * No more than 256 positions per lexeme Only the first 256 locations are stored; later occurrences are ignored. Again, as RT does not present the locations of the matches, this information loss is irrelevant to the application. [1] http://www.postgresql.org/docs/9.1/static/textsearch-limitations.html

If you’re running pre-4.0.3, you certainly want to upgrade.

Alex

Peter_Viskup · March 31, 2016, 7:47am

Fortunately our version is 4.2.10.
Just discovered we didn’t ran rt-fulltext-indexer enough many times.
It proceeded with only 200 attachments at once. Is that some
limitation of PgSQL or rt-fulltext-indexer? Didn’t read that in the
documentation.

I ran it in a loop, till it didn’t report any other indexed
attachments. This solved our issue with full text search.
Thank you.

Peter ViskupOn Thu, Mar 31, 2016 at 6:49 AM, Alex Vandiver alex@chmrr.net wrote:

On Wed, 30 Mar 2016 21:32:37 -0700 Alex Vandiver alex@chmrr.net wrote:

What version of RT? That’s a symptom of RT 4.0.9 or earlier; 4.0.10
contains a fix that makes RT simply skip that attachment.

Actually, my mistake – it was fixed in 4.0.3:
In Postgres, simply skip attachments whose tsvectors are too large · bestpractical/rt@692b5bc · GitHub

If you’re running pre-4.0.3, you certainly want to upgrade.

Alex

RT 4.4 and RTIR Training Sessions https://bestpractical.com/training

Washington DC - May 23 & 24, 2016

chmrr · March 31, 2016, 9:28am

Just discovered we didn’t ran rt-fulltext-indexer enough many times.
It proceeded with only 200 attachments at once. Is that some
limitation of PgSQL or rt-fulltext-indexer? Didn’t read that in the
documentation.

See
https://docs.bestpractical.com/rt/4.4.0/full_text_indexing.html#Updating-the-index

You can pass --all to index all attachments, which looks to not be
documented.

Alex