Docs improvement suggestion for full-text searching

So the documentation at https://www.bestpractical.com/docs/rt/4.2/UPGRADING-4.0.html says:

Since 4.0.0, RT’s ticket content search is disabled by default because of performance issues when used without full text indexing. For details on how to re-enable it with (or without) full text indexing, see docs/full_text_indexing.pod.

However this URL https://www.bestpractical.com/docs/rt/4.2/full_text_indexing.html contains only information on how to enable full text indexing, and no information on how to enable it without the indexing.

Seems like you need to add the information from http://blog.bestpractical.com/2011/06/full-text-searching.html to the top of this page.

Jo Rhett
+1 (415) 999-1798
Skype: jorhett
Net Consonance : net philanthropy to improve open source and internet projects.

However [doc/full_text_indexing] contains only information on how to
enable full text indexing, and no information on how to enable it
without the indexing.

Mostly because we’re afraid of users shooting themselves in the foot.
Regardless, I’ve committed 2551dca5 which details how to do so.

  • Alex

However [doc/full_text_indexing] contains only information on how to
enable full text indexing, and no information on how to enable it
without the indexing.

Mostly because we’re afraid of users shooting themselves in the foot.
Regardless, I’ve committed 2551dca5 which details how to do so.

  • Alex

Understood. Just a clarity nitpick :slight_smile: Although I am confused by your statements that full text indexing isn’t available in MySQL. I’ve used this quite successfully in the past http://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html

Likewise, MariaDB has Sphinx support compiled in and available in their packages, which made this process almost trivial to enable.
https://mariadb.com/kb/en/mariadb/documentation/storage-engines/sphinx-storage-engine/about-sphinxse/

I found three problems in the config pushed out by the sbin/rt-setup-fulltext-index command:

  1. You need to create and chown the var/sphinx directory it references
    mkdir /opt/rt4/var/sphinx
    chown -R sphinx:sphinx /opt/rt4/var/sphinx

  2. You need to add this binlog path line
    binlog_path = /opt/rt4/var/sphinx

  3. You need to add this compat line or searchd won’t start
    compat_sphinxql_magics = 0

Easy fixes…

Jo Rhett
+1 (415) 999-1798
Skype: jorhett
Net Consonance : net philanthropy to improve open source and internet projects.

Understood. Just a clarity nitpick :slight_smile: Although I am confused by your
statements that full text indexing isn�t available in MySQL. I�ve used
this quite successfully in the past
http://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html

At the time the documentation was written, MySQL only had FTS support
using MyISAM, not InnoDB. Commits 84066c4 and 77641fc on the unmerged
4.2/mysql-native-fts branch clarify the topic, as well as implement
native FTS support.

Likewise, MariaDB has Sphinx support compiled in and available in
their packages, which made this process almost trivial to enable.
https://mariadb.com/kb/en/mariadb/documentation/storage-engines/sphinx-storage-engine/about-sphinxse/

RT doesn’t officially support MariaDB – though this is almost purely
because of lack of documentation and testing infrastucture, not because
it is known to have failure modes. When MariaDB support is added, the
Sphinx integration will be noted.

I found three problems in the config pushed out by the sbin/rt-setup-fulltext-index command:

  1. You need to create and chown the var/sphinx directory it references
    mkdir /opt/rt4/var/sphinx
    chown -R sphinx:sphinx /opt/rt4/var/sphinx

Noted in

  1. You need to add this binlog path line
    binlog_path = /opt/rt4/var/sphinx

  2. You need to add this compat line or searchd won�t start
    compat_sphinxql_magics = 0

What version of sphinx? Neither of those are necessary with 2.0.

  • Alex

version 2.0.8-1 rpm package for EL6On Dec 1, 2014, at 9:21 AM, Alex Vandiver alexmv@bestpractical.com wrote:

On 11/28/2014 03:12 AM, Jo Rhett wrote:

Understood. Just a clarity nitpick :slight_smile: Although I am confused by your
statements that full text indexing isn’t available in MySQL. I’ve used
this quite successfully in the past
http://dev.mysql.com/doc/refman/5.6/en/innodb-fulltext-index.html

At the time the documentation was written, MySQL only had FTS support
using MyISAM, not InnoDB. Commits 84066c4 and 77641fc on the unmerged
4.2/mysql-native-fts branch clarify the topic, as well as implement
native FTS support.

Likewise, MariaDB has Sphinx support compiled in and available in
their packages, which made this process almost trivial to enable.
https://mariadb.com/kb/en/mariadb/documentation/storage-engines/sphinx-storage-engine/about-sphinxse/

RT doesn’t officially support MariaDB – though this is almost purely
because of lack of documentation and testing infrastucture, not because
it is known to have failure modes. When MariaDB support is added, the
Sphinx integration will be noted.

I found three problems in the config pushed out by the sbin/rt-setup-fulltext-index command:

  1. You need to create and chown the var/sphinx directory it references
    mkdir /opt/rt4/var/sphinx
    chown -R sphinx:sphinx /opt/rt4/var/sphinx

Noted in

  1. You need to add this binlog path line
    binlog_path = /opt/rt4/var/sphinx

  2. You need to add this compat line or searchd won’t start
    compat_sphinxql_magics = 0

What version of sphinx? Neither of those are necessary with 2.0.

  • Alex

Jo Rhett
+1 (415) 999-1798
Skype: jorhett
Net Consonance : net philanthropy to improve open source and internet projects.

version 2.0.8-1 rpm package for EL6

compat_sphinxql_magics was added in 2.0.1-beta, defaults to 0 in
2.1.1-beta, and was removed in 2.2.1-beta. I’m hesitant to add
something to the documented configuration which will cause sphinx to
fail on all other versions. Can you explain the failure mode of not
having it more clearly?

binlog_path was added in 1.10 – I’ve added it, with a comment to the
versions it’s pertinent to.

  • Alex

Sphinx refuses to run without that parameter. Which given that it wasn’t defined in the file they really should have set the default appropriately and not whined at the user, but this is the version of Sphinx currently in RHEL EPEL so there’s going to be a lot of RHEL/CentOS users running into this problem.On Dec 5, 2014, at 8:07 AM, Alex Vandiver alexmv@bestpractical.com wrote:

On 12/01/2014 04:11 PM, Jo Rhett wrote:

version 2.0.8-1 rpm package for EL6

compat_sphinxql_magics was added in 2.0.1-beta, defaults to 0 in
2.1.1-beta, and was removed in 2.2.1-beta. I’m hesitant to add
something to the documented configuration which will cause sphinx to
fail on all other versions. Can you explain the failure mode of not
having it more clearly?

binlog_path was added in 1.10 – I’ve added it, with a comment to the
versions it’s pertinent to.

  • Alex

Jo Rhett
+1 (415) 999-1798
Skype: jorhett
Net Consonance : net philanthropy to improve open source and internet projects.

Sphinx refuses to run without that parameter. Which given that it
wasn�t defined in the file they really should have set the default
appropriately and not whined at the user, but this is the version of
Sphinx currently in RHEL EPEL so there�s going to be a lot of
RHEL/CentOS users running into this problem.

I can’t replicate the compat_sphinxql_magics problems you report with a
stock Sphinx 2.0.8 from EPEL on CentOS 6. With a stock configuration as
provided by 4.2-trunk, indexer runs with no errors (see below). If it
refuses to run, please show your configuration file, the sphinx version,
and the actual error when running without compat_sphinxql_magics.

  • Alex

-bash-4.1# rm /opt/rt4/var/sphinx/*
-bash-4.1# indexer --config tmp.conf rt
Sphinx 2.0.8-id64-release (r3831)
Copyright © 2001-2012, Andrew Aksyonoff
Copyright © 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file ‘tmp.conf’…
indexing index ‘rt’…
WARNING: Attribute count is 0: switching to none docinfo
collected 1 docs, 0.0 MB
sorted 0.0 Mhits, 100.0% done
total 1 docs, 626 bytes
total 0.009 sec, 64073 bytes/sec, 102.35 docs/sec
total 2 reads, 0.000 sec, 0.3 kb/call avg, 0.0 msec/call avg
total 6 writes, 0.000 sec, 0.3 kb/call avg, 0.0 msec/call avg

-bash-4.1# cat tmp.conf
source rt {
type = mysql

sql_host        = 127.0.0.1
sql_db          = rt4
sql_user        = root
sql_pass        =

sql_query_pre   = SET NAMES utf8
sql_query       = \
    SELECT a.id, a.content FROM Attachments a \
    JOIN Transactions txn ON a.TransactionId = txn.id AND

txn.ObjectType = ‘RT::Ticket’
JOIN Tickets t ON txn.ObjectId = t.id
WHERE a.ContentType = ‘text/plain’ AND t.Status != ‘deleted’

sql_query_info  = SELECT * FROM Attachments WHERE id=$id

}

index rt {
source = rt
path = /opt/rt4/var/sphinx/index
docinfo = extern
charset_type = utf-8
}

indexer {
mem_limit = 32M
}

searchd {
port = 3312
log = /opt/rt4/var/sphinx/searchd.log
query_log = /opt/rt4/var/sphinx/query.log
read_timeout = 5
max_children = 30
pid_file = /opt/rt4/var/sphinx/searchd.pid
max_matches = 1000
seamless_rotate = 1
preopen_indexes = 0
unlink_old = 1
}

Sphinx refuses to run without that parameter. Which given that it
wasn’t defined in the file they really should have set the default
appropriately and not whined at the user, but this is the version of
Sphinx currently in RHEL EPEL so there’s going to be a lot of
RHEL/CentOS users running into this problem.

I can’t replicate the compat_sphinxql_magics problems you report with a
stock Sphinx 2.0.8 from EPEL on CentOS 6. With a stock configuration as
provided by 4.2-trunk, indexer runs with no errors (see below). If it

RT’s use of sphinx requires the daemon to answer, which you didn’t start. With your example file when I start the daemon I get this error:

service searchd start

Starting searchd: Sphinx 2.0.8-id64-release (r3831)
Copyright © 2001-2012, Andrew Aksyonoff
Copyright © 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file ‘/etc/sphinx/sphinx.conf’…
WARNING: compat_sphinxql_magics=1 is deprecated; please update your application and config

Note that your file doesn’t contain magics=1. So I totally agree that it’s an annoying bug that the developers should fix, but it’s also something you can avoid by putting it in the file you output.

Jo Rhett
+1 (415) 999-1798
Skype: jorhett
Net Consonance : net philanthropy to improve open source and internet projects.

RT’s use of sphinx requires the daemon to answer, which you didn�t
start. With your example file when I start the daemon I get this
error:

service searchd start Starting searchd: Sphinx 2.0.8-id64-release

(r3831) Copyright © 2001-2012, Andrew Aksyonoff Copyright ©
2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file ‘/etc/sphinx/sphinx.conf’… WARNING:
compat_sphinxql_magics=1 is deprecated; please update your
application and config

As the output notes, that is merely a warning. It is not an error.
searchd starts up just fine for me:

service searchd start

Starting searchd: Sphinx 2.0.8-id64-release (r3831)
Copyright © 2001-2012, Andrew Aksyonoff
Copyright © 2008-2012, Sphinx Technologies Inc (http://sphinxsearch.com)

using config file ‘/etc/sphinx/sphinx.conf’…
WARNING: compat_sphinxql_magics=1 is deprecated; please update your
application and config
listening on all interfaces, port=3312
precaching index 'rt’
precached 1 indexes in 0.001 sec
[ OK ]

service searchd status

searchd (pid 15041) is running…

Note that your file doesn�t contain magics=1. So I totally agree that
it�s an annoying bug that the developers should fix, but it�s also
something you can avoid by putting it in the file you output.

Since it starts up fine without the line, and would fail to start on
Sphinx < 2 or > 2.2 with the line, I’m unconvinced on adding it to the
default configuration. Administrators who are running Sphinx 2.0.x can
simply heed the warning and add compat_sphinxql_magics=0 to their
configuration.

  • Alex