Performance problems with 3.8.1

Sahlberg_Mauri · September 17, 2008, 10:13am

Hi,

Just upgraded from 3.6 to 3.8.1 (via 3.8) and at the same time moved the database to it’s own server. We also removed all closed tickets from the database. The move and upgrade was done as our old installation got too slow. Unfortunately the new installation is also very slow despite the ticket removal and dedicated database server. Especially search building view and ticket view take time to open.

We have:

     Checked that weI have newest DBIx:search builder installed

     Checked that both our database server (CentOS 5.2 final) and web server (CentOS 5.2 final)  fulfill the minimum shared memory settings suggested at the wiki

     Have FastCGI installed on Apache 2.2.3, standard CentOS 5.2 final binary (standard CentOS 5.2 perl 5.8.8 with make fixdeps installed modules)

     Our Postgresql 8.1.11 (Centos 5.2 standard binary) has more shared_buffers and temp_buffers than suggested at the wiki

     Installed pg_top and tried to isolate the query that hogs the machine

Any suggestions what we could yet do to speed things up?

Sven_Sternberger · September 17, 2008, 11:01am

try if it is faster with MS IE, firefox has problems
with the new CSS

regards

svenOn Mi, 2008-09-17 at 13:13 +0300, Sahlberg Mauri wrote:

Hi,

Just upgraded from 3.6 to 3.8.1 (via 3.8) and at the same time moved
the database to it’s own server. We also removed all closed tickets
from the database. The move and upgrade was done as our old
installation got too slow. Unfortunately the new installation is also
very slow despite the ticket removal and dedicated database server.
Especially search building view and ticket view take time to open.

We have:
    Checked that weI have newest DBIx:search builder installed
    Checked that both our database server (CentOS 5.2 final) and
web server (CentOS 5.2 final) fulfill the minimum shared memory
settings suggested at the wiki
    Have FastCGI installed on Apache 2.2.3, standard CentOS 5.2
final binary (standard CentOS 5.2 perl 5.8.8 with make fixdeps
installed modules)
    Our Postgresql 8.1.11 (Centos 5.2 standard binary) has more
shared_buffers and temp_buffers than suggested at the wiki
    Installed pg_top and tried to isolate the query that hogs
the machine

Any suggestions what we could yet do to speed things up?

The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

Curtis_Bruneau · September 17, 2008, 1:57pm

For FastCGI, I changed my configuration to allow multiple processes so
other requests aren’t getting queued up when big queries are hitting the
server. Without this setting the site was pretty slow to respond but
once the request got in it was fast. I’ve also changed the default idle
timeout to something similar to mod_perl2, the default was triggering on
normal stuff, some un-indexed fields in search were causing really slow
count()s…

FastCgiServer /opt/rt3/bin/mason_handler.fcgi -idle-timeout 300 -processes 4

Also the recommended shared buffers would be tailored to your instance,
in our case we have nearly 1GB in indexes alone so ours is set fairly
high. Generally you want it high enough so it doesn’t have to swap out
indexes all the time.

Curtis

Sahlberg Mauri wrote:

Kenneth_Marshall · September 17, 2008, 2:19pm

Hi,

Just upgraded from 3.6 to 3.8.1 (via 3.8) and at the same time moved the database to it’s own server. We also removed all closed tickets from the database. The move and upgrade was done as our old installation got too slow. Unfortunately the new installation is also very slow despite the ticket removal and dedicated database server. Especially search building view and ticket view take time to open.

We have:
     Checked that weI have newest DBIx:search builder installed
     Checked that both our database server (CentOS 5.2 final) and web server (CentOS 5.2 final)  fulfill the minimum shared memory settings suggested at the wiki
     Have FastCGI installed on Apache 2.2.3, standard CentOS 5.2 final binary (standard CentOS 5.2 perl 5.8.8 with make fixdeps installed modules)
     Our Postgresql 8.1.11 (Centos 5.2 standard binary) has more shared_buffers and temp_buffers than suggested at the wiki
     Installed pg_top and tried to isolate the query that hogs the machine
Any suggestions what we could yet do to speed things up?

I just checked the latest DBIx::SearchBuilder and the fix that I
thought had been incorporated into the Handle/Pg.pm has not as of
yet. The problem line is line number 245:

$$statementref = “SELECT DISTINCT main.* FROM $$statementref”;

If you replace it with a line like the one in the Oracle.pm at
line number 279:

$$statementref = "SELECT main.* FROM ( SELECT DISTINCT main.id FROM $$statementref ) distinctquery, $table main WHERE (main.id = distinctquery.id) ";

you should see a nice performance boost on the page loads. I have sent
this change in before, but it has not been adopted even though it is
being used in the Oracle.pm file. I examined the query used by mysql.pm
for this query, and the Oracle.pm version of the line duplicates the
query used by mysql.pm exactly. The query as currently written spends
a lot of sorting fields that are not used. The alternate query is
much faster. Maybe BP can re-examine this suggestion again.

Cheers,
Ken

Jesse_Vincent · September 17, 2008, 4:16pm

Hi,

Just upgraded from 3.6 to 3.8.1 (via 3.8) and at the same time moved
the database to it’s own server. We also removed all closed tickets
from the database. The move and upgrade was done as our old
installation got too slow. Unfortunately the new installation is
also very slow despite the ticket removal and dedicated database
server. Especially search building view and ticket view take time to
open.

So, what is slow and how slow is it?

Also, what can you tell us about your hardware specs?

I’ve worked with users for whom “slow” was “1.2s to load RT’s
homepage” and with users for whom “slow” was “10 minutes to load a
ticket page. It used to be five and that was ok. but 10 is too long.”

Jesse_Vincent · September 17, 2008, 4:20pm

try if it is faster with MS IE, firefox has problems
with the new CSS

As far as we can figure it out, the issue you’re referring to is
pretty rare. A very few Linux+Firefox+X Server combinations run into
trouble with the rounded corners. Any brilliant discoveries would be
hugely appreciated.

-j

Hossein_Rafighi · September 18, 2008, 5:34pm

Hi All,

We have been using RT in a class B network (142.90.0.0) for few years. I
just realized when users from different IP addresses inside our network
log into RT’s web interface the log file records ALL connections from
98.240.133.0! The exact message is: "RT: Successful login for USER_NAME
from 98.240.133.0 (/opt/rt3/share/html/autohandler:258). Traceroute of
this IP takes me to Comcast Cable Communications, Inc. MINNESOTA, USA.

Has anybody else seen this behavior? What do you see in your log files?

Cheers – and tanks,
Hossein

_____ _____ _____ _ _ _ _ ____ Hossein Rafighi
|_ || _ \ | || | | || _/ || __|TRIUMF, 4004 Wesbrook Mall
| | | || ) | | | | | || || |__ Vancouver BC, Canada, V6T 2A3
| | | _ / | | | _/ || _/ || |Voice: (604) 222-1047
| | | | \ \ | | | || | | || | Fax: (604) 222-1074
|| || _|_| _/ || |||_| Website: http://www.triumf.ca

Sahlberg_Mauri · September 19, 2008, 7:34am

Hi,

Thank you all for the advice.

I asked the same question little differently phrased on postgresql-admin list and got several good answers from there as well. As result I implemented following changes:

     Added 5 more processes to fastcgi, now we have 10 instead of 5 (thanks Curtis)

     Changed search builder as Kenneth advised, thanks

     Updated database server to version 8.4 (yeah, I know it is a development version and should not used for production purposes)

     Heavily tuned the postgresql. The performance tuning advice on RT wiki is rather outdated but on Postgresql manual at http://www.postgresql.org/docs/8.3/interactive/runtime-config-resource.html is one very helpful comment (by Steven Citron-Pousty) on the subject.

The first user comment from this morning, roughly translated: “The RT now runs nearly with speed of light! Thank you”.

The parameters for our 4 gig server are now:
shared_buffers=320MB
maintenance_work_mem=384MB
effective_cache_size = 2048MB
wal_buffres=8MB

The work_mem is the “default” 1MB but I got recommendation (from Scott Marlowe) to change that to 4-8MB which I will do on the next maintenance. I did not change the max_fsm or checkpoint_segments so there is more room for improvement.

Thank you.

Lähettäjä: Jesse Vincent [mailto:jesse@bestpractical.com]
Lähetetty: 17. syyskuuta 2008 19:17
Vastaanottaja: Sahlberg Mauri
Kopio: rt-users@lists.bestpractical.com
Aihe: Re: [rt-users] Performance problems with 3.8.1On Sep 17, 2008, at 6:13 AM, Sahlberg Mauri wrote:

Hi,

Just upgraded from 3.6 to 3.8.1 (via 3.8) and at the same time moved the database to it’s own server. We also removed all closed tickets from the database. The move and upgrade was done as our old installation got too slow. Unfortunately the new installation is also very slow despite the ticket removal and dedicated database server. Especially search building view and ticket view take time to open.

So, what is slow and how slow is it?

Also, what can you tell us about your hardware specs?

I’ve worked with users for whom “slow” was “1.2s to load RT’s homepage” and with users for whom “slow” was “10 minutes to load a ticket page. It used to be five and that was ok. but 10 is too long.”