We presently have our RT installation running on the same hardware as the database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB RAM. We’ve been plagued with speed issues even after upgrading to this. We realized initial performance gains when we installed to the new hardware about two and a half years ago but eventually that faded.
Our engineering director now is suggesting another tech refresh which would add dedicated hardware for the database leaving the frontend on the existing hardware. I’m skeptical that this will improve anything due to RT’s small footprint and what I perceive as inherent obstacles to performance.
My skepticism is based on the fact that the some of the ways we use RT cause ticket load times to be slow regardless of any changes (I wish I could convince one person in particular that RT isn’t meant to be a document versioning repository but he’s quite retarded at times). Additionally, the page refresh for every click doesn’t help. When a ticket has hundreds of transactions it has to gather them up before the Mason libraries even build the page. Add to that often numerous attachments and things get even worse.
Has anyone else found dedicated hardware to be a significant factor in boosting performance? How powerful did you make it? I’m still evaluating the mysqltuner.pl script Ruslan suggested in another thread I created so I’ve yet to see what improvements can be made on the software side.
Keep up with my goings on at The Life and Times of a Slacker
What type of RAID system are you using and how fast are the disks?
James Moseley
Mathew
<mathew.snyder@gm
ail.com> To
Sent by: RT Users
rt-users-bounces@ <rt-users@bestpractical.com>
lists.bestpractic cc
al.com
Subject
[rt-users] Hardware Config
01/16/2009 01:37
PM
We presently have our RT installation running on the same hardware as the
database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB RAM.
We’ve been plagued with speed issues even after upgrading to this. We
realized initial performance gains when we installed to the new hardware
about two and a half years ago but eventually that faded.
Our engineering director now is suggesting another tech refresh which would
add dedicated hardware for the database leaving the frontend on the
existing hardware. I’m skeptical that this will improve anything due to
RT’s small footprint and what I perceive as inherent obstacles to
performance.
My skepticism is based on the fact that the some of the ways we use RT
cause ticket load times to be slow regardless of any changes (I wish I
could convince one person in particular that RT isn’t meant to be a
document versioning repository but he’s quite retarded at times).
Additionally, the page refresh for every click doesn’t help. When a ticket
has hundreds of transactions it has to gather them up before the Mason
libraries even build the page. Add to that often numerous attachments and
things get even worse.
Has anyone else found dedicated hardware to be a significant factor in
boosting performance? How powerful did you make it? I’m still evaluating
the mysqltuner.pl script Ruslan suggested in another thread I created so
I’ve yet to see what improvements can be made on the software side.
Hardware is RAID 5 on LSI card. I’m working on getting the disk specs.
jmoseley@corp.xanadoo.com wrote:
What type of RAID system are you using and how fast are the disks?
James Moseley
Mathew
<mathew.snyder@gm
ail.com> To
Sent by: RT Users
rt-users-bounces@ <rt-users@bestpractical.com>
lists.bestpractic cc
al.com
Subject
[rt-users] Hardware Config
01/16/2009 01:37
PM
We presently have our RT installation running on the same hardware as the
database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB RAM.
We’ve been plagued with speed issues even after upgrading to this. We
realized initial performance gains when we installed to the new hardware
about two and a half years ago but eventually that faded.
Our engineering director now is suggesting another tech refresh which would
add dedicated hardware for the database leaving the frontend on the
existing hardware. I’m skeptical that this will improve anything due to
RT’s small footprint and what I perceive as inherent obstacles to
performance.
My skepticism is based on the fact that the some of the ways we use RT
cause ticket load times to be slow regardless of any changes (I wish I
could convince one person in particular that RT isn’t meant to be a
document versioning repository but he’s quite retarded at times).
Additionally, the page refresh for every click doesn’t help. When a ticket
has hundreds of transactions it has to gather them up before the Mason
libraries even build the page. Add to that often numerous attachments and
things get even worse.
Has anyone else found dedicated hardware to be a significant factor in
boosting performance? How powerful did you make it? I’m still evaluating
the mysqltuner.pl script Ruslan suggested in another thread I created so
I’ve yet to see what improvements can be made on the software side.
Keep up with my goings on at The Life and Times of a Slacker
We presently have our RT installation running on the same hardware as the database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB RAM. We’ve been plagued with speed issues even after upgrading to this. We realized initial performance gains when we installed to the new hardware about two and a half years ago but eventually that faded.
Can you tell us about how you’ve tuned mysql, how many concurrent users
you have working with RT, how many tickets you have in RT, etc?
No matter how beefy a server you’ve got, if you don’t spend some time
tuning mysql, it will assume it’s running on a single-core Pentium 133
with 128 megs of RAM which is also your primary mail server. And the
performance you see will reflect that.
I would strongly recommend that you invest some time in profiling and
tuning before just throwing a bigger box at your problem. RT runs just
great for many of our clients on hardware much more modest than what
you’ve got already.
We presently have our RT installation running on the same hardware as
the database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB
RAM. We’ve been plagued with speed issues even after upgrading to this.
We realized initial performance gains when we installed to the new
hardware about two and a half years ago but eventually that faded.
Can you tell us about how you’ve tuned mysql, how many concurrent users
you have working with RT, how many tickets you have in RT, etc?
No matter how beefy a server you’ve got, if you don’t spend some time
tuning mysql, it will assume it’s running on a single-core Pentium 133
with 128 megs of RAM which is also your primary mail server. And the
performance you see will reflect that.
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
James
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
-j
I absolutely agree. I’ve already told him that new hardware isn’t going to make a difference even before asking about it here. However, in order to be able to cover my bases in proving him wrong (admittedly a task I chomp at the bit for) I decided to ask about it here.
I don’t have direct access to the my.cnf file as I’m only a consultant these days but once I’m able to get that I’ll give some more info.
As also mentioned, I still need to take a look at the tuning scrip Ruslan pointed me to.
Jesse Vincent wrote:> On Fri, Jan 16, 2009 at 02:37:30PM -0500, Mathew wrote:
We presently have our RT installation running on the same hardware as the database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB RAM. We’ve been plagued with speed issues even after upgrading to this. We realized initial performance gains when we installed to the new hardware about two and a half years ago but eventually that faded.
Can you tell us about how you’ve tuned mysql, how many concurrent users
you have working with RT, how many tickets you have in RT, etc?
No matter how beefy a server you’ve got, if you don’t spend some time
tuning mysql, it will assume it’s running on a single-core Pentium 133
with 128 megs of RAM which is also your primary mail server. And the
performance you see will reflect that.
I would strongly recommend that you invest some time in profiling and
tuning before just throwing a bigger box at your problem. RT runs just
great for many of our clients on hardware much more modest than what
you’ve got already.
Keep up with my goings on at The Life and Times of a Slacker
I don’t have the exact specs but I know they are SCSI and likely 10k. I don’t know the seek times though.
jmoseley@corp.xanadoo.com wrote:
We presently have our RT installation running on the same hardware as
the database: an Intel 1550 box with 4 cores of about 2.5GHz each and 8GB
RAM. We’ve been plagued with speed issues even after upgrading to this.
We realized initial performance gains when we installed to the new
hardware about two and a half years ago but eventually that faded.
Can you tell us about how you’ve tuned mysql, how many concurrent users
you have working with RT, how many tickets you have in RT, etc?
No matter how beefy a server you’ve got, if you don’t spend some time
tuning mysql, it will assume it’s running on a single-core Pentium 133
with 128 megs of RAM which is also your primary mail server. And the
performance you see will reflect that.
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
James
Keep up with my goings on at The Life and Times of a Slacker
Can I sob quietly into a hard drink just for the sake of having the drink?
Jesse Vincent wrote:
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
-j
Keep up with my goings on at The Life and Times of a Slacker
I don’t have direct access to the my.cnf file as I’m only a consultant these days but once I’m able to get that I’ll give some more info.
As also mentioned, I still need to take a look at the tuning scrip Ruslan pointed me to.
Start with that script. It just queries the database. It doesn’t make any
changes. But really, if you haven’t done that yet, further discussion
probably doesn’t make much sense.
Jesse Vincent wrote:
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
-j
Yeah agreed, It should rarely go to disc. The iowait on my server is
very low. If it is there’s either missing indexes or not enough memory
pool for innodb to keep it cached, you can see that with the hit rate.
He can adjust that with the ‘innodb_buffer_pool_size’ , I’ve also set
‘innodb_flush_log_at_trx_commit’ to 0 which isn’t as safe but it makes
writes really fast. There are other settings also.
The one area that is prone to issues is the Search due to all the fields
it can search and a lot of them aren’t indexed so it’s doing a lot of
row scans.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
True, but the database server might have many other busy databases as well,
not just RT’s 
I’m definitely not an expert on how mysql utilizes system memory, but on
32-bit Linux systems, isn’t the max amount of memory a single process can
use 2 GB? Can it even take advantage of the extra 6 GB?
James
I’m definitely not an expert on how mysql utilizes system memory, but on
32-bit Linux systems, isn’t the max amount of memory a single process can
use 2 GB?
There are (fairly standard) patches to the kernel to remove that limit.
And there are known issues with mysql on 32 bit linux and using more
than 2 gigs of ram. But Mathew also didn’t say they’d gone and installed
32-bit Linux (which can’t properly take advantage of their hardware)
instead of a proper 64-bit linux distribution.
There are certainly plenty of issues one should be aware of here. Then
again, if everyone was fully up to speed on all these issues, I might be
out a job
Best,
Jesse
Curtis Bruneau wrote:
Jesse Vincent wrote:
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
-j
Yeah agreed, It should rarely go to disc. The iowait on my server is
very low. If it is there’s either missing indexes or not enough memory
pool for innodb to keep it cached, you can see that with the hit rate.
He can adjust that with the ‘innodb_buffer_pool_size’ , I’ve also set
‘innodb_flush_log_at_trx_commit’ to 0 which isn’t as safe but it makes
writes really fast. There are other settings also.
The one area that is prone to issues is the Search due to all the fields
it can search and a lot of them aren’t indexed so it’s doing a lot of
row scans.
Writes aren’t an issue. It doesn’t take long to write one transaction compared to reading every transaction on a ticket before displaying it. I still need to take a look-see at the config file to see what I’ve got going on there before I can say it isn’t the buffer pool size.
Additionally, we do have about six or seven custom fields which wouldn’t be indexed.
Keep up with my goings on at The Life and Times of a Slacker
Fair enough.
James Moseley
Jesse Vincent
<jesse@bestpracti
cal.com>
I’m definitely not an expert on how mysql utilizes system memory, but on
32-bit Linux systems, isn’t the max amount of memory a single process can
use 2 GB?
There are (fairly standard) patches to the kernel to remove that limit.
And there are known issues with mysql on 32 bit linux and using more
than 2 gigs of ram. But Mathew also didn’t say they’d gone and installed
32-bit Linux (which can’t properly take advantage of their hardware)
instead of a proper 64-bit linux distribution.
There are certainly plenty of issues one should be aware of here. Then
again, if everyone was fully up to speed on all these issues, I might be
out a job
Best,
Jesse
I was finally able to log into the system I’m consulting on. Using all of this discussion as a jumping off point to figure out other steps to take I’ve found a couple things which are clear problems. There are only 4GB RAM as opposed to the 8GB which I thought it had. This would be moot anyway as the system is 32-bit and the PAE kernel isn’t in use.
I found a query which gave me the database size. Turns out to be much larger than I thought: 5.7GB vs the couple of GB I assumed. Add this to the 4GB RAM and the my.cnf can’t be tuned properly anyway. Kinda hard to fit 5.7GB of database into 4GB of RAM. Essentially swaps are being done and in the present config there isn’t anything that can really be done about it.
I’ve recommended that, instead of building out a separate DB server, they upgrade to RHEL 5.2 64-bit and upping the RAM. Once that is done the MySQL config can be adjusted to make use of the greater capacity eliminating at least that aspect of the problem.
I made other points as well. I recommended upgrading MySQL from 5.0.27 which I recall having minor issues which have caused problems in the past. I also suggested having someone create indexes for the custom fields.
Thanks for all the input folks. It’s appreciated.
-Mathew
Jesse Vincent wrote:
I don’t have direct access to the my.cnf file as I’m only a consultant these days but once I’m able to get that I’ll give some more info.
As also mentioned, I still need to take a look at the tuning scrip Ruslan pointed me to.
Start with that script. It just queries the database. It doesn’t make any
changes. But really, if you haven’t done that yet, further discussion
probably doesn’t make much sense.
Keep up with my goings on at The Life and Times of a Slacker
I’m currently running a roughly 8GB database on 2gb of ram (quad core xeon
2.4), webserver and db on the same machine. The indexes sit just over a 1GB
and I have 768MB on the pool, iowait is very low and the index hit rate is
high. The shredder indexes account for a large portion of it so it caches
them back in at that time which is late at night. I still think you could
get more out of it with some tuning. The indexes would probably help if you
are doing a lot of queries against those fields though.----- Original Message -----
From: “Mathew” mathew.snyder@gmail.com
To: “Jesse Vincent” jesse@bestpractical.com
Cc: “RT Users” rt-users@bestpractical.com
Sent: Friday, January 16, 2009 7:35 PM
Subject: Re: [rt-users] Hardware Config
I was finally able to log into the system I’m consulting on. Using all of
this discussion as a jumping off point to figure out other steps to take
I’ve found a couple things which are clear problems. There are only 4GB
RAM as opposed to the 8GB which I thought it had. This would be moot
anyway as the system is 32-bit and the PAE kernel isn’t in use.
I found a query which gave me the database size. Turns out to be much
larger than I thought: 5.7GB vs the couple of GB I assumed. Add this to
the 4GB RAM and the my.cnf can’t be tuned properly anyway. Kinda hard to
fit 5.7GB of database into 4GB of RAM. Essentially swaps are being done
and in the present config there isn’t anything that can really be done
about it.
I’ve recommended that, instead of building out a separate DB server, they
upgrade to RHEL 5.2 64-bit and upping the RAM. Once that is done the
MySQL config can be adjusted to make use of the greater capacity
eliminating at least that aspect of the problem.
I made other points as well. I recommended upgrading MySQL from 5.0.27
which I recall having minor issues which have caused problems in the past.
I also suggested having someone create indexes for the custom fields.
Thanks for all the input folks. It’s appreciated.
-Mathew
Jesse Vincent wrote:
I don’t have direct access to the my.cnf file as I’m only a consultant
these days but once I’m able to get that I’ll give some more info.
As also mentioned, I still need to take a look at the tuning scrip
Ruslan pointed me to.
Start with that script. It just queries the database. It doesn’t make any
changes. But really, if you haven’t done that yet, further discussion
probably doesn’t make much sense.
–
Keep up with my goings on at The Life and Times of a Slacker
The rt-users Archives
Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com
Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com
Jesse Vincent wrote:
I agree completely with the above, but more important to me than just RAM
and processing power is the speed of disk access. He mentioned using RAID
5 in a follow-up post. That’s fine, but are these IDE or 15k SCSI drives?
Faster drives should always speed up database performance.
At 8 gigs of RAM on a well-tuned system, most of what RT is pulling out
of the database should always be cached in memory. If MySQL is going to
disk on every query, the game’s over and you’re better off sobbing
quietly into a stiff drink than getting faster disks.
Well, yeah. Bump up your query_cache_size and that will be true for frequently
run SELECTs. Bump up tmp_table_size / max_heap_query_size while you’re about it
so sorts are all done in RAM as well. (RT seems to generate quite a lot of
sorting…)
However, databases having this thing about committing changes to persistent storage
means that they are always going to be doing disk IO, and slow disks are going
to hurt performance on INSERT, UPDATE, DELETE or anything else that modifies
data or schema. And that’s going to delay any subsequent SELECTs that can’t be
satisfied out of the query cache.
On RAID5 – unless you’ve got a really, really expensive RAID controller card,
RAID5 is always going to be sub-optimal for the sort of small, randomized IOs
that databases generate. Of course, if you can afford a good enough RAID
controller that it makes RAID5 fast enough, you can almost certainly afford a
few extra disks and use RAID10 instead, and it will be even faster…
Cheers,
Matthew
Dr Matthew Seaman The Bunker, Ash Radar Station
PGP: 0x60AE908C on servers Marshborough Rd
Tel: +44 1304 814890 Sandwich
Fax: +44 1304 814899 Kent, CT13 0PL, UK
signature.asc (259 Bytes)
I’ve been following this discussion (rather belatedly) with some
interest. One thing I haven’t heard anyone discuss yet:
Has anyone tried running RT in a virtual machine? I’m about to move
our RT 3.4.2 server onto a virtual machine. I’ve configured the RT VM
to be actually really quite weedy (32-bit and only 2 GB RAM), and
perhaps I need to change that, but I figure that VMware ESX probably
does quite a good job of caching the disk its using anyway. The
actual physical hardware is a quad socket quad core machine with 64 GB
of RAM. The underlying storage is StorageWorks EVA on a SAN, so it
should be pretty quick.
Has anyone tried this sort of thing? Am I about to burn myself
badly? Our turnover of tickets is pretty low - only a few hundred
tickets a week.
Tim
The Wellcome Trust Sanger Institute is operated by Genome Research
Limited, a charity registered in England with number 1021457 and a
company registered in England with number 2742969, whose registered
office is 215 Euston Road, London, NW1 2BE.
I’ve been following this discussion (rather belatedly) with some
interest. One thing I haven’t heard anyone discuss yet:
Has anyone tried running RT in a virtual machine?
Our current production system is a VM. It currently is configured with
4GB of RAM and 2 processors (32-bit). We use a SAN on the backend for
storage as well (XioTech, IIRC.)
Our system has been live for about three weeks and it just passed 1K
tickets.
Things seem snappy, though we experienced some slowdown after our
initial cut. This was due to memory leaks in mod-perl (we restart apache
every night now) and some queries that could have used some indexes
(which BP, operating within a support contract, helped us create.)
HTH
Matt Zagrabelny - mzagrabe@d.umn.edu - (218) 726 8844
University of Minnesota Duluth
Information Technology Systems & Services
PGP key 1024D/84E22DA2 2005-11-07
Fingerprint: 78F9 18B3 EF58 56F5 FC85 C5CA 53E7 887F 84E2 2DA2
He is not a fool who gives up what he cannot keep to gain what he cannot
lose.
-Jim Elliot
signature.asc (197 Bytes)