What software is recommended for high-loaded RT3.8-latest?

RT Users
1

Hi all.

I’m going to migrate our rt installation to latest version. We’ll install clean
RT on new hardware and them migrate DB and custom modifications.

Some points about our rt installation:

  • db size - more than 30G;
  • mostly 100000 tickets;
  • 4000 transactions per day.

Can you please advice software for serving such high-loaded system:

  • FreeBSD or Linux?
  • File system: Ext3/XFS/JFS/…?
  • apache 2.2 or nginx?
  • MySQL or Postgresql?

Any advice will be appreciated.

Agnislav Onufrijchuk
PortaOne, Inc., RT Developer
Tel: +1-866-SIP VOIP (+1 866 747 8647) ext. 7670

Meet us on April 14-15 at Booth 1202
Billing & OSS World Conference & Expo
Rio All-Suite Hotel & Casino, Las Vegas

2

Thu 02 Apr 2009 09:57:57 AM GMT
Agnislav Onufrijchuk wrote:

Hi all.

I’m going to migrate our rt installation to latest version. We’ll install clean
RT on new hardware and them migrate DB and custom modifications.

Some points about our rt installation:

  • db size - more than 30G;
  • mostly 100000 tickets;
  • 4000 transactions per day.

Can you please advice software for serving such high-loaded system:

  • FreeBSD or Linux?

Slackware Linux. Perfect balance of security and stability and with a
custom-generic kernel the RAM footprint is comparatively tiny and makes
for a very responsive server.

Whatever OS you choose, make sure you do a manual RT install, don’t rely
on someone’s pre-packaged system. Also, I recommend making sure all of
your perl modules are installed via CPAN not a packaging system to
ensure no upstream modifications and a simple upgrade path. In Slackware
both of the above are a given.

  • File system: Ext3/XFS/JFS/…?

Debatable. I would probably say Ext3 myself, but then for the level of
transactions you’re talking about you are on the border where J/X/Reiser
could prove themselves useful. Wouldn’t hurt to do some benchmarking.
For what it’s worth, don’t take recoverability into account in your
decision, just make backups. Trying to perform file-system data recovery
in that type of environment is a waste of time on any FS.

  • apache 2.2 or nginx?

Apache. No Question.

  • MySQL or Postgresql?

Debatable. I think for me it would depend on what is in use in the rest
of your architecture. If you are a fully MySQL house, as we are here,
then it makes sense to keep it all the same since you can share
primary/failover servers and your people-processes are harmonious. If
you don’t really have a dependency on either then… well it’s up to
you. I’m used to MySQL and having it at the core of nearly all
DB-dependant applications here has been useful, but many would argue
that for a larger system like yours PG wouyld give you better
performance. Again, a bit of benchmarking wouldn’t go amiss.

Kind Regards,

Mike Peachey, IT
Tel: +44 114 281 2655
Fax: +44 114 281 2951
Jennic Ltd, Furnival Street, Sheffield, S1 4QT, UK
Comp Reg No: 3191371 - Registered In England

3

Hi,

I have included my comments below. It is important to consider
your skills/strengths when making these choices. i.e. If you
have experience with one database or OS, you should consider
using them instead of trying to build expertise in a new
environment. That being said…

Hi all.

I’m going to migrate our rt installation to latest version. We’ll install clean
RT on new hardware and them migrate DB and custom modifications.

Some points about our rt installation:

  • db size - more than 30G;
  • mostly 100000 tickets;
  • 4000 transactions per day.

Are these 4000 tickets per day or 4000 updates total? 100000 tickets
is not very many if you actually generate 4000 tickets per day. Do
you “shred” old tickets to remove them from your DB?

Can you please advice software for serving such high-loaded system:

  • FreeBSD or Linux?
    Either would be acceptable, given 4000 tickets per day = 500 per hour
    for an 8 hour day = less than 10 tickets per minute is not much of a
    CPU load for today’s hardware if the I/O subsystem is up to the task…
  • File system: Ext3/XFS/JFS/…?
    Use the supported/recommended one for your chosen OS.
  • apache 2.2 or nginx?
    Apache all the way.
  • MySQL or Postgresql?
    We use PostgreSQL here because the release quality does not vary
    as wildly and MySQL. Check the mailing list for problems caused by
    particular versions of MySQL. If you pick a tested version, it will
    work well. PostgreSQL also support full text index support that make
    searching ticket body content extremely fast. We also use the Slony
    replication software to keep a warm spare RT system ready to go, in
    case the primary system has a hardware problem. We really want to
    have redundancy in our ticket system because it should be up even if
    everything else is down. :slight_smile:

Hope this helps.
Cheers,
Ken

4

Mike Peachey wrote:

Slackware Linux. Perfect balance of security and stability and with a
custom-generic kernel the RAM footprint is comparatively tiny and makes
for a very responsive server.

I have no enough experience with Slackware. AFAIK, it’s simple as BSD. Is it true?

Whatever OS you choose, make sure you do a manual RT install, don’t rely
on someone’s pre-packaged system. Also, I recommend making sure all of
your perl modules are installed via CPAN not a packaging system to
ensure no upstream modifications and a simple upgrade path.

Sure :slight_smile:

  • File system: Ext3/XFS/JFS/…?

Debatable. I would probably say Ext3 myself, but then for the level of
transactions you’re talking about you are on the border where J/X/Reiser
could prove themselves useful. Wouldn’t hurt to do some benchmarking.
For what it’s worth, don’t take recoverability into account in your
decision, just make backups. Trying to perform file-system data recovery
in that type of environment is a waste of time on any FS.

AFAIK, they’re all provide good data safety. Now we’re using MySQL && InnoDB, I
think XFS should be fast enough. But we may migrate to PostgreSQL. AFAIK it uses
a number of files (I may be wrong) to serve its DB. So, there can be Reiser/JFS.

  • apache 2.2 or nginx?

Apache. No Question.

Why? nginx supports FastCGI too and it is recommended to use on dedicated projects.

  • MySQL or Postgresql?

Debatable. I think for me it would depend on what is in use in the rest
of your architecture. If you are a fully MySQL house, as we are here,
then it makes sense to keep it all the same since you can share
primary/failover servers and your people-processes are harmonious. If
you don’t really have a dependency on either then… well it’s up to
you. I’m used to MySQL and having it at the core of nearly all
DB-dependant applications here has been useful, but many would argue
that for a larger system like yours PG wouyld give you better
performance. Again, a bit of benchmarking wouldn’t go amiss.

Thank you!

Agnislav Onufrijchuk
PortaOne, Inc., RT Developer
Tel: +1-866-SIP VOIP (+1 866 747 8647) ext. 7670

Meet us on April 14-15 at Booth 1202
Billing & OSS World Conference & Expo
Rio All-Suite Hotel & Casino, Las Vegas

5

Thu 02 Apr 2009 14:51:06 GMT
Agnislav Onufrijchuk wrote:

Mike Peachey wrote:

Slackware Linux. Perfect balance of security and stability and with a
custom-generic kernel the RAM footprint is comparatively tiny and makes
for a very responsive server.

I have no enough experience with Slackware. AFAIK, it’s simple as BSD. Is it true?

It is the oldest and most unix-like and vanilla distributions of linux.
Simplicity is at its heart along with security and stability. I use it
on Servers, Desktops and Laptops alike.

AFAIK, they’re all provide good data safety. Now we’re using MySQL && InnoDB, I
think XFS should be fast enough. But we may migrate to PostgreSQL. AFAIK it uses
a number of files (I may be wrong) to serve its DB. So, there can be Reiser/JFS.

Whatever you pick, build it into your kernel and you’ll be fine :slight_smile:

  • apache 2.2 or nginx?
    Apache. No Question.

Why? nginx supports FastCGI too and it is recommended to use on dedicated projects.

Let me put it this way… when you run into trouble, you want to be on
the same server that 99.9% of RT users are running.
Kind Regards,

Mike Peachey, IT
Tel: +44 114 281 2655
Fax: +44 114 281 2951
Jennic Ltd, Furnival Street, Sheffield, S1 4QT, UK
Comp Reg No: 3191371 - Registered In England

6

Kenneth Marshall wrote:

Are these 4000 tickets per day or 4000 updates total? 100000 tickets
is not very many if you actually generate 4000 tickets per day. Do
you “shred” old tickets to remove them from your DB?

4000 Transactions, not tickets per day.

  • File system: Ext3/XFS/JFS/…?
    Use the supported/recommended one for your chosen OS.

Even if I’ll choose FreeBSD, I will not use UFS :slight_smile: It’s too slow.

  • apache 2.2 or nginx?
    Apache all the way.

What advantages does it have?

  • MySQL or Postgresql?
    We use PostgreSQL here because the release quality does not vary
    as wildly and MySQL. Check the mailing list for problems caused by
    particular versions of MySQL. If you pick a tested version, it will
    work well. PostgreSQL also support full text index support that make
    searching ticket body content extremely fast. We also use the Slony
    replication software to keep a warm spare RT system ready to go, in
    case the primary system has a hardware problem. We really want to
    have redundancy in our ticket system because it should be up even if
    everything else is down. :slight_smile:

FTS - is one of the advantages of PostgreSQL we look for.

Thank you for help!

Agnislav Onufrijchuk
PortaOne, Inc., RT Developer
Tel: +1-866-SIP VOIP (+1 866 747 8647) ext. 7670

Meet us on April 14-15 at Booth 1202
Billing & OSS World Conference & Expo
Rio All-Suite Hotel & Casino, Las Vegas

7

Are these 4000 tickets per day or 4000 updates total? 100000 tickets
is not very many if you actually generate 4000 tickets per day. Do
you “shred” old tickets to remove them from your DB?

One more thing: we have 4000 transactions, but we have a number of long SELECT
queries every day. No, we didn’t shred any of tickets yet, because currently we
use 3.4.4 version :frowning:

Agnislav Onufrijchuk
PortaOne, Inc., RT Developer
Tel: +1-866-SIP VOIP (+1 866 747 8647) ext. 7670

Meet us on April 14-15 at Booth 1202
Billing & OSS World Conference & Expo
Rio All-Suite Hotel & Casino, Las Vegas

8

Kenneth Marshall wrote:

Are these 4000 tickets per day or 4000 updates total? 100000 tickets
is not very many if you actually generate 4000 tickets per day. Do
you “shred” old tickets to remove them from your DB?

4000 Transactions, not tickets per day.

Good, that means every alternative will meet your needs performance-wise.

  • File system: Ext3/XFS/JFS/…?
    Use the supported/recommended one for your chosen OS.

Even if I’ll choose FreeBSD, I will not use UFS :slight_smile: It’s too slow.

Makes sense.

  • apache 2.2 or nginx?
    Apache all the way.

What advantages does it have?

The biggest advantage is the userbase for when you have a problem.

  • MySQL or Postgresql?
    We use PostgreSQL here because the release quality does not vary
    as wildly and MySQL. Check the mailing list for problems caused by
    particular versions of MySQL. If you pick a tested version, it will
    work well. PostgreSQL also support full text index support that make
    searching ticket body content extremely fast. We also use the Slony
    replication software to keep a warm spare RT system ready to go, in
    case the primary system has a hardware problem. We really want to
    have redundancy in our ticket system because it should be up even if
    everything else is down. :slight_smile:

FTS - is one of the advantages of PostgreSQL we look for.

I am partial to FTS and it definitely rocks on PostgreSQL.

Cheers,
Ken

9
  • apache 2.2 or nginx?
    Apache. No Question.

Why? nginx supports FastCGI too and it is recommended to use on dedicated projects.

Let me put it this way… when you run into trouble, you want to be on
the same server that 99.9% of RT users are running.
Agree :slight_smile:

Thanks for help!

Agnislav Onufrijchuk
PortaOne, Inc., RT Developer
Tel: +1-866-SIP VOIP (+1 866 747 8647) ext. 7670

Meet us on April 14-15 at Booth 1202
Billing & OSS World Conference & Expo
Rio All-Suite Hotel & Casino, Las Vegas

10

Are these 4000 tickets per day or 4000 updates total? 100000 tickets
is not very many if you actually generate 4000 tickets per day. Do
you “shred” old tickets to remove them from your DB?

One more thing: we have 4000 transactions, but we have a number of long SELECT
queries every day. No, we didn’t shred any of tickets yet, because currently we
use 3.4.4 version :frowning:

Have you run EXPLAIN or the MySQL equivalent to see what is taking
the time in the long queries? Maybe adjusting your indexes would
help.

Cheers,
Ken

11

I have no enough experience with Slackware. AFAIK, it’s simple as BSD. Is it true?

I second the Slackware vote. I use it for everything. It’s the ultimate
distro for reliability-through-simplicity. Install nothing you don’t need.
If you don’t need xxgdb on your production server, don’t install it. (I
don’t even have X installed on my servers, but that’s just me).
Incidentally I frequently have servers go 200 days + without so much as a
reboot, even high volume FTP servers like this one:

root@----:~# uptime
13:13:47 up 246 days, 1:53, 2 users, load average: 0.23, 0.14, 0.05

AFAIK, they’re all provide good data safety. Now we’re using MySQL && InnoDB, I
think XFS should be fast enough. But we may migrate to PostgreSQL. AFAIK it uses
a number of files (I may be wrong) to serve its DB. So, there can be Reiser/JFS.

Filesystems are something I’ve spent a LOT of time on, so I know something
about this.

XFS has not-so-good safety. The fsck / repair tools don’t work on very
large filesystems because they need massive amount of memory – more often
than not, more than you have. If you need to fsck XFS, odds are you’ll be
formatting it instead. That said, it is deliciously fast and scalable when
properly optioned. Use a RAID controller with battery backup and you should
be fine; otherwise turn off write-back caching. Or, test your backups
frequently for restorability. :slight_smile:

reiserfs is similiar to XFS with safety. A fsck almost never works because
everything’s a tree – once the tree is scrambled, everything in the tree
below that point is scrambled too. This is even a bigger risk if you don’t
make the filesystem with notail. You’ll be formatting, not fsck’ing. It
also doesn’t scale well and its performance with large files is horrid.

JFS on the other hand, has wonderful repair tools and decent scalability.
Unfortunately, the performance of JFS degrades exponentially with the number
of inodes used (files & directories) as it searches everything rather
linearly, and the inode structure is necessarily inefficient to make it
easily repairable. Not recommended for a filesystem with gobs and gobs of
small files. (Unfortunately, I have two 12TB RAID arrays formatted JFS with
over 14 million small files on them, and if they weren’t in production, I’d
change it in a heart-beat).

ext4 is no longer in development mode and is considered production quality
(in kernel 2.6.28 and newer). I highly recommend using it over the other
options. It is extent-based rather than block-mapped (if you format it as
such), it has the reliability of ext3 and then some (as the journal is
checksummed), and its even faster with lots of small files than reiser if
you create the filesystem with the dir_index option (which creates a hash of
directory entries that is even faster than reiser’s b-tree). It even fsck’s
faster than ext3 because it skips unallocated space.

In short: XFS is fast and not reliable. JFS is very reliable, but slow.
Reiserfs is a pitiful joke (which can used successfully by the daring &
lucky). ext4 gives you everything you always asked for: the speed of XFS
(ok, almost), the fast lookups of reiser, and the reliability of JFS :slight_smile:

– ============================
Tom Lahti
BIT Statement LLC

(425)251-0833 x 117
http://www.bitstatement.net/
– ============================

12

Publicly faced:
Web server : 22:05:01 up 566 days, 10:38, 0 users, load average: 0.04, 0.03, 0.02
Mail server: 09:04:10 up 573 days, 15:16, 0 users, load average: 0.16, 0.04, 0.01
Name server: 22:04:16 up 573 days, 21:34, 0 users, load average: 0.02, 0.05, 0.00

RHEL5.

Simplicity isn’t always better. Management of distribution counts for a lot.

In any case, choose which ever platform you’re comfortable with.

StuartFrom: rt-users-bounces@lists.bestpractical.com [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Tom Lahti
Sent: Friday, 3 April 2009 07:17
To: Agnislav Onufrijchuk
Cc: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] What software is recommended for high-loaded RT3.8-latest?

I have no enough experience with Slackware. AFAIK, it’s simple as BSD. Is it true?

I second the Slackware vote. I use it for everything. It’s the ultimate
distro for reliability-through-simplicity. Install nothing you don’t need.
If you don’t need xxgdb on your production server, don’t install it. (I
don’t even have X installed on my servers, but that’s just me).
Incidentally I frequently have servers go 200 days + without so much as a
reboot, even high volume FTP servers like this one:

root@----:~# uptime
13:13:47 up 246 days, 1:53, 2 users, load average: 0.23, 0.14, 0.05

AFAIK, they’re all provide good data safety. Now we’re using MySQL && InnoDB, I
think XFS should be fast enough. But we may migrate to PostgreSQL. AFAIK it uses
a number of files (I may be wrong) to serve its DB. So, there can be Reiser/JFS.

Filesystems are something I’ve spent a LOT of time on, so I know something
about this.

XFS has not-so-good safety. The fsck / repair tools don’t work on very
large filesystems because they need massive amount of memory – more often
than not, more than you have. If you need to fsck XFS, odds are you’ll be
formatting it instead. That said, it is deliciously fast and scalable when
properly optioned. Use a RAID controller with battery backup and you should
be fine; otherwise turn off write-back caching. Or, test your backups
frequently for restorability. :slight_smile:

reiserfs is similiar to XFS with safety. A fsck almost never works because
everything’s a tree – once the tree is scrambled, everything in the tree
below that point is scrambled too. This is even a bigger risk if you don’t
make the filesystem with notail. You’ll be formatting, not fsck’ing. It
also doesn’t scale well and its performance with large files is horrid.

JFS on the other hand, has wonderful repair tools and decent scalability.
Unfortunately, the performance of JFS degrades exponentially with the number
of inodes used (files & directories) as it searches everything rather
linearly, and the inode structure is necessarily inefficient to make it
easily repairable. Not recommended for a filesystem with gobs and gobs of
small files. (Unfortunately, I have two 12TB RAID arrays formatted JFS with
over 14 million small files on them, and if they weren’t in production, I’d
change it in a heart-beat).

ext4 is no longer in development mode and is considered production quality
(in kernel 2.6.28 and newer). I highly recommend using it over the other
options. It is extent-based rather than block-mapped (if you format it as
such), it has the reliability of ext3 and then some (as the journal is
checksummed), and its even faster with lots of small files than reiser if
you create the filesystem with the dir_index option (which creates a hash of
directory entries that is even faster than reiser’s b-tree). It even fsck’s
faster than ext3 because it skips unallocated space.

In short: XFS is fast and not reliable. JFS is very reliable, but slow.
Reiserfs is a pitiful joke (which can used successfully by the daring &
lucky). ext4 gives you everything you always asked for: the speed of XFS
(ok, almost), the fast lookups of reiser, and the reliability of JFS :slight_smile:

– ============================
Tom Lahti
BIT Statement LLC

(425)251-0833 x 117
http://www.bitstatement.net/
– ============================
The rt-users Archives

Community help: http://wiki.bestpractical.com
Commercial support: sales@bestpractical.com

Discover RT’s hidden secrets with RT Essentials from O’Reilly Media.
Buy a copy at http://rtbook.bestpractical.com

13

Stuart Browne stuart.browne@ausregistry.com.au writes:

In short: XFS is fast and not reliable.

I think you exaggerate a bit. We have hundreds of terabytes of storage
on XFS. We see very few reliability problems.

Leif Nixon - Systems expert
National Supercomputer Centre - Linkoping University