Problems with ! in html format emails

Hi,
I am sorry I just noticed that this question didn’t have a proper Subject, which I have now added.

Regards

Kevin Curtis
FarSite CommunicationsFrom: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: 29 May 2014 15:05
To: rt-users@lists.bestpractical.commailto:rt-users@lists.bestpractical.com
Subject: [rt-users] (no subject)

Hi,
I have searched hard for the answer to this one, but haven’t seen it yet, maybe someone can point me in the right direction.

We have RT version 4.2.1 installed on Ubuntu 12.04. The main mailbox is on a Windows Exchange server, and we use fetchmail to get the mail every minute or so. Mail sent by RT goes through sendmail.

We use RT as a Ticket handling tool for customer problems. Our inhouse support staff use Office Outlook version 12. The email client composes html formatted email.

The problem that I have spent the last week trying to solve is that the emails sent out have an exclamation marks “!” inserted into them at various places.

Now I have tracked down that the sendmail tool is doing this because some of the lines are longer than the maximum line length supported by SMTP (990 characters).

And I have also tracked down that the Office Outlook email client is creating the long lines when html is chosen as the message format. (It doesn’t appear to be a problem with rtf or plain text).

I know the problem isn’t in RT itself, but our configuration must so typical of many RT installations that I can’t believe that we are the first to see this problem, and that there isn’t a solution already out there somewhere. If someone knows what it is then I’d be pleased to hear it.

I am not an expert in any of the component parts (fetchmail, sendmail or RT), but it seemed to me the best place to try and solve the issue was in the fetchmail/mailgate interface. So I have added a new method to EmailParser.pm.

I have used the RescueOutlook method as a template and I have tried to break lines (using the perl Text::Wrap) but this doesn’t seem to be doing the job. It looks like what I really need to do is process just the text/html section of the email and be a bit more intelligent about where the line breaks are placed. At the moment it’s just if the line is greater than 132 characters.

It’s been quite a steep learning curve this week! And it looks like it will take me a long time to get this fixed using the method I have chosen. I hope that there is already a fix.

Thanks in Advance

Kevin Curtis
Farsite Communications.

Hi,
In case anyone else comes across this same problem (and I can’t see why we should be the only users affected, I am sure that other RT installations must be processing emails from MS Office Outlook clients!), I have solved the issue for out particular case.

If this is a real problem it would be nice to have the fix in a future release of RT, so should I report it as a bug?

In the end I incorporated the solution into RescueOutlook as follows:

sub RescueOutlook {
my $self = shift;
my $mime = $self->Entity();

return unless $mime && $self->LooksLikeMSEmail($mime);

my $text_part;
my $html_part;
my $retval = 0;

my $tmp = $mime->head->get('Content-Type');
$RT::Logger->debug("Header content type $tmp");

if ( $mime->head->get('Content-Type') =~ m{multipart/related} ) {
    $RT::Logger->debug("processing multipart/related");
    my $first = $mime->parts(0);
  $tmp = $first->head->get('Content-Type');
    $RT::Logger->debug($tmp);

    if ( $first->head->get('Content-Type') =~ m{multipart/alternative} )
    {
    $RT::Logger->debug("processing related/alternative");
        my $inner_first = $first->parts(0);
        if ( $inner_first->head->get('Content-Type') =~ m{text/plain} )
        {
            $RT::Logger->debug("releated/alternative/plain");
            $text_part = $inner_first;
        }
        $inner_first = $first->parts(1);
        if ( $inner_first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("releated/alternative/html");
            $html_part = $inner_first;
        }
    }
}
elsif ( $mime->head->get('Content-Type') =~ m{multipart/mixed} ) {
    $RT::Logger->debug("processing multipart/mixed");
    my $first = $mime->parts(0);
    if ( $first->head->get('Content-Type') =~ m{multipart/alternative} )
    {
    $RT::Logger->debug("processing multipart/alternative");
        my $inner_first = $first->parts(0);
        if ( $inner_first->head->get('Content-Type') =~ m{text/plain} )
        {
            $RT::Logger->debug("mixex/alternative/plain");
            $text_part = $inner_first;
        }
        $inner_first = $first->parts(1);
        if ( $inner_first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("multipart/alternative/html");
            $html_part = $inner_first;
        }
    }
}
elsif ( $mime->head->get('Content-Type') =~ m{multipart/alternative} ) {
    my $first = $mime->parts(0);
    if ( $first->head->get('Content-Type') =~ m{text/plain} ) {
    $RT::Logger->debug("alternative/plain");
        $text_part = $first;
    }
  $first = $mime->parts(1);
        if ( $first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("alternative/html");
            $html_part = $first;
        }
}

# Add base64 since we've seen examples of double newlines with
# this type too. Need an example of a multi-part base64 to
# handle that permutation if it exists.
elsif ( $mime->head->get('Content-Transfer-Encoding') =~ m{base64} ) {
    $text_part = $mime;    # Assuming single part, already decoded.
}

if ($text_part) {
    # use the unencoded string
    my $content = $text_part->bodyhandle->as_string;
    if ( $content =~ s/\n\n/\n/g ) {

        # Outlook puts a space on extra newlines, remove it
        $content =~ s/\ +$//mg;

        # only write only if we did change the content
        if ( my $io = $text_part->open("w") ) {
            $io->print($content);
            $io->close;
            $RT::Logger->debug(
                "Removed extra newlines from MS Outlook message.");
            $retval = 1;
        }
        else {
            $RT::Logger->error("Can't write to body to fix newlines");
        }
    }
}
else
{
  $RT::Logger->debug("No text_part to fix newlines");
}

if ($html_part) {
    # use the unencoded string
    my $html_content = $html_part->bodyhandle->as_string;
    if ( $html_content =~ s/<o:p>/<o:p>\n/g ) {
        # only write only if we did change the content
        if ( my $io = $html_part->open("w") ) {
            $io->print($html_content);
            $io->close;
            $RT::Logger->debug(
                "Added linebreaks to Outlook message.");
            $retval = 1;
        }
        else {
            $RT::Logger->error("Can't write to html body to add linebreaks");
        }
    }
}
else
{
  $RT::Logger->debug("No html_part to add line breaks");
}

return $retval;

}

Regards

KevinFrom: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: 30 May 2014 17:18
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,
I am sorry I just noticed that this question didn’t have a proper Subject, which I have now added.

Regards

Kevin Curtis
FarSite Communications

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: 29 May 2014 15:05
To: rt-users@lists.bestpractical.commailto:rt-users@lists.bestpractical.com
Subject: [rt-users] (no subject)

Hi,
I have searched hard for the answer to this one, but haven’t seen it yet, maybe someone can point me in the right direction.

We have RT version 4.2.1 installed on Ubuntu 12.04. The main mailbox is on a Windows Exchange server, and we use fetchmail to get the mail every minute or so. Mail sent by RT goes through sendmail.

We use RT as a Ticket handling tool for customer problems. Our inhouse support staff use Office Outlook version 12. The email client composes html formatted email.

The problem that I have spent the last week trying to solve is that the emails sent out have an exclamation marks “!” inserted into them at various places.

Now I have tracked down that the sendmail tool is doing this because some of the lines are longer than the maximum line length supported by SMTP (990 characters).

And I have also tracked down that the Office Outlook email client is creating the long lines when html is chosen as the message format. (It doesn’t appear to be a problem with rtf or plain text).

I know the problem isn’t in RT itself, but our configuration must so typical of many RT installations that I can’t believe that we are the first to see this problem, and that there isn’t a solution already out there somewhere. If someone knows what it is then I’d be pleased to hear it.

I am not an expert in any of the component parts (fetchmail, sendmail or RT), but it seemed to me the best place to try and solve the issue was in the fetchmail/mailgate interface. So I have added a new method to EmailParser.pm.

I have used the RescueOutlook method as a template and I have tried to break lines (using the perl Text::Wrap) but this doesn’t seem to be doing the job. It looks like what I really need to do is process just the text/html section of the email and be a bit more intelligent about where the line breaks are placed. At the moment it’s just if the line is greater than 132 characters.

It’s been quite a steep learning curve this week! And it looks like it will take me a long time to get this fixed using the method I have chosen. I hope that there is already a fix.

Thanks in Advance

Kevin Curtis
Farsite Communications.

Hi Kevin, it sounds like you have an issue with your Outlook client
configuration, maybe a standard signature your company uses or some
configuration pushed by GP or your build is causing it. We use Outlook
as our mail client and HTML is our standard format. We have never seen
any issues like this.From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: Wednesday, June 04, 2014 11:54 AM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,

In case anyone else comes across this same problem (and I can't see

why we should be the only users affected, I am sure that other RT
installations must be processing emails from MS Office Outlook
clients!), I have solved the issue for out particular case.

If this is a real problem it would be nice to have the fix in a future
release of RT, so should I report it as a bug?

In the end I incorporated the solution into RescueOutlook as follows:

sub RescueOutlook {

my $self = shift;

my $mime = $self->Entity();



return unless $mime && $self->LooksLikeMSEmail($mime);



my $text_part;

my $html_part;

my $retval = 0;



my $tmp = $mime->head->get('Content-Type');

$RT::Logger->debug("Header content type $tmp");



if ( $mime->head->get('Content-Type') =~ m{multipart/related} ) {

    $RT::Logger->debug("processing multipart/related");

    my $first = $mime->parts(0);

  $tmp = $first->head->get('Content-Type');

    $RT::Logger->debug($tmp);



    if ( $first->head->get('Content-Type') =~

m{multipart/alternative} )

    {

    $RT::Logger->debug("processing related/alternative");

        my $inner_first = $first->parts(0);

        if ( $inner_first->head->get('Content-Type') =~

m{text/plain} )

        {

            $RT::Logger->debug("releated/alternative/plain");

            $text_part = $inner_first;

        }

        $inner_first = $first->parts(1);

        if ( $inner_first->head->get('Content-Type') =~ m{text/html}

)

        {

            $RT::Logger->debug("releated/alternative/html");

            $html_part = $inner_first;

        }

    }

}

elsif ( $mime->head->get('Content-Type') =~ m{multipart/mixed} ) {

    $RT::Logger->debug("processing multipart/mixed");

    my $first = $mime->parts(0);

    if ( $first->head->get('Content-Type') =~

m{multipart/alternative} )

    {

    $RT::Logger->debug("processing multipart/alternative");

        my $inner_first = $first->parts(0);

        if ( $inner_first->head->get('Content-Type') =~

m{text/plain} )

        {

            $RT::Logger->debug("mixex/alternative/plain");

            $text_part = $inner_first;

        }

        $inner_first = $first->parts(1);

        if ( $inner_first->head->get('Content-Type') =~ m{text/html}

)

        {

            $RT::Logger->debug("multipart/alternative/html");

            $html_part = $inner_first;

        }

    }

}

elsif ( $mime->head->get('Content-Type') =~ m{multipart/alternative}

) {

    my $first = $mime->parts(0);

    if ( $first->head->get('Content-Type') =~ m{text/plain} ) {

    $RT::Logger->debug("alternative/plain");

        $text_part = $first;

    }

  $first = $mime->parts(1);

        if ( $first->head->get('Content-Type') =~ m{text/html} )

        {

            $RT::Logger->debug("alternative/html");

            $html_part = $first;

        }

}



# Add base64 since we've seen examples of double newlines with

# this type too. Need an example of a multi-part base64 to

# handle that permutation if it exists.

elsif ( $mime->head->get('Content-Transfer-Encoding') =~ m{base64} )

{

    $text_part = $mime;    # Assuming single part, already decoded.

}



if ($text_part) {

    # use the unencoded string

    my $content = $text_part->bodyhandle->as_string;

    if ( $content =~ s/\n\n/\n/g ) {



        # Outlook puts a space on extra newlines, remove it

        $content =~ s/\ +$//mg;



        # only write only if we did change the content

        if ( my $io = $text_part->open("w") ) {

            $io->print($content);

            $io->close;

            $RT::Logger->debug(

                "Removed extra newlines from MS Outlook message.");

            $retval = 1;

        }

        else {

            $RT::Logger->error("Can't write to body to fix

newlines");

        }

    }

}

else

{

  $RT::Logger->debug("No text_part to fix newlines");

}



if ($html_part) {

    # use the unencoded string

    my $html_content = $html_part->bodyhandle->as_string;

    if ( $html_content =~ s/<o:p>/<o:p>\n/g ) {

        # only write only if we did change the content

        if ( my $io = $html_part->open("w") ) {

            $io->print($html_content);

            $io->close;

            $RT::Logger->debug(

                "Added linebreaks to Outlook message.");

            $retval = 1;

        }

        else {

            $RT::Logger->error("Can't write to html body to add

linebreaks");

        }

    }

}

else

{

  $RT::Logger->debug("No html_part to add line breaks");

}



return $retval;

}

Regards

Kevin

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: 30 May 2014 17:18
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,

I am sorry I just noticed that this question didn't have a proper

Subject, which I have now added.

Regards

Kevin Curtis

FarSite Communications

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: 29 May 2014 15:05
To: rt-users@lists.bestpractical.com
Subject: [rt-users] (no subject)

Hi,

I have searched hard for the answer to this one, but haven't seen it

yet, maybe someone can point me in the right direction.

We have RT version 4.2.1 installed on Ubuntu 12.04. The main mailbox is
on a Windows Exchange server, and we use fetchmail to get the mail every
minute or so. Mail sent by RT goes through sendmail.

We use RT as a Ticket handling tool for customer problems. Our inhouse
support staff use Office Outlook version 12. The email client composes
html formatted email.

The problem that I have spent the last week trying to solve is that the
emails sent out have an exclamation marks “!” inserted into them at
various places.

Now I have tracked down that the sendmail tool is doing this because
some of the lines are longer than the maximum line length supported by
SMTP (990 characters).

And I have also tracked down that the Office Outlook email client is
creating the long lines when html is chosen as the message format. (It
doesn’t appear to be a problem with rtf or plain text).

I know the problem isn’t in RT itself, but our configuration must so
typical of many RT installations that I can’t believe that we are the
first to see this problem, and that there isn’t a solution already out
there somewhere. If someone knows what it is then I’d be pleased to
hear it.

I am not an expert in any of the component parts (fetchmail, sendmail or
RT), but it seemed to me the best place to try and solve the issue was
in the fetchmail/mailgate interface. So I have added a new method to
EmailParser.pm.

I have used the RescueOutlook method as a template and I have tried to
break lines (using the perl Text::Wrap) but this doesn’t seem to be
doing the job. It looks like what I really need to do is process just
the text/html section of the email and be a bit more intelligent about
where the line breaks are placed. At the moment it’s just if the line
is greater than 132 characters.

It’s been quite a steep learning curve this week! And it looks like it
will take me a long time to get this fixed using the method I have
chosen. I hope that there is already a fix.

Thanks in Advance

Kevin Curtis

Farsite Communications.

------------------------------------------------------------------------------This message contains information that may be confidential and proprietary. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail, and delete the message immediately.

Hi Jason,
Thanks for the response. The “!” characters are being inserted (with a space as well) by sendmail. Is that the MTA that you are using?

I kind of agree that that it could be a configuration issue somewhere, but the “where” is difficult to track down. Everything looks ok.

We have two other problems that I haven’t seen any reference to either in the mailing list:

  1.  Inline images not displayed in emails sent out by rt, because they had been converted to text
    
  2.  Not being able to make an attachment when replying to correspondence through the GUI
    

All of these things are pretty noticeable, so it must be something unique with our installation.

It has been quite a frustrating few weeks.

Regards

KevinFrom: Hubbard, Jason [mailto:Jason.Hubbard@circles.com]
Sent: 04 June 2014 19:38
To: Kevin Curtis; rt-users@lists.bestpractical.com
Subject: RE: [rt-users] Problems with ! in html format emails

Hi Kevin, it sounds like you have an issue with your Outlook client configuration, maybe a standard signature your company uses or some configuration pushed by GP or your build is causing it. We use Outlook as our mail client and HTML is our standard format. We have never seen any issues like this.

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: Wednesday, June 04, 2014 11:54 AM
To: rt-users@lists.bestpractical.commailto:rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,
In case anyone else comes across this same problem (and I can’t see why we should be the only users affected, I am sure that other RT installations must be processing emails from MS Office Outlook clients!), I have solved the issue for out particular case.

If this is a real problem it would be nice to have the fix in a future release of RT, so should I report it as a bug?

In the end I incorporated the solution into RescueOutlook as follows:

sub RescueOutlook {
my $self = shift;
my $mime = $self->Entity();

return unless $mime && $self->LooksLikeMSEmail($mime);

my $text_part;
my $html_part;
my $retval = 0;

my $tmp = $mime->head->get('Content-Type');
$RT::Logger->debug("Header content type $tmp");

if ( $mime->head->get('Content-Type') =~ m{multipart/related} ) {
    $RT::Logger->debug("processing multipart/related");
    my $first = $mime->parts(0);
  $tmp = $first->head->get('Content-Type');
    $RT::Logger->debug($tmp);

    if ( $first->head->get('Content-Type') =~ m{multipart/alternative} )
    {
    $RT::Logger->debug("processing related/alternative");
        my $inner_first = $first->parts(0);
        if ( $inner_first->head->get('Content-Type') =~ m{text/plain} )
        {
            $RT::Logger->debug("releated/alternative/plain");
            $text_part = $inner_first;
        }
        $inner_first = $first->parts(1);
        if ( $inner_first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("releated/alternative/html");
            $html_part = $inner_first;
        }
    }
}
elsif ( $mime->head->get('Content-Type') =~ m{multipart/mixed} ) {
    $RT::Logger->debug("processing multipart/mixed");
    my $first = $mime->parts(0);
    if ( $first->head->get('Content-Type') =~ m{multipart/alternative} )
    {
    $RT::Logger->debug("processing multipart/alternative");
        my $inner_first = $first->parts(0);
        if ( $inner_first->head->get('Content-Type') =~ m{text/plain} )
        {
            $RT::Logger->debug("mixex/alternative/plain");
            $text_part = $inner_first;
        }
        $inner_first = $first->parts(1);
        if ( $inner_first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("multipart/alternative/html");
            $html_part = $inner_first;
        }
    }
}
elsif ( $mime->head->get('Content-Type') =~ m{multipart/alternative} ) {
    my $first = $mime->parts(0);
    if ( $first->head->get('Content-Type') =~ m{text/plain} ) {
    $RT::Logger->debug("alternative/plain");
        $text_part = $first;
    }
  $first = $mime->parts(1);
        if ( $first->head->get('Content-Type') =~ m{text/html} )
        {
            $RT::Logger->debug("alternative/html");
            $html_part = $first;
        }
}

# Add base64 since we've seen examples of double newlines with
# this type too. Need an example of a multi-part base64 to
# handle that permutation if it exists.
elsif ( $mime->head->get('Content-Transfer-Encoding') =~ m{base64} ) {
    $text_part = $mime;    # Assuming single part, already decoded.
}

if ($text_part) {
    # use the unencoded string
    my $content = $text_part->bodyhandle->as_string;
    if ( $content =~ s/\n\n/\n/g ) {

        # Outlook puts a space on extra newlines, remove it
        $content =~ s/\ +$//mg;

        # only write only if we did change the content
        if ( my $io = $text_part->open("w") ) {
            $io->print($content);
            $io->close;
            $RT::Logger->debug(
                "Removed extra newlines from MS Outlook message.");
            $retval = 1;
        }
        else {
            $RT::Logger->error("Can't write to body to fix newlines");
        }
    }
}
else
{
  $RT::Logger->debug("No text_part to fix newlines");
}

if ($html_part) {
    # use the unencoded string
    my $html_content = $html_part->bodyhandle->as_string;
    if ( $html_content =~ s/<o:p>/<o:p>\n/g ) {
        # only write only if we did change the content
        if ( my $io = $html_part->open("w") ) {
            $io->print($html_content);
            $io->close;
            $RT::Logger->debug(
                "Added linebreaks to Outlook message.");
            $retval = 1;
        }
        else {
            $RT::Logger->error("Can't write to html body to add linebreaks");
        }
    }
}
else
{
  $RT::Logger->debug("No html_part to add line breaks");
}

return $retval;

}

Regards

Kevin

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: 30 May 2014 17:18
To: rt-users@lists.bestpractical.commailto:rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,
I am sorry I just noticed that this question didn’t have a proper Subject, which I have now added.

Regards

Kevin Curtis
FarSite Communications

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On Behalf Of Kevin Curtis
Sent: 29 May 2014 15:05
To: rt-users@lists.bestpractical.commailto:rt-users@lists.bestpractical.com
Subject: [rt-users] (no subject)

Hi,
I have searched hard for the answer to this one, but haven’t seen it yet, maybe someone can point me in the right direction.

We have RT version 4.2.1 installed on Ubuntu 12.04. The main mailbox is on a Windows Exchange server, and we use fetchmail to get the mail every minute or so. Mail sent by RT goes through sendmail.

We use RT as a Ticket handling tool for customer problems. Our inhouse support staff use Office Outlook version 12. The email client composes html formatted email.

The problem that I have spent the last week trying to solve is that the emails sent out have an exclamation marks “!” inserted into them at various places.

Now I have tracked down that the sendmail tool is doing this because some of the lines are longer than the maximum line length supported by SMTP (990 characters).

And I have also tracked down that the Office Outlook email client is creating the long lines when html is chosen as the message format. (It doesn’t appear to be a problem with rtf or plain text).

I know the problem isn’t in RT itself, but our configuration must so typical of many RT installations that I can’t believe that we are the first to see this problem, and that there isn’t a solution already out there somewhere. If someone knows what it is then I’d be pleased to hear it.

I am not an expert in any of the component parts (fetchmail, sendmail or RT), but it seemed to me the best place to try and solve the issue was in the fetchmail/mailgate interface. So I have added a new method to EmailParser.pm.

I have used the RescueOutlook method as a template and I have tried to break lines (using the perl Text::Wrap) but this doesn’t seem to be doing the job. It looks like what I really need to do is process just the text/html section of the email and be a bit more intelligent about where the line breaks are placed. At the moment it’s just if the line is greater than 132 characters.

It’s been quite a steep learning curve this week! And it looks like it will take me a long time to get this fixed using the method I have chosen. I hope that there is already a fix.

Thanks in Advance

Kevin Curtis
Farsite Communications.

------------------------------------------------------------------------------This message contains information that may be confidential and proprietary. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail, and delete the message immediately.

Hi Kevin,

Sorry to hear that it’s been frustrating (I’ve been there).

We use Send mail for emails sent by RT. We use fetchmail to get
messages from our Exchange environment and push them to RT.

Our sendmail config is vanilla as well as most of our RT config. We are
sending html mail from RT using HTML in the templates. I haven’t tried
to insert inline images but I would imagine that it could be done in the
template.

I would try looking at outlook to discover why it is creating lines
longer than 990 characters when html is the format. It shouldn’t do
this by default. Also if your mail is processed by another MTA before
it reaches RT that could be adding the extra lines as well. The content
of the long lines should give some clue as to where they are coming
from, ie. a standard disclaimer added to outgoing emails, standard
signature added to every message by a mail delivery agent between
outlook and Exchange, etc.From: Kevin Curtis [mailto:Kevin.Curtis@farsite.com]
Sent: Thursday, June 05, 2014 6:27 AM
To: Hubbard, Jason; rt-users@lists.bestpractical.com
Subject: RE: [rt-users] Problems with ! in html format emails

Hi Jason,

Thanks for the response.  The "!" characters are being inserted

(with a space as well) by sendmail. Is that the MTA that you are using?

I kind of agree that that it could be a configuration issue somewhere,
but the “where” is difficult to track down. Everything looks ok.

We have two other problems that I haven’t seen any reference to either
in the mailing list:

  1.  Inline images not displayed in emails sent out by rt, because
    

they had been converted to text

  1.  Not being able to make an attachment when replying to
    

correspondence through the GUI

All of these things are pretty noticeable, so it must be something
unique with our installation.

It has been quite a frustrating few weeks.

Regards

Kevin

From: Hubbard, Jason [mailto:Jason.Hubbard@circles.com]
Sent: 04 June 2014 19:38
To: Kevin Curtis; rt-users@lists.bestpractical.com
Subject: RE: [rt-users] Problems with ! in html format emails

Hi Kevin, it sounds like you have an issue with your Outlook client
configuration, maybe a standard signature your company uses or some
configuration pushed by GP or your build is causing it. We use Outlook
as our mail client and HTML is our standard format. We have never seen
any issues like this.

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: Wednesday, June 04, 2014 11:54 AM
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,

In case anyone else comes across this same problem (and I can't see

why we should be the only users affected, I am sure that other RT
installations must be processing emails from MS Office Outlook
clients!), I have solved the issue for out particular case.

If this is a real problem it would be nice to have the fix in a future
release of RT, so should I report it as a bug?

In the end I incorporated the solution into RescueOutlook as follows:

sub RescueOutlook {

my $self = shift;

my $mime = $self->Entity();



return unless $mime && $self->LooksLikeMSEmail($mime);



my $text_part;

my $html_part;

my $retval = 0;



my $tmp = $mime->head->get('Content-Type');

$RT::Logger->debug("Header content type $tmp");



if ( $mime->head->get('Content-Type') =~ m{multipart/related} ) {

    $RT::Logger->debug("processing multipart/related");

    my $first = $mime->parts(0);

  $tmp = $first->head->get('Content-Type');

    $RT::Logger->debug($tmp);



    if ( $first->head->get('Content-Type') =~

m{multipart/alternative} )

    {

    $RT::Logger->debug("processing related/alternative");

        my $inner_first = $first->parts(0);

        if ( $inner_first->head->get('Content-Type') =~

m{text/plain} )

        {

            $RT::Logger->debug("releated/alternative/plain");

            $text_part = $inner_first;

        }

        $inner_first = $first->parts(1);

        if ( $inner_first->head->get('Content-Type') =~ m{text/html}

)

        {

            $RT::Logger->debug("releated/alternative/html");

            $html_part = $inner_first;

        }

    }

}

elsif ( $mime->head->get('Content-Type') =~ m{multipart/mixed} ) {

    $RT::Logger->debug("processing multipart/mixed");

    my $first = $mime->parts(0);

    if ( $first->head->get('Content-Type') =~

m{multipart/alternative} )

    {

    $RT::Logger->debug("processing multipart/alternative");

        my $inner_first = $first->parts(0);

        if ( $inner_first->head->get('Content-Type') =~

m{text/plain} )

        {

            $RT::Logger->debug("mixex/alternative/plain");

            $text_part = $inner_first;

        }

        $inner_first = $first->parts(1);

        if ( $inner_first->head->get('Content-Type') =~ m{text/html}

)

        {

            $RT::Logger->debug("multipart/alternative/html");

            $html_part = $inner_first;

        }

    }

}

elsif ( $mime->head->get('Content-Type') =~ m{multipart/alternative}

) {

    my $first = $mime->parts(0);

    if ( $first->head->get('Content-Type') =~ m{text/plain} ) {

    $RT::Logger->debug("alternative/plain");

        $text_part = $first;

    }

  $first = $mime->parts(1);

        if ( $first->head->get('Content-Type') =~ m{text/html} )

        {

            $RT::Logger->debug("alternative/html");

            $html_part = $first;

        }

}



# Add base64 since we've seen examples of double newlines with

# this type too. Need an example of a multi-part base64 to

# handle that permutation if it exists.

elsif ( $mime->head->get('Content-Transfer-Encoding') =~ m{base64} )

{

    $text_part = $mime;    # Assuming single part, already decoded.

}



if ($text_part) {

    # use the unencoded string

    my $content = $text_part->bodyhandle->as_string;

    if ( $content =~ s/\n\n/\n/g ) {



        # Outlook puts a space on extra newlines, remove it

        $content =~ s/\ +$//mg;



        # only write only if we did change the content

        if ( my $io = $text_part->open("w") ) {

            $io->print($content);

            $io->close;

            $RT::Logger->debug(

                "Removed extra newlines from MS Outlook message.");

            $retval = 1;

        }

        else {

            $RT::Logger->error("Can't write to body to fix

newlines");

        }

    }

}

else

{

  $RT::Logger->debug("No text_part to fix newlines");

}



if ($html_part) {

    # use the unencoded string

    my $html_content = $html_part->bodyhandle->as_string;

    if ( $html_content =~ s/<o:p>/<o:p>\n/g ) {

        # only write only if we did change the content

        if ( my $io = $html_part->open("w") ) {

            $io->print($html_content);

            $io->close;

            $RT::Logger->debug(

                "Added linebreaks to Outlook message.");

            $retval = 1;

        }

        else {

            $RT::Logger->error("Can't write to html body to add

linebreaks");

        }

    }

}

else

{

  $RT::Logger->debug("No html_part to add line breaks");

}



return $retval;

}

Regards

Kevin

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: 30 May 2014 17:18
To: rt-users@lists.bestpractical.com
Subject: Re: [rt-users] Problems with ! in html format emails

Hi,

I am sorry I just noticed that this question didn't have a proper

Subject, which I have now added.

Regards

Kevin Curtis

FarSite Communications

From: rt-users [mailto:rt-users-bounces@lists.bestpractical.com] On
Behalf Of Kevin Curtis
Sent: 29 May 2014 15:05
To: rt-users@lists.bestpractical.com
Subject: [rt-users] (no subject)

Hi,

I have searched hard for the answer to this one, but haven't seen it

yet, maybe someone can point me in the right direction.

We have RT version 4.2.1 installed on Ubuntu 12.04. The main mailbox is
on a Windows Exchange server, and we use fetchmail to get the mail every
minute or so. Mail sent by RT goes through sendmail.

We use RT as a Ticket handling tool for customer problems. Our inhouse
support staff use Office Outlook version 12. The email client composes
html formatted email.

The problem that I have spent the last week trying to solve is that the
emails sent out have an exclamation marks “!” inserted into them at
various places.

Now I have tracked down that the sendmail tool is doing this because
some of the lines are longer than the maximum line length supported by
SMTP (990 characters).

And I have also tracked down that the Office Outlook email client is
creating the long lines when html is chosen as the message format. (It
doesn’t appear to be a problem with rtf or plain text).

I know the problem isn’t in RT itself, but our configuration must so
typical of many RT installations that I can’t believe that we are the
first to see this problem, and that there isn’t a solution already out
there somewhere. If someone knows what it is then I’d be pleased to
hear it.

I am not an expert in any of the component parts (fetchmail, sendmail or
RT), but it seemed to me the best place to try and solve the issue was
in the fetchmail/mailgate interface. So I have added a new method to
EmailParser.pm.

I have used the RescueOutlook method as a template and I have tried to
break lines (using the perl Text::Wrap) but this doesn’t seem to be
doing the job. It looks like what I really need to do is process just
the text/html section of the email and be a bit more intelligent about
where the line breaks are placed. At the moment it’s just if the line
is greater than 132 characters.

It’s been quite a steep learning curve this week! And it looks like it
will take me a long time to get this fixed using the method I have
chosen. I hope that there is already a fix.

Thanks in Advance

Kevin Curtis

Farsite Communications.

------This message contains information that may be confidential and
proprietary. Unless you are the intended recipient (or authorized to
receive this message for the intended recipient), you may not use, copy,
disseminate or disclose to anyone the message or any information
contained in the message. If you have received the message in error,
please advise the sender by reply e-mail, and delete the message
immediately.

------------------------------------------------------------------------------This message contains information that may be confidential and proprietary. Unless you are the intended recipient (or authorized to receive this message for the intended recipient), you may not use, copy, disseminate or disclose to anyone the message or any information contained in the message. If you have received the message in error, please advise the sender by reply e-mail, and delete the message immediately.

We are also seeing this issue, but we are not using Outlook/Exchange for our mail server. @Kevin_Curtis, did you find a solution of any kind? I was actually thinking of opening a bug report with RT, but I really wanted to check around a bit first to see if it is some weird configuration option conflict. The "! " insertions seem to be at random for us - at the surface, I’m not seeing a discernible pattern in terms of characters, but this could be because they are getting added repeatedly over replies… They are ending up all over the place - inside mailto: links, inline styles, To:/From: lists - you name it. Sometimes, it looks like they’ve just been added into the content. Other times, it looks like they’ve replaced one or more characters (or lines) of text. Might need to look a little closer to verify that last observation, but it did seem that way when I was looking at this last.