What format certificate to use with --ca-file

Hi,

I am finding that after my upgrade to 4.4.2 and moving to an SSL website that rt-mailgate is unhappy with my certificate.

I am using certbot and the site is responding to port 443 and all the major browsers are happy with the cert as Apache delivers it when browsing to the site https://tracker-software.support however replies to tickets are not making it through for delivery. /var/log/mail.log says:

Oct 10 10:30:29 rt postfix/qmgr[1354]: EC23C2C1F85: from=paulororke@gmail.com, size=16789, nrcpt=1 (queue active)
Oct 10 10:30:29 rt postfix/local[1675]: 6D2332C1F7B: to=support_comment@tracker-software.support, relay=local, delay=1744, delays=1744/0.02/0/0.23, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 500 Can’t connect to tracker-software.support:443 (certificate verify failed). Your webserver logs may have more information or there may be a network problem. )
Oct 10 10:30:29 rt postfix/local[1675]: using backwards-compatible default setting relay_domains=$mydestination to update fast-flush logfile for domain “tracker-software.support”

“certificate verify failed” - Apache config shows the following paths for the site’s certificates in it’s Virtual Host definition:

SSLCertificateFile /etc/letsencrypt/live/tracker-software.support/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/tracker-software.support/privkey.pem

I tried adding the path to the cert file for mailgate in /etc/aliases:

support_comment: "|/opt/rt4/bin/rt-mailgate --queue Support --action comment --ca-file /etc/letsencrypt/live/tracker-software.support/fullchain.pem --url https://tracker-software.support/"
support_correspond: “|/opt/rt4/bin/rt-mailgate --queue Support --action correspond --ca-file /etc/letsencrypt/live/tracker-software.support/fullchain.pem --url https://tracker-software.support/

but RT does not recognise it:

Oct 10 10:55:41 rt postfix/local[1858]: D96A92C1F86: to=support_comment@tracker-software.support, relay=local, delay=0.57, delays=0.36/0/0/0.21, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 500 SSL_ca_file /etc/letsencrypt/live/tracker-software.support/fullchain.pem does not exist. Your webserver logs may have more information or there may be a network problem. )

I am not seeing anything in the Apache logs about this. What format certificate is mailgate expecting to see and what should my --ca-file path point to?

I even tried clubbing it with –no-verify-ssl but it seemed to still be looking for one with the same error “certificate verify failed” - yes I did run newaliases when testing this :slight_smile:

Please and thanks

Paul

I had the same problem when I moved rt on ssl.
–no-verify-ssl helped me

support: |"/opt/rt4/bin/rt-mailgate --no-verify-ssl --queue General --action correspond --url https://rt.my.com"

Thanks Anton,

I am not having any success with this. I tried:

support: |"/opt/rt4/bin/rt-mailgate --queue Support --action correspond --no-verify-ssl --url https://tracker-software.support/"

and I get:

Oct 12 07:30:36 rt postfix/local[23972]: AD08D2C1F3B: to=support_comment@tracker-software.support, relay=local, delay=1028, delays=1027/0.02/0/0.26, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 500 Can’t connect to tracker-software.support:443 (certificate verify failed). Your webserver logs may have more information or there may be a network problem. )
Oct 12 07:30:36 rt postfix/local[23972]: using backwards-compatible default setting relay_domains=$mydestination to update fast-flush logfile for domain “tracker-software.support”

What is the process RT uses to “Verify” the certificate? Is it a problem with this cert? Is there any way to simulate this test and get feedback/error messages?

If I can get it to verify the SSL cert that would be ideal, however, getting --n0-verify-ssl to work would at least get my users on the new RT. I am stuck way behind schedule and I need to find a way to move this forward.

Does anyone have any thoughts on how to further troubleshoot this?

Respectfully

Paul

OK - I am not getting anywhere with this SSL issue and I need the new RT for my users so I tried ditching SSL entirely with the thought to return to the SSL later but I cannot get my new RT to accept mail.

Now /etc/aliases simply says:

Support: |"/opt/rt4/bin/rt-mailgate --queue Support --action correspond --url http://tracker-software.support/
Support_comment: |”/opt/rt4/bin/rt-mailgate --queue Support --action comment --url http://tracker-software.support/"

Yet when mail for an existing ticket hits RT it is not being added to the ticket.

/var/log/mail.log

Oct 16 10:56:02 rt postfix/qmgr[1359]: 796FF2C1ED8: from=paulororke@hotmail.com, size=7979, nrcpt=1 (queue active)
Oct 16 10:56:02 rt postfix/local[1794]: 796FF2C1ED8: to=Support_comment@tracker-software.support, relay=local, delay=959, delays=959/0.02/0/0.21, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 501 Not Implemented. Your webserver logs may have more information or there may be a network problem. )
Oct 16 10:56:02 rt postfix/local[1794]: using backwards-compatible default setting relay_domains=$mydestination to update fast-flush logfile for domain "tracker-software.support"
Oct 16 11:08:04 rt postfix/smtpd[1806]: connect from unknown[121.201.110.228]
Oct 16 11:08:05 rt postfix/smtpd[1806]: disconnect from unknown[121.201.110.228] ehlo=1 auth=0/1 rset=1 quit=1 commands=3/4

501 not implemented? https://developer.mozilla.org/en-US/docs/Web/HTTP/Status

501 Not Implemented
The request method is not supported by the server and cannot be handled. The only methods that servers are required to support (and therefore that must not return this code) are GET and HEAD.

Perhaps what ever is causing RT to not talk to postfix here is also the root of the issue when using SSL?

Is there any extended debugging/logging I can enable to help find out why I am getting this?

I live in hope.

Paul

Hi Paul,

did you found a solution for your problem? If have the same issue here on my server.
For me the Option --no-verify-ssl do not work. I always get the error message
HTTP request failed: 500 Can't connect to domain:443 (certificate verify failed). Your webserver logs may have more information or there may be a network problem.

I also tried to use the option --ca-file but the message is the same.
Just for Information, I use a PKI and sign the server certificate there. With several browsers I have no issues detected, so the certificate works.

Any help / idea is welcome.

Regards,
Tom

UPDATE:
@Paul, please have a look at thist post Rt-mailgate ignoring --no-verify-ssl?

I’m able to get it working with this modification.

use IO::Socket::SSL;
...
   sub get_useragent {
        my $self = shift;
        my $opts = shift;
        my $ua   = LWP::UserAgent->new();
        $ua->agent("rt-mailgate/4.4.2 ");
        $ua->cookie_jar( { file => $opts->{'jar'} } ) if $opts->{'jar'};

        $ua->ssl_opts( SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE );
        $ua->ssl_opts( verify_hostname => $opts->{'verify-ssl'} );
        $ua->ssl_opts( SSL_ca_file => $opts->{'ca-file'} )
            if $opts->{'ca-file'};

        return $ua;
    }

Test the mail flow:
$ echo "Subject: Testmail" | sendmail -v helpdesk@domain

I got a new ticket on the webpage. happy

Hi Tom,

thanks for finding that for me! Unfortunately SSL_VERIFY_NONE is still not being honoured.

in rt-mailgate:

sub get_useragent {
    my $self = shift;
    my $opts = shift;
    my $ua   = LWP::UserAgent->new();
    $ua->agent("rt-mailgate/4.4.2 ");
    $ua->cookie_jar( { file => $opts->{'jar'} } ) if $opts->{'jar'};

    $ua->ssl_opts( SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE );
    $ua->ssl_opts( verify_hostname => $opts->{'verify-ssl'} );
    $ua->ssl_opts( SSL_ca_file => $opts->{'ca-file'} )
        if $opts->{'ca-file'};

    return $ua;
}

I am pretty confident the line $ua->ssl_opts( SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE ); is OK because the first time I tried it I forgot to declare

use IO::Socket::SSL;

and I got this until I did.

Command output: Bareword “IO::Socket::SSL::SSL_VERIFY_NONE” not allowed while “strict subs” in use at /opt/rt4/bin/rt-mailgate line 157. Execution of /opt/rt4/bin/rt-mailgate aborted due to compilation errors.
again.

after declaring

use IO::Socket::SSL;

I am back to :

Oct 17 10:43:59 rt postfix/local[1657]: 1DF4A2C0918: to=Support_comment@tracker-software.support, relay=local, delay=531, delays=531/0.01/0/0.24, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 500 Can’t connect to tracker-software.support:443 (certificate verify failed). Your webserver logs may have more information or there may be a network problem. )

I even removed the line

$ua->ssl_opts( verify_hostname => $opts->{‘verify-ssl’} );

in the hope that this would stop the check but it changed nothing and now I am just floundering… :frowning:

I was really excited Tom, but my excitement was short lived. I’m really struggling with this one.

Hi Paul,

could you show us the command you entered on the command line when you got these message after the change of the rt-mailgate script?
Did you change the /etc/aliases file to something like that?

Support: |"/opt/rt4/bin/rt-mailgate --no-verify-ssl --queue Support --action correspond --url https://tracker-software.support/"
Support_comment: |"/opt/rt4/bin/rt-mailgate --no-verify-ssl  --queue Support --action comment --url https://tracker-software.support/"

Note: I copy your original commands and did some modification on it.

Let me know if it is now working.

Kind regards,
T0m

Thanks for this Tom,

we are making progress. I copy/pasted your version of /etc/aliases, ran newaliases, confirmed the non-SSL site is redirecting to the SSL site using:

RewriteEngine on
RewriteCond %{SERVER_NAME} =tracker-software.support
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

current /etc/aliases:

Support: |"/opt/rt4/bin/rt-mailgate --no-verify-ssl --queue Support --action correspond --url https://tracker-software.support/
Support_comment: |”/opt/rt4/bin/rt-mailgate --no-verify-ssl --queue Support --action comment --url https://tracker-software.support/"

Now I get this in my mail logs:

Oct 18 10:30:25 rt postfix/qmgr[1340]: E911A2C0927: from=paulororke@hotmail.com, size=10611, nrcpt=1 (queue active)
Oct 18 10:30:25 rt postfix/local[1615]: E911A2C0927: to=Support_comment@tracker-software.support, relay=local, delay=919, delays=918/0.03/0/0.31, dsn=4.3.0, status=deferred (temporary failure. Command output: HTTP request failed: 500 Status read failed: Connection reset by peer. Your webserver logs may have more information or there may be a network problem. )

I am not using a mail command to test, I am actually replying from Outlook.com to a test ticket.

This with /opt/rt4/bin/rt-mailgate using this:

use IO::Socket::SSL;

sub get_useragent {
my $self = shift;
my $opts = shift;
my $ua = LWP::UserAgent->new();
$ua->agent("rt-mailgate/4.4.2 ");
$ua->cookie_jar( { file => $opts->{‘jar’} } ) if $opts->{‘jar’};

$ua->ssl_opts( SSL_verify_mode => IO::Socket::SSL::SSL_VERIFY_NONE );
$ua->ssl_opts( verify_hostname => $opts->{'verify-ssl'} );
$ua->ssl_opts( SSL_ca_file => $opts->{'ca-file'} )
    if $opts->{'ca-file'};
return $ua;

}

This has me thinking it may be the redirect?
HTTP request failed: 500 Status read failed: Connection reset by peer.

yet a web browser redirects OK for me.

I feel so close I can smell it. I really appreciate the help Tom.

Hi Paul,

maybe the Problem is located in the apache2 configuration now.
I have in the /etc/apache2/sites-available/000-default.conf the following configuration:

<VirtualHost *:80>
ServerName RequestTrackerName:80
Redirect / https://RequestTrackerName/
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>

Then in the /etc/apach2/sites-available/rt.conf I use these settings.

<IfModule mod_ssl.c>
<VirtualHost _default_:443>
# Request Tracker
ServerName RequestTrackerName:443
AddDefaultCharset UTF-8
DocumentRoot /opt/rt4/share/html
Alias /NoAuth/images/ /opt/rt4/share/html/NoAuth/images/
ScriptAlias / /opt/rt4/sbin/rt-server.fcgi/
<Location />
## Apache version >= 2.4
Require all granted
</Location>
<Directory "/opt/rt4/sbin">
SSLOptions +StdEnvVars
</Directory>
ErrorLog ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLEngine on
SSLCertificateFile /path/to/certification.crt
SSLCertificateKeyFile /path/to/server.key
<FilesMatch "\.(cgi|shtml|phtml|php)$">
SSLOptions +StdEnvVars
</FilesMatch>
<Directory /usr/lib/cgi-bin>
SSLOptions +StdEnvVars
</Directory>
</VirtualHost>
</IfModule>

Could you check this?

Hopefully it will help you.

Thanks for this Tom,

I am still unable to get this to work. I stepped away from it for the weekend in the hope that I would see something afresh this morning but alas I am still seeing the same errors, either the certificate verification fails

HTTP request failed: 500 Can’t connect to tracker-software.support:443 (certificate verify failed).

or there is the

HTTP request failed: 500 Status read failed: Connection reset by peer.

error.

So I am going to go back to basics and perform a clean install on Ubuntu 16.04 LTS without data imported from my production instance. Perhaps something from my import is effecting the way this works. My hope is that if I cannot make a vanilla install work that maybe at least the issue is reproducible.

I took a break from this and came back with GoDaddy certs thinking the issue may have been the ones I was using from certbot.

I had hoped to not have to go right back to the beginning. I get the exact same errors with the GoDaddy certs so at this point I am assuming the issue is with my set up not the cert itself. Has anyone else used GoDaddy with RT 4.4 and Ubuntu 16.04?

This is killing me… :wink:

I resolved this and feel the need to apologize for wasting so much of other people’s time. The issue was not with any of the Perl scripts or RT config but my networking. :frowning:

curl showed me that I could not access my certificate even from localhost. The FQDN I used in /etc/aliases was sending the verify-ssl request through my gateway and back to the same interface and then port forwarded again to the rt host.

Simply changing /etc/hosts from the hostname rt.tracker-software.support to tracker-software.support (which matches the cert) and mail is flowing in all the loving ways it does on my old instance.

Thank you Tom and Anton for your input. It was most appreciated. Sorry I let my frustration was show.

I have a fairly comprehensive RT Upgrade on Ubuntu 16.04 notes having done this so many times over I can do it in my sleep. If anyone is interested I could post them on the wiki.