JIRA, Confluence and Lets Encrypt

2017/02/17

I recently had to move a JIRA and Confluence environment to a new infrastructure stack. During the move, we also changed the TLS Certificates and instead of using one of the paid-for incumbents we decided to give Lets Encrypt a go.

Everything with the migration went smoothly. The first hurdle we hit was when we were checking the Application Integration between the two systems. The integration wasnt functioning and no amount of delete, change, recreate would fix it. The admin pages in JIRA and Confluence were both reporting SSL errors. When I dug into the actual Tomcat logs for each instance, the following errors were appearing:

Confluence:

Caused by: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException
: unable to find valid certification path to requested target

JIRA:

Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification
path to requested target

Some quick googlephoo found a few items on the internet about this, not specific to JIRA and Confluence though.

ttps://community.letsencrypt.org/t/will-the-cross-root-cover-trust-by-the-default-list-in-the-jdk-jre/134/3

http://stackoverflow.com/questions/34110426/does-java-support-lets-encrypt-certificates

The root cause of the problem is that the JRE thats included with our version JIRA and Confluence is too old and doesn’t include the Lets Encrypt root keychain in its included keystore. The above articles had references and code snippets to help get the Lets Encrypt certificates into the JRE keystore but they were all very ugly.

Its worth mentioning that Oracle JAVA JRE 1.8.0_101 DOES include the Lets Encrypt certificates.

Options at this point were:

  • Find a way to get the required certificates into the JRE keystore (the CLI method to do this is described in the Lets Encrypt community post above).
  • Install a new JRE on the servers and make JIRA and Confluence work with that. Most likely putting us out of support with Atlassian.
  • Find out if current JIRA and Confluence include the required JRE version and then upgrade JIRA and Confluence. This would need another round of testing to properly do the upgrade.

Moving to unsupported configuration was undesirable and I didnt have the time to properly dive into a new round of testing to see if newer JIRA and Confluence had the required JRE version. I did look for some detail on the Atlassian pages to determine the answer to this and wasn’t able to locate anything.

What I did find on an Atlassian page was this article https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-779355358.html which shows how to use a free JIRA and Confluence plugin to get third party root certificates into the JRE keystore using a simple web page GUI. Some small CLI steps (a cp) are still required after the plug-in does its thing but it does make the fix less likely to fail.

Recommended.


Introduction to strace | The Road to Elysium

2016/12/05

HT to @patickkelso for this. Adding to the stash for future reference.

Source: Introduction to strace | The Road to Elysium


Internet Performance Measurement Tools

2016/11/23

This post contains a catalogue of useful Internet performance testing tools.

ICSI Netalyzr

http://netalyzr.icsi.berkeley.edu/cli.html

sivel/speedtest-cli: Command line interface for testing internet bandwidth using speedtest.net

A nifty tool for the CLI jockeys to test the internet performance of their machine.

speedtest-cli – Command line interface for testing internet bandwidth using speedtest.net

Source: sivel/speedtest-cli: Command line interface for testing internet bandwidth using speedtest.net


Combining PTP with NTP to Get the Best of Both Worlds – Red Hat Enterprise Linux Blog

2016/11/10

There are two supported protocols in Red Hat Enterprise Linux for synchronization of computer clocks over a network. The older and more well-known protocol is the Network Time Protocol (NTP). In it…

Source: Combining PTP with NTP to Get the Best of Both Worlds – Red Hat Enterprise Linux Blog

 


Install vSphere CLI on Ubuntu 14.04 LTS

2016/01/07

ARGGGH. What a painful experience this was.

Mainly due to unresolved dependancies in the vmware-install.pl script.

After a half a day shaving this yak, the method to fix the dependancies is:

sudo apt-get install libxml-libxml-perl libdevel-stacktrace-perl libclass-data-inheritable-perl libconvert-asn1-perl libcrypt-openssl-rsa-perl libcrypt-x509-perl libexception-class-perl libarchive-zip-perl libpath-class-perl libtry-tiny-perl libclass-methodmaker-perl libdata-dump-perl libnet-inet6glue-perl

Lastly, you need to install UUID and UUID::Random from CPAN. There doesnt appear to be suitable packages in 14.04.

  • install cpan (find your own link to an article on doing this on Ubuntu)
  • Then:
perl -MCPAN -e 'install UUID'
perl -MCPAN -e 'install UUID::Random'

You can then run your vmware-install.pl.


Home Internet Monitoring Appliance

2013/07/02

My home internet has been pretty flakey of late and I needed a way to monitor the performance of the connection over long periods of time so that I could gather evidence to escalate to my ISP for support and troubleshooting.

Due to the absence of a suitable machine at home to run the tools on I decided to build an Ubuntu VM with smokeping and Cacti. This would allow me to build the environment quickly and then move the VM to a temporary laptop running VMware Workstation at home.

The following steps describe the sequence of events and references I used to complete the work.


Upgrading Request Tracker

2010/10/14

I recently upgraded a Request Tracker (RT) environment from version 3.2.2 to version 3.8.7. These are my learnings and thoughts from the experience.

The environment that RT 3.2.2 was running on was an old hand rolled from source deployment on Fedora Core 3. The database was on a separate Debian 4 host running a backported MySQL 4.1.

A new web hosting and database back end was available based on Ubuntu 10.04 LTS and MySQL 5.1. This was the target environment for the upgraded RT.

Based on the available target environment the simplest approach to deliver this upgrade was to:

  • Install and configure RT 3.8.7 from packages on Ubuntu 10.04 LTS. Instructions here.
  • Dump the database using mysqldump from the old MySQL server.
  • Load the DB into the new database server.
  • Run the upgrade steps.
  • Validate the data and functionality.

As I tested and read up on the upgrade steps I soon found that this wasn’t going to be the big hassle I first imagined it to be. The details of the migration went like this:

  • Turn off all inbound SMTP email to the primary mail server. This is necessary as I want any emails bound for our ticketing system to queue up at the senders end as my migration involved a change of host for the ultimate destination for ticket emails.
  • Shutdown the old RT apache instance on the old web server and its MTA as well.
  • Shutdown the new RT apache instance on the new web server and its MTA as well.
  • Use mysqldump to copy the prod RT database to a .sql file on the new database server.

mysqldump –user={rtuser} –password={rtuserpasswd} –host={oldrtdbhost} –opt –skip-lock-tables –single-transaction –default-character-set=binary –databases {rtdbname} > {pathto .sql file}

  • Use mysql to load the .sql file into the new database server.

mysql –user={rtuser} –password={rtuserpasswd} –host={newrtdbhost} < {pathto .sql file}

  • Use the rt-setup-database script on the new RT web server to upgrade the DB from 3.2.2 to 3.7.87.

rt-setup-database –action upgrade –dba {rtuser} –prompt-for-dba-password

  • Build the schema upgrade script.

perl /usr/share/request-tracker3.8/etc/upgrade/upgrade-mysql-schema.pl {rtdbname}:{dbhost} {rtusername} {rtuserpasswd} > {path to schema .sql}

  • Apply the schema upgrade script.

mysql –user {rtuser} –password={rtuserpasswd} -h {newrtdbhost} {rtdbname} < {path to schema .sql}

  • Use the rt-setup-database script on the new RT web server to upgrade the DB from 3.7.87 to 3.8.7.

rt-setup-database –action upgrade –dba {rtuser} –prompt-for-dba-password

  • Start the web server and MTA on the new RT web server
  • Logged in to the new RT instance and made sure it all looked okay. I had to tweak settings in the following locations:
    • The rights all needed redoing. I blew them all away and reassigned rights to my groups from scratch.
    • Because I had started using Exim4 integration I had to go through each queues config and update its correspondence and comment address.
    • I tuned the default dashboard to look more like our original 3.2.2 dashboard. All the new stuff can be turned on later. I didnt want to move the agents cheese too much at first.
  • I then had to jump on our primary MX and fix up all its forwarder configs for our queue email addresses so that emails addressed to our queues would be delivered to the new host.
  • Turned in bound email on the primary mail server back on.
  • Sent and received test messages.

That was it. Of course there were teething problems from the migration. These included:

  • I hadn’t checked the $Organization value in the config of our new RT to make sure it matched the value in the old config. I ended up having to fix it.
  • I had a few problems with the change of the queue correspondence and comment address. The exim4 integration expects a comment address of {queuename}-comment@host.domain.com.au. Our comment addresses didn’t match this and I had to quickly update our forwarder configs to deliver to the correct address on the RT host.
  • I needed to give all the agents the ModifyCustomField right.
  • In my original exim4 reconfigure I hadnt configured exim to rewrite outbound emails from the host so all emails sent from RT left with an address of queuename@host.domain.com.au when what I wanted was queuename@domain.com.au. A quick `dpkg–reconfigure exim4-config` allowed me to turn on the rewriting of sent emails and the address they should be rewritten as.
  • RT will truncate ticket transactions with large bodies. See the MaxInlineBody RTConfig option or have your agents adjust their settings in their preferences.
  • Agents didnt have the right to modify requestor details. Had to grant them the AdminUsers and ShowConfigTab rights.

Somethings I learnt and knowledge I want to retain:

  • To test how exim4 will handle an email address on a host use this command

exim -d -bt {email address}

That’s it. So far things have been running really well.