Yet Another James Green Web Developer, Aquarist

A Network of SSL-Connected ActiveMQ Brokers

Message queues are simple yet fundamentally critical to distributing an application across multiple computers. If you're like us you may need to distribute some of these messages between your servers over a public Internet connection.

To prevent eavesdropping we can encrypt the traffic between the servers using industry-standard SSL. Specifically we use RSA.

ActiveMQ supports SSL and supports what we call "a network of brokers" meaning multiple servers each running at least one ActiveMQ instance, talking to each other.

To get SSL connectivity up and running could be considered a bit of a dark art however and having spent a few hours on the topic I'd thought I'd document what I learned.

Firstly let me explain the topology I'm using - we have an office machine that we call the "hub" containing various administrative details. Next, we have the servers each of which does various jobs. We connect each server to the hub. This particular topology is the hub-spoke architecture.

Each machine (hub and servers) needs two files when running:

1. A private  keystore
2. A shared keystore

A "keystore" is a database containing keys. A key is a digital fingerprint - an identity if you like. Each entry within the keystore can be referenced by an alias. The aliases you choose should for consistency's sake match the hostname of your machines which might make them "hub", "server-a", "server-b",  etc.

Let's Get Started

Within your ActiveMQ installation you should have a conf directory. Within here you will be creating both keystores and a third file.

Firstly, on each machine, create a private keystore with a machine fingerprint. Let's take server-a as an example:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -genkey -alias server-a -keyalg RSA -keystore server-a.ks
jamesg@server-b:/opt/activemq/conf$ sudo keytool -genkey -alias server-a -keyalg RSA -keystore server-b.ks
[ etc ]

You'll need to set a password so open up your favorite password management utility and create an entry for each machine's private keystore file. Enter the details needed on each machine one-by-one.

Once you have each machine with it's own private keystore you should export the private key you just built into a public key file:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -export -alias server-a -keystore server-a.ks -file server-a_cert
jamesg@server-b:/opt/activemq/conf$ sudo keytool -export -alias server-a -keystore server-b.ks -file server-b_cert
[ etc ]

So now each machine has a keystore with a private signature, and a file with a public signature (the cert file).

Now, securely copy each server's public certificate file to the hub machine. On the hub you should end up with server-a_cert, server-b_cert, server-c_cert, etc. You need to import each file into a shared keystore file. You'll be prompted for a new password for this new keystore file on the hub:

jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-a -keystore shared.ks -file /home/jamesg/server-a_cert
jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-b -keystore shared.ks -file /home/jamesg/server-b_cert
jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-c -keystore shared.ks -file /home/jamesg/server-c_cert
[ etc ]

Each server (a, b, c, etc.) now needs it's own shared keystore for inbound connections to be authenticated against. On each server, import it's own public signature:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -import -keystore shared.ks -file server-a_cert
jamesg@server-b:/opt/activemq/conf$ sudo keytool -import -keystore shared.ks -file server-b_cert

Remember, this is on each server, not the hub which has a shared.ks keystore holding every server's signature.

Duplex Requirements

At this point you should have two keystores on each server and on the hub. The servers only know about themselves so far - the hub knows about itself and the servers. To allow the hub to talk to the servers we now need to be add the hub's own public key to the shared keystore on each server.

To do this, on the hub machine export your public key just as you did on the servers and add it to a shared key file on the hub. Next, securely copy your newly minted hub_cert file to each server and import them just as you did with the server certificates themselves to each server's shared.ks keystore file. When you add each one, remember to alias it according to the hostname of the machine it comes from so when adding the hub's hub_cert to server-a's shared.ks, the alias should be "hub".

XML Configuration

Back on to each server you need to edit the conf/activemq.xml file. Remember, as of ActiveMQ 5.4.0 the elements need to be listed in alphabetical order!

A child of <broker> needs to be added:  sslContext. There are no attributes but there is a single child: sslContext (yes, again):

<sslContext>
<sslContext keyStore="server-a.ks" keyStorePassword="server-a-password"
trustStore="shared.ks" trustStorePassword="shared-password"/>
</sslContext>

Yes, there really is an sslContext child of an sslContext. Repeat on each server. Then on the hub:

<sslContext>
<sslContext keyStore="hub.ks" keyStorePassword="hub-private-password"
trustStore="shared.ks" trustStorePassword="hub-shared-password"/>
</sslContext>

You now have the SSL keys shared and configured. Now we need to configure our transports so the servers actually connect to the hub. On each server add the following to the activemq.xml configuration (remember, alphabetical order!):

<networkConnectors>
<networkConnector uri="static://(ssl://hub.mycompany.com:61616)"
name="hub"
duplex="true"
dynamicOnly="false">
</networkConnector>
</networkConnectors>

Notice above we have duplex="true". This enables two-way communication over the same connection. Obvious!

Finally, on the hub itself modify your activemq.xml:

<transportConnector name="openwire" uri="ssl://0.0.0.0:61616?needClientAuth=true"/>

If you wish, you can add the above on port 61617 or similar provided that chosen port is not used and you reflect the change of port on each connecting server.

Finishing Up

Note that you should not adjust the transporConnectors on your server unless clients that connect to your servers require something non-standard (ssl, Stomp, etc) or you need to adjust the settings for security reasons. Remember, the IP 0.0.0.0 means "listen on each interface, public and private".

Naturally you must ensure you have opened your hub's firewall to allow inbound connections to this port. Restart the hub, then your servers. Fingers crossed!

Debugging

If you're wondering what is happening take a look within data/activemq.log. Sadly, the errors themselves aren't automatically prompting humans where to look to correct them so a little Googling may be needed. Generally, check that the right signatures are listed each server's and the hub's shared.ks and that you have configured the right passwords. If you really don't see anything, check that ActiveMQ is actually still running - if there's an XML file problem it will quit without logging the fault! You can invoke using java manually if you need to.

As to the keystores themselves the good news is that keytool -list -keystore thefile.ks will show the contents when given the password. Compare the MD5 signatures of the aliases found within. If you've mis-named one, don't worry - just keytool -delete -alias badalias -keystore thefile.ks. It really is pretty simple when you get used to it.

Extending Futher

There is no reason why server-a could not talk SSL to server-b. Ensure the public certificates are added to each other's shared keystore and add the relevant networkConnector node to the XML file. Remember, the server with the networkConnector is considered by the connected-to server as a standard client.