Yet Another James Green Web Developer, Aquarist

Early Message Queue Experiences

Earlier this month I rolled out changes across our small fleet of services that made ActiveMQ an essential element backing much of our software. Hopefully I can note that the things that went well and the things that didn't go according to plan, at least in respect to the message queue software and our use of it.

Firstly, actually installing ActiveMQ is really not very difficult (we run Ubuntu GNU/Linux). However, there are some configuration tweaks needed to ensure a restart occurs in a timely manner. Also, since security of the data within is essential all traffic between AMQ instances goes via SSL. This latter aspect proves a bit of a pain given you need to set up mutual key store trusts between each linkage. However, it does work.

An area that we could not easily predict would be memory use. Here's an area that really could use some clarification from the AMQ documentation. There's three limits to be set - an in-RAM limit, a temporary storage (overflow for in-RAM), and a persistence storage limit. This seems to be on top of any Producer Flow Control - which occurs per thread. It may look pretty easy to configure this stuff up however there are enough questions about behaviour to make me wonder what I've missed. One of our servers actually began issuing Java out of memory exceptions - something that is documented on AMQ's site and which we managed to adjust.

Another thing - when using the PHP PECL Stomp library to connect to localhost for some reason it can fail to connect. The solution appears to be to specify 127.0.0.1 instead. The library's author did note the same thing as something to be investigated.

So launch evening night came along and having made the necessary adjustments I published our internal set of customer accounts to one of our servers. Publication itself only took a few seconds however consumption speed was dire. Given I'd tested on a small subset beforehand without even noticing a delay I was rather surprised. Either way, I was not about to subject our staff or customers to potential performance problems elsewhere and backed out. Subsequent testing proved the problem had nothing to do with AMQ at all but MySQL and ext4 performing way too many fsyncs. We adjusted MySQL and the consumption of accounts was orders of magnitude faster.

A second roll-forward was scheduled, this one went smoother although verification that all systems were operating normally took far too long - again absolutely nothing to do with AMQ more ensuring staff were aware of any new ways of operating our systems and repercussions for our customers.

So now that the dust has settled how have things been? AMQ has - touch wood - not been a source of trouble. We're not exactly pushing tens of thousands of messages per second through this stuff but we're seeing the enqueue/dequeue counters flying fast enough to be coping with things nicely.

If I were to do things differently: well, we got hit with at least two major architectural changes that needed to be pushed out simultaneously. I can't tell you how painful that is to deal with.

What about AMQ use? Give the broker plenty of CPU and RAM. It will fly. And don't write a line of code before you read Enterprise Integration Patterns.

A Network of SSL-Connected ActiveMQ Brokers

Message queues are simple yet fundamentally critical to distributing an application across multiple computers. If you're like us you may need to distribute some of these messages between your servers over a public Internet connection.

To prevent eavesdropping we can encrypt the traffic between the servers using industry-standard SSL. Specifically we use RSA.

ActiveMQ supports SSL and supports what we call "a network of brokers" meaning multiple servers each running at least one ActiveMQ instance, talking to each other.

To get SSL connectivity up and running could be considered a bit of a dark art however and having spent a few hours on the topic I'd thought I'd document what I learned.

Firstly let me explain the topology I'm using - we have an office machine that we call the "hub" containing various administrative details. Next, we have the servers each of which does various jobs. We connect each server to the hub. This particular topology is the hub-spoke architecture.

Each machine (hub and servers) needs two files when running:

1. A private  keystore
2. A shared keystore

A "keystore" is a database containing keys. A key is a digital fingerprint - an identity if you like. Each entry within the keystore can be referenced by an alias. The aliases you choose should for consistency's sake match the hostname of your machines which might make them "hub", "server-a", "server-b",  etc.

Let's Get Started

Within your ActiveMQ installation you should have a conf directory. Within here you will be creating both keystores and a third file.

Firstly, on each machine, create a private keystore with a machine fingerprint. Let's take server-a as an example:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -genkey -alias server-a -keyalg RSA -keystore server-a.ks
jamesg@server-b:/opt/activemq/conf$ sudo keytool -genkey -alias server-a -keyalg RSA -keystore server-b.ks
[ etc ]

You'll need to set a password so open up your favorite password management utility and create an entry for each machine's private keystore file. Enter the details needed on each machine one-by-one.

Once you have each machine with it's own private keystore you should export the private key you just built into a public key file:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -export -alias server-a -keystore server-a.ks -file server-a_cert
jamesg@server-b:/opt/activemq/conf$ sudo keytool -export -alias server-a -keystore server-b.ks -file server-b_cert
[ etc ]

So now each machine has a keystore with a private signature, and a file with a public signature (the cert file).

Now, securely copy each server's public certificate file to the hub machine. On the hub you should end up with server-a_cert, server-b_cert, server-c_cert, etc. You need to import each file into a shared keystore file. You'll be prompted for a new password for this new keystore file on the hub:

jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-a -keystore shared.ks -file /home/jamesg/server-a_cert
jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-b -keystore shared.ks -file /home/jamesg/server-b_cert
jamesg@hub:/opt/activemq/conf$ sudo keytool -import -alias server-c -keystore shared.ks -file /home/jamesg/server-c_cert
[ etc ]

Each server (a, b, c, etc.) now needs it's own shared keystore for inbound connections to be authenticated against. On each server, import it's own public signature:

jamesg@server-a:/opt/activemq/conf$ sudo keytool -import -keystore shared.ks -file server-a_cert
jamesg@server-b:/opt/activemq/conf$ sudo keytool -import -keystore shared.ks -file server-b_cert

Remember, this is on each server, not the hub which has a shared.ks keystore holding every server's signature.

Duplex Requirements

At this point you should have two keystores on each server and on the hub. The servers only know about themselves so far - the hub knows about itself and the servers. To allow the hub to talk to the servers we now need to be add the hub's own public key to the shared keystore on each server.

To do this, on the hub machine export your public key just as you did on the servers and add it to a shared key file on the hub. Next, securely copy your newly minted hub_cert file to each server and import them just as you did with the server certificates themselves to each server's shared.ks keystore file. When you add each one, remember to alias it according to the hostname of the machine it comes from so when adding the hub's hub_cert to server-a's shared.ks, the alias should be "hub".

XML Configuration

Back on to each server you need to edit the conf/activemq.xml file. Remember, as of ActiveMQ 5.4.0 the elements need to be listed in alphabetical order!

A child of <broker> needs to be added:  sslContext. There are no attributes but there is a single child: sslContext (yes, again):

<sslContext>
<sslContext keyStore="server-a.ks" keyStorePassword="server-a-password"
trustStore="shared.ks" trustStorePassword="shared-password"/>
</sslContext>

Yes, there really is an sslContext child of an sslContext. Repeat on each server. Then on the hub:

<sslContext>
<sslContext keyStore="hub.ks" keyStorePassword="hub-private-password"
trustStore="shared.ks" trustStorePassword="hub-shared-password"/>
</sslContext>

You now have the SSL keys shared and configured. Now we need to configure our transports so the servers actually connect to the hub. On each server add the following to the activemq.xml configuration (remember, alphabetical order!):

<networkConnectors>
<networkConnector uri="static://(ssl://hub.mycompany.com:61616)"
name="hub"
duplex="true"
dynamicOnly="false">
</networkConnector>
</networkConnectors>

Notice above we have duplex="true". This enables two-way communication over the same connection. Obvious!

Finally, on the hub itself modify your activemq.xml:

<transportConnector name="openwire" uri="ssl://0.0.0.0:61616?needClientAuth=true"/>

If you wish, you can add the above on port 61617 or similar provided that chosen port is not used and you reflect the change of port on each connecting server.

Finishing Up

Note that you should not adjust the transporConnectors on your server unless clients that connect to your servers require something non-standard (ssl, Stomp, etc) or you need to adjust the settings for security reasons. Remember, the IP 0.0.0.0 means "listen on each interface, public and private".

Naturally you must ensure you have opened your hub's firewall to allow inbound connections to this port. Restart the hub, then your servers. Fingers crossed!

Debugging

If you're wondering what is happening take a look within data/activemq.log. Sadly, the errors themselves aren't automatically prompting humans where to look to correct them so a little Googling may be needed. Generally, check that the right signatures are listed each server's and the hub's shared.ks and that you have configured the right passwords. If you really don't see anything, check that ActiveMQ is actually still running - if there's an XML file problem it will quit without logging the fault! You can invoke using java manually if you need to.

As to the keystores themselves the good news is that keytool -list -keystore thefile.ks will show the contents when given the password. Compare the MD5 signatures of the aliases found within. If you've mis-named one, don't worry - just keytool -delete -alias badalias -keystore thefile.ks. It really is pretty simple when you get used to it.

Extending Futher

There is no reason why server-a could not talk SSL to server-b. Ensure the public certificates are added to each other's shared keystore and add the relevant networkConnector node to the XML file. Remember, the server with the networkConnector is considered by the connected-to server as a standard client.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          Apache ActiveMQ
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Apache ActiveMQ Messageing Queue
# Description:       Manages the Apache ActiveMQ messaging queue java daemon
### END INIT INFO

# Author: James Green <james.mk.green@gmail.com>

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="Apache Messaging Queue"
NAME=activemq
DIR=/usr/share/activemq
BIN_DIR=$DIR/bin/linux-x86-32
DAEMON=$BIN_DIR/$NAME
DAEMON_START="start xbean:conf/activemq.xml"
DAEMON_STOP="stop"
PIDFILE=$BIN_DIR/ActiveMQ.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

cd $DIR

#
# Function that starts the daemon/service
#
do_start()
{
 # Return
 #   0 if daemon has been started
 #   1 if daemon was already running
 #   2 if daemon could not be started
 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
 || return 1
 start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON -- \
 $DAEMON_START \
 || return 2
 # Add code here, if necessary, that waits for the process to be ready
 # to handle requests from services started subsequently which depend
 # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
 # Return
 #   0 if daemon has been stopped
 #   1 if daemon was already stopped
 #   2 if daemon could not be stopped
 #   other if a failure occurred
 #start-stop-daemon --stop --name $NAME --pidfile $PIDFILE --exec $DAEMON -- $DAEMON_STOP
 RETVAL="$?"
 $DAEMON stop
 [ "$RETVAL" = 2 ] && return 2
 # Wait for children to finish too if this is a daemon that forks
 # and if the daemon is only ever run from this initscript.
 # If the above conditions are not satisfied then add some other code
 # that waits for the process to drop all resources that could be
 # needed by services started subsequently.  A last resort is to
 # sleep for some time.
 start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
 [ "$?" = 2 ] && return 2
 # Many daemons don't delete their pidfiles when they exit.
 rm -f $PIDFILE
 return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
 #
 # If the daemon can reload its configuration without
 # restarting (for example, when it is sent a SIGHUP),
 # then implement that here.
 #
 start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
 return 0
}

case "$1" in
 start)
 [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
 do_start
 case "$?" in
 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
 stop)
 [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
 do_stop
 case "$?" in
 0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
 2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
 esac
 ;;
 status)
 status_of_proc "$DAEMON" "$NAME" && exit 0 || exit $?
 ;;
 #reload|force-reload)
 #
 # If do_reload() is not implemented then leave this commented out
 # and leave 'force-reload' as an alias for 'restart'.
 #
 #log_daemon_msg "Reloading $DESC" "$NAME"
 #do_reload
 #log_end_msg $?
 #;;
 restart|force-reload)
 #
 # If the "reload" option is implemented then remove the
 # 'force-reload' alias
 #
 log_daemon_msg "Restarting $DESC" "$NAME"
 do_stop
 case "$?" in
 0|1)
 do_start
 case "$?" in
 0) log_end_msg 0 ;;
 1) log_end_msg 1 ;; # Old process is still running
 *) log_end_msg 1 ;; # Failed to start
 esac
 ;;
 *)
 # Failed to stop
 log_end_msg 1
 ;;
 esac
 ;;
 *)
 #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
 echo "Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}" >&2
 exit 3
 ;;
esac

: