Besides this, there’s a couple of new features and close to two hundred bug fixes in this release. Some of the prominent new features are:

• Secure WebSocket (wss) transport – which means that you can now securely connect to the broker directly from your browser. You can find more information about it here
• Broker Redelivery – which allows you to define redelivery policy such that broker will resend a message to a different consumer in case that processing fails. Here you can find more information on when you’d want to use this feature and how.

We also improved our storage locking mechanism, so now it’s completely pluggable. This means that locking is not tied to the store itself, but it’s separately configured. And also you can tune it or implement new locking strategies to suit your environment. We also provided a new database locker, called Lease Database Locker, which should do much better job for JDBC master slave scenarios.

So, that’s about it. Give 5.7.0 a try and let us know what do you want to see in 5.8.0.

And while I have your attention, there’re two upcoming sessions where you can learn more about ActiveMQ:

• I’ll be talking about ActiveMQ in the cloud at Red Hat Integration and BPM Week on October 15th
• and about Apollo and the future of ActiveMQ at ApacheCon EU on November 8th

For the upcoming 5.7.0 release we introduced pluggable storage lockers, which means that message storage locking is totally separated from the store itself. That means that you can now use any locking strategy with any store. An example configuration is shown below:

<persistenceAdapter>
    <kahaDB directory = "target/activemq-data">
        <locker>
            <shared-file-locker lockAcquireSleepInterval="5000"/>
        </locker>
    </kahaDB>
</persistenceAdapter>

You can find more details on this new feature here. It will also allow us to implement new locking strategies, based on ZooKeeper for example, which will make high-availability setups even easier. So stay tuned.

Using appropriate message type

So let’s start with the first principles of messaging. Why we want to use a message broker in our architecture? The most probable answer is to exchange data in asynchronous and lously-coupled way between our systems. In that terms we should see how our data are best represented in terms of messages. JMS specification defines a few types of message types to be used which I’d classify in two groups. In the first group I’ll put TextMessages and ByteMessages which provides a kind of a plain-sheet in terms of what kind of data is carried in the message body. I think that you should strongly consider using only these message types as they provide a framework for loosely-coupled data exchange without introducing unnecessary complexity. Let’s cover briefly other message types and discuss what they bring to the picture:

ObjectMessages

If you haven’t already, be sure to read a post Jeff Mesnil wrote on this subject. It sums up pretty well what’s wrong ObjectMessage type. In the nutshell:

• You can get into a classloading mess as your systems need to share a classpath information as they need to be able to (de)serialize same objects. This increases coupling between the systems which we wanted to avoid in the first place.

• It adds unnecessary performance penalties for serializing and transferring the whole objects, instead of only valuable data

StreamMessages

In a similar fashion, StreamMessage adds some semantics over the basic ByteMessages. Instead of treating all bytes equally, you can now write and read strings, integers, objects, etc. This was very valuable in times when JMS API was designed, but today in the age of all these advanced serialization frameworks, both binary (Protobuf and co) and text-based ones (Jackson and co.), I think you’ll better forget about it. Encode your data with the some of the tools you’re probably already using in your applications and transfer them using ByteMessages or TextMessages.

MapMessages

MapMessages are just one more example of a too specialized interface. It’s true that map (properties) collection format is commonly used, but it’s just one of many, so why depend and couple your application to it.

Additionally usage of ObjectMessages, StreamMessages and MapMessage ties your solution to the JMS land and prevents you to exchange messages with Stomp-based clients for example.

Avoid fat-messages (Blobs)

While it’s certainly possible to move large messages through message brokers, you should think twice if you want to do that. And when I say large messages, I mean gigabytes and gigabytes of data in a single message. If you find yourself with the requirement like that, ask yourself do you really need a messaging service to move this data. Broker internals are optimized to move large amount of messages and adding big blobs in the combination can cause various side effects (connection controls kicking in, blocking other clients, exhausting resources, etc).

The usual pattern suggested for this use case is that you should use some traditional transport, like FTP, for moving large data; and use messaging service to notify clients when data is ready for download (and where to find it). ActiveMQ even provides and [API which will do this for you under the cover of JMS API] (http://activemq.apache.org/blob-messages.html)

That’s it for today, choose your message type and size wisely so you don’t end up with tightly coupled systems and brokers/clients struggling with oversized messages.

Short-lived connections

One thing that reoccur regularly is folks (knowingly or unknowingly) creating and tearing connections to the broker for every message they produce and consume.

This anti-pattern is especially common in two environments: Stomp and Spring. Stomp is very lightweight text-oriented messaging protocol. Which makes it really easy to write clients in almost any programming language available. This is one of the main strengths of Stomp and if you’re not JVM-exclusive shop I strongly recommend you to take a detailed look at it. But this impose a problem of course; large number of clients are poorly written and/or used inappropriately. For example, you can have a script which when ran will send or consume some messages from the broker. So if you don’t care much, you’ll open a new connection every time, send and consume messages and (hopefully) disconnect from the broker. Then you’ll put your script under load which will then transfer that load to the broker.

Spring, on the other hand, uses nice abstractions for dealing with JMS brokers, but the problem is that it was designed to be ran in a container of some kind and it is expected that container will manage resources for it. So when you write your standalone Java application and don’t care about this you end up with messaging clients behaving badly. A new connection, session, producer, consumer objects will be created for every message exchange and that far from optimal.

Why is all this such a problem? Well, first of all opening and closing connections requires from broker to do some work and having large number of clients opening and closing connections to exchange a single message produces a huge overhead and steals resources broker could use to do some other useful work. This is not specific to messaging and broker. You don’t open and close a database connection for every query (hopefully) for the same reasons. And also a spike in load in this case can spike in number of sockets used (as they need some time shutdown on the system) and eventual system resources exhaustion.

Besides this, messaging services are all about long lasting connections and clients. ActiveMQ implements various concepts, like producer flow control and consumer message prefetch, which are aimed to improve messaging experience and performance of the whole system. Not only that these messaging mechanisms are meaningless in a short-lived connections scenario, but can also introduce additional overhead in the system.

Finally if you have network brokers, information on large number of message consumers coming and going will be propagated through the network. This can significantly increase a broker to broker traffic and even impact the stability of remote brokers.

So what’s to be done? In Spring it’s easy, just use some kind of a cached connection factory and you’ll be sorted. There’ll be no more a new connection, session, producer, consumer for every message exchanged. More resources on ActiveMQ Spring support can be found here so please give it read if you’re using JMS Spring clients in your applications.

For Stomp, there’s no silver bullet. But for starters be aware of what you’re doing and try to reuse your resources smartly. If you do it, messages will flow smoothly and you’ll have a stable system.

In this and couple of follow-up posts I’ll try sum up some things we saw in mailing lists, Jiras, etc related to improper usage of messaging systems (ActiveMQ in particular). Hopefully, it will help people that consider using messaging in their architecture, see if it is the right tool for solving their particular problem.

So let’s kick off with one common mistake people make

Using message queue as a database

Messaging systems are built to asynchronously connect multiple systems, by passing messages between them. So everything is designed with that in mind; how to most efficiently pass messages from producers to consumers. This means that messages are expected to be reasonably short-lived, and not stored in a queue.

From time to time we see people trying keep application state in the broker. Put some messages in a queue, than browse them, cherry-pick just some of them, delete others and similar stuff. While most of the messaging systems have some kind of support to do this it’s not what they’re designed to support primarily. Client APIs, internal storage system, client-server contracts, etc. are optimized for entirely different set of tasks.

An example could be a system that wants to keep a single most-recent data as a queue message (timestamped or versioned somehow). So that application can find that message (usually by browsing) and do the house keeping by deleting stale data. People are sometimes inclined to do this as brokers provide high-availability, reconnection logic, can be geographically distributed which is all fine and well. But brokers are a poor choice for maintaining application state.

A workaround is not to keep any state in the broker, of course. Either use a centralized high-availability database or a local copy of data and use messaging system to propagate changes.

So if you find yourself wanting to store some messages in a queue and then later browse them, query them or maintain them, please don’t. Get yourself a database of some kind (relational or not) and manipulate data there. Queues (and topics) are for data that should be consumed as they come and moved from one system to another as fast as possible. They should live in the broker only as long as it takes to consume them or if something has gone wrong and they cannot be consumed. Which should be an exceptional situation rather than what we design for.

If there’s any messaging anti-patterns you observed (or designed yourself – c’mon don’t be ashamed), send them to me. I’ll gladly put them on my list and document them in coming days.

CamelOne was packed with FuseSource customers and users with a great feedback on the things we do. There were a lot of interest in Fuse Fabric which should help folks provision their integration infrastructure with ease. I covered how Fabric can help with complex ActiveMQ deployments and you can find the slides embedded below.

If you have to deploy and manage more than one broker I strongly recommend you taking a look at Fabric and FuseESB Enterprise 7.0 which incorporates it.

JEEConf was a more general Java conference; a quite larger than last year. Although most of the sessions were in Russian so I couldn’t actually follow, at least I had time to write this blog post
I gave a talk on ActiveMQ Apollo subproject, which is packed with cool new stuff that should bring open source messaging to the next level. Take a look at the slides here

It was well received and I’m looking forward to more feedback as people start playing with it more.
All in all, it was a great week and it’s always a pleasure to present and get a feedback on the our projects. Can’t wait for Monday when the new cycle of development begins.

Also, next week I’ll be at CamelOne in Boston and JEEConf In Kiev (a bit too much traveling for my taste, but there you go). I’ll talk about Enterprise deployment of ActiveMQ using Fuse Fabric (a topic already covered here in a nutshell) and Apache Apollo, the next generation of the broker. All these projects that are coming in the pipeline are pushing the possibilities of our integration infrastructure one step further. It will allow people to do complex deployments easier and connect to the infrastructure from virtually everywhere. It was a very exciting first half of the year, and it looks like things are going to be even more interesting going forward.

Provisioning

While classic server software (ActiveMQ included) deployment scenarios, which include unpacking a distro, setting a config and starting/stopping software, works OK for small deployments, there are a lot of challenges people encounter when trying to set and manage a large number of instances. Here are some of them:

• Enormous amount of work needed to set everything – imagine a large cluster you need to set, ssh-ing, unpacking, copying and tweaking config files is very tedious and error-prone process
• Changing configuration during runtime – you need again to manually tweak every file and restart the server, which is all but fun
• Upgrading – can also be a challenge and time consuming

Some of the things folks usually do to make their life easier is to:

• Keep XML config part of the spec as a template and in version control system and tweak as much things as possible with properties – this makes it easier to keep things under control and minimize the potential of an error when managing a configuration on large number of instances.
• Keep configuration separate from distribution to make it all easier to upgrade
• Use tools like Puppet or Chef to even further make things easier in these scenarios

One of the things where Fabric excels is centralized configuration management. In Fabric it’s really easy to spin new container instances on your servers (using ssh) or to the public cloud. Additionally it’s really easy to deploy ActiveMQ broker to those containers,
by simply applying appropriate profile to them. So generally, you can predefine a template for your broker, like XML configuration template and all necessary properties in a profile
and then with a simple Karaf command (or even mouse click in our tools, like FuseIDE and Fuse Management Console) deploy as many instances as you like of that broker (no matter if it’s a physical box in your data center or a VM at the public could provider).

Discovery

Fabric uses Apache ZooKeeper as it’s central registry of container instances running. This same registry is used to keep track of all brokers running inside a Fabric instance. This means that we can use this registry to discover all brokers inside a certain group. Therefore we created a new discovery protocol (called fabric of course) that can do that. So clients can connect to the broker group (think of it as a cluster) without any need to know exact location of the brokers. So now you can see how Fabric helps deploying your brokers to the cloud. First, you can use its ability to start containers (and deploy brokers on them) on any server or cloud provider available. And than clients can use Fabric to discover brokers and connect to them, all location agnostic.

Topologies

ZooKeeper as a central registry allows us to do some more nice things in domain of broker topologies. For example, currently the master-slave topology of ActiveMQ brokers depends on shared storage (either shared file system or enterprise JDBC database). In this scenario the master election and slave locking depends on the locking ability of the shared storage, limiting it to certain type of hardware. With ZooKeeper as a distributed registry, battle-proven for this kind of use cases, it’s easy to create a master-slave topologies with master election done on ZooKeeper locks. If you want to have a persistent master-slave, you still gonna need to store your messages in a some kind of a shared storage, but locking is not store’s job any more. And on the other hand it’s really easy to create non-persistent cluster of brokers with shared-nothing philosophy. Creating a master-slave is as simple as creating a multiple brokers with the same name in the group. The first one started will become a master, while others will be slaves waiting for the master to fail.

Another example of enhanced topology possibilities with ActiveMQ and Fabric is new ways you can set networks of brokers. Just as clients can use ZooKeeper registry to discover brokers, network connectors can use the same discovery protocol to connect the broker with all other brokers in the certain group. Again, brokers are mutually totally location agnostic, which is something desired in deployment scenarios with modern infrastructure. Could it be done any easier? Finally, you might want to mix these two topologies and create connected networks of master/slave broker, which with Fabric, is just easy to do as those basic topologies.

Upgrades and Patching

And now you say you need to upgrade all of the dozen brokers you have? No problem, with Fabric you can centrally update any bundle you’re using in your profile and that change will be propagated to all brokers. Now it’s time to talk about risk management of upgrades and this is where profile versions come into play. Instead of messing with profiles that are used in production at the moment, you can create a new version of it with updated versions of all bundles you plan to use. Once you have that profile ready, you can test it out on one or a few instances. And only after you’re certain that everything is OK, you can apply a new profile to all brokers. Of course, it’s easy to rollback to the previous version if anything goes wrong.

Fabric also provides you with the mechanism to provide a simple incremental patch jar with only certain classes modified (a single bug-fix). These classes will be then used instead of old ones and will allow us to provide a fix for a critical bug without need to upgrade the project version. This can be useful for dealing with critical production bugs, where waiting for the next release is not an option.

More resources

This post was just meant to give you a glance view of what’s coming from FuseSource regarding easier deployment, provisioning and managing of your messaging (and generally integration) infrastructure. You can check some more docs on the projects. Especially, there’s some more info that explains ActiveMQ concepts explained here in more details and shows how to run examples. You will definitely hear more from us on this topic in coming days, so stay tuned.

First there’s a new feature, called Priority Backup. It’s described in details here, but in a nutshell it provides you with the mechanism of prioritizing your failover urls and keep your clients connected to them as soon as they are available. The most obvious use case for this is to keep your clients connected to the broker in local data center whenever you can. By doing this, you can both have better performances and stability of your clients, but also save on your bandwidth bills.

Another improvement is coming for automatic broker cluster feature. Although this feature is not new, I spent some time hardening it and thought to share some more insight in how (and when) to use it in your projects.

In search of high availability, people often default to master-slave architecture. This makes sense in most use cases, but if your flow is purely non-persistent you can probably come up with more optimal architecture. Instead of having one broker at the time handling all your load, and other one just waiting for it to fail, you’ll get more efficient system with some kind of active-active configuration where (possibly multiple) brokers share the load all the time. Ideally clients would be evenly distributed and would rebalance if anything changes. Brokers don’t need to share any messages as clients are distributed and messages are non-persistent so they will be lost if broker fails. So can you achieve this kind of architecture with ActiveMQ?

Sure you do. That’s where automatic rebalance and clustering shines. First of all, brokers should be networked but only so they can exchange information on their availability. They shouldn’t exchange the messages (but of course can if your use case needs it). In 5.6 you do that with pure static networks, using configuration like

<networkConnector uri="static:(tcp://host)" staticBridge="true"/>

So now imagine three brokers A,B and C forming a full mesh. In addition every broker uses rebalance options on their transport connectors

<transportConnector name="openwire" uri="tcp://localhost:61616"
                    updateClusterClients="true" updateClusterClientsOnRemove="true"
                    rebalanceClusterClients="true"
/>

All that is left for the client to do is connect to one of the brokers it knows like

failover:(brokerA)

and the broker will fill it with all information on other brokers in the cluster and whether it should reconnect to one of them or not. So having a large number of clients connecting like this, very soon they’ll rebalance over available brokers. You can stop one of the brokers in the cluster for updates and clients will rebalance over remaining ones. You can even add a new broker to the cluster and everything will get rebalanced without any need for you to touch your clients.

So, basically in this way you have both load balancing and high availability for your non-persistent messages. Additionally, your clients are automatically updated with all information they need, and no manual intervention is needed.

Although the basic support for clustering was there since 5.4, I did some more hardening and better rebalancing, so it’s coming in the Apache ActiveMQ 5.6 (and the next Fuse 5.5.1) release. Also, there are some more great stuff regarding broker clustering coming soon, so stay tuned and happy messaging.