ActiveMQ producer flow control with timeout

ActiveMQ producer flow control feature allows you to prevent overflowing the broker by slowing down fast producers. Thus far you had two options, the producer send() method will either wait until space becomes available, or fail if it’s configured through sendFailIfNoSpace property, like this:

<systemUsage>
 <systemUsage sendFailIfNoSpace="true">
   <memoryUsage>
     <memoryUsage limit="20 mb"/>
   </memoryUsage>
 </systemUsage>
</systemUsage>

For 5.3.1 version of the broker we added a third option. You can now configure the broker to do the producer flow control with timeout. If you configure <systemUsage> like this (using sendFailIfNoSpaceAfterTimeout property),

<systemUsage>
 <systemUsage sendFailIfNoSpaceAfterTimeout="3000">
   <memoryUsage>
     <memoryUsage limit="20 mb"/>
   </memoryUsage>
 </systemUsage>
</systemUsage>

the producer’s send() method will wait for space for 3 seconds and if it don’t get it, it will fail. This feature can be very useful if you don’t want your producers wait for the space for the unspecified amount of time.

Jettison 1.2 Released

Jettison version 1.2 has been released. The highlights of this release are:

  • Refactored MappedXMLStreamWriter – which solved many issues with serializing arrays
  • Plugable type converters and support for 64-bit long conversion

You can find the full change log for this release on the Download page. Enjoy

Handling IOException in ActiveMQ

When you encounter IOException trying to persist a message on the disk, you know something has gone wrong. One of two things usually happen:

  • Either your disk is dead, or disconnected in case you use network disks
  • Or you just don’t have more space on the disk

Depending on the type of the exception you may encounter, you might want to take a different action. For upcoming 5.3.1 release we implemented fully configurable IOException handling which can be tuned to your need.

A default handler stops the broker when the message persistence fails in all cases except when it detects that the failure is caused by the lack of space on the disk. In the later case, it just wait for the space to be reclaimed and continues.

For more information on this topic, check http://activemq.apache.org/configurable-ioexception-handling.html

Fuse Stomp project

In our attempt to provide better future for Stomp protocol, we already moved specification under ActiveMQ project umbrella, where all future specification improvements will be done. Another necessary step is to provide high-quality Stomp clients for use in different development environments. For that purpose, we created Stomp project on FUSE Forge. It should provide a project hosting environment for various Stomp clients.

For starters I released PHP Stomp Client 1.0.0. It’s a natural extension of all the work previously hosted on Codehaus + Google Code. It is now stabilized, with a lot of bug fixes and released as 1.0.0. All future development will be continued on the FUSE Forge as it provides much better hosting environment than we used thus far.

So, if you’re interested in Stomp for your application development, join the community and help us create high-quality clients for your environment.

Python messaging: ActiveMQ and RabbitMQ

I found myself recently with a task of testing various Python messaging clients. In this article I’ll present my findings regarding performance and scalability on two Python clients connecting to ActiveMQ and RabbitMQ message brokers.

For ActiveMQ Python client, I used pyactivemq library version 0.1.0. It’s basically a Python wrapper for ActiveMQ-CPP library, allowing Python clients to communicate with the broker using both OpenWire (ActiveMQ specific binary protocol) and Stomp (Simple text-oriented protocol) protocols. Since it uses ActiveMQ-CPP, installation requires special attention and some of those details you can find in a related blog post.

As a RabbitMQ client I used py-amqplib version 0.6, since that’s currently the most natural solution if you don’t want to use frameworks like Twisted.

I used the latest broker versions, which means ActiveMQ 5.3.0 and RabbitMQ 1.7. Also, all numbers shown below are for tests executed on an “average linux desktop box” running Ubuntu 9.0.4.

Now let’s go testing.

Performance

The performance test I used is really simple: One producer, one consumer and one queue. Both producer and consumer tries to do their best in terms of performances and we’re not going to use any transactions. I used small text messages, with text Example message _num_. The full source code of tests can be found at http://github.com/dejanb/pymsg/. For every client (and broker) there are two Python scripts:

  • test_receive_async.py – which starts an asynchronous consumer in a thread and samples consuming rates every 10 seconds
  • test_send.py – which starts a producer in a thread and samples producing rates every 10 seconds

Both of these scripts try to get 100 rate samples, which means tests last for about 15 minutes.

Persistent messages

ActiveMQ 5.3.0 comes with the number of example configuration files that helps you configure it for different usage scenarios. One of those configuration files is called conf/activemq-throughput.xml and it is the one that should be used for high performance scenarios.

To start a broker with this configuration file, just execute:

bin/activemq xbean:conf/activemq-throughput.xml

Now you can run the consumer and producer with

python test_receive_async.py

and

python test_send.py

in separate console windows.

When ran like this, the client will use OpenWire protocol. The following diagram shows the result (the full output can be found here).

openwire persistent

As you can see, after the initial spike, both producer and consumer have consistent rates of around 1300 msg/sec.

To repeat the same test, now using Stomp wire protocol, we’ll execute the scripts with one additional parameter, stomp (of course, a clean broker start is needed)

python test_receive_async.py stomp

and

python test_send.py stomp

The results for pyactivemq client using Stomp protocols can be found here and the diagram visualizing those data is shown below.

stomp persistent

Again, after the initial glitches, producer and consumer settles on a bit more than 1150 msg/sec. This is an interesting result that shows that with persistent messages the limit is not in the network protocol, but the ability of the broker to successfully persist messages.

Now let’s try do the same with RabbitMQ and py-amqplib client. The output with the result samples can be found here and they are visualized on the following diagram.

amqp persistent

On the first look you can notice the high message producing rates, starting with more than 3000 msg/sec and slowly decreasing toward 2000 messages per second. On the other hand the consumer is considerably slower at rates around 1000 msg/sec. This situation causes more and more messages being stored in the broker, when it finally crashes after 12-13 minutes. I tried setting Memory-based flow control, but without much success.

The high value on a producer side have one more important drawback regarding reliability. The AMQP protocol send operation is strictly asynchronous, meaning that producer does not have any confirmation from the broker that the message was actually queued. So what I observed is that the number of messages sent shown by the producer and messages stored in the broker are not in sync. You can check this by starting the producer that sends a large number of messages for some time and check the number of messages in the broker. You’ll notice that messages arrive to the broker, “long” after the producer has finished its job. So if broker crashes in the meantime, all messages producer thought it sent are lost. I didn’t test this with transactions, so I’m not sure if the behavior is somewhat different in this case. My guess it that it is and that is one of the things that I’d like to test further.

Also, RabbitMQ allows you to set the “returned listener” on the channel which can be used by the broker to return messages that were not routed (if the mandatory parameter is used during publishing) or could be immediately delivered (if the immediate parameter is used during publishing). py-amqplib library is currently missing this functionality, which can be another reliability issue for concern.

So generally, it seems like RabbitMQ have a very high producing rate (sacrificing reliability), while consuming rates are pretty standard and a bit lower than those seen by ActiveMQ consumers.

Non-persistent messages

Now let’s try to do all that again with non-persistent messages. To modify pyactivemq producer to send non-persistent messages, we have to uncomment the following line

#self.producer.deliveryMode = DeliveryMode.NON_PERSISTENT

in perftest.py library.

Now we can rerun our tests. For OpenWire protocol results now quite different

openwire non-persistent

showing that the whole throughput of non-persistent messages through the broker is around 3000 msg/sec.

Similarly, the results for Stomp are visualized in the diagram below.

stomp non-persistent

This shows that at around 2000 msg/sec the limitations of Stomp text-oriented protocol kicks in and limit the throughput.

To modify RabbitMQ client to send non-persistent messages, we have to comment the following line

msg.properties["delivery_mode"] = 2

in the PerfProducer class.

Results now look like this

amqp non-persistent

which is pretty much the same as in the persistent case. This is probably due to the nature of queuing and persisting messages in the broker discussed above. The only difference is that now the broker keep the rates at steady numbers of around 3200 msg/sec for producer and 950 for consumer.

Scalability

I also tried a simple scalability test against two brokers: try to send a message to as many queues as you can. The test source can be found in the appropriate test_scale.py file for both py-amqplib and pyactivemq libraries.

When you run appropriate

python test_scale.py

against RabbitMQ you’ll get that the message was sent to the around 32000 queues before it crashes, which is very good result.

ActiveMQ, again, comes with the predefined configuration file which should be used if you want to scale your broker (conf/activemq-scalability.xml). As it’s stated in the configuration file, there a couple of minor changes needed to be made to the startup script in order to achieve maximum scalability. We’ll use these

ACTIVEMQ_OPTS="-Xmx1024M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"

properties in order to give broker a bit more memory and turn of dedicated task runner.

When started with the configuration like this, the

python test_scale.py

shows that message was sent to impressive 64000 queues before it crashed.

We can try the scalability of Stomp protocol as well. In order to do that, we should add

<transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613"/>

to the list of available connectors.

Now we can start the test again

python test_scale.py stomp

and we’ll get the result very similar to the one of the default OpenWire connector.

Conclusion

To conclude, both ActiveMQ and RabbitMQ are decent brokers that will serve their purpose well in normal conditions, but put to their extremes in terms of throughput, scalability and reliablilty, ActiveMQ currently outperforms RabbitMQ for messaging usage in Python. All tests performed here are done with honest intentions of providing realistic results. If you spot any problems or have any ideas what more can be tested (transactions, etc.), please let me know.

Apache ActiveMQ 5.3.0 Released

After more than a year in development, we finally released 5.3.0 version of ActiveMQ. It contains over 300 bug fixes, but also some of the neat new features. The full list of features and bug fixes, as well as the download link can be found at the release page

Here, I would like to emphasize just some of the new features:

  • New KahaDB persisten store, which improves recovery times and scalability in a great deal
  • A default configuration now focused more on production then demoing all features. You can also find some useful configuration examples for different scenarios, like vertical scaling or throughput
  • Camel 2.0.0 integrated along with it’s web console for managing endpoints and routes
  • … and much, much more

You can expect more useful resources on ActiveMQ 5.3.0 in coming days.

XML to JSON

While working on Jettison, I often have need to convert XML to JSON. Here’s a simple method that uses XStream and Jettison to do this conversion.

import java.io.StringReader;
import java.io.StringWriter;

import com.thoughtworks.xstream.io.HierarchicalStreamReader;
import com.thoughtworks.xstream.io.HierarchicalStreamWriter;
import com.thoughtworks.xstream.io.copy.HierarchicalStreamCopier;
import com.thoughtworks.xstream.io.json.JettisonMappedXmlDriver;
import com.thoughtworks.xstream.io.xml.XppReader;

public class Test {

	public static void main(String[] args) throws Exception {

		String xml = "ToveJaniReminder" +
				"Don't forget me this weekend!";

        HierarchicalStreamReader sourceReader = new XppReader(new StringReader(xml));

        StringWriter buffer = new StringWriter();
        JettisonMappedXmlDriver jettisonDriver = new JettisonMappedXmlDriver();
        jettisonDriver.createWriter(buffer);
        HierarchicalStreamWriter destinationWriter = jettisonDriver.createWriter(buffer);

        HierarchicalStreamCopier copier = new HierarchicalStreamCopier();
        copier.copy(sourceReader, destinationWriter);

        System.out.println(buffer.toString());
	}

}

All conversion problems, like lists representation are handled by Jetttison. This example uses Mapped convention, but of course, BadgerFish notation can be used as well.

Stomp Future

While thinking about the future of Stomp protocol, we concluded that the best place for future maintenance of the specification is under umbrella of ActiveMQ project. There are a couple reasons for this, but it basically boils down to two facts:

  • Apache infrastructure is much better suited for this task than the one currently used at Codehaus
  • Most of the Stomp contributors are also ActiveMQ committers, so this transition feels natural in this respect.

As a starting point, we created an ActiveMQ sub-project http://activemq.apache.org/stomp/ which will be the home of the all future work on the protocol. At the moment, we only ported information related to Stomp version 1.0 (specification and related articles) to this new project. Soon, we’ll start working on protocol specification version 1.1, using the current idea pool as a base.

The whole documentation for the project is created using Webgen, so it consists of Markdown pages stored in an appropriate SVN repository (for example, Stomp 1.0 specification is located in here). You can also use Jira to file any tasks, issues and requests for the specification related work. So please feel free to contribute :)

This project is not meant to contain any Stomp clients, but only to be a home of the Stomp specification. This doesn’t mean that we don’t think about how to provide the better infrastructure to Stomp clients developers. But that is the topic for another discussion and you can expect more news regarding that soon.

pyactivemq on Ubuntu

pyactivemq is a Python library that enables Python clients to communicate with ActiveMQ using both OpenWire and Stomp protocols. To do that it wraps ActiveMQ-CPP library, so it involves some compiling to make it all work. While building instructions for both ActiveMQ-CPP and pyactivemq and clear and concise, I found some additional steps were need to make it all work on Ubuntu box. I’ll post it here for my future reference and hopefully it can be useful to other folks.

This is tested with ActiveMQ-CPP version 2.2.6 and activemqpy 0.1.0

Let’s start with ActiveMQ-CPP. ActiveMQ-CPP depends on a couple of libraries listed on the wiki page. You can easily obtain them using something like:

$ sudo apt-get install autoconf
$ sudo apt-get install automake
$ sudo apt-get install libtool
$ sudo apt-get install uuid-dev
$ sudo apt-get install libcppunit-dev

But you also need to install the build-essential package with

sudo apt-get install build-essential

This is now added to the wiki as well

Before starting the build, you’ll also need to install APR-util and APR libraries. Steps are pretty forward, so I’ll skip them here.

Now we’re finally ready to start the ActiveMQ-CPP build. By following instructions on the wiki page, you should have ActiveMQ-CPP installed into /usr/local/include/activemq-cpp-x.x.x/.

So, let’s get back to the pyactivemq. First thing you need to do is to edit setup.py and point it to the location where ActiveMQ-CPP is installed. You can do that by changing all /opt/activemq-cpp-x.x.x references to your /usr/local/include/activemq-cpp-x.x.x/

For successful build of pyactivemq you need to install one more dependency (Boost.Python) and you can do that with the following command:

sudo apt-get install libboost-python-dev

After these steps you can build and install pyactivemq.

There is just one more thing to do in order to successfully run scripts that use pyactivemq. You need to add ActiveMQ-CPP location to the list of shared libraries. To do that add the following line:

/usr/local/include/activemq-cpp-x.x.x/

to the /etc/ld.so.conf file and run

$ sudo ldconfig

It’s finally ready. To test it, start your broker, go to the src/examples directory and run

$ python asynclistener.py

Enjoy writing your Python application using ActiveMQ and pyactivemq.