Messaging for IoT

Deploying message broker for an IoT use case introduces some new challenges to the broker scalability. We’re talking now about thousands of connections, consumers and destinations, which make us think of how we provision, configure and monitor our messaging infrastructure much more carefully. In this post I’ll try to sum up some of techniques that can be used with the current Apache ActiveMQ in terms to scale it better for IoT deployment. I’ll also describe some of the new features we developed for the 5.12.0 release to make it a better fit in this new world. And finally I’ll try to explain where we can go from here and what can we work on in the future.

ActiveMQ Vertical Scaling

The two most common messaging protocols used for IoT are MQTT and AMQP and we spent a significant effort to make them rock-solid in the latest releases. But protocols aren’t everything and every time someone starts play with the broker, the same question arise: How can I get the maximum scalability from the single broker instance? So, here are some tips:

  • Always start with the general broker scaling techniques. That basically means to try using as minimal threads as possible no matter how much connections and destinations your broker handles. So use NIO transport and turn of thread-per-destination setup.
  • The first implementation of MQTT protocol for ActiveMQ assumed that QoS1 and QoS2 subscribers are mapped internally to JMS durable subscribers. JMS durable subscribers have a heavy state and don’t scale very well. In later versions, you can opt for another implementation that uses virtual topics instead and should scale much better. 
    
<transportConnectors>
  <transportConnector name="mqtt+nio"
    uri="mqtt+nio://0.0.0.0:1883&transport.subscriptionStrategy=mqtt-virtual-topic-subscriptions"/>
</transportConnector>
  • One more reason to try the new 5.12.0 release is the new improvements of KahaDB message store that can now preallocate journal files. You can find more information about this in this post, but these tweaks on some file systems can gain significant performance improvements

    • All these little configuration tweaks are summed in the new example configuration file, which you can find in the 

    examples/conf/activemq-mqtt.xml

    file in the new 5.12.0 distribution.

    Vertically scaling the broker is only one part of the equation. There a couple more important questions that needs to be met for the successful IoT deployment.

    SSL

    Many IoT devices depend on SSL certificates for authentication purposes. This is not something new and we saw that in traditional messaging setups as well (and supported it), but the difference is, again, in the scale. It’s easy to manually maintain a keystore with a handful of certificates in it. It’s entirely different story when the number of certificates start to raise.

    In 5.12.0 we added some new features to help people deal with this. There are a couple of tools that are standardised for solving these problems and supported in JDK. So now, you can use Certificate Revocation List, which provides an easy way to revoke invalid certificates during the runtime.

    You can also use OCSP (Online Certificate Status Protocol) which provides even more automated way to communicate with your certificate authority. You can find more information about these features here.

    I think that SSL certificate provisioning is much bigger story for IoT deployments (and clouds in general) and there are already some interesting projects emerging solve it, like pki.io. We’ll try to support whatever people use in this space and right now with the support of CRL and OCSP you can have a little more flexibility when dealing with your certificates.

    Monitoring under stress

    One topic I often encounter when talking about IoT deployments is how to monitor the broker that is on its limits of the vertical scaling (whatever they are). We usually monitor the broker behaviour using JMX or advisory messages. While these are perfect tools when the broker is under its limits, things get less optimal when you’re on the edge.

    With large number of destinations and connections coming and going, registering MBeans and firing advisory messages can become very expensive, especially when done in high volume. It can get in the way of the actual work the broker needs to do.

    Folks that want to get maximum from their broker instance usually just turn everything off, like

    <broker useJmx="false" advisorySupport="false" ...>

    but that leave us with no insight in the state of the broker, beyond peaking into the logs.

    One solution we came up with is making an mbean registration more selective

    Starting with 5.12.0, you can define mbean names you want to suppress from registering, by defining them on the management context, like 

    <managementContext>
  <managementContext
    suppressMBean="endpoint=dynamicProducer,endpoint=Consumer,
                   connectionName=*,destinationName=ActiveMQ.Advisory.*"/>
</managementContext>

    Using this feature, you can customize what view of the broker you need and by suppressing registration/unregistration of high frequency ephemeral mbeans, you can help your broker deal with the load. So even in the high scale/load scenario, you can get the basic metrics of the broker. There is much more we can do in this space, by defining custom views and such, so stay tuned.

    Legacy MQTT

    Apache ActiveMQ implements MQTT 3.1.1 specification, but MQTT is not a new protocol and there is already a vast amount of devices deployed that use older (3.1) clients. We made an effort to enable known use cases where older clients expect different behaviour than what’s in the 3.1.1 specification. So for example, you can enable publishing to the “dollar topics” and see a difference in behaviour on unsuccessful subscription attempts. We’ll try to cover all these corner cases and provide the support for legacy clients where it is sensible thing to do so.

    ActiveMQ Artemis

    In case you weren’t paying attention, there have been a bit of a consolidation in the Java message brokers land. The HornetQ broker has been donated to the Apache and is now part of the ActiveMQ project. Its asynchronous core gives us a good base for the next generation ActiveMQ that should scale better and have better performances than the current broker. It already have an initial support for AMQP and MQTT protocols. With these protocols hardened and features above fully implemented, it should be a very good message broker for the backbone of your IoT infrastructure.

    Help from the friends

    Having a good message a broker is sure an important piece of the puzzle. But for truly large IoT deployments we’ll need more than that. We need to have a more complex infrastructure that will allow us to partition our traffic (in terms of connections, destination, etc.), provide fault tolerance and high availability capabilities. There are a couple of interesting projects that can help build an elastic messaging infrastructure for IoT needs.

    Qpid Dispatch Router provides a broker-less routing of messages between clients, brokers and other endpoints based on AMQP. It helps build optimal topologies and routing messages from clients to their final destination. For example, dispatch router can serve as gateway between clients and broker, helping the large number of connections or destinations be concentrated and partitioned over multiple brokers without client awareness. That’s only one of the examples where adding a router to your messaging network can help. It’s an interesting topic and you’ll hear much more interesting things in this space in the future.

    Fabric8 and OpenShift, on the other hand provide us with an easy way to provision and manage this messaging infrastructure. You can use them to easily deploy new brokers, routers, gateways and discover existing components. Fabric8 also provide a gateway that can be used to partition the traffic among the endpoints.

    There are a many ways to slice this problem and the final solution is certainly dependent on the actual use case. But having all these components rock solid and working nicely together is crucial in architecting the final solution.

    Conclusion

    With this post I tried to give some perspective on where we are and where we’re heading with the supporting IoT cases. I hoped you enjoyed it and found it useful. I’ll cover this topic in much more details at Voxxed Days Belgrade and ApacheCon in October. For the end, take a look at the awesome Red Hat Summit IoT demo, which shows some of the this stuff in practice.