If you have a hammer everything looks like a nail, right? So we all witnessed that people sometimes try to solve the problem with wrong technology. Heck we probably all did it at one point or another. Common reasons are familiarity with an exiting technology stack we have at hand or perception that some of the features will justify it all. But using any technology in a way it’s not designed for, will lead to all kind of problems and you’ll eventually be forced to do it properly (probably cursing at the project in question because it hasn’t met you wrong expectations).
In this and couple of follow-up posts I’ll try sum up some things we saw in mailing lists, Jiras, etc related to improper usage of messaging systems (ActiveMQ in particular). Hopefully, it will help people that consider using messaging in their architecture, see if it is the right tool for solving their particular problem.
So let’s kick off with one common mistake people make
Using message queue as a database
Messaging systems are built to asynchronously connect multiple systems, by passing messages between them. So everything is designed with that in mind; how to most efficiently pass messages from producers to consumers. This means that messages are expected to be reasonably short-lived, and not stored in a queue.
From time to time we see people trying keep application state in the broker. Put some messages in a queue, than browse them, cherry-pick just some of them, delete others and similar stuff. While most of the messaging systems have some kind of support to do this it’s not what they’re designed to support primarily. Client APIs, internal storage system, client-server contracts, etc. are optimized for entirely different set of tasks.
An example could be a system that wants to keep a single most-recent data as a queue message (timestamped or versioned somehow). So that application can find that message (usually by browsing) and do the house keeping by deleting stale data. People are sometimes inclined to do this as brokers provide high-availability, reconnection logic, can be geographically distributed which is all fine and well. But brokers are a poor choice for maintaining application state.
A workaround is not to keep any state in the broker, of course. Either use a centralized high-availability database or a local copy of data and use messaging system to propagate changes.
So if you find yourself wanting to store some messages in a queue and then later browse them, query them or maintain them, please don’t. Get yourself a database of some kind (relational or not) and manipulate data there. Queues (and topics) are for data that should be consumed as they come and moved from one system to another as fast as possible. They should live in the broker only as long as it takes to consume them or if something has gone wrong and they cannot be consumed. Which should be an exceptional situation rather than what we design for.
If there’s any messaging anti-patterns you observed (or designed yourself – c’mon don’t be ashamed), send them to me. I’ll gladly put them on my list and document them in coming days.
I’m guilty. Actually, I use the message broker exactly how it is intended, except for one thing. My app tracks states of progress. When the status changes, my client app get notified and it lights up some indicators. Works great! But what if the user stops and relaunches my app? There’s no history of progress because all the messages have been acknowledged. So my apps sits blankly until another status message arrives. Users don’t like this. They want to see the latest state of progress instantly. (I get paid when users are happy! Then I’m happy too!)
Ah, so if I don’t ack the messages, they’ll be redelivered upon restart. I’m so clever!! Then one day, my all instances of my app hang. Everything interacting with ActiveMQ is locked up! WTF! I’m not so smart now! Kahadb shows lots and lots of db-N.log files consuming all free space. Not acknowledging messages causes ActiveMQ to save everything indefinitely even messages that expired.
How do I solve this problem? Is there a config setting that tells ActiveMQ to go ahead and clean-up unack’d messages if they’ve expired? If not, what to you recommend to fix my dilemma?
Mark
Hi Mark, I saw your question on mailing list as well. Looks to me it’s sorted. Cheers
I agree queues shouldn’t be used like databases, however, I strongly believe that queues should be very reliably stateful in a DB-ish way.
The use case for this is passing state between separate systems. The sender sends something and notes that is done, it could get rid of that state now if it wanted. The message passes through the messaging system (durable/persistent) and gets read by the recipient(s) and popped into their databases.
There is a time when the only place the business state is living is in the messaging system. At this point, for a short while the messaging system is a system of record.
Now this point of view irritates some people. They say that the end points should be able to replay, etc. But frankly, if I can do that, why am I using middleware UDP would do 🙂 Doing it this way makes it easier for ‘corporate programmers’.
Also, SEDA kind of depends on this pattern.
On the same point its also useful if you can replay already read messages (something almost no JMS brokers do).
Fantastic for disaster recovery (e.g. SA does a Drop Table in prod and the backups are a few hours old).
Cheers
John
Hi John, thanks for the comment. Of course, I agree that durability and reliability are crucial parts of the message brokers. I argued just not to use it as a database.
Also, replaying messages is a valid use case. ActiveMQ already supports archiving journal files, which should be enough to replay all the state. We still miss necessary tools. Hopefully we’ll find resources to implement them soon.
Does delaying message deliver count as incorporating state in the queue?
For example, if I want a message to not be delivered for 1 day, and I put it on the queue with setDeliveryDelay to 24 hours.
Note: I work for CodeStreet
Our ‘ReplayService for JMS’ product is built exactly for this use case: effectively search and retrieve previously published messages (n-times delivery). A JMS message queue is really built for a rather time-constrained 1-time delivery.
We had customers asking for a replayable audit of all their messages over the last 5 days or 3 months or so. Given today’s message volume, this could run into hundreds of gigabytes and given today’s throughput, databases were way too slow and expensive.
See http://www.codestreet.com/marketdata/jms/jms_details.php for further details. Don’t hesitate to contact me on technical questions: axel.podehl@codestreet.com