RabbitMQ Quorum queues definition arguments

I recently went through the process of migrating some of our classic queues to the new quorum queues and in this post I will explain some of the arguments that we chose for the RabbitMQ queue definition and why we decided on which one to choose.

For those who are not aware, the RabbiMQ classic queues were deprecated and are planned to be removed in RabbitMQ version 4.

RabbitMQ Quorum queues

So, for most use cases, the classic queues will be replaced by the Quorum queues. Quorum queues are optimized for set of use cases where data safety is a top priority. So you will replace all those ha-mode policies with a queue made with data safety in mind.

Quorum queues are durable, replicated FIFO queues based on the Raft consensus algorithm. It was first available in version RabbitMQ 3.8.0.

This guide considers that you use at least RabbiMQ version 3.10 as there are a lot of improvements in this version compared to the 3.8. Also, some arguments will only work on the 3.10 forward.

RabbitMQ Quorum queues arguments

Quorum queues share most of the arguments of Classic queues but a few differ or have limited options.

To easily see the difference, let’s take a look at what the Dashboard shows when creating the different queues.

The image below shows the argument options for a Classic queue.

And the image below shows the ones for the Quorum queues

As you can easily notice you don’t have the durable option, as Quorum queues are always durable.

Let’s see some of the arguments and sample values that I used. Of course, they will vary depending on your use case, but take it as an example.

x-overflow

Official definition

This determines what happens to messages when the maximum length of a queue is reached. Valid values are drop-head, reject-publish or reject-publish-dlx. The quorum queue type only supports drop-head and reject-publish."

Value selected: reject-publish
Why we need it: It defaults to drop-head so we want to change the behavior. If we use drop-head the message will be published successfully and the publisher will confirm it, as it is going to be delivered to the queue and the first message will be dropped to the DLQ. Using reject-publish allows the publisher to know that the message was not delivered to all queues as it will be rejected. This way the consumer can handle it in the of their choosing. The overflow size is defined by x-max-length-bytes below.

x-max-length-bytes

Official definition

Total body size for ready messages a queue can contain before it starts to drop them from its head.

Value selected: 1_073_741_824 (1GB) Why we need it: to protect Brokers from overload. This prevents the queue from getting too big and impacts the disk space of the node.

x-message-ttl

Official definition

How long a message published to a queue can live before it is discarded (milliseconds).

Value selected: 604_800_000 (7 days) Why we need it: To avoid long lasting messages. This is only meant to be a safeguard, if the message spends a long time in the queue like 7 days, it is probably due to some problems and there are high chances that it is not needed, so having an expiry helps us remove it from the queue.

x-delivery-limit

Official definition

The number of allowed unsuccessful delivery attempts. Once a message has been delivered unsuccessfully more than this many times it will be dropped or dead-lettered, depending on the queue configuration.

Value selected: 10 Why we need it: To avoid poison messages. If the message is not delivered for 10 retries, it is probably not deliverable so that we can route it to the DLQ.

x-dead-letter-exchange

Official definition

Optional name of an exchange to which messages will be republished if they are rejected or expire.

Value selected: mainDeadLetterExchange Why we need it: To be able to route messages that are dropped from the queues. This exchange is responsible for receiving all rejected or expired messages and will be responsible for routing then to the dead letter queues.

x-dead-letter-routing-key

Official definition

Optional replacement routing key to use when a message is dead-lettered. If this is not set, the message’s original routing key will be used.

Value selected: ${service-name}-dlq

Why we need it: to route the messages to the correct queue. Here you can go with one queue for all services or one queue per service. We decided to go with one queue per microservice as it microservice has an owner team and they would be responsible for it.

x-dead-letter-strategy

Official definition

Valid values are at-most-once or at-least-once. It defaults to at-most-once. This setting is understood only by quorum queues. If at-least-once is set, Overflow behaviour must be set to reject-publish. Otherwise, dead letter strategy will fall back to at-most-once.

Value selected: at-least-once Why we need it: The default value at-most-once can lose some messages, so we decided to go with at-least-once which can deliver duplicated messages but does not lose it.

RabbitMQ Quorum definitions

Just an example, below are the example definitions using Spring and also javascript.

The Spring example is based on Spring Boot 2.7.x and spring-boot-starter-amqp

@Bean
public Queue quorumQueue() {
 return QueueBuilder.durable("queue-name")
   .quorum()
   .withArgument("x-overflow", "reject-publish")
   .withArgument("x-message-ttl", Duration.ofDays(7).toMillis())
   .withArgument("x-max-length-bytes", 1_073_741_824)
   .withArgument("x-delivery-limit", 10)
   .withArgument("x-dead-letter-exchange", "mainDeadLetterExchange")
   .withArgument("x-dead-letter-strategy", "at-least-once")
   .withArgument("x-dead-letter-routing-key", "queue-name-dlq")
   .build();
}

The javascript example uses the library amqplib

await channel.assertQueue('queue-name', {
 overflow: 'reject-publish',
 messageTtl: 604800000,
 deadLetterExchange: 'mainDeadLetterExchange',
 deadLetterRoutingKey: 'queue-name-dlq',
 arguments: {
  'x-queue-type': 'quorum',
  'x-dead-letter-strategy': 'at-least-once',
  'x-delivery-limit': 10,
  'x-max-length-bytes': 1073741824,
 },
})

I hope these examples help you have a good RabbitMQ experience.

Happy coding!