Spring AMQP
  1. Spring AMQP
  2. AMQP-209

BlockingQueueConsumer.InternalConsumer does not override handleCancel which is called when consumer is cancelled by broker

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Complete
    • Affects Version/s: 1.0.0 GA, 1.1.0.GA
    • Fix Version/s: 1.1.0.GA
    • Component/s: Core
    • Labels:
    • Environment:
      RAbbitMQ Server 2.7.1 (earlier versions as well) on Linux.

      Description

      The rabbit server is allowed to issue a basicCancel to a consumer. Spring-AMQP should handle this and at minimum allow a user to redeclare queue, bindings and reconsume. See RabbitMQ docoumentation http://www.rabbitmq.com/amqp-0-9-1-quickref.html#basic.cancel.

      "It may also be sent from the server to the client in the event of the consumer being unexpectedly cancelled (i.e. cancelled for any reason other than the server receiving the corresponding basic.cancel from the client). This allows clients to be notified of the loss of consumers due to events such as queue deletion."

      basicCancel is issued from the broker to consumers in two important cases when using a cluster.

      1. When a cluster node goes down, all consumers which were consuming from this queue and not directly connected to the downed node are issued basicCancel. In order to conform to the behavior of consumers which ARE directly connected to the downed node, these consumers must redeclare the queues, bindings and reconsume. Current, in 1.0.1-SNAPSHOT on github, after this node is taken down, all consumers directly connected to this node are automatically reconnected by the CachingConnectionFactory. Consumers not directly connected silently stop consuming. This is a very serious shortcoming and is preventing us from going live with the spring-amqp client.

      2. basicCancel is issued from the broker to all consumers of an HA-queue on slave promotion to master. Current, in 1.0.1-SNAPSHOT on github, when a slave is promoted to master, all consumers not directly connected to the master node silently stop consuming. This is currently preventing us from using HA queues in production with the spring-amqp client. We are using spring "java config" and are thus not dependent on the current ongoing development to "Support Rabbit HA".

      "Clients that were consuming from the mirrored-queue and support our Consumer Cancellation Notifications extension will receive a notification that their subscription to the mirrored-queue has been abruptly cancelled. At this point they should re-consume from the queue, which will pick up the new master. The reason for sending this notification is that informing clients of the loss of the master is essential: otherwise the client may continue to issue acknowledgements for messages they were sent by the old, failed master, and not expect that they might be about to see the same messages again, this time sent by the new master. Of course, clients that were connected to the failed node will find their connections failed, and will need to reconnect to a surviving node of the cluster."

      I forked spring-amqp on github https://github.com/drstevens/spring-amqp/commit/187eb30066ee8125932866756a553d2ac95b2e92 in order to add overload to BlockingQueueConsumer.InternalConsumer.handleCancel. After reproducing, I now get debug logging identifying cancellation by the broker.

      11:47:52.018 [AMQP Connection xxx.xxx.xxx.xxx:5672] DEBUG o.s.a.r.l.BlockingQueueConsumer - Received cancellation notice for Consumer: tag=[amq.ctag-AcmfdKARPeoy5kA-s-1Lv6], channel=Cached Rabbit Channel: AMQChannel(amqp://guest@xxx.xxx.xxx.xxx:5672/,2), acknowledgeMode=AUTO local queue size=0

        Activity

        Show
        Dave Stevens added a comment - I have created a new thread on the forum regarding this. http://forum.springsource.org/showthread.php?122129-handling-basic-cancel%28consumer-tag-consumer-tag-no-wait-no-wait%29-sent-from-Server
        Hide
        Gary Russell added a comment -

        ...are thus not dependent on the current ongoing development to "Support Rabbit HA".

        I assume you are not using any of the rabbitTemplate .*sendAndReceive() methods because that effort also enables adding HA arguments to temporary reply queues.

        If you have interest in contributing to the project, please sign the CLA (https://support.springsource.com/spring_committer_signup) before issuing any pull requests.

        Thanks

        Show
        Gary Russell added a comment - ...are thus not dependent on the current ongoing development to "Support Rabbit HA". I assume you are not using any of the rabbitTemplate .*sendAndReceive() methods because that effort also enables adding HA arguments to temporary reply queues. If you have interest in contributing to the project, please sign the CLA ( https://support.springsource.com/spring_committer_signup ) before issuing any pull requests. Thanks
        Hide
        Dave Stevens added a comment -

        That is correct. We have kept it as basic as possible because we need to talk to Akka and a custom .NET client.

        I'm actually a little concerned about just calling the existing auto queue declaration logic in RabbitAdmin when these consumers get unexpectedly canceled.

        When a node goes down containing a durable non-HA queue with consumers, these consumers must not attempt to redeclare this queue. If they do, they will exception. I'm not sure what should happen. I would guess they should try again and keep retrying until that node with the original durable queue comes back up.

        I'm thinking that there is no generic auto-declaration strategy that can work across the board for all types of queues.

        Show
        Dave Stevens added a comment - That is correct. We have kept it as basic as possible because we need to talk to Akka and a custom .NET client. I'm actually a little concerned about just calling the existing auto queue declaration logic in RabbitAdmin when these consumers get unexpectedly canceled. When a node goes down containing a durable non-HA queue with consumers, these consumers must not attempt to redeclare this queue. If they do, they will exception. I'm not sure what should happen. I would guess they should try again and keep retrying until that node with the original durable queue comes back up. I'm thinking that there is no generic auto-declaration strategy that can work across the board for all types of queues.
        Show
        Dave Stevens added a comment - Please see response to this thread containing log output http://forum.springsource.org/showthread.php?122129-handling-basic-cancel(consumer-tag-consumer-tag-no-wait-no-wait)-sent-from-Server
        Hide
        Gary Russell added a comment -

        This was resolved in the recent 1.1.0 release but I inadvertently left the JIRA open.

        Show
        Gary Russell added a comment - This was resolved in the recent 1.1.0 release but I inadvertently left the JIRA open.

          People

          • Assignee:
            Gary Russell
            Reporter:
            Dave Stevens
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: