Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-3100

module.*.count > 1 duplicates messages on taps

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.1
    • Fix Version/s: None
    • Component/s: Runtime
    • Story Points:
      5
    • Rank (Obsolete):
      9223372036854775807
    • Acceptance Criteria:
      Hide

      Using module.name.count > 1 should not duplicate messages when used in taps while deploying it with these specific properties.

      Show
      Using module.name.count > 1 should not duplicate messages when used in taps while deploying it with these specific properties.

      Description

      Using module.name.count > 1 when deploying taps causes duplication of messages in those modules. This impacts balancing of the containers and modules in a cluster as messages should not be duplicated across modules if the same module is deployed twice to two containers in order to spread the load.

      We use taps quite heavily in our project mainly for analytics of the life feed in real time but due to issue we have discovered and described in this bug we are currently facing a limitation where heavily processing modules can not be load balanced across the cluster as they are causing duplication of the messages and therefore the same module deployed to two
      containers would still process the same message twice.

      To demonstrate the problem please see test case scenario set up below:

      1. Environment

      • Spring-XD version 1.1.1-RELEASE
      • Running two spring-xd containers and one spring-xd admin

      2. Set up

      Stream definition is as follows:

      stream create --name test-module-count --definition "syslog-udp --port=5140 | transform | log"
      stream deploy --name test-module-count --properties "module.*.count=2"
      stream create --name tap-test-module-count --definition "tap:stream:test-module-count.syslog-udp > transform --expression='payload.toString() + \"TAPPED\"' | log"
      stream deploy --name tap-test-module-count --properties "module.*.count=2"

      Please refer to the screen shots attached to see that after deploying those two streams we have:

      • streams successfully deployed ( module-count-spring-xd-streams.png )
      • streams successfully deployed with count=2 to both containers ( module-count-spring-xd-containers.png )
      • 5 queues created in Rabbit ( module-count-rabbit.png ) where two were created for the syslog-udp collector as a result of using module.syslog-udp.count=2 - this is causing messages to be duplicated. Normally the expectation would be to have only one queue for the tap

      3. Test input data

      I have sent a very simple UDP message to the listening udp collector running on second container:

      echo test-module-count >> /dev/udp/host02/5140

      4. Test output data in the logs ( module-count-container01.log and module-count-container02.log )

      Expected result:

      Below messages logged only on 1 container (it does not matter which one)

      2015-05-26 09:52:21,630 1.1.1.RELEASE INFO xdbus.test-module-count.1-1 sink.test-module-count -

      Unknown macro: {UNDECODED=test-module-count}

      Below message logged only on one container (it does not matter which one)

      2015-05-26 09:52:21,843 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count
      }TAPPED

      Actual result:

      Stream that has been create as a tap has duplicated the same message and as a result the same message was proccessed twice on both containers by the same module ( transformer ) and logged twice to the console on both containers

      Container01:

      2015-05-26 14:52:21,143 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count
      }TAPPED

      Container02:

      2015-05-26 09:52:21,630 1.1.1.RELEASE INFO xdbus.test-module-count.1-1 sink.test-module-count - {UNDECODED=test-module-count
      }
      2015-05-26 09:52:21,843 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count
      }TAPPED

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              lukasz.nowanski@emc.com Lukasz Nowanski
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: