Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2996

Align Spring XD partitioning with Kafka partitioning for the Kafka message bus



    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: None
    • Fix Version/s: 1.2 GA
    • Component/s: None
    • Labels:


      As a developer, I want that the Spring XD partitioning process targets Kafka bus partitions directly, so that the design of my stream processing application is easier to understand and the order of messages is not altered

      Current situation

      • Spring XD partitioning logic that builds on top of Kafka partitioning;
      • The number of Spring XD partitions is not explicitly configured (it's inferred from the number of consumer modules)
      • If the concurrency of the consumer modules is 1, then Spring XD partitions are matched 1:1 with Kafka partitions;
      • If the concurrency of the consumer modules is n, then a Spring XD partition uses n Kafka partitions, and the message bus distributes messages across the Kafka partitions that correspond to the Spring XD partitions;
      • this could be confusing to the end user, especially if they are used to the Kafka partitioning process;
      • this can also lead to changes of ordering between messages, as messages within the same Spring XD partitions will be sent to different Kafka partitions (this only happens if the concurrency of the receiving module is higher than 1)


      • For the Kafka message bus the number of Spring XD partitions does not need to be equal to the number of modules (must be higher or equal, though, so that consumers can be created), and should be configured explicitly - using the `partitionCount` property - (as an option, the module count * concurrency can be used as a default)
      • as a result, in the case of Kafka there will always be a 1:1 match between Kafka partitions and Spring XD partitions, optionally processed by fewer modules than the partition count;




            mbogoevici Marius Bogoevici
            mbogoevici Marius Bogoevici
            0 Vote for this issue
            1 Start watching this issue