Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2868

Support Partitioned Batch Jobs with a LocalMessageBus

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Priority: Minor
    • Resolution: Complete
    • Affects Version/s: None
    • Fix Version/s: 1.2 M1
    • Component/s: Runtime
    • Labels:
      None
    • Story Points:
      5
    • Rank (Obsolete):
      9223372036854775807
    • Acceptance Criteria:
      Hide

      The ability to run partitioned batch jobs on a LocalMessageBus with configurable concurrency.

      Show
      The ability to run partitioned batch jobs on a LocalMessageBus with configurable concurrency.
    • Sprint:
      Sprint 46

      Description

      Initial support for partitioned batch jobs (initially tested with a local bus) had an ExecutorChannel in the job context to enable multiple partitions to run. Otherwise, with a local bus, only one partition would run at a time.

      When further work was done to support other buses, this was removed and the bus was used to control partition concurrency.

      The LocalMessageBus was changed to use an unbounded task executor; this was wrong because now all partitions ran at once.

      Further changes to the local bus changed the task executor to be pooled, but with default properties that mean only one thread is used.

      Further, the pool configuration is bus-wide so you can't use that configuration to select the concurrency for an individual job.

      The bottom line is that the local bus is not suitable for partitioned batch jobs; it was not anticipated that it would be used for this scenario. With 1.0.x too many partitions run (all); with 1.1.x only one thread runs (by default).

      In the local bus, we need to use a configurable, dedicated, bounded task executor for each batch job.

        Attachments

          Activity

            People

            Assignee:
            grussell Gary Russell
            Reporter:
            grussell Gary Russell
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: