Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2755

Scala processor module executor trims messages

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: 1.1 GA
    • Fix Version/s: 1.1.1, 1.2 M1
    • Component/s: Runtime
    • Labels:
      None

      Description

      How to reproduce:

      1. Run xd-singlenode (for which setting the Spark master URL to 'local' is a requirement). Use more than 1 worker thread. e.g. local[4]

      2. Deploy the word-count example

      3. Create a stream
      stream create spark-streaming-word-count --definition "http | word-count | log" --deploy

      4. Send data
      xd:>http post --data "a b c d e f g"

      xd:>http post --data "a b c"

      5.Observe the result

      2015-02-24 15:12:46,018 1.2.0.SNAP INFO Executor task launch worker-3 sink.spark-streaming-word-count - (e,1)
      2015-02-24 15:12:46,018 1.2.0.SNAP INFO Executor task launch worker-1 sink.spark-streaming-word-count - (d,1)
      2015-02-24 15:12:46,019 1.2.0.SNAP INFO Executor task launch worker-2 sink.spark-streaming-word-count - (b,1)
      2015-02-24 15:12:46,020 1.2.0.SNAP INFO Executor task launch worker-1 sink.spark-streaming-word-count - (g,1)
      2015-02-24 15:13:40,020 1.2.0.SNAP INFO Executor task launch worker-1 sink.spark-streaming-word-count - (a,1)
      2015-02-24 15:13:40,020 1.2.0.SNAP INFO Executor task launch worker-2 sink.spark-streaming-word-count - (b,1)
      2015-02-24 15:13:40,021 1.2.0.SNAP INFO Executor task launch worker-3 sink.spark-streaming-word-count - (c,1)

      (the last three results are coming from the second invocation))

      Note: there seems to be a correlation between the number of values emitted and the number of workers, as, in all the attempts, there aren't more values emitted than the number of workers.

        Attachments

          Activity

            People

            Assignee:
            iperumal Ilayaperumal Gopinathan
            Reporter:
            mbogoevici Marius Bogoevici
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: