Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2748

Implement Reliable spark streaming receiver

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Priority: Minor
    • Resolution: Complete
    • Affects Version/s: None
    • Fix Version/s: 1.1.1, 1.2 M1
    • Component/s: None
    • Labels:
      None

      Description

      The spark streaming message bus receiver isn't reliable yet. The receiver needs to handle data loss in case of worker node that has it running.

      We currently handle the driver failure automatically by re-deploying spark streaming module. But, this is about the data loss when the worker node dies.

      Please see the documents here:

      https://databricks.com/blog/2015/01/15/improved-driver-fault-tolerance-and-zero-data-loss-in-spark-streaming.html

      http://spark.apache.org/docs/latest/streaming-custom-receivers.html

        Attachments

          Activity

            People

            Assignee:
            iperumal Ilayaperumal Gopinathan
            Reporter:
            iperumal Ilayaperumal Gopinathan
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: