Details
-
Type:
Story
-
Status: Done
-
Priority:
Minor
-
Resolution: Complete
-
Affects Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:3
-
Rank (Obsolete):9223372036854775807
-
Pull Request URL:
-
Acceptance Criteria:
- Merge PR to produce SI Kafka SNAPSHOT build
- Upgrade XD's SI Kafka dependency
-
Sprint:Sprint 46
Description
The spark streaming message bus receiver isn't reliable yet. The receiver needs to handle data loss in case of worker node that has it running.
We currently handle the driver failure automatically by re-deploying spark streaming module. But, this is about the data loss when the worker node dies.
Please see the documents here:
http://spark.apache.org/docs/latest/streaming-custom-receivers.html