Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2046

Source:file module read file line by line

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0 RC1, 1.0 GA
    • Fix Version/s: Waiting for Triage
    • Component/s: Stream Module
    • Labels:
      None

      Description

      I have a stream that watch output of multi file in a directory, process data and put it to HDFS. Here is my stream creat command:

      stream create -name fileHdfs --definition "file --dir=/var/log/supervisor/ --pattern=tracker.out*.log --outputType=text/plain | logHdfsTransformer | hdfs --fsUri=hdfs://192.168.1.115:8020 --directory=/data/log/appsync --fileName=log --partitionPath=path(dateFormat('yyyy/MM/dd'))" --deploy

      Problem is source:file module send all data read from file to log processing module instead of one line each turn, becase of that, payload string have millions of char, i can't process it. Ex:

      — PAYLOAD LENGTH---- 9511284

      Please tell me how to read line by line when use source:file module, thanks !!!

        Attachments

          Activity

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            phamptu Tu Pham
            Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: