Uploaded image for project: 'Spring Batch'
  1. Spring Batch
  2. BATCH-1850

Streaming API based ItemWriter for JSON, with restart support

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.1.8
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:

      Description

      Since JSON has become a widely adapted format among Integration technologies, a Streaming API based ItemWriter with restart support for JSON has to be developed. On the similar lines to StaxEventItemWriter.

      Usecase:

      1) Online Application initiating the batch job
      2) ItemReader reads the list of IDs to be process from underlying DB or Service
      3) ItemProcessor does further processing using DB and/or Service
      4) ItemWriter writes JSON file out of the processed records
      5) Callback mechanism would drop the file into underlying rabbitMQ
      6) Consumers would parse the JSON File for downstream applications.

      Writing a simple Itemwriter wouldnt suffice. As in most batch process the datasets could be enormous,so the job should support restarts.

      If the above has to be achieved with XML as the output format, it can be easily achieved using StaxEventItemWriter.

      A similar implementation for JSON would be highly valuable.

        Activity

        Hide
        david_syer Dave Syer added a comment -

        I didn't get the RabbitMQ piece (5), but can't you do everything with a FlatFileItemWriter plus a custom LineAggregator (and maybe header/footer callbacks to complete the JSON object)? We could definietly add one of those (to complement the existing JsonLineMapper).

        Show
        david_syer Dave Syer added a comment - I didn't get the RabbitMQ piece (5), but can't you do everything with a FlatFileItemWriter plus a custom LineAggregator (and maybe header/footer callbacks to complete the JSON object)? We could definietly add one of those (to complement the existing JsonLineMapper).
        Hide
        vasanth Vasanth Raj added a comment -

        RabbitMQ part isnt related to the issue, but mentioned it for use case to look complete, end to end. the output file needs to be placed in a Queue, probably a JobListener should do the job.

        Using the technique you suggested above should work for simple scenarios, but letting a parser do the work would make the solution more accommodating for future changes.

        eg: If there will be 2 item processors writing 2 different fragments to the same file ??

        Show
        vasanth Vasanth Raj added a comment - RabbitMQ part isnt related to the issue, but mentioned it for use case to look complete, end to end. the output file needs to be placed in a Queue, probably a JobListener should do the job. Using the technique you suggested above should work for simple scenarios, but letting a parser do the work would make the solution more accommodating for future changes. eg: If there will be 2 item processors writing 2 different fragments to the same file ??
        Hide
        vasanth Vasanth Raj added a comment -

        Apologies, I meant the approach wont accommodate changes such as multiple steps working on the same file.

        Show
        vasanth Vasanth Raj added a comment - Apologies, I meant the approach wont accommodate changes such as multiple steps working on the same file.

          People

          • Assignee:
            Unassigned
            Reporter:
            vasanth Vasanth Raj
          • Votes:
            2 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated: