Spring Batch
  1. Spring Batch
  2. BATCH-540

Skippable items : getting item read even if errors occured on read

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: 1.0.0
    • Fix Version/s: 1.1.0
    • Component/s: Infrastructure
    • Labels:
      None

      Description

      SkipListener allows us to get the skipped item when the exception is thrown during the writing process (SkipListener#onSkipInWrite(Object,Throwable)).

      It would be interesting to get the skipped item even if the exception is thrown during the read process (this can be common when using a ValidatingItemReader).

      One solution would be to have an aspect adding every object read in a step context - the context would be available for skipListener.
      Another solution is using an itemReader decorator instead of an aspect as Dave suggested.
      The aspect solution is perhaps intrusive.
      The decorator solution can clutter configuration (validatingReader -> itemRecorderReader -> fileInputReader) and perhaps less powerfull (we can only record itemReaders). If we want to record FieldSetMapper, we should create a FieldSetRecorderMapper ?

      see also http://forum.springframework.org/showthread.php?t=51947

        Activity

        Hide
        Lucas Ward added a comment -

        I understand the motivation for this issue, but I can't see how we could implement it. An ItemReader either returns an Item or throws an exception. There was a recent change to the validator to throw an exception with the item in it which can help. There is also BATCH-554, which would let you move the validation to the ItemWriter but not cause a rollback for validation failures.

        Show
        Lucas Ward added a comment - I understand the motivation for this issue, but I can't see how we could implement it. An ItemReader either returns an Item or throws an exception. There was a recent change to the validator to throw an exception with the item in it which can help. There is also BATCH-554 , which would let you move the validation to the ItemWriter but not cause a rollback for validation failures.
        Hide
        Anatoly Polinsky added a comment -

        "An ItemReader either returns an Item or throws an exception" - that is true for most of the cases (simple cases, really). However imagine a "smart" ItemReader, that composes one item of off several reads. Here is an example:

        • Each item has a "start section" / "end section" identifiers
        • Each section can consist out of "one" to "several" lines - due to a singe line length limitaion (e.g. usually 80 characters for mainframes, or other). If it is one line - that means it is "start" and "end" section in one.
        • A full item to be passed on to the ItemWriter would be all lines that capture all sections from "start" to "end"
        • ItemReader reads the first 80 bytes (since it can be the only line to represent a full item), buffers it.
        • Figures out whether it needs to read more to get a full item
        • Reads more, adds it to the buffer
        • And so on, until it gets the full item to give to ItemWriter to process (write).

        Now if the exception is raised during reading, there is a high probability that ItemReader has a well defined (now) partial item, which by the way can be enough to identify the whole record (e.g. it can have a messageID, SSN, etc...).

        Hence the ItemReaderListener would be able to take this "partial item" in "onReadError" method.

            • The above is not an "imaginary case" - it is something I had to solve on a client site.
        Show
        Anatoly Polinsky added a comment - "An ItemReader either returns an Item or throws an exception" - that is true for most of the cases (simple cases, really). However imagine a "smart" ItemReader, that composes one item of off several reads. Here is an example: Each item has a "start section" / "end section" identifiers Each section can consist out of "one" to "several" lines - due to a singe line length limitaion (e.g. usually 80 characters for mainframes, or other). If it is one line - that means it is "start" and "end" section in one. A full item to be passed on to the ItemWriter would be all lines that capture all sections from "start" to "end" ItemReader reads the first 80 bytes (since it can be the only line to represent a full item), buffers it. Figures out whether it needs to read more to get a full item Reads more, adds it to the buffer And so on, until it gets the full item to give to ItemWriter to process (write). Now if the exception is raised during reading, there is a high probability that ItemReader has a well defined (now) partial item, which by the way can be enough to identify the whole record (e.g. it can have a messageID, SSN, etc...). Hence the ItemReaderListener would be able to take this "partial item" in "onReadError" method. The above is not an "imaginary case" - it is something I had to solve on a client site.
        Hide
        Lucas Ward added a comment -

        The problem still stands, either the ItemReader reads an item successfully, or it throws an exception. I understand the use-case where an 'item' may be composed of several elements, I have also ran into it at clients, and the Multi-line samples jobs illustrate it one way as well. However, from the framework standpoint, either read returns successfully or it does not. If it doesn't, the framework has no way to know what the item was, nor anyway to determine what boundary the 'partial item' would follow, since it would have to be entirely an application concern. My recommendation would be to not return from your ItemReader until you have a completely composed item, and if you have an exception, to be sure you include all relevant information in your exception before throwing. Either that or log out the error in your ItemReader directly.

        Show
        Lucas Ward added a comment - The problem still stands, either the ItemReader reads an item successfully, or it throws an exception. I understand the use-case where an 'item' may be composed of several elements, I have also ran into it at clients, and the Multi-line samples jobs illustrate it one way as well. However, from the framework standpoint, either read returns successfully or it does not. If it doesn't, the framework has no way to know what the item was, nor anyway to determine what boundary the 'partial item' would follow, since it would have to be entirely an application concern. My recommendation would be to not return from your ItemReader until you have a completely composed item, and if you have an exception, to be sure you include all relevant information in your exception before throwing. Either that or log out the error in your ItemReader directly.
        Hide
        Lucas Ward added a comment -

        As mentioned in my last comment, I don't see anyway that this feature could be implemented. It still seems to make more sense to either move validation to the writer (and not rollback for validation exceptions) or to simply log out the validation exception in the reader itself. Closing as 'Won't Fix'

        Show
        Lucas Ward added a comment - As mentioned in my last comment, I don't see anyway that this feature could be implemented. It still seems to make more sense to either move validation to the writer (and not rollback for validation exceptions) or to simply log out the validation exception in the reader itself. Closing as 'Won't Fix'

          People

          • Assignee:
            Unassigned
            Reporter:
            adrian
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: