Spring Batch
  1. Spring Batch
  2. BATCH-194

Incorrect exception handling when using Hibernate

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.0-m3
    • Fix Version/s: 1.0-m3
    • Component/s: Core
    • Security Level: Public (Public Issues)
    • Labels:
      None

      Description

      ItemProcessor uses Hibernate for persitence. During processing of chunk the Database raises an exception (i.e. duplicate key - DataIntegrityViolationException).

      Error description:
      Hibernate caches all updates until commit point. This causes that the database exception is raised during database commit (in TransactionTemplate) not in ChunkOperations.iterate(). ExceptionHandler in ChunkOperatons (RepeatTemplate) cannot handle this exception because it occurs after chunk is finnished (this is not synchronous with processing of the bad record).
      This way we are unable to handle bad records (to skip them) and the whole batch job fails.

      See example in attached zip file (hibernate job).

        Issue Links

          Activity

          Hide
          Dave Syer added a comment -

          This presents a similar problem to BATCH-76. Any approach that works there will probably work here.

          Peter: can you attach a patch with a test case that fails? The attached .zip file looks like the whole samples project, so it's hard to know what is important and what is not.

          Show
          Dave Syer added a comment - This presents a similar problem to BATCH-76 . Any approach that works there will probably work here. Peter: can you attach a patch with a test case that fails? The attached .zip file looks like the whole samples project, so it's hard to know what is important and what is not.
          Hide
          Robert Kasanicky added a comment -

          Attached patch is sufficient to reproduce the issue (HibernateJobFunctionalTests is the failing testcase)

          Show
          Robert Kasanicky added a comment - Attached patch is sufficient to reproduce the issue (HibernateJobFunctionalTests is the failing testcase)
          Hide
          Dave Syer added a comment -

          Added my own HibernateFailureJobFunctionalTests - a simpler version of the patch. Commented out the assertion that shows that it fails for now.

          Show
          Dave Syer added a comment - Added my own HibernateFailureJobFunctionalTests - a simpler version of the patch. Commented out the assertion that shows that it fails for now.
          Hide
          Dave Syer added a comment -

          Added some extra features to the HibernateCreditWriter sample - it needs to be a RepeatInterceptor (as per Joris's solution in BATCH-76) so it can force a flush inside the RepeatContext of the chunk.

          The skip is still impossible, but we are getting to the point where we can think about it properly. What we need is a CompletionPolicy that kicks into binary search mode on a failure, and zooms in on the failed records so they can be skipped individually. Easy to design the policy, hard to apply it because it has to be bound to the next repeat context (not the current one). "Next" is meaningless since there is no guarantee that an ItemProvider will provide the same items in the same order when we come back, or that the failed items will not be processed in another thread.

          Show
          Dave Syer added a comment - Added some extra features to the HibernateCreditWriter sample - it needs to be a RepeatInterceptor (as per Joris's solution in BATCH-76 ) so it can force a flush inside the RepeatContext of the chunk. The skip is still impossible, but we are getting to the point where we can think about it properly. What we need is a CompletionPolicy that kicks into binary search mode on a failure, and zooms in on the failed records so they can be skipped individually. Easy to design the policy, hard to apply it because it has to be bound to the next repeat context (not the current one). "Next" is meaningless since there is no guarantee that an ItemProvider will provide the same items in the same order when we come back, or that the failed items will not be processed in another thread.
          Hide
          Dave Syer added a comment -

          Added step operations to RepeatContextHolder (and changed name of implementation so it isn't tied to chunks). This was needed anyway, otherwise you can't control the exception handling in the step operations properly.

          Had to make HibernateCursorInputSource Skippable. Plus the samples were configured to use SimpleStepExecutor which doesn't skip by default (makes me wonder if the restart sample actually does anything sensible), so upgraded to DefaultStepExecutor.

          Added Restartable capabilities to HibernateCursorInputSource.

          Created HibernateAwareItemWriter to abstract the flushing concerns into a framework class - to use it you have to register it with the chunkOperations as an interceptor and inject it with an ItemWriter for the step concerned. There might be an AOP approach to this that would reduce the burden of having to confugure it - we can look into that as part of the namespace / DSL work. See hibernateJob.xml for sample.

          Please try it out and tell us if it works!

          Show
          Dave Syer added a comment - Added step operations to RepeatContextHolder (and changed name of implementation so it isn't tied to chunks). This was needed anyway, otherwise you can't control the exception handling in the step operations properly. Had to make HibernateCursorInputSource Skippable. Plus the samples were configured to use SimpleStepExecutor which doesn't skip by default (makes me wonder if the restart sample actually does anything sensible), so upgraded to DefaultStepExecutor. Added Restartable capabilities to HibernateCursorInputSource. Created HibernateAwareItemWriter to abstract the flushing concerns into a framework class - to use it you have to register it with the chunkOperations as an interceptor and inject it with an ItemWriter for the step concerned. There might be an AOP approach to this that would reduce the burden of having to confugure it - we can look into that as part of the namespace / DSL work. See hibernateJob.xml for sample. Please try it out and tell us if it works!
          Hide
          Jens Eickmeyer added a comment -

          The approach looks nice. Unfortunately, it does not work out-of-the-box in our environment. The reason is that we do not use implementations based on ItemWriter most of the time. Instead we are focused on reusing existing DAOs which are simply based on HibernateDaoSupport. Because these DAOs are also used in other applications we can not add any dependencies to Spring Batch. It would be very nice if a more general solution for this problem could be offered in the short term.

          Show
          Jens Eickmeyer added a comment - The approach looks nice. Unfortunately, it does not work out-of-the-box in our environment. The reason is that we do not use implementations based on ItemWriter most of the time. Instead we are focused on reusing existing DAOs which are simply based on HibernateDaoSupport. Because these DAOs are also used in other applications we can not add any dependencies to Spring Batch. It would be very nice if a more general solution for this problem could be offered in the short term.
          Hide
          Jens Eickmeyer added a comment -

          After talking to Lucas we have created a slightly different approach for ensuring that we can use Hibernate in our Batch scenario. We just flush the Hibernate session after processing each item. This could be done by implementing and configuring a RepeatInterceptor or by employing AOP. The attached file contains an advice that flushes the Hibernate session. It can be configured easily we the following Spring configuration:

          <bean
          id="hibernateFlushAdvice"
          class="org.springframework.batch.sample.advice.HibernateFlushAdvice"
          p:sessionFactory-ref="sessionFactory" />

          <aop:config>
          <aop:aspect
          id="hibernateFlushing"
          ref="hibernateFlushAdvice">

          <aop:after
          pointcut="execution( * org.springframework.batch.item.ItemProcessor+.process(Object))"
          method="doFlush" />

          </aop:aspect>
          </aop:config>

          This way it is not necessary to use the HibernateAwareItemWriter.

          Show
          Jens Eickmeyer added a comment - After talking to Lucas we have created a slightly different approach for ensuring that we can use Hibernate in our Batch scenario. We just flush the Hibernate session after processing each item. This could be done by implementing and configuring a RepeatInterceptor or by employing AOP. The attached file contains an advice that flushes the Hibernate session. It can be configured easily we the following Spring configuration: <bean id="hibernateFlushAdvice" class="org.springframework.batch.sample.advice.HibernateFlushAdvice" p:sessionFactory-ref="sessionFactory" /> <aop:config> <aop:aspect id="hibernateFlushing" ref="hibernateFlushAdvice"> <aop:after pointcut="execution( * org.springframework.batch.item.ItemProcessor+.process(Object))" method="doFlush" /> </aop:aspect> </aop:config> This way it is not necessary to use the HibernateAwareItemWriter.
          Hide
          Dave Syer added a comment -

          Assume closed as resolved and released

          Show
          Dave Syer added a comment - Assume closed as resolved and released

            People

            • Assignee:
              Dave Syer
              Reporter:
              Peter Zozom
            • Votes:
              3 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: