Spring Batch
  1. Spring Batch
  2. BATCH-940

Possible Memory Leak in 'prototype' data beans used by org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Minor Minor
    • Resolution: Invalid
    • Affects Version/s: 1.1.2
    • Fix Version/s: None
    • Component/s: Infrastructure
    • Labels:
      None
    • Environment:
      Red Hat Enterprise Linux AS release 4 (Nahant Update 4). Apache Tomcat/6.0.16. JVM 1.6.0_06-b02. Tomcat is run with -Xmx1024m option. Only 1 application is published on this Tomcat instance.

      Description

      I am using org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper to process 3 fixed length flat files in 1 job. The fixed length flat files has the following characteristics (no of lines/file size): File 1: 5000 lines/ 1MB, File 2: 120k lines/42MB and File 3: 510k lines and 210MB file size.

      The Job is configured as described by this forum's thread: http://forum.springsource.org/showthread.php?t=60421
      Each job run should use separate sub application context (created by ClassPathXmlApplicationContextJobFactory) which should be automatically destroyed after each run.

      For testing, I run the job once for every 2 hours. After repeated tried, out of memory error always occurs on the 4th run at the start of the step to process the 3rd flat file.:
      java.lang.OutOfMemoryError: Java heap space at org.apache.catalina.loader.WebappClassLoader.findResourceInternal(WebappClassLoader.java:2053) at org.apache.catalina.loader.WebappClassLoader.findResource(WebappClassLoader.java:934) at org.apache.catalina.loader.WebappClassLoader.getResource(WebappClassLoader.java:1069) at org.springframework.core.io.ClassPathResource.getURL(ClassPathResource.java:159) at org.springframework.core.io.ClassPathResource.getFile(ClassPathResource.java:174) at org.springframework.core.io.AbstractResource.exists(AbstractResource.java:51) at org.springframework.batch.core.resource.StepExecutionResourceProxy.exists(StepExecutionResourceProxy.java:112) at org.springframework.batch.item.file.FlatFileItemReader.doOpen(FlatFileItemReader.java:226) at org.springframework.batch.item.support.AbstractBufferedItemReaderItemStream.open(AbstractBufferedItemReaderItemStream.java:154) at org.springframework.batch.item.support.CompositeItemStream.open(CompositeItemStream.java:103) at org.springframework.batch.core.step.item.ItemOrientedStep.open(ItemOrientedStep.java:462) at org.springframework.batch.core.step.AbstractStep.execute(AbstractStep.java:167) at org.springframework.batch.core.job.SimpleJob.execute(SimpleJob.java:100) at org.springframework.batch.core.configuration.support.ClassPathXmlApplicationContextJobFactory$ContextClosingJob.execute(ClassPathXmlApplicationContextJobFactory.java:107) at org.springframework.batch.core.launch.support.SimpleJobLauncher$1.run(SimpleJobLauncher.java:86) at java.lang.Thread.run(Thread.java:619)

      Using Profiler4J, I can see that the used memory size is creeping up after each run until it fills up the available memory as shown on DGIS-DBIZLINK-MemoryProfile1.PNG

      After a restart, the memory is emptied (DGIS-DBIZLINK-MemoryProfile1.PNG between 5.40 and 5.50) but again the used memory size is creeping up again as shown on DGIS-DBIZLINK-MemoryProfile1.PNG (after 5.50) and DGIS-DBIZLINK-MemoryProfile2.PNG

      1. applicationContext-SpringBatch-Base.xml
        1 kB
        Vito Limandibhrata
      2. applicationContext-SpringBatch-Base-Parent.xml
        3 kB
        Vito Limandibhrata
      3. applicationContext-SpringBatch-Job-GISDEB.xml
        4 kB
        Vito Limandibhrata
      4. applicationContext-SpringBatch-Job-GISDEB-GISSOH-GISSOL.xml
        1 kB
        Vito Limandibhrata
      5. applicationContext-SpringBatch-Job-GISSOH.xml
        4 kB
        Vito Limandibhrata
      6. applicationContext-SpringBatch-Job-GISSOL.xml
        4 kB
        Vito Limandibhrata
      7. applicationContext-SpringBatch-Jobs.xml
        2 kB
        Vito Limandibhrata
      8. GZipUncompress.java
        2 kB
        Vito Limandibhrata
      9. ProcessCobolFlatFilesInSeparateApplicationContexts.java
        5 kB
        Vito Limandibhrata
      1. DGIS-DBIZLINK-MemoryProfile1.PNG
        124 kB
      2. DGIS-DBIZLINK-MemoryProfile2.PNG
        68 kB

        Activity

        Hide
        Vito Limandibhrata added a comment -

        Pardon me, the screen shots are from Probe Lambada screens. However, using Profiler4J, it shows the same result.

        Show
        Vito Limandibhrata added a comment - Pardon me, the screen shots are from Probe Lambada screens. However, using Profiler4J, it shows the same result.
        Hide
        Lucas Ward added a comment -

        Can you post your configuration? I don't think that the out of memory error is likely due to a memory leak in the bean wrapper (although, anything is possible). Usually when something like this happens, a commit interval is way too high (or it's in a tasklet) which will cause the above result.

        Show
        Lucas Ward added a comment - Can you post your configuration? I don't think that the out of memory error is likely due to a memory leak in the bean wrapper (although, anything is possible). Usually when something like this happens, a commit interval is way too high (or it's in a tasklet) which will cause the above result.
        Hide
        Vito Limandibhrata added a comment -

        Hi Lucas,

        I have uploaded the configuration files.

        The following configuration files which are loaded in the Parent Application Context
        applicationContext-SpringBatch-Base.xml
        applicationContext-SpringBatch-Base-Parent.xml

        The following configuration files which are loaded in the Subcontext Application Context
        applicationContext-SpringBatch-Jobs.xml
        applicationContext-SpringBatch-Job-GISDEB.xml
        applicationContext-SpringBatch-Job-GISSOH.xml
        applicationContext-SpringBatch-Job-GISSOL.xml
        applicationContext-SpringBatch-Job-GISDEB-GISSOH-GISSOL.xml

        There is a custom tasklet to unzipped the flat files before the files are being processed
        GZipUncompress.java

        The job which is called is jobImportCobolGISDEBGISSOHGISSOL (defined in applicationContext-SpringBatch-Job-GISDEB-GISSOH-GISSOL.xml)
        It is triggered using a QuartzJobBean. The code is in this file
        ProcessCobolFlatFilesInSeparateApplicationContexts.java

        Thank you

        Show
        Vito Limandibhrata added a comment - Hi Lucas, I have uploaded the configuration files. The following configuration files which are loaded in the Parent Application Context applicationContext-SpringBatch-Base.xml applicationContext-SpringBatch-Base-Parent.xml The following configuration files which are loaded in the Subcontext Application Context applicationContext-SpringBatch-Jobs.xml applicationContext-SpringBatch-Job-GISDEB.xml applicationContext-SpringBatch-Job-GISSOH.xml applicationContext-SpringBatch-Job-GISSOL.xml applicationContext-SpringBatch-Job-GISDEB-GISSOH-GISSOL.xml There is a custom tasklet to unzipped the flat files before the files are being processed GZipUncompress.java The job which is called is jobImportCobolGISDEBGISSOHGISSOL (defined in applicationContext-SpringBatch-Job-GISDEB-GISSOH-GISSOL.xml) It is triggered using a QuartzJobBean. The code is in this file ProcessCobolFlatFilesInSeparateApplicationContexts.java Thank you
        Hide
        Dave Syer added a comment -

        I think the problem is much more likely to be in the ZIP deflation Tasklet than in Spring Batch itself. Can you try the same job but without the tasklet (unzip the file manually first)? I bet it works just fine.

        The only obvious error I see in the Tasklet is that you do not need to create a new buffer every time you read from the zip stream - I can't see why it would be harmful, but the IO libraries are native, and you never know what they might be doing. You should protect the close() calls in a finally block as well, but that's unlikely to be the issue here since there are no IO exceptions.

        Show
        Dave Syer added a comment - I think the problem is much more likely to be in the ZIP deflation Tasklet than in Spring Batch itself. Can you try the same job but without the tasklet (unzip the file manually first)? I bet it works just fine. The only obvious error I see in the Tasklet is that you do not need to create a new buffer every time you read from the zip stream - I can't see why it would be harmful, but the IO libraries are native, and you never know what they might be doing. You should protect the close() calls in a finally block as well, but that's unlikely to be the issue here since there are no IO exceptions.
        Hide
        Dave Syer added a comment -

        There's been no feedback on this for quite a while, and it didn't look like a real bug, so I'll mark as resolved. Please update if you disagree.

        Show
        Dave Syer added a comment - There's been no feedback on this for quite a while, and it didn't look like a real bug, so I'll mark as resolved. Please update if you disagree.

          People

          • Assignee:
            Unassigned
            Reporter:
            Vito Limandibhrata
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: