Uploaded image for project: 'Spring Web Services'
  1. Spring Web Services
  2. SWS-707

Recievign very large attachments in the clent will cause a OutOfMemoryError

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: 1.5.9, 2.0.1
    • Fix Version/s: 3.0.0.RELEASE
    • Component/s: Core
    • Labels:
      None

      Description

      AbstractHttpSenderConnectio.hasResponse reads the enter message into memory in order to ensure a message exists. When dealing with large messages this causes an OutOfMemoryError:

      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
      at java.util.Arrays.copyOf(Arrays.java:2786)
      at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
      at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
      at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
      at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
      at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
      at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
      at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:548)
      at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:496)
      at gov.ic.dia.cdir.client.CdirFetchServicesClientImpl.fetchWithAttachments(CdirFetchServicesClientImpl.java:160)
      at gov.ic.dia.cdir.client.CommandLineClient.testFetchEnclosure(CommandLineClient.java:283)
      at gov.ic.dia.cdir.client.CommandLineClient.main(CommandLineClient.java:129)

        Activity

        karthikramachandran Karthik Ramacahndran created issue -
        Hide
        cwong15 Christopher Wong added a comment -

        This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages.

        Show
        cwong15 Christopher Wong added a comment - This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages.
        arjen.poutsma Arjen Poutsma made changes -
        Field Original Value New Value
        Assignee Arjen Poutsma [ arjen.poutsma ]
        Fix Version/s 2.1 M1 [ 11748 ]
        arjen.poutsma Arjen Poutsma made changes -
        Fix Version/s 2.1 M1 [ 11748 ]
        Fix Version/s 2.2 M1 [ 12850 ]
        Hide
        arjen.poutsma Arjen Poutsma added a comment -

        AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read.

        Show
        arjen.poutsma Arjen Poutsma added a comment - AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read.
        arjen.poutsma Arjen Poutsma made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Fix Version/s 2.2 [ 12850 ]
        Resolution Won't Fix [ 2 ]
        Hide
        gyszalai Gyula Szalai added a comment -

        How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes.

        Show
        gyszalai Gyula Szalai added a comment - How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes.
        Hide
        gyszalai Gyula Szalai added a comment -

        I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error:

        java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary.
        	at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223)
        	at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68)
        	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163)
        	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130)
        	at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198)
        	at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181)
        	at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106)
        	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122)
        	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133)
        	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210)
        	at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444)
        	at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950)
        	at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363)
        	at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323)
        	at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109)
        	at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
        	at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324)
        	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307)
        	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127)
        	at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738)
        	at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62)
        	at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409)
        	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598)
        	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
        	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
        	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

        Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original):

        java.lang.OutOfMemoryError: Java heap space
        	at java.util.Arrays.copyOf(Arrays.java:2271)
        	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
        	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
        	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
        	at org.springframework.util.StreamUtils.copy(StreamUtils.java:125)
        	at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109)
        	at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156)
        	at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
        	at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
        	at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
        	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591)
        	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
        	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
        	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
        	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

        I used an EndpointInterceptor implementation to set the Content-Lenght header as follows:

         
            ...
         
            @Override
            public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception {
                WebServiceMessage responseMessage = messageContext.getResponse();
                AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage;
                SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage;
                SOAPMessage soapMessage = saajSoapMessage.getSaajMessage();
                
                MimeHeaders mimeHeaders = soapMessage.getMimeHeaders();
                mimeHeaders.addHeader("Content-Length", "10000000000");
                return true;
            }
         
            ....

        Am I doing something wrong?

        Show
        gyszalai Gyula Szalai added a comment - I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error: java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary. at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223) at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68) at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163) at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130) at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198) at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181) at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210) at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444) at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950) at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363) at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323) at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109) at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957) at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100) at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75) at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250) at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127) at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738) at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62) at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409) at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598) at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372) at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66) at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19) Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original): java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at org.springframework.util.StreamUtils.copy(StreamUtils.java:125) at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109) at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156) at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72) at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46) at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86) at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591) at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372) at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66) at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19) I used an EndpointInterceptor implementation to set the Content-Lenght header as follows:   ...   @Override public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception { WebServiceMessage responseMessage = messageContext.getResponse(); AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage; SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage; SOAPMessage soapMessage = saajSoapMessage.getSaajMessage(); MimeHeaders mimeHeaders = soapMessage.getMimeHeaders(); mimeHeaders.addHeader( "Content-Length" , "10000000000" ); return true ; }   .... Am I doing something wrong?
        Hide
        kalthoff Marcus Kalthoff added a comment -

        I would try to set it to a value less than Integer.MAX_VALUE
        2147483647 <-- max int
        10000000000 <--- your value

        Show
        kalthoff Marcus Kalthoff added a comment - I would try to set it to a value less than Integer.MAX_VALUE 2147483647 <-- max int 10000000000 <--- your value
        Hide
        gyszalai Gyula Szalai added a comment -

        Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore.
        However, I think, an attachment (at least in theory) can be larger than 2 GB, so it can be a problem.

        Show
        gyszalai Gyula Szalai added a comment - Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore. However, I think, an attachment (at least in theory) can be larger than 2 GB, so it can be a problem.
        Hide
        stuckert Ralf Stuckert added a comment -

        If you do streaming, you often don't know any content-lenght in advance.

        And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back.

        Show
        stuckert Ralf Stuckert added a comment - If you do streaming, you often don't know any content-lenght in advance. And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back.
        Hide
        stuckert Ralf Stuckert added a comment -

        Implementation using a PushbackInputStream in order to avoid reading the whole content

        Show
        stuckert Ralf Stuckert added a comment - Implementation using a PushbackInputStream in order to avoid reading the whole content
        stuckert Ralf Stuckert made changes -
        Hide
        stuckert Ralf Stuckert added a comment -

        Here is an implemenation using a PushbackInputStream: PatchedAbstractHttpSenderConnection.java

        It works fine and allows streaming of large content without content-length header being set.

        Show
        stuckert Ralf Stuckert added a comment - Here is an implemenation using a PushbackInputStream: PatchedAbstractHttpSenderConnection.java It works fine and allows streaming of large content without content-length header being set.
        Hide
        stuckert Ralf Stuckert added a comment -

        And since the method hasResponse() is final, there is no chance to fix by inheritance.

        Show
        stuckert Ralf Stuckert added a comment - And since the method hasResponse() is final, there is no chance to fix by inheritance.
        gregturn Greg Turnquist made changes -
        Resolution Won't Fix [ 2 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Assignee Arjen Poutsma [ arjen.poutsma ] Greg Turnquist [ gregturn ]
        gregturn Greg Turnquist made changes -
        Fix Version/s 3.0.0.RELEASE [ 16498 ]
        Pull Request URL https://github.com/spring-projects/spring-ws/pull/99
        gregturn Greg Turnquist made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Complete [ 8 ]
        gregturn Greg Turnquist made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1053d 17h 7m 1 Arjen Poutsma 18/Mar/14 3:53 AM
        Resolved Resolved Reopened Reopened
        1316d 12h 26m 1 Greg Turnquist 24/Oct/17 4:19 PM
        Reopened Reopened Resolved Resolved
        1m 52s 1 Greg Turnquist 24/Oct/17 4:21 PM
        Resolved Resolved Closed Closed
        2s 1 Greg Turnquist 24/Oct/17 4:21 PM

          People

          • Assignee:
            gregturn Greg Turnquist
            Reporter:
            karthikramachandran Karthik Ramacahndran
          • Votes:
            4 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: