[#SWS-707] Recievign very large attachments in the clent will cause a OutOfMemoryError

Comment by Christopher Wong [ 09/Jun/11 ]

This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages.

Comment by Arjen Poutsma [ 18/Mar/14 ]

AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read.

Comment by Gyula Szalai [ 23/Apr/14 ]

How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes.

Comment by Gyula Szalai [ 24/Apr/14 ]

I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error:

java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary.
	at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223)
	at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68)
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163)
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130)
	at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198)
	at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181)
	at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133)
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210)
	at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444)
	at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950)
	at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363)
	at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323)
	at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109)
	at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957)
	at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100)
	at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75)
	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
	at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127)
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324)
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307)
	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127)
	at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738)
	at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62)
	at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409)
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598)
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original):

java.lang.OutOfMemoryError: Java heap space
	at java.util.Arrays.copyOf(Arrays.java:2271)
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
	at org.springframework.util.StreamUtils.copy(StreamUtils.java:125)
	at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109)
	at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156)
	at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
	at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
	at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591)
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

I used an EndpointInterceptor implementation to set the Content-Lenght header as follows:

    ...

    @Override
    public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception {
        WebServiceMessage responseMessage = messageContext.getResponse();
        AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage;
        SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage;
        SOAPMessage soapMessage = saajSoapMessage.getSaajMessage();
        
        MimeHeaders mimeHeaders = soapMessage.getMimeHeaders();
        mimeHeaders.addHeader("Content-Length", "10000000000");
        return true;
    }

    ....

Am I doing something wrong?

Comment by Marcus Kalthoff [ 24/Apr/14 ]

I would try to set it to a value less than Integer.MAX_VALUE
2147483647 <-- max int
10000000000 <--- your value

Comment by Gyula Szalai [ 24/Apr/14 ]

Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore.
However, I think, an attachment (at least in theory) can be larger than 2 GB, so it can be a problem.

Comment by Ralf Stuckert [ 04/Sep/15 ]

If you do streaming, you often don't know any content-lenght in advance.

And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back.

Comment by Ralf Stuckert [ 04/Sep/15 ]

Implementation using a PushbackInputStream in order to avoid reading the whole content

Comment by Ralf Stuckert [ 04/Sep/15 ]

Here is an implemenation using a PushbackInputStream: PatchedAbstractHttpSenderConnection.java

It works fine and allows streaming of large content without content-length header being set.

Comment by Ralf Stuckert [ 04/Sep/15 ]

And since the method hasResponse() is final, there is no chance to fix by inheritance.

[SWS-707] Recievign very large attachments in the clent will cause a OutOfMemoryError Created: 29/Apr/11 Updated: 24/Oct/17 Resolved: 24/Oct/17
Status:	Closed
Project:	Spring Web Services
Component/s:	Core
Affects Version/s:	1.5.9, 2.0.1
Fix Version/s:	3.0.0.RELEASE

[SWS-707] Recievign very large attachments in the clent will cause a OutOfMemoryError Created: 29/Apr/11 Updated: 24/Oct/17 Resolved: 24/Oct/17