[SWS-707] Recievign very large attachments in the clent will cause a OutOfMemoryError Created: 29/Apr/11  Updated: 04/Sep/15  Resolved: 18/Mar/14

Status: Resolved
Project: Spring Web Services
Component/s: Core
Affects Version/s: 1.5.9, 2.0.1
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Karthik Ramacahndran Assignee: Arjen Poutsma
Resolution: Won't Fix Votes: 4
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: Java Source File PatchedAbstractHttpSenderConnection.java    

 Description   

AbstractHttpSenderConnectio.hasResponse reads the enter message into memory in order to ensure a message exists. When dealing with large messages this causes an OutOfMemoryError:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:113)
at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:164)
at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:548)
at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:496)
at gov.ic.dia.cdir.client.CdirFetchServicesClientImpl.fetchWithAttachments(CdirFetchServicesClientImpl.java:160)
at gov.ic.dia.cdir.client.CommandLineClient.testFetchEnclosure(CommandLineClient.java:283)
at gov.ic.dia.cdir.client.CommandLineClient.main(CommandLineClient.java:129)



 Comments   
Comment by Christopher Wong [ 09/Jun/11 ]

This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages.

Comment by Arjen Poutsma [ 18/Mar/14 ]

AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read.

Comment by Gyula Szalai [ 23/Apr/14 ]

How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes.

Comment by Gyula Szalai [ 24/Apr/14 ]

I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error:

java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary.
 
	at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223)
 
	at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68)
 
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163)
 
	at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130)
 
	at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198)
 
	at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181)
 
	at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106)
 
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122)
 
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133)
 
	at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210)
 
	at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444)
 
	at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950)
 
	at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363)
 
	at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323)
 
	at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109)
 
	at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957)
 
	at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100)
 
	at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75)
 
	at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250)
 
	at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127)
 
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324)
 
	at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307)
 
	at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127)
 
	at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738)
 
	at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62)
 
	at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409)
 
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598)
 
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
 
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
 
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original):

java.lang.OutOfMemoryError: Java heap space
 
	at java.util.Arrays.copyOf(Arrays.java:2271)
 
	at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113)
 
	at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
 
	at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140)
 
	at org.springframework.util.StreamUtils.copy(StreamUtils.java:125)
 
	at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109)
 
	at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156)
 
	at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72)
 
	at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46)
 
	at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86)
 
	at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591)
 
	at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380)
 
	at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372)
 
	at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66)
 
	at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19)

I used an EndpointInterceptor implementation to set the Content-Lenght header as follows:

 
 
    ...
 
 
 
    @Override
 
    public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception {
 
        WebServiceMessage responseMessage = messageContext.getResponse();
 
        AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage;
 
        SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage;
 
        SOAPMessage soapMessage = saajSoapMessage.getSaajMessage();
 
        
        MimeHeaders mimeHeaders = soapMessage.getMimeHeaders();
 
        mimeHeaders.addHeader("Content-Length", "10000000000");
 
        return true;
 
    }
 
 
 
    ....

Am I doing something wrong?

Comment by Marcus Kalthoff [ 24/Apr/14 ]

I would try to set it to a value less than Integer.MAX_VALUE
2147483647 <-- max int
10000000000 <--- your value

Comment by Gyula Szalai [ 24/Apr/14 ]

Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore.
However, I think, an attachment (at least in theory) can be larger than 2 GB, so it can be a problem.

Comment by Ralf Stuckert [ 04/Sep/15 ]

If you do streaming, you often don't know any content-lenght in advance.

And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back.

Comment by Ralf Stuckert [ 04/Sep/15 ]

Implementation using a PushbackInputStream in order to avoid reading the whole content

Comment by Ralf Stuckert [ 04/Sep/15 ]

Here is an implemenation using a PushbackInputStream: PatchedAbstractHttpSenderConnection.java

It works fine and allows streaming of large content without content-length header being set.

Comment by Ralf Stuckert [ 04/Sep/15 ]

And since the method hasResponse() is final, there is no chance to fix by inheritance.

Generated at Fri Sep 22 22:24:20 UTC 2017 using JIRA 6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5.