[SWS-707] Recievign very large attachments in the clent will cause a OutOfMemoryError Created: 29/Apr/11 Updated: 24/Oct/17 Resolved: 24/Oct/17 |
|
Status: | Closed |
Project: | Spring Web Services |
Component/s: | Core |
Affects Version/s: | 1.5.9, 2.0.1 |
Fix Version/s: | 3.0.0.RELEASE |
Type: | Bug | Priority: | Major |
Reporter: | Karthik Ramacahndran | Assignee: | Greg Turnquist |
Resolution: | Complete | Votes: | 4 |
Labels: | None | ||
Remaining Estimate: | Not Specified | ||
Time Spent: | Not Specified | ||
Original Estimate: | Not Specified |
Attachments: |
![]() |
Pull Request URL: | https://github.com/spring-projects/spring-ws/pull/99 |
Description |
AbstractHttpSenderConnectio.hasResponse reads the enter message into memory in order to ensure a message exists. When dealing with large messages this causes an OutOfMemoryError: Exception in thread "main" java.lang.OutOfMemoryError: Java heap space |
Comments |
Comment by Christopher Wong [ 09/Jun/11 ] |
This rather defeats the point of the various underlying streaming message facilities, such as StreamingWebServiceMessage or the Axiom message factory with payloadCaching disabled. It makes WebServiceTemplate unusable for large messages. |
Comment by Arjen Poutsma [ 18/Mar/14 ] |
AbstractHttpSenderConnection only reads the message to determine the content-length of the message. If a Content-Length header is set in the response, it is used and the message is not read. |
Comment by Gyula Szalai [ 23/Apr/14 ] |
How can I set the Content-Length header to the appropriate value on the server side? I tried it with an EndpointInterceptor, but I don't know how to get the full size of the response message in bytes. |
Comment by Gyula Szalai [ 24/Apr/14 ] |
I have a response that contains a 600MB attachment, since I was not able to calculate the full size of the response (see my previous comment) I just set the Content-Length to a relatively small value (10000) and tried the client. I got the following error: java.lang.RuntimeException: org.jvnet.mimepull.MIMEParsingException: Reached EOF, but there is no closing MIME boundary. at org.jvnet.mimepull.MIMEParser.readBody(MIMEParser.java:223) at org.jvnet.mimepull.MIMEParser.access$600(MIMEParser.java:68) at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:163) at org.jvnet.mimepull.MIMEParser$MIMEEventIterator.next(MIMEParser.java:130) at org.jvnet.mimepull.MIMEMessage.makeProgress(MIMEMessage.java:198) at org.jvnet.mimepull.MIMEMessage.parseAll(MIMEMessage.java:181) at org.jvnet.mimepull.MIMEMessage.getAttachments(MIMEMessage.java:106) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parseAll(MimePullMultipart.java:122) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimePullMultipart.parse(MimePullMultipart.java:133) at com.sun.xml.messaging.saaj.packaging.mime.internet.MimeMultipart.getCount(MimeMultipart.java:210) at com.sun.xml.messaging.saaj.soap.MessageImpl.initializeAllAttachments(MessageImpl.java:1444) at com.sun.xml.messaging.saaj.soap.MessageImpl.getAttachments(MessageImpl.java:950) at org.springframework.ws.soap.saaj.Saaj13Implementation.getAttachment(Saaj13Implementation.java:363) at org.springframework.ws.soap.saaj.SaajSoapMessage.getAttachment(SaajSoapMessage.java:323) at org.springframework.ws.support.MarshallingUtils$MimeMessageContainer.getAttachment(MarshallingUtils.java:109) at org.springframework.oxm.jaxb.Jaxb2Marshaller$Jaxb2AttachmentUnmarshaller.getAttachmentAsDataHandler(Jaxb2Marshaller.java:957) at com.sun.xml.bind.v2.runtime.unmarshaller.MTOMDecorator.startElement(MTOMDecorator.java:100) at com.sun.xml.bind.v2.runtime.unmarshaller.InterningXmlVisitor.startElement(InterningXmlVisitor.java:75) at com.sun.xml.bind.v2.runtime.unmarshaller.SAXConnector.startElement(SAXConnector.java:150) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:244) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:281) at com.sun.xml.bind.unmarshaller.DOMScanner.visit(DOMScanner.java:250) at com.sun.xml.bind.unmarshaller.DOMScanner.scan(DOMScanner.java:127) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:324) at com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:307) at javax.xml.bind.helpers.AbstractUnmarshallerImpl.unmarshal(AbstractUnmarshallerImpl.java:127) at org.springframework.oxm.jaxb.Jaxb2Marshaller.unmarshal(Jaxb2Marshaller.java:738) at org.springframework.ws.support.MarshallingUtils.unmarshal(MarshallingUtils.java:62) at org.springframework.ws.client.core.WebServiceTemplate$3.extractData(WebServiceTemplate.java:409) at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:598) at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372) at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66) at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19) Then I set the Content-Length header to a very large value (10,000,000,000) I get the following error (I think it's the same as the original): java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:2271) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:113) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:140) at org.springframework.util.StreamUtils.copy(StreamUtils.java:125) at org.springframework.util.FileCopyUtils.copy(FileCopyUtils.java:109) at org.springframework.util.FileCopyUtils.copyToByteArray(FileCopyUtils.java:156) at org.springframework.ws.transport.http.AbstractHttpSenderConnection.hasResponse(AbstractHttpSenderConnection.java:72) at org.springframework.ws.transport.AbstractSenderConnection.createTransportInputStream(AbstractSenderConnection.java:46) at org.springframework.ws.transport.AbstractWebServiceConnection.receive(AbstractWebServiceConnection.java:86) at org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(WebServiceTemplate.java:591) at org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(WebServiceTemplate.java:539) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:386) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:380) at org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(WebServiceTemplate.java:372) at hu.vanio.springwsmtom.client.SaajMtomClient.loadContent(SaajMtomClient.java:66) at hu.vanio.springwsmtom.client.SaajMtomClientIT.testLoad(SaajMtomClientIT.java:19) I used an EndpointInterceptor implementation to set the Content-Lenght header as follows: ... @Override public boolean handleResponse(MessageContext messageContext, Object endpoint) throws Exception { WebServiceMessage responseMessage = messageContext.getResponse(); AbstractSoapMessage abstractSoapMessage = (AbstractSoapMessage) responseMessage; SaajSoapMessage saajSoapMessage = (SaajSoapMessage) abstractSoapMessage; SOAPMessage soapMessage = saajSoapMessage.getSaajMessage(); MimeHeaders mimeHeaders = soapMessage.getMimeHeaders(); mimeHeaders.addHeader("Content-Length", "10000000000"); return true; } .... Am I doing something wrong? |
Comment by Marcus Kalthoff [ 24/Apr/14 ] |
I would try to set it to a value less than Integer.MAX_VALUE |
Comment by Gyula Szalai [ 24/Apr/14 ] |
Thanks Marcus, that helped. If I set it to 2147483647 I don't get OOM on the client anymore. |
Comment by Ralf Stuckert [ 04/Sep/15 ] |
If you do streaming, you often don't know any content-lenght in advance. And you do not need to read the whole content into memory just to decide, whether there is content. This could easily be done using a PushbackInputStream, reading one byte and pushing it back. |
Comment by Ralf Stuckert [ 04/Sep/15 ] |
Implementation using a PushbackInputStream in order to avoid reading the whole content |
Comment by Ralf Stuckert [ 04/Sep/15 ] |
Here is an implemenation using a PushbackInputStream: PatchedAbstractHttpSenderConnection.java It works fine and allows streaming of large content without content-length header being set. |
Comment by Ralf Stuckert [ 04/Sep/15 ] |
And since the method hasResponse() is final, there is no chance to fix by inheritance. |