[SWS-1000] java.util.Hashtable limits the throughput of org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive Created: 23/Oct/17  Updated: 07/Nov/17  Resolved: 07/Nov/17

Status: Closed
Project: Spring Web Services
Component/s: Core
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Major
Reporter: Sasidhar Sekar Assignee: Unassigned
Resolution: Works as Designed Votes: 0
Labels: marshalling
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: PNG File Contention.png     PNG File Contention.png     PNG File Contention_removed.png     PNG File E2ELatency.png    

 Description   

org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive uses java.util.Hashtable.get(Object) implementation to the get url stream handler for marshaling.

This HashTable implementation is known to be quite poor in terms of performance in a highly concurrent environment. Latency/throughput of the implementing application can be found below.

E2ELatency.png

I'm attaching the JFR stack trace of the related calls, showing contention. This increases as the concurrency increases.

Contention.png

"Stack Trace";"Count";"Duration"
"java.util.Hashtable.get(Object)";"1,903";"11,916,392,176"
"   java.net.URL.getURLStreamHandler(String)";"1,903";"11,916,392,176"
"      java.net.URL.<init>(URL, String, URLStreamHandler)";"1,903";"11,916,392,176"
"         java.net.URL.<init>(URL, String)";"1,903";"11,916,392,176"
"            java.net.URL.<init>(String)";"1,903";"11,916,392,176"
"               java.net.JarURLConnection.parseSpecs(URL)";"1,903";"11,916,392,176"
"                  java.net.JarURLConnection.<init>(URL)";"1,903";"11,916,392,176"
"                     org.springframework.boot.loader.jar.JarURLConnection.<init>(URL, JarFile, JarURLConnection$JarEntryName)";"1,903";"11,916,392,176"
"                        org.springframework.boot.loader.jar.JarURLConnection.get(URL, JarFile)";"1,903";"11,916,392,176"
"                           org.springframework.boot.loader.jar.Handler.openConnection(URL)";"1,903";"11,916,392,176"
"                              java.net.URL.openConnection()";"1,903";"11,916,392,176"
"                                 sun.misc.URLClassPath$Loader.findResource(String, boolean)";"1,903";"11,916,392,176"
"                                    sun.misc.URLClassPath$1.next()";"1,903";"11,916,392,176"
"                                       sun.misc.URLClassPath$1.hasMoreElements()";"1,903";"11,916,392,176"
"                                          java.net.URLClassLoader$3$1.run()";"1,903";"11,916,392,176"
"                                             java.net.URLClassLoader$3$1.run()";"1,903";"11,916,392,176"
"                                                java.security.AccessController.doPrivileged(PrivilegedAction, AccessControlContext)";"1,903";"11,916,392,176"
"                                                   java.net.URLClassLoader$3.next()";"1,903";"11,916,392,176"
"                                                      java.net.URLClassLoader$3.hasMoreElements()";"1,903";"11,916,392,176"
"                                                         sun.misc.CompoundEnumeration.next()";"1,903";"11,916,392,176"
"                                                            sun.misc.CompoundEnumeration.hasMoreElements()";"1,903";"11,916,392,176"
"                                                               sun.misc.CompoundEnumeration.next()";"1,903";"11,916,392,176"
"                                                                  sun.misc.CompoundEnumeration.hasMoreElements()";"1,903";"11,916,392,176"
"                                                                     java.util.ServiceLoader$LazyIterator.hasNextService()";"1,903";"11,916,392,176"
"                                                                        java.util.ServiceLoader$LazyIterator.hasNext()";"1,903";"11,916,392,176"
"                                                                           java.util.ServiceLoader$1.hasNext()";"1,903";"11,916,392,176"
"                                                                              javax.xml.transform.FactoryFinder$1.run()";"1,795";"11,303,335,460"
"                                                                                 java.security.AccessController.doPrivileged(PrivilegedAction)";"1,795";"11,303,335,460"
"                                                                                    javax.xml.transform.FactoryFinder.findServiceProvider(Class)";"1,795";"11,303,335,460"
"                                                                                       javax.xml.transform.FactoryFinder.find(Class, String)";"1,795";"11,303,335,460"
"                                                                                          javax.xml.transform.TransformerFactory.newInstance()";"1,795";"11,303,335,460"
"                                                                                             com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.<init>()";"1,795";"11,303,335,460"
"                                                                                                com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.newTransformer()";"1,795";"11,303,335,460"
"                                                                                                   com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream)";"1,701";"10,734,490,000"
"                                                                                                      com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream, boolean)";"1,701";"10,734,490,000"
"                                                                                                         com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.getContentAsStream()";"1,701";"10,734,490,000"
"                                                                                                            com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getHeaderBytes()";"1,701";"10,734,490,000"
"                                                                                                               com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges()";"1,701";"10,734,490,000"
"                                                                                                                  org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(OutputStream)";"1,701";"10,734,490,000"
"                                                                                                                     org.springframework.ws.transport.AbstractWebServiceConnection.send(WebServiceMessage)";"1,701";"10,734,490,000"
"                                                                                                                        org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceConnection, WebServiceMessage)";"1,701";"10,734,490,000"
"                                                                                                                           org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(MessageContext, WebServiceConnection, WebServiceMessageCallback, WebServiceMessageExtractor)";"1,701";"10,734,490,000"
"                                                                                                                              org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(String, WebServiceMessageCallback, WebServiceMessageExtractor)";"1,701";"10,734,490,000"
"                                                                                                                                 org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(String, Object, WebServiceMessageCallback)";"1,701";"10,734,490,000"
"                                                                                                                                    org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(String, Object)";"1,701";"10,734,490,000"
"                                                                                                                                       com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.sendAndReceive(AbstractLoyaltyRequest)";"1,701";"10,734,490,000"
"                                                                                                                                          com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.retrieveApplicableRewards(LoyaltyApplicableRewardRetrievalRequest)";"1,660";"10,525,361,114"
"                                                                                                                                             sun.reflect.GeneratedMethodAccessor216.invoke(Object, Object[])";"1,660";"10,525,361,114"
"                                                                                                                                                sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[])";"1,660";"10,525,361,114"
"                                                                                                                                                   java.lang.reflect.Method.invoke(Object, Object[])";"1,660";"10,525,361,114"
"                                                                                                                                                      com.hotels.thermos.proxy.GenericProxyCircuitBreaker.execute()";"1,660";"10,525,361,114"
"                                                                                                                                                         com.hotels.thermos.AbstractCircuitBreaker.run()";"1,660";"10,525,361,114"
"                                                                                                                                                            com.hotels.thermos.HystrixCommandWrapper.run()";"1,660";"10,525,361,114"
"                                                                                                                                                               com.netflix.hystrix.HystrixCommand$1.call(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                  com.netflix.hystrix.HystrixCommand$1.call(Object)";"1,660";"10,525,361,114"
"                                                                                                                                                                     rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                        rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
"                                                                                                                                                                           rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                              rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
"                                                                                                                                                                                 rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                                    rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
"                                                                                                                                                                                       rx.Observable.unsafeSubscribe(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                                          com.netflix.hystrix.AbstractCommand$5.call(Subscriber)";"1,660";"10,525,361,114"
"                                                                                                                                                                                             com.netflix.hystrix.AbstractCommand$5.call(Object)";"1,660";"10,525,361,114"

Can an alternative implementation, like the one mentioned at https://issues.apache.org/jira/browse/SPARK-16826, be used to circumvent this issue?



 Comments   
Comment by Greg Turnquist [ 24/Oct/17 ]

For starters, the gap between WebServiceTemplate, where our library code is, and the actual usage of a Hashtable is over 30 layers deep. So to say that Spring WS is using a Hashtable is a faulty statement. Spring Web Services is leveraging SAAJ or the SOAP with Attachments API for Java, hence the code using a Hashtable would be standard Java.

In essence, Spring Web Services is taking a string-based URI and asking Java to take if from there in constructing a SAAJ message. Any alteration in the efficiency of SAAJ would entail a bug report with the SAAJ spec lead.

WebServiceTemplate has a DestinationProvider interface, which includes a cached solution that supports caching URIs after they are parsed from objects, but your stack trace doesn't appear to invoke the getDefaultUri() call that would invoke that.

If you provided a little more information about your use case and how you think this is Spring Web Services making an inefficient call, then perhaps we can unravel the real issue you are running into.

Comment by Sasidhar Sekar [ 25/Oct/17 ]

Hi Greg,
Thank you for the explanation. Below is the use case we are trying to implement.

  • We have a client that calls a backend microservice
  • The client can make two types of POST requests
  • The URL of the requests is the same (i.e.) a single endpoint
  • The client code invokes WebServiceTemplate.marshalSendAndReceive(String microserviceUrl, RequestObject request)

The client code looks like this:

class MicroserviceGateway extends WebServiceGatewaySupport implements MicroserviceGateway {
 
    private String microserviceUrl;
    private MicroserviceRequestResponseLogCollector requestResponseLogCollector;
 
    @Override
    public MicroserviceRequestOneResponse retrieveRequestTypeOne(final MicroserviceApplicableRewardRetrievalRequest microserviceApplicableRewardRetrievalRequest) {
        requestResponseLogCollector.addRequestLogEntries(microserviceApplicableRewardRetrievalRequest);
        MicroserviceRequestOneResponse microserviceRequestTypeOneResponse = (MicroserviceRequestOneResponse) sendAndReceive(microserviceApplicableRewardRetrievalRequest);
        requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeOneResponse);
        return microserviceRequestTypeOneResponse;
    }
 
    @Override
    public MicroserviceRequestTwoResponse retrieveRequestTypeTwo(final MicroserviceRequestTwoRetrievalRequest microserviceRequestTypeTwoRetrievalRequest) {
        requestResponseLogCollector.addRequestLogEntries(microserviceRequestTypeTwoRetrievalRequest);
        MicroserviceRequestTwoResponse microserviceRequestTypeTwoResponse = (MicroserviceRequestTwoResponse) sendAndReceive(microserviceRequestTypeTwoRetrievalRequest);
        requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeTwoResponse);
        return microserviceRequestTypeTwoResponse;
    }
 
    public void setMicroserviceUrl(final String microserviceUrl) {
        this.microserviceUrl = microserviceUrl;
    }
 
    public void setRequestResponseLogCollector(final MicroserviceRequestResponseLogCollector requestResponseLogCollector) {
        this.requestResponseLogCollector = requestResponseLogCollector;
    }
 
    private <T extends AbstractMicroserviceRequest> Object sendAndReceive(final T request) {
        return getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request);
    }
}

When a request comes in for RequestTypeOne, the client invokes retrieveRequestTypeOne and likewise for RequestTypeTwo, which in turn pass the request object and the endpoint url to getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request).

Comment by Greg Turnquist [ 27/Oct/17 ]

I guess what you have described sounds like a conventional approach. I don't know what your load factor is, but we are talking about blocking APIs when we discuss SOAP. (People have put in requests for Reactor-based APIs that I have had to decline).

Since we are talking microservices, do you have a cloud-based solution in place? I didn't know if you were using Spring Cloud components like Spring Cloud Netflix Eureka or Spring Cloud Ribbon. Because if you have a pool of microservices, it would make sense to round robin between them for various calls. Whereas Spring Web Services doesn't support Eureka-based URL resolution, it probably wouldn't be too difficult to programmatically get that URL and hence simplify remote calls to a pool of SOAP microservices through service discovery.

Comment by Sasidhar Sekar [ 30/Oct/17 ]

We are not on cloud yet. The servers are physical, within our datacenter. But, we do have multiple instances behind a VIP to cover for this at the moment. The only concern is that we are unable to use the physical servers to their full capacity at the moment.
I'm hoping we could implement the DestinationProvider interface in one of the subsequent sprints, to see if that helps our cause.

Comment by Sasidhar Sekar [ 07/Nov/17 ]

Implementing the AbstractCachingDestinationProvider has infact resolved the contention observed on URL resolution. A comparison of the url resolution attempts before and after could be found below.

Before

After

As can be seen, only one call is made to resolve the url (marked by the HashTable.get())

Thank you so much for your help with this, Greg. I think this is resolved now.

Comment by Greg Turnquist [ 07/Nov/17 ]

Glad to help you resolve your issue.

Generated at Sat Dec 16 05:14:07 UTC 2017 using JIRA 6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5.