Uploaded image for project: 'Spring Web Services'
  1. Spring Web Services
  2. SWS-1000

java.util.Hashtable limits the throughput of org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Works as Designed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Core
    • Labels:

      Description

      org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive uses java.util.Hashtable.get(Object) implementation to the get url stream handler for marshaling.

      This HashTable implementation is known to be quite poor in terms of performance in a highly concurrent environment. Latency/throughput of the implementing application can be found below.

      E2ELatency.png

      I'm attaching the JFR stack trace of the related calls, showing contention. This increases as the concurrency increases.

      Contention.png

      "Stack Trace";"Count";"Duration"
      "java.util.Hashtable.get(Object)";"1,903";"11,916,392,176"
      "   java.net.URL.getURLStreamHandler(String)";"1,903";"11,916,392,176"
      "      java.net.URL.<init>(URL, String, URLStreamHandler)";"1,903";"11,916,392,176"
      "         java.net.URL.<init>(URL, String)";"1,903";"11,916,392,176"
      "            java.net.URL.<init>(String)";"1,903";"11,916,392,176"
      "               java.net.JarURLConnection.parseSpecs(URL)";"1,903";"11,916,392,176"
      "                  java.net.JarURLConnection.<init>(URL)";"1,903";"11,916,392,176"
      "                     org.springframework.boot.loader.jar.JarURLConnection.<init>(URL, JarFile, JarURLConnection$JarEntryName)";"1,903";"11,916,392,176"
      "                        org.springframework.boot.loader.jar.JarURLConnection.get(URL, JarFile)";"1,903";"11,916,392,176"
      "                           org.springframework.boot.loader.jar.Handler.openConnection(URL)";"1,903";"11,916,392,176"
      "                              java.net.URL.openConnection()";"1,903";"11,916,392,176"
      "                                 sun.misc.URLClassPath$Loader.findResource(String, boolean)";"1,903";"11,916,392,176"
      "                                    sun.misc.URLClassPath$1.next()";"1,903";"11,916,392,176"
      "                                       sun.misc.URLClassPath$1.hasMoreElements()";"1,903";"11,916,392,176"
      "                                          java.net.URLClassLoader$3$1.run()";"1,903";"11,916,392,176"
      "                                             java.net.URLClassLoader$3$1.run()";"1,903";"11,916,392,176"
      "                                                java.security.AccessController.doPrivileged(PrivilegedAction, AccessControlContext)";"1,903";"11,916,392,176"
      "                                                   java.net.URLClassLoader$3.next()";"1,903";"11,916,392,176"
      "                                                      java.net.URLClassLoader$3.hasMoreElements()";"1,903";"11,916,392,176"
      "                                                         sun.misc.CompoundEnumeration.next()";"1,903";"11,916,392,176"
      "                                                            sun.misc.CompoundEnumeration.hasMoreElements()";"1,903";"11,916,392,176"
      "                                                               sun.misc.CompoundEnumeration.next()";"1,903";"11,916,392,176"
      "                                                                  sun.misc.CompoundEnumeration.hasMoreElements()";"1,903";"11,916,392,176"
      "                                                                     java.util.ServiceLoader$LazyIterator.hasNextService()";"1,903";"11,916,392,176"
      "                                                                        java.util.ServiceLoader$LazyIterator.hasNext()";"1,903";"11,916,392,176"
      "                                                                           java.util.ServiceLoader$1.hasNext()";"1,903";"11,916,392,176"
      "                                                                              javax.xml.transform.FactoryFinder$1.run()";"1,795";"11,303,335,460"
      "                                                                                 java.security.AccessController.doPrivileged(PrivilegedAction)";"1,795";"11,303,335,460"
      "                                                                                    javax.xml.transform.FactoryFinder.findServiceProvider(Class)";"1,795";"11,303,335,460"
      "                                                                                       javax.xml.transform.FactoryFinder.find(Class, String)";"1,795";"11,303,335,460"
      "                                                                                          javax.xml.transform.TransformerFactory.newInstance()";"1,795";"11,303,335,460"
      "                                                                                             com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.<init>()";"1,795";"11,303,335,460"
      "                                                                                                com.sun.xml.internal.messaging.saaj.util.transform.EfficientStreamingTransformer.newTransformer()";"1,795";"11,303,335,460"
      "                                                                                                   com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream)";"1,701";"10,734,490,000"
      "                                                                                                      com.sun.xml.internal.messaging.saaj.soap.impl.EnvelopeImpl.output(OutputStream, boolean)";"1,701";"10,734,490,000"
      "                                                                                                         com.sun.xml.internal.messaging.saaj.soap.SOAPPartImpl.getContentAsStream()";"1,701";"10,734,490,000"
      "                                                                                                            com.sun.xml.internal.messaging.saaj.soap.MessageImpl.getHeaderBytes()";"1,701";"10,734,490,000"
      "                                                                                                               com.sun.xml.internal.messaging.saaj.soap.MessageImpl.saveChanges()";"1,701";"10,734,490,000"
      "                                                                                                                  org.springframework.ws.soap.saaj.SaajSoapMessage.writeTo(OutputStream)";"1,701";"10,734,490,000"
      "                                                                                                                     org.springframework.ws.transport.AbstractWebServiceConnection.send(WebServiceMessage)";"1,701";"10,734,490,000"
      "                                                                                                                        org.springframework.ws.client.core.WebServiceTemplate.sendRequest(WebServiceConnection, WebServiceMessage)";"1,701";"10,734,490,000"
      "                                                                                                                           org.springframework.ws.client.core.WebServiceTemplate.doSendAndReceive(MessageContext, WebServiceConnection, WebServiceMessageCallback, WebServiceMessageExtractor)";"1,701";"10,734,490,000"
      "                                                                                                                              org.springframework.ws.client.core.WebServiceTemplate.sendAndReceive(String, WebServiceMessageCallback, WebServiceMessageExtractor)";"1,701";"10,734,490,000"
      "                                                                                                                                 org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(String, Object, WebServiceMessageCallback)";"1,701";"10,734,490,000"
      "                                                                                                                                    org.springframework.ws.client.core.WebServiceTemplate.marshalSendAndReceive(String, Object)";"1,701";"10,734,490,000"
      "                                                                                                                                       com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.sendAndReceive(AbstractLoyaltyRequest)";"1,701";"10,734,490,000"
      "                                                                                                                                          com.hotels.booking.discos.loyalty.client.LoyaltyWebServiceGateway.retrieveApplicableRewards(LoyaltyApplicableRewardRetrievalRequest)";"1,660";"10,525,361,114"
      "                                                                                                                                             sun.reflect.GeneratedMethodAccessor216.invoke(Object, Object[])";"1,660";"10,525,361,114"
      "                                                                                                                                                sun.reflect.DelegatingMethodAccessorImpl.invoke(Object, Object[])";"1,660";"10,525,361,114"
      "                                                                                                                                                   java.lang.reflect.Method.invoke(Object, Object[])";"1,660";"10,525,361,114"
      "                                                                                                                                                      com.hotels.thermos.proxy.GenericProxyCircuitBreaker.execute()";"1,660";"10,525,361,114"
      "                                                                                                                                                         com.hotels.thermos.AbstractCircuitBreaker.run()";"1,660";"10,525,361,114"
      "                                                                                                                                                            com.hotels.thermos.HystrixCommandWrapper.run()";"1,660";"10,525,361,114"
      "                                                                                                                                                               com.netflix.hystrix.HystrixCommand$1.call(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                  com.netflix.hystrix.HystrixCommand$1.call(Object)";"1,660";"10,525,361,114"
      "                                                                                                                                                                     rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                        rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
      "                                                                                                                                                                           rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                              rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
      "                                                                                                                                                                                 rx.Observable$2.call(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                                    rx.Observable$2.call(Object)";"1,660";"10,525,361,114"
      "                                                                                                                                                                                       rx.Observable.unsafeSubscribe(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                                          com.netflix.hystrix.AbstractCommand$5.call(Subscriber)";"1,660";"10,525,361,114"
      "                                                                                                                                                                                             com.netflix.hystrix.AbstractCommand$5.call(Object)";"1,660";"10,525,361,114"
      

      Can an alternative implementation, like the one mentioned at https://issues.apache.org/jira/browse/SPARK-16826, be used to circumvent this issue?

      1. E2ELatency.png
        25 kB
      2. Contention.png
        29 kB
      3. Contention.png
        29 kB
      4. Contention_removed.png
        42 kB

        Activity

        Hide
        gregturn Greg Turnquist added a comment -

        For starters, the gap between WebServiceTemplate, where our library code is, and the actual usage of a Hashtable is over 30 layers deep. So to say that Spring WS is using a Hashtable is a faulty statement. Spring Web Services is leveraging SAAJ or the SOAP with Attachments API for Java, hence the code using a Hashtable would be standard Java.

        In essence, Spring Web Services is taking a string-based URI and asking Java to take if from there in constructing a SAAJ message. Any alteration in the efficiency of SAAJ would entail a bug report with the SAAJ spec lead.

        WebServiceTemplate has a DestinationProvider interface, which includes a cached solution that supports caching URIs after they are parsed from objects, but your stack trace doesn't appear to invoke the getDefaultUri() call that would invoke that.

        If you provided a little more information about your use case and how you think this is Spring Web Services making an inefficient call, then perhaps we can unravel the real issue you are running into.

        Show
        gregturn Greg Turnquist added a comment - For starters, the gap between WebServiceTemplate, where our library code is, and the actual usage of a Hashtable is over 30 layers deep. So to say that Spring WS is using a Hashtable is a faulty statement. Spring Web Services is leveraging SAAJ or the SOAP with Attachments API for Java, hence the code using a Hashtable would be standard Java. In essence, Spring Web Services is taking a string-based URI and asking Java to take if from there in constructing a SAAJ message. Any alteration in the efficiency of SAAJ would entail a bug report with the SAAJ spec lead. WebServiceTemplate has a DestinationProvider interface, which includes a cached solution that supports caching URIs after they are parsed from objects, but your stack trace doesn't appear to invoke the getDefaultUri() call that would invoke that. If you provided a little more information about your use case and how you think this is Spring Web Services making an inefficient call, then perhaps we can unravel the real issue you are running into.
        Hide
        sasidharsmit Sasidhar Sekar added a comment -

        Hi Greg,
        Thank you for the explanation. Below is the use case we are trying to implement.

        • We have a client that calls a backend microservice
        • The client can make two types of POST requests
        • The URL of the requests is the same (i.e.) a single endpoint
        • The client code invokes WebServiceTemplate.marshalSendAndReceive(String microserviceUrl, RequestObject request)

        The client code looks like this:

        class MicroserviceGateway extends WebServiceGatewaySupport implements MicroserviceGateway {
         
            private String microserviceUrl;
            private MicroserviceRequestResponseLogCollector requestResponseLogCollector;
         
            @Override
            public MicroserviceRequestOneResponse retrieveRequestTypeOne(final MicroserviceApplicableRewardRetrievalRequest microserviceApplicableRewardRetrievalRequest) {
                requestResponseLogCollector.addRequestLogEntries(microserviceApplicableRewardRetrievalRequest);
                MicroserviceRequestOneResponse microserviceRequestTypeOneResponse = (MicroserviceRequestOneResponse) sendAndReceive(microserviceApplicableRewardRetrievalRequest);
                requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeOneResponse);
                return microserviceRequestTypeOneResponse;
            }
         
            @Override
            public MicroserviceRequestTwoResponse retrieveRequestTypeTwo(final MicroserviceRequestTwoRetrievalRequest microserviceRequestTypeTwoRetrievalRequest) {
                requestResponseLogCollector.addRequestLogEntries(microserviceRequestTypeTwoRetrievalRequest);
                MicroserviceRequestTwoResponse microserviceRequestTypeTwoResponse = (MicroserviceRequestTwoResponse) sendAndReceive(microserviceRequestTypeTwoRetrievalRequest);
                requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeTwoResponse);
                return microserviceRequestTypeTwoResponse;
            }
         
            public void setMicroserviceUrl(final String microserviceUrl) {
                this.microserviceUrl = microserviceUrl;
            }
         
            public void setRequestResponseLogCollector(final MicroserviceRequestResponseLogCollector requestResponseLogCollector) {
                this.requestResponseLogCollector = requestResponseLogCollector;
            }
         
            private <T extends AbstractMicroserviceRequest> Object sendAndReceive(final T request) {
                return getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request);
            }
        }
        

        When a request comes in for RequestTypeOne, the client invokes retrieveRequestTypeOne and likewise for RequestTypeTwo, which in turn pass the request object and the endpoint url to getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request).

        Show
        sasidharsmit Sasidhar Sekar added a comment - Hi Greg, Thank you for the explanation. Below is the use case we are trying to implement. We have a client that calls a backend microservice The client can make two types of POST requests The URL of the requests is the same (i.e.) a single endpoint The client code invokes WebServiceTemplate.marshalSendAndReceive(String microserviceUrl, RequestObject request) The client code looks like this: class MicroserviceGateway extends WebServiceGatewaySupport implements MicroserviceGateway {   private String microserviceUrl; private MicroserviceRequestResponseLogCollector requestResponseLogCollector;   @Override public MicroserviceRequestOneResponse retrieveRequestTypeOne( final MicroserviceApplicableRewardRetrievalRequest microserviceApplicableRewardRetrievalRequest) { requestResponseLogCollector.addRequestLogEntries(microserviceApplicableRewardRetrievalRequest); MicroserviceRequestOneResponse microserviceRequestTypeOneResponse = (MicroserviceRequestOneResponse) sendAndReceive(microserviceApplicableRewardRetrievalRequest); requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeOneResponse); return microserviceRequestTypeOneResponse; }   @Override public MicroserviceRequestTwoResponse retrieveRequestTypeTwo( final MicroserviceRequestTwoRetrievalRequest microserviceRequestTypeTwoRetrievalRequest) { requestResponseLogCollector.addRequestLogEntries(microserviceRequestTypeTwoRetrievalRequest); MicroserviceRequestTwoResponse microserviceRequestTypeTwoResponse = (MicroserviceRequestTwoResponse) sendAndReceive(microserviceRequestTypeTwoRetrievalRequest); requestResponseLogCollector.addResponseLogEntries(microserviceRequestTypeTwoResponse); return microserviceRequestTypeTwoResponse; }   public void setMicroserviceUrl( final String microserviceUrl) { this .microserviceUrl = microserviceUrl; }   public void setRequestResponseLogCollector( final MicroserviceRequestResponseLogCollector requestResponseLogCollector) { this .requestResponseLogCollector = requestResponseLogCollector; }   private <T extends AbstractMicroserviceRequest> Object sendAndReceive( final T request) { return getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request); } } When a request comes in for RequestTypeOne, the client invokes retrieveRequestTypeOne and likewise for RequestTypeTwo, which in turn pass the request object and the endpoint url to getWebServiceTemplate().marshalSendAndReceive(microserviceUrl, request).
        Hide
        gregturn Greg Turnquist added a comment -

        I guess what you have described sounds like a conventional approach. I don't know what your load factor is, but we are talking about blocking APIs when we discuss SOAP. (People have put in requests for Reactor-based APIs that I have had to decline).

        Since we are talking microservices, do you have a cloud-based solution in place? I didn't know if you were using Spring Cloud components like Spring Cloud Netflix Eureka or Spring Cloud Ribbon. Because if you have a pool of microservices, it would make sense to round robin between them for various calls. Whereas Spring Web Services doesn't support Eureka-based URL resolution, it probably wouldn't be too difficult to programmatically get that URL and hence simplify remote calls to a pool of SOAP microservices through service discovery.

        Show
        gregturn Greg Turnquist added a comment - I guess what you have described sounds like a conventional approach. I don't know what your load factor is, but we are talking about blocking APIs when we discuss SOAP. (People have put in requests for Reactor-based APIs that I have had to decline). Since we are talking microservices, do you have a cloud-based solution in place? I didn't know if you were using Spring Cloud components like Spring Cloud Netflix Eureka or Spring Cloud Ribbon. Because if you have a pool of microservices, it would make sense to round robin between them for various calls. Whereas Spring Web Services doesn't support Eureka-based URL resolution, it probably wouldn't be too difficult to programmatically get that URL and hence simplify remote calls to a pool of SOAP microservices through service discovery.
        Hide
        sasidharsmit Sasidhar Sekar added a comment -

        We are not on cloud yet. The servers are physical, within our datacenter. But, we do have multiple instances behind a VIP to cover for this at the moment. The only concern is that we are unable to use the physical servers to their full capacity at the moment.
        I'm hoping we could implement the DestinationProvider interface in one of the subsequent sprints, to see if that helps our cause.

        Show
        sasidharsmit Sasidhar Sekar added a comment - We are not on cloud yet. The servers are physical, within our datacenter. But, we do have multiple instances behind a VIP to cover for this at the moment. The only concern is that we are unable to use the physical servers to their full capacity at the moment. I'm hoping we could implement the DestinationProvider interface in one of the subsequent sprints, to see if that helps our cause.
        Hide
        sasidharsmit Sasidhar Sekar added a comment -

        Implementing the AbstractCachingDestinationProvider has infact resolved the contention observed on URL resolution. A comparison of the url resolution attempts before and after could be found below.

        Before

        After

        As can be seen, only one call is made to resolve the url (marked by the HashTable.get())

        Thank you so much for your help with this, Greg. I think this is resolved now.

        Show
        sasidharsmit Sasidhar Sekar added a comment - Implementing the AbstractCachingDestinationProvider has infact resolved the contention observed on URL resolution. A comparison of the url resolution attempts before and after could be found below. Before After As can be seen, only one call is made to resolve the url (marked by the HashTable.get()) Thank you so much for your help with this, Greg. I think this is resolved now.
        Hide
        gregturn Greg Turnquist added a comment -

        Glad to help you resolve your issue.

        Show
        gregturn Greg Turnquist added a comment - Glad to help you resolve your issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            sasidharsmit Sasidhar Sekar
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: