[SWS-563] Provide support for Apache HttpClient 4.0 Created: 03/Sep/09  Updated: 09/May/12  Resolved: 09/May/12

Status: Resolved
Project: Spring Web Services
Component/s: Core
Affects Version/s: None
Fix Version/s: 2.1 RC1

Type: New Feature Priority: Major
Reporter: Alan Stewart (personal account) Assignee: Arjen Poutsma
Resolution: Complete Votes: 27
Labels: None
Remaining Estimate: 0d
Time Spent: 6h 8m
Original Estimate: Not Specified

Attachments: Java Source File HttpClientConnection.java     Java Source File HttpClientConnection.java     Java Source File HttpClientMessageSender.java     Java Source File HttpClientMessageSender.java     Java Source File HttpClientMessageSender.java     Java Source File ProtocolExceptionOverrideInterceptor.java    
Reference URL: http://forum.springsource.org/showthread.php?t=77023

 Description   

I would like to be able to use the Apache HttpClient 4.0 in Spring WS, now that it is out of beta



 Comments   
Comment by Oleg Kalnichevski [ 10/Sep/09 ]

I am willing to give a helping hand with that.

Oleg (Apache HttpComponents committer)

Comment by Alan Stewart (personal account) [ 10/Sep/09 ]

Oleg - that's great! I mentioned in the forum entry that I would have a go at it. I created a class called HttpClientMessageSender (a copy of the CommonHttpMessageSender class) that also extends org.springframework.ws.transport.http.AbstractHttpWebServiceMessageSender . I converted most of the methods but there seems to be in some cases no direct equivalent from Commons to the new client. I got stuck on the setMaxConnectionsPerHost method. It may not be relevant anymore anyway but with your expertise in this, it should be quite trivial. An implementation of the org.springframework.ws.transport .WebServiceConnection is also required as well.
Thanks
Alan

Comment by Arjen Poutsma [ 10/Sep/09 ]

Thanks for all the contributions everbody!

Comment by Oleg Kalnichevski [ 10/Sep/09 ]

Alan

public HttpClientMessageSender()

{ httpClient = new DefaultHttpClient(); setConnectionTimeout(DEFAULT_CONNECTION_TIMEOUT_MILLISECONDS); setReadTimeout(DEFAULT_READ_TIMEOUT_MILLISECONDS); }

I would strongly recommend using ThreadSafeClientConnManager instead of the one created by HttpClient per default

http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e596

> Commons to the new client. I got stuck on the setMaxConnectionsPerHost method. It may not be relevant anymore
> anyway but with your expertise in this, it should be quite trivial

One can use ConnPerRouteBean class to enforce max connection limit for individual HTTP routes, as described here:

http://hc.apache.org/httpcomponents-client/tutorial/html/connmgmt.html#d4e596
http://hc.apache.org/httpcomponents-client/httpclient/apidocs/org/apache/http/conn/params/ConnPerRouteBean.html

> An implementation of the org.springframework.ws.transport .WebServiceConnection is also required as well.

That's where things may get tricky. In any way I'll happily help with the HttpClient specific stuff.

Oleg

Comment by Oleg Kalnichevski [ 10/Sep/09 ]

One more thing. This bit looks ugly:

public void afterPropertiesSet() throws Exception {
if (getCredentials() != null)

{ ((DefaultHttpClient) getHttpClient()).getCredentialsProvider().setCredentials(getAuthScope(), getCredentials()); // ((DefaultHttpClient) getHttpClient()).getCredentialsProvider().setAuthenticationPreemptive(true); }

}

There are better ways of providing auth credentials at run-time, for instance by binding a custom credentials provider to the local execution context or by using a custom protocol interceptor. For details see:

http://hc.apache.org/httpcomponents-client/tutorial/html/authentication.html#d4e909
http://hc.apache.org/httpcomponents-client/tutorial/html/authentication.html#d4e942

Oleg

Comment by Alan Stewart (personal account) [ 14/Sep/09 ]

> I would strongly recommend using ThreadSafeClientConnManager instead of the one created by HttpClient per default

I followed the same pattern as the CommonHttpMessageSender by allowing the default constructor to create a default client, and allow the user to inject a multi-threaded connection manager (like I do in my production systems) if desired. If you think this is still a bad idea, I can change it.

>> An implementation of the org.springframework.ws.transport .WebServiceConnection is also required as well.
>That's where things may get tricky. In any way I'll happily help with the HttpClient specific stuff.

I created a HttpClientConnection class which extends the Spring WS framework's org.springframework.ws.transport.http.AbstractSenderConnection as does the Commons class.
There are some abstract methods which must be implemented:
/** Returns the HTTP status code of the response. */
protected abstract int getResponseCode() throws IOException;

/** Returns the HTTP status message of the response. */
protected abstract String getResponseMessage() throws IOException;

/** Returns the length of the response. */
protected abstract long getResponseContentLength() throws IOException;

/** Returns the raw, possibly compressed input stream to read the response from. */
protected abstract InputStream getRawResponseInputStream() throws IOException;

I can get the last 2 working by returning httpPost.getEntity().getContentLength() and httpPost.getEntity().getContent() respectively, but can't find equivalents for getResponseCode() and getResponseMessage in HttpPost or HttpEntity.

Attached are new versions of the code
Alan

Comment by Oleg Kalnichevski [ 16/Sep/09 ]

> I followed the same pattern as the CommonHttpMessageSender by allowing the default constructor
> to create a default client, and allow the user to inject a multi-threaded connection manager
> (like I do in my production systems) if desired. If you think this is still a bad idea, I can change it.

I believe (I may be wrong here, though) CommonHttpMessageSender can be used by multiple threads to execute requests concurrently. Therefore a multi-threaded connection manager should be used by default to enable proper handling of concurrent request execution.

> I can get the last 2 working by returning httpPost.getEntity().getContentLength() and
> httpPost.getEntity().getContent() respectively, but can't find equivalents for
> getResponseCode() and getResponseMessage in HttpPost or HttpEntity.

Please note some responses may have no enclosing response entity, so you should be checking for HttpResponse#getEntity being non null just in case.

Use HttpResponse#getStatusLine()#getStatusCode() in order to obtain the response code. Use HttpResponse#getStatusLine()#getReasonPhrase() in order to obtain the response message

Also, make sure to call HttpEntity#consumeContent() in HttpClientConnection#onClose() to ensure correct re-use of the underlying HTTP connection and deallocation of system resources.

Last but not the least, we should eliminate in memory buffering of the request content body. For that we might have to change the subclass of HttpClientConnection from AbstractHttpSenderConnection to something else or just implement WebServiceConnection directly. In memory buffering of request content pretty much eliminates all the benefits of using HttpClient in the first place.

Oleg

Comment by Alan Stewart (personal account) [ 21/Sep/09 ]

> Last but not the least, we should eliminate in memory buffering of the request content body. For that we might have to change the subclass of
> HttpClientConnection from AbstractHttpSenderConnection to something else or just implement WebServiceConnection directly. In memory buffering of request
> content pretty much eliminates all the benefits of using HttpClient in the first place.

To eliminate in memory buffering, do you mean to use something other than the ByteArrayEntity in the onSendAfterWrite method? The CommonsHttpConnection uses the older ByteArrayRequestEntity. What do you suggest instead?

Alan

Comment by Oleg Kalnichevski [ 21/Sep/09 ]

Alan

The problem is the implementation of the WebServiceConnection#send method by AbstractWebServiceConnection class [1]

In order to utilize HttpClient content streaming capability one should provide a custom implementation of HttpEntity interface that wraps the WebServiceMessage instance instead of writing its content out to an intermediate byte array and using ByteArrayEntity. This will enable HttpClient to write out the message content directly to the socket of the underlying HTTP connection.

The catch is AbstractWebServiceConnection can no longer be a base class for HttpClientConnection and therefore most of its functionality may need to be duplicated.

Oleg

[1] https://src.springframework.org/svn/spring-ws/trunk/core/src/main/java/org/springframework/ws/transport/AbstractWebServiceConnection.java

Comment by Arjen Poutsma [ 10/Nov/10 ]

To do proper justice to the new capabilities of HttpClient 4.0, we will probably have to rewrite some of the transport logic in Spring-WS. Unfortunately, I do not have time to do that before 2.0, so I am rescheduling for 2.1.

A shame, but that's the way it is.

Comment by Morten Andersen-Gott [ 27/Mar/11 ]

Have a preliminary date been set for 2.1? Or for the milestone in which this will be included?

Comment by Piotr Bazan [ 29/Jun/11 ]

Guys,

I needed to do this integration (httpclient 4 and spring-ws) now and faced some issues with jdk's 6 saaj and httpclient 4. It's not so much the spring issue but you have to deal with this anyway.

The problem is the httpclient 4 is very restrictive in its RequestContent interceptor and throws ProtocolException when a Content-Length header has been set prior to calling its process method. And this is a case with a saaj's com.sun.xml.messaging.saaj.soap.MessageImpl. The class calculates a message size itself and sets a Content-Length header in saveChanges.

Although I agree it's a transport layer responsibility to calculate it, for compatibility purposes httpclient should consider logging a warning and maybe overriding the header instead of throwing an exception. Otherwise it must be worked around by an integration code, Spring in this case.

Piotr

Comment by Barry [ 24/Aug/11 ]

I also needed this feature (my application requires multiple HttpClient instances, each of which is configured with different client X509 keyStores at startup. As far as I know that can't be done with commons-httpclient but can with Apache HttpClient 4).

I took the attachments provided by Alan and applied all of Oleg's suggestions as well as working around the problem mentioned by Piotr (with a custom 'HttpRequestInterceptor' that clears the Content-Length and Transfer-Encoding headers set by SAAJ).

I'll attach the files to this issue in case anyone else needs this sooner rather than later. I have tested it in my test environment and its working fine.

Comment by Barry [ 24/Aug/11 ]

Working HttpClient-based WebServiceMessageSender implementation

Comment by Arjen Poutsma [ 08/May/12 ]

This is now in SVN, as HttpComponentsMessageSender.

Thanks for the patches, guys, it really made my job a lot easier!

Comment by Barry [ 08/May/12 ]

Hi Arjen,

One of my use-cases is that I need to be able to specify different HTTP Basic Auth credentials per request, based on the current user (the context). Looking at the changes at https://fisheye.springsource.org/browse/~br=trunk/spring-ws/trunk/core/src/main/java/org/springframework/ws/transport/http/HttpComponentsConnection.java?r=1981&r=1981, I don't think that will be possible, because I need to specify a org.apache.http.protocol.HttpContext when making the request.

Oleg alluded to using the HttpContext to supply credentials earlier:

There are better ways of providing auth credentials at run-time, for instance by binding a custom credentials provider to the local execution context or by using a custom protocol interceptor. For details see:

http://hc.apache.org/httpcomponents-client/tutorial/html/authentication.html#d4e909
http://hc.apache.org/httpcomponents-client/tutorial/html/authentication.html#d4e942

The problem is that there is no way to provide a org.apache.http.protocol.HttpContext for httpClient.execute(httpPost, httpContext).

What I have been doing in the past is subclassing HttpComponentsMessageSender to override createConnection(URI uri), creating a HttpComponentsConnection with a HttpContext instance like so:

public abstract class BasicAuthHttpComponentsMessageSender extends HttpComponentsMessageSender {
 
    @Override
    public WebServiceConnection createConnection(URI uri) throws IOException {
        HttpPost httpPost = new HttpPost(uri);
        if (isAcceptGzipEncoding()) {
            httpPost.addHeader(HttpTransportConstants.HEADER_ACCEPT_ENCODING, HttpTransportConstants.CONTENT_ENCODING_GZIP);
        }
 
        HttpHost targetHost = new HttpHost(uri.getHost(), uri.getPort(), uri.getScheme());
 
        ((DefaultHttpClient) getHttpClient()).getCredentialsProvider().setCredentials(
                new AuthScope(targetHost.getHostName(), targetHost.getPort()), getCredentials());
 
 
        AuthCache authCache = new BasicAuthCache();
        BasicScheme basicAuth = new BasicScheme();
        authCache.put(targetHost, basicAuth);
 
        BasicHttpContext localContext = new BasicHttpContext();
        localContext.setAttribute(ClientContext.AUTH_CACHE, authCache);
 
        return new HttpComponentsConnection(getHttpClient(), httpPost, localContext);
    }
 
    protected abstract UsernamePasswordCredentials getCredentials();
}

Would it be possible to include a constructor for HttpComponentsConnection that takes a org.apache.http.protocol.HttpContext?

Thanks a lot

Comment by Arjen Poutsma [ 08/May/12 ]

I'll take a look tomorrow.

Comment by Arjen Poutsma [ 09/May/12 ]

I've added a createContext(URI) template method that allows you to create a http context for a given URI. The default implementation returns null.

Hope that helps!

Comment by Barry [ 09/May/12 ]

Great, thanks!

Generated at Mon Dec 18 09:04:53 UTC 2017 using JIRA 6.4.14#64029-sha1:ae256fe0fbb912241490ff1cecfb323ea0905ca5.