Spring Framework
  1. Spring Framework
  2. SPR-5516

RestTemplate should encode the url variables

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Complete
    • Affects Version/s: 3.0 M2
    • Fix Version/s: 3.0 M2
    • Component/s: Web
    • Labels:
      None
    • Last commented by a User:
      false

      Description

      The RestTemplate does not encode the url variables. The following line:

      template.postForLocation("http://twitter.com/statuses/update.xml?status={status}", "", "Ho Ho");
      

      Results in an exception:

      Exception in thread "main" java.lang.IllegalArgumentException
      	at java.net.URI.create(URI.java:842)
      	at org.springframework.web.util.UriTemplate.expand(UriTemplate.java:140)
      	at org.springframework.web.client.core.RestTemplate.execute(RestTemplate.java:266)
      	at org.springframework.web.client.core.RestTemplate.postForLocation(RestTemplate.java:203)
      	at test.RestClientTest.main(RestClientTest.java:30)
      Caused by: java.net.URISyntaxException: Illegal character in query at index 48: http://twitter.com/statuses/update.xml?status=Ho Ho
      	at java.net.URI$Parser.fail(URI.java:2809)
      	at java.net.URI$Parser.checkChars(URI.java:2982)
      	at java.net.URI$Parser.parseHierarchical(URI.java:3072)
      	at java.net.URI$Parser.parse(URI.java:3014)
      	at java.net.URI.<init>(URI.java:578)
      	at java.net.URI.create(URI.java:840)
      	... 4 more
      

        Issue Links

          Activity

          Hide
          Chantal Ackermann added a comment -

          Ok, I am getting your point. Thanks for that helpful explanation.

          I've tried around with the plus sign in my browsers. It is never escaped as a literal plus sign.

          If you enter http://example.com/ jvg+37xRX4tKphEsdgtSMg== in your browser, you will see that it does not escape anything, because it does not have to.

          Yuk. I'm not sure about that. I am not escaping spaces with + or %20 by default. It's not something the average browser user would do, wouldn't they? Wouldn't they rather mean a plus sign if they put it in....?
          Thinking about it - no, you could never know. The URL might have been copied from somewhere. So, it's ALWAYS interpreted as a space. But how then input a literal plus - as %2B? But the clients do not scan automatically for %2B, yet. They do have to know that it might come along.

          If you submit "foo+bar", your browser will encode the + and do a request on http://www.google.nl/search?q=foo%2Bbar.

          No, I've tried that. I am not able to trigger that conversion, neither on Firefox nor on Safari. (There are possiblity a lot of combinations I haven't tried, though.)

          If anything, the issue you are having seem to be on the server side (in DispatcherServlet), where we seem to do too aggressive decoding by using URLDecoder for any request URL (see UrlPathHelper#decodeRequestString). Hence jvg+37xRX4tKphEsdgtSMg== that is decoded into jvg 37xRX4tKphEsdgtSMg==. I will do some further investigation on this, and create a new JIRA for that if necessary.

          Is it really an error on the server side? I wondered about that when I described the test. But after your last comment I am not sure anymore.

          This makes me look forward to a Spring based solution that takes care of these conversion problems... But as it requires a solution that works for different types of web server / client combinations (some of them not using spring), and frontends that dynamically construct URLs in different layers including JavaScript, this certainly is not an easy task.

          Show
          Chantal Ackermann added a comment - Ok, I am getting your point. Thanks for that helpful explanation. I've tried around with the plus sign in my browsers. It is never escaped as a literal plus sign. If you enter http://example.com/ jvg+37xRX4tKphEsdgtSMg== in your browser, you will see that it does not escape anything, because it does not have to. Yuk. I'm not sure about that. I am not escaping spaces with + or %20 by default. It's not something the average browser user would do, wouldn't they? Wouldn't they rather mean a plus sign if they put it in....? Thinking about it - no, you could never know. The URL might have been copied from somewhere. So, it's ALWAYS interpreted as a space. But how then input a literal plus - as %2B? But the clients do not scan automatically for %2B, yet. They do have to know that it might come along. If you submit "foo+bar", your browser will encode the + and do a request on http://www.google.nl/search?q=foo%2Bbar . No, I've tried that. I am not able to trigger that conversion, neither on Firefox nor on Safari. (There are possiblity a lot of combinations I haven't tried, though.) If anything, the issue you are having seem to be on the server side (in DispatcherServlet), where we seem to do too aggressive decoding by using URLDecoder for any request URL (see UrlPathHelper#decodeRequestString). Hence jvg+37xRX4tKphEsdgtSMg== that is decoded into jvg 37xRX4tKphEsdgtSMg==. I will do some further investigation on this, and create a new JIRA for that if necessary. Is it really an error on the server side? I wondered about that when I described the test. But after your last comment I am not sure anymore. This makes me look forward to a Spring based solution that takes care of these conversion problems... But as it requires a solution that works for different types of web server / client combinations (some of them not using spring), and frontends that dynamically construct URLs in different layers including JavaScript, this certainly is not an easy task.
          Hide
          Arjen Poutsma added a comment -

          I've confirmed the bug in Spring MVC, and created SPR-6291 to fix it. It seems that the server-side components are too aggressive when decoding request URIs, which is probably the cause of your problems. Specifically, the UrlPathHelper should not decode + into space.

          As for your other comments: this is the behavior I see when typing URLs in Safari (following links is not good enough, because HTML interpretation might get in the way):
          http://example.com/foo bar becomes http://example.com/foo%20bar, i.e. Safari (not the person typing it) encodes the space into %20.
          http://example.com/foo+bar= remains http://example.com/foo+bar=, i.e. + and = are legal URL characters, and don't need to be encoded.

          RestTemplate executes the same behavior in these two cases, and I see no reason to change it.

          When using Safari as a HTML interpreter (using Google search as an example), I see the following behavior:
          Entering "foo bar" in the search box in http://google.com results in a request for http://www.google.com/search?q=foo+bar (there are some other query parameters, but this is the main part), i.e. Safari forms a URL with a query component, and encodes the space in this query component as a +.
          Entering "foo+bar" in the search box results in a request for http://www.google.com/search?q=foo%2Bbar, i.e. Safari forms a URL with a query component, and encodes the + in this query component as %2B.

          Show
          Arjen Poutsma added a comment - I've confirmed the bug in Spring MVC, and created SPR-6291 to fix it. It seems that the server-side components are too aggressive when decoding request URIs, which is probably the cause of your problems. Specifically, the UrlPathHelper should not decode + into space. As for your other comments: this is the behavior I see when typing URLs in Safari (following links is not good enough, because HTML interpretation might get in the way): http://example.com/foo bar becomes http://example.com/foo%20bar , i.e. Safari (not the person typing it) encodes the space into %20. http://example.com/foo+bar= remains http://example.com/foo+bar= , i.e. + and = are legal URL characters, and don't need to be encoded. RestTemplate executes the same behavior in these two cases, and I see no reason to change it. When using Safari as a HTML interpreter (using Google search as an example), I see the following behavior: Entering "foo bar" in the search box in http://google.com results in a request for http://www.google.com/search?q=foo+bar (there are some other query parameters, but this is the main part), i.e. Safari forms a URL with a query component, and encodes the space in this query component as a +. Entering "foo+bar" in the search box results in a request for http://www.google.com/search?q=foo%2Bbar , i.e. Safari forms a URL with a query component, and encodes the + in this query component as %2B.
          Hide
          Chantal Ackermann added a comment -

          Hi Arjen,

          thanks a lot for clearing things up!

          I'm sorry. I reproduced the test with the Google URL incorrectly. I used the URL address field of the browser directly instead of using the input field of Google's web page...
          When using the Google's input field it works exactly as you descibe it.

          Thanks again, great work!
          Chantal

          Show
          Chantal Ackermann added a comment - Hi Arjen, thanks a lot for clearing things up! I'm sorry. I reproduced the test with the Google URL incorrectly. I used the URL address field of the browser directly instead of using the input field of Google's web page... When using the Google's input field it works exactly as you descibe it. Thanks again, great work! Chantal
          Hide
          Matthijs Bierman added a comment -

          How then would you send a '+' in your URL template?

          The encoding is not done properly if '+' occurs in a query parameter. Say you want to do a Google search for "+obama +president":

          http://www.google.com/search?q=%2Bobama+%2Bpresident

          You would pass the URL http://www.google.com/search?q=

          {query}

          and pass the parameter in a map:

          Map<String,String> params = new HashMap<String,String>();
          params.put("query","+obama +president");
          

          This does not work as expected. The '+' is not encoded, and therefore treated as a space. Manually encoding the parameter results in double encoding.

          After a little more research, the problem appears to be in UriUtils, where the '+' sign is being cleared from the BitSet of characters to be encoded for the QUERY_PARAM BitSet.

          QUERY_PARAM.clear('+');
          
          Show
          Matthijs Bierman added a comment - How then would you send a '+' in your URL template? The encoding is not done properly if '+' occurs in a query parameter. Say you want to do a Google search for "+obama +president": http://www.google.com/search?q=%2Bobama+%2Bpresident You would pass the URL http://www.google.com/search?q= {query} and pass the parameter in a map: Map< String , String > params = new HashMap< String , String >(); params.put( "query" , "+obama +president" ); This does not work as expected. The '+' is not encoded, and therefore treated as a space. Manually encoding the parameter results in double encoding. After a little more research, the problem appears to be in UriUtils, where the '+' sign is being cleared from the BitSet of characters to be encoded for the QUERY_PARAM BitSet. QUERY_PARAM.clear('+');
          Hide
          Arjen Poutsma added a comment -

          URI encoding is a difficult process. UriUtils tries to guess the various URI components (path, query, etc) by using a regular expression. This guessing is by no means foolproof, but it happens to work in most cases, and it's the best we can do.

          The way to work around the problem you encountered is to construct an java.net.URI object with the desired URL, like so:

          RestTemplate restTemplate = new RestTemplate();
          URI u = new URI("http://www.google.com/search?q=+obama+president");
          String result = restTemplate.getForObject(u, String.class);
          

          When a URI is passed to the RestTemplate, it is not encoded, but treated as is.

          Read http://blogs.msdn.com/b/oldnewthing/archive/2010/03/31/9987779.aspx for more information on URI encodings.

          Show
          Arjen Poutsma added a comment - URI encoding is a difficult process. UriUtils tries to guess the various URI components (path, query, etc) by using a regular expression. This guessing is by no means foolproof, but it happens to work in most cases, and it's the best we can do. The way to work around the problem you encountered is to construct an java.net.URI object with the desired URL, like so: RestTemplate restTemplate = new RestTemplate(); URI u = new URI( "http: //www.google.com/search?q=+obama+president" ); String result = restTemplate.getForObject(u, String .class); When a URI is passed to the RestTemplate, it is not encoded, but treated as is. Read http://blogs.msdn.com/b/oldnewthing/archive/2010/03/31/9987779.aspx for more information on URI encodings.

            People

            • Assignee:
              Arjen Poutsma
              Reporter:
              Tareq Abedrabbo
              Last updater:
              Trevor Marshall
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                3 years, 29 weeks, 6 days ago