Uploaded image for project: 'Spring Framework'
  1. Spring Framework
  2. SPR-9832

Erroneous "0" returned where empty string expected in call through the RestTemplate

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Complete
    • Affects Version/s: 3.1 RC1, 3.1.2
    • Fix Version/s: 3.1.3, 3.2 RC1
    • Component/s: Web
    • Labels:
      None
    • Last commented by a User:
      false

      Description

      The RestTemplate erroneously infers "0" for an empty string values.

      1. Execute the attached test RestTemplateTest using the attached pom.xml file.
      2. The test fails, the response from the RestTemplate is

      {"lhs":"1 U.S. dollar","rhs":"1 U.S. dollar","error":"0","icc":true}

      For comparison, change the spring version to any 3.0.x version, or perform a request to http://www.google.com/ig/calculator?q=1USD=?EUR from a browser. The response is

      {lhs: "1 U.S. dollar",rhs: "0.774833411 Euros",error: "",icc: true}
      1. pom.xml
        3 kB
        Mattias Severson
      2. RestTemplateTest.java
        1 kB
        Mattias Severson

        Issue Links

          Activity

          Show
          rstoya05-aop Rossen Stoyanchev added a comment - Repro project: https://github.com/SpringSource/spring-framework-issues/tree/master/SPR-9832
          Hide
          rstoya05-aop Rossen Stoyanchev added a comment -

          From an brief look the response is of content type "text/html" and is processed with StringHttpMessageConverter, which simply turns the response body into a String. At the moment I fail to see what could be causing this issue.

          Also I'm not sure why the subject is about json-lib or how this is connected to JSON deserialization?

          Show
          rstoya05-aop Rossen Stoyanchev added a comment - From an brief look the response is of content type "text/html" and is processed with StringHttpMessageConverter, which simply turns the response body into a String. At the moment I fail to see what could be causing this issue. Also I'm not sure why the subject is about json-lib or how this is connected to JSON deserialization?
          Hide
          rstoya05-aop Rossen Stoyanchev added a comment - - edited

          Mattias Severson, could you clarify the subject? I.e. how json-lib and JSON deserialization are related? The server response is of Content-Type "text/html" (`curl -v http://www.google.com/ig/calculator?q=1USD=?EUR') and is processed with StringHttpMessageConverter.

          If you can confirm that I will update the title. Also have you found any other ways in which the issue can be reproduced or exhibits itself?

          Show
          rstoya05-aop Rossen Stoyanchev added a comment - - edited Mattias Severson , could you clarify the subject? I.e. how json-lib and JSON deserialization are related? The server response is of Content-Type "text/html" (`curl -v http://www.google.com/ig/calculator?q=1USD=?EUR ') and is processed with StringHttpMessageConverter. If you can confirm that I will update the title. Also have you found any other ways in which the issue can be reproduced or exhibits itself?
          Hide
          matsev Mattias Severson added a comment -

          Sorry about the late reply. I am currently on vacation and I don't have my laptop with me, so unfortunately I cannot do any investigation at this time. You are probably right that the title is misleading, feel free to update it.

          Show
          matsev Mattias Severson added a comment - Sorry about the late reply. I am currently on vacation and I don't have my laptop with me, so unfortunately I cannot do any investigation at this time. You are probably right that the title is misleading, feel free to update it.
          Hide
          matsev Mattias Severson added a comment -

          Rossen Stoyanchev, as you stated, the json-lib can be ruled out.

          Additionally, it seems like the error was introduced in the 3.1.0.RELEASE (I have verified that the test passes using the 3.0.0.RELEASE and 3.0.7.RELEASE, but fails using the 3.1.0.RELEASE, 3.1.1.RELEASE and 3.1.2.RELEASE).

          Show
          matsev Mattias Severson added a comment - Rossen Stoyanchev , as you stated, the json-lib can be ruled out. Additionally, it seems like the error was introduced in the 3.1.0.RELEASE (I have verified that the test passes using the 3.0.0.RELEASE and 3.0.7.RELEASE, but fails using the 3.1.0.RELEASE, 3.1.1.RELEASE and 3.1.2.RELEASE).
          Hide
          rstoya05-aop Rossen Stoyanchev added a comment -

          I'm a bit stumped by this issue. What's odd is the server returns "text/html" content and the StringHttpMessageConverter simply turns the InputStream into a String without ever parsing it or trying to interpret it.

          I created a project to try to replicate the behavior by creating a controller method that returns the same content as the Google API but I wasn't able to reproduce it. See RestTemplateTests and TestController.

          Show
          rstoya05-aop Rossen Stoyanchev added a comment - I'm a bit stumped by this issue. What's odd is the server returns "text/html" content and the StringHttpMessageConverter simply turns the InputStream into a String without ever parsing it or trying to interpret it. I created a project to try to replicate the behavior by creating a controller method that returns the same content as the Google API but I wasn't able to reproduce it. See RestTemplateTests and TestController .
          Hide
          rstoya05-aop Rossen Stoyanchev added a comment -

          Root cause: thanks to Phil Webb for pointing out that the issue is with the way the query string is parsed. Given "q=1USD=?EUR", the value of "q" is "1USD=?EUR", which contains a reserved character "=". Ideally, for such cases, where reserved characters are expected, it is recommend to use a URI variable:

          "http://www.google.com/ig/calculator?q={value}"

          This ensures the URI can be parsed correctly and the value encoded so the resulting URI would be:

          http://www.google.com/ig/calculator?q=1USD%3D?EUR

          Spring 3.0 vs 3.1: in Spring 3.0, the UriTemplate encoded the query string as a whole without attempting to parse it, which meant that query parameters could not be fully encoded. Spring 3.1 introduced UriComponentsBuilder (also used inside UriTemplate), which does parse the query string and fully encodes query parameter names and values. However, when the query string contains reserved characters there is ambiguity in the parsing.

          Fix: I've updated the regular expression used to parse query params to give more intuitive behavior in some common cases and to fix the regression. Below is the comment from commit 072114.

          Improve regex for parsing query params

          Previously UriComponentsBuilder used a regular expression for parsing
          query name-value pairs where both name and value were expected to not
          contain neither '&', not '='. The idea is that the presence of reserved
          characters makes it impossible to guess correctly how to parse the
          query string (e.g. a=b&c).

          This change relaxes the constraint on query param values, allowing them
          to contain '='. In effect '&' is the ultimate separator of name-value
          pairs, and any '=' in values is ignored. For example "q=1USD=?EUR" is
          interpreted as "q equals '1USD=?EUR'".

          Show
          rstoya05-aop Rossen Stoyanchev added a comment - Root cause : thanks to Phil Webb for pointing out that the issue is with the way the query string is parsed. Given "q=1USD=?EUR", the value of "q" is "1USD=?EUR", which contains a reserved character "=". Ideally, for such cases, where reserved characters are expected, it is recommend to use a URI variable: "http://www.google.com/ig/calculator?q={value}" This ensures the URI can be parsed correctly and the value encoded so the resulting URI would be: http://www.google.com/ig/calculator?q=1USD%3D?EUR Spring 3.0 vs 3.1 : in Spring 3.0, the UriTemplate encoded the query string as a whole without attempting to parse it, which meant that query parameters could not be fully encoded. Spring 3.1 introduced UriComponentsBuilder (also used inside UriTemplate), which does parse the query string and fully encodes query parameter names and values. However, when the query string contains reserved characters there is ambiguity in the parsing. Fix : I've updated the regular expression used to parse query params to give more intuitive behavior in some common cases and to fix the regression. Below is the comment from commit 072114 . Improve regex for parsing query params Previously UriComponentsBuilder used a regular expression for parsing query name-value pairs where both name and value were expected to not contain neither '&', not '='. The idea is that the presence of reserved characters makes it impossible to guess correctly how to parse the query string (e.g. a=b&c). This change relaxes the constraint on query param values, allowing them to contain '='. In effect '&' is the ultimate separator of name-value pairs, and any '=' in values is ignored. For example "q=1USD=?EUR" is interpreted as "q equals '1USD=?EUR'".

            People

            • Assignee:
              rstoya05-aop Rossen Stoyanchev
              Reporter:
              matsev Mattias Severson
              Last updater:
              Rossen Stoyanchev
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:
                Days since last comment:
                5 years, 5 weeks, 4 days ago