Uploaded image for project: 'Spring Data MongoDB'
  1. Spring Data MongoDB
  2. DATAMONGO-994

QueryDsl findAll(Predicate, Pageable) is expensive

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Works as Designed
    • Affects Version/s: 1.5.1 (Dijkstra SR1)
    • Fix Version/s: None
    • Component/s: Repository
    • Labels:
      None
    • Sprint:
      51 - Dijkstra SR2

      Description

      Page<T> findAll(Predicate, Pageable) is first doing a count query (to get total count for the Page). Then the actual query is performed.

      Querydsl MongodbQuery.java#list() method is used in spring data mongodb code. This method call cursor.size() to construct ArrayList<>(). cursor.size() seems to be making an additional count query call. What looks like a simple findAll(…) call is translating into multiple count queries before the actual query is called.

      I opened a bug against querydsl https://github.com/querydsl/querydsl/issues/856

      I also opened a improvement for spring-data-commons to add a findAll(Predicate, Pageable) that returns Slice so that all count calls can be avoided.

        Issue Links

          Activity

          Hide
          olivergierke Oliver Gierke added a comment -

          So what exactly is it that you think we can do about it? Our count query is indeed necessary to calculate the Page. So by deciding to call the paging method, you know what you get. That Querydsl is triggering another count (important: a different one, counting the number of the subset) is hardly something we can do something about, right?

          I saw the ticket with the Slice method but right now there's hardly anything we can do about it as we can't have an overload with an alternative return type.

          Show
          olivergierke Oliver Gierke added a comment - So what exactly is it that you think we can do about it? Our count query is indeed necessary to calculate the Page . So by deciding to call the paging method, you know what you get. That Querydsl is triggering another count (important: a different one, counting the number of the subset) is hardly something we can do something about, right? I saw the ticket with the Slice method but right now there's hardly anything we can do about it as we can't have an overload with an alternative return type.
          Hide
          olivergierke Oliver Gierke added a comment -

          The related ticket in Querydsl has been closed and is due to be released with Querydsl 3.4.2. Upgrading to that version once it has been release should do the trick.

          Show
          olivergierke Oliver Gierke added a comment - The related ticket in Querydsl has been closed and is due to be released with Querydsl 3.4.2. Upgrading to that version once it has been release should do the trick.
          Hide
          spasam Seshu Pasam added a comment -

          Sorry, somehow missed the comment updates. Few things can be done to help users. It was not very obvious that calling this method is 3 times more expensive. And required debugging code in 3 different projects.

          a.) Document the behavior. Javadoc, online spring-data-mongodb reference etc

          b.) Although the improvement I proposes in spring-data-common is not applicable to this project, since it was assigned to you, the following could be done.
          Add Slice<T> findSlice(Predicate predicate, Pageable page) method in spring-data-commons. Of course this will have implications on other sub-projects.
          And eventually implement that in spring-data-mongodb

          Show
          spasam Seshu Pasam added a comment - Sorry, somehow missed the comment updates. Few things can be done to help users. It was not very obvious that calling this method is 3 times more expensive. And required debugging code in 3 different projects. a.) Document the behavior. Javadoc, online spring-data-mongodb reference etc b.) Although the improvement I proposes in spring-data-common is not applicable to this project, since it was assigned to you, the following could be done. Add Slice<T> findSlice(Predicate predicate, Pageable page) method in spring-data-commons. Of course this will have implications on other sub-projects. And eventually implement that in spring-data-mongodb
          Hide
          olivergierke Oliver Gierke added a comment -

          As indicated, we have a separate ticket for the suggestion you made. This however is not a fix but effectively new API that needs more exhaustive consideration than a bug fix that could've worked around a third party library bug. See we're effectively at the following point:

          1. Pagination might require an additional query. This - and how to avoid it if you don't need the page metadata but access to the page content - is documented in the reference documentation. I've added DATACMNS-548 to improve this and include the mentioning of Slice as return value.
          2. The additional count query Querydsl triggers (which is what I considered this ticket to be about) is nothing we can do about but rely on users upgrading to 3.4.2.
          3. The request for a sliceAll(…) or the like is covered in DATACMNS-541.

          Especially 2 is why marked this as works as designed as we can't do anything in Spring Data MongoDB's code base to reduce the price of a pagination query. I agree on the documentation side of things as well as the request for a slicing CRUD method, but that's got to be tracked in the according tickets of Spring Data Commons.

          Does that make sense?

          Show
          olivergierke Oliver Gierke added a comment - As indicated, we have a separate ticket for the suggestion you made. This however is not a fix but effectively new API that needs more exhaustive consideration than a bug fix that could've worked around a third party library bug. See we're effectively at the following point: 1. Pagination might require an additional query. This - and how to avoid it if you don't need the page metadata but access to the page content - is documented in the reference documentation . I've added DATACMNS-548 to improve this and include the mentioning of Slice as return value. 2. The additional count query Querydsl triggers (which is what I considered this ticket to be about) is nothing we can do about but rely on users upgrading to 3.4.2. 3. The request for a sliceAll(…) or the like is covered in DATACMNS-541 . Especially 2 is why marked this as works as designed as we can't do anything in Spring Data MongoDB's code base to reduce the price of a pagination query. I agree on the documentation side of things as well as the request for a slicing CRUD method, but that's got to be tracked in the according tickets of Spring Data Commons. Does that make sense?
          Hide
          spasam Seshu Pasam added a comment -

          Sounds good. Querydsl issue is already resolved. So there is just one additional call now for the total count. I guess spring-data-commons issue has wider implications.

          Show
          spasam Seshu Pasam added a comment - Sounds good. Querydsl issue is already resolved. So there is just one additional call now for the total count. I guess spring-data-commons issue has wider implications.

            People

            • Assignee:
              olivergierke Oliver Gierke
              Reporter:
              spasam Seshu Pasam
              Last updater:
              Seshu Pasam
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Agile