Spring Integration
  1. Spring Integration
  2. INT-1857

File Adapter: FileReadingMessageSource - Sorted order of list of files in a FileListFilter not preserved by Spring Integration

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Complete
    • Affects Version/s: 2.0.3
    • Fix Version/s: 2.0.4
    • Component/s: Core, File Support
    • Labels:
      None

      Description

      I have a filter (implements FileListFilter) that implements a custom sorting algorithm for a given list of files.
      This filter is defined as follows:

      <file:inbound-channel-adapter channel="filesIn"  filter="customFilterBean"
      	                              directory="file:${java.io.tmpdir}/spring-integration-samples/input">
      	    <integration:poller id="poller" fixed-delay="5000"/>
      	</file:inbound-channel-adapter>
      

      The filter then sorts the list of files. The filter implementation returns a list of sorted files. However, the sorted list of files is ignored by the FileReadingMessageSource

      Looking at the implementation (also have a test case to prove it) of FileReadingMessageSource, it appears to convert the 'sorted list' into a 'set' - thereby ignoring
      the sort in the FileListFilter

      private void scanInputDirectory() {
      		List<File> filteredFiles = scanner.listFiles(directory);  --> This list is sorted
      		Set<File> freshFiles = new HashSet<File>(filteredFiles);  --> Converted to a SET losing the sort order
      		if (!freshFiles.isEmpty()) {
      			toBeReceived.addAll(freshFiles);
      			if (logger.isDebugEnabled()) {
      				logger.debug("Added to queue: " + freshFiles);
      			}
      		}
      	}
      

      Our use case calls for processing the incoming files in a specific order.

        Activity

        Hide
        Benjamin Wootton added a comment -

        It's nothing to do with the set.

        Looks like you need to specify a comparator on the inbound channel adapter:

        "If none is provided, the order will be determined by the java.io.File implementation of Comparable."

        Show
        Benjamin Wootton added a comment - It's nothing to do with the set. Looks like you need to specify a comparator on the inbound channel adapter: "If none is provided, the order will be determined by the java.io.File implementation of Comparable."
        Hide
        Anuj Kathuria added a comment -

        Thanks for your reply. I have specified a filter on the inbound channel adapter - where I have a custom comparator that does the sorting, and returns the sorted list.

        However, in FileReadingMessageSource.scanInputDirectory(), the sorted ordered list that I return from the custom filter implementation is converted to a SET
        Set<File> freshFiles = new HashSet<File>(filteredFiles)

        It appears from the code that the sort order for the list of files is not preserved,when the queue is created.

        Show
        Anuj Kathuria added a comment - Thanks for your reply. I have specified a filter on the inbound channel adapter - where I have a custom comparator that does the sorting, and returns the sorted list. However, in FileReadingMessageSource.scanInputDirectory(), the sorted ordered list that I return from the custom filter implementation is converted to a SET Set<File> freshFiles = new HashSet<File>(filteredFiles) It appears from the code that the sort order for the list of files is not preserved,when the queue is created.
        Hide
        Benjamin Wootton added a comment -

        I don't have a lot of time to test but looks as though the behaviour is to provide the ordering at the point they are added into the blocking queue - after they've come out of the set.

        toBeReceived.addAll(freshFiles);

        Because no comparator is provided, this falls back onto the default java.io.File comparable.

        Show
        Benjamin Wootton added a comment - I don't have a lot of time to test but looks as though the behaviour is to provide the ordering at the point they are added into the blocking queue - after they've come out of the set. toBeReceived.addAll(freshFiles); Because no comparator is provided, this falls back onto the default java.io.File comparable.
        Hide
        Anuj Kathuria added a comment -

        Thanks for looking. If that is the case, the code in 'FileReadingMessageSource.scanInputDirectory()' should be modified. The comparator property can also be exposed. On the other hand, Mark did propose sorting it in earlier in the chain (in the filter) which I did, but as you are aware, that did not work because of the issue mentioned in the defect.

        Show
        Anuj Kathuria added a comment - Thanks for looking. If that is the case, the code in 'FileReadingMessageSource.scanInputDirectory()' should be modified. The comparator property can also be exposed. On the other hand, Mark did propose sorting it in earlier in the chain (in the filter) which I did, but as you are aware, that did not work because of the issue mentioned in the defect.
        Hide
        Mark Fisher added a comment -

        Can you please try out the latest and let me know if it's preserving order for you now? All I did was switch from HashSet to LinkedHashSet within the code that you referenced above. That way the List order should be preserved. Here's the commit:
        http://git.springsource.org/spring-integration/spring-integration/commit/bd81fe9db2ff140294b428145e3f99c2ff9ca55e

        Show
        Mark Fisher added a comment - Can you please try out the latest and let me know if it's preserving order for you now? All I did was switch from HashSet to LinkedHashSet within the code that you referenced above. That way the List order should be preserved. Here's the commit: http://git.springsource.org/spring-integration/spring-integration/commit/bd81fe9db2ff140294b428145e3f99c2ff9ca55e
        Hide
        Anuj Kathuria added a comment -

        Mark, I built using your commit. However, as Benjamin mentioned earlier, and upon further debugging the code , it appears that the ordering is actually done at the time the items are added to the PriorityBlockingQueue.

        Apparently, if we specify a comparator in the file adapter discussion, then it works fine.
        <file:inbound-channel-adapter channel="filesIn" filter="customFilterBean"
        directory="file:$

        {java.io.tmpdir}

        /spring-integration-samples/input"
        comparator="customComparatorBean">
        ......
        </file:inbound-channel-adapter>

        I have verified that the ordering is actually a property of inbound adapter, and can be specified using a comparator.

        Show
        Anuj Kathuria added a comment - Mark, I built using your commit. However, as Benjamin mentioned earlier, and upon further debugging the code , it appears that the ordering is actually done at the time the items are added to the PriorityBlockingQueue. Apparently, if we specify a comparator in the file adapter discussion, then it works fine. < file:inbound-channel-adapter channel="filesIn" filter="customFilterBean" directory="file:$ {java.io.tmpdir} /spring-integration-samples/input" comparator="customComparatorBean"> ...... </ file:inbound-channel-adapter > I have verified that the ordering is actually a property of inbound adapter, and can be specified using a comparator.
        Hide
        Mark Fisher added a comment -

        So now that you pass a reference to your Comparator implementation there, everything is okay?

        If so, I will close the issue.

        Thanks,
        Mark

        Show
        Mark Fisher added a comment - So now that you pass a reference to your Comparator implementation there, everything is okay? If so, I will close the issue. Thanks, Mark
        Hide
        Anuj Kathuria added a comment -

        Thanks, Mark!

        Show
        Anuj Kathuria added a comment - Thanks, Mark!

          People

          • Assignee:
            Mark Fisher
            Reporter:
            Anuj Kathuria
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved: