Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-3171

Composed modules not working on YARN

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: None
    • Fix Version/s: 1.2 GA
    • Component/s: None
    • Labels:
      None

      Description

      From https://github.com/spring-projects/spring-xd/issues/1704

      I am trying to use composed modules when running on YARN.

      In ZK, each child module definition of the composed module gets serialized as follows:

      ```

      {"@class":"org.springframework.xd.module.SimpleModuleDefinition","name":"transform","type":"processor","location":"file:/tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1433789137218_0001/filecache/17/spring-xd-yarn-1.1.2.RELEASE.zip/modules/processor/transform/"}

      ```

      When I try to use the composed module on YARN, it may be deployed to a different container where the "location" file path is not valid. In this case I get the following exception:

      ```
      java.lang.IllegalArgumentException: File must exist
      at org.springframework.boot.loader.data.RandomAccessDataFile.<init>(RandomAccessDataFile.java:67)
      at org.springframework.boot.loader.data.RandomAccessDataFile.<init>(RandomAccessDataFile.java:51)
      at org.springframework.boot.loader.jar.JarFile.<init>(JarFile.java:95)
      at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:61)
      at org.springframework.boot.loader.archive.JarFileArchive.<init>(JarFileArchive.java:57)
      at org.springframework.xd.module.support.ModuleUtils.createModuleClassLoader(ModuleUtils.java:54)
      at org.springframework.xd.module.support.ModuleUtils.createModuleClassLoader(ModuleUtils.java:47)
      at org.springframework.xd.module.options.DefaultModuleOptionsMetadataResolver.resolveNormalMetadata(DefaultModuleOptionsMetadataResolver.java:186)
      at org.springframework.xd.module.options.DefaultModuleOptionsMetadataResolver.resolve(DefaultModuleOptionsMetadataResolver.java:164)
      at org.springframework.xd.module.options.DelegatingModuleOptionsMetadataResolver.resolve(DelegatingModuleOptionsMetadataResolver.java:44)
      at org.springframework.xd.module.options.EnvironmentAwareModuleOptionsMetadataResolver.resolve(EnvironmentAwareModuleOptionsMetadataResolver.java:127)
      at org.springframework.xd.module.options.DefaultModuleOptionsMetadataResolver.resolveComposedModuleMetadata(DefaultModuleOptionsMetadataResolver.java:175)
      at org.springframework.xd.module.options.DefaultModuleOptionsMetadataResolver.resolve(DefaultModuleOptionsMetadataResolver.java:167)
      at org.springframework.xd.module.options.DelegatingModuleOptionsMetadataResolver.resolve(DelegatingModuleOptionsMetadataResolver.java:44)
      at org.springframework.xd.module.options.EnvironmentAwareModuleOptionsMetadataResolver.resolve(EnvironmentAwareModuleOptionsMetadataResolver.java:127)
      at org.springframework.xd.dirt.stream.XDStreamParser.parse(XDStreamParser.java:174)
      ```

      I think the issue may be related to the following line in ArchiveModuleRegistry:

      ```
      String filename = resource.getFile().getCanonicalFile().getName();
      ```

        Attachments

          Activity

            People

            • Assignee:
              eric.bottard Eric Bottard
              Reporter:
              iperumal Ilayaperumal Gopinathan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: