Uploaded image for project: 'Spring for Apache Hadoop'
  1. Spring for Apache Hadoop
  2. SHDP-95

DistributedCacheFactoryBean fails to add JAR into classpath

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Complete
    • Affects Version/s: 1.0.0.RC1
    • Fix Version/s: 1.0.0.RC1
    • Component/s: MR
    • Labels:
      None
    • Environment:
      Hadoop 0.23.3 branch - CDH4

      Description

      Do not append #sss if you call DistributedCache.addFileToClassPath, it ends with following error:

      2/08/04 00:40:21 ERROR security.UserGroupInformation: PriviledgedActionException as:crawler (auth:SIMPLE) cause:java.io.FileNotFoundException: File does not exist: hdfs://ponto.amerinoc.com:9000/cznewgen/config.jar#config.jar
      12/08/04 00:40:21 ERROR spring.SpringHadoopRunner: Tool fetch failed
      java.io.FileNotFoundException: File does not exist: hdfs://ponto.amerinoc.com:9000/cznewgen/config.jar#config.jar
      at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:729)
      at org.apache.hadoop.fs.FileSystem.resolvePath(FileSystem.java:636)

      This code works:
      for(String config:parsed_configs)

      { DistributedCache.addFileToClassPath(new Path(config), conf); }

      Following code caused error:
      DistributedCacheFactoryBean dcache = new DistributedCacheFactoryBean();
      dcache.setConfiguration(conf);
      dcache.setCreateSymlink(false);
      Set <DistributedCacheFactoryBean.CacheEntry> jars = new HashSet <DistributedCacheFactoryBean.CacheEntry> ();
      for(String config:parsed_configs)

      { DistributedCacheFactoryBean.CacheEntry entry = new DistributedCacheFactoryBean.CacheEntry(DistributedCacheFactoryBean.CacheEntry.EntryType.CP, config); jars.add(entry); }

      dcache.setEntries(jars);
      dcache.afterPropertiesSet();

        Attachments

          Activity

            People

            Assignee:
            costin Costin Leau
            Reporter:
            hsn Radim Kolar
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: