Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-3132

Sqoop job doesn't run when deployed on YARN on Ambari deployed HDP

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: None
    • Fix Version/s: 1.2 GA
    • Component/s: None
    • Labels:
      None

      Description

      Got this error when submitting Sqoop job:

      2015-06-01 19:09:42,932 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(75)) - Sqoop command: import
      2015-06-01 19:09:42,939 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(76)) - Using args: [--table, XD_JOB_REGISTRY, --target-dir, /xd/sqoop2, -m=1]
      2015-06-01 19:09:42,939 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:main(77)) - Mapreduce home: /usr/hdp/2.2.4.2-2/hadoop-mapreduce
      2015-06-01 19:09:42,977 WARN [main] conf.Configuration (Configuration.java:(646)) - DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
      2015-06-01 19:09:42,984 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: fs.defaultFS=hdfs://hawaii:8020
      2015-06-01 19:09:43,743 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.hostname=hawaii
      2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.address=hawaii:8050
      2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: yarn.resourcemanager.scheduler.address=hawaii
      2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: mapreduce.framework.name=yarn
      2015-06-01 19:09:43,758 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:setConfigurationProperty(168)) - Setting configuration property: mapreduce.jobhistory.address=hawaii
      2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: yarn.resourcemanager.scheduler.address=hawaii:8030
      2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: mapreduce.application.classpath=$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*:$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*:$PWD/mr-framework/hadoop/share/hadoop/common/*:$PWD/mr-framework/hadoop/share/hadoop/common/lib/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/*:$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/*:$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*:/usr/hdp/2.2.4.2-2/hadoop/lib/hadoop-lzo-0.6.0.2.2.4.2-2.jar:/etc/hadoop/conf/secure
      2015-06-01 19:09:43,760 INFO [main] sqoop.SqoopRunner (SqoopRunner.java:createConfiguration(158)) - Setting configuration property: mapreduce.application.framework.path=/hdp/apps/2.2.4.2-2/mapreduce/mapreduce.tar.gz#mr-framework
      2015-06-01 19:09:44,141 INFO [main] sqoop.Sqoop (Sqoop.java:(92)) - Running Sqoop version: 1.4.5
      2015-06-01 19:09:44,691 INFO [main] manager.SqlManager (SqlManager.java:initOptionDefaults(98)) - Using default fetchSize of 1000
      2015-06-01 19:09:44,691 INFO [main] tool.CodeGenTool (CodeGenTool.java:generateORM(92)) - Beginning code generation
      2015-06-01 19:09:45,057 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM XD_JOB_REGISTRY AS t WHERE 1=0
      2015-06-01 19:09:45,074 INFO [main] manager.SqlManager (SqlManager.java:execute(749)) - Executing SQL statement: SELECT t.* FROM XD_JOB_REGISTRY AS t WHERE 1=0
      2015-06-01 19:09:45,148 INFO [main] orm.CompilationManager (CompilationManager.java:findHadoopJars(94)) - HADOOP_MAPRED_HOME is /usr/hdp/2.2.4.2-2/hadoop-mapreduce
      2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(184)) - It seems as though you are running sqoop with a JRE.
      2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(185)) - Sqoop requires a JDK that can compile Java code.
      2015-06-01 19:09:45,230 ERROR [main] orm.CompilationManager (CompilationManager.java:compile(186)) - Please install a JDK and set $JAVA_HOME to use it.
      2015-06-01 19:09:45,232 ERROR [main] tool.ImportTool (ImportTool.java:run(609)) - Encountered IOException running import job: java.io.IOException: Could not start Java compiler.
      at org.apache.sqoop.orm.CompilationManager.compile(CompilationManager.java:187)
      at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:97)
      at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
      at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:601)
      at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
      at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
      at org.springframework.xd.sqoop.SqoopRunner.main(SqoopRunner.java:87)
      

        Attachments

          Activity

            People

            • Assignee:
              thomas.risberg Thomas Risberg
              Reporter:
              thomas.risberg Thomas Risberg
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: