Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-2912

Sqoop Module not running

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Major
    • Resolution: Works as Designed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Story Points:
      0
    • Rank (Obsolete):
      9223372036854775807
    • Sprint:
      Sprint 47

      Description

      Can not get any jobs created using Scoop Module to run. Have tried both the direct implementation using module definition and job create from shell.

      Job creates , deploys and goes into execution mode. Runs endlessly no return. Attaching container log and job definition:

      job create sqoop96 -definition "sqoop --command=import --args='-table INV.MTL_CATEGORY_SETS_B -m=1 --connect jdbc:oracle:thin:@xxxxx.xxx.xxx --username ****** --password *********'" --deploy

      --Container Log
      2015-04-06 15:38:23,720 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1428324095255_0006_000001
      2015-04-06 15:38:24,041 WARN [main] org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
      2015-04-06 15:38:24,058 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
      2015-04-06 15:38:24,058 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id

      { id: 6 cluster_timestamp: 1428324095255 }

      attemptId: 1 } keyId: 723784990)
      2015-04-06 15:38:24,213 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
      2015-04-06 15:38:24,811 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config null
      2015-04-06 15:38:24,864 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
      2015-04-06 15:38:24,895 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
      2015-04-06 15:38:24,897 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
      2015-04-06 15:38:24,898 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
      2015-04-06 15:38:24,899 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
      2015-04-06 15:38:24,900 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
      2015-04-06 15:38:24,906 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
      2015-04-06 15:38:24,906 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
      2015-04-06 15:38:24,907 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
      2015-04-06 15:38:24,926 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
      2015-04-06 15:38:24,941 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
      2015-04-06 15:38:24,960 INFO [main] org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils: Default file system is set solely by core-default.xml therefore - ignoring
      2015-04-06 15:38:24,979 INFO [main] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Emitting job history data to the timeline server is not enabled
      2015-04-06 15:38:25,032 INFO [main] org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
      2015-04-06 15:38:25,295 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
      2015-04-06 15:38:25,353 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
      2015-04-06 15:38:25,353 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system started
      2015-04-06 15:38:25,362 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for job_1428324095255_0006 to jobTokenSecretManager
      2015-04-06 15:38:25,476 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing job_1428324095255_0006 because: not enabled;
      2015-04-06 15:38:25,489 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job job_1428324095255_0006 = 0. Number of splits = 1
      2015-04-06 15:38:25,489 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job job_1428324095255_0006 = 0
      2015-04-06 15:38:25,489 INFO [main] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1428324095255_0006Job Transitioned from NEW to INITED
      2015-04-06 15:38:25,490 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal, non-uberized, multi-container job job_1428324095255_0006.
      2015-04-06 15:38:25,515 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
      2015-04-06 15:38:25,522 INFO Socket Reader #1 for port 51164 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 51164
      2015-04-06 15:38:25,543 INFO [main] org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
      2015-04-06 15:38:25,543 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2015-04-06 15:38:25,543 INFO [IPC Server listener on 51164] org.apache.hadoop.ipc.Server: IPC Server listener on 51164: starting
      2015-04-06 15:38:25,544 INFO [main] org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated MRClientService at sandbox.hortonworks.com/192.168.3.128:51164
      2015-04-06 15:38:25,598 INFO [main] org.mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
      2015-04-06 15:38:25,601 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http request log for http.requests.mapreduce is not defined
      2015-04-06 15:38:25,609 INFO [main] org.apache.hadoop.http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
      2015-04-06 15:38:25,614 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context mapreduce
      2015-04-06 15:38:25,614 INFO [main] org.apache.hadoop.http.HttpServer2: Added filter AM_PROXY_FILTER (class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context static
      2015-04-06 15:38:25,616 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /mapreduce/*
      2015-04-06 15:38:25,616 INFO [main] org.apache.hadoop.http.HttpServer2: adding path spec: /ws/*
      2015-04-06 15:38:25,624 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty bound to port 38343
      2015-04-06 15:38:25,624 INFO [main] org.mortbay.log: jetty-6.1.26.hwx
      2015-04-06 15:38:25,662 INFO [main] org.mortbay.log: Extract jar:file:/usr/hdp/2.2.0.0-2041/hadoop-yarn/hadoop-yarn-common-2.6.0.2.2.0.0-2041.jar!/webapps/mapreduce to /tmp/Jetty_0_0_0_0_38343_mapreduce____86ppbg/webapp
      2015-04-06 15:38:25,901 INFO [main] org.mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:38343
      2015-04-06 15:38:25,901 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web app /mapreduce started at 38343
      2015-04-06 15:38:26,261 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Registered webapp guice modules
      2015-04-06 15:38:26,265 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.speculate.DefaultSpeculator: JOB_CREATE job_1428324095255_0006
      2015-04-06 15:38:26,267 INFO [main] org.apache.hadoop.ipc.CallQueueManager: Using callQueue class java.util.concurrent.LinkedBlockingQueue
      2015-04-06 15:38:26,267 INFO Socket Reader #1 for port 58807 org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 58807
      2015-04-06 15:38:26,272 INFO [IPC Server Responder] org.apache.hadoop.ipc.Server: IPC Server Responder: starting
      2015-04-06 15:38:26,272 INFO [IPC Server listener on 58807] org.apache.hadoop.ipc.Server: IPC Server listener on 58807: starting
      2015-04-06 15:38:26,300 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: nodeBlacklistingEnabled:true
      2015-04-06 15:38:26,300 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: maxTaskFailuresPerNode is 3
      2015-04-06 15:38:26,300 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: blacklistDisablePercent is 33
      2015-04-06 15:38:26,343 INFO [main] org.apache.hadoop.yarn.client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
      2015-04-06 15:38:26,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: maxContainerCapability: <memory:2250, vCores:32>
      2015-04-06 15:38:26,410 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: queue: default
      2015-04-06 15:38:26,415 INFO [main] org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Upper limit on the thread pool size is 500
      2015-04-06 15:38:26,417 INFO [main] org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy: yarn.client.max-cached-nodemanagers-proxies : 0
      2015-04-06 15:38:26,428 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1428324095255_0006Job Transitioned from INITED to SETUP
      2015-04-06 15:38:26,430 INFO CommitterEvent Processor #0 org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler: Processing the event EventType: JOB_SETUP
      2015-04-06 15:38:26,442 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1428324095255_0006Job Transitioned from SETUP to RUNNING
      2015-04-06 15:38:26,457 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskImpl: task_1428324095255_0006_m_000000 Task Transitioned from NEW to SCHEDULED
      2015-04-06 15:38:26,459 INFO [AsyncDispatcher event handler] org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: attempt_1428324095255_0006_m_000000_0 TaskAttempt Transitioned from NEW to UNASSIGNED
      2015-04-06 15:38:26,459 INFO [Thread-50] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: mapResourceRequest:<memory:1024, vCores:1>
      2015-04-06 15:38:26,531 INFO [eventHandlingThread] org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Event Writer setup for JobId: job_1428324095255_0006, File: hdfs://sandbox:8020/tmp/hadoop-yarn/staging/root/.staging/job_1428324095255_0006/job_1428324095255_0006_1.jhist
      2015-04-06 15:38:27,413 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before Scheduling: PendingReds:0 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:0 AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:0 ContRel:0 HostLocal:0 RackLocal:0
      2015-04-06 15:38:27,466 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources() for application_1428324095255_0006: ask=1 release= 0 newContainers=0 finishedContainers=0 resourcelimit=<memory:500, vCores:0> knownNMs=1

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            monasj1 Jared Monast
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: