Uploaded image for project: 'Spring for Apache Hadoop'
  1. Spring for Apache Hadoop
  2. SHDP-394

Can't run mapreduce job in local mode

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Complete
    • Affects Version/s: 2.0.0.GA
    • Fix Version/s: 2.0.3, 2.1.0.M2
    • Component/s: MR
    • Labels:
      None

      Description

      I would like to run the mapreduce jobs without running any hadoop server or having HDFS to test them. So I use mapreduce.framework.name=local. Unfortunately the org.springframework.data.hadoop.mapreduce.JobUtils#getRunningJob sets this configuration like this:

      				Configuration cfg = job.getConfiguration();
      				cfg.set("mapreduce.framework.name", "yarn");
      

      I could override this while debugging and could run the job but can't do this from configuration. Can you tell me why is this here and can you remove it or make it configurable or tell me why should I not use local mode?

      I only tested this with 2.0.0-RELEASE but the code is there in version 2.1.0.M1 too.

        Attachments

          Activity

            People

            Assignee:
            thomas.risberg Thomas Risberg
            Reporter:
            tewe tewe
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: