Uploaded image for project: 'Spring for Apache Hadoop'
  1. Spring for Apache Hadoop
  2. SHDP-135

Introduce ParentFirstURLClassLoader in HadoopCodeExecutor

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 1.0.0.GA
    • Fix Version/s: None
    • Component/s: MR
    • Labels:
      None

      Description

      We've been experiencing interesting classloading errors while try to launch an "uberjar" with help of Spring Hadoop. We're using JCascalog in our project and as such we depend on Clojure's jar being present on the classpath. One of Clojure's peculiarities is that its runtime initialization fails if the main RT class was loaded by a classloader that is different from the context classloader of the executing thread. This is discussed on Google Groups here: https://groups.google.com/forum/?hl=en&fromgroups=#!topic/scala-user/Bh_YmI6e-wY.

      The way ParentLastURLClassLoader is implemented is that it delegates up the classloader hierarchy first and only then tries to load classes from the supplied uberjar. So, in case a clojure.jar is also present on the main path (i.e. as can be the case during integration tests or we may simply need it to be in the WEB-INF/lib for other reasons) the submission of the clojure containing "uberjar" will fail with the following exception: Caused by: java.lang.IllegalStateException: Attempting to call unbound fn: #'clojure.core/refer

      It may be reasonable, in my opinion, to make class loading delegation configurable in HadoopCodeExecutor.

      I think this is an important RFE, and I am curious to hear your thoughts on this. On a more general note, was there a particular reason to choose "parent first" style of class loading delegation especially given that this is not how delegation typically works in application servers?

      Thank you very much for your attention to this issue.

        Attachments

          Activity

            People

            Assignee:
            thomas.risberg Thomas Risberg
            Reporter:
            kyrill007 Kyrill Alyoshin
            Votes:
            2 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: