Uploaded image for project: 'Spring XD'
  1. Spring XD
  2. XD-3176

Using HDFS for custom module home doesn't work with Kerberized Hadoop cluster

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Priority: Critical
    • Resolution: Complete
    • Affects Version/s: 1.2 RC1
    • Fix Version/s: 1.2 GA
    • Component/s: Configuration, Runtime
    • Labels:
      None

      Description

      I tried setting the xd.customModule.home property to point to a Kerberized Hadoop cluster with all usual security config settings provided. It failed with the following exception:

      org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'moduleRegistry' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Cannot create inner bean 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#19f459aa' of type [org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean] while setting constructor argument with key [1]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#19f459aa' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Invocation of init method failed; nested exception is org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:313) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:122) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveManagedList(BeanDefinitionValueResolver.java:382) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveValueIfNecessary(BeanDefinitionValueResolver.java:157) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.ConstructorResolver.resolveConstructorArguments(ConstructorResolver.java:648) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.ConstructorResolver.autowireConstructor(ConstructorResolver.java:140) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.autowireConstructor(AbstractAutowireCapableBeanFactory.java:1139) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1042) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:755) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:757) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:480) ~[spring-context-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:686) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
      	at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
      	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:139) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
      	at org.springframework.boot.builder.SpringApplicationBuilder.run(SpringApplicationBuilder.java:129) [spring-boot-1.2.3.RELEASE.jar:1.2.3.RELEASE]
      	at org.springframework.xd.dirt.server.admin.AdminServerApplication.run(AdminServerApplication.java:95) [spring-xd-dirt-1.2.0.BUILD-SNAPSHOT.jar:1.2.0.BUILD-SNAPSHOT]
      	at org.springframework.xd.dirt.server.admin.AdminServerApplication.main(AdminServerApplication.java:79) [spring-xd-dirt-1.2.0.BUILD-SNAPSHOT.jar:1.2.0.BUILD-SNAPSHOT]
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean#19f459aa' defined in class path resource [META-INF/spring-xd/internal/repositories.xml]: Invocation of init method failed; nested exception is org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1574) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:539) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:299) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	... 22 common frames omitted
      Caused by: org.apache.hadoop.security.AccessControlException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[na:1.7.0_67]
      	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) ~[na:1.7.0_67]
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[na:1.7.0_67]
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:526) ~[na:1.7.0_67]
      	at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2755) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2724) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:870) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:866) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:866) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:859) ~[hadoop-hdfs-2.6.0.jar:na]
      	at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:1817) ~[hadoop-common-2.6.0.jar:na]
      	at org.springframework.xd.dirt.module.ExtendedResource$HdfsExtendedResource.mkdirs(ExtendedResource.java:127) ~[spring-xd-dirt-1.2.0.BUILD-SNAPSHOT.jar:1.2.0.BUILD-SNAPSHOT]
      	at org.springframework.xd.dirt.module.WritableResourceModuleRegistry.afterPropertiesSet(WritableResourceModuleRegistry.java:123) ~[spring-xd-dirt-1.2.0.BUILD-SNAPSHOT.jar:1.2.0.BUILD-SNAPSHOT]
      	at org.springframework.xd.dirt.module.CustomModuleRegistryFactoryBean.afterPropertiesSet(CustomModuleRegistryFactoryBean.java:79) ~[spring-xd-dirt-1.2.0.BUILD-SNAPSHOT.jar:1.2.0.BUILD-SNAPSHOT]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1633) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1570) ~[spring-beans-4.1.6.RELEASE.jar:4.1.6.RELEASE]
      	... 25 common frames omitted
      Caused by: org.apache.hadoop.ipc.RemoteException: SIMPLE authentication is not enabled.  Available:[TOKEN, KERBEROS]
      	at org.apache.hadoop.ipc.Client.call(Client.java:1468) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.ipc.Client.call(Client.java:1399) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232) ~[hadoop-common-2.6.0.jar:na]
      	at com.sun.proxy.$Proxy79.mkdirs(Unknown Source) ~[na:na]
      	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:539) ~[hadoop-hdfs-2.6.0.jar:na]
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_67]
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_67]
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_67]
      	at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_67]
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) ~[hadoop-common-2.6.0.jar:na]
      	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[hadoop-common-2.6.0.jar:na]
      	at com.sun.proxy.$Proxy80.mkdirs(Unknown Source) ~[na:na]
      	at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2753) ~[hadoop-hdfs-2.6.0.jar:na]
      	... 37 common frames omitted
      2015-06-10T14:49:20-0400 1.2.0.SNAP ERROR main boot.SpringApplication - Application startup failed
      

        Attachments

          Activity

            People

            Assignee:
            thomas.risberg Thomas Risberg
            Reporter:
            thomas.risberg Thomas Risberg
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: