Details
Description
Running Spring-XD singlenode with a kerberized hadoop cluster on CDH 5.3.2. with JDK 1.7 and JCE 1.7.
The kerberos ticket policies are:
- expiration: 24 hours
- renew: 7 days
I need to keep the Spring XD server running constantly because my flows are always waiting for incoming files to be ingested into the HDFS, but the kerberos session expires if there aren't jobs to run before the expiration date. The expiration policies can't be changed due internal company policies.
Is there a way which Spring XD can generate a new ticket instead of renew the current one when a job or stream start executing?
The Spring XD server has configured the hadoop.properties like:
- Use servers.yml to change URI for namenode
- You can add additional properties in this file
dfs.namenode.kerberos.principal=hdfs/[email protected]
yarn.resourcemanager.principal=yarn/[email protected]
yarn.application.classpath=/opt/cloudera/parcels/CDH/lib/hadoop/,/opt/cloudera/parcels/CDH/lib/hadoop/lib/,/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/,/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/,/opt/cloudera/parcels/CDH/lib/hadoop-yarn/,/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/
hadoop.security.authorization=true
hadoop.security.authentication=kerberos
spring.hadoop.userKeytab=file:///export/home/user/user.keytab
[email protected]
#Connecting to Kerberized Hadoop (Spring XD doc configuration Appendix D)
spring.hadoop.security.authMethod=kerberos
spring.hadoop.security.userKeytab=/export/home/user/user.keytab
[email protected]
spring.hadoop.security.namenodePrincipal=hdfs/[email protected]
spring.hadoop.security.rmManagerPrincipal=yarn/[email protected]