Kerberos Integration with Hadoop
Change the hostname if not already changed
Run the command
> sudo hostnamectl set-hostname hadoop.os.com # Here we are setting the hostname as
os.com
> sudo systemctl restart systemd-hostnamed
> sudo reboot
Once the system is rebooted, check the hostname
Kerberos installation
/etc/krb5.conf
[libdefaults]
default_realm = OS.COM
# The following krb5.conf variables are only for MIT Kerberos.
kdc_timesync = 1
ccache_type = 4
forwardable = true
proxiable = true
# The following encryption type specification will be used by MIT Kerberos
# if uncommented. In general, the defaults in the MIT Kerberos code are
# correct and overriding these specifications only serves to disable new
# encryption types as they are added, creating interoperability problems.
#
# The only time when you might need to uncomment these lines and change
# the enctypes is if you have local software that will break on ticket
# caches containing ticket encryption types it doesn't know about (such as
# old versions of Sun Java).
# default_tgs_enctypes = des3-hmac-sha1
# default_tkt_enctypes = des3-hmac-sha1
# permitted_enctypes = des3-hmac-sha1
# The following libdefaults parameters are only for Heimdal Kerberos.
fcc-mit-ticketflags = true
[realms]
OS.COM = {
kdc = localhost
admin_server = localhost
}
[domain_realm]
.os.com = OS.COM
os.com=OS.COM
Installing JSVC
> echo $JAVA_HOME (Make sure this command is pointing to right JDK)
> wget https://dlcdn.apache.org//commons/daemon/source/commons-daemon-1.4.0-src.tar.gz
> tar -zxvf commons-daemon-1.4.0-src.tar.gz
> cd commons-daemon-1.4.0-src/src/native/unix
> ./configure
> make
>sudo cp jsvc /usr/bin
Edit hadoop-env.sh
export JSVC_HOME=/usr/bin
export HADOOP_OPTS="-Djava.security.krb5.conf=/etc/krb5.conf"
export HDFS_DATANODE_SECURE_USER=root
Reference:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SecureMode.html
Add the following properties in core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/home/hadoop/tmpdata</value>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://127.0.0.1:9000</value>
</property>
<property>
<name>hadoop.security.authentication</name>
<value>kerberos</value>
</property>
<property>
<name>hadoop.security.authorization</name>
<value>true</value>
</property>
</configuration>
Add the following properties in hdfs-site.xml (Please check the hostname and realm )
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/dfsdata/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/dfsdata/datanode</value>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.namenode.kerberos.principal</name>
<value>hdfs/hadoop.os.com@OS.COM</value>
</property>
<property>
<name>dfs.namenode.keytab.file</name>
<value>/etc/nn.keytab</value>
</property>
<property>
<name>dfs.datanode.kerberos.principal</name>
<value>hdfs/hadoop.os.com@OS.COM</value>
</property>
<property>
<name>dfs.datanode.keytab.file</name>
<value>/etc/nn.keytab</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.block.access.token.enable</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>0.0.0.0:1004</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>0.0.0.0:1006</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
</configuration>
Create new realm with the new hostname
sudo krb5_newrealm
NOTE: if any error comes then delete the files present in /var/lib/krb5kdc folder
sudo rm -rf /var/lib/krb5kdc/*
Now again add the newrealm
Generating the keytab
sudo kadmin.local -q "addprinc -randkey hdfs/hadoop.os.com@OS.COM"
sudo kadmin.local -q "ktadd -k /etc/nn.keytab hdfs/hadoop.os.com@OS.COM"
→ Change the permission of hadoop user directory to 755
Run name node and datanode from root user
