Introduction

SSH, or secure shell, is a secure protocol and the most common way of safely
administering remote servers. Using a number of encryption technologies, SSH
provides a mechanism for establishing a cryptographically secured connection
between two parties, authenticating each side to the other, and passing
commands and output back and forth.
In this guide, we will be examining the underlying encryption techniques that
SSH employs and the methods it uses to establish secure connections. This
information can be useful for understanding the various layers of encryption
and the different steps needed to form a connection and authenticate both
parties.

Understanding Symmetric Encryption, Asymmetric Encryption, and

Hashes
In order to secure the transmission of information, SSH employs a number of
different types of data manipulation techniques at various points in the
transaction. These include forms of symmetrical encryption, asymmetrical
encryption, and hashing.

Symmetrical Encryption
The relationship of the components that encrypt and decrypt data determines
whether an encryption scheme is symmetrical or asymmetrical.
Symmetrical encryption is a type of encryption where one key can be used to
encrypt messages to the opposite party, and also to decrypt the messages
received from the other participant. This means that anyone who holds the key
can encrypt and decrypt messages to anyone else holding the key.
This type of encryption scheme is often called “shared secret” encryption, or
“secret key” encryption. There is typically only a single key that is used for all
operations or a pair of keys where the relationship is discoverable and it’s trivial
to derive the opposite key.
Symmetric keys are used by SSH in order to encrypt the entire connection.
Contrary to what some users assume, public/private asymmetrical key pairs
that can be created are only used for authentication, not encrypting the
connection. The symmetrical encryption allows even password authentication
to be protected against snooping.
The client and server both contribute toward establishing this key, and the
resulting secret is never known to outside parties. The secret key is created
through a process known as a key exchange algorithm. This exchange results in
the server and client both arriving at the same key independently by sharing
certain pieces of public data and manipulating them with certain secret data.
This process is explained in greater detail later on.
The symmetrical encryption key created by this procedure is session-based and
constitutes the actual encryption for the data sent between server and client.
Once this is established, the rest of the data must be encrypted with this shared
secret. This is done prior to authenticating a client.
SSH can be configured to use a variety of different symmetrical cipher systems,
including Advanced Encryption Standard (AES), Blowfish, 3DES, CAST128, and
Arcfour. The server and client can both decide on a list of their supported
ciphers, ordered by preference. The first option from the client’s list that is
available on the server is used as the cipher algorithm in both directions.
On Ubuntu 20.04, both the client and the server are defaulted like the following:
• chacha20-poly1305@openssh.com
• aes128-ctr
• aes192-ctr
• aes256-ctr
• aes128-gcm@openssh.com
• aes256-gcm@openssh.com
This means that if two Ubuntu 20.04 machines are connecting to each other
(without overriding the default ciphers through configuration options), they will
always default to using the chacha20-poly1305@openssh.com cipher to encrypt
their connection.

Asymmetrical Encryption
Asymmetrical encryption is different from symmetrical encryption because to
send data in a single direction, two associated keys are needed. One of these
keys is known as the private key, while the other is called the public key.
The public key can be freely shared with any party. It is associated with its
paired key, but the private key cannot be derived from the public key. The
mathematical relationship between the public key and the private key allows
the public key to encrypt messages that can only be decrypted by the private
key. This is a one-way ability, meaning that the public key has no ability to
decrypt the messages it writes, nor can it decrypt anything the private key may
send it.
The private key should be kept entirely secret and should never be shared with
another party. This is a key requirement for the public key paradigm to work.
The private key is the only component capable of decrypting messages that
were encrypted using the associated public key. By virtue of this fact, any entity
capable of decrypting these messages has demonstrated that they are in
control of the private key.
SSH uses asymmetric encryption in a few different places. During the initial key
exchange process used to set up the symmetrical encryption (used to encrypt
the session), asymmetrical encryption is used. In this stage, both parties
produce temporary key pairs and exchange the public key in order to produce
the shared secret that will be used for symmetrical encryption.
The more well-discussed use of asymmetrical encryption with SSH comes from
SSH key-based authentication. SSH key pairs can be used to authenticate a
client to a server. The client creates a key pair and then uploads the public key
to any remote server it wishes to access. This is placed in a file called
authorized_keys within the ~/.ssh directory in the user account’s home
directory on the remote server.
After the symmetrical encryption is established to secure communications
between the server and client, the client must authenticate to be allowed
access. The server can use the public key in this file to encrypt a challenge
message to the client. If the client can prove that it was able to decrypt this
message, it has demonstrated that it owns the associated private key. Then the
server can set up the environment for the client.

Hashing
Another form of data manipulation that SSH takes advantage of is
cryptographic hashing. Cryptographic hash functions are methods of creating a
succinct “signature” or summary of a set of information. Their main
distinguishing attributes are that they are never meant to be reversed, they are
virtually impossible to influence predictably, and they are practically unique.
Using the same hashing function and message should produce the same hash;
modifying any portion of the data should produce an entirely different hash. A
user should not be able to produce the original message from a given hash, but
they should be able to tell if a given message produced a given hash.
Given these properties, hashes are mainly used for data integrity purposes and
to verify the authenticity of communication. The main use in SSH is with HMAC,
or hash-based message authentication codes. These are used to ensure the
message text that’s received is intact and unmodified.
As part of the symmetrical encryption negotiation outlined previously, a
message authentication code (MAC) algorithm is selected. The algorithm is
chosen by working through the client’s list of acceptable MAC choices. The first
one on this list that the server supports will be used.
Each message sent after the encryption is negotiated must contain a MAC so
that the other party can verify the packet integrity. The MAC is calculated from
the symmetrical shared secret, the packet sequence number of the message,
and the actual message content.
The MAC itself is sent outside of the symmetrically encrypted area as the final
part of the packet. Researchers generally recommend this method of encrypting
the data first and then calculating the MAC.

Understanding How SSH Works

You probably already have a basic understanding of how SSH works. The SSH
protocol employs a client-server model to authenticate two parties and encrypt
the data between them.
The server component listens on a designated port for connections. It is
responsible for negotiating the secure connection, authenticating the
connecting party, and spawning the correct environment if the credentials are
accepted.
The client is responsible for beginning the initial transmission control protocol
(TCP) handshake with the server, negotiating the secure connection, verifying
that the server’s identity matches previously recorded information, and
providing credentials to authenticate.
An SSH session is established in two separate stages. The first is to agree upon
and establish encryption to protect future communication. The second stage is
to authenticate the user and discover whether access to the server should be
granted.

Negotiating Encryption for the Session

When a TCP connection is made by a client, the server responds with the
protocol versions it supports. If the client can match one of the acceptable
protocol versions, the connection continues. The server also provides its public
host key, which the client can use to check whether this was the intended host.
At this point, both parties negotiate a session key using a version of something
called the Diffie-Hellman algorithm. This algorithm (and its variants) make it
possible for each party to combine their own private data with public data from
the other system to arrive at an identical secret session key.
The session key will be used to encrypt the entire session. The public and
private key pairs used for this part of the procedure are completely separate
from the SSH keys used to authenticate a client to the server.
The basis of this procedure for classic Diffie-Hellman are:
• Both parties agree on a large prime number, which will serve as a seed
value.
• Both parties agree on an encryption generator (typically AES), which will
be used to manipulate the values in a predefined way.
• Independently, each party comes up with another prime number which is
kept secret from the other party. This number is used as the private key
 for this interaction (different from the private SSH key used for
authentication).
• The generated private key, the encryption generator, and the shared
prime number are used to generate a public key that is derived from the
private key, but which can be shared with the other party.
• Both participants then exchange their generated public keys.
• The receiving entity uses their own private key, the other party’s public
key, and the original shared prime number to compute a shared secret
key. Although this is independently computed by each party, using
opposite private and public keys, it will result in the same shared secret
key.
• The shared secret is then used to encrypt all communication that follows.
This process allows each party to equally participate in generating the shared
secret, which does not allow one end to control the secret. It also accomplishes
the task of generating an identical shared secret without ever having to send
that information over insecure channels. The shared secret encryption that is
used for the rest of the connection is called binary packet protocol.
The generated secret is a symmetric key, meaning that the same key used to
encrypt a message can be used to decrypt it on the other side. The purpose of
this is to wrap all further communication in an encrypted tunnel that cannot be
deciphered by outsiders.
After the session encryption is established, the user authentication stage
begins.

Authenticating the User’s Access to the Server

The next step involves authenticating the user and deciding on access. There
are a few methods that can be used for authentication, based on what the
server accepts.
The general method is password authentication, which is when the server
prompts the client for the password of the account they are attempting to log in
with. The password is sent through the negotiated encryption, so it is secure
from outside parties.
Even though the password will be encrypted, this method is not generally
recommended due to the limitations on the complexity of the password.
Automated scripts can break passwords of normal lengths very easily compared
to other authentication methods.
The most popular and recommended alternative is the use of SSH key pairs.
SSH key pairs are asymmetric keys, meaning that the two associated keys
serve different functions.
The public key is used to encrypt data that can only be decrypted with the
private key. The public key can be freely shared, because, although it can
encrypt for the private key, there is no method of deriving the private key from
the public key.
Authentication using SSH key pairs begins after the symmetric encryption has
been established as described in the previous section. The procedure happens
as follows:
• The client begins by sending an ID for the key pair it would like to
authenticate with to the server.
• The server checks the authorized_keys file of the account that the client is
attempting to log into for the key ID.
• If a public key with a matching ID is found in the file, the server generates
a random number and uses the public key to encrypt the number.
• The server sends the client this encrypted message.
• If the client actually has the associated private key, it will be able to
decrypt the message using that key, revealing the original number.
• The client combines the decrypted number with the shared session key
that is being used to encrypt the communication, and calculates the MD5
hash of this value. MD5 is a message-digest algorithm that uses the hash
function to generate a 128-bit hash value.
• The client then sends this MD5 hash back to the server as an answer to
the encrypted number message.
• The server uses the same shared session key and the original number that
it sent to the client to calculate the MD5 value on its own. It compares its
own calculation to the one that the client sent back. If these two values
match, it proves that the client was in possession of the private key and
the client is authenticated.
In sum, the asymmetry of the keys allows the server to encrypt messages to
the client using the public key. The client can then prove that it holds the
private key by decrypting the message correctly. The two types of encryption
that are used (symmetric shared secret and asymmetric public/private keys)
are each able to leverage their specific strengths in this model.

Conclusion
Learning about the connection negotiation steps and the layers of encryption at
work in SSH can help you better understand what is happening when you log in
to a remote server.
what is ssh? Why we need to set up it during hadoop installation

What is SSH?
SSH (Secure Shell) is a cryptographic network protocol used for securely
accessing and managing network services over an unsecured network. It
provides a secure channel over an unsecured network by using encryption to
ensure that all communication between the client and server is confidential and
tamper-proof. SSH is commonly used for remote login, command execution, and
file transfers.

Why is SSH Needed During Hadoop Installation?

In a Hadoop cluster, SSH is essential for the following reasons:
1. Cluster Management:

• Hadoop consists of multiple nodes (machines) that need to

communicate with each other. The master node (NameNode and
ResourceManager) needs to start and stop processes on the worker
nodes (DataNodes and NodeManagers).
• SSH allows the master node to securely connect to the worker nodes
and execute commands remotely.
 2. Passwordless Login:

• During Hadoop installation, passwordless SSH is configured between

the master node and the worker nodes. This is necessary because
Hadoop scripts (like start-dfs.sh and start-yarn.sh) need to
automatically log in to the worker nodes without requiring manual
password input.
• Passwordless SSH is achieved by generating an SSH key pair on the
master node and distributing the public key to all worker nodes.
3. Secure Communication:

• SSH ensures that all communication between nodes in the Hadoop

cluster is encrypted, preventing unauthorized access or
eavesdropping.
4. Fault Tolerance and High Availability:

• In case of node failures, Hadoop relies on SSH to restart processes on

other nodes or to perform failover operations.

Steps to Set Up SSH for Hadoop Installation

1. Install SSH:

• Ensure that SSH is installed on all nodes in the cluster.

• On most Linux distributions, you can install SSH using:

sudo apt-get install openssh-server openssh-client

2. Generate SSH Keys:

• On the master node, generate an SSH key pair (if not already
generated):

ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa

• This creates a public key (id_rsa.pub) and a private key (id_rsa).

3. Copy the Public Key to Worker Nodes:

• Use the ssh-copy-id command to copy the public key to all worker
nodes:
 ssh-copy-id user@worker-node

• Replace user with your username and worker-node with the

hostname or IP address of the worker node.
4. Test Passwordless SSH:

• Verify that you can log in to each worker node from the master node
without entering a password:
ssh user@worker-node

5. Repeat for All Nodes:

• Ensure that the master node can SSH into all worker nodes and vice
versa (if required).
6. Configure Hadoop:

• Once SSH is set up, you can proceed with the Hadoop installation
and configuration, as the Hadoop scripts will rely on SSH to manage
the cluster.
By setting up SSH, you ensure that your Hadoop cluster can operate efficiently
and securely, with seamless communication between nodes.

so is it required for internal working of the system

Yes, SSH is required for the internal working of the Hadoop system. It
plays a critical role in enabling communication and coordination between the
nodes in a Hadoop cluster. Here's why SSH is essential for Hadoop's internal
operations:

Why SSH is Required for Hadoop's Internal Working

1. Starting and Stopping Daemons:

• Hadoop consists of several daemons (background processes) such

as:
• NameNode and DataNode (for HDFS).

• ResourceManager and NodeManager (for YARN).

• When you run Hadoop startup scripts like start-dfs.sh or start-

yarn.sh, the master node (NameNode or ResourceManager) needs to
 connect to the worker nodes (DataNodes or NodeManagers) to start
these daemons.
• SSH is used to remotely execute commands on the worker nodes to
start or stop these processes.
2. Cluster Coordination:

• Hadoop relies on distributed coordination between nodes. For

example:
• The NameNode needs to communicate with DataNodes to
manage file storage.
• The ResourceManager needs to communicate with
NodeManagers to manage resources and schedule tasks.
• SSH ensures secure and seamless communication between these
nodes.
3. Fault Tolerance and Recovery:

• If a node fails, Hadoop may need to restart processes on other nodes

or reassign tasks. SSH allows the master node to remotely execute
recovery commands on the worker nodes.

4. Logging and Monitoring:

• Hadoop administrators often use SSH to access worker nodes for

debugging, checking logs, or monitoring system performance.
5. Secure Communication:

• SSH encrypts all communication between nodes, ensuring that

sensitive data (like configuration files, keys, or commands) is not
exposed to unauthorized users.

What Happens Without SSH?

If SSH is not set up or configured correctly:
• The master node will not be able to start or stop processes on the worker
nodes.
• The Hadoop cluster will fail to initialize, and you will encounter errors
when running startup scripts.
• Communication between nodes will be disrupted, and the cluster will not
function as intended.
Is SSH Used Continuously During Hadoop Operation?
Once the Hadoop daemons are up and running, SSH is not actively used for
day-to-day operations. However, it remains essential for:
• Restarting the cluster.
• Adding or removing nodes.
• Performing maintenance tasks.
• Handling failures or recovery scenarios.

Conclusion
SSH is a fundamental requirement for Hadoop's internal working because it
enables secure and efficient communication between the master and worker
nodes. Without SSH, the Hadoop cluster cannot be managed or operated
effectively. Setting up passwordless SSH ensures that Hadoop scripts can
automate tasks without manual intervention, making the cluster more reliable
and easier to manage.

when we run startup scripts like start-dfs.sh, how does the master
connects to the worker node? does it sets up a TCP connection or any
other method?
When you run Hadoop startup scripts like start-dfs.sh or start-yarn.sh, the
master node connects to the worker nodes using SSH. Here's a detailed
explanation of how this process works:

How the Master Node Connects to Worker Nodes

1. SSH Connection:

• The master node uses SSH to establish a secure connection with

each worker node.
 • SSH operates over a TCP connection (typically on port 22) to
communicate with the worker nodes.
• During the connection, the master node authenticates itself using
the SSH key pair (public and private keys) that you configured
earlier.
2. Remote Command Execution:

• Once the SSH connection is established, the master node executes

the necessary commands on the worker nodes to start the Hadoop
daemons (e.g., DataNode or NodeManager).
• For example, the master node might run a command like:
bash
Copy
hadoop-daemon.sh start datanode

on each worker node to start the DataNode process.

3. Passwordless SSH:

• To automate this process, Hadoop relies on passwordless SSH. This

means the master node's public key must be added to the
~/.ssh/authorized_keys file on each worker node.
• This allows the master node to log in to the worker nodes without
requiring manual password input.

Does Hadoop Use TCP Connections for Internal Communication?

Yes, but not during the startup process. Here's the distinction:
1. During Startup:

• The master node uses SSH over TCP to connect to worker nodes
and start the Hadoop daemons.
• SSH is only used for initializing the cluster (starting processes).
2. After Startup:

• Once the Hadoop daemons are running, the nodes communicate

with each other using Hadoop's internal protocols, which also rely
on TCP connections.
• For example:
• The NameNode and DataNodes communicate over TCP for
HDFS operations.
 • The ResourceManager and NodeManagers communicate
over TCP for YARN operations.
• These internal communications use specific ports configured in
Hadoop's configuration files (e.g., core-site.xml, hdfs-site.xml, yarn-
site.xml).

Summary of the Process

1. The master node uses SSH over TCP to connect to worker nodes.

2. It executes commands remotely to start Hadoop daemons (e.g.,

DataNode, NodeManager).
3. Once the daemons are running, the nodes switch to Hadoop's internal
protocols (also over TCP) for ongoing communication.

Why SSH and Not Direct TCP for Startup?

• Security: SSH encrypts the communication, ensuring that sensitive
information (like commands and configurations) is not exposed.
• Authentication: SSH uses key-based authentication to verify the identity
of the master node.
• Automation: Passwordless SSH allows Hadoop scripts to automate the
startup process without manual intervention.
In conclusion, SSH is used during the startup phase to establish a secure
connection and execute commands, while TCP is used for both SSH and
Hadoop's internal communication.
what protocol do master uses for remote login? Does it sends
concerned commands to the worker node by sending a http request or
what the message or communication happens for sending a remote
command to the worker node
When the master node connects to worker nodes for remote login and
command execution during Hadoop startup, it uses the SSH protocol. SSH
(Secure Shell) is specifically designed for secure remote login and command
execution over an unsecured network. Here's how the communication works:

Protocol Used for Remote Login

• The master node uses the SSH protocol to log in to the worker nodes.
 • SSH operates over a TCP connection (typically on port 22) and provides
encryption for secure communication.

How Commands Are Sent to Worker Nodes

1. SSH Connection Establishment:

• The master node initiates an SSH connection to the worker node

using the worker node's IP address or hostname.
• The master node authenticates itself using the SSH key pair (public
and private keys) that were configured during the setup process.
2. Remote Command Execution:

• Once the SSH connection is established, the master node sends the
command to the worker node over the SSH channel.
• For example, if the master node needs to start the DataNode process
on a worker node, it sends the following command:
bash
Copy
hadoop-daemon.sh start datanode

• This command is executed on the worker node's shell.

3. Communication Mechanism:

• The communication between the master node and the worker node
happens over the SSH protocol, not HTTP or any other protocol.
• SSH provides a secure channel for sending commands and receiving
outputs.

What Happens Under the Hood?

1. TCP Connection:

• The master node establishes a TCP connection to the worker node on

port 22 (default SSH port).
2. SSH Handshake:

• The master and worker nodes perform an SSH handshake to

establish a secure connection. This involves:
• Exchanging encryption keys.
 • Authenticating the master node using its SSH key pair.
3. Command Execution:

• The master node sends the command (e.g., hadoop-daemon.sh start

datanode) to the worker node over the encrypted SSH channel.
• The worker node executes the command in its shell and returns the
output (if any) to the master node over the same SSH channel.
4. Termination:

• Once the command is executed, the SSH connection is terminated

(unless configured to remain open).

Why Not HTTP or Other Protocols?

• SSH is Purpose-Built: SSH is specifically designed for secure remote
login and command execution. It provides encryption, authentication, and
integrity checking, which are essential for managing a distributed system
like Hadoop.
• HTTP is Not Suitable: HTTP is designed for web-based communication
and does not provide the same level of security or functionality for remote
command execution.
• Efficiency: SSH is lightweight and efficient for the task of running
commands on remote machines.

Summary
• The master node uses the SSH protocol for remote login and command
execution on worker nodes.
• Commands are sent over an encrypted SSH channel, not via HTTP or
any other protocol.
• The communication involves:
1. Establishing a TCP connection.
2. Performing an SSH handshake.
3. Sending the command over the SSH channel.
4. Executing the command on the worker node and returning the
output (if any).
This process ensures secure and efficient management of the Hadoop cluster
during startup and maintenance.
so ssh is used for remote login but here what do you mean by 'login'
login for what service. Means you are getting access to terminal? or
Gui or whats the basic behind it
When we say the master node uses SSH for remote login to worker nodes in a
Hadoop cluster, it means the master node is accessing the command-line
interface (CLI) or terminal of the worker nodes. Let me break this down for
clarity:

What Does "Login" Mean in This Context?

1. Accessing the Worker Node's Shell:
 • SSH allows the master node to log in to the worker node's
command-line interface (CLI).
• This is similar to opening a terminal on the worker node and typing
commands manually, but it's done remotely and programmatically.
2. No GUI Involved:

• In most Hadoop setups, especially in production environments,

worker nodes are headless servers (no graphical user interface or
GUI).
• SSH provides access to the text-based terminal of the worker
node, where commands can be executed.
3. Purpose of Login:

• The master node logs in to the worker node to:

• Start or stop Hadoop daemons (e.g., DataNode, NodeManager).

• Check the status of processes.

• Perform administrative tasks (e.g., checking logs,
troubleshooting).

How Does SSH Enable This?

1. SSH Provides a Secure Shell:

• SSH stands for Secure Shell. It creates a secure, encrypted channel

between the master node and the worker node.
• Once the connection is established, the master node can execute
commands on the worker node as if it were physically present at the
worker node's terminal.
2. Command Execution:

• For example, when you run start-dfs.sh on the master node:

• The script uses SSH to log in to each worker node.

• It then runs the command hadoop-daemon.sh start datanode
on the worker node's terminal to start the DataNode process.
3. No Human Intervention:

• Since passwordless SSH is configured, the master node can log in

and execute commands without requiring a human to type a
password or interact with the worker node.
What Happens During the Login Process?
1. Authentication:

• The master node authenticates itself to the worker node using its
SSH key pair (public and private keys).
• The worker node verifies the master node's identity by checking the
public key in its ~/.ssh/authorized_keys file.
2. Shell Access:

• Once authenticated, the master node gains access to the worker

node's shell (command-line interface).
• This shell is the same as what you would get if you logged in to the
worker node manually.
3. Command Execution:

• The master node sends commands (e.g., hadoop-daemon.sh start

datanode) to the worker node's shell.
• The worker node executes these commands and returns the output
(if any) to the master node over the SSH connection.

Why Not GUI?

• Efficiency: Hadoop clusters are typically managed through the command
line because it is faster and more efficient for automation.
• Resource Usage: GUIs consume more resources (CPU, memory) and are
unnecessary for server environments.
• Scalability: Managing hundreds or thousands of nodes via a GUI is
impractical, whereas CLI-based tools and scripts can scale easily.

Summary
• "Login" in this context means accessing the command-line interface
(CLI) or terminal of the worker node.
• SSH provides a secure way for the master node to remotely execute
commands on the worker node's terminal.
• This is done programmatically (without human intervention) to start, stop,
or manage Hadoop daemons.
• There is no GUI involved; everything happens through the text-based
terminal.
This approach is efficient, secure, and scalable, making it ideal for managing
distributed systems like Hadoop.

Summary

• SSH is used for remote login to access the command-line interface

(CLI) of worker nodes.
• The master node uses SSH to execute commands on worker nodes (e.g.,
starting/stopping Hadoop daemons).
• This process is secure, efficient, and automated, which is essential for
managing a distributed system like Hadoop.
If you have more questions about Hadoop, SSH, or anything else, feel free to
ask. Happy learning

Linux Crash Course - Connecting to Linux Servers via SSH

https://www.youtube.com/watch?v=kjFz7Lp8Qjk