Big Data Analytics

Unit 1: Introduction to Big Data

1.1 What is Big Data?​

1.2 Why Big Data?​
1.3 Convergence of Key Trends​
1.4 Structured vs. Unstructured Data​
1.5 Industry Examples of Big Data​
1.6 Web Analytics and Marketing Analytics​
1.7 Fraud Detection and Risk Management​
1.8 Big Data in Healthcare and Medicine​
1.9 Algorithmic Trading and Advertising​
1.10 Big Data Technologies Overview​
1.11 Introduction to Hadoop​
1.12 Open-Source Big Data Technologies​
1.13 Cloud Computing and Big Data​
1.14 Mobile Business Intelligence​
1.15 Crowd-Sourcing Analytics​
1.16 Inter and Trans-Firewall Analytics

Unit 2: NoSQL Databases

2.1 Introduction to NoSQL​

2.2 Aggregate Data Models​
2.3 Types of NoSQL Databases

●​ Key-Value Stores​

●​ Document Stores​

●​ Graph Databases​
2.4 Relationships in NoSQL Databases​
2.5 Schema-less Databases​
2.6 Materialized Views​
2.7 Data Distribution Models​
2.8 Sharding and Partitioning​
2.9 Replication Strategies​

●​ Master-Slave Replication​
 ●​ Peer-to-Peer Replication​
2.10 Consistency and Availability​
2.11 Relaxing Consistency Constraints​
2.12 Version Stamps in NoSQL​
2.13 MapReduce Integration with NoSQL​
2.14 Partitioning and Combining Data​
2.15 Composing MapReduce Calculations​

Unit 3: Hadoop and HDFS

3.1 Understanding Data Formats in Big Data​

3.2 Introduction to Hadoop Framework​
3.3 Scaling Out with Hadoop​
3.4 Hadoop Streaming and Pipes​
3.5 HDFS (Hadoop Distributed File System)​
3.6 HDFS Design Principles​
3.7 HDFS Concepts and Architecture​
3.8 Java Interface for HDFS​
3.9 Data Flow in HDFS​
3.10 Hadoop I/O Operations​
3.11 Ensuring Data Integrity​
3.12 Data Compression in Hadoop​
3.13 Serialization in Hadoop​
3.14 Avro – A Data Serialization System​
3.15 File-Based Data Structures in Hadoop

Unit 4: MapReduce – The Processing Engine

4.1 Understanding MapReduce Workflows​

4.2 Writing and Running MapReduce Jobs​
4.3 Unit Testing with MRUnit​
4.4 Test Data and Local Testing​
4.5 Anatomy of a MapReduce Job Run​
4.6 Classic MapReduce vs. YARN​
4.7 Handling Failures in MapReduce and YARN​
4.8 Job Scheduling in Hadoop​
4.9 The Shuffle and Sort Phase​
4.10 Task Execution in MapReduce​
4.11 MapReduce Data Types and Structures​
4.12 Input Formats in MapReduce​
4.13 Output Formats in MapReduce

Unit 5: HBase and Cassandra – NoSQL in Action

5.1 Introduction to HBase​

5.2 HBase Data Model and Architecture​
5.3 HBase Client Operations​
5.4 Practical Examples of HBase Usage​
5.5 Understanding Cassandra​
5.6 Cassandra Data Model and Features​
5.7 Cassandra Use Cases and Examples​
5.8 Cassandra Client Operations​
5.9 Comparing HBase and Cassandra​
5.10 Integration of HBase and Cassandra with Hadoop

Unit 6: Pig and Hive – High-Level Data Processing

6.1 Introduction to Pig​

6.2 Pig Latin – The Scripting Language for Hadoop​
6.3 Understanding Pig’s Data Model​
6.4 Writing and Executing Pig Scripts​
6.5 Grunt Shell – Interactive Pig Execution​
6.6 Developing and Testing Pig Latin Scripts​
6.7 Introduction to Hive​
6.8 Data Types and File Formats in Hive​
6.9 HiveQL – SQL for Big Data​
6.10 Hive Data Definition Language (DDL)​
6.11 Hive Data Manipulation Language (DML)​
6.12 Writing HiveQL Queries​
6.13 Partitioning and Bucketing in Hive​
6.14 Comparing Pig and Hive – When to Use What?
Sure! Let’s dive into Unit 1: Big Data Concepts step by step. I'll break it into smaller sections to
ensure clarity and give relevant examples for easy understanding.

1. What is Big Data?

Big Data refers to a large and complex set of data that traditional data processing tools cannot
handle.

●​ Examples of Big Data:​

○​ Social media platforms like Facebook and Twitter generate huge amounts of user
posts, likes, and interactions daily.​

○​ E-commerce websites like Amazon analyze millions of customer reviews,

transactions, and product details.​

Key Characteristics of Big Data (The 5 V's):

1.​ Volume: The amount of data generated (e.g., petabytes or terabytes).​

2.​ Velocity: The speed at which data is generated (e.g., real-time data from sensors).​

3.​ Variety: The different types of data (structured, unstructured, semi-structured).​

4.​ Veracity: The reliability or accuracy of the data.​

5.​ Value: The insights or benefits extracted from the data.​

2. Why Big Data?

The world generates massive amounts of data every second. Big Data helps organizations:

●​ Improve decision-making (e.g., predicting customer buying patterns).​

●​ Detect fraud (e.g., unusual credit card transactions).​

●​ Enhance customer experiences (e.g., personalized recommendations on Netflix).​

3. Convergence of Key Trends

Big Data evolved because of advancements in:

●​ Technology: Faster processors and cheap storage.​

●​ Connectivity: Billions of devices connected via the internet.​

●​ Data Sources: Social media, IoT devices, cloud computing, etc.​

4. Unstructured Data

Structured data is organized in tables (e.g., Excel sheets).​

Unstructured data lacks a fixed format (e.g., videos, emails, social media posts).

●​ Example:​

○​ Structured: A database table of student records.​

○​ Unstructured: A folder containing text files, images, and videos.​

Big Data tools (like Hadoop) help manage and analyze unstructured data.

5. Industry Examples of Big Data

Let’s see how different industries use Big Data:

1.​ Web Analytics:​

○​ Google uses Big Data to analyze search trends and improve ad targeting.​

2.​ Marketing:​

○​ Amazon uses purchase history to recommend products.​

3.​ Fraud Detection:​

 ○​ Banks analyze unusual transaction patterns to detect fraud.​

4.​ Healthcare:​

○​ Hospitals predict disease outbreaks using patient data and trends.​

5.​ Algorithmic Trading:​

○​ Stock markets use AI to make rapid trading decisions based on data patterns.​

6.​ Advertising:​

○​ Platforms like YouTube use Big Data to show targeted ads.​

6. Big Data Technologies

Technologies that support Big Data include:

1.​ Hadoop: A framework for storing and processing large datasets.​

2.​ Cloud Computing: Tools like AWS and Google Cloud store Big Data.​

3.​ Open Source Tools: Tools like Spark, Hive, and Pig simplify data analysis.​

7. Mobile Business Intelligence (BI)

●​ Mobile BI allows decision-makers to access analytics on their phones.​

●​ Example: A manager checks real-time sales data on their mobile app while traveling.​

8. Crowdsourcing Analytics

Crowdsourcing involves gathering data or insights from a large group of people.

●​ Example: Wikipedia collects knowledge contributions from volunteers worldwide.​

9. Inter and Trans-Firewall Analytics

●​ Inter-firewall analytics: Sharing data securely between two organizations.​

●​ Trans-firewall analytics: Analyzing data that moves across multiple organizations, while
ensuring privacy and security.​

I’ll pause here to check your understanding and address questions. Let me know when you’re
ready to proceed to the next part or need clarifications.

Let’s continue with Unit 1 and explore the remaining concepts in detail.

10. Hadoop: An Introduction

Hadoop is an open-source framework designed to process and store huge datasets. It is one of
the most popular tools for Big Data processing.

Key Components of Hadoop

1.​ HDFS (Hadoop Distributed File System):​

○​ Stores large data files across multiple computers.​

○​ Breaks data into smaller chunks and replicates them for reliability.​

○​ Example: A 1TB file can be split into smaller 64MB or 128MB blocks and stored
on different machines.​

2.​ MapReduce:​

○​ A programming model for processing large datasets in parallel.​

○​ Example: Counting word frequency in a large document is broken into two steps:​

■​ Map Step: Count words in smaller chunks of data.​

■​ Reduce Step: Combine the counts from all chunks.​

 3.​ YARN (Yet Another Resource Negotiator):​

○​ Manages and allocates system resources for running jobs.​

4.​ Hadoop Ecosystem Tools:​

○​ Hive: For querying data using SQL-like language.​

○​ Pig: For scripting and handling unstructured data.​

○​ HBase: A database for real-time read/write access.​

○​ Cassandra: A distributed database system.​

11. Open Source Technologies

●​ Big Data thrives on open-source software due to its cost-effectiveness and flexibility.​

●​ Examples:​

○​ Apache Spark: Faster alternative to Hadoop for data processing.​

○​ Kafka: Real-time data streaming.​

○​ Elasticsearch: Searching and analyzing data in real-time.​

12. Cloud and Big Data

Cloud computing provides scalable and cost-effective infrastructure for Big Data.

●​ Benefits:​

○​ Scalability: Easily add more storage or processing power.​

○​ Cost-effective: Pay only for what you use.​

●​ Example:​
 ○​ Netflix uses Amazon Web Services (AWS) to manage its Big Data for streaming
optimization.​

13. Mobile Business Intelligence (BI)

Mobile BI gives users access to analytics tools on smartphones or tablets.

●​ Example: A sales executive can track daily sales performance via a mobile dashboard.​

●​ Importance: Helps managers and executives make decisions in real-time, even on the
go.​

14. Crowdsourcing Analytics

This is when data or insights are gathered from a large group of people (the crowd).

●​ Example 1: Google Maps collects traffic data from user devices to estimate travel times.​

●​ Example 2: Platforms like Kaggle allow data scientists worldwide to contribute solutions
for real-world problems.​

15. Inter and Trans-Firewall Analytics

●​ Inter-firewall Analytics: Analyzing data between two firewalls (e.g., collaboration

between two organizations sharing secure data).​

○​ Example: A logistics company shares delivery data with its partner to improve
shipment tracking.​

●​ Trans-firewall Analytics: Processing and analyzing data across multiple firewalls

securely.​

○​ Example: Governments analyzing pandemic-related data shared across borders.​

Summary of Key Concepts from Unit 1

1.​ Big Data is about managing and analyzing vast amounts of data to gain valuable
insights.​

2.​ Key Big Data technologies include Hadoop, cloud computing, and tools like Hive and
Pig.​

3.​ Industries use Big Data for marketing, fraud detection, healthcare, and more.​

4.​ Innovations like mobile BI and crowdsourcing analytics enable better decision-making.​

Unit 1 is complete! Let me know if you’d like me to clarify anything or move to Unit 2:
Introduction to NoSQL.
Let’s start with Unit 2: Introduction to NoSQL in a simple and structured manner.

1. What is NoSQL?

NoSQL (Not Only SQL) is a type of database designed to handle large-scale, unstructured, or
semi-structured data efficiently. Unlike traditional relational databases (SQL), NoSQL databases
are flexible, scalable, and capable of processing massive amounts of data.

2. Why NoSQL?

Traditional SQL databases (like MySQL or Oracle) may struggle with:

1.​ High data volumes: Managing petabytes of data.​

2.​ Variety in data: Handling unstructured data like social media posts, images, or videos.​

3.​ Scalability: Expanding databases across multiple servers.​

4.​ Fast access: Delivering real-time results.​

NoSQL databases solve these issues with their distributed and non-relational structure.

3. Types of NoSQL Databases

1.​ Key-Value Databases​

○​ Store data as key-value pairs.​

○​ Example:​

■​ Key: "user123"​

■​ Value: {name: "John", age: 30, city: "New York"}​

○​ Use Case: Caching, session storage (e.g., Redis, DynamoDB).​

2.​ Document Databases​

 ○​ Store data as documents (usually JSON or XML).​

Example:​
​
{
"userID": "123",
"name": "John Doe",
"purchases": ["laptop", "phone"]
}

○​
○​ Use Case: E-commerce (e.g., MongoDB, CouchDB).​

3.​ Column-Family Databases​

○​ Organize data into rows and columns, but columns are grouped into families.​

○​ Example: Cassandra, HBase.​

○​ Use Case: Analytics and high-speed write operations.​

4.​ Graph Databases​

○​ Represent data as nodes (entities) and edges (relationships).​

○​ Example:​

■​ Nodes: "John," "Jane."​

■​ Edge: "John is friends with Jane."​

○​ Use Case: Social networks, recommendation systems (e.g., Neo4j, ArangoDB).​

4. Key Concepts in NoSQL

a) Aggregates

●​ Collections of related data grouped together.​

 ●​ Example: A document in MongoDB that contains customer details and their order
history.​

b) Schema-less Databases

●​ No fixed table structure; you can add fields as needed.​

●​ Example: In MongoDB, you can store a document with {name, age} and another with
{name, city, email} in the same collection.​

c) Materialized Views

●​ Precomputed query results stored for quick access.​

●​ Example: Precomputing monthly sales totals to avoid recalculating them repeatedly.​

5. Distribution Models

a) Sharding

●​ Breaking data into smaller pieces (shards) and storing them across multiple servers.​

●​ Example: Splitting a database of 1 million users into shards of 100,000 users each.​

b) Replication

●​ Copying data across multiple servers to ensure reliability and availability.​

●​ Types:​

1.​ Master-Slave Replication: One server (master) writes data, and others (slaves)
read it.​

2.​ Peer-to-Peer Replication: All servers can read and write data.​

6. Consistency in NoSQL
Consistency refers to how databases handle simultaneous changes.

●​ Strong Consistency: Ensures all nodes have the same data immediately.​

●​ Eventual Consistency: Updates are propagated gradually but guarantee consistency

over time.​

●​ Example: A social media post might take a few seconds to appear on all devices.​

7. Version Stamps

NoSQL databases use version stamps to track changes in data.

●​ Example: A document update might include a version number to resolve conflicts in

distributed systems.​

8. MapReduce in NoSQL

MapReduce is a programming model used to process large datasets in parallel.

●​ Map Phase: Breaks data into smaller chunks.​

●​ Reduce Phase: Aggregates results from the map phase.​

●​ Example: Counting the frequency of each word in a large book.​

○​ Map: Break the book into smaller parts and count words.​

○​ Reduce: Combine all counts to get the final result.​

9. Partitioning and Combining

●​ Partitioning: Dividing data for distributed storage.​

○​ Example: Splitting customer data by region.​

 ●​ Combining: Merging results after processing partitions.​

○​ Example: Summing regional sales to get global sales totals.​

Summary of Key Concepts in Unit 2

1.​ NoSQL databases are designed for large-scale, flexible, and distributed data
management.​

2.​ Types of NoSQL databases include key-value, document, column-family, and graph
databases.​

3.​ Core features include schema-less design, sharding, replication, and eventual
consistency.​

4.​ Tools like MapReduce help process data efficiently.​

Let me know if you'd like me to expand on any topic or move to Unit 3: Hadoop and HDFS.
Let’s begin Unit 3: Hadoop and HDFS step by step for a complete understanding.

1. Introduction to Hadoop

Hadoop is an open-source framework for storing and processing large datasets across multiple
computers.

●​ Why Hadoop?​

○​ It is cost-effective, scalable, and fault-tolerant.​

○​ Traditional systems can’t handle the volume, velocity, and variety of Big Data.​

2. Hadoop Ecosystem Components

Hadoop has several components for data storage, processing, and analysis. The core
components are:

1.​ HDFS (Hadoop Distributed File System): For storing large datasets.​

2.​ MapReduce: For processing data in parallel.​

3.​ YARN (Yet Another Resource Negotiator): For resource management.​

3. HDFS Design and Concepts

HDFS is the storage layer of Hadoop. It is designed to handle large files and distribute them
across a cluster of machines.

Key Features of HDFS

1.​ Distributed Storage:​

○​ Files are split into blocks (default size: 128MB or 64MB) and distributed across
multiple machines.​
 ○​ Example: A 1GB file is split into 8 blocks of 128MB each and stored on different
nodes.​

2.​ Fault Tolerance:​

○​ Each block is replicated (default: 3 copies) on different nodes to ensure data

availability even if one node fails.​

3.​ Scalability:​

○​ HDFS can easily scale by adding more nodes to the cluster.​

4.​ Write Once, Read Many Times:​

○​ Data can only be written once but read multiple times to ensure consistency.​

4. Data Flow in HDFS

1.​ Write Operation:​

○​ When a client uploads a file, HDFS splits it into blocks and stores them across
multiple nodes.​

2.​ Read Operation:​

○​ When a client requests a file, HDFS retrieves the blocks from respective nodes
and combines them.​

Example:

If you upload a video file, HDFS will split it into chunks and store them on Node A, Node B, and
Node C. When you access the file, it retrieves all chunks and reassembles them.

5. Hadoop I/O (Input/Output)

Hadoop has its own mechanisms for reading and writing data efficiently.

●​ Compression: Reduces file size for faster processing.​

 ●​ Serialization: Converts data into a format that can be stored or transmitted.​

●​ Avro: A serialization system used in Hadoop for structured data exchange.​

6. Data Format in Hadoop

Hadoop supports various file formats, including:

1.​ Text Format: Simple but inefficient for large datasets.​

2.​ Sequence Files: Binary format with key-value pairs, used for intermediate MapReduce
outputs.​

3.​ Avro Files: For data serialization.​

4.​ Parquet/ORC: Optimized formats for analytics.​

Example:

In a retail store dataset, Parquet format allows querying only the required columns (e.g., "sales"
instead of the entire table), saving time.

7. Scaling Out

Scaling in Hadoop means adding more nodes to the cluster for better performance.

●​ Horizontal Scaling: Adding more machines (preferred in Hadoop).​

●​ Vertical Scaling: Adding more resources (RAM, CPU) to a single machine.​

8. Hadoop Streaming

Hadoop Streaming allows writing MapReduce programs in languages like Python or Ruby
(instead of Java).

●​ Example: Use a Python script to process log files and find the most visited URLs.​
9. Hadoop Pipes

Hadoop Pipes allows writing MapReduce programs in C++. It is useful for developers familiar
with C++ rather than Java.

10. Data Integrity in Hadoop

Hadoop ensures data integrity by:

1.​ Checksum Verification: Ensures blocks are not corrupted during storage or transfer.​

2.​ Replication: Provides multiple copies of data to avoid data loss.​

Summary of Unit 3

1.​ Hadoop is a framework for storing and processing massive datasets across clusters.​

2.​ HDFS is its storage layer, designed for scalability and fault tolerance.​

3.​ Key Concepts: Distributed storage, block replication, write-once-read-many.​

4.​ Hadoop I/O: Efficient file formats (e.g., Parquet) and serialization (e.g., Avro).​

5.​ Scalability: Hadoop supports horizontal scaling for better performance.​

Let me know if you’d like a deeper explanation of any concept or if you’re ready to move to Unit
4: MapReduce Workflows.

Let’s start with Unit 4: MapReduce Workflows step by step.

1. What is MapReduce?

MapReduce is a programming model used in Hadoop to process and analyze large datasets in
parallel.
 ●​ Why MapReduce?: Traditional methods struggle with large datasets; MapReduce splits
the work into smaller tasks that can run simultaneously.​

2. Key Steps in MapReduce

a) Map Phase

●​ Splits the input data into smaller chunks.​

●​ Processes each chunk to produce intermediate key-value pairs.​

b) Shuffle and Sort Phase

●​ Groups all intermediate key-value pairs with the same key together.​

c) Reduce Phase

●​ Processes the grouped data to produce the final output.​

Example: Word Count

●​ Input: A text file: apple banana apple.​

●​ Map Phase: Break into key-value pairs:​

apple: 1, banana: 1, apple: 1.​

●​ Shuffle and Sort Phase: Group by key:​

apple: [1, 1], banana: [1].​

●​ Reduce Phase: Count occurrences:​

apple: 2, banana: 1.​

3. Anatomy of a MapReduce Job

1.​ Input Splits: Data is divided into chunks, one per mapper.​
 2.​ Mapper: Processes each chunk and outputs key-value pairs.​

3.​ Partitioner: Determines which reducer will process each key.​

4.​ Shuffle and Sort: Groups keys and sends them to reducers.​

5.​ Reducer: Aggregates the values for each key and outputs the result.​

4. YARN (Yet Another Resource Negotiator)

YARN is Hadoop’s resource management layer.

●​ Role: Allocates system resources (CPU, memory) to MapReduce jobs.​

●​ Components:​

1.​ Resource Manager: Oversees resources across the cluster.​

2.​ Node Manager: Monitors resources on individual machines.​

Difference Between Classic MapReduce and YARN

Feature Classic MapReduce YARN

Resource Managed by Managed by

Management JobTracker ResourceManager

Scalability Limited Highly scalable

5. Job Scheduling in MapReduce

Hadoop ensures efficient task management through job scheduling.

1.​ FIFO Scheduler: Executes jobs in the order they arrive.​

2.​ Fair Scheduler: Shares resources equally among all jobs.​

3.​ Capacity Scheduler: Allocates resources based on predefined queues.​

6. Shuffle and Sort Phase

●​ This is the intermediate phase between Map and Reduce.​

●​ Shuffle: Transfers map outputs to reducers.​

●​ Sort: Sorts keys to group them for the reducer.​

●​ Example:​
If map outputs are:​
(apple, 1), (banana, 1), (apple, 1),​
shuffle and sort produce:​
(apple, [1, 1]), (banana, [1]).​

7. Failures in MapReduce

Failures in Hadoop are common due to distributed nature, but it is designed to handle them.

1.​ Mapper Failure: The task is retried on another node.​

2.​ Reducer Failure: Only the failed task is retried.​

3.​ Node Failure: HDFS replicates data to prevent loss.​

8. MapReduce Workflows

●​ MapReduce workflows involve chaining multiple MapReduce jobs.​

●​ Example:​

○​ Job 1: Calculate total sales per region.​

○​ Job 2: Use the output of Job 1 to calculate the region with the highest sales.​

9. Unit Testing with MRUnit

MRUnit is a testing framework for MapReduce programs.

●​ Purpose: Test mapper, reducer, and workflows independently.​

●​ Example: Test if the mapper correctly outputs key-value pairs for input data.​

10. Input and Output Formats

Hadoop supports various input and output formats to process different data types.

Input Formats

1.​ TextInputFormat: Default format; processes plain text files.​

2.​ KeyValueTextInputFormat: Splits lines into key-value pairs.​

3.​ SequenceFileInputFormat: Handles binary key-value pairs.​

Output Formats

1.​ TextOutputFormat: Writes plain text output (default).​

2.​ SequenceFileOutputFormat: Writes binary output.​

Example:

For a CSV file:

●​ Input Format: Split rows into individual records.​

●​ Output Format: Combine results into a summarized CSV.​

Summary of Unit 4

1.​ MapReduce processes data in three steps: Map, Shuffle & Sort, and Reduce.​

2.​ YARN manages cluster resources for efficient execution.​

 3.​ Key phases include job scheduling, shuffle & sort, and handling failures.​

4.​ Input/output formats ensure compatibility with various data types.​

5.​ Testing frameworks like MRUnit validate MapReduce workflows.​

Let me know if you’d like more details on any topic or if we should move to Unit 5: HBase and
Cassandra.
Let’s start with Unit 5: HBase and Cassandra step by step.

1. Introduction to HBase and Cassandra

Both HBase and Cassandra are NoSQL databases designed for storing and managing
massive amounts of structured and semi-structured data.

●​ HBase: Built on top of Hadoop’s HDFS, it is optimized for random read/write operations
on large datasets.​

●​ Cassandra: A distributed database designed for high availability and scalability without
a single point of failure.​

2. HBase: Data Model and Implementations

HBase Data Model

HBase stores data in a table-like format with rows and columns, but it is more flexible than
traditional relational databases.

1.​ Tables: Organized into rows and columns.​

2.​ Column Families: Groups of related columns stored together.​

3.​ Rows: Identified by a unique row key.​

4.​ Cells: Contain values and are versioned by timestamps.​

Example:
Row Key Column Family: Info Column Family: Stats

Row1 Name: "John" Score: 85

Row2 Name: "Alice" Score: 90

HBase Architecture
 1.​ HMaster: Manages the cluster and assigns regions to RegionServers.​

2.​ RegionServer: Handles read/write requests for specific regions of a table.​

3.​ Zookeeper: Ensures coordination and fault tolerance.​

Features of HBase

●​ Scales horizontally by adding nodes.​

●​ Supports random read/write operations.​

●​ Integrates seamlessly with Hadoop for batch processing.​

3. HBase Clients and Examples

●​ HBase clients connect to the database using APIs to perform CRUD operations.​

Example:

●​ Insert data: Add a new row with row key "123" and name "John".​

●​ Read data: Fetch all rows where the score is greater than 80.​

4. Cassandra: Data Model and Implementations

Cassandra Data Model

Cassandra organizes data into keyspaces, which contain tables.

1.​ Keyspace: A namespace similar to a database in SQL.​

2.​ Tables: Store data in rows and columns.​

3.​ Partitions: Data is distributed across the cluster using a partition key.​

4.​ Clustering Columns: Define the order of data within a partition.​

Example:
Partition Key Clustering Column Name Scor
e

101 Math John 85

101 Science John 90

Cassandra Architecture

1.​ Nodes: Individual servers in the cluster.​

2.​ Replication: Data is replicated across nodes for fault tolerance.​

○​ Replication Factor: Number of copies of the data.​

3.​ Consistency Levels: Controls how many nodes must respond for a read/write
operation.​

Features of Cassandra

●​ Decentralized with no single point of failure.​

●​ High write and read performance.​

●​ Tunable consistency (eventual or strong).​

5. HBase vs Cassandra
Feature HBase Cassandra

Storage Layer Built on HDFS Standalone, custom storage engine

Read/Write Optimized for consistent reads Optimized for high write throughput

Architecture Master-slave Peer-to-peer

Consistency Strong consistency Tunable consistency

Use Case Random read/write, batch High availability, low latency

processing
6. Hadoop Integration

Both HBase and Cassandra can integrate with Hadoop for data processing.

●​ HBase: Processes data stored in HDFS directly.​

●​ Cassandra: Uses connectors to process data with Hadoop or Spark.​

Example:

●​ Use HBase for storing patient records in a hospital system.​

●​ Use Cassandra for a messaging system requiring real-time updates.​

Summary of Unit 5

1.​ HBase is a NoSQL database designed for random read/write on massive datasets and
integrates with Hadoop for batch processing.​

2.​ Cassandra is a distributed NoSQL database optimized for high availability and fault
tolerance.​

3.​ Comparison: HBase uses HDFS as its storage layer, while Cassandra is standalone
and peer-to-peer.​

4.​ Use Cases:​

○​ HBase: Analytical workloads with Hadoop.​

○​ Cassandra: Real-time applications like messaging or IoT.​

Let me know if you’d like to dive deeper into any part or move on to Unit 6: Pig and Hive.
Let’s start with Unit 6: Pig and Hive step by step.

1. Introduction to Pig and Hive

Both Pig and Hive are tools built on top of Hadoop for processing and analyzing large datasets.

●​ Pig: Provides a scripting language called Pig Latin for data transformations.​

●​ Hive: Offers an SQL-like interface called HiveQL for querying and managing data in
Hadoop.​

2. Pig

What is Pig?

●​ A platform for analyzing large datasets using its scripting language, Pig Latin.​

●​ Designed for ETL (Extract, Transform, Load) operations.​

Key Components

1.​ Pig Latin: High-level scripting language.​

2.​ Pig Engine: Converts Pig Latin scripts into MapReduce jobs.​

Pig Data Model

Data in Pig is represented as nested structures:

1.​ Atom: A single value (e.g., a number or string).​

2.​ Tuple: A collection of fields (e.g., (101, "Alice")).​

3.​ Bag: A collection of tuples (e.g., {(101, "Alice"), (102, "Bob")}).​

4.​ Map: Key-value pairs (e.g., [“name”#”Alice”]).​

Pig Latin Commands
Load Data:​
​
data = LOAD 'data.txt' USING PigStorage(',') AS (id: int, name: chararray);

1.​

Transform Data:​
​
filtered = FILTER data BY id > 100;

2.​

Group Data:​
​
grouped = GROUP data BY name;

3.​

Store Results:​
​
STORE grouped INTO 'output' USING PigStorage(',');

4.​

Features of Pig

●​ Handles semi-structured and unstructured data.​

●​ Automatically translates scripts into MapReduce jobs.​

●​ Supports user-defined functions (UDFs).​

Example Use Case

Analyze log files to find the number of errors in a system.

3. Hive

What is Hive?
 ●​ A data warehouse system for querying and managing large datasets stored in Hadoop.​

●​ Designed for users familiar with SQL.​

Key Components

1.​ HiveQL: SQL-like query language.​

2.​ Metastore: Stores schema information about tables.​

3.​ Execution Engine: Converts HiveQL queries into MapReduce jobs.​

Hive Data Types

1.​ Primitive Types: INT, STRING, FLOAT, etc.​

2.​ Complex Types: ARRAY, MAP, STRUCT.​

HiveQL Basics
Create Table:​
​
CREATE TABLE students (id INT, name STRING, score FLOAT)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ',';

1.​

Load Data:​
​
LOAD DATA INPATH '/path/to/data.csv' INTO TABLE students;

2.​

Query Data:​
​
SELECT name, AVG(score) FROM students GROUP BY name;

3.​
Insert Data:​
​
INSERT INTO TABLE students VALUES (101, 'Alice', 90.0);

4.​

Features of Hive

●​ SQL-like interface for querying data.​

●​ Integrates with BI tools for analytics.​

●​ Handles structured data effectively.​

Example Use Case

Generate reports on sales data stored in HDFS.

4. Pig vs Hive
Feature Pig Hive

Language Pig Latin HiveQL (SQL-like)

Use Case ETL operations Data warehousing

Data Type Semi-structured/unstructure Structured

d

User Developers Analysts

Execution Converts to MapReduce Converts to MapReduce

jobs jobs

5. Integration of Pig and Hive with Hadoop

●​ Pig: Automatically translates scripts into MapReduce jobs that run on Hadoop.​

●​ Hive: Integrates with Hadoop for batch processing and supports analytics over large
datasets.​
Summary of Unit 6

1.​ Pig uses Pig Latin scripts for ETL and works well with semi-structured data.​

2.​ Hive uses HiveQL for SQL-like queries and is ideal for structured data analysis.​

3.​ Comparison: Pig is more suitable for developers, while Hive is aimed at analysts.​

4.​ Applications:​

○​ Pig: Log analysis, data cleansing.​

○​ Hive: Reporting, data warehousing.​

Let me know if you’d like to dive deeper into any part or review the entire syllabus together!
Compressed Short Notes for Big Data Analytics (OEC-CS801A)

Unit 1: Big Data Concepts

●​ Big Data: Large, complex datasets, challenging traditional processing tools.​

●​ Key Trends: Increased data from unstructured sources (social media, IoT).​

●​ Industry Examples: Web analytics, fraud detection, credit risk, algorithmic trading,
healthcare, advertising.​

●​ Big Data Technologies:​

○​ Hadoop: Framework for distributed data storage/processing.​

○​ Cloud: Scalable storage/computing.​

○​ Mobile BI: Access insights on mobile devices.​

○​ Crowdsourcing Analytics: Leverages user-contributed data.​

○​ Trans/Inter Firewall Analytics: Analyzing data across secured environments.​

Unit 2: NoSQL

●​ NoSQL: Non-relational databases for big data.​

○​ Types: Key-value, Document, Graph databases.​

●​ Aggregate Models: Grouping data for quick access.​

●​ Schemaless Databases: No fixed schema; dynamic data.​

●​ Replication:​

○​ Master-Slave: Single primary node.​

○​ Peer-Peer: Equal nodes share the load.​

 ●​ Consistency Models:​

○​ Eventual Consistency: Data synchronizes over time.​

○​ Strong Consistency: Immediate synchronization.​

●​ MapReduce: Programming model for distributed data processing (Partitioning,

Combining).​

Unit 3: Hadoop and HDFS

●​ Hadoop: Framework for distributed storage/processing.​

○​ HDFS: Hadoop Distributed File System, stores big data across multiple nodes.​

●​ Data Flow: Reads/writes occur through NameNode and DataNodes.​

●​ Key Concepts:​

○​ Compression: Saves storage.​

○​ Serialization: Converts data for storage (Avro).​

○​ Hadoop I/O: Ensures data integrity and efficiency.​

○​ File-based Structures: Stores files in chunks for distribution.​

Unit 4: MapReduce

●​ MapReduce: Framework for parallel processing.​

○​ Map Phase: Filters and sorts data.​

○​ Reduce Phase: Aggregates data.​

●​ Workflow: Input → Split → Map → Shuffle/Sort → Reduce → Output.​

●​ YARN: Resource manager for MapReduce.​

 ●​ Job Scheduling: Ensures task execution order.​

●​ Failure Handling: Retries tasks if nodes fail.​

●​ Input/Output Formats: Handle structured, semi-structured, or custom data.​

Unit 5: HBase and Cassandra

●​ HBase: Column-oriented NoSQL database on Hadoop.​

○​ Data Model: Tables, column families, row keys, timestamps.​

○​ Architecture: HMaster, RegionServer, Zookeeper.​

○​ Use Case: Real-time read/write for analytics.​

●​ Cassandra: Distributed NoSQL database for scalability.​

○​ Data Model: Keyspaces, tables, partitions, clustering columns.​

○​ Architecture: Nodes, replication (peer-to-peer), tunable consistency.​

○​ Use Case: Messaging systems, IoT.​

●​ Comparison:​

○​ HBase: Strong consistency, Hadoop integration.​

○​ Cassandra: High availability, fault-tolerant.​

Unit 6: Pig and Hive

●​ Pig: High-level scripting language (Pig Latin) for ETL.​

○​ Data Model: Atom, Tuple, Bag, Map.​

○​ Commands: LOAD, FILTER, GROUP, STORE.​

 ○​ Use Case: Log analysis, data cleansing.​

●​ Hive: SQL-like interface (HiveQL) for querying structured data.​

○​ Key Components: Metastore, execution engine.​

○​ Commands: CREATE TABLE, LOAD DATA, SELECT, INSERT.​

○​ Use Case: Reporting, data warehousing.​

●​ Comparison:​

○​ Pig: Developer-focused, supports unstructured data.​

○​ Hive: Analyst-focused, for structured data.​

Key Takeaways

1.​ Big Data systems handle Volume, Velocity, Variety, Veracity, Value.​

2.​ Hadoop and NoSQL databases like HBase and Cassandra are core technologies.​

3.​ MapReduce simplifies distributed processing.​

4.​ Tools like Pig and Hive ease analytics on big data.​

Let me know if you'd like further clarification or deeper explanations!

Unit 1: Big Data Concepts

Let’s dive into Big Data like it’s a thrilling adventure. Imagine you're an explorer in a vast,
ever-growing digital jungle. Each step takes you through fascinating technologies, real-world
applications, and innovative ideas. Ready? Let’s go!

Chapter 1: What is Big Data?

Imagine a bustling city where every second, people are creating an endless stream of
data—social media posts, online transactions, GPS locations, medical records, and more. This
massive amount of data that’s too large and complex for traditional tools to handle is what we
call Big Data.

●​ Why Big Data?​

Think of solving a giant puzzle. Big Data allows us to connect the pieces of information
to understand patterns, predict outcomes, and make smarter decisions.​

Chapter 2: Key Trends – The Spark That Ignited Big Data

Why is Big Data such a big deal now?

●​ Devices like smartphones, IoT sensors, and cameras create mountains of data every
day.​

●​ Businesses need to make sense of this to serve customers better, improve operations,
and stay competitive.​

For example, imagine Netflix recommending your favorite movies. It uses Big Data to study what
you like and gives you spot-on suggestions.

Chapter 3: Types of Data

Big Data isn’t always neat and tidy. It comes in three types:

1.​ Structured Data: Think spreadsheets, where every piece of data has a specific place
(e.g., sales records).​
 2.​ Unstructured Data: Raw and messy, like tweets, videos, or photos.​

3.​ Semi-Structured Data: A mix of both, like an email with metadata (sender, time) and a
message body.​

Chapter 4: Big Data in Action – Real-Life Adventures

Big Data is the superhero behind many industries:

●​ Web Analytics: Imagine tracking millions of website visitors to improve their experience.​

●​ Marketing: Ever noticed how ads seem tailored to you? That’s Big Data studying your
behavior.​

●​ Fraud Detection: Banks analyze transaction patterns to catch suspicious activities.​

●​ Healthcare: Doctors predict diseases and recommend treatments by analyzing patient

histories.​

●​ Advertising: Brands decide where and how to advertise by studying what grabs
people’s attention.​

Chapter 5: Tools of the Trade – The Big Data Arsenal

Big Data explorers need tools to process and store all this information. Here are the main ones:

●​ Hadoop: Like a giant warehouse where data is stored and processed.​

●​ Cloud: Your storage space in the sky—accessible, scalable, and reliable.​

●​ Mobile BI (Business Intelligence): Insights on your phone, anytime, anywhere.​

Chapter 6: Big Data Technologies – The Magic Behind the Scenes

Let’s peek behind the curtain and see how Big Data works:
 1.​ Hadoop: Divides data into smaller chunks and processes it on multiple computers. Think
of cutting a large cake into slices to share with friends.​

2.​ Crowdsourcing Analytics: People contribute data, like in Waze (a traffic app), where
users report road conditions.​

3.​ Trans/Inter Firewall Analytics: Analyzing data across secure networks while keeping it
safe.​

Chapter 7: Big Data in the Future – Endless Possibilities

Picture a world where Big Data makes life better:

●​ Cities become smarter, managing traffic and energy efficiently.​

●​ Healthcare becomes predictive, stopping illnesses before they start.​

●​ Businesses anticipate your needs, making every interaction seamless.​

Story Summary

Big Data is like a giant treasure hunt in the digital world. Every industry is using it to uncover
patterns, solve problems, and innovate. Tools like Hadoop, the cloud, and mobile BI make the
journey possible, while applications like web analytics, healthcare, and marketing show its
power.

Now, you’re not just learning Big Data—you’re experiencing its magic! Let me know if you want
to dive deeper into any part!
Unit 2: NoSQL – The Story of Non-Relational Databases

Imagine you're in a bustling marketplace where vendors sell all sorts of items—some neatly
stacked, some scattered, and some grouped in creative ways. Now think of this marketplace as
your database. Traditional databases (like relational ones) only work with items that are
perfectly organized (structured). But what about the messy and unique data? That’s where
NoSQL steps in—a revolutionary system that says, “Bring it on!”

Chapter 1: The Rise of NoSQL

Once upon a time, relational databases ruled the world. But as data exploded, businesses faced
new challenges:

1.​ Unstructured and Semi-Structured Data: Social media posts, images, and logs didn’t
fit neatly into rows and columns.​

2.​ Scalability Issues: Traditional databases struggled to handle massive growth.​

Thus, NoSQL was born—a flexible, scalable, and fast way to handle Big Data.

Chapter 2: What is NoSQL?

NoSQL stands for “Not Only SQL.” It’s a new type of database that doesn’t follow the rigid
structure of traditional databases. Think of it as a playground where creativity and flexibility are
key.

Key features:

1.​ Schemaless Databases: No predefined structure—data can evolve over time.​

2.​ High Scalability: Easily handles growing data by adding more servers.​

3.​ Flexible Data Models: Works with key-value pairs, documents, graphs, and more.​

Chapter 3: Types of NoSQL Databases

1.​ Key-Value Stores: Like a dictionary—data is stored as key-value pairs.​

 ○​ Example: Redis, DynamoDB.​

○​ Use Case: Caching, real-time applications.​

2.​ Document Databases: Stores data as documents (JSON, BSON).​

○​ Example: MongoDB, CouchDB.​

○​ Use Case: Content management, catalog systems.​

3.​ Graph Databases: Represents data as nodes and relationships.​

○​ Example: Neo4j.​

○​ Use Case: Social networks, recommendation engines.​

4.​ Column-Family Databases: Stores data in columns instead of rows.​

○​ Example: Cassandra, HBase.​

○​ Use Case: Large-scale analytics.​

Chapter 4: Behind the Scenes – NoSQL Concepts

●​ Aggregate Models: Groups related data into collections or aggregates for easy access.​

○​ Example: A product and its reviews stored together.​

●​ Sharding: Divides data across multiple servers for scalability.​

○​ Think of splitting a big cake among friends.​

●​ Replication: Makes copies of data for fault tolerance.​

○​ Types:​

■​ Master-Slave: One master controls updates; others follow.​

■​ Peer-to-Peer: All servers are equal.​

Chapter 5: Relaxing Rules – The CAP Theorem

In the world of databases, you can’t have it all! The CAP Theorem says you can only choose
two out of three:

1.​ Consistency: All users see the same data at the same time.​

2.​ Availability: The system is always online.​

3.​ Partition Tolerance: The system works even if some parts fail.​

NoSQL often relaxes consistency for better scalability and availability.

Chapter 6: The Magic of MapReduce

MapReduce is like assembling a jigsaw puzzle:

1.​ Map Phase: Breaks data into smaller pieces and processes them.​

2.​ Reduce Phase: Combines results into a final answer.​

●​ Example: Counting words in a book.​

○​ Map: Divide the book into chapters, count words in each.​

○​ Reduce: Add up the counts to get the total.​

Chapter 7: Real-World Examples of NoSQL

1.​ Facebook: Uses Cassandra for messaging.​

2.​ Netflix: Uses Redis for user sessions.​

3.​ Amazon: Uses DynamoDB for product catalogs.​

4.​ Twitter: Uses Graph Databases for social connections.​

Story Summary

NoSQL databases broke the rules of traditional systems to handle the dynamic, unstructured
world of Big Data. With types like key-value, document, and graph databases, NoSQL offers
flexibility, scalability, and performance. Concepts like sharding, replication, and MapReduce
make it a powerful tool for modern applications.

Let’s continue this journey if you’re ready for the next chapter!
Unit 3: Hadoop and HDFS – The Story of a Distributed Data Superhero

Imagine you’re building a massive library to store millions of books. A single shelf (traditional
system) isn’t enough to hold everything, and a single librarian can’t manage it all. You need a
solution that divides the workload and stores books across multiple shelves. That’s exactly what
Hadoop and HDFS (Hadoop Distributed File System) do for data—a superhero duo for Big
Data storage and processing!

Chapter 1: Hadoop – The Big Data Framework

Hadoop is like a manager of your massive data library. It divides huge data tasks into smaller
pieces, assigns them to different servers, and collects the results.

●​ Why Hadoop?​

○​ Handles unstructured data.​

○​ Scales easily by adding more servers.​

○​ Fault-tolerant, meaning it continues to work even if some servers fail.​

Chapter 2: HDFS – Storing Data Across Servers

Think of HDFS as a magical bookshelf system:

●​ Distributed Storage: Data is stored across multiple servers (called nodes).​

●​ Replication: Every piece of data has copies stored on different nodes for safety.​

○​ Example: If one node (server) fails, its data is still available from another node.​

●​ Scalability: Need more space? Just add more nodes!​

Chapter 3: Anatomy of HDFS

1.​ Blocks: Data is broken into smaller chunks called blocks (default size: 128MB).​
 ○​ Example: A 1GB file is split into 8 blocks.​

2.​ Nodes:​

○​ NameNode: The master node that keeps track of where data is stored.​

○​ DataNodes: Worker nodes that store the actual data.​

Chapter 4: How Hadoop Works

1.​ Data Input: You upload your data to Hadoop.​

2.​ Storage in HDFS:​

○​ Data is split into blocks and distributed across DataNodes.​

○​ NameNode keeps a record of where each block is stored.​

3.​ Processing with MapReduce:​

○​ Hadoop breaks tasks into smaller pieces (Map).​

○​ Combines the results (Reduce).​

Chapter 5: Hadoop Streaming and Pipes

Sometimes, you need to interact with Hadoop using different programming languages.

●​ Hadoop Streaming: Allows you to write MapReduce programs in any language (e.g.,
Python).​

●​ Hadoop Pipes: Enables the use of C++ for MapReduce programs.​

Chapter 6: Data Formats and I/O in Hadoop

 ●​ Data Formats: Hadoop supports multiple data formats like CSV, JSON, Avro, and
Parquet.​

●​ Hadoop I/O: Handles input and output operations efficiently.​

○​ Compression: Reduces storage size.​

○​ Serialization: Converts data into a format that can be stored and processed.​

Chapter 7: Ensuring Data Integrity

Hadoop uses checksums to verify that data hasn’t been corrupted during storage or transfer. If
an error is detected, it retrieves a copy from a replicated block.

Chapter 8: Real-World Applications of Hadoop

1.​ Retail: Walmart uses Hadoop to analyze customer behavior and improve sales.​

2.​ Healthcare: Processes patient records for better diagnoses.​

3.​ Finance: Detects fraud in credit card transactions.​

4.​ Social Media: Facebook processes petabytes of data daily for features like friend
suggestions.​

Story Summary

Hadoop and HDFS revolutionize data storage and processing. They break data into blocks,
distribute it across nodes, and ensure fault tolerance with replication. Tools like Hadoop
Streaming, Hadoop Pipes, and support for various data formats make it versatile. This system is
the backbone of Big Data solutions in industries like retail, healthcare, and finance.

Excited to explore more? Let’s continue to the next unit when you’re ready!
Unit 4: MapReduce – The Engine of Big Data Processing

Imagine you’re organizing a massive carnival with thousands of stalls. To keep everything
running smoothly, you divide the work among teams—one group collects data, another
processes it, and another combines everything into a final result. This is how MapReduce, the
heart of Hadoop, works: splitting big tasks into small ones, processing them in parallel, and
merging the results. Let’s dive into this story!

Chapter 1: What is MapReduce?

MapReduce is a programming model used to process vast amounts of data efficiently.

●​ Map Phase: Breaks the data into smaller parts and processes them independently.​

●​ Reduce Phase: Combines the outputs of the Map phase to generate a final result.​

Analogy: Imagine counting the number of apples in a massive orchard:

●​ Map: Each worker counts apples in their assigned section.​

●​ Reduce: All counts are combined to get the total number of apples.​

Chapter 2: MapReduce Workflow – Step by Step

1.​ Input Split: The data is divided into chunks.​

2.​ Mapping: Each chunk is processed to produce key-value pairs.​

3.​ Shuffling and Sorting: The key-value pairs are grouped by key and sorted.​

4.​ Reducing: The grouped data is processed to produce the final output.​

Example: Word Count in a Document

●​ Input: A text document.​

●​ Map: Split the document into words and count each occurrence.​
 ●​ Shuffle/Sort: Group counts by word.​

●​ Reduce: Add up the counts for each word to get the final total.​

Chapter 3: Anatomy of a MapReduce Job

1.​ Job Tracker: The master that oversees task execution.​

2.​ Task Tracker: Workers that process chunks of data.​

3.​ Data Nodes: Store the data being processed.​

4.​ YARN (Yet Another Resource Negotiator): Manages resources across a cluster.​

YARN’s Role: If MapReduce is the engine, YARN is the driver ensuring resources are allocated
efficiently.

Chapter 4: Handling Failures in MapReduce

Failures happen, but MapReduce is prepared:

●​ If a node fails, the task is reassigned to another node.​

●​ Intermediate results are saved, so the process doesn’t have to start over.​

Chapter 5: MapReduce Types

●​ Simple MapReduce: One Map and one Reduce phase.​

●​ Multi-stage MapReduce: Multiple Map and Reduce phases for complex tasks.​

Chapter 6: Input and Output Formats

MapReduce can handle various types of data:

 1.​ Input Formats: Text files, JSON, CSV, or even custom formats.​

2.​ Output Formats: Text, key-value pairs, or structured data.​

Chapter 7: Testing MapReduce Jobs

Before running MapReduce on large clusters, developers test their programs:

●​ MRUnit: A testing framework for MapReduce.​

●​ Local Tests: Simulates small datasets to debug errors.​

Chapter 8: Real-World Uses of MapReduce

1.​ Search Engines: Google uses MapReduce to index web pages.​

2.​ Social Media: Facebook processes billions of posts daily for features like trending
topics.​

3.​ Finance: Detects anomalies in financial transactions.​

4.​ Weather Prediction: Processes massive datasets from satellites.​

Story Summary

MapReduce is like a highly efficient factory line, breaking large tasks into manageable pieces,
processing them in parallel, and assembling the results. Its workflow—Map, Shuffle/Sort,
Reduce—makes it the backbone of Big Data processing. With features like fault tolerance,
support for multiple data formats, and real-world applications in search, social media, and
finance, it’s an indispensable tool in the Big Data world.

Let me know when you're ready to continue to the next unit!

Unit 5: HBase and Cassandra – Big Data’s Dynamic Duo

Welcome to the chapter where we explore HBase and Cassandra, the superheroes of
managing large-scale, distributed databases. If Hadoop is like the library of data, HBase and
Cassandra are like the librarians who specialize in handling specific kinds of books efficiently.
Let’s dive into their story!

Chapter 1: HBase – The NoSQL Table Master

Imagine you have a gigantic spreadsheet where you need to store billions of rows and columns
of data. HBase, built on top of Hadoop, is designed to manage such a spreadsheet.

●​ What is HBase?​

○​ A column-oriented NoSQL database.​

○​ Built for real-time reading and writing of data.​

○​ Operates on top of HDFS for distributed storage.​

Key Features:

1.​ Column-Oriented: Data is stored in columns instead of rows.​

○​ Example: Instead of storing all details of a customer in one row, HBase splits
attributes (name, age, purchases) into columns for faster access.​

2.​ Scalability: Easily handles data that grows over time.​

3.​ Random Access: Quickly fetches specific pieces of data.​

HBase Architecture:

1.​ Region Servers: Handle read/write requests.​

2.​ Regions: Data is divided into smaller chunks called regions for easy management.​

3.​ ZooKeeper: Coordinates and manages distributed nodes.​

Chapter 2: Cassandra – The Peer-to-Peer NoSQL Titan

Now, imagine a library where there’s no single master librarian, but every librarian has equal
authority and works independently. That’s Cassandra!

●​ What is Cassandra?​

○​ A distributed, peer-to-peer NoSQL database.​

○​ Designed for handling massive data across multiple servers with no single point
of failure.​

Key Features:

1.​ Decentralized: Every server (node) is equal; no master-slave relationship.​

2.​ Replication: Data is automatically copied across nodes for fault tolerance.​

3.​ Scalability: Handles millions of requests per second.​

4.​ High Availability: Even if some nodes fail, the system remains operational.​

Cassandra’s Data Model:

●​ Tables: Organize data in tables like a traditional database.​

●​ Partitions: Divide tables into smaller pieces for better performance.​

●​ Clusters: Groups of nodes that store data.​

Chapter 3: HBase vs. Cassandra

Feature HBase Cassandra

Architecture Master-Slave Peer-to-Peer

Data Column-Oriented Wide-Row Storage Model

Storage

Use Case Real-time analytics High availability & scalability

Consistency Strong consistency Tunable consistency

Chapter 4: Integrating with Hadoop

Both HBase and Cassandra integrate seamlessly with Hadoop for big data analytics:

●​ HBase with Hadoop: Uses HDFS for storage and MapReduce for data processing.​

●​ Cassandra with Hadoop: Processes data using Hadoop’s tools while leveraging
Cassandra’s distributed nature.​

Chapter 5: Real-World Applications

1.​ HBase Applications:​

○​ Facebook: Stores messaging data.​

○​ OpenTSDB: Time-series database for monitoring systems.​

2.​ Cassandra Applications:​

○​ Netflix: Stores user viewing histories.​

○​ Spotify: Manages playlists and user data.​

Story Summary

HBase and Cassandra are two NoSQL databases designed to handle massive amounts of data.

●​ HBase is perfect for real-time analytics and operates in a master-slave architecture.​

●​ Cassandra excels in high availability and scalability with its peer-to-peer setup.​
Both play vital roles in big data ecosystems and are used by companies like Facebook,
Netflix, and Spotify to power their services.​

Let me know when you're ready to explore the next unit!

Unit 6: Pig and Hive – Simplifying Big Data Processing

Welcome to the story of Pig and Hive, the tools that make working with big data as easy as
writing simple scripts or SQL queries. Imagine Pig and Hive as translators that help you interact
with Hadoop in your preferred language—whether you’re a developer or someone familiar with
databases.

Chapter 1: Pig – The Data Scriptwriter

Pig is like a scripting genius for Hadoop. It lets you process large datasets using a high-level
language called Pig Latin.

Why Pig?

●​ Writing MapReduce programs can be complex. Pig simplifies it.​

●​ Processes unstructured and semi-structured data.​

Key Components:

1.​ Pig Latin: A simple scripting language for processing data.​

○​ Example: Filtering out all users above the age of 25 from a dataset.​

2.​ Grunt Shell: Interactive environment for writing and running Pig scripts.​

Pig Workflow:

1.​ Write a Pig script in Pig Latin.​

2.​ Submit the script to the Grunt shell.​

3.​ Pig translates the script into MapReduce jobs and runs them on Hadoop.​

Example Task: Word Count

1.​ Load a file: A = LOAD 'input.txt' AS (word:chararray);​

2.​ Group by word: B = GROUP A BY word;​

 3.​ Count words: C = FOREACH B GENERATE group, COUNT(A);​

4.​ Store the result: STORE C INTO 'output';​

Chapter 2: Hive – The SQL Wizard

Hive is like a magical SQL translator for Hadoop, designed for people comfortable with
databases.

Why Hive?

●​ Lets you query Hadoop-stored data using SQL-like commands called HiveQL.​

●​ Ideal for structured data.​

Key Features:

1.​ Data Types and File Formats: Supports various types (INT, STRING, etc.) and formats
(Text, ORC, Parquet).​

2.​ HiveQL: A SQL-like language for querying data.​

3.​ Tables and Partitions:​

○​ Tables organize data logically.​

○​ Partitions split large datasets for faster queries.​

Hive Workflow:

1.​ Define the schema: Create a table to represent your data.​

2.​ Load data into the table.​

3.​ Query the data using HiveQL.​

Example Task: Count Users by Age

Create a table:​
​
CREATE TABLE users (name STRING, age INT);

1.​

Load data:​
​
LOAD DATA INPATH 'hdfs:/user/data.txt' INTO TABLE users;

2.​

Run a query:​
​
SELECT age, COUNT(*) FROM users GROUP BY age;

3.​

Chapter 3: Pig vs. Hive

Feature Pig Hive

Language Pig Latin HiveQL (SQL-like)

Data Type Semi-structured/Unstructured Structured

Use Case Data processing and Querying and analyzing

transformation data

Learning Curve Easier for programmers Easier for SQL users

Chapter 4: Real-World Applications

1.​ Pig Applications:​

○​ Data transformation and cleaning.​

○​ Processing logs for analytics.​

2.​ Hive Applications:​

○​ Business Intelligence (BI) tools for reporting.​

 ○​ Querying structured data in retail and finance.​

Story Summary

Pig and Hive simplify big data processing in Hadoop:

●​ Pig is perfect for developers who want to write scripts for data transformation.​

●​ Hive is ideal for SQL users who need to query and analyze structured data.​
These tools make it easier for people from different backgrounds to harness the power
of big data, enabling applications in business intelligence, data cleaning, and reporting.​

Let me know if you’d like to explore anything in more detail or summarize all units into a single
cohesive note!
The Big Data Adventure: A Cohesive Story of All Units

Imagine you’re on an exciting journey into the world of Big Data, where you’ll explore massive
datasets, innovative tools, and powerful technologies that make sense of it all. Let’s travel
through each chapter of this story, uncovering the secrets of how Big Data is managed,
processed, and utilized.

Chapter 1: The Big Data Landscape (Unit 1)

Once upon a time, data was simple and structured. But as the world became more connected,
data exploded into an overwhelming mix of structured, unstructured, and semi-structured forms.
Welcome to the world of Big Data!

Key Trends:

●​ Unstructured data grew from sources like social media, IoT devices, and logs.​

●​ Industries like healthcare, finance, and marketing started using Big Data for insights.​

Technologies to the Rescue:

●​ Hadoop emerged as a knight in shining armor for storing and processing data.​

●​ Cloud Computing provided the infrastructure to scale.​

●​ Crowd-sourcing Analytics allowed data collection from diverse sources.​

Big Data started solving real-world problems: detecting fraud, managing credit risks, and
improving healthcare outcomes.

Chapter 2: NoSQL Databases – The Data Architects (Unit 2)

Traditional databases struggled with Big Data. Enter NoSQL databases, designed to handle
massive, complex datasets.

Types of NoSQL Databases:

1.​ Key-Value Stores: Like a dictionary, perfect for simple lookups.​

 2.​ Document Stores: Store data as JSON or XML (e.g., MongoDB).​

3.​ Graph Databases: Ideal for relationship-heavy data, like social networks.​

Key Concepts:

●​ Sharding: Splitting data across servers for scalability.​

●​ Replication: Copying data to ensure fault tolerance.​

●​ Consistency: Balancing strong and eventual consistency for data reliability.​

With NoSQL, Big Data gained the flexibility it needed to grow.

Chapter 3: Hadoop and HDFS – The Backbone of Big Data (Unit 3)

To store and manage Big Data, the Hadoop Distributed File System (HDFS) became the
foundation. Think of HDFS as a giant warehouse for all types of data.

HDFS Features:

1.​ Distributed Storage: Splits data into blocks stored across multiple nodes.​

2.​ Fault Tolerance: Automatically replicates data to prevent loss.​

3.​ Scalability: Can handle petabytes of data.​

Additional Tools:

●​ Hadoop Streaming: Allows using non-Java languages like Python for processing.​

●​ Serialization: Optimizes how data is stored and transmitted.​

HDFS became the backbone of Big Data storage.

Chapter 4: MapReduce – The Engine of Processing (Unit 4)

HDFS is great for storage, but how do you process all that data? Meet MapReduce, the
superhero of Big Data processing.
How It Works:

1.​ Map Phase: Splits data into smaller chunks and processes them in parallel.​

2.​ Reduce Phase: Combines results to generate insights.​

Workflow:

●​ Input -> Map -> Shuffle/Sort -> Reduce -> Output​

Key Tools:

●​ YARN: Manages resources efficiently across clusters.​

●​ MRUnit: For testing MapReduce jobs.​

With MapReduce, tasks like counting website hits or analyzing customer reviews became faster
and scalable.

Chapter 5: HBase and Cassandra – The Dynamic Duo (Unit 5)

For real-time data access, HBase and Cassandra stepped in as specialized databases.

HBase:

●​ Column-oriented, works on top of HDFS.​

●​ Great for real-time analytics, like monitoring stock prices.​

Cassandra:

●​ Peer-to-peer architecture, ensuring no single point of failure.​

●​ Ideal for distributed systems like Netflix’s user data management.​

Both databases enhanced Big Data’s ability to serve diverse, high-speed needs.

Chapter 6: Pig and Hive – Simplifying the Complex (Unit 6)

Finally, we meet the tools that make Big Data processing accessible to everyone.

Pig:

●​ A scripting platform using Pig Latin for data transformation.​

●​ Example: Cleaning messy log files.​

Hive:

●​ A SQL-like tool using HiveQL for querying structured data.​

●​ Example: Analyzing sales data for trends.​

Comparison:

●​ Pig is for programmers; Hive is for SQL-savvy users.​

With these tools, Big Data became approachable for both technical and non-technical users.

The Big Data Adventure in a Nutshell

1.​ Big Data Concepts: Understanding the challenges and opportunities of massive data.​

2.​ NoSQL: Flexible databases for diverse data types.​

3.​ HDFS: Distributed, fault-tolerant storage.​

4.​ MapReduce: Parallel processing for insights.​

5.​ HBase and Cassandra: Real-time and distributed databases.​

6.​ Pig and Hive: Tools that simplify data processing and querying.​

Together, these technologies have transformed how industries harness data, making Big Data
not just a technical marvel but an essential tool for solving real-world problems.

Let me know if you'd like me to focus on any specific concept or example!