Ad3364 - Dev Lab Manual Final
Ad3364 - Dev Lab Manual Final
LAB MANUAL
(Regulation 2023)
(THIRD SEMESTER)
Prepared By
Mr.B.Yogesh Kumar, Asst. Prof. (O.G) / AI&DS
1
INDEX
E.NO EXPERIMENT NAME Pg. No.
B Syllabus 6
E Mode of Assessment 8
Perform Exploratory Data Analysis (EDA) on with datasets like email data set.
2 Export all your emails as a dataset, import them inside a pandas data frame, 13-16
visualize them and get different insights from the data.
3 Working with Numpy arrays, Pandas data frames, Basic Plots using Matplotlib. 17-21
Explore various variable and row filters in R for cleaning data. Apply various
4
plot features in R on sample data sets and visualize. 22-23
5 Perform Time Series Analysis and apply the various visualizations techniques. 24-26
Perform Data Analysis and representation on map using map data sets with
6
Mouse Rollover effect, user interaction, etc. 27-28
Use a case study on a data set and apply the various EDA and visualizations
9
techniques and present an analysis report. 37
2
PROGRAMME EDUCATIONAL OBJECTIVES (PEOs)
1. To afford the necessary background in the field of Information Technology to deal with engineering problems
to excel as engineering professionals in industries.
2. To improve the qualities like creativity, leadership, teamwork and skill thus contributing towards the growth
and development of society.
3. To develop ability among students towards innovation and entrepreneurship that caters to the needs of Industry
and society.
4. To inculcate and attitude for life-long learning process through the use of information technology sources.
5. To prepare then to be innovative and ethical leaders, both in their chosen profession and in other activities.
3
Create, select, and apply appropriate techniques, resources, and
modern engineering and IT tools, including prediction and
5 Modern tool usage
modeling to complex engineering activities, with an
understanding of the limitations.
Apply reasoning informed by the contextual knowledge to assess
societal, health, safety, legal, and cultural issues and the
6 The engineer and society
consequent responsibilities relevant to the professional
engineering practice
Understand the impact of the professional engineering solutions
Environment and
7 in societal and environmental contexts, and demonstrate the
sustainability
knowledge of, and need for sustainable development.
Apply ethical principles and commit to professional ethics and
8 Ethics
responsibilities and norms of the engineering practice
Function effectively as an individual, and as a member or leader
9 Individual and team work
in diverse teams, and in multidisciplinary settings
Communicate effectively on complex engineering activities with
the engineering community and with the society at large, such
10 Communication as, being able to comprehend and write effective reports and
design documentation, make effective presentations, and give
and receive clear instructions
Demonstrate knowledge and understanding of the engineering
Project management and and management principles and apply these to one’s own work,
11
finance as a member and leader in a team, to manage projects and in
multidisciplinary environments
Recognize the need for, and have the preparation and ability to
12 Life-long learning engage in independent and life-long learning in the broadest
context of technological change
4
PROGRAMME SPECIFIC OUTCOMES (PSOs)
After the completion of Bachelor of Technology in Artificial Intelligence and Data Science
programme the student will have following Program specific outcomes
1. Design and develop secured database applications with data analytical approaches of data preprocessing,
optimization, visualization techniques and maintenance using state of the art methodologies based on
ethical values.
2. Design and develop intelligent systems using computational principles, methods and systems for extracting
knowledge from data to solve real time problems using advanced technologies and tools.
3. Design, plan and setting up the network that is helpful for contemporary business environments using latest
software and hardware.
4. Planning and defining test activities by preparing test cases that can predict and correct errors ensuring a
socially transformed product catering all technological needs.
5
AD3364 DATA EXPLORATION AND VISUALIZATION LABORATORY LTPC
0 0 3 1.5
OBJECTIVES:
LIST OF EXPERIMENTS
TOTAL: 45 PERIODS
6
LIST OF EQUIPMENTS FOR A BATCH OF 30 STUDENTS
SOFTWARE:
Standalone desktops with Python 3 interpreter for Windows / Linux 30 Nos. (or) Server with Python 3
interpreter for Windows/Linux supporting 30 terminals or more.
HARDWARE:
Standalone Desktops: 30 Nos.
COURSE OUTCOMES
AD3364.5 Use data exploration and visualization techniques for multivariate and time series data.
PO PSO
CO
1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4
1 3 - 3 - - - - - - - - - 2 - - 1
2 - 2 - - 1 - - - - 2 - - 1 - - -
3 - - 3 1 - - - - - 3 - 2 2 - - -
4 2 - - - - - - - 1 - - - 2 - 1 2
5 - 3 1 3 1 - - - 1 - - 2 3 1 - -
Average 2.5 2.5 2.3 2.0 1.0 - - - 1.0 2.5 - 2.0 2.0 1.0 1.0 1.5
7
EVALUATION PROCEDURE FOR EACH EXPERIMENT
S. No Description Mark
2. Observation 30
5. Viva 10
Total 100
S. No Description Mark
2. Record 10
3. Model Test 15
Total 50
8
Ex. No: 1 INSTALLING DATA ANALYSIS AND VISUALIZATION TOOL
AIM
To write a step to install data analysis and visualization tool: R / Python / Tableau Public / Power BI.
PROCEDURE
1. R:
Download R:
Visit the official R website (https://cran.r-project.org/) and download the installer for
your operating system (Windows, macOS, or Linux).
Install R by following the instructions provided in the installer.
2. Python:
Download Python:
Visit the official website (https://www.python.org/downloads/) and download the Python
installer for your OS (Windows, macOS, or Linux).
Install Python by running the installer and making sure to check the option to add Python to your
system’s PATH during installation.
NumPy (Numerical Python) is an open-source core Python library for scientific computations. It is
a general-purpose array and matrices processing package.
jupyter-lab
9
(iii) JUPYTER NOTEBOOK
Jupyter notebook
Scipy is a Python library that is useful in solving many mathematical equations and algorithms. It
is designed on the top of Numpy library. SCIPY means Scientific Python.
Pandas is a Python Package that provides fast, flexible, and expressive data structures designed to
make working with “relational” or “labeled” data botheasy and intuitive.
Matplotlib is a comphrehensive library for creating static, animated, and interactive visualizations
in Python. Working with “relational” or “labeled” data botheasy and intuitive.
3. Tableau Public:
Tableau Public
It is a web-based tool, so there’s no installation required. Simply visit the Tableau Public
Website (https”//public.tableau.com/s/gallery) and create an account to start using it.
10
4. Power Bi:
Download Power BI Desktop:
Go to the official Power BI wenbsite (https://powerbi.microsoft.com/en-us/desktop/) and
download Power BI Desktop.
Installer Power BI Desktop by running the installer.
PROGRAM 1
import numpy as np
import pandas as pd
hafeez = [‘Hafeez’, 19]
aslam = [‘Aslam’, 21]
kareem = [‘kareem’, 18]
dataframe = pd.DataFrame([hafeez, aslam, kareem], columns = [‘Name’, ‘Age’])
print(dataframe)
Output 1
PROGRAM 1
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
data = pd.read_csv(“CountryData.csv”)
plt.hist(data)
plt.xlabel(“code”)
plt.ylabel(“Total_personal_income”)
plt.show()
First create a CSV file in excel with attributes ‘code’ and ‘Total_personal_income’.
Save the file with filename mentioned above “CountryData” with extension as .csv file.
11
Output 2
RESULT
Thus, the python program to install data analysis and visualization tools like R, Python, Tableau Public,
or Power BI, and their features were explored successfully.
12
Ex. No: 2 EXPLORATORY DATA ANALYSIS (EDA) ON WITH DATASETS
AIM
To perform exploratory data analysis (EDA) on with datasets like email data set.
PROCEDURE
Exploratory Data Analysis (EDA) on email datasets involves importing the data, cleaning it, visualizing it, and
extracting insights. Here's a step-by-step guide on how to perform EDA on an email dataset using Python and
Pandas
1. Import Necessary Libraries:
Import the required Python libraries for data analysis and visualization.
3. Data Cleaning:
Depending on your dataset, you may need to clean and preprocess the data. Common cleaning steps
include handling missing values, converting dates to datetime format, and removing duplicates.
4. Data Exploration:
Now, you can start exploring the dataset using various techniques. Here are some common EDA
tasks:
Basic Statistics:
Get summary statistics of the dataset.
Distribution of Dates:
Visualize the distribution of email dates.
PROGRAM
# Import necessary libraries
import pandas as pd
import matplotlib.pyplot as plt
13
import seaborn as sns
# Load the dataset
df = pd.read_csv('D:\ARCHANA\dxv\LAB\DXV\Emaildataset.csv')
# Display basic information about the dataset
print(df.info())
# Display the first few rows of the dataset
print(df.head())
# Descriptive statistics
print(df.describe())
# Check for missing values
print(df.isnull().sum())
# Visualize the distribution of numerical variables
sns.pairplot(df) plt.show()
# Visualize the distribution of categorical variables
sns.countplot(x='label', data=df) plt.show()
# Correlation matrix for numerical variables
correlation_matrix = df.corr()
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm')
plt.show()
# Word cloud for text data (if you have a column with text data) from wordcloud
import WordCloud text_data = ' '.join(df['text_column']) wordcloud =
WordCloud(width=800, height=400, random_state=21,
max_font_size=110).generate(text_data)
plt.figure(figsize=(10, 7))
plt.imshow(wordcloud, interpolation="bilinear")
plt.axis('off')
plt.show()
OUTPUT
Data columns (total 4 columns):
# Column Non-Null Count Dtype
14
0 Unnamed: 0 5171 non-null int64
1 label 5171 non-null object
2 text 5171 non-null object
3 label_num 5171 non-null int64
dtypes: int64(2), object(2)
memory usage: 161.7+ KB
None
Unnamed: 0 label text label_num
0 605 ham Subject: enron methanol; meter #: 988291\r\n... 0
1 2349 ham Subject: hpl nom for january 9 , 2001\r\n( see... 0
2 3624 ham Subject: neon retreat\r\nho ho ho , we ' re ar... 0
3 4685 spam Subject: photoshop , windows , office . cheap ... 1
4 2030 ham Subject: re : indian springs\r\nthis deal is t... 0
Unnamed: 0 label_num
count 5171.000000 5171.000000
mean 2585.000000 0.289886
std 1492.883452 0.453753
min 0.000000 0.000000
25% 1292.500000 0.000000
50% 2585.000000 0.000000
75% 3877.500000 1.000000
max 5170.000000 1.000000
Unnamed: 0 0
label 0
text 0
label_num 0
dtype: int64
15
RESULT
Thus, the above Performing exploratory data analysis (EDA) on with datasets like email data set has been
performed successfully.
16
Ex. No:3 WORKING WITH NUMPY ARRAYS, PANDAS DATA FRAMES, BASIC PLOTS
USING MATPLOTLIB
AIM
To write the steps for Working with Numpy arrays, Pandas data frames, Basic plots using Matplotlib
PROCEDURE
1. NumPy:
NumPy is a fundamental library for numerical computing in Python. It provides support for multi-
dimensional arrays and various mathematical functions. To get started, you’ll first need to install NumPy
if you haven’t already (you can use pip):
17
OUTPUT
2. Pandas:
Pandas is a powerful library for data manipulation and analysis.
You can install pandas using pip:
OUTPUT
19
3. Matplotlib:
Matplotlib is a popular library for creating static, animated, or interactive plots and graphs.
Install Matplotlib using pip:
pip install matplotlib
OUTPUT
RESULT
Thus, the above working with numpy, pandas, and matplotlib has been completed successfully.
21
Ex. No: 4 EXPLORING VARIOUS VARIABLE AND ROW FILTERS IN R FOR CLEANING
DATA
AIM
To exploring various variable and row filters in R for cleaning data.
PROCEDURE
Data Preparation and Cleaning
First, let’s create a sample dataset and then explore various variable and row filters to clean the data.
# Create a sample dataset
set.seed(123)
data <- data.frame( ID = 1:10, Age = sample(18:60, 10, replace = TRUE), Gender = sample(c("Male",
"Female"), 10, replace = TRUE), Score = sample(1:100, 10) )
# Print the sample data
print(data)
OUTPUT
Variable Filters
1. Filtering by s Specific Value:
To filter rows based on a specific value in a variable (e.g., only show rows where Age is greater than
30): filtered_data <-data [data$Age>30, ]
2. Filtering by Multiple Conditions:
You can filter row based on multiple conditions using the & (AND) or | (OR) operators (e.g., show rows
22
where Age is greater than 30 and Gender is “Male”): filtered_data <- data [data$Age > 30 &
data$Gender == “Male”,]
Row Filters
1. Removing Duplicate Rows:
To remove duplicate rows based on certain columns (e.g., remove duplicates based on ‘ID’):
cleaned_data <- unique (data [, c(“ID”, “Age”, “Gender”)])
2. Removing Rows with Missing Values:
To remove rows with missing values (NA):
cleaned_data <-na.omit (data)
Data Visualization
Apply various plot features using the ggplot2 package to visualize the cleaned data.
# Load the ggplot2 package library (ggplot2)
# Create a scatterplot of Age vs. Score with points colored by Gender
Ggplot (data = cleaned_data, aes(x = Age, y = Score, color = Gender)) + geom_point () + labs(title = "Scatterplot
of Age vs. Score", x = "Age", y = "Score")
# Create a histogram of Age
Ggplot (data = cleaned_data, aes(x = Age)) + geom_histogram (binwidth = 5, fill = "blue", alpha = 0.5) + labs
(title = "Histogram of Age", x = "Age", y = "Frequency")
# Create a bar chart of Gender distribution
Ggplot (data = cleaned_data, aes(x = Gender)) + geom_bar (fill = "green", alpha = 0.7) + labs (title = "Gender
Distribution", x = "Gender", y = "Count")
RESULT
Thus, the above exploring various variable and row filters in R for cleaning data was successfully
completed.
23
Ex. No: 5 TIME SERIES ANALYSIS USING VARIOUS VISUALIZATION TECHNIQUES
AIM
To perform time series analysis and apply the various visualization techniques.
PROCEDURE
DOWNLOAD DATASET
Step 1: Open google and type the following path in the address bar and download a dataset.
http://github.com/jbrownlee/Datasets.
Step 2: Write the following code to get the details.
from pandas import read_csv
from matplotlib import pyplot
series = read_csv (‘pathname’)
print (series.head ( ))
series.plot ( )
pyplot.show ( )
OUTPUT
24
Step 3: To get the time series line plot:
series.plot (style=’-.’)
pyplot.show ( )
OUTPUT
OUTPUT
25
Step 5: To create density plot:
series.plot (kind = ‘kde’)
pyplot.show ( )
OUTPUT
RESULT
Thus, the above time analysis has been checked with various visualization techniques.
26
Ex. No: 6 DATA ANALYSIS AND REPRESENTATION ON MAP
AIM
Write a program to perform data analysis and representation on a map using various map data sets with
mouse rollover effect, user interaction etc.
PROCEDURE
STEP 1:
Make sure to install the necessary libraries.
pip install geopandas folium bokeh
PROGRAM
from bokeh.io import show
from bokeh.models import ColumnDataSource, HoverTool
from bokeh.plotting import figure
from bokeh.layouts import column
import pandas as pd
import folium
# Load your data
data = pd.read_csv('D:\ARCHANA\dxv\LAB\DXV\geographic.csv')
# Create a Bokeh figure
p = figure(width=800, height=400, tools='pan,wheel_zoom,reset')
# Create a ColumnDataSource to hold data
source = ColumnDataSource(data)
# Add circle markers to the figure
p.circle(x='Longitude', y='Latitude', size=10, source=source, color='orange')
# Create a hover tool for mouse rollover effect
hover = HoverTool() hover.tooltips = [("Info", "@Info"), ("Latitude", "@Latitude"), ("Longitude",
"@Longitude")]
p.add_tools(hover)
# Display the Bokeh plot
layout = column(p)
27
show(layout)
# Create a map centered at a specific location
m = folium.Map(location=[latitude, longitude], zoom_start=10)
# Add markers for your data points
for index, row in data.iterrows():
folium.Marker( location=[row['Latitude'], row['Longitude']], popup=row['Info'], # Display additional
info on mouse click ).add_to(m)
# Save the map to an HTML file
m.save('map.html')
OUTPUT
RESULT
Thus, the data analysis and representation on a map using various map data sets with mouse rollover effect,
use interaction has been completed successfully.
28
Ex. No: 7 BUILDING CARTOGRAPHIC VISUALIZATION
AIM
Build cartographic visualization for multiple datasets involving various countries of the world; states
and districts in India etc.
PROCEDURE
STEP 1:
Collect Datasets
Gather the datasets containing geographical information for countries, states, or districts. Make sure these
datasets include the necessary attributes for mapping (e.g., country/state/district names, codes, and relevant
data).
STEP 2:
Install Required Libraries:
pip install geopandas matplotlib
STEP 3:
Load Geographic Data:
Use Geopandas to load the geographic data for countries, states, or districts. Make sure to match the
geographical data with your datasets based on the common attributes.
STEP 4:
Merge Datasets:
Merge your datasets with the geographic data based on common attributes. This step is crucial for linking your
data to the corresponding geographic regions.
STEP 5:
Create Cartographic Visualizations:
Use Matplotlib to create cartographic visualizations. You can create separate plots for different datasets or
overlay them on a single map.
29
STEP 6:
Customize and Enhance:
Customize your visualizations based on your needs. You can add legends, labels, titles, and other elements to
enhance the interpretability of your maps.
STEP 7:
Save and Share:
Save your visualizations as image files or interactive plots if needed. You can then share these visualizations
with others.
PROGRAM:
import pandas as pd
import geopandas as gpd
import shapely
# needs 'descartes'
import matplotlib.pyplot as plt
df = pd.DataFrame({'city': ['Berlin', 'Paris', 'Munich'], 'latitude': [52.518611111111, 48.856666666667,
48.137222222222], 'longitude': [13.408333333333, 2.3516666666667, 11.575555555556]})
gdf = gpd.GeoDataFrame(df.drop(['latitude', 'longitude'], axis=1), crs={'init': 'epsg:4326'},
geometry=[shapely.geometry.Point(xy) for xy in zip(df.longitude, df.latitude)])
print (gdf)
world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
base = world.plot(color='white', edgecolor='black')
gdf.plot (ax=base, marker='o', color='red', markersize=5)
plt.show ( )
OUTPUT
city geometry
0 Berlin POINT (13.40833 52.51861)
1 Paris POINT (2.35167 48.85667)
2 Munich POINT (11.57556 48.13722)
30
RESULT
Build cartographic visualization for multiple datasets involving various countries of the world; has
been visualized successfully.
31
Ex. No: 8 PERFORM EDA ON WINE QUALITY DATA SET.
AIM
To write a program to peform EDA on Wine Quality Data Set.
PROGRAM
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
# Load the dataset
data = pd.read_csv("pathname")
# Display the first few rows of the dataset
print(data.head())
# Get information about the dataset
print(data.info())
# Summary statistics
print (data.describe())
# Distribution of wine quality
sns.countplot (data ['quality'])
plt.title (" Wine Quality data set")
plt.show ( )
# Box plots for selected features by wine quality
features = ['alcohol', 'volatile acidity', 'citric acid', 'residual sugar']
for feature in features:
plt.figure (figsize=(8, 6))
sns.boxplot(x='quality', y=feature, data=data)
plt.title (f'{feature} by Wine Quality')
plt.show ( )
# Pair plot of selected features
sns.pairplot (data, vars= ['alcohol', 'volatile acidity', 'citric acid', 'residual sugar'], hue='quality', diag_kind='kde')
plt.suptitle ("Pair Plot of Selected Features")
plt.show ( )
32
# Correlation heatmap
corr_matrix = data.corr ( )
plt.figure (figsize = (10, 8))
sns.heatmap (corr_matrix, annot=True, cmap="coolwarm", fmt=".2f")
plt.title ("Correlation Heatmap")
plt.show ( )
# Histograms of selected features
features = ['alcohol', 'volatile acidity', 'citric acid', 'residual sugar']
for feature in features:
plt.figure (figsize = (6, 4))
sns.histplot (data [feature], kde=True, bins=20)
plt.title (f"Distribution of {feature}")
plt.show ( )
OUTPUT
33
34
35
RESULT
Thus the above program to perform EDA on Wine Quality Data Set.
36
Ex. No: 9 VISUALIZING VARIOUS EDA TECHNIQUES AS CASE STUDY FOR IRIS
DATASET
AIM
The Mini Project to predict the time taken to solve a problem given the current status of the user using
Random Forest Regressor Model.
PROCEDURE
Import Libraries:
Start by importing the necessary libraries and loading the dataset.
Descriptive Statistics:
Compute and display descriptive statistics.
python
Check for Missing Values:
Verify if there are any missing values in the dataset.
Visualize Data Distributions:
Visualize the distribution of numerical variables.
python
Correlation Heatmap:
Examine the correlation between numerical variables.
Boxplots for Categorical Variables:
Use boxplots to visualize the distribution of features by species.
Violin Plots:
Combine box plots with kernel density estimation for better visualization.
Correlation between Features:
Visualize pair-wise feature correlations.
37