SUBJECT CODE : 310251
As per Revised Syllabus of
Savitribai Phule Pune University
Choice Based Credit System (CBCS)
T.E. (Computer) Semester - VI
Data Science
and Big Data Analytics
Iresh A. Dhotre
M.E. (Information Technology)
Ex-Faculty, Sinhgad College of Engineering,
Pune.
Dr. Kalpana V. Metre
(Ph.D.),
Associate Professor,
MET’s Institute of Engineering,
Nashik.
® ®
TECHNICAL
PUBLICATIONS
SINCE 1993 An Up-Thrust for Knowledge
(i)
Data Science and Big Data Analytics
Subject Code : 310251
T.E. (Computer Engineering) Semester - VI
ã Copyright with I.A.Dhotre
All publishing rights (printed and ebook version) reserved with Technical Publications. No part of this book
should be reproduced in any form, Electronic, Mechanical, Photocopy or any information storage and
retrieval system without prior permission in writing, from Technical Publications, Pune.
Published by :
® ®
Amit Residency, Office No.1, 412, Shaniwar Peth,
TECHNICAL Pune - 411030, M.S. INDIA, Ph.: +91-020-24495496/97
PUBLICATIONS
SINCE 1993 An Up-Thrust for Knowledge Email : sales@technicalpublications.org Website : www.technicalpublications.org
Printer :
Yogiraj Printers & Binders
Sr.No. 10/1A,
Ghule Industrial Estate, Nanded Village Road,
Tal. - Haveli, Dist. - Pune - 411041.
ISBN 978-93-91567-92-7
SPPU 19
9 789391 567927
9789391567927 [1] (ii)
(iv)
(v)
TM
TECHNICAL PUBLICATIONS - An up thrust for knowledge
?
Clickstream logs Emails
Application logs Contracts
Geographical
Data information
Machine data systems and
volume
geo-spatial data
Mobile
Sensor data
networks
Amazon,
facebook,
Data Social
Yahoo, Google
velocity media
(Web based
companies)
Structured
Data
variety Unstructured
Semi-structured
Business
understanding
Data Data
mining exploration
Data science
Data
Feature visualization
engineering
Data Predictive
cleaning modeling
Printed
books
101010101110010101
010100001110001100
Videos 011010101000011100
011110000011100010
hp
101010101010000111
000101010001100110
Photos and 101010101010101000
drawing
Types of data
Structured Unstructured
Attributes
Qualitative/Categorical Quantitative/Numeric
Nominal Ordinal Intervel Ratio
1 st
Wrangling
Raw data Evaluate usability Analyze Visualize
Cleanse Usable data Findings
V1 V2 V3 V4
V – Min A
V ( New – MaxA – New – Min A ) New – Min A
MaxA – Min A
V – min marks
( New marks – New min) + New min
Max marks – Min marks
8– 8
(1 – 0) 0
20 – 8
( 0)
1
12
(10 – 8)
(1 – 0) 0
20 – 8
2
1
12
(15 – 8)
(1 – 0) 0
20 – 8
( 3)
1
12
( 20 – 8)
(1 – 0) 0
20 – 8
12
1
12
V i 10 j
Vi
Vi [{( V i – min) (max – min) } { new_ max – new_ min} ] new_ min
(76, 300 – Mean) (54000)
Standard deviation 16000
54000
16000
min A maxA new_ min A
new_ maxA v
v – min A
v (new_ maxA – new_ min A ) new_ min A
maxA – min A
73600 – 12000
v (1.0 – 0.0) 0.0
98000 – 12000
v
min A maxA new_ min A
new_ maxA v
v – min A
v (new_ maxA – new_ min A ) new_ min A
maxA – min A
600 – 200
v (1.0 – 0.0) 0.0
1000 – 200
v 600
j
10 10 3
State
University
Branch
College
TM
TECHNICAL PUBLICATIONS - An up thrust for knowledge
Probability
Used to summarize data
Population
Descriptive
statistics
Sample
Inferential statistics
Used to make
inferences about the population
x
Sum of the values of the n observations xi
x
Number of observations in the sample n
Sum of the values of the N observations xi
Number of observations in the population n
(19 26)
2
Q1
Q1
Q3
Q3
Q1
Q3 Q3 Q1
xi x
(x i x) 2
S2
n 1
2
(x i )2
N
2
(x )2
N
(x x) 2
S2
n 1
(x M) 2 (x M) 2
n n
(x)
x 5 10 15 20 50
x
n 4 4
x –x (x – x) 2
(– 7.5) 2
(– 2.5) 2
(2.5) 2
(7.5) 2
(x – x) 2
(x – x) 2 125.01
6.455
n –1 4 –1
600 470 170 430 300
5
(206) 2 (76) 2 (– 224) 2 (36) 2 (– 94) 2
5
42436 5776 50176 1296 8836
5
B 1 , B 2 , B 3 , ... , B n
1 k n
P(A B k ) P(B k )
P(B k A )
n
P(A B i ) P(B i )
i 1
P(D A ) P(A)
P(D A ) P(A) P(D B) P(B) P(D C) P(C)
0.05 0.25
0.05 0.25 0.04 0.35 0.02 0.4
0.0125
0.0125 0.014 0.008
P(D B) P(B)
P(D A ) P(A) P(D B) P(B) P(D C) P(C)
0.04 0.35
0.05 0.25 0.04 0.35 0.02 0.4
0.014 0.014
0.0125 0.014 0.008 0.0345
P(D C) P(C)
P(D A ) P(A) P(D B) P(B) P(D C) P(C)
0.02 0.4
0.05 0.25 0.04 0.35 0.02 0.4
0.008 0.008
0.0125 0.014 0.008 0.0345
P(T F) P(F)
P(T F) P(F) P(T M) P(M)
1 3 3
100 5 500
1 3 4 2 3 8
100 5 100 5 500 500
3
11
62 36
1
9
A B
2
P(A B)
36 18
P(A B) P(A) P(B A )
P(A B)
P(A)
1 18 1 9
19 18 1
1 3
P(T F)P(F) 100 5 3
P(T F) P(F) P(T M) P(M) 1 3 4 2 11
100 5 100 5
H0
H1
Reject H0 Reject H0
Middle 95 %
high-probability values
if H0 is true
80
from H0
Z = – 1.96 0 Z = 1.96
Critical region;
Extreme 5 %
( )
P
Z cal
P
(Z cal )
( )
(z )
Formulate H0 and H1
Select appropriate test
Choose level of significance
Calculate test statistic TSCAL
Determine probability Determine critical
association with test stat value of test stat TSCR
Determine if TSCR
Compare with level
of significance, falls into (Non)
Rejection Region
Reject / Do not Reject H0
Draw marketing research conclusion
f(x, )
Rejection Rejection
Acceptance
region region
region
(1 – )
/2 /2
1 2
f(x, )
(1 – )
/2 /2
X
1 2
(H0 )
H0
(H1 )
H1
H1
H1
1 – (1 – 0.05) 20
H0 H1
( x i – x) ( y i – y)
(x i – x) 2 (y i – y) 2
xi
yi
(a) Positive correlation (b) Negative correlation (c) No correlation
X2 Y2
X Y X2 Y2 XY
X 6500
X
n 10
Y 660
Y
n 10
1
cov (X , Y) XY XY
n
x y 1 2 1 2
X2 X Y2 Y
n n
456040
650 66
10
4764800 45784
650 2 66 2
10 10
X2 Y2
X Y X2 Y2 XY
X 420
X
n 7
Y 476
Y
n 7
1
cov (X , Y) XY XY
n
x y 1 2 1 2
X2 X Y2 Y
n n
29464
60 68
7
28096 34376
60 2 68 2
7 7
3
1 2 3
d1 d2 d3 d12 d 22 d 32
6 d 12 6 74
( x, y) 1 1
n (n 2 1) 10 (100 1)
6 d 22 6 44
( y , z) 1 1
n (n 2 1) 10 (100 1)
6 d 23 6 156
( z , x) 1 1
n ( n 2 1) 10 (100 1)
( y , z) 2 3
2
( )
2 (Observed Expected) 2
Expected
2
k (O E) 2
E
i 1
xi i
2 2
i
2 2 2
2 (x 1 1) (x 2 2) (x v v)
...
2 2 2
1 2 v
v (x i 2
i)
2
i 1 i
2
d.f. = 2
0.2
d.f. = 5
0.1 d.f. = 10
d.f. = 15
d.f. = 30
10 20 30 40 50
Critical region
area =
2
X
(Critical value)
0
xi 2.15 1.99 2.05 2.12 2.17 2.01 1.98 2.03 2.25 1.93
x
n 10
20.68
x 2.068
10
(x x) 2
S2
n
x x (x x) 2
(x x) 2 0.09096
S2
n 10
2 2
(H 0 ) : 0
2 2
(H 1 ) : 0
n s2 10 0.009096
X2
2 (0.145) 2
0
X2
(H 0 )
n
2 (observed expected) 2
expected
B1 B2
A1 a b r1
A2 a|c b|d r2
c1 c2 a|b||
cd n
X2
n( ad bc) 2
X2
c 1 c 2 r1 r2
X 0
Z Z2
n
2
(Of) ij (Ef) ij
2
(Ef) ij
( 31 30) 2 ( 42 30) 2 ( 22 30) 2 ( 25 30) 2
30 30 30 30
(19 20) 2 ( 8 20) 2 ( 28 20) 2 ( 25 20) 2
20 20 20 20
n1 n2
X1 – X2
1 1
SP
n1 n2
( n 1 – 1) S 21 ( n 2 – 1) S 22
S 2P
n1 n2 – 2
n1 n2
n 1 (N 1)
w
2
n 1 n 2 (N 1)
w
2
