Join our channel if you haven’t joined yet https://t.me/fresco_milestone (https://t.
me/fresco_milestone) ( @fresco_milestone )
In [1]: import pandas as pd
import numpy as np
import csv
Fill in the Command to load your CSV dataset "imdb.csv" with pandas
In [2]: #Data Loading
imdb=pd.read_csv('imdb.csv')
imdb.columns = ["index","text","label"]
print(imdb.head(5))
index text label
0 0 A very, very, very slow-moving, aimless movie ... 0
1 1 Not sure who was more lost - the flat characte... 0
2 2 Attempting artiness with black & white and cle... 0
3 3 Very little music or anything to speak of. 0
4 4 The best scene in the movie was when Gerardo i... 1
Data Analysis
Get the shape of the dataset and print it.
Get the column names in list and print it.
Group the dataset by label and describe the dataset to understand the basic statistics of the dataset.
Print the first three rows of the dataset
In [3]: data_size =imdb.shape
print(data_size)
imdb_col_names =list(imdb.columns)
print(imdb_col_names)
print(imdb.describe(include='all'))
print(imdb.head(3))
(1000, 3)
['index', 'text', 'label']
index text label
count 1000.000000 1000 1000.00000
unique NaN 997 NaN
top NaN 10/10 NaN
freq NaN 2 NaN
mean 499.500000 NaN 0.50000
std 288.819436 NaN 0.50025
min 0.000000 NaN 0.00000
25% 249.750000 NaN 0.00000
50% 499.500000 NaN 0.50000
75% 749.250000 NaN 1.00000
max 999.000000 NaN 1.00000
index text label
0 0 A very, very, very slow-moving, aimless movie ... 0
1 1 Not sure who was more lost - the flat characte... 0
2 2 Attempting artiness with black & white and cle... 0
Target Identification
Execute the below cell to identify the target variables. If 0 it is a bad review,if it is 1 it is a good review.
In [4]: imdb_target=imdb['label']
print(imdb_target)
Tokenization
Convert the text into lower.
Tokenize the text using word_tokenize
Apply the function split_tokens for the column text in the imdb dataset with axis =1
In [6]: from nltk.tokenize import word_tokenize
import nltk
nltk.download('all')
def split_tokens(text):
message = text.lower()
word_tokens = word_tokenize(message)
return word_tokens
imdb['tokenized_message'] = imdb.text.apply(split_tokens)
[nltk_data] Downloading collection 'all'
[nltk_data] |
[nltk_data] | Downloading package abc to /home/user/nltk_data...
Lemmatization
Apply the function split_into_lemmas for the column tokenized_message with axis=1
Print the 55th row from the column tokenized_message.
Print the 55th row from the column lemmatized_message
In [7]: from nltk.stem.wordnet import WordNetLemmatizer
def split_into_lemmas(text):
lemma = []
lemmatizer = WordNetLemmatizer()
for word in text:
a=lemmatizer.lemmatize(word)
lemma.append(a)
return lemma
imdb['lemmatized_message'] = imdb.tokenized_message.apply(split_into_lemmas)
print('Tokenized message:',imdb.tokenized_message[54] )
print('Lemmatized message:',imdb.lemmatized_message[54] )
Tokenized message: ['long', ',', 'whiny', 'and', 'pointless', '.']
Lemmatized message: ['long', ',', 'whiny', 'and', 'pointless', '.']
Stop Word Removal
Set the stop words language as english in the variable stop_words
Apply the function stopword_removal to the column lemmatized_message with axis=1
Print the 55th row from the column preprocessed_message
In [8]: from nltk.corpus import stopwords
def stopword_removal(text):
stop_words = stopwords.words('english')
filtered_sentence = []
filtered_sentence = ' '.join([word for word in text if word not in stop_words])
return filtered_sentence
imdb['preprocessed_message'] = imdb.lemmatized_message.apply(stopword_removal)
print('Preprocessed message:',imdb.preprocessed_message[54])
Training_data=pd.Series(list(imdb['preprocessed_message']))
Training_label=pd.Series(list(imdb['label']))
Preprocessed message: long , whiny pointless .
Term Document Matrix
Apply CountVectorizer with following parameters
ngram_range = (1,2)
min_df = (1/len(Training_label))
max_df = 0.7
Fit the tf_vectorizer with the Training_data
Transform the Total_Dictionary_TDM with the Training_data
In [9]: from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
tf_vectorizer = CountVectorizer(ngram_range = (1,2),min_df = (1/len(Training_label)),max_df = 0.7 )
Total_Dictionary_TDM = tf_vectorizer.fit(Training_data)
message_data_TDM = tf_vectorizer.transform(Training_data)
Term Frequency Inverse Document Frequency (TFIDF)
Apply TfidfVectorizer with following parameters
ngram_range = (1,2)
min_df = (1/len(Training_label))
max_df = 0.7
Fit the tfidf_vectorizer with the Training_data
Transform the Total_Dictionary_TFIDF with the Training_data
In [10]: from sklearn.feature_extraction.text import CountVectorizer,TfidfVectorizer
tfidf_vectorizer = TfidfVectorizer( ngram_range = (1,2),min_df = (1/len(Training_label)),max_df = 0.7 )
Total_Dictionary_TFIDF = tfidf_vectorizer.fit(Training_data)
message_data_TFIDF = tfidf_vectorizer.transform(Training_data)
Train and Test Data
Splitting the data for training and testing(90% train,10% test)
Perform train-test split on message_data_TDM and Training_label with 90% as train data and 10% as test data.
In [11]: from sklearn.model_selection import train_test_split
train_data,test_data, train_label, test_label = train_test_split(message_data_TDM,Training_label,test_size=0.1)
Join our channel if you haven’t joined yet https://t.me/fresco_milestone (https://t.me/fresco_milestone) ( @fresco_milestone )
Support Vector Machine
Get the shape of the train-data and print the same.
Get the shape of the test-data and print the same.
Initialize SVM classifier with following parameters
kernel = linear
C= 0.025
random_state=seed
Train the model with train_data and train_label
Now predict the output with test_data
Evaluate the classifier with score from test_data and test_label
Print the predicted score
In [12]: seed=9
from sklearn.svm import SVC
train_data_shape = train_data.shape
test_data_shape = test_data.shape
print("The shape of train data:", train_data_shape)
print("The shape of test data:", test_data_shape)
classifier = SVC( kernel='linear',C=0.025,random_state=seed )
classifier = classifier.fit(train_data,train_label)
target = classifier.predict(test_data)
score = classifier.score(test_data,test_label)
print('SVM Classifier : ',score)
with open('output.txt', 'w') as file:
file.write(str((imdb['tokenized_message'][55],imdb['lemmatized_message'][55])))
The shape of train data: (900, 9051)
The shape of test data: (100, 9051)
SVM Classifier : 0.73
Stochastic Gradient Descent Classifier
Perform train-test split on message_data_TDM and Training_label with this time 80% as train data and 20% as test data.
Get the shape of the train-data and print the same.
Get the shape of the test-data and print the same.
Initialize SVM classifier with following parameters
loss = modified_huber
shuffle= True
random_state=seed
Train the model with train_data and train_label
Now predict the output with test_data
Evaluate the classifier with score from test_data and test_label
Print the predicted score
In [13]: from sklearn.linear_model import SGDClassifier
train_data,test_data, train_label, test_label = train_test_split(message_data_TDM,Training_label,test_size=0.2)
train_data_shape = train_data.shape
test_data_shape = test_data.shape
print("The shape of train data:", train_data_shape )
print("The shape of test data:", test_data_shape )
classifier = SGDClassifier( loss='modified_huber',shuffle=True,random_state=seed )
classifier = classifier.fit(train_data,train_label)
target = classifier.predict(test_data)
score = classifier.score(test_data,test_label)
print('SGD classifier : ',score)
with open('output1.txt', 'w') as file:
file.write(str((imdb['preprocessed_message'][55])))
The shape of train data: (800, 9051)
The shape of test data: (200, 9051)
SGD classifier : 0.7
In [ ]: "@fresco_milestone"
