AI Simplified by Aiforkids.

in

NPL Class
roPje 10
AI Notes
 NATURAL
LANGUAGE
PROCESSING
Start
Process to simplify human
The ability of a computer to lang. to make it
understand text and spoken
NLP Process
Data
Ex. Mitsuku Bot, Clever
Wh Processing
Bot, Jabberwacky, and
at Text Normalisation
Sentence Segmentation
Tokenisation
Chat
Removal of Stop word
Bots Smart Converting into same case
Application
Bot Script s of NLP Stemming and
Bot Lemmatization
Wh
Automatic Bag of word Algorithm
Summarization
y
TFIDF
Sentiment Term Frequency
Analysis Text Inverse Document Frequency

classification
Virtual Applications of TFIDF
Assistants Problems in Understanding
human languages by
computers. CLICK TEXT TO OPEN THE LINK
Human
Language Computer Download Revision Notes Pdf
Huma
Language Arrangement of Solve Important Questions
n VS
words & meanings Comput Practice VIP Questions PDF
er
(Structure) Practice Sample Papers
Syntax (Meaning)
Ask and Solve Doubts
Semantics
at Aiforkids Doubts
Multiple Meanings of a corner
word Perfect Syntax, no NLP Explanation Video
Meaning Practice

Youtube.com/ Aiforkids.in/class-
aiforkids 10/nlp
" Learning is not a course, Its a path from passio
 1

WHAT IS NLP?

Natural Language Processing (NLP) is the sub-field of AI that focuses

on the ability of a computer to understand human language (command)
as spoken or written and to give an output by processing it.

APPLICATIONS OF NLP
→ Feel Deta h, NLP KA

Automatic Summarization
Summarizing the meaning of documents and information
Extract the key emotional information from the text to understand the
reactions (Social Media)

Sentiment Analysis

Identify sentiments and emotions from one or more posts

Companies use it to identify opinions and sentiments to get
feedback Can be Positive, Negative or Neutral

Text classification

Assign predefined categories to a document and organize it to help you find

the information you need or simplify some activities.
Eg: Spam filtering in email.
 2

Virtual Assistants

By accessing our data, they can help us in keeping notes of our

tasks, making calls for us, sending messages, and a lot more.
With speech recognition, these assistants can not only detect our
speech but can also make sense of it.
A lot more advancements are expected in this field in the near
future Eg: Google Assistant, Cortana, Siri, Alexa, etc

REVISING AI PROJECT CYCLE

Project Cycle is a step-by-step process to solve problems using

proven scientific methods and drawing inferences about them.
1COMPONENTS OF PROJECT CYCLE

Problem Scoping - Understanding the problem

Data Acquisition - Collecting accurate and reliable
data Data Exploration - Arranging the data
uniformly Modelling - Creating Models from the
data
Evaluation - Evaluating the project

The Stakeholder Who

[Problem
Have a problem Issue/Problem What
Statement
When/While Context/Situation/Location Wher Template]
e
Ideal Solution How the Solution will help Stakeholders
 3

CHATBOTS

One of the most common applications of Natural Language Processing is a

chatbot.

Exampl
Mitsuku Jabberwack Rose
e:
Bot Clever y Haptic OChatb
Bot ot

Types of ChatBots

SCRIPT BOTS SMART

BOTS
Easy to make Comparatively difficult to make

Work on the script of the Work on bigger databases

programmed set.
Limited functionality Wide functionality

No or little language processing skills Coding is required

Example: Customer Care Bots. Example: Google Assistant, Alexa,
Cortana, Siri, etc.
 4

HUMAN LANGUAGE VS COMPUTER LANGUAGE

1 HUMAN LANGUAGE

Humans communicate through language which we process all the

time. Our brain keeps on processing the sounds that it hears
around itself and tries to make sense out of them all the time.
Communications made by humans are complex.
2 COMPUTER LANGUAGE

The computer understands the language of numbers.

Everything that is sent to the machine has to be converted
to numbers. A single mistake is made, the computer throws an
error and does not process that part.
The communications made by the machines are very basic and simple

ERRORS IN PROCESSING HUMAN LANGUAGE

Arrangement of words and

meaning Different Syntax, Our Brain
Same meaning Different
Listen Prioritize
Meaning, Same Syntax
Multiple Meanings of the Word
Process
Perfect Syntax, No Meaning
 5

ARRANGEMENT OF THE WORDS AND MEANING

Syntax: Syntax refers to the grammatical structure of a

sentence. Semantics: It refers to the meaning of the

Different syntax, same semantics: 2+3 = 3+2
sentence.

Here the way these statements are written is different, but their
meanings are the same that is 5.
Different semantics, same syntax: 3/2 (Python 2.7) ≠ 3

Here we have the same syntax but their meanings are different. In
Python 2.7, this statement would result in 1 while in Python 3, it
would give an output of 1.5.

1
MULTIPLE MEANINGS OF A WORD

To understand let us have an example of the following three sentences:

1. "His face turned red after he found out that he had taken the wrong
bag" Possibilities: He feels ashamed because he took another
person’s bag instead of his OR he's feeling angry because he did not
manage to steal the bag that he has been targeting.
2. "The red car zoomed past his nose"
Possibilities: Probably talking about the color of the car, that
traveled close to him in a flash.
 6

3. "His face turns red after consuming the medicine"

Possibilities: Is he having an allergic reaction? Or is he not able to bear the
taste of that medicine?

2
PERFECT SYNTAX, NO MEANING

1. "Chickens feed extravagantly while the moon drinks tea"

Meaning: This statement is correct grammatically but makes no
sense. In Human language, a perfect balance of syntax and semantics
is important for better understanding.

DATA PROCESSING

Since we all know that the language of computers is Numerical, the

very first step that comes to our mind is to convert our language to
numbers.
This conversion takes a few steps to happen. The first step to it is
Text Normalisation.

TEXT NORMALISATION

In Text Normalization, we undergo several steps to normalize the

text to a lower level. That is, we will be working on text from multiple
documents and the term used for the whole textual data from all the
documents altogether is known as "Corpus".
 7

1
SENTENCE SEGMENTATION

Under sentence segmentation, the whole corpus is divided into

sentences. Each sentence is taken as a different data so now the whole
corpus gets reduced to sentences.

Example:

BEFORE SENTENCE AFTER SENTENCE

SEGMENTATION SEGMENTATION
“You want to see the dreams with You want to see the dreams with close
eyes
close eyes and achieve them?
and achieve them?
They’ll remain dreams, look for
They’ll remain dreams, look for
AIMs and your eyes have to stay AIMs and your eyes have to stay
open for a change to be open for a change
seen.” to be seen

2 TOTKOEKNENISISAATTIOIONN

A “Token” is a term used for any word or number or special character

occurring in a sentence.

Under Tokenisation, every word, number, and special character is

considered separately and each of them is now a separate token.
 8

Corpus: A corpus can be defined as a collection of t

Example: You want to see the dreams with close eyes and achieve them?

You want to see the dreams with close

eyes and acheive them ?

4
REMOVAL OF STOPWORDS

Stopwords: Stopwords are the words that occur very

frequently in the corpus but do not add any value to it.

Examples: a, an, and, are, as, for, it, is, into, in, if, on, or, such, the, there,
to.

In this step, the tokens which are not necessary are removed
from the token list. To make it easier for the computer to focus on
meaningful terms, these words are removed.

Along with these words, a lot of times our corpus might have
special characters and/or numbers.

if you are working on a document containing email IDs, then y

 9

Example: You want to see the dreams with close eyes and
achieve them? the removed words would be
to, the, and, ?

The outcome would be:

-> You want see dreams with close ey es achieve them

5 CONVERTING TEXT TO A COMMON CASE

We convert the whole text into a similar case, preferably lower

case. This ensures that the case sensitivity of the machine does not
consider the same words as different just because of different
cases.

6 STEMMING

Stemming is a technique used to extract the base form of the

words by removing affixes from them. It is just like cutting down
the branches of a tree to its stems.
Might not be meaningful.
 10

Example:

Wor Affix Ste

es
ds m
i
heali heal
n
ng drea
g

7
LEMMATIZATION

In lemmatization, the word we get after affix removal (also

known as lemma) is a meaningful one and it takes a longer
time to execute than stemming.

Lemmatization makes sure that a lemma is a wor

Example:

Wor Affix lem

es
ds ma
i
heali heal
n
ng drea
g
 11

DIFFERENCE BETWEEN STEMMING AND LEMMAT

Stemmin lemmatization
g
The lemma word is a meaningful
The stemmed words might one.
not be meaningful. Caring ➔ Care
Caring ➔ Car

BAG OF WORD ALGORITHM

Bag of Words just creates a set of vectors containing the count of

word occurrences in the document (reviews). Bag of Words
vectors is easy to interpret.

The bag of words gives us two things:

A vocabulary of words for the corpus
The frequency of these words (number of times it has occurred in
the whole corpus).

Here calling this algorithm a “bag” of words symbolizes that th

 12

STEPS OF THE BAG OF WORDS ALGORITHM

1. Text Normalisation: Collecting data and pre-processing it

2. Create Dictionary: Making a list of all the unique
words occurring in the corpus. (Vocabulary)
3. Create document vectors: For each document in the
corpus, find out how many times the word from the unique
list of words has occurred.
4. Create document vectors for all the documents.

Example:
Step 1: Collecting data and pre-processing it.

Raw
Processed Data
Data
Document 1: [aman, and, anil, are,
Document 1: Aman and Anil
stressed ]
are stressed
Document 2: [aman, went, to, a,
Document 2: Aman went to
therapist]
a therapist
Document 3: [anil, went, to,
Document 3: Anil went to
download, a, health, chatbot]
download a health chatbot

Step 2: Create Dictionary

Dictionary in NLP means a list of all the unique words
occurring in the corpus. If some words are repeated in
different documents, they are all written just once while
creating the dictionary.
 13

aman an anil are stress we

d ed nt

downlo chatb therap

cuments, they are all written just once, while creating the d

Step 3: Create a document vector

The document Vector contains the frequency of each word of
the vocabulary in a particular document.

In the document, vector vocabulary is written in the

top row. Now, for each word in the document,
if it matches the vocabulary, put a 1 under it.
If the same word appears again, increment
the previous value by 1.
And if the word does not occur in that document,
put a 0 under it.

am a a a stress we t a therap downlo heal chatb

an n ni r ed nt o ist ad th ot
d l e 0
 14

Step 4: Creating a document vector table for all documents

am a a a stress we t a therap downlo heal chat

an n ni r ed nt o ist ad th bot
d l e
0
1 1 0 0 0 0 0 0
1 1 1 1
1 0 1 1 1 0 0 0
0 0 0 1

In this table, the header row contains the vocabulary of the corpus
and three rows correspond to three different documents.
Finally, this gives us the document vector table for our
corpus. But the tokens have still not converted to numbers.
This leads us to the final steps of our algorithm: TFIDF.

TFIDF
TFIDF stands for Term Frequency & Inverse Document Fr

1
TERM FREQUENCY

1. Term frequency is the frequency of a word in one document.

2. Term frequency can easily be found in the document vector
table

Example:
 15

am a a a stress we t a therap downlo heal chat

an n ni r ed nt o ist ad th bot
d l e
0
1 1 0 0 0 0 0 0
1 1 1 1
1 0 1 1 1 0 0 0
0 0 0 1

Here, as we can see that the frequency of each word for

each document has been recorded in the table. These numbers
are nothing but the Term Frequencies!

2
DOCUMENT FREQUENCY

Document Frequency is the number of documents in which the

word occurs irrespective of how many times it has occurred in
those documents.

am a a a stress we t a therap downlo heal chatb

an n ni r ed nt o ist ad th ot
d l e 2

We can observe from the table is:

1. Document frequency of ‘aman’, ‘anil’, ‘went’, ‘to’ and
‘a’ is 2 as they have occurred in two documents.
2. Rest of them occurred in just one document
hence the document frequency for them is one.
 16

3
INVERSE DOCUMENT FREQUENCY

In the case of inverse document frequency, we need to

put the
document frequency in the denominator
while the total number of documents is the
numerator.
am a a a stress we t a therap downlo heal chatb
an n ni r ed nt o ist ad th ot
d l e 3

FORMULA OF TFIDF

The formula of TFIDF for any word W becomes:

TFIDF(W) = TF(W) * log( IDF(W) )

We don’t need to calculate the log va

aman and a are stress we t a thera downlo heal chat

ni ed nt o pist ad th bot
l

0*l
1*log(3 1*log 1*log
0*l 0*l og
/2) (3) (3) 1*log 0*log 0*log( 0*log 0*log
1*l og og (3/
(3) (3) 3) (3) (3)
og (3/ (3/ 2)
(3/ 2) 2)
2)
1*log(3 0*log 0*log
/2) (3) (3) 0*log 1*l 1*log 0*log( 0*log 0*log
(3) 1*l 1*l og (3) 3) (3) (3)
0*l og og (3/
og (3/ (3/ 2)
(3/ 2) 2)
 17

After calculating all the values, we get:

am a anil are stress we to a therap downlo heal chat

an n ed nt ist ad th bot
d
0.1 0.4 0 0
0.1 76 77 0.477 0 0 0 0 0
76 .4
7
7 0.1 0.1
0 0 0 0.1 76 76 0.477 0 0 0
0.1 76
76
0

Finally, the words have been converted to numbers. These

numbers are the values of each document.

Here, we can see that since we have less amount of data,

words like ‘are’ and ‘and’ also have a high value. But as the
IDF value increases, the value of that word decreases.

That is, for example:

Total Number of documents: 10
Number of documents in which ‘and’ occurs: 10
Therefore, IDF(and) = 10/10 = 1

Which means: log(1) = 0. Hence, the value of ‘and’ becomes 0.

On the other hand, the number of documents in which ‘pollution’
occurs: 3 IDF(pollution) = 10/3 = 3.3333…
This means log(3.3333) = 0.522; which shows that the word
‘pollution’ has considerable value in the corpus.
 18

Important concepts to remember:

Words that occur in all the documents with high term frequencies
For a word to have a high TFIDF value, the word needs to have a
These values help the computer understand which words are to b

APPLICATIONS OF TFIDF

TFIDF is commonly used in the Natural Language Processing

domain. Some of its applications are:

1. Document Classification – Helps in classifying the

type and genre of a document.
2. Topic Modelling – It helps in predicting the topic for a corpus.
3. Information Retrieval System – To extract the important
information out of a corpus.
4. Stop word filtering – Helps in removing the unnecessary
words from a text body.