Lecture 2:

Natural Language Challenges

1/13/2020
Review: Course Logistics
• https://dijkstra.eecs.umich.edu/eecs498
Review: Course Introduction
• Conversational AI seeks to build software that can converse with
human users to help accomplish tasks or acquire information
• We take an utterance as input, classify its intent, extract semantic slot
information, take action with business logic, then generate a response

• Project-based course in which your team will pitch an idea for a

virtual assistant, then implement it during the semester over three
sprints, culminating in a final demonstration

• You will have access to the Clinc Conversational AI Platform for

building your project
 Review: Conversational Flow
Natural language understanding
Spoken language
utterance
(i.e., sound)
Automated Speech Intent Classification:
“Please pay Dr. transfer_money
Recognition
Leach $1000.”
(ASR)
Slot mapping:
recipient: “Dr. Leach”
amount: “$1000”

“OK, I gave Dr. Response Business Logic:

Leach $1000.” Generation
Text-to-speech Deduct $1000 from account
(TTS) “Sorry fam, you Add $1000 to recipient account
Template
don’t have
Responses
enough cash”
Review: Conversational Flow
• 2 or more interlocutors exchanging information
• What assumptions must we make to create virtual assistants?

• In Conversational AI, a human user speaks or writes an utterance that

gets processed by a natural language understanding engine
• The utterance implies an intent (what the user intends to do) and contains
slot values (useful semantic information)

• In our Gaming demo from last class, what intents and slots do you
think are relevant?
One-Slide Summary: NLP Challenges
• Natural Language Processing is the field of CS, AI, and CL concerned with
interactions in natural languages
• NLP is hard because human language is messy, ambiguous, and constantly
changing
• We rely on a lot of contextual information as humans
• Implicit references
• Implicit memory
• Colloquialism
• Rule-based NLP requires human-created criteria for extracting semantics
from an utterance
• Deep learning NLP allows data-driven models to extract semantics without
rules and language-agnostically
Conversational AI Building Blocks
• Automated Speech Recognition (ASR) is a separate step
• Google reigns supreme
• Other specialty providers (e.g., Julius, Sphinx)

• We use Natural Language Processing techniques to help understand

the semantics of the textual utterance

7
Natural Language Processing
• Wiki: Natural language processing (NLP) is a field of computer
science, artificial intelligence, and computational linguistics
concerned with the interactions between computers and human
(natural) languages.

8
What is NLP?
• “Natural” languages
• English, Mandarin, French, Swahili, Arabic, Nahuatl, ….
• NOT Java, C++, Perl, … (take compilers lol)

• Ultimate goal: Natural human-to-computer communication

• Sub-field of Artificial Intelligence, but very interdisciplinary

• Computer science, human-computer interaction (HCI), linguistics, cognitive
psychology, speech signal processing (EE), …

9
Applications: Text Classification

www.wired.com

10
Applications: Machine Translation

11
Applications: Sentiment/Opinion

12
Applications: Sentiment/Opinion

13
Applications: Information Extraction

14
Applications: Text Generation SciGen: A tool for automatically
generating scientific papers

15
Applications: Text Generation

16
 NLP in Conversational AI
• NLP is the major workhorse of a conversational AI
system

Natural language understanding

• In this course, you will apply NLP solutions as
part of a conversational AI Intent Classification:
transfer_money

Slot mapping:
recipient: “Dr. Leach”
amount: “$1000”

17
Challenges in NLP
• NLP is hard
• Human languages are messy, ambiguous, and ever-changing

• What challenges get in the way of understanding and responding to

natural language?
• Implicit references
• Ambiguous references / semantics
• Implicit memory
• Imprecise rules
• Myriad languages
• Scale
Challenges in NLP

Christopher Robin is alive and well. He is the same

person that you read about in the book, Winnie the Pooh.
As a boy, Chris lived in a pretty home called Cotchfield
Farm. When Chris was three years old, his father wrote
a poem about him. The poem was printed in a magazine
for others to read. Mr. Robin then wrote a book

• Who wrote Winnie the Pooh?

• Where did Chris live?

19
Challenges: Ambiguity

20
Challenges: Ambiguity
• Word sense / meaning ambiguity

21
Challenges: Ambiguity

Credit: Mark Liberman, http://languagelog.ldc.upenn.edu/nll/?p=17711

22
Challenges: Ambiguity

23
Challenges: Ambiguity
• Ambiguous headlines:
• Include your children when baking cookies
• Local High School Dropouts Cut in Half
• Hospitals are Sued by 7 Foot Doctors
• Iraqi Head Seeks Arms

• Safety Experts Say School Bus Passengers Should Be Belted

• Teacher Strikes Idle Kids

24
Challenges: Implicit context
Challenges: Language is dynamic

26
Challenges: Language is Compositional

Carefully
Slide

27
Challenges: Language is Compositional
小心: 地滑:
Carefully Slide
Careful Landslip
Take Wet Floor
Care Smooth
Caution

28
Challenges: Scale
• Examples:
• Bible (King James version): ~700K
• Penn Tree bank ~1M from Wall street journal
• Newswire collection: 500M+
• Wikipedia: 2.9 billion word (English)
• Web: several billions of words

29
Challenges: Summary
• Natural language is replete with messy issues!

30
NLP Terminology
• Morphology: What is a word?
• 奧林匹克運動會（希臘語：Ολυμπιακοί Αγώνες，簡稱奧運會或奧運）是國際奧林匹克委員會主
辦的包含多種體育運動項目的國際性運動會，每四年舉行一次。
• ‫“ = ﻛﺑﯾوﺗﮭﺎ‬to her houses”

• Lexicography: What does each word mean?

• He plays bass guitar.
• That bass was delicious!

• Syntax: How do the words relate to each other?

• The dog bit the man. ≠ The man bit the dog.
• But in Russian: человек собаку съел = человек съел собаку

31
NLP Terminology
• Semantics: How can we infer meaning from sentences?
• I saw the man on the hill with the telescope.

• The ipod is so small! 

• The monitor is so small! 

• Discourse: How about across many sentences?

• President Bush met with President-Elect Obama today at the White House.
He welcomed him, and showed him around.

• Who is “he”? Who is “him”? How would a computer figure that out?

32
Classic NLP Pipeline
Digital Signal Processing meets NLP

Examples from Prof. Julia Hirschberg’s slides

Spoken Language Processing
• Speech Recognition
• Automatic dictation, assistance for blind people, automatic
411, …

• Related things we study…

• How does intonation affect semantic meaning?
• Detecting uncertainty and emotions
• Detecting deception!

• Why is this hard?

• Each speaker has a different voice (male vs female, child versus
older person)
• Many different accents (Scottish, American, non-native
speakers) and ways of speaking
• Conversation: turn taking, interruptions, …
Examples from Prof. Julia Hirschberg’s slides
Spoken Language Processing
• Text-to-Speech / Spoken dialog systems
• Call response centers, tutoring systems, …

• Related things we study…

• Making computer voices sound more human
• Making computer speech acts more human-like
Part of speech tagging

Tag Meaning English Examples

ADJ adjective new, good, high, special, big, local
ADP adposition on, of, at, with, by, into, under
ADV adverb really, already, still, early, now
CONJ conjunction and, or, but, if, while, although
DET determiner, article the, a, some, most, every, no, which
NOUN noun year, home, costs, time, Africa
NUM numeral twenty-four, fourth, 1991, 14:24
PRT particle at, on, out, over per, that, up, with
PRON pronoun he, their, her, its, my, I, us
VERB verb is, say, told, given, playing, would
. punctuation marks .,;!
X other ersatz, esprit, dunno, gr8, univeristy

37
Syntactic (Constituency) parsing

38
Syntactic structure => meaning

39
Dependency Parsing

40
Semantic analysis
• Word sense disambiguation
• Semantic role labeling

Credit: Ivan Titov

41
Q: [Chris] = [Mr. Robin] ?
Christopher Robin is alive and well. He is the
same person that you read about in the book,
Winnie the Pooh. As a boy, Chris lived in a
pretty home called Cotchfield Farm. When
Chris was three years old, his father wrote a
poem about him. The poem was printed in a
magazine for others to read. Mr. Robin then
wrote a book
Slide modified from Dan Roth

42
Co-reference Resolution

Christopher Robin is alive and well. He is the

same person that you read about in the book,
Winnie the Pooh. As a boy, Chris lived in a
pretty home called Cotchfield Farm. When
Chris was three years old, his father wrote a
poem about him. The poem was printed in a
magazine for others to read. Mr. Robin then
wrote a book

43
So what about ML and Deep Learning?

• Conversational AI uses Machine Learning (ML) and Deep Learning

(DL) to complete many high-level NLP tasks
Machine Learning Basics
Machine learning is a field of computer science that gives computers the ability to
learn without being explicitly programmed

Machine Learning
Labeled Data algorithm

Training
Prediction

Labeled / Unlabled Learned model Prediction

Data

Methods that can learn from and make predictions on data

Types of Learning
Supervised: Learning with a labeled training set
Example: email classification with already labeled emails

Unsupervised: Discover patterns in unlabeled data

Example: cluster similar documents based on text

Reinforcement learning: learn to act based on feedback/reward

Example: learn to play Go, reward: win or lose

class A

class A

Classification Regression Clustering

Anomaly Detection
Sequence labeling
…
ML vs Deep Learning
Most machine learning methods work well because of human-designed
representations and input features
ML becomes just optimizing weights to best make a final prediction
What is Deep Learning?
A machine learning subfield of learning representations of data. Exceptional effective
at learning patterns.
Deep learning algorithms attempt to learn (multiple levels of) representation by using
a hierarchy of multiple layers
If you provide the system tons of information, it begins to understand it and respond
in useful ways.
Why is DL useful?
o Manually designed features are often over-specified, incomplete and take a long
time to design and validate
o Learned Features are easy to adapt, fast to learn
o Deep learning provides a very flexible, (almost?) universal, learnable framework for
representing world, visual and linguistic information.
o Can learn both unsupervised and supervised
o Effective end-to-end joint system learning
o Utilize large amounts of training data

In ~2010 DL started outperforming other

ML techniques
first in speech and vision, then NLP
State of the art for…

Several big improvements in recent years in NLP

 Machine Translation
 Sentiment Analysis Leverage different levels of representation
 Dialogue Agents o words & characters
 Question Answering o syntax & semantics
 Text Classification …