Download as PDF, PPTX
Uploaded byArvind Devaraj
NLP using transformers
This document discusses neural network models for natural language processing tasks like machine translation. It describes how recurrent neural networks (RNNs) were used initially but had limitations in capturing long-term dependencies and parallelization. The encoder-decoder framework addressed some issues but still lost context. Attention mechanisms allowed focusing on relevant parts of the input and using all encoded states. Transformers replaced RNNs entirely with self-attention and encoder-decoder attention, allowing parallelization while generating a richer representation capturing word relationships. This revolutionized NLP tasks like machine translation.
More Related Content
![[Paper Reading] Attention is All You Need](https://cdn.slidesharecdn.com/ss_thumbnails/reading20181228-190111054908-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to NLP using transformers
NLP using transformers
- 1. NLP using Transformermodels How translation works
- 2. Questions to ponder Whatis Deep learning for NLP How Machine translation (NMT) works How Google translate drastically improved after 2017 Why Transformer (gave rise to BERT, GPT, XLNet)
- 3.
- 4. Neural Machine Translation Majorimprovement from the earlier SMT (Statistical MT) Transformer model introduced in 2017 revolutionized and became SOTA
- 6. Models 1. Neural Networks 2.Recurrent Neural Networks 3. Encoder-Decoder 4. Attention Models 5. Transformer
- 7. Models 1. Neural Networks 2.Recurrent Neural Networks 3. Encoder-Decoder 4. Attention Models 5. Transformer
- 8. Models with Applications 1.Neural Networks - Word representation (word2vec) 2. Recurrent Neural Networks - Language Generation 3. Encoder-Decoder - Translation 4. Attention Models - Translation, Image recognition 5. Transformer - Translation
- 9.
- 10. Neural Network Neural Networksact as a ‘black box’ that takes inputs and predicts an output.
- 11.
- 12.
- 13.
- 14. Neural Network ‘Black box’that takes inputs and predicts an output. Trained using known (input,output) , approximates the function and maps new inputs Learns the function mapping inputs to output by adjusting the internal parameters (weights)
- 15. Word2vec using neuralnetworks
- 16. Word2vec using neuralnetworks
- 17. Word2vec using neuralnetworks
- 18. From words tosentences Word embedding captures meaning of words How about sentences? Can we encode meaning of sentences You can't cram the meaning of the entire $%!!* sentence into a !!$!$ vector Sentences are variable length Need fixed length representation
- 19.
- 20. RNN RNNs are usedwhen the inputs have some state information Examples include time series and word sequences Can captures the essence of the sequence in its hidden state (context) https://towardsdatascience.com/illustrated-guide-to-recurrent-neural-networks-79e5eb8049c9
- 21. RNN - unrolledin time
- 22.
- 23. RNN The unreasonable effectivenessof neural networks by Andrej Karpathy
- 24. RNN can learnsentence representations
- 25. RNNs can predict RNNcan learn a probability function p( word | previous-words )
- 26.
- 27. RNN - Summary RNNcan encode sentence meaning Can predict next word given a sequence of words. Can be trained to learn a language model
- 28. RNN - problems ●Hard to parallelize efficiently ● Back propagation through variable length sequence ● Transmitting information through one bottleneck (final hidden state) ● longer sentences lose context Example I was born in France..I speak fluent French (There are improvements to RNN structure that retain context)
- 29. Translation - FirstAttempt Now we can have two RNNs jointly trained. The first RNN creates context The second RNN uses the final context of the first RNN. Training using Parallel corpora
- 30.
- 31.
- 32.
- 33. Encoder Decoder models theconditional probability p(y/x) of translating a source X into target Y
- 34.
- 35. Encoder - Decodersummary 2014 -Google replaced their Statistical model with NMT Due to its flexibility it is the goto framework for NLG with different models taking roles of encoder and decoder The decoder can not only be conditioned on a sequence but on arbitrary representation enabling lot of use cases (like generating caption from image)
- 36. Translation using Encoder-Decoder Issues ●Meaning is crammed ● Long term dependencies ● Word alignment ● Word interdependencies
- 37. Translation issues (longterm dependencies) The model cannot remember enough Only final Encoder state is used Each encoder state has valuable information difficult to retain >30 words. "Born in France..went to EPFL Switzerland..I speak fluent ..."
- 38. Translation issues (Word alignment )
- 39. Translation issues (wordalignment) The European Economic Area → Le zone économique européenne The European Economic Area Le 1 Zone 1 Economique 1 Europeene 1
- 41. Translation issues (Interdepencies )
- 42. Translation issues Interdependencies Crammed meaning(due to context) Word alignment Words may or may not compose ( green apple, hot dog) Word ordering is important ( I hit him on the eye yesterday)
- 43. How to retainmore meaning (Context) European 1 Economic 2 Area 3 → zone économique européenne Decoding is done using only final context. The final context doesn’t capture entire information Can we use intermediate contexts ?
- 44.
- 45. Need for Attention Attentionremoves bottleneck of Encoder-Decoder model ● Pay selective attention to relevant parts of input ● Utilize all intermediate states while decoding (Don’t encode the whole input into one vector losing information) ● Do Alignment while translating
- 46.
- 47. Pay selective attention Firstintroduced in Image Recognition
- 48. Without attention Decode usingonly final state
- 49.
- 50. Attention captures alignment Alignwhile translating
- 51. Attention summary Pay selectiveattention to words we need to translate Compute weighted sum of all hidden states (called attention weight) Use attention weights while decoding Attention also take care of alignment
- 52.
- 53. Transformers Can we getrid of RNN completely ? RNN are too sequential. Parallelization is not possible since these intermediate contexts are generated sequentially
- 54.
- 55.
- 56. Transformers There are threekinds of attention 1. within encoder 2. encoder decoder 3. within decoder
- 57. Transformers (Self attention) Anew representation Z
- 58. Transformers (Self attention) Calculatefor each word , its dependencies on others ‘it’ = 0.7 * street + .2 * animal + …..
- 59. Transformers (Self attention) Rememberthat Encoder creates input representation Transformers create a rich representation that captures interdependencies (This rich representation captures relationship between words compared to simple word embeddings) This rich representation is called Self-attention
- 60. Transformers (Self attention) Self-attentionmechanism directly models relationships between all words in a sentence, regardless of their respective position Self attention allows connections between words within a sentence. Eg. I arrived at the bank after crossing the river
- 61. Transformers Use self attentioninstead of RNNs, CNN Computation can be parallelized Ability to learn long term dependencies
- 62.
- 63.
- 64.
- 65. Two types ofattention 1. Source Target Attention 2. Self Attention
- 66. Thank you foryou attention Attention is all you need
- 67. Summary NN can encodewords RNN can encode sentences Long sentences need changes to RNN architecture Two RNNs can act as encoder and decoder (of any representation) Encoding everything into single context loses information Selectively pay attention to the inputs we need. Get rid of RNN and use only attention mechanism. Make it parallelizable Richer representation of inputs using Self-attention. Use Encoder-Decoder attention as usual for translation