Downloaded 12 times
More Related Content
Similar to Deep learning lecture - part 1 (basics, CNN)
Deep learning lecture - part 1 (basics, CNN)
- 1. Deep Learning Lecture (1) 19.10.22You Sung Min Bengio, Yoshua, Ian Goodfellow, and Aaron Courville. Deep learning. Vol. 1. MIT press, 2017.
- 2. 0. Introduction 1. Whyneural networks? 1. What is the neural network? 2. Universal approximation theorem 3. Why deep neural network? 2. How the network learns 1. Gradient descent 2. Backpropagation 3. Modern deep learning 1. Convolutional neural network 2. Recurrent neural network Contents
- 3. ๏ถExample of deeplearning model Introduction Image source : Zeiler & Fergus, 2014
- 4.
- 5. ๏ถHistory of deeplearning Introduction Backpropagation Distributed representation (1986) Deep learning (2006) LSTM (1997)Biological learning (1943) Neocognitron (1980) Perceptron (1958) Stochastic gradient descent (1960)
- 6. ๏ถHistory of deeplearning ๏ถ Size of dataset Introduction
- 7. ๏ถHistory of deeplearning ๏ถ Connections per neuron Introduction 10: GoogleNet (2014)
- 8. ๏ถHistory of deeplearning ๏ถ Number of neurons Introduction 1. Perceptron 20. GoogleNet
- 9. ๏ถStructure of perceptron(Developed in 1950s) Why neural networks? = ๐ ๐๐ ๐ ๐๐ ๐๐ โค ๐ป ๐ ๐๐ ๐ ๐๐ ๐๐ > ๐ป ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐๐ ๐๐Binary Inputs Threshold T ๐ ๐๐ ๐๐ โ ๐ป โค ๐ or ๐ ๐๐ ๐๐ โ ๐ป > ๐ ๐ = ๐ ๐๐ ๐๐ + ๐ ๐๐๐๐๐๐ ๐ = ๐(๐), where ๐ is called activation ftn. output of a single neuron ๐ = ๐( ๐ ๐๐ ๐๐ + ๐)
- 10. ๏ถMultilayer perceptron (MLP) Whyneural networks? ๐ ๐ ๐ ๐๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐๐ ๐๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐๐ ๐ = ๐( ๐ ๐๐ ๐ ๐๐ + ๐๐ ๐ ) ๐ ๐ ๐ = ๐( ๐ ๐๐ ๐ ๐๐ ๐ + ๐๐ ๐ ) ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ = ๐( ๐ ๐๐ ๐ ๐๐ ๐ + ๐๐ ๐ ) ๐ญ ๐ = ๐ ๐ ๐๐ ๐ ๐( ๐ ๐๐ ๐ ๐( ๐ ๐๐ ๐ ๐๐ + ๐๐ ๐ ) + ๐๐ ๐ ) + ๐๐ ๐ Output of a network
- 11. ๏ถUniversal approximation theorem(๋ณดํธ ๊ทผ์ฌ์ ๋ฆฌ) โ For any subset of โ ๐, any continuous function f can be approximated with a feedforward neural network that has at least a single hidden layer โ ํ๋์ ์๋์ธต์ ๊ฐ๋ ์ ๊ฒฝ๋ง์ ์์์ ์ฐ์์ธ ๋ค๋ณ์ ํจ ์๋ฅผ ์ํ๋ ์ ๋๋ก ๊ทผ์ฌ ํ ์ ์๋ค Why neural networks? ๐ญ ๐ = ๐=๐ ๐ต ๐๐ ๐ ๐พ๐ ๐ป ๐ + ๐๐ , where ฯ is โ โ โ, nonconstant, bounded , continuous function ๐ญ ๐ โ ๐ ๐ < ๐ for all ๐ โ ๐๐๐๐๐ ๐๐ โ ๐ด
- 12. ๏ถUniversal approximation theorem(๋ณดํธ ๊ทผ์ฌ์ ๋ฆฌ) โ Regardless of what function we are trying to learn, a large MLP will be able to represent that function But not guaranteed that the training algorithm is able to learn that function 1. Optimization algorithm may fail to find parameters (weight) 2. Training algorithm might choose wrong function due to overfitting (fail generalization) : There is no universal procedure to train and generalize a function (no free lunch theorem; Wolpert, 1996) Why neural networks?
- 13. ๏ถUniversal approximation theorem(๋ณดํธ ๊ทผ์ฌ์ ๋ฆฌ) โ A feed forward with a single hidden layer is sufficient to represent any function. But the layer may be large and may fail to learn and generalize correctly ๏ถ Why deep neural network? In many case, deeper model can reduce the required number of units (neuron) and the amount of generalization error Why neural networks?
- 14. ๏ถWhy deep neuralnetwork? ๏ถEffect of depth (Goodfellow et al., 2014) ๏ถ Street View House Numbers (SVHN) database Why neural networks? Number of depth Goodfellow, Ian J., et al. "Multi-digit number recognition from street view imagery using deep convolutional neural networks." arXiv preprint arXiv:1312.6082 (2013)
- 15. ๏ถWhy deep neuralnetwork? ๏ถCurse of dimensionality (โ statistical challenge) ๏ถLet dimension of data space as d ๏ถRequired number of sample to inference : n ๏ถGenerally in practical task: ๐ โซ ๐ ๐ Why neural networks? Image source : Nicolas Chapados d = 10 ๐ ๐ d = ๐๐ ๐ ๐ ๐ d = ๐๐ ๐ ๐ ๐ ๐ ๐ < ๐ ๐ โช ๐ ๐
- 16. ๏ถWhy deep neuralnetwork? ๏ถLocal constancy prior (smoothness prior) ๏ถ For ๐ as an input sample and small change of ฮต, the well-trained function ๐ should satisfy Why neural networks? ๐โ ๐ โ ๐โ ๐ + ๐
- 17. ๏ถWhy deep neuralnetwork? ๏ถLocal constancy prior (smoothness prior) ๏ถModels with local kernel at samples ๏ถ๐ถ(๐) sample is required to distinguish ๐ถ(๐) regions ๏ถDeep learning spans data into subspaces (Distributed representation) ๏ถData was generated by the composition of factors (or features), potentially at multiple levels in a hierarchy Why neural networks? Voronoi diagram (nearest-neighborhood)
- 18. ๏ถWhy deep neuralnetwork? ๏ถManifold hypothesis ๏ถManifold : a connected set of points that can be approximated well by considering only a small number of degree of freedom (or dimensions) in a higher-dimensional space Why neural networks?
- 19. ๏ถWhy deep neuralnetwork? ๏ถManifold hypothesis ๏ถReal world data(sound, image, text etc.) are highly concentrated Why neural networks? Random samples in the image space
- 20. ๏ถWhy deep neuralnetwork? ๏ถManifold hypothesis ๏ถEven though the data space is โ ๐, we donโt have to consider all the space ๏ถWe may consider only neighborhood of the observed samples along with some manifolds ๏ถA transfer may exist along the manifold ๏ถFor example, intensity change in images ๏ถ Manifolds related human face and those related with cat may different Why neural networks?
- 21. ๏ถWhy deep neuralnetwork? ๏ถManifold hypothesis Why neural networks? Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." arXiv preprint arXiv:1511.06434 (2015)
- 22. ๏ถWhy deep neuralnetwork? ๏ถ Non-linear transform by learning ๏ถLinear model: linear combination of input ๐ฟ โ Linear model with non-linear transform ๐(๐ฟ) as input ๏ถFinding an optimal ๐ ๐ฟ ๏ถPrevious: human knowledge-based transform (i.e., handcrafted features) ๏ถDeep learning: learning inside the network ๏ถ๐ = ๐ ๐; ๐ฝ, ๐ = ๐(๐; ๐ฝ) ๐ป ๐ Why neural networks?
- 23. ๏ถWhy deep neuralnetwork? Why neural networks? A hidden layer ๐ = ๐ ๐; ๐ฝ, ๐ = ๐(๐; ๐ฝ) ๐ป ๐
- 24. ๏ถWhy deep neuralnetwork? ๏ถSummary ๏ถCurse of dimensionality ๏ถLocal constancy prior ๏ถManifold hypothesis ๏ถNonlinear transform by learning ๏ถDimension of the data space can be reduced as subsets of manifold ๏ถThe number of decision regions can be spanned with the subspaces as composition of factors Why neural networks?
- 25. ๏ถLearning of thenetwork ๏ถTo approximate a function ๐โ ๏ถClassifier ๐ = ๐โ(๐), where ๐๐ โ ๐๐๐๐๐๐ ๐๐๐ ๏ถRegression ๐ = ๐โ (๐), where ๐๐ โ โ ๐ ๏ถ A network defines a mapping ๐ = ๐(๐; ๐ฝ) and learns parameters ๐ฝ which approximate the function ๐โ ๏ถDue to the non-linearity, the global optimization algorithm (such as convex optimization) is not proper to the deep learning โ Update cost function ๐ช ๏ถGradient descent ๏ถBackpropagation How the network learns
- 26. ๏ถLearning of thenetwork ๏ถGradient descent How the network learns ๐ ๐: โ โ โ ๐ ๐: โ ๐ โ โ
- 27. ๏ถLearning of thenetwork ๏ถDirectional derivative of ๐ at ๐ direction ๐ ๐๐ถ ๐ ๐ + ๐ถ๐ = ๐ ๐ป ๐๐ ๐(๐) โ min ๐ cos ๐ฝ , ๐๐๐๐๐ ๐ถ = ๐ ๏ถMoving toward negative gradient decreases ๐ How the network learns ๐ ๐โฒ = ๐ โ ๐ผ๐๐ ๐(๐) (๐ผ โถ ๐๐๐๐๐๐๐๐ ๐๐๐๐)
- 28. ๏ถLearning of thenetwork ๏ถBackpropagation How the network learns Error backpropagation path ๐ ๐ = ๐(๐) ๐ ๐ ๐ ๐ = ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ = ๐ ๐ ๐ = ๐(๐)y ๐ by chain-rule
- 29. ๏ถLearning of thenetwork ๏ถBackpropagation ๏ถFor ๐ โ โ ๐ , ๐ โ โ ๐ and ๐: โ ๐ โ โ ๐ , ๐: โ ๐ โ โ ๏ถFrom gradient descent, How the network learns ๐ ๐ ๐ ๐ = ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐ ๐๐ ๐๐๐ = ๐ ๐๐ ๐๐๐ ๐๐๐ ๐๐๐ ๐๐ ๐ = ( ๐๐ ๐๐ ) ๐ป ๐๐ ๐ ๐๐ ๐๐ : ๐ ร ๐ Jacobian matrix of ๐ ๐โฒ = ๐ โ ๐ผ( ๐๐ ๐๐ ) ๐ป ๐๐ ๐ ๐ฝโฒ = ๐ฝ โ ๐ผ( ๐๐ ๐๐ฝ ) ๐ป ๐๐ ๐
- 30. ๏ถLearning of thenetwork ๏ถUniversal approximation theorem ๏ถGradient descent & Backpropagation ๏ถPractical reason of fail ๏ถOptimization ๏ถOptimizer (SGD, AdaGrad, RMSprop, Adam, etc.) ๏ถWeight initialization ๏ถRegularization ๏ถParameter norm penalty (๐ณ ๐ , ๐ณ ๐ ) ๏ถAugmentation / Noise input (weight noise, label smoothing) ๏ถMultitask learning ๏ถParameter sharing (CNN) ๏ถEnsemble / Dropout ๏ถAdversarial training How the network learns Domain specific prior
- 31. ๏ถConvolutional neural network ๏ถConvolutionvs cross-correlation ๏ถConvolution ๏ถCross-correlation Modern deep learning ๐บ ๐, ๐ = ๐ฐ โ ๐ฒ ๐, ๐ = ๐ ๐ ๐ฐ ๐, ๐ ๐ฒ(๐ โ ๐, ๐ โ ๐) = ๐ฒ โ ๐ฐ ๐, ๐ = ๐ ๐ ๐ฐ ๐ โ ๐, ๐ โ ๐ ๐ฒ(๐, ๐) ๐บ ๐, ๐ = ๐ฐ โ ๐ฒ ๐, ๐ = ๐ ๐ ๐ฐ ๐ + ๐, ๐ + ๐ ๐ฒ(๐, ๐) Most of CNN actually uses cross-correlation not convolution
- 32. ๏ถConvolutional neural network ๏ถSignificantcharacteristics of CNN ๏ถ Sparse interaction ๏ถ Parameter sharing ๏ถ Equivariant representation ๏ถSparse interaction ๏ถ Kernel size โช input size (e.g., 128-by-128 image and 3-by-3 kernel) ๏ถ For ๐ โ ๐๐๐๐๐ and ๐ โ ๐๐๐๐๐๐, fully connected network: ๐ถ ๐ ร ๐ CNN: ๐ถ ๐ ร ๐ , ๐ฐ๐ก๐๐ซ๐ ๐ค ๐ข๐ฌ ๐ง๐ฎ๐ฆ๐๐๐ซ ๐จ๐ ๐๐จ๐ง๐ง๐๐๐ญ๐ข๐จ๐ง๐ฌ ๏ถ Practically, k has several orders of magnitude smaller than m Modern deep learning CNN fully connected network Receptive field of CNN
- 33. ๏ถConvolutional neural network ๏ถParametersharing ๏ถ Learning only a set of parameters (kernel) for every location ๏ถ Reduce the required amount of memory Modern deep learning fully connected networkCNN Calculation : 4 billion times efficient Memory storage: 178,640 for matrix multiplication Vertical edge
- 34. ๏ถConvolutional neural network ๏ถEquivariantrepresentation (translation equivariant) ๏ถ Translation in input โ translation in output Modern deep learning Location of output (feature) related to cat
- 35. ๏ถConvolutional neural network ๏ถPooling(translation invariance) ๏ถTasks that care more about whether some features exist than exactly where they are Modern deep learning
- 36. ๏ถConvolutional neural network ๏ถPriorbelief of convolution and pooling ๏ถFtn. the layer should learn contains only local interactions and is equivariant to translation ๏ถFtn. the layers learns must be invariant to small translations ๏ถC.f.) Inception module(Szegedy. 2015) Capsule network(Hinton, 2017) Modern deep learning
- 37. ๏ถConvolutional neural network ๏ถHistoricalmeaning of CNN ๏ถSince the imageNet challenge of AlexNet(2012) Modern deep learning
- 38. ๏ถConvolutional neural network ๏ถHistoricalmeaning of CNN ๏ถFirst deep network that is trained and operated well with backpropagation ๏ถReason of success is not entirely clear ๏ถEfficiency of the computation time might give chances to perform more experiments for the tuning of the implementation and hyperparameters ๏ถCNN achieved states of the arts with the data that has a clear grid-structured topology(such as image) Modern deep learning
- 39.
Editor's Notes
- #10ย A simple model to emulate a single neuron A perceptron takes binary inputs (๐_๐,๐_๐,๐_๐โฆ) and produce a single binary output (0, 1)
- #32ย By Cmglee - Own work, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=20206883
- #35ย Image source: https://www.cc.gatech.edu/~san37/post/dlhc-cnn/
- #36ย Image source: https://www.cc.gatech.edu/~san37/post/dlhc-cnn/
- #37ย Image source: https://www.cc.gatech.edu/~san37/post/dlhc-cnn/
- #38ย Image source: https://www.topbots.com/14-design-patterns-improve-convolutional-neural-network-cnn-architecture/
- #40ย 13์ธต์ ์ปจ๋ณผ๋ฃจ์ ์ ๊ฒฝ๋ง์ ๊ฐ์ ์ฐ์ถํ๊ธฐ ์ํด์ ์ฝ 300์ต ๋ฒ์ ์ฐ์ฐ์ ํ์