Download as PDF, PPTX
More Related Content
![Big Data [sorry] & Data Science: What Does a Data Scientist Do?](https://cdn.slidesharecdn.com/ss_thumbnails/dslatcloudmsevent20130125-130126065651-phpapp01-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Hands-on Deep Learning in Python
Hands-on Deep Learning in Python
- 1. Hands-on Deep Learning inPython Imry Kissos Deep Learning Meetup TLV August 2015
- 2. Outline ● Problem Definition ●Training a DNN ● Improving the DNN ● Open Source Packages ● Summary 2
- 3.
- 4. Tutorial ● Goal: Detectfacial landmarks on (normal) face images ● Data set provided by Dr. Yoshua Bengio ● Tutorial code available: https://github.com/dnouri/kfkd-tutorial/blob/master/kfkd.py 4
- 5. Flow 5 Predict Points on TestSet Train Model General Train Model “Nose Tip” Train Model “Mouth Corners”
- 6. Flow 6 Train Images Train Points FitTrained Net
- 7.
- 8. Python Deep LearningFramework nolearn - Wrapper to Lasagne Lasagne - Theano extension for Deep Learning Theano - Define, optimize, and mathematical expressions Efficient Cuda GPU for DNN 8 Low Level High Level HW Supports: GPU & CPU OS: Linux, OS X, Windows
- 9. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN 9
- 10. Training a DeepNeural Network 1. Data Analysis a. Exploration + Validation b. Pre-Processing c. Batch and Split 2. Architecture Engineering 3. Optimization 4. Training the DNN 10
- 11. Data Exploration +Validation Data: ● 7K gray-scale images of detected faces ● 96x96 pixels per image ● 15 landmarks per image (?) Data validation: ● Some Landmarks are missing 11 1
- 12.
- 13. Batch - - t -train batch - validation batch - - test batch ⇐One Epoch’s data 13train/valid/test splits are constant
- 14. Train / ValidationSplit 14 Classification - Train/Validation preserve classes proportion
- 15. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering a. Layers Definition b. Layers Implementation 3. Optimization 4. Training 15
- 16. Architecture 16 X Y Conv PoolDense Output
- 17.
- 18.
- 19.
- 20.
- 21.
- 22. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization a. Back Propagation b. Objective c. SGD d. Updates e. Convergence Tuning 4. Training the DNN 22
- 23. Back Propagation Forward Path 23 ConvDense X Y Output Points
- 24. Back Propagation Forward Path 24 XY Conv Output PointsDense X Y Training Points
- 25. Back Propagation Backward Path 25 XY Conv Dense
- 26. Back Propagation Update 26 Conv Dense ForAll Layers:
- 27.
- 28. S.G.D 28 Updates the networkafter each batch Karpathy - “Babysitting”: weights/updates ~1e3
- 29.
- 30. Adjusting Learning Rate& Momentum 30 Linear in epoch
- 31. Convergence Tuning 31 stops accordingto validation loss returns best weights
- 32. Training a DeepNeural Network 1. Data Analysis 2. Architecture Engineering 3. Optimization 4. Training the DNN a. Fit b. Fine Tune Pre-Trained c. Learning Curves 32
- 33. Fit 33 Loop over validationbatchs Forward Loop over train batchs Forward+BackProp
- 34. Fine Tune Pre-Trained fgd 34 changeoutput layer load pre-trained weight fine tune specialist
- 35. Learning Curves Loop over6 Nets: 35 Epochs
- 36. Learning Curves Analysis 36 Net1 Net 2 OverfittingConvergence Jittering EpochsEpochs RMSE RMSE
- 37. Part 1 Summary Traininga DNN: 37
- 38.
- 39.
- 40. Outline ● Problem Definition ●Motivation ● Training a DNN ● Improving the DNN ● Open Source Packages ● Summary 40
- 41. Beyond Training 1. Improvingthe DNN a. Analysis Capabilities b. Augmentation c. Forward - Backward Path d. Monitor Layers’ Training 2. Open Source Packages 3. Summary 41
- 42. Improving the DNN Verytempting: ● >1M images ● >1M parameters ● Large gap: Theory ↔ Practice ⇒Brute force experiments?! 42
- 43. Analysis Capabilities 1. Theoreticalexplanation a. Eg. dropout and augmentation decrease overfit 2. Empirical claims about a phenomena a. Eg. normalization improves convergence 3. Numerical understanding a. Eg. exploding / vanishing updates 43
- 44. Reduce Overfitting Solution: Data Augmentation 44 Net1 Net 2 Overfitting Epochs
- 45. Data Augmentation Horizontal FlipPerturbation 45 1
- 46.
- 47. Convergence Challenges 47 Need tomonitor forward + backward path EpochsEpochs RMSE Data ErrorNormalization
- 48. Forward - BackwardPath Forward Backward: Gradient w.r.t parameters 48
- 49. Monitor Layers’ Training nolearn- visualize.py 49
- 50. Monitor Layers’ Training 50 X.Glorot ,Y. Bengio, Understanding the difficulty of training deep feedforward neural networks: “Monitoring activation and gradients across layers and training iterations is a powerful investigation tool” Easy to monitor in Theano Framework
- 51. Weight Initialization matters(1) 51 Layer 1- Gradient are close to zero - vanishing gradients
- 52. Weight Initialization matters(2) 52 Network returns close to zero values for all inputs
- 53. Monitoring Activation plateaus sometimesseen when training neural networks 53 For most epochs the network returns close to zero output for all inputs Objective plateaus sometimes can be explained by saturation
- 54. Max of Weightsof Conv1: Max of Updates of Conv1: 54http://cs231n.github.io/neural-networks-3/#baby Monitoring weights/update ratio 3e-1 2e-1 1e-1 0 3e-3 2e-3 1e-3 0 Epoch Epoch
- 55. Beyond Training 1. Improvingthe DNN 2. Open Source Packages a. Hardware and OS b. Python Framework c. Deep Learning Open Source Packages d. Effort Estimation 3. Summary 55
- 56. Hardware and OS ●Amazon Cloud GPU: AWS Lasagne GPU Setup Spot ~ $0.0031 per GPU Instance Hour ● IBM Cloud GPU: http://www-03.ibm.com/systems/platformcomputing/products/symphony/gpuharvesting.html ● Your Linux machine GPU: pip install -r https://raw.githubusercontent.com/dnouri/kfkd- tutorial/master/requirements.txt ● Window install http://deeplearning.net/software/theano/install_windows.html#install-windows 56
- 57. Starting Tips ● SanityChecks: ○ DNN Architecture : “Overfit a tiny subset of data” Karpathy ○ Check Regularization ↗ Loss ↗ ● Use pre-trained VGG as a base line ● Start with ~3 conv layer with ~16 filter each - quickly iterate 57
- 58. Python ● Rich eco-system ●State-of-the-art ● Easy to port from prototype to production 58 Podcast : http://www.reversim.com/2015/10/277-scientific-python.html
- 59. Python Deep Learning Framework 59Keras,pylearn2, OpenDeep, Lasagne - common base
- 60. Tips from DeepLearning Packages Torch code organization Caffe’s separation configuration ↔code NeuralNet → YAML text format defining experiment’s configuration 60
- 61. Deep Learning Open SourcePackages 61 Caffe for applications Torch and Theano for research on Deep Learning itself http://fastml.com/torch-vs-theano/ Black BoxWhite Box Open source progress rapidly→ impossible to predict industry’s standard
- 62. Disruptive Effort Estimation FeatureEng Deep Learning 62Still requires algorithmic expertise
- 63. Summary ● Dove intoTraining a DNN ● Presented Analysis Capabilities ● Reviewed Open Source Packages 63
- 64. References Hinton Coursera NeuronalNetwork https://www.coursera.org/course/neuralnets Technion Deep Learning course http://moodle.technion.ac.il/course/view.php?id=4128 Oxford Deep Learning course https://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53Fu CS231n CNN for Visual Recognition http://cs231n.github.io/ Deep Learning Book http://www.iro.umontreal.ca/~bengioy/dlbook/ Montreal DL summer school http://videolectures.net/deeplearning2015_montreal/ 64
- 65.