Download as PDF, PPTX
![● Build a graph computing a neural net inference.
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
x = tf.placeholder("float", shape=[None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
Example TensorFlow fragment](https://image.slidesharecdn.com/k2jeffdean-160609173832/85/Large-Scale-Deep-Learning-with-TensorFlow-23-320.jpg)
![● Automatically add ops to calculate symbolic gradients
of variables w.r.t. loss function.
● Apply these gradients with an optimization algorithm
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
opt = tf.train.GradientDescentOptimizer(0.01)
train_op = opt.minimize(cross_entropy)
Python API for Machine Learning](https://image.slidesharecdn.com/k2jeffdean-160609173832/85/Large-Scale-Deep-Learning-with-TensorFlow-24-320.jpg)
![● Build a graph computing a neural net inference.
import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets('MNIST_data', one_hot=True)
x = tf.placeholder("float", shape=[None, 784])
W = tf.Variable(tf.zeros([784,10]))
b = tf.Variable(tf.zeros([10]))
y = tf.nn.softmax(tf.matmul(x, W) + b)
Example TensorFlow fragment](https://image.slidesharecdn.com/k2jeffdean-160609173832/75/Large-Scale-Deep-Learning-with-TensorFlow-23-2048.jpg)
![● Automatically add ops to calculate symbolic gradients
of variables w.r.t. loss function.
● Apply these gradients with an optimization algorithm
y_ = tf.placeholder(tf.float32, [None, 10])
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
opt = tf.train.GradientDescentOptimizer(0.01)
train_op = opt.minimize(cross_entropy)
Python API for Machine Learning](https://image.slidesharecdn.com/k2jeffdean-160609173832/75/Large-Scale-Deep-Learning-with-TensorFlow-24-2048.jpg)
More Related Content
![[모두의연구소] 쫄지말자딥러닝](https://cdn.slidesharecdn.com/ss_thumbnails/20160528-160529034614-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Large Scale Deep Learning with TensorFlow
![RTP_AR_Basic_Learners' Workbook_KS2 [FOR REPRODUCTION] (1).pdf](https://cdn.slidesharecdn.com/ss_thumbnails/rtparbasiclearnersworkbookks2forreproduction1-251016024943-e51a16ac-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
![Matrix and determinant URT [Autosaved].pptx](https://cdn.slidesharecdn.com/ss_thumbnails/matrixanddeterminanturtautosaved-251018190340-9e6a6deb-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Large Scale Deep Learning with TensorFlow
- 1. Large-Scale Deep Learning WithTensorFlow Jeff Dean Google Brain team g.co/brain In collaboration with many other people at Google
- 2. We can nowstore and perform computation on large datasets MapReduce
- 3. We can nowstore and perform computation on large datasets But what we really want is not just raw data, but computer systems that understand this data MapReduce
- 4. What do Imean by understanding?
- 5. What do Imean by understanding?
- 6. What do Imean by understanding?
- 7.
- 8. “cat” ● A powerfulclass of machine learning model ● Modern reincarnation of artificial neural networks ● Collection of simple, trainable mathematical functions What is Deep Learning?
- 9. “cat” ● Loosely basedon (what little) we know about the brain What is Deep Learning?
- 10. What is DeepLearning? ● Each neuron is connected to a small subset of other neurons. ● Based on what it sees, it decides what it wants to say. ● Neurons learn to cooperate to accomplish the task. Commonalities with real brains:
- 11. What is DeepLearning? Each neuron implements a relatively simple mathematical function. But the composition of 106 - 109 such functions is surprisingly powerful.
- 12.
- 13. Important Property ofNeural Networks Results get better with more data + bigger models + more computation (Better algorithms, new insights and improved techniques always help, too!)
- 14. Growing Use ofDeep Learning at Google Android Apps drug discovery Gmail Image understanding Maps Natural language understanding Photos Robotics research Speech Translation YouTube … many others ... Across many products/areas: # of directories containing model description files
- 15. Open, standard softwarefor general machine learning Great for Deep Learning in particular First released Nov 2015 Apache 2.0 license http://tensorflow.org/ and https://github.com/tensorflow/tensorflow
- 16. Strong External Adoption GitHubLaunch Nov. 2015 GitHub Launch Sep. 2013 GitHub Launch Jan. 2012 GitHub Launch Jan. 2008 50,000+ binary installs in 72 hours, 500,000+ since Nov, 2015 Most forks of any GitHub repo in 2015, despite only being available starting in Nov, 2015 (source: http://donnemartin.com/viz/pages/2015)
- 17. Motivations ● DistBelief (our1st system) was scalable and good for production deployment, but not as flexible as we wanted for research purposes ● Better understanding of problem space allowed us to make some dramatic simplifications ● Define a standard way of expressing machine learning ideas and computations ● Short circuit the MapReduce/Hadoop inefficiency
- 18.
- 19.
- 20. TensorFlow: Expressing High-LevelML Computations ● Core in C++ ○ Very low overhead Core TensorFlow Execution System CPU GPU Android iOS ...
- 21. TensorFlow: Expressing High-LevelML Computations ● Core in C++ ○ Very low overhead ● Different front ends for specifying/driving the computation ○ Python and C++ today, easy to add more Core TensorFlow Execution System CPU GPU Android iOS ...
- 22. TensorFlow: Expressing High-LevelML Computations ● Core in C++ ○ Very low overhead ● Different front ends for specifying/driving the computation ○ Python and C++ today, easy to add more Core TensorFlow Execution System CPU GPU Android iOS ... C++ front end Python front end ...
- 23. ● Build agraph computing a neural net inference. import tensorflow as tf from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets('MNIST_data', one_hot=True) x = tf.placeholder("float", shape=[None, 784]) W = tf.Variable(tf.zeros([784,10])) b = tf.Variable(tf.zeros([10])) y = tf.nn.softmax(tf.matmul(x, W) + b) Example TensorFlow fragment
- 24. ● Automatically addops to calculate symbolic gradients of variables w.r.t. loss function. ● Apply these gradients with an optimization algorithm y_ = tf.placeholder(tf.float32, [None, 10]) cross_entropy = -tf.reduce_sum(y_*tf.log(y)) opt = tf.train.GradientDescentOptimizer(0.01) train_op = opt.minimize(cross_entropy) Python API for Machine Learning
- 25. ● Launch thegraph and run the training ops in a loop init = tf.initialize_all_variables() sess = tf.Session() sess.run(init) for i in range(1000): batch_xs, batch_ys = mnist.train.next_batch(100) sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys}) Python API for Machine Learning
- 26. MatMul Add Relu biases weights examples labels Xent Graph ofNodes, also called Operations or ops. Computation is a dataflow graph
- 27. with tensors MatMul Add Relu biases weights examples labels Xent Edgesare N-dimensional arrays: Tensors Computation is a dataflow graph
- 28. with state Add Mul biases ... learningrate −=... 'Biases' is a variable −= updates biasesSome ops compute gradients Computation is a dataflow graph
- 29. GPU 0 CPU AddMul biases learning rate Assign Sub ... ... distributedComputation is a dataflow graph
- 30. GPU 0 CPU AddMul biases learning rate Assign Sub ... ... Assign Devices to Ops ● TensorFlow inserts Send/Recv Ops to transport tensors across devices ● Recv ops pull data from Send ops Send Recv
- 31. GPU 0 CPU AddMul biases learning rate Assign Sub ... ... Assign Devices to Ops ● TensorFlow inserts Send/Recv Ops to transport tensors across devices ● Recv ops pull data from Send ops Send Recv Send Recv Send Recv Send Recv
- 32. phones single machines(CPU and/or GPUs) … distributed systems of 100s of machines and/or GPU cards Automatically Runs on Variety of Platforms custom ML hardware
- 33. What are someways that deep learning is having a significant impact at Google?
- 34. “How cold is itoutside?” Deep Recurrent Neural Network Acoustic Input Text Output Reduced word errors by more than 30% Speech Recognition Google Research Blog - August 2012, August 2015
- 35. “ocean” Deep Convolutional Neural Network Your Photo AutomaticTag Search personal photos without tags. Google Photos Search Google Research Blog - June 2013
- 36.
- 37. Google Photos Search “Wow.The new Google photo search is a bit insane. I didn’t tag those… :)”
- 38. We have tonsof vision problems Image search, StreetView, Satellite Imagery, Translation, Robotics, Self-driving Cars, www.google.com/sunroof
- 39.
- 40. April 1, 2009:April Fool’s Day joke Nov 5, 2015: Launched Real Product Feb 1, 2016: >10% of mobile Inbox replies Smart Reply
- 41. Small Feed- Forward Neural Network IncomingEmail Activate Smart Reply? yes/no Smart Reply Google Research Blog - Nov 2015
- 42. Small Feed- Forward Neural Network IncomingEmail Activate Smart Reply? Deep Recurrent Neural Network Generated Replies yes/no Smart Reply Google Research Blog - Nov 2015
- 43. Combined Vision +Translation
- 44. Model: A closeup of a child holding a stuffed animal. Human: A young girl asleep on the sofa cuddling a stuffed bear. Model: A baby is asleep next to a teddy bear. Image Captions Research
- 46. Experiment Turnaround Timeand Research Productivity ● Minutes, Hours: ○ Interactive research! Instant gratification! ● 1-4 days ○ Tolerable ○ Interactivity replaced by running many experiments in parallel ● 1-4 weeks ○ High value experiments only ○ Progress stalls ● >1 month ○ Don’t even try
- 48. Image Model TrainingTime Hours 10 GPUs 50 GPUs 1 GPU
- 49. Hours 2.6 hours vs.79.3 hours (30.5X) 10 GPUs 50 GPUs 1 GPU Image Model Training Time
- 50. How Can YouGet Started with Machine Learning? Four ways, with varying complexity: (1) Use a Cloud-based API (Vision, Speech, etc.) (2) Run your own pretrained model (3) Use an existing model architecture, and retrain it or fine tune on your dataset (4) Develop your own machine learning models for new problems More flexible, but more effort required
- 51. (1) Use Cloud-basedAPIs cloud.google.com/translate cloud.google.com/speech cloud.google.com/vision cloud.google.com/text
- 52. (1) Use Cloud-basedAPIs cloud.google.com/translate cloud.google.com/speech cloud.google.com/vision cloud.google.com/text
- 53. Google Cloud VisionAPI https://cloud.google.com/vision/
- 54. (2) Using aPre-trained Image Model yourself with TensorFlow www.tensorflow.org/tutorials/image_recognition/index.html
- 55. For training yourown models (3 & 4), two choices: cloud.google.com/ml or Run open-source release on your own physical machines or virtual machines in a cloud hosting environment
- 56. (3) Training aModel on Your Own Image Data www.tensorflow.org/versions/master/how_tos/image_retraining/index.html
- 57. (4) Develop yourown machine learning models https://www.tensorflow.org/versions/master/get_started/basic_usage.html
- 58. Example: Combining Visionwith Robotics “Deep Learning for Robots: Learning from Large-Scale Interaction”, Google Research Blog, March, 2016 “Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection”, Sergey Levine, Peter Pastor, Alex Krizhevsky, & Deirdre Quillen, Arxiv, arxiv.org/abs/1603.02199 https://www.youtube.com/watch?v=iaF43Ze1oeI
- 59. What Does theFuture Hold? Deep learning usage will continue to grow and accelerate: ● Across more and more fields and problems: ○ robotics, self-driving vehicles, ... ○ health care ○ video understanding ○ dialogue systems ○ personal assistance ○ ...
- 60. Conclusions Deep neural networksare making significant strides in understanding: In speech, vision, language, search, … If you’re not considering how to use deep neural nets to solve your vision or understanding problems, you almost certainly should be
- 61. Further Reading ● Dean,et al., Large Scale Distributed Deep Networks, NIPS 2012, research.google. com/archive/large_deep_networks_nips2012.html. ● TensorFlow white paper, tensorflow.org/whitepaper2015.pdf (clickable links in bibliography) ● TensorFlow: A System for Large-Scale Machine Learning, http://arxiv.org/abs/1605.08695 tensorflow.org research.google.com/people/jeff We’re Hiring! See g.co/brain Questions?