TensorFlow Workshop
Core TF Model
Yet another dataflow system
biases Graph of Nodes, also called Operations or ops.
weights Add Relu
MatMul Xent
examples
labels
s o rs
ith ten
Yet another dataflow systemw
biases Edges are N-dimensional arrays: Tensors
weights Add Relu
MatMul Xent
examples
labels
t a t e
ith s
Yet another dataflow systemw
'Biases' is a variable Some ops compute gradients −= updates biases
biases
... Add ... Mul −=
learning rate
ut e d
is t r i b
Yet another dataflow systemd
Device A Device B
biases
... Add ... Mul −=
learning rate
Devices: Processes, Machines, GPUs, etc
What's not in the Core Model
● Anything about neural networks, machine learning, ...
● Anything about backpropagation, differentiation, ...
● Anything about gradient descent, parameter servers…
These are built by combining existing operations, or defining new operations.
Core system can be applied to other problems than machine learning.
Core TF API
API Families
Graph Construction
● Assemble a Graph of Operations.
Graph Execution
● Deploy and execute operations in a Graph.
Hello, world!
from google3.learning.brain.public import tensorflow as tf
# Create an operation.
hello = tf.Constant("Hello, world!")
# Create a session with the "local" Tensorflow runtime.
sess = tf.Session("local")
# Execute that operation and print its result.
print sess.Run(hello)
Graph Construction
Library of predefined Ops
● Constant, Variables, Math ops, etc.
Functions to add Ops for common needs
● Gradients: Add Ops to compute derivatives.
● Training methods: Add Ops to update variables (SGD, Adagrad, etc.)
All operations are added to a global Default Graph.
Slightly more advanced calls let you control the Graph more precisely.
Value
Constant Constant
Op that outputs a constant value when run. (Surprising?)
from google3.learning.brain.public.tensorflow import *
a = Constant([1.0, 2.0, 3.0, 4.0]) # float vector
b = Constant([[5, 6], [7, 8]]) # int32 2x2 matrix
import numpy as np
c = Constant(np.random.rand(2, 4, 6, 8)) # double 2x4x6x8 tensor
Value Reference
Variable Variable
State
Op that holds state that persists across calls to Run()
v = Variable(shape=[4, 3], dtype=DT_FLOAT) # float 4x3 matrix
Value Reference
Variables Variable
State
Some Ops modify the Variable state: InitVariable, Assign, AssignSub, AssignAdd.
init = Assign(v, RandomParameters(shape=v.shape))
Updates the variable value when run.
Variable
State Assign Outputs the value for convenience
Random
Parameters
Math Ops
A variety of Operations for linear algebra, convolutions, etc.
c = Constant(...) c
w = Variable(...) MatMul
w Add
b = Variable(...)
b
y = Add(MatMul(c, w), b)
Overloaded Python operators help with construction: y = MatMul(c, w) + b
Operations, plenty of them
Documentation at http://go/tensorflow-ops
● Array ops ● Neural network ops
○ Concat ○ Non-linearities (Relu, …)
○ Slice ○ Convolutions (Conv2D, …)
○ Reshape ○ Pooling (AvgPool, …)
○ ... ● ...and many more
● Math ops ○ Constants, Data flow, Control flow,
○ Linear algebra (MatMul, …) Embedding, Initialization, I/O, Legacy
○ Component-wise ops (Mul, ...) Input Layers, Logging, Random,
○ Reduction ops (Sum, …) Sparse, State, Summary, Lua, etc.
Graph Construction Helpers
● Gradients
● Optimizers
● Higher-Level APIs in core TF
● Higher-Level libraries outside core TF
Gradients
Given a loss, add Ops to compute gradients for Variables.
many ops
var0 Op
Op loss
var1 Op
Gradients
Gradients(loss, [var0, var1]) # Generate gradients
many ops
var0 Op
Op loss
var1 Op
Op
Gradients for var0 Op
many ops
Gradients for var1 Op
x
MatMul y
w
Example
Gradients for MatMul
x Transpose
MatMul gw
gy
MatMul gx
w Transpose
Optimizers
Apply gradients to Variables: SGD(var, grad, learning_rate)
var
AssignSub
grad
Mul
learning_rate
Note: learning_rate is just output of an Op, it can easily be decayed
Easily Add Optimizers
Builtin
● SGD, Adagrad, Momentum, Adam, …
Contributed
● LazyAdam, NAdam, YellowFin, ...
Putting all together to train a Neural
Net
Build a Graph by adding Operations:
● For Variables to hold the parameters of the Neural Net.
● To compute the Neural Net output: e.g. classification predictions.
● To compute a training loss: e.g. cross entropy, parameter L2 norms.
● To calculate gradients for the parameters to train.
● To apply gradients with a training function.
MNIST Example
tutorials/mnist/mnist.py
● Shows both training and evaluation
● Also shows InputLayers: Ops that
read data from files
Distributed Execution
Graph Execution
Session API
● Stubby based API to deploy a Graph in a Tensorflow runtime
● Can run any subset of the graph
● Can add Ops to an existing Graph (for interactive use in colab for example)
Training Utilities
● Checkpoint, Recovery, Summaries, Avisu, Replicas, etc.
Tensorflow Runtimes
● "local": In address space of the Python program.
● Remote: In servers typically running on Borg.
Local Runtime
Python Program
Session
create graph Runtime
create session
sess.Run() CPU
GPU
Remote Runtime
Master RunSubGraph()
Python Program CPU
()
raph Worker
create graph
ateG
s])
create session
([op
Cre
sess.Run() GetTensor()
Run
Session CPU CPU
Worker Worker
GPU GPU
Deploying Graph, Running Ops
# ...Add ops to the graph...
sess = session.Session("local") # Deploy graph
Running and fetching output
an op Fetch
# Run an Op and fetch its output.
# "values" is a numpy ndarray.
values = sess.Run(<an op output>)
Running and fetching output
an op Fetch
Transitive closure of needed ops is Run
Execution happens in parallel
Feeding input, Running, and Fetching
an op Fetch
Feed
a_val = ...a numpy ndarray...
values = sess.Run(<an op output>,
feed_input({<a output>: a_val})
Feeding input, Running, and Fetching
an op Fetch
Feed
Only the required Ops are run.
Higher-Level Core TF API
Training Utilities
Training program typically runs multiple threads
● Execute the training op in a loop.
● Checkpoint every so often.
● Gather summaries for the Visualizer.
● Other, eg. monitors Nans, costs, etc.
Training Coordinator, Training Threads
Helper objects to help multithreaded training
● Thread classes to execute training op, summaries, etc
● Coordinator to start/stop them together, manage summaries
Makes it easy to train single or multiple replicas
Example:
learning/brain/models/mnist/mnist_replicas.py
Layers are ops that create Variables
def embedding(x, vocab_size, dense_size,
name=None, reuse=None, multiplier=1.0):
"""Embed x of type int64 into dense vectors."""
with tf.variable_scope( # Use scopes like this.
name, default_name="emb", values=[x], reuse=reuse):
embedding_var = tf.get_variable(
"kernel", [vocab_size, dense_size])
return tf.gather(embedding_var, x)
Models are built from Layers
def bytenet(inputs, targets, hparams):
final_encoder = common_layers.residual_dilated_conv(
inputs, hparams.num_block_repeat, "SAME", "encoder", hparams)
shifted_targets = common_layers.shift_left(targets)
kernel = (hparams.kernel_height, hparams.kernel_width)
decoder_start = common_layers.conv_block(
tf.concat([final_encoder, shifted_targets], axis=3),
hparams.hidden_size, [((1, 1), kernel)], padding="LEFT")
return common_layers.residual_dilated_conv(
decoder_start, hparams.num_block_repeat,
"LEFT", "decoder", hparams)
Estimator
Estimator and Experiment
estimator = tf.contrib.learn.Estimator(
model_fn=model_builder(model_name, hparams=hparams),
model_dir=output_dir,
config=tf.contrib.learn.RunConfig(master=...))
experiment = tf.contrib.learn.Experiment(
estimator=estimator,
train_input_fn=f1, eval_input_fn=f2,
eval_metrics=eval_metrics, train_steps=train_steps,
eval_steps=eval_steps, train_monitors=train_monitors)
model_fn
def model_fn(features, targets, mode):
"""Creates the prediction, loss, and train ops.
Args:
features: A dictionary of tensor by feature name.
targets: A tensor representing the labels (targets).
mode: The execution mode, tf.contrib.learn.ModeKeys.
Returns:
A tuple: prediction, loss, and train_op.
input_fn
def input_fn():
"""Supplies input to our model.
This function supplies input to our model, where this input is a
function of the mode. For example, we supply different data if
we're performing training versus evaluation.
Returns:
A tuple consisting of 1) a dictionary of tensors whose keys are
the feature names, and 2) a tensor of target labels if the mode
is not INFER (and None, otherwise).
"""
High-Level Exteral Libraries:
Tensor2Tensor
Tensor2Tensor (github)
Define, train, and evaluate ML tasks and models (especially sequence tasks).
● Many datasets (WMT, MSCoco, LM1B, etc.) and models (Transformer,
ByteNet, NeuralGPU, LSTM) already built in - mix and match!
● Eminently extensible - add a new Problem, T2TModel, or Modality
● Easy distributed training, both sync and async (and with support for multiple
GPUs per machine)
● Easy hyperparameter tuning
Tensor2Tensor Organization
● data_generators/ : generators for datasets which subclass Problem
● models/ : layers and models, models must subclass T2TModel
● utils/ : utilities, t2t_model class, etc.
● google/ : internal stuff (organized in the same way)
● t2t_trainer.py: main binary called to train:
t2t-trainer
--data_dir=$DATA_DIR --problems=$PROBLEM --model=$MODEL \
--hparams_set=$HPARAMS --output_dir=$TRAIN_DIR
Tensor2Tensor Models
@registry.register_model
class ByteNet(t2t_model.T2TModel):
def model_fn_body(self, features):
return bytenet_internal(
features["inputs"], features["targets"], self._hparams)
Notes:
● T2TModels are registered in a registry, get name (byte_net)
● May implement model_fn_body, default model_fn.
Adding a Problem
@registry.register_problem("wmt_ende_tokens_8k")
class WMTEnDeTokens8k(WMTProblem):
"""Problem spec for WMT En-De translation."""
@property
def targeted_vocab_size(self):
return 2**13 # 8192
def train_generator(self, data_dir, tmp_dir, train):
yield {“inputs”: [1,2], “targets”: [3, 4]}
Let the Tensors Flow!
