TensorFlow/MNIST - Revision history

Admin at 01:44, 28 October 2017

2017-10-28T01:44:11Z

Admin: /* MNIST Convolutional Neural Network */

2017-10-28T01:34:43Z

MNIST Convolutional Neural Network

← Older revision		Revision as of 01:34, 28 October 2017
Line 1:		Line 1:
	=MNIST Convolutional ~~Neural~~ Network=		=Complicated MNIST Convolutional Network=

	Concept: Simple, end-to-end, LeNet-5-like convolutional MNIST model example. Meant as a tutorial for simple convolutional models.		Concept: Simple, end-to-end, LeNet-5-like convolutional MNIST model example. Meant as a tutorial for simple convolutional models.

Admin: /* Flags */

2017-10-28T01:22:20Z

Flags

← Older revision		Revision as of 01:22, 28 October 2017
Line 534:		Line 534:
	print('Test error: %.1f%%' % test_error)		print('Test error: %.1f%%' % test_error)
	</pre>		</pre>

			=Why Keras=

			Even this simple "hello world" of machine learning ends up being a total mess in TensorFlow. This is not a great example - there's way too much going on, and it's really difficult to extend the example to other cases.

			Now you can see why people use Keras.

	=Flags=		=Flags=

Admin: /* Evaluation Model */

2017-10-28T01:20:54Z

Evaluation Model

@@ Line 411: / Line 411: @@
 </pre>
-===Evaluation Model===
+Lastly, there is the prediction function, which finishes the last layer of the neural network with a softmax function:
 <pre>
    # Predictions for the current training minibatch.
    train_prediction = tf.nn.softmax(logits)
    # Predictions for the test and validation, which we'll compute less often.
    eval_prediction = tf.nn.softmax(model(eval_data))
 </pre>

Admin: /* Training Model */

2017-10-28T01:02:02Z

Training Model

← Older revision		Revision as of 01:02, 28 October 2017
Line 358:		Line 358:

	===Training Model===		===Training Model===

			The training procedure starts by creating a model (calling the model() function):

	<pre>		<pre>
	# Training computation: logits + cross-entropy loss.		# Training computation: logits + cross-entropy loss.
	logits = model(train_data_node, True)		logits = model(train_data_node, True)
			</pre>

			The loss function is defined as a function that seeks to minimize the cross-entropy:

			<pre>
	loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(		loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
	labels=train_labels_node, logits=logits))		labels=train_labels_node, logits=logits))
			</pre>

			L2 regulizers are used to ensure we do not get spurious fits of constants (i.e., it will minimize the number of constants needed to fit data):

			<prE>
	# L2 regularization for the fully connected parameters.		# L2 regularization for the fully connected parameters.
	regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +		regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +
Line 370:		Line 381:
	# Add the regularization term to the loss.		# Add the regularization term to the loss.
	loss += 5e-4 * regularizers		loss += 5e-4 * regularizers
			</pre>

			The L2 loss function is part of the neural network (nn) module in TensorFlow.

			Link to nn module documentation: https://www.tensorflow.org/versions/master/api_docs/python/tf/nn

			Last, the training schedule is set: an exponential decay schedule is used, wherein each round/batch will reduce the learning rate.

			<pre>
	# Optimizer: set up a variable that's incremented once per batch and		# Optimizer: set up a variable that's incremented once per batch and
	# controls the learning rate decay.		# controls the learning rate decay.
Line 381:		Line 400:
	0.95, # Decay rate.		0.95, # Decay rate.
	staircase=True)		staircase=True)
			</pre>

			The optimizer object is set, and it works by minimizing the loss function (the mean of the cross entropy function):

			<pre>
	# Use simple momentum for the optimization.		# Use simple momentum for the optimization.
	optimizer = tf.train.MomentumOptimizer(learning_rate,		optimizer = tf.train.MomentumOptimizer(learning_rate,

Admin: /* Evaluation Model */

2017-10-28T00:53:05Z

Evaluation Model

← Older revision		Revision as of 00:53, 28 October 2017
Line 388:		Line 388:

	===Evaluation Model===		===Evaluation Model===

			<pre>
			# Predictions for the current training minibatch.
			train_prediction = tf.nn.softmax(logits)

			# Predictions for the test and validation, which we'll compute less often.
			eval_prediction = tf.nn.softmax(model(eval_data))
			</pre>

	=Flags=		=Flags=

Admin: /* Training Model */

2017-10-28T00:52:44Z

Training Model

← Older revision		Revision as of 00:52, 28 October 2017
Line 358:		Line 358:

	===Training Model===		===Training Model===

			<pre>
			# Training computation: logits + cross-entropy loss.
			logits = model(train_data_node, True)
			loss = tf.reduce_mean(tf.nn.sparse_softmax_cross_entropy_with_logits(
			labels=train_labels_node, logits=logits))

			# L2 regularization for the fully connected parameters.
			regularizers = (tf.nn.l2_loss(fc1_weights) + tf.nn.l2_loss(fc1_biases) +
			tf.nn.l2_loss(fc2_weights) + tf.nn.l2_loss(fc2_biases))
			# Add the regularization term to the loss.
			loss += 5e-4 * regularizers

			# Optimizer: set up a variable that's incremented once per batch and
			# controls the learning rate decay.
			batch = tf.Variable(0, dtype=data_type())
			# Decay once per epoch, using an exponential schedule starting at 0.01.
			learning_rate = tf.train.exponential_decay(
			0.01, # Base learning rate.
			batch * BATCH_SIZE, # Current index into the dataset.
			train_size, # Decay step.
			0.95, # Decay rate.
			staircase=True)
			# Use simple momentum for the optimization.
			optimizer = tf.train.MomentumOptimizer(learning_rate,
			0.9).minimize(loss,
			global_step=batch)
			</pre>

	===Evaluation Model===		===Evaluation Model===

Admin: /* Neural Network Layers */

2017-10-28T00:52:12Z

Neural Network Layers

← Older revision		Revision as of 00:52, 28 October 2017
Line 349:		Line 349:

	return tf.matmul(hidden, fc2_weights) + fc2_biases		return tf.matmul(hidden, fc2_weights) + fc2_biases
	<pre>		</pre>

	===Why Separate Variable Definitions and Model Creation===		===Why Separate Variable Definitions and Model Creation===

Admin: /* Variables for Fully Connected Layers */

2017-10-28T00:51:55Z

Variables for Fully Connected Layers

@@ Line 268: / Line 268: @@
 .1, shape=[NUM_LABELS], dtype=data_type()))
 </pre>
 =Flags=

Admin: /* Convolutional Layers */

2017-10-27T23:41:27Z

Convolutional Layers

← Older revision		Revision as of 23:41, 27 October 2017
Line 210:		Line 210:
	Next, we create a set of tensors that will hold the weights and biases of the convolutional neural network layers. (When we use high-level neural net APIs like [[Keras]], this is all taken care of for us, but here we do it all manually.)		Next, we create a set of tensors that will hold the weights and biases of the convolutional neural network layers. (When we use high-level neural net APIs like [[Keras]], this is all taken care of for us, but here we do it all manually.)

	====Convolutional Layers====		====Variables for Convolutional Layers====

	First, convolutional layer 1: in this layer we are defining a 5 x 5 filter - meaning the 28 x 28 pixel images in the NIST data set will be convoluted down to a 5 x 5 image set. But the number of feature maps is 32, meaning we're going to convolute the 28 x 28 pixel images down into 32 different 5 x 5 pixel images. The idea is that each of those feature maps will pick out a particular feature about the image topology that is important, and "represent" it, or "allow" that signal to pass through the first layer of the convolutional neural network.		First, convolutional layer 1: in this layer we are defining a 5 x 5 filter - meaning the 28 x 28 pixel images in the NIST data set will be convoluted down to a 5 x 5 image set. But the number of feature maps is 32, meaning we're going to convolute the 28 x 28 pixel images down into 32 different 5 x 5 pixel images. The idea is that each of those feature maps will pick out a particular feature about the image topology that is important, and "represent" it, or "allow" that signal to pass through the first layer of the convolutional neural network.