Keras

Neural network package in Python.

See https://github.com/charlesreid1/circe - specifically the examples using the NIST handwriting digit classification data set. This shows how to utilize Keras to train a neural network to perform dimensionality reduction, and further explores the manifold that the neural network identified for the different digits to better understand the neural network model.

Errors

Notes on errors with Keras

Binary Categorization

Accuracy stays at exactly 50% and does not change

Was running into the issue with Keras that, when running a binary categorization model, I was seeing predictions of exactly 50%, and nothing was changing at all.

This turned out to be due to a couple of reasons:

• Poor choice of activation function
• Poor choice of optimizer

Check out the first example here: https://keras.io/getting-started/sequential-model-guide/#training

This gives an example of a simple binary categorization network:

# For a single-input model with 2 classes (binary classification):

model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Generate dummy data
import numpy as np
data = np.random.random((1000, 100))
labels = np.random.randint(2, size=(1000, 1))

# Train the model, iterating on the data in batches of 32 samples
model.fit(data, labels, epochs=10, batch_size=32)

note a few things being done here:

• Optimizer being used is "rmsprop"
• Activation function of last layer, Dense(1), is sigmoid
• The last layer is a dense layer with a single neuron, which (when on) represents yes, and (when off) represents no

What I was doing wrong:

• I had set the activation function of the last Dense layer as "softmax", which meant that I was ALWAYS predicting "yes" (everything was always rounded up). Because my training set had a 50/50 mix of yes/no cases, I was getting predictions of exactly 50% because I was always guessing "yes".
• (Even earlier) I had initially been using TWO Dense neurons, e.g., Dense(2), to get yes/no. This is not correct!
• I was trying to use an SGD optimizer... something fancy-pants that I should not have been using