#DeepLearning #SupervisedLearning #CNN
By Billy Gustave
Fasion MNIST ¶
Goal
- Classify Fashion MNIST Images using CNN
- Tensorflow with Keras
- using hyperas for hyperparameters search
Data Cleaning and Exploration ¶
Import the data from keras datasets
In [1]:
import tensorflow as tf, numpy as np
#load fashion mnist
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
print(X_train.shape)
print(y_train.shape)
print(X_test.shape)
print(y_test.shape)
(60000, 28, 28) (60000,) (10000, 28, 28) (10000,)
Train-Test-Validation split
In [2]:
# get validation set
from sklearn.model_selection import train_test_split
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=43)
print(X_train.shape)
print(y_train.shape)
print(X_val.shape)
print(y_val.shape)
print(X_test.shape)
print(y_test.shape)
(48000, 28, 28) (48000,) (12000, 28, 28) (12000,) (10000, 28, 28) (10000,)
Visualizing the data
In [3]:
import matplotlib.pyplot as plt
plt.figure(figsize=(10,10))
for i in range(10):
plt.subplot(5,5,i+1)
plt.xticks([])
plt.yticks([])
plt.imshow(X_train[i], cmap=plt.cm.binary)
plt.xlabel(y_train[i])
Reshaping: Normalizing X data and One-Hot_Encoding y data
In [4]:
#reshaping (a, b , c) -> (a, b, c, 1) -> (a, 28, 28, 1)
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], X_train.shape[2], 1)
X_val = X_val.reshape(X_val.shape[0], X_val.shape[1], X_val.shape[2], 1)
X_test = X_test.reshape(X_test.shape[0], X_test.shape[1], X_test.shape[2], 1)
# rescale
X_train = X_train.astype('float32')
X_val = X_val.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_val /= 255
X_test /= 255
import numpy as np
# one-hot-encoding y values
n_class = len(np.unique(y_train))
y_train = tf.keras.utils.to_categorical(y_train, n_class)
y_val = tf.keras.utils.to_categorical(y_val, n_class)
y_test = tf.keras.utils.to_categorical(y_test, n_class)
print(X_train.shape)
print(y_train.shape)
print(X_val.shape)
print(y_val.shape)
print(X_test.shape)
print(y_test.shape)
(48000, 28, 28, 1) (48000, 10) (12000, 28, 28, 1) (12000, 10) (10000, 28, 28, 1) (10000, 10)
Base Model Architecture selection ¶
In [5]:
# param initialization
input_shape = X_train.shape[1:]
learning_rate = 0.001
opt = tf.keras.optimizers.Adam(learning_rate=learning_rate)
batch_size = 256
epochs = 20
conv-pool-conv-pool model
In [6]:
# keras model
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(8, kernel_size=(3, 3), activation='relu', padding='same', input_shape=input_shape),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Conv2D(16, kernel_size=(5, 5), activation='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(120, activation='relu'),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.summary()
WARNING:tensorflow:From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers. Model: "sequential" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d (Conv2D) (None, 28, 28, 8) 80 _________________________________________________________________ max_pooling2d (MaxPooling2D) (None, 14, 14, 8) 0 _________________________________________________________________ conv2d_1 (Conv2D) (None, 10, 10, 16) 3216 _________________________________________________________________ max_pooling2d_1 (MaxPooling2 (None, 5, 5, 16) 0 _________________________________________________________________ flatten (Flatten) (None, 400) 0 _________________________________________________________________ dense (Dense) (None, 120) 48120 _________________________________________________________________ dense_1 (Dense) (None, 64) 7744 _________________________________________________________________ dense_2 (Dense) (None, 10) 650 ================================================================= Total params: 59,810 Trainable params: 59,810 Non-trainable params: 0 _________________________________________________________________
In [7]:
%%time
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=[X_val, y_val])
score, acc = model.evaluate(X_val, y_val)
Train on 48000 samples, validate on 12000 samples Epoch 1/20 48000/48000 [==============================] - 24s 498us/sample - loss: 0.8241 - acc: 0.7025 - val_loss: 0.5419 - val_acc: 0.7954 Epoch 2/20 48000/48000 [==============================] - 25s 513us/sample - loss: 0.4801 - acc: 0.8255 - val_loss: 0.4370 - val_acc: 0.8409 Epoch 3/20 48000/48000 [==============================] - 24s 509us/sample - loss: 0.4090 - acc: 0.8539 - val_loss: 0.4161 - val_acc: 0.8455 Epoch 4/20 48000/48000 [==============================] - 28s 592us/sample - loss: 0.3725 - acc: 0.8658 - val_loss: 0.3585 - val_acc: 0.8704 Epoch 5/20 48000/48000 [==============================] - 26s 549us/sample - loss: 0.3485 - acc: 0.8736 - val_loss: 0.3504 - val_acc: 0.8737 Epoch 6/20 48000/48000 [==============================] - 29s 598us/sample - loss: 0.3297 - acc: 0.8808 - val_loss: 0.3387 - val_acc: 0.8797 Epoch 7/20 48000/48000 [==============================] - 30s 624us/sample - loss: 0.3144 - acc: 0.8857 - val_loss: 0.3212 - val_acc: 0.8837 Epoch 8/20 48000/48000 [==============================] - 26s 536us/sample - loss: 0.2958 - acc: 0.8925 - val_loss: 0.3102 - val_acc: 0.8868 Epoch 9/20 48000/48000 [==============================] - 25s 512us/sample - loss: 0.2891 - acc: 0.8945 - val_loss: 0.2957 - val_acc: 0.8932 Epoch 10/20 48000/48000 [==============================] - 26s 537us/sample - loss: 0.2760 - acc: 0.8995 - val_loss: 0.2946 - val_acc: 0.8952 Epoch 11/20 48000/48000 [==============================] - 26s 539us/sample - loss: 0.2642 - acc: 0.9043 - val_loss: 0.2828 - val_acc: 0.8968 Epoch 12/20 48000/48000 [==============================] - 29s 597us/sample - loss: 0.2573 - acc: 0.9058 - val_loss: 0.2843 - val_acc: 0.9003 Epoch 13/20 48000/48000 [==============================] - 25s 527us/sample - loss: 0.2469 - acc: 0.9097 - val_loss: 0.2791 - val_acc: 0.8978 Epoch 14/20 48000/48000 [==============================] - 25s 528us/sample - loss: 0.2396 - acc: 0.9125 - val_loss: 0.2712 - val_acc: 0.9032 Epoch 15/20 48000/48000 [==============================] - 26s 531us/sample - loss: 0.2311 - acc: 0.9149 - val_loss: 0.2867 - val_acc: 0.8975 Epoch 16/20 48000/48000 [==============================] - 25s 530us/sample - loss: 0.2260 - acc: 0.9168 - val_loss: 0.2617 - val_acc: 0.9064 Epoch 17/20 48000/48000 [==============================] - 26s 532us/sample - loss: 0.2156 - acc: 0.9201 - val_loss: 0.2593 - val_acc: 0.9093 Epoch 18/20 48000/48000 [==============================] - 26s 536us/sample - loss: 0.2125 - acc: 0.9222 - val_loss: 0.2608 - val_acc: 0.9043 Epoch 19/20 48000/48000 [==============================] - 25s 528us/sample - loss: 0.2067 - acc: 0.9235 - val_loss: 0.2706 - val_acc: 0.9008 Epoch 20/20 48000/48000 [==============================] - 26s 534us/sample - loss: 0.1991 - acc: 0.9268 - val_loss: 0.2622 - val_acc: 0.9062 12000/12000 [==============================] - 3s 239us/sample - loss: 0.2622 - acc: 0.9062 Wall time: 8min 44s
conv-pool * 3
In [12]:
# keras model
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation='relu', input_shape=input_shape),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Conv2D(64, kernel_size=(3,3), activation='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Conv2D(128, kernel_size=(3, 3), activation='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(64, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.summary()
Model: "sequential_1" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_2 (Conv2D) (None, 26, 26, 32) 320 _________________________________________________________________ max_pooling2d_2 (MaxPooling2 (None, 13, 13, 32) 0 _________________________________________________________________ conv2d_3 (Conv2D) (None, 11, 11, 64) 18496 _________________________________________________________________ max_pooling2d_3 (MaxPooling2 (None, 5, 5, 64) 0 _________________________________________________________________ conv2d_4 (Conv2D) (None, 3, 3, 128) 73856 _________________________________________________________________ max_pooling2d_4 (MaxPooling2 (None, 1, 1, 128) 0 _________________________________________________________________ flatten_1 (Flatten) (None, 128) 0 _________________________________________________________________ dense_3 (Dense) (None, 64) 8256 _________________________________________________________________ dense_4 (Dense) (None, 10) 650 ================================================================= Total params: 101,578 Trainable params: 101,578 Non-trainable params: 0 _________________________________________________________________
In [17]:
%%time
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=[X_val, y_val])
score, acc = model.evaluate(X_val, y_val)
Train on 48000 samples, validate on 12000 samples Epoch 1/20 48000/48000 [==============================] - 81s 2ms/sample - loss: 1.3774 - acc: 0.7065 - val_loss: 0.5414 - val_acc: 0.8000 Epoch 2/20 48000/48000 [==============================] - 84s 2ms/sample - loss: 0.4950 - acc: 0.8181 - val_loss: 0.4789 - val_acc: 0.8204 Epoch 3/20 48000/48000 [==============================] - 86s 2ms/sample - loss: 0.4364 - acc: 0.8388 - val_loss: 0.4313 - val_acc: 0.8382 Epoch 4/20 48000/48000 [==============================] - 86s 2ms/sample - loss: 0.4017 - acc: 0.8511 - val_loss: 0.4302 - val_acc: 0.8386 Epoch 5/20 48000/48000 [==============================] - 87s 2ms/sample - loss: 0.3756 - acc: 0.8604 - val_loss: 0.4035 - val_acc: 0.8487 Epoch 6/20 48000/48000 [==============================] - 87s 2ms/sample - loss: 0.3564 - acc: 0.8673 - val_loss: 0.3953 - val_acc: 0.8535 Epoch 7/20 48000/48000 [==============================] - 89s 2ms/sample - loss: 0.3402 - acc: 0.8734 - val_loss: 0.4047 - val_acc: 0.8512 Epoch 8/20 48000/48000 [==============================] - 88s 2ms/sample - loss: 0.3249 - acc: 0.8783 - val_loss: 0.3819 - val_acc: 0.8627 Epoch 9/20 48000/48000 [==============================] - 88s 2ms/sample - loss: 0.3097 - acc: 0.8856 - val_loss: 0.3898 - val_acc: 0.8565 Epoch 10/20 48000/48000 [==============================] - 89s 2ms/sample - loss: 0.2990 - acc: 0.8876 - val_loss: 0.3966 - val_acc: 0.8568 Epoch 11/20 48000/48000 [==============================] - 89s 2ms/sample - loss: 0.2870 - acc: 0.8918 - val_loss: 0.3734 - val_acc: 0.8609 Epoch 12/20 48000/48000 [==============================] - 97s 2ms/sample - loss: 0.2768 - acc: 0.8953 - val_loss: 0.3843 - val_acc: 0.8645 Epoch 13/20 48000/48000 [==============================] - 93s 2ms/sample - loss: 0.2679 - acc: 0.8984 - val_loss: 0.3825 - val_acc: 0.8629 Epoch 14/20 48000/48000 [==============================] - 93s 2ms/sample - loss: 0.2622 - acc: 0.9001 - val_loss: 0.3921 - val_acc: 0.8637 Epoch 15/20 48000/48000 [==============================] - 91s 2ms/sample - loss: 0.2509 - acc: 0.9060 - val_loss: 0.3979 - val_acc: 0.8627 Epoch 16/20 48000/48000 [==============================] - 95s 2ms/sample - loss: 0.2447 - acc: 0.9084 - val_loss: 0.3886 - val_acc: 0.8673 Epoch 17/20 48000/48000 [==============================] - 96s 2ms/sample - loss: 0.2399 - acc: 0.9084 - val_loss: 0.3950 - val_acc: 0.8667 Epoch 18/20 48000/48000 [==============================] - 95s 2ms/sample - loss: 0.2355 - acc: 0.9114 - val_loss: 0.3829 - val_acc: 0.8702 Epoch 19/20 48000/48000 [==============================] - 93s 2ms/sample - loss: 0.2294 - acc: 0.9121 - val_loss: 0.4043 - val_acc: 0.8650 Epoch 20/20 48000/48000 [==============================] - 93s 2ms/sample - loss: 0.2167 - acc: 0.9174 - val_loss: 0.4105 - val_acc: 0.8657 12000/12000 [==============================] - 9s 790us/sample - loss: 0.4105 - acc: 0.8657 Wall time: 30min 11s
conv-conv-pool * 2
In [19]:
# keras model
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(16, kernel_size=(3, 3), activation='relu', input_shape=input_shape),
tf.keras.layers.Conv2D(16, kernel_size=(3,3), activation='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Conv2D(32, kernel_size=(3,3), activation='relu'),
tf.keras.layers.Conv2D(23, kernel_size=(3,3), activation='relu'),
tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=2),
tf.keras.layers.Flatten(),
tf.keras.layers.Dense(512, activation='relu'),
tf.keras.layers.Dense(10, activation='softmax')
])
model.summary()
Model: "sequential_2" _________________________________________________________________ Layer (type) Output Shape Param # ================================================================= conv2d_5 (Conv2D) (None, 26, 26, 16) 160 _________________________________________________________________ conv2d_6 (Conv2D) (None, 24, 24, 16) 2320 _________________________________________________________________ max_pooling2d_5 (MaxPooling2 (None, 12, 12, 16) 0 _________________________________________________________________ conv2d_7 (Conv2D) (None, 10, 10, 32) 4640 _________________________________________________________________ conv2d_8 (Conv2D) (None, 8, 8, 23) 6647 _________________________________________________________________ max_pooling2d_6 (MaxPooling2 (None, 4, 4, 23) 0 _________________________________________________________________ flatten_2 (Flatten) (None, 368) 0 _________________________________________________________________ dense_5 (Dense) (None, 512) 188928 _________________________________________________________________ dense_6 (Dense) (None, 10) 5130 ================================================================= Total params: 207,825 Trainable params: 207,825 Non-trainable params: 0 _________________________________________________________________
In [20]:
%%time
model.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=epochs, batch_size=batch_size, validation_data=[X_val, y_val])
score, acc = model.evaluate(X_val, y_val)
Train on 48000 samples, validate on 12000 samples Epoch 1/20 48000/48000 [==============================] - 79s 2ms/sample - loss: 1.0877 - acc: 0.7575 - val_loss: 0.4446 - val_acc: 0.8390 Epoch 2/20 48000/48000 [==============================] - 85s 2ms/sample - loss: 0.3931 - acc: 0.8555 - val_loss: 0.3790 - val_acc: 0.8607 Epoch 3/20 48000/48000 [==============================] - 85s 2ms/sample - loss: 0.3379 - acc: 0.8756 - val_loss: 0.3572 - val_acc: 0.8691 Epoch 4/20 48000/48000 [==============================] - 89s 2ms/sample - loss: 0.3064 - acc: 0.8867 - val_loss: 0.3418 - val_acc: 0.8752 Epoch 5/20 48000/48000 [==============================] - 84s 2ms/sample - loss: 0.2844 - acc: 0.8952 - val_loss: 0.3569 - val_acc: 0.8748 Epoch 6/20 48000/48000 [==============================] - 82s 2ms/sample - loss: 0.2655 - acc: 0.9013 - val_loss: 0.3309 - val_acc: 0.8817 Epoch 7/20 48000/48000 [==============================] - 83s 2ms/sample - loss: 0.2476 - acc: 0.9074 - val_loss: 0.3143 - val_acc: 0.8867 Epoch 8/20 48000/48000 [==============================] - 82s 2ms/sample - loss: 0.2325 - acc: 0.9134 - val_loss: 0.3168 - val_acc: 0.8871 Epoch 9/20 48000/48000 [==============================] - 82s 2ms/sample - loss: 0.2200 - acc: 0.9179 - val_loss: 0.3210 - val_acc: 0.8858 Epoch 10/20 48000/48000 [==============================] - 84s 2ms/sample - loss: 0.2101 - acc: 0.9220 - val_loss: 0.3243 - val_acc: 0.8848 Epoch 11/20 48000/48000 [==============================] - 100s 2ms/sample - loss: 0.1985 - acc: 0.9249 - val_loss: 0.3148 - val_acc: 0.8923 Epoch 12/20 48000/48000 [==============================] - 98s 2ms/sample - loss: 0.1903 - acc: 0.9300 - val_loss: 0.3367 - val_acc: 0.8903 Epoch 13/20 48000/48000 [==============================] - 104s 2ms/sample - loss: 0.1802 - acc: 0.9344 - val_loss: 0.3307 - val_acc: 0.8863 Epoch 14/20 48000/48000 [==============================] - 103s 2ms/sample - loss: 0.1716 - acc: 0.9365 - val_loss: 0.3290 - val_acc: 0.8909 Epoch 15/20 48000/48000 [==============================] - 107s 2ms/sample - loss: 0.1623 - acc: 0.9397 - val_loss: 0.3472 - val_acc: 0.8918 Epoch 16/20 48000/48000 [==============================] - 100s 2ms/sample - loss: 0.1531 - acc: 0.9434 - val_loss: 0.3488 - val_acc: 0.8917 Epoch 17/20 48000/48000 [==============================] - 103s 2ms/sample - loss: 0.1452 - acc: 0.9455 - val_loss: 0.3579 - val_acc: 0.8902 Epoch 18/20 48000/48000 [==============================] - 102s 2ms/sample - loss: 0.1391 - acc: 0.9472 - val_loss: 0.3673 - val_acc: 0.8873 Epoch 19/20 48000/48000 [==============================] - 105s 2ms/sample - loss: 0.1287 - acc: 0.9524 - val_loss: 0.3773 - val_acc: 0.8888 Epoch 20/20 48000/48000 [==============================] - 109s 2ms/sample - loss: 0.1185 - acc: 0.9563 - val_loss: 0.4068 - val_acc: 0.8832 12000/12000 [==============================] - 7s 544us/sample - loss: 0.4068 - acc: 0.8832 Wall time: 31min 13s
VGG (conv-conv-pool * 2) is best base model
Model hyperparameters tuning with Hyperas ¶
In [25]:
from hyperopt import Trials, STATUS_OK, tpe
from hyperas import optim
from hyperas.distributions import choice, uniform
Using TensorFlow backend.
In [31]:
def data():
import tensorflow as tf
(X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=43)
#reshaping (a, b , c) -> (a, b, c, 1) -> (a, 28, 28, 1)
X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], X_train.shape[2], 1)
X_val = X_val.reshape(X_val.shape[0], X_val.shape[1], X_val.shape[2], 1)
# rescale
X_train = X_train.astype('float32')
X_val = X_val.astype('float32')
X_train /= 255
X_val /= 255
# one-hot-encoding y values
n_class = len(np.unique(y_train))
y_train = tf.keras.utils.to_categorical(y_train, n_class)
y_val = tf.keras.utils.to_categorical(y_val, n_class)
return X_train, y_train, X_val, y_val
In [32]:
def model(X_train, y_train, X_val, y_val):
input_shape = X_train.shape[1:]
model = tf.keras.Sequential()
model_choice = {{choice(['one','two'])}}
if model_choice == 'one':
model.add(tf.keras.layers.Conv2D(16, kernel_size=3, padding='same', activation='relu', input_shape=input_shape))
model.add(tf.keras.layers.Conv2D(16, kernel_size=3, padding='same', activation='relu'))
model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
model.add(tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'))
model.add(tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
elif model_choice =='two':
model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu', input_shape=input_shape))
model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu'))
model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
model.add(tf.keras.layers.Flatten())
model.add(tf.keras.layers.Dense({{choice([256, 512,1024])}}, activation='relu'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
choiceval = {{choice(['one', 'two'])}}
if choiceval == 'two':
model.add(tf.keras.layers.Dense({{choice([256, 512,1024])}}, activation='relu'))
model.add(tf.keras.layers.BatchNormalization())
model.add(tf.keras.layers.Dropout({{uniform(0, 1)}}))
model.add(tf.keras.layers.Dense(10, activation='softmax'))
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
model.fit(X_train, y_train, epochs=15, batch_size=256, verbose=2, validation_data=[X_val, y_val])
score, acc = model.evaluate(X_val, y_val, verbose=0)
print('Validation accuracy:', acc)
return {'loss': -acc, 'status': STATUS_OK, 'model': model}
In [33]:
best_run, best_model = optim.minimize(model=model, data=data, algo=tpe.suggest, max_evals=30,
trials=Trials(), notebook_name='Fashion_Mnist_CNN_vgg')
>>> Imports:
#coding=utf-8
try:
import tensorflow, numpy as tfnp
except:
pass
try:
from sklearn.model_selection import train_test_split
except:
pass
try:
import numpy as np
except:
pass
try:
from hyperopt import Trials, STATUS_OK, tpe
except:
pass
try:
from hyperas import optim
except:
pass
try:
from hyperas.distributions import choice, uniform
except:
pass
>>> Hyperas search space:
def get_space():
return {
'model_choice': hp.choice('model_choice', ['one','two']),
'Dropout': hp.uniform('Dropout', 0, 1),
'Dropout_1': hp.uniform('Dropout_1', 0, 1),
'Dropout_2': hp.uniform('Dropout_2', 0, 1),
'Dropout_3': hp.uniform('Dropout_3', 0, 1),
'Dense': hp.choice('Dense', [256, 512,1024]),
'Dropout_4': hp.uniform('Dropout_4', 0, 1),
'choiceval': hp.choice('choiceval', ['one', 'two']),
'Dense_1': hp.choice('Dense_1', [256, 512,1024]),
'Dropout_5': hp.uniform('Dropout_5', 0, 1),
}
>>> Data
1:
2: import tensorflow as tf
3: (X_train, y_train), (X_test, y_test) = tf.keras.datasets.fashion_mnist.load_data()
4: X_train, X_val, y_train, y_val = train_test_split(X_train, y_train, test_size=0.2, random_state=43)
5: #reshaping (a, b , c) -> (a, b, c, 1) -> (a, 28, 28, 1)
6: X_train = X_train.reshape(X_train.shape[0], X_train.shape[1], X_train.shape[2], 1)
7: X_val = X_val.reshape(X_val.shape[0], X_val.shape[1], X_val.shape[2], 1)
8: # rescale
9: X_train = X_train.astype('float32')
10: X_val = X_val.astype('float32')
11: X_train /= 255
12: X_val /= 255
13: # one-hot-encoding y values
14: n_class = len(np.unique(y_train))
15: y_train = tf.keras.utils.to_categorical(y_train, n_class)
16: y_val = tf.keras.utils.to_categorical(y_val, n_class)
17:
18:
19:
>>> Resulting replaced keras model:
1: def keras_fmin_fnct(space):
2:
3: input_shape = X_train.shape[1:]
4: model = tf.keras.Sequential()
5: model_choice = space['model_choice']
6: if model_choice == 'one':
7: model.add(tf.keras.layers.Conv2D(16, kernel_size=3, padding='same', activation='relu', input_shape=input_shape))
8: model.add(tf.keras.layers.Conv2D(16, kernel_size=3, padding='same', activation='relu'))
9: model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
10: model.add(tf.keras.layers.Dropout(space['Dropout']))
11:
12: model.add(tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'))
13: model.add(tf.keras.layers.Conv2D(32, kernel_size=3, activation='relu'))
14: model.add(tf.keras.layers.BatchNormalization())
15: model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
16: model.add(tf.keras.layers.Dropout(space['Dropout_1']))
17:
18: elif model_choice =='two':
19: model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu', input_shape=input_shape))
20: model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu'))
21: model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
22: model.add(tf.keras.layers.Dropout(space['Dropout_2']))
23:
24: model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
25: model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
26: model.add(tf.keras.layers.BatchNormalization())
27: model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
28: model.add(tf.keras.layers.Dropout(space['Dropout_3']))
29:
30: model.add(tf.keras.layers.Flatten())
31: model.add(tf.keras.layers.Dense(space['Dense'], activation='relu'))
32: model.add(tf.keras.layers.BatchNormalization())
33: model.add(tf.keras.layers.Dropout(space['Dropout_4']))
34: choiceval = space['choiceval']
35: if choiceval == 'two':
36: model.add(tf.keras.layers.Dense(space['Dense_1'], activation='relu'))
37: model.add(tf.keras.layers.BatchNormalization())
38: model.add(tf.keras.layers.Dropout(space['Dropout_5']))
39: model.add(tf.keras.layers.Dense(10, activation='softmax'))
40:
41: optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
42:
43: model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
44: model.fit(X_train, y_train, epochs=15, batch_size=256, verbose=2, validation_data=[X_val, y_val])
45: score, acc = model.evaluate(X_val, y_val, verbose=0)
46: print('Validation accuracy:', acc)
47: return {'loss': -acc, 'status': STATUS_OK, 'model': model}
48:
0%| | 0/30 [00:00<?, ?it/s, best loss: ?]WARNING:tensorflow:Large dropout rate: 0.73717 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:Large dropout rate: 0.651797 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 116s - loss: 0.8684 - acc: 0.7134 - val_loss: 2.5457 - val_acc: 0.1839
Epoch 2/15
48000/48000 - 119s - loss: 0.4733 - acc: 0.8289 - val_loss: 1.0601 - val_acc: 0.6524
Epoch 3/15
48000/48000 - 124s - loss: 0.3947 - acc: 0.8572 - val_loss: 0.3418 - val_acc: 0.8738
Epoch 4/15
48000/48000 - 122s - loss: 0.3543 - acc: 0.8733 - val_loss: 0.2836 - val_acc: 0.8932
Epoch 5/15
48000/48000 - 123s - loss: 0.3243 - acc: 0.8826 - val_loss: 0.2910 - val_acc: 0.8937
Epoch 6/15
48000/48000 - 126s - loss: 0.3025 - acc: 0.8899 - val_loss: 0.2806 - val_acc: 0.8917
Epoch 7/15
48000/48000 - 125s - loss: 0.2865 - acc: 0.8963 - val_loss: 0.2468 - val_acc: 0.9084
Epoch 8/15
48000/48000 - 126s - loss: 0.2688 - acc: 0.9031 - val_loss: 0.2705 - val_acc: 0.9010
Epoch 9/15
48000/48000 - 127s - loss: 0.2589 - acc: 0.9079 - val_loss: 0.2552 - val_acc: 0.9076
Epoch 10/15
48000/48000 - 127s - loss: 0.2492 - acc: 0.9114 - val_loss: 0.2416 - val_acc: 0.9128
Epoch 11/15
48000/48000 - 126s - loss: 0.2403 - acc: 0.9132 - val_loss: 0.2711 - val_acc: 0.9040
Epoch 12/15
48000/48000 - 126s - loss: 0.2325 - acc: 0.9171 - val_loss: 0.2262 - val_acc: 0.9167
Epoch 13/15
48000/48000 - 128s - loss: 0.2208 - acc: 0.9212 - val_loss: 0.2235 - val_acc: 0.9187
Epoch 14/15
48000/48000 - 126s - loss: 0.2153 - acc: 0.9223 - val_loss: 0.2643 - val_acc: 0.9062
Epoch 15/15
48000/48000 - 128s - loss: 0.2080 - acc: 0.9254 - val_loss: 0.2406 - val_acc: 0.9127
Validation accuracy:
0.9126667
3%|█▌ | 1/30 [31:18<15:08:00, 1878.62s/it, best loss: -0.9126666784286499]WARNING:tensorflow:Large dropout rate: 0.966268 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
WARNING:tensorflow:Large dropout rate: 0.836667 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 124s - loss: 2.9716 - acc: 0.2510 - val_loss: 1.8182 - val_acc: 0.2462
Epoch 2/15
48000/48000 - 126s - loss: 1.4760 - acc: 0.4347 - val_loss: 1.3550 - val_acc: 0.5770
Epoch 3/15
48000/48000 - 123s - loss: 1.2490 - acc: 0.5127 - val_loss: 0.9363 - val_acc: 0.6590
Epoch 4/15
48000/48000 - 123s - loss: 1.1288 - acc: 0.5664 - val_loss: 0.7866 - val_acc: 0.6997
Epoch 5/15
48000/48000 - 122s - loss: 1.0274 - acc: 0.6085 - val_loss: 0.6986 - val_acc: 0.7293
Epoch 6/15
48000/48000 - 122s - loss: 0.9585 - acc: 0.6382 - val_loss: 0.6901 - val_acc: 0.7228
Epoch 7/15
48000/48000 - 121s - loss: 0.9073 - acc: 0.6583 - val_loss: 0.6204 - val_acc: 0.7519
Epoch 8/15
48000/48000 - 124s - loss: 0.8627 - acc: 0.6766 - val_loss: 0.6123 - val_acc: 0.7541
Epoch 9/15
48000/48000 - 122s - loss: 0.8315 - acc: 0.6887 - val_loss: 0.5799 - val_acc: 0.7777
Epoch 10/15
48000/48000 - 123s - loss: 0.8095 - acc: 0.6965 - val_loss: 0.5596 - val_acc: 0.7744
Epoch 11/15
48000/48000 - 122s - loss: 0.7896 - acc: 0.7066 - val_loss: 0.5412 - val_acc: 0.8013
Epoch 12/15
48000/48000 - 122s - loss: 0.7664 - acc: 0.7155 - val_loss: 0.5299 - val_acc: 0.8006
Epoch 13/15
48000/48000 - 124s - loss: 0.7503 - acc: 0.7234 - val_loss: 0.5343 - val_acc: 0.7987
Epoch 14/15
48000/48000 - 123s - loss: 0.7368 - acc: 0.7265 - val_loss: 0.5103 - val_acc: 0.8060
Epoch 15/15
48000/48000 - 123s - loss: 0.7234 - acc: 0.7320 - val_loss: 0.5054 - val_acc: 0.8125
Validation accuracy:
0.8125
7%|██▉ | 2/30 [1:02:10<14:32:54, 1870.53s/it, best loss: -0.9126666784286499]WARNING:tensorflow:Large dropout rate: 0.562382 (>0.5). In TensorFlow 2.x, dropout() uses dropout rate instead of keep_prob. Please ensure that this is intended.
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 276s - loss: 4.5155 - acc: 0.2323 - val_loss: 2.4388 - val_acc: 0.1008
Epoch 2/15
48000/48000 - 282s - loss: 1.7922 - acc: 0.3273 - val_loss: 1.6886 - val_acc: 0.3048
Epoch 3/15
48000/48000 - 271s - loss: 1.5676 - acc: 0.3728 - val_loss: 1.0738 - val_acc: 0.6632
Epoch 4/15
48000/48000 - 274s - loss: 1.4629 - acc: 0.3954 - val_loss: 0.8648 - val_acc: 0.7395
Epoch 5/15
48000/48000 - 283s - loss: 1.4122 - acc: 0.4138 - val_loss: 0.8238 - val_acc: 0.7288
Epoch 6/15
48000/48000 - 291s - loss: 1.3578 - acc: 0.4355 - val_loss: 0.7725 - val_acc: 0.7479
Epoch 7/15
48000/48000 - 314s - loss: 1.3197 - acc: 0.4417 - val_loss: 0.7178 - val_acc: 0.7592
Epoch 8/15
48000/48000 - 313s - loss: 1.3033 - acc: 0.4502 - val_loss: 0.7000 - val_acc: 0.7551
Epoch 9/15
48000/48000 - 312s - loss: 1.2720 - acc: 0.4655 - val_loss: 0.6787 - val_acc: 0.7568
Epoch 10/15
48000/48000 - 315s - loss: 1.2476 - acc: 0.4657 - val_loss: 0.6667 - val_acc: 0.7674
Epoch 11/15
48000/48000 - 325s - loss: 1.2318 - acc: 0.4731 - val_loss: 0.6563 - val_acc: 0.7741
Epoch 12/15
48000/48000 - 313s - loss: 1.2034 - acc: 0.4850 - val_loss: 0.6202 - val_acc: 0.7838
Epoch 13/15
48000/48000 - 313s - loss: 1.1872 - acc: 0.4889 - val_loss: 0.6167 - val_acc: 0.7856
Epoch 14/15
48000/48000 - 312s - loss: 1.1675 - acc: 0.4981 - val_loss: 0.6013 - val_acc: 0.7889
Epoch 15/15
48000/48000 - 319s - loss: 1.1416 - acc: 0.5109 - val_loss: 0.5667 - val_acc: 0.7946
Validation accuracy:
0.7945833
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 331s - loss: 1.8486 - acc: 0.3933 - val_loss: 3.1812 - val_acc: 0.1008
Epoch 2/15
48000/48000 - 316s - loss: 1.1775 - acc: 0.5578 - val_loss: 7.5816 - val_acc: 0.1008
Epoch 3/15
48000/48000 - 331s - loss: 1.0216 - acc: 0.6076 - val_loss: 8.8348 - val_acc: 0.1008
Epoch 4/15
48000/48000 - 329s - loss: 0.9578 - acc: 0.6290 - val_loss: 9.1659 - val_acc: 0.1008
Epoch 5/15
48000/48000 - 329s - loss: 0.9121 - acc: 0.6457 - val_loss: 10.2469 - val_acc: 0.1008
Epoch 6/15
48000/48000 - 311s - loss: 0.8914 - acc: 0.6560 - val_loss: 12.5752 - val_acc: 0.1008
Epoch 7/15
48000/48000 - 331s - loss: 0.8631 - acc: 0.6628 - val_loss: 16.0045 - val_acc: 0.1008
Epoch 8/15
48000/48000 - 327s - loss: 0.8371 - acc: 0.6759 - val_loss: 17.3240 - val_acc: 0.1008
Epoch 9/15
48000/48000 - 335s - loss: 0.8174 - acc: 0.6850 - val_loss: 19.1224 - val_acc: 0.1008
Epoch 10/15
48000/48000 - 334s - loss: 0.8101 - acc: 0.6857 - val_loss: 21.1678 - val_acc: 0.1008
Epoch 11/15
48000/48000 - 329s - loss: 0.7943 - acc: 0.6943 - val_loss: 22.7165 - val_acc: 0.1008
Epoch 12/15
48000/48000 - 329s - loss: 0.7849 - acc: 0.6924 - val_loss: 24.3014 - val_acc: 0.1008
Epoch 13/15
48000/48000 - 327s - loss: 0.7766 - acc: 0.6985 - val_loss: 25.1858 - val_acc: 0.1008
Epoch 14/15
48000/48000 - 326s - loss: 0.7695 - acc: 0.7006 - val_loss: 24.1785 - val_acc: 0.1008
Epoch 15/15
48000/48000 - 324s - loss: 0.7592 - acc: 0.7048 - val_loss: 23.5503 - val_acc: 0.1008
Validation accuracy:
0.100833334
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 323s - loss: 0.6763 - acc: 0.7675 - val_loss: 1.7766 - val_acc: 0.3904
Epoch 2/15
48000/48000 - 315s - loss: 0.4172 - acc: 0.8478 - val_loss: 0.7290 - val_acc: 0.8064
Epoch 3/15
48000/48000 - 313s - loss: 0.3532 - acc: 0.8704 - val_loss: 0.3495 - val_acc: 0.8694
Epoch 4/15
48000/48000 - 311s - loss: 0.3083 - acc: 0.8860 - val_loss: 0.2678 - val_acc: 0.9003
Epoch 5/15
48000/48000 - 310s - loss: 0.2862 - acc: 0.8952 - val_loss: 0.2495 - val_acc: 0.9064
Epoch 6/15
48000/48000 - 314s - loss: 0.2630 - acc: 0.9031 - val_loss: 0.2461 - val_acc: 0.9095
Epoch 7/15
48000/48000 - 298s - loss: 0.2492 - acc: 0.9075 - val_loss: 0.2636 - val_acc: 0.9021
Epoch 8/15
48000/48000 - 275s - loss: 0.2404 - acc: 0.9112 - val_loss: 0.2160 - val_acc: 0.9215
Epoch 9/15
48000/48000 - 272s - loss: 0.2258 - acc: 0.9159 - val_loss: 0.2021 - val_acc: 0.9254
Epoch 10/15
48000/48000 - 272s - loss: 0.2157 - acc: 0.9204 - val_loss: 0.2165 - val_acc: 0.9200
Epoch 11/15
48000/48000 - 271s - loss: 0.2104 - acc: 0.9210 - val_loss: 0.2121 - val_acc: 0.9222
Epoch 12/15
48000/48000 - 271s - loss: 0.2013 - acc: 0.9259 - val_loss: 0.1950 - val_acc: 0.9297
Epoch 13/15
48000/48000 - 271s - loss: 0.1941 - acc: 0.9280 - val_loss: 0.1895 - val_acc: 0.9300
Epoch 14/15
48000/48000 - 269s - loss: 0.1903 - acc: 0.9298 - val_loss: 0.1817 - val_acc: 0.9346
Epoch 15/15
48000/48000 - 269s - loss: 0.1876 - acc: 0.9297 - val_loss: 0.1857 - val_acc: 0.9303
Validation accuracy:
0.9303333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 126s - loss: 1.0351 - acc: 0.6487 - val_loss: 2.7456 - val_acc: 0.2049
Epoch 2/15
48000/48000 - 125s - loss: 0.5848 - acc: 0.7768 - val_loss: 1.0466 - val_acc: 0.6595
Epoch 3/15
48000/48000 - 125s - loss: 0.4950 - acc: 0.8136 - val_loss: 0.4481 - val_acc: 0.8306
Epoch 4/15
48000/48000 - 124s - loss: 0.4446 - acc: 0.8356 - val_loss: 0.3523 - val_acc: 0.8683
Epoch 5/15
48000/48000 - 125s - loss: 0.4086 - acc: 0.8486 - val_loss: 0.3209 - val_acc: 0.8852
Epoch 6/15
48000/48000 - 125s - loss: 0.3814 - acc: 0.8605 - val_loss: 0.3192 - val_acc: 0.8811
Epoch 7/15
48000/48000 - 125s - loss: 0.3602 - acc: 0.8681 - val_loss: 0.2887 - val_acc: 0.8891
Epoch 8/15
48000/48000 - 124s - loss: 0.3453 - acc: 0.8737 - val_loss: 0.2700 - val_acc: 0.8985
Epoch 9/15
48000/48000 - 125s - loss: 0.3294 - acc: 0.8790 - val_loss: 0.2776 - val_acc: 0.8998
Epoch 10/15
48000/48000 - 125s - loss: 0.3218 - acc: 0.8807 - val_loss: 0.2714 - val_acc: 0.9032
Epoch 11/15
48000/48000 - 125s - loss: 0.3122 - acc: 0.8849 - val_loss: 0.2692 - val_acc: 0.8997
Epoch 12/15
48000/48000 - 125s - loss: 0.3067 - acc: 0.8877 - val_loss: 0.2520 - val_acc: 0.9097
Epoch 13/15
48000/48000 - 148s - loss: 0.2965 - acc: 0.8910 - val_loss: 0.2489 - val_acc: 0.9109
Epoch 14/15
48000/48000 - 125s - loss: 0.2939 - acc: 0.8920 - val_loss: 0.2676 - val_acc: 0.9022
Epoch 15/15
48000/48000 - 124s - loss: 0.2861 - acc: 0.8944 - val_loss: 0.2629 - val_acc: 0.9079
Validation accuracy:
0.90791667
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 275s - loss: 0.6861 - acc: 0.7571 - val_loss: 2.4284 - val_acc: 0.1709
Epoch 2/15
48000/48000 - 272s - loss: 0.4134 - acc: 0.8480 - val_loss: 1.1657 - val_acc: 0.5813
Epoch 3/15
48000/48000 - 274s - loss: 0.3418 - acc: 0.8749 - val_loss: 0.3022 - val_acc: 0.8922
Epoch 4/15
48000/48000 - 274s - loss: 0.3060 - acc: 0.8874 - val_loss: 0.2636 - val_acc: 0.9004
Epoch 5/15
48000/48000 - 274s - loss: 0.2796 - acc: 0.8965 - val_loss: 0.2496 - val_acc: 0.9101
Epoch 6/15
48000/48000 - 275s - loss: 0.2605 - acc: 0.9037 - val_loss: 0.2311 - val_acc: 0.9174
Epoch 7/15
48000/48000 - 274s - loss: 0.2488 - acc: 0.9083 - val_loss: 0.2333 - val_acc: 0.9131
Epoch 8/15
48000/48000 - 273s - loss: 0.2372 - acc: 0.9119 - val_loss: 0.2273 - val_acc: 0.9183
Epoch 9/15
48000/48000 - 271s - loss: 0.2301 - acc: 0.9153 - val_loss: 0.2097 - val_acc: 0.9238
Epoch 10/15
48000/48000 - 273s - loss: 0.2178 - acc: 0.9210 - val_loss: 0.2047 - val_acc: 0.9247
Epoch 11/15
48000/48000 - 272s - loss: 0.2099 - acc: 0.9218 - val_loss: 0.2141 - val_acc: 0.9228
Epoch 12/15
48000/48000 - 272s - loss: 0.2025 - acc: 0.9240 - val_loss: 0.2067 - val_acc: 0.9247
Epoch 13/15
48000/48000 - 271s - loss: 0.1965 - acc: 0.9260 - val_loss: 0.2197 - val_acc: 0.9202
Epoch 14/15
48000/48000 - 273s - loss: 0.1936 - acc: 0.9276 - val_loss: 0.2235 - val_acc: 0.9218
Epoch 15/15
48000/48000 - 273s - loss: 0.1864 - acc: 0.9312 - val_loss: 0.2119 - val_acc: 0.9224
Validation accuracy:
0.9224167
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 122s - loss: 0.9787 - acc: 0.6359 - val_loss: 2.5227 - val_acc: 0.2008
Epoch 2/15
48000/48000 - 121s - loss: 0.6840 - acc: 0.7361 - val_loss: 3.3773 - val_acc: 0.2369
Epoch 3/15
48000/48000 - 121s - loss: 0.6033 - acc: 0.7658 - val_loss: 2.9111 - val_acc: 0.3610
Epoch 4/15
48000/48000 - 121s - loss: 0.5560 - acc: 0.7858 - val_loss: 2.1269 - val_acc: 0.4129
Epoch 5/15
48000/48000 - 121s - loss: 0.5212 - acc: 0.7992 - val_loss: 2.2580 - val_acc: 0.3509
Epoch 6/15
48000/48000 - 121s - loss: 0.5004 - acc: 0.8067 - val_loss: 1.9084 - val_acc: 0.4093
Epoch 7/15
48000/48000 - 121s - loss: 0.4798 - acc: 0.8169 - val_loss: 1.9196 - val_acc: 0.4105
Epoch 8/15
48000/48000 - 121s - loss: 0.4691 - acc: 0.8195 - val_loss: 2.0754 - val_acc: 0.4356
Epoch 9/15
48000/48000 - 122s - loss: 0.4540 - acc: 0.8265 - val_loss: 2.7466 - val_acc: 0.3808
Epoch 10/15
48000/48000 - 121s - loss: 0.4440 - acc: 0.8286 - val_loss: 1.8901 - val_acc: 0.4533
Epoch 11/15
48000/48000 - 120s - loss: 0.4362 - acc: 0.8336 - val_loss: 2.0641 - val_acc: 0.4406
Epoch 12/15
48000/48000 - 121s - loss: 0.4222 - acc: 0.8393 - val_loss: 1.5386 - val_acc: 0.5153
Epoch 13/15
48000/48000 - 120s - loss: 0.4164 - acc: 0.8417 - val_loss: 2.2129 - val_acc: 0.4018
Epoch 14/15
48000/48000 - 120s - loss: 0.4127 - acc: 0.8419 - val_loss: 2.4557 - val_acc: 0.4059
Epoch 15/15
48000/48000 - 120s - loss: 0.4011 - acc: 0.8478 - val_loss: 2.4829 - val_acc: 0.4268
Validation accuracy:
0.42683333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 122s - loss: 3.8339 - acc: 0.2265 - val_loss: 2.4563 - val_acc: 0.1726
Epoch 2/15
48000/48000 - 122s - loss: 1.5277 - acc: 0.4270 - val_loss: 1.6972 - val_acc: 0.2992
Epoch 3/15
48000/48000 - 120s - loss: 1.1765 - acc: 0.5221 - val_loss: 1.2495 - val_acc: 0.4348
Epoch 4/15
48000/48000 - 121s - loss: 1.0533 - acc: 0.5757 - val_loss: 0.9289 - val_acc: 0.6155
Epoch 5/15
48000/48000 - 121s - loss: 0.9833 - acc: 0.6078 - val_loss: 0.7522 - val_acc: 0.7002
Epoch 6/15
48000/48000 - 120s - loss: 0.9150 - acc: 0.6434 - val_loss: 0.6822 - val_acc: 0.7309
Epoch 7/15
48000/48000 - 121s - loss: 0.8668 - acc: 0.6687 - val_loss: 0.6522 - val_acc: 0.7563
Epoch 8/15
48000/48000 - 120s - loss: 0.8234 - acc: 0.6859 - val_loss: 0.6378 - val_acc: 0.7557
Epoch 9/15
48000/48000 - 121s - loss: 0.7979 - acc: 0.6998 - val_loss: 0.6676 - val_acc: 0.7502
Epoch 10/15
48000/48000 - 120s - loss: 0.7543 - acc: 0.7134 - val_loss: 0.5938 - val_acc: 0.7721
Epoch 11/15
48000/48000 - 121s - loss: 0.7426 - acc: 0.7216 - val_loss: 0.5862 - val_acc: 0.7764
Epoch 12/15
48000/48000 - 121s - loss: 0.7171 - acc: 0.7284 - val_loss: 0.5575 - val_acc: 0.7827
Epoch 13/15
48000/48000 - 121s - loss: 0.7033 - acc: 0.7342 - val_loss: 0.5958 - val_acc: 0.7682
Epoch 14/15
48000/48000 - 120s - loss: 0.6863 - acc: 0.7416 - val_loss: 0.5698 - val_acc: 0.7795
Epoch 15/15
48000/48000 - 121s - loss: 0.6689 - acc: 0.7464 - val_loss: 0.5631 - val_acc: 0.7820
Validation accuracy:
0.782
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 264s - loss: 0.5793 - acc: 0.7899 - val_loss: 1.9495 - val_acc: 0.2074
Epoch 2/15
48000/48000 - 263s - loss: 0.3599 - acc: 0.8691 - val_loss: 0.8444 - val_acc: 0.7510
Epoch 3/15
48000/48000 - 261s - loss: 0.3057 - acc: 0.8859 - val_loss: 0.3296 - val_acc: 0.8958
Epoch 4/15
48000/48000 - 260s - loss: 0.2755 - acc: 0.8985 - val_loss: 0.2357 - val_acc: 0.9114
Epoch 5/15
48000/48000 - 260s - loss: 0.2552 - acc: 0.9066 - val_loss: 0.2476 - val_acc: 0.9089
Epoch 6/15
48000/48000 - 258s - loss: 0.2339 - acc: 0.9152 - val_loss: 0.2355 - val_acc: 0.9113
Epoch 7/15
48000/48000 - 259s - loss: 0.2240 - acc: 0.9178 - val_loss: 0.2312 - val_acc: 0.9173
Epoch 8/15
48000/48000 - 259s - loss: 0.2154 - acc: 0.9212 - val_loss: 0.2275 - val_acc: 0.9156
Epoch 9/15
48000/48000 - 259s - loss: 0.2037 - acc: 0.9233 - val_loss: 0.1986 - val_acc: 0.9261
Epoch 10/15
48000/48000 - 258s - loss: 0.1978 - acc: 0.9249 - val_loss: 0.2349 - val_acc: 0.9181
Epoch 11/15
48000/48000 - 258s - loss: 0.1904 - acc: 0.9295 - val_loss: 0.1953 - val_acc: 0.9276
Epoch 12/15
48000/48000 - 258s - loss: 0.1829 - acc: 0.9311 - val_loss: 0.1984 - val_acc: 0.9276
Epoch 13/15
48000/48000 - 257s - loss: 0.1764 - acc: 0.9331 - val_loss: 0.2041 - val_acc: 0.9263
Epoch 14/15
48000/48000 - 257s - loss: 0.1701 - acc: 0.9368 - val_loss: 0.1924 - val_acc: 0.9316
Epoch 15/15
48000/48000 - 258s - loss: 0.1639 - acc: 0.9386 - val_loss: 0.1905 - val_acc: 0.9322
Validation accuracy:
0.9321667
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 270s - loss: 0.8280 - acc: 0.7052 - val_loss: 2.9635 - val_acc: 0.1055
Epoch 2/15
48000/48000 - 271s - loss: 0.4702 - acc: 0.8239 - val_loss: 1.2086 - val_acc: 0.5670
Epoch 3/15
48000/48000 - 272s - loss: 0.3858 - acc: 0.8580 - val_loss: 0.3710 - val_acc: 0.8575
Epoch 4/15
48000/48000 - 271s - loss: 0.3438 - acc: 0.8721 - val_loss: 0.3088 - val_acc: 0.8832
Epoch 5/15
48000/48000 - 274s - loss: 0.3155 - acc: 0.8850 - val_loss: 0.2787 - val_acc: 0.8972
Epoch 6/15
48000/48000 - 272s - loss: 0.2942 - acc: 0.8916 - val_loss: 0.2497 - val_acc: 0.9103
Epoch 7/15
48000/48000 - 273s - loss: 0.2788 - acc: 0.8974 - val_loss: 0.2494 - val_acc: 0.9085
Epoch 8/15
48000/48000 - 272s - loss: 0.2640 - acc: 0.9027 - val_loss: 0.2277 - val_acc: 0.9154
Epoch 9/15
48000/48000 - 272s - loss: 0.2540 - acc: 0.9057 - val_loss: 0.2228 - val_acc: 0.9169
Epoch 10/15
48000/48000 - 272s - loss: 0.2485 - acc: 0.9066 - val_loss: 0.2475 - val_acc: 0.9114
Epoch 11/15
48000/48000 - 274s - loss: 0.2369 - acc: 0.9128 - val_loss: 0.2364 - val_acc: 0.9146
Epoch 12/15
48000/48000 - 273s - loss: 0.2264 - acc: 0.9162 - val_loss: 0.2122 - val_acc: 0.9228
Epoch 13/15
48000/48000 - 272s - loss: 0.2265 - acc: 0.9166 - val_loss: 0.2009 - val_acc: 0.9269
Epoch 14/15
48000/48000 - 274s - loss: 0.2201 - acc: 0.9181 - val_loss: 0.2051 - val_acc: 0.9243
Epoch 15/15
48000/48000 - 271s - loss: 0.2127 - acc: 0.9209 - val_loss: 0.2085 - val_acc: 0.9233
Validation accuracy:
0.92325
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 274s - loss: 1.8528 - acc: 0.6296 - val_loss: 2.0832 - val_acc: 0.2662
Epoch 2/15
48000/48000 - 273s - loss: 0.7131 - acc: 0.7554 - val_loss: 1.3780 - val_acc: 0.5089
Epoch 3/15
48000/48000 - 273s - loss: 0.5644 - acc: 0.8005 - val_loss: 0.5801 - val_acc: 0.7972
Epoch 4/15
48000/48000 - 272s - loss: 0.5002 - acc: 0.8245 - val_loss: 0.3596 - val_acc: 0.8688
Epoch 5/15
48000/48000 - 272s - loss: 0.4560 - acc: 0.8389 - val_loss: 0.3489 - val_acc: 0.8763
Epoch 6/15
48000/48000 - 274s - loss: 0.4322 - acc: 0.8482 - val_loss: 0.3356 - val_acc: 0.8825
Epoch 7/15
48000/48000 - 275s - loss: 0.4056 - acc: 0.8587 - val_loss: 0.3077 - val_acc: 0.8903
Epoch 8/15
48000/48000 - 273s - loss: 0.3852 - acc: 0.8667 - val_loss: 0.2648 - val_acc: 0.9054
Epoch 9/15
48000/48000 - 272s - loss: 0.3630 - acc: 0.8730 - val_loss: 0.2642 - val_acc: 0.9037
Epoch 10/15
48000/48000 - 270s - loss: 0.3544 - acc: 0.8773 - val_loss: 0.2500 - val_acc: 0.9090
Epoch 11/15
48000/48000 - 270s - loss: 0.3506 - acc: 0.8798 - val_loss: 0.2510 - val_acc: 0.9096
Epoch 12/15
48000/48000 - 271s - loss: 0.3364 - acc: 0.8824 - val_loss: 0.2427 - val_acc: 0.9139
Epoch 13/15
48000/48000 - 287s - loss: 0.3311 - acc: 0.8864 - val_loss: 0.2347 - val_acc: 0.9183
Epoch 14/15
48000/48000 - 274s - loss: 0.3174 - acc: 0.8904 - val_loss: 0.2496 - val_acc: 0.9114
Epoch 15/15
48000/48000 - 275s - loss: 0.3100 - acc: 0.8940 - val_loss: 0.2362 - val_acc: 0.9162
Validation accuracy:
0.91625
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 274s - loss: 3.1406 - acc: 0.1013 - val_loss: 2.3054 - val_acc: 0.0996
Epoch 2/15
48000/48000 - 268s - loss: 2.6286 - acc: 0.1069 - val_loss: 2.3024 - val_acc: 0.0995
Epoch 3/15
48000/48000 - 267s - loss: 2.4518 - acc: 0.1102 - val_loss: 2.3014 - val_acc: 0.1037
Epoch 4/15
48000/48000 - 270s - loss: 2.3670 - acc: 0.1151 - val_loss: 2.3025 - val_acc: 0.0997
Epoch 5/15
48000/48000 - 266s - loss: 2.3330 - acc: 0.1150 - val_loss: 2.2995 - val_acc: 0.1775
Epoch 6/15
48000/48000 - 267s - loss: 2.3193 - acc: 0.1175 - val_loss: 2.2982 - val_acc: 0.1013
Epoch 7/15
48000/48000 - 264s - loss: 2.3083 - acc: 0.1192 - val_loss: 2.2935 - val_acc: 0.0997
Epoch 8/15
48000/48000 - 268s - loss: 2.2905 - acc: 0.1231 - val_loss: 2.2922 - val_acc: 0.0997
Epoch 9/15
48000/48000 - 265s - loss: 2.2550 - acc: 0.1298 - val_loss: 2.2646 - val_acc: 0.0997
Epoch 10/15
48000/48000 - 267s - loss: 2.2059 - acc: 0.1393 - val_loss: 2.2017 - val_acc: 0.0969
Epoch 11/15
48000/48000 - 266s - loss: 2.1690 - acc: 0.1529 - val_loss: 2.1391 - val_acc: 0.1243
Epoch 12/15
48000/48000 - 267s - loss: 2.1570 - acc: 0.1556 - val_loss: 2.1508 - val_acc: 0.1625
Epoch 13/15
48000/48000 - 266s - loss: 2.1467 - acc: 0.1575 - val_loss: 2.1612 - val_acc: 0.1011
Epoch 14/15
48000/48000 - 267s - loss: 2.1364 - acc: 0.1603 - val_loss: 2.1686 - val_acc: 0.1001
Epoch 15/15
48000/48000 - 265s - loss: 2.1349 - acc: 0.1599 - val_loss: 2.1609 - val_acc: 0.0998
Validation accuracy:
0.09983333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 131s - loss: 2.0598 - acc: 0.3487 - val_loss: 3.8757 - val_acc: 0.1008
Epoch 2/15
48000/48000 - 128s - loss: 1.4168 - acc: 0.4733 - val_loss: 5.7200 - val_acc: 0.1008
Epoch 3/15
48000/48000 - 128s - loss: 1.2149 - acc: 0.5413 - val_loss: 8.9026 - val_acc: 0.1008
Epoch 4/15
48000/48000 - 131s - loss: 1.0912 - acc: 0.5782 - val_loss: 15.1288 - val_acc: 0.1008
Epoch 5/15
48000/48000 - 129s - loss: 1.0227 - acc: 0.6052 - val_loss: 15.6343 - val_acc: 0.1008
Epoch 6/15
48000/48000 - 129s - loss: 0.9893 - acc: 0.6170 - val_loss: 18.1014 - val_acc: 0.1008
Epoch 7/15
48000/48000 - 129s - loss: 0.9570 - acc: 0.6286 - val_loss: 20.7870 - val_acc: 0.1008
Epoch 8/15
48000/48000 - 129s - loss: 0.9475 - acc: 0.6310 - val_loss: 20.8913 - val_acc: 0.1008
Epoch 9/15
48000/48000 - 129s - loss: 0.9247 - acc: 0.6403 - val_loss: 17.0720 - val_acc: 0.1008
Epoch 10/15
48000/48000 - 129s - loss: 0.9158 - acc: 0.6437 - val_loss: 20.0259 - val_acc: 0.1008
Epoch 11/15
48000/48000 - 130s - loss: 0.9020 - acc: 0.6480 - val_loss: 21.7241 - val_acc: 0.1008
Epoch 12/15
48000/48000 - 128s - loss: 0.8905 - acc: 0.6552 - val_loss: 21.2193 - val_acc: 0.1008
Epoch 13/15
48000/48000 - 128s - loss: 0.8857 - acc: 0.6559 - val_loss: 24.9372 - val_acc: 0.1008
Epoch 14/15
48000/48000 - 129s - loss: 0.8754 - acc: 0.6594 - val_loss: 24.2325 - val_acc: 0.1008
Epoch 15/15
48000/48000 - 128s - loss: 0.8676 - acc: 0.6614 - val_loss: 24.2183 - val_acc: 0.1008
Validation accuracy:
0.100833334
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 265s - loss: 1.2118 - acc: 0.5556 - val_loss: 1.9552 - val_acc: 0.2033
Epoch 2/15
48000/48000 - 261s - loss: 0.7129 - acc: 0.7295 - val_loss: 0.8112 - val_acc: 0.6887
Epoch 3/15
48000/48000 - 260s - loss: 0.6133 - acc: 0.7693 - val_loss: 0.4873 - val_acc: 0.8058
Epoch 4/15
48000/48000 - 264s - loss: 0.5550 - acc: 0.7919 - val_loss: 0.4150 - val_acc: 0.8508
Epoch 5/15
48000/48000 - 259s - loss: 0.5068 - acc: 0.8138 - val_loss: 0.3769 - val_acc: 0.8692
Epoch 6/15
48000/48000 - 262s - loss: 0.4736 - acc: 0.8293 - val_loss: 0.3418 - val_acc: 0.8810
Epoch 7/15
48000/48000 - 262s - loss: 0.4395 - acc: 0.8430 - val_loss: 0.3378 - val_acc: 0.8824
Epoch 8/15
48000/48000 - 260s - loss: 0.4173 - acc: 0.8518 - val_loss: 0.3106 - val_acc: 0.8885
Epoch 9/15
48000/48000 - 260s - loss: 0.3988 - acc: 0.8602 - val_loss: 0.3164 - val_acc: 0.8868
Epoch 10/15
48000/48000 - 261s - loss: 0.3807 - acc: 0.8676 - val_loss: 0.3015 - val_acc: 0.8890
Epoch 11/15
48000/48000 - 260s - loss: 0.3686 - acc: 0.8706 - val_loss: 0.2797 - val_acc: 0.8959
Epoch 12/15
48000/48000 - 261s - loss: 0.3597 - acc: 0.8740 - val_loss: 0.2742 - val_acc: 0.9039
Epoch 13/15
48000/48000 - 259s - loss: 0.3462 - acc: 0.8789 - val_loss: 0.2511 - val_acc: 0.9115
Epoch 14/15
48000/48000 - 258s - loss: 0.3388 - acc: 0.8811 - val_loss: 0.2594 - val_acc: 0.9081
Epoch 15/15
48000/48000 - 258s - loss: 0.3306 - acc: 0.8840 - val_loss: 0.2464 - val_acc: 0.9126
Validation accuracy:
0.91258335
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 254s - loss: 1.3005 - acc: 0.5293 - val_loss: 2.0639 - val_acc: 0.3112
Epoch 2/15
48000/48000 - 257s - loss: 0.7436 - acc: 0.7212 - val_loss: 1.0386 - val_acc: 0.5829
Epoch 3/15
48000/48000 - 256s - loss: 0.6281 - acc: 0.7643 - val_loss: 0.6083 - val_acc: 0.7497
Epoch 4/15
48000/48000 - 260s - loss: 0.5580 - acc: 0.7922 - val_loss: 0.3964 - val_acc: 0.8515
Epoch 5/15
48000/48000 - 258s - loss: 0.5091 - acc: 0.8163 - val_loss: 0.3760 - val_acc: 0.8601
Epoch 6/15
48000/48000 - 257s - loss: 0.4690 - acc: 0.8332 - val_loss: 0.3251 - val_acc: 0.8823
Epoch 7/15
48000/48000 - 255s - loss: 0.4350 - acc: 0.8468 - val_loss: 0.3046 - val_acc: 0.8903
Epoch 8/15
48000/48000 - 255s - loss: 0.4094 - acc: 0.8566 - val_loss: 0.2989 - val_acc: 0.8972
Epoch 9/15
48000/48000 - 257s - loss: 0.3917 - acc: 0.8635 - val_loss: 0.2800 - val_acc: 0.9003
Epoch 10/15
48000/48000 - 256s - loss: 0.3810 - acc: 0.8681 - val_loss: 0.2712 - val_acc: 0.9047
Epoch 11/15
48000/48000 - 256s - loss: 0.3656 - acc: 0.8756 - val_loss: 0.2685 - val_acc: 0.9012
Epoch 12/15
48000/48000 - 255s - loss: 0.3550 - acc: 0.8785 - val_loss: 0.2560 - val_acc: 0.9112
Epoch 13/15
48000/48000 - 257s - loss: 0.3470 - acc: 0.8813 - val_loss: 0.2496 - val_acc: 0.9126
Epoch 14/15
48000/48000 - 256s - loss: 0.3342 - acc: 0.8876 - val_loss: 0.2381 - val_acc: 0.9144
Epoch 15/15
48000/48000 - 255s - loss: 0.3283 - acc: 0.8899 - val_loss: 0.2352 - val_acc: 0.9177
Validation accuracy:
0.9176667
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 271s - loss: 1.5405 - acc: 0.6129 - val_loss: 2.4108 - val_acc: 0.1072
Epoch 2/15
48000/48000 - 268s - loss: 0.6693 - acc: 0.7621 - val_loss: 0.9699 - val_acc: 0.6430
Epoch 3/15
48000/48000 - 265s - loss: 0.5348 - acc: 0.8133 - val_loss: 0.4663 - val_acc: 0.8222
Epoch 4/15
48000/48000 - 266s - loss: 0.4580 - acc: 0.8406 - val_loss: 0.3332 - val_acc: 0.8751
Epoch 5/15
48000/48000 - 266s - loss: 0.4112 - acc: 0.8599 - val_loss: 0.2964 - val_acc: 0.8920
Epoch 6/15
48000/48000 - 267s - loss: 0.3805 - acc: 0.8683 - val_loss: 0.2919 - val_acc: 0.8917
Epoch 7/15
48000/48000 - 274s - loss: 0.3568 - acc: 0.8781 - val_loss: 0.2624 - val_acc: 0.9031
Epoch 8/15
48000/48000 - 275s - loss: 0.3340 - acc: 0.8860 - val_loss: 0.2531 - val_acc: 0.9114
Epoch 9/15
48000/48000 - 269s - loss: 0.3173 - acc: 0.8932 - val_loss: 0.2723 - val_acc: 0.9021
Epoch 10/15
48000/48000 - 269s - loss: 0.3059 - acc: 0.8966 - val_loss: 0.2289 - val_acc: 0.9178
Epoch 11/15
48000/48000 - 276s - loss: 0.2991 - acc: 0.8990 - val_loss: 0.2327 - val_acc: 0.9156
Epoch 12/15
48000/48000 - 287s - loss: 0.2879 - acc: 0.9035 - val_loss: 0.2257 - val_acc: 0.9181
Epoch 13/15
48000/48000 - 281s - loss: 0.2740 - acc: 0.9077 - val_loss: 0.2179 - val_acc: 0.9231
Epoch 14/15
48000/48000 - 282s - loss: 0.2630 - acc: 0.9090 - val_loss: 0.2121 - val_acc: 0.9233
Epoch 15/15
48000/48000 - 279s - loss: 0.2566 - acc: 0.9131 - val_loss: 0.2126 - val_acc: 0.9220
Validation accuracy:
0.922
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 266s - loss: 1.1092 - acc: 0.6312 - val_loss: 1.5157 - val_acc: 0.4192
Epoch 2/15
48000/48000 - 277s - loss: 0.6253 - acc: 0.7631 - val_loss: 0.8074 - val_acc: 0.7443
Epoch 3/15
48000/48000 - 276s - loss: 0.5242 - acc: 0.8042 - val_loss: 0.4714 - val_acc: 0.8185
Epoch 4/15
48000/48000 - 261s - loss: 0.4587 - acc: 0.8312 - val_loss: 0.3491 - val_acc: 0.8703
Epoch 5/15
48000/48000 - 268s - loss: 0.4222 - acc: 0.8453 - val_loss: 0.3269 - val_acc: 0.8768
Epoch 6/15
48000/48000 - 265s - loss: 0.3949 - acc: 0.8559 - val_loss: 0.2920 - val_acc: 0.8937
Epoch 7/15
48000/48000 - 267s - loss: 0.3731 - acc: 0.8639 - val_loss: 0.2815 - val_acc: 0.8959
Epoch 8/15
48000/48000 - 265s - loss: 0.3540 - acc: 0.8689 - val_loss: 0.2625 - val_acc: 0.9043
Epoch 9/15
48000/48000 - 269s - loss: 0.3425 - acc: 0.8746 - val_loss: 0.2799 - val_acc: 0.8961
Epoch 10/15
48000/48000 - 268s - loss: 0.3300 - acc: 0.8783 - val_loss: 0.2600 - val_acc: 0.9052
Epoch 11/15
48000/48000 - 265s - loss: 0.3242 - acc: 0.8812 - val_loss: 0.2459 - val_acc: 0.9103
Epoch 12/15
48000/48000 - 274s - loss: 0.3127 - acc: 0.8854 - val_loss: 0.2471 - val_acc: 0.9070
Epoch 13/15
48000/48000 - 270s - loss: 0.3071 - acc: 0.8878 - val_loss: 0.2288 - val_acc: 0.9172
Epoch 14/15
48000/48000 - 268s - loss: 0.2999 - acc: 0.8896 - val_loss: 0.2282 - val_acc: 0.9159
Epoch 15/15
48000/48000 - 266s - loss: 0.2907 - acc: 0.8940 - val_loss: 0.2295 - val_acc: 0.9153
Validation accuracy:
0.91533333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 129s - loss: 1.2068 - acc: 0.5671 - val_loss: 2.3611 - val_acc: 0.1272
Epoch 2/15
48000/48000 - 126s - loss: 0.7225 - acc: 0.7235 - val_loss: 1.1352 - val_acc: 0.5559
Epoch 3/15
48000/48000 - 126s - loss: 0.6360 - acc: 0.7545 - val_loss: 0.6743 - val_acc: 0.7184
Epoch 4/15
48000/48000 - 129s - loss: 0.5878 - acc: 0.7769 - val_loss: 0.5432 - val_acc: 0.7840
Epoch 5/15
48000/48000 - 128s - loss: 0.5632 - acc: 0.7842 - val_loss: 0.4572 - val_acc: 0.8192
Epoch 6/15
48000/48000 - 127s - loss: 0.5381 - acc: 0.7966 - val_loss: 0.4368 - val_acc: 0.8332
Epoch 7/15
48000/48000 - 129s - loss: 0.5164 - acc: 0.8039 - val_loss: 0.4050 - val_acc: 0.8438
Epoch 8/15
48000/48000 - 131s - loss: 0.4952 - acc: 0.8111 - val_loss: 0.3773 - val_acc: 0.8551
Epoch 9/15
48000/48000 - 144s - loss: 0.4782 - acc: 0.8190 - val_loss: 0.3802 - val_acc: 0.8550
Epoch 10/15
48000/48000 - 129s - loss: 0.4673 - acc: 0.8239 - val_loss: 0.3726 - val_acc: 0.8587
Epoch 11/15
48000/48000 - 128s - loss: 0.4500 - acc: 0.8310 - val_loss: 0.3589 - val_acc: 0.8714
Epoch 12/15
48000/48000 - 130s - loss: 0.4468 - acc: 0.8337 - val_loss: 0.3736 - val_acc: 0.8583
Epoch 13/15
48000/48000 - 129s - loss: 0.4323 - acc: 0.8368 - val_loss: 0.3476 - val_acc: 0.8686
Epoch 14/15
48000/48000 - 129s - loss: 0.4278 - acc: 0.8397 - val_loss: 0.3360 - val_acc: 0.8719
Epoch 15/15
48000/48000 - 130s - loss: 0.4172 - acc: 0.8450 - val_loss: 0.3166 - val_acc: 0.8817
Validation accuracy:
0.88166666
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 271s - loss: 0.6648 - acc: 0.7697 - val_loss: 1.9499 - val_acc: 0.2960
Epoch 2/15
48000/48000 - 274s - loss: 0.3862 - acc: 0.8624 - val_loss: 0.9795 - val_acc: 0.7437
Epoch 3/15
48000/48000 - 268s - loss: 0.3250 - acc: 0.8827 - val_loss: 0.3697 - val_acc: 0.8668
Epoch 4/15
48000/48000 - 268s - loss: 0.2911 - acc: 0.8949 - val_loss: 0.2589 - val_acc: 0.9023
Epoch 5/15
48000/48000 - 266s - loss: 0.2678 - acc: 0.9029 - val_loss: 0.2979 - val_acc: 0.8937
Epoch 6/15
48000/48000 - 267s - loss: 0.2538 - acc: 0.9075 - val_loss: 0.2320 - val_acc: 0.9154
Epoch 7/15
48000/48000 - 266s - loss: 0.2363 - acc: 0.9146 - val_loss: 0.2219 - val_acc: 0.9180
Epoch 8/15
48000/48000 - 269s - loss: 0.2286 - acc: 0.9164 - val_loss: 0.2264 - val_acc: 0.9170
Epoch 9/15
48000/48000 - 271s - loss: 0.2186 - acc: 0.9194 - val_loss: 0.2045 - val_acc: 0.9276
Epoch 10/15
48000/48000 - 268s - loss: 0.2081 - acc: 0.9240 - val_loss: 0.2110 - val_acc: 0.9223
Epoch 11/15
48000/48000 - 270s - loss: 0.2022 - acc: 0.9254 - val_loss: 0.2102 - val_acc: 0.9247
Epoch 12/15
48000/48000 - 272s - loss: 0.1956 - acc: 0.9293 - val_loss: 0.2098 - val_acc: 0.9257
Epoch 13/15
48000/48000 - 269s - loss: 0.1880 - acc: 0.9301 - val_loss: 0.1912 - val_acc: 0.9306
Epoch 14/15
48000/48000 - 268s - loss: 0.1828 - acc: 0.9324 - val_loss: 0.2390 - val_acc: 0.9214
Epoch 15/15
48000/48000 - 270s - loss: 0.1772 - acc: 0.9344 - val_loss: 0.2004 - val_acc: 0.9282
Validation accuracy:
0.9281667
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 277s - loss: 0.5169 - acc: 0.8139 - val_loss: 1.7856 - val_acc: 0.2938
Epoch 2/15
48000/48000 - 277s - loss: 0.3240 - acc: 0.8812 - val_loss: 0.8334 - val_acc: 0.7551
Epoch 3/15
48000/48000 - 280s - loss: 0.2698 - acc: 0.9015 - val_loss: 0.2821 - val_acc: 0.9077
Epoch 4/15
48000/48000 - 278s - loss: 0.2361 - acc: 0.9133 - val_loss: 0.2330 - val_acc: 0.9129
Epoch 5/15
48000/48000 - 278s - loss: 0.2132 - acc: 0.9203 - val_loss: 0.2172 - val_acc: 0.9201
Epoch 6/15
48000/48000 - 282s - loss: 0.1942 - acc: 0.9302 - val_loss: 0.2209 - val_acc: 0.9208
Epoch 7/15
48000/48000 - 276s - loss: 0.1778 - acc: 0.9333 - val_loss: 0.2658 - val_acc: 0.9115
Epoch 8/15
48000/48000 - 277s - loss: 0.1642 - acc: 0.9397 - val_loss: 0.1926 - val_acc: 0.9291
Epoch 9/15
48000/48000 - 277s - loss: 0.1520 - acc: 0.9435 - val_loss: 0.2231 - val_acc: 0.9231
Epoch 10/15
48000/48000 - 277s - loss: 0.1418 - acc: 0.9474 - val_loss: 0.2058 - val_acc: 0.9261
Epoch 11/15
48000/48000 - 277s - loss: 0.1312 - acc: 0.9512 - val_loss: 0.2141 - val_acc: 0.9285
Epoch 12/15
48000/48000 - 279s - loss: 0.1209 - acc: 0.9546 - val_loss: 0.1993 - val_acc: 0.9315
Epoch 13/15
48000/48000 - 280s - loss: 0.1101 - acc: 0.9594 - val_loss: 0.2165 - val_acc: 0.9313
Epoch 14/15
48000/48000 - 279s - loss: 0.1042 - acc: 0.9620 - val_loss: 0.2049 - val_acc: 0.9346
Epoch 15/15
48000/48000 - 279s - loss: 0.0980 - acc: 0.9625 - val_loss: 0.1991 - val_acc: 0.9325
Validation accuracy:
0.9325
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 288s - loss: 0.6997 - acc: 0.7467 - val_loss: 1.9375 - val_acc: 0.3307
Epoch 2/15
48000/48000 - 292s - loss: 0.4234 - acc: 0.8437 - val_loss: 0.8600 - val_acc: 0.7141
Epoch 3/15
48000/48000 - 303s - loss: 0.3625 - acc: 0.8669 - val_loss: 0.3845 - val_acc: 0.8665
Epoch 4/15
48000/48000 - 288s - loss: 0.3235 - acc: 0.8812 - val_loss: 0.3382 - val_acc: 0.8745
Epoch 5/15
48000/48000 - 302s - loss: 0.2934 - acc: 0.8917 - val_loss: 0.2863 - val_acc: 0.8966
Epoch 6/15
48000/48000 - 288s - loss: 0.2770 - acc: 0.8975 - val_loss: 0.2432 - val_acc: 0.9114
Epoch 7/15
48000/48000 - 286s - loss: 0.2582 - acc: 0.9057 - val_loss: 0.2681 - val_acc: 0.8990
Epoch 8/15
48000/48000 - 288s - loss: 0.2462 - acc: 0.9088 - val_loss: 0.2599 - val_acc: 0.9053
Epoch 9/15
48000/48000 - 288s - loss: 0.2353 - acc: 0.9114 - val_loss: 0.2307 - val_acc: 0.9181
Epoch 10/15
48000/48000 - 288s - loss: 0.2248 - acc: 0.9169 - val_loss: 0.2257 - val_acc: 0.9180
Epoch 11/15
48000/48000 - 287s - loss: 0.2165 - acc: 0.9189 - val_loss: 0.2138 - val_acc: 0.9223
Epoch 12/15
48000/48000 - 291s - loss: 0.2074 - acc: 0.9235 - val_loss: 0.2376 - val_acc: 0.9143
Epoch 13/15
48000/48000 - 294s - loss: 0.2029 - acc: 0.9239 - val_loss: 0.2126 - val_acc: 0.9237
Epoch 14/15
48000/48000 - 296s - loss: 0.1963 - acc: 0.9262 - val_loss: 0.2139 - val_acc: 0.9221
Epoch 15/15
48000/48000 - 299s - loss: 0.1919 - acc: 0.9280 - val_loss: 0.2080 - val_acc: 0.9262
Validation accuracy:
0.92616665
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 312s - loss: 0.5516 - acc: 0.8030 - val_loss: 1.9073 - val_acc: 0.3297
Epoch 2/15
48000/48000 - 304s - loss: 0.3438 - acc: 0.8745 - val_loss: 0.8752 - val_acc: 0.7742
Epoch 3/15
48000/48000 - 321s - loss: 0.2855 - acc: 0.8953 - val_loss: 0.3317 - val_acc: 0.8771
Epoch 4/15
48000/48000 - 307s - loss: 0.2481 - acc: 0.9093 - val_loss: 0.2620 - val_acc: 0.9035
Epoch 5/15
48000/48000 - 302s - loss: 0.2261 - acc: 0.9169 - val_loss: 0.2286 - val_acc: 0.9170
Epoch 6/15
48000/48000 - 342s - loss: 0.2056 - acc: 0.9233 - val_loss: 0.2228 - val_acc: 0.9165
Epoch 7/15
48000/48000 - 317s - loss: 0.1918 - acc: 0.9289 - val_loss: 0.2108 - val_acc: 0.9243
Epoch 8/15
48000/48000 - 296s - loss: 0.1757 - acc: 0.9342 - val_loss: 0.2112 - val_acc: 0.9268
Epoch 9/15
48000/48000 - 303s - loss: 0.1694 - acc: 0.9370 - val_loss: 0.2179 - val_acc: 0.9255
Epoch 10/15
48000/48000 - 288s - loss: 0.1558 - acc: 0.9415 - val_loss: 0.2015 - val_acc: 0.9267
Epoch 11/15
48000/48000 - 379s - loss: 0.1454 - acc: 0.9457 - val_loss: 0.2143 - val_acc: 0.9281
Epoch 12/15
48000/48000 - 315s - loss: 0.1381 - acc: 0.9489 - val_loss: 0.2095 - val_acc: 0.9264
Epoch 13/15
48000/48000 - 327s - loss: 0.1273 - acc: 0.9527 - val_loss: 0.2074 - val_acc: 0.9295
Epoch 14/15
48000/48000 - 329s - loss: 0.1239 - acc: 0.9528 - val_loss: 0.1929 - val_acc: 0.9335
Epoch 15/15
48000/48000 - 334s - loss: 0.1156 - acc: 0.9564 - val_loss: 0.2120 - val_acc: 0.9311
Validation accuracy:
0.9310833
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 341s - loss: 0.4965 - acc: 0.8198 - val_loss: 1.8954 - val_acc: 0.3555
Epoch 2/15
48000/48000 - 339s - loss: 0.3168 - acc: 0.8846 - val_loss: 0.9792 - val_acc: 0.6956
Epoch 3/15
48000/48000 - 320s - loss: 0.2618 - acc: 0.9042 - val_loss: 0.3663 - val_acc: 0.8587
Epoch 4/15
48000/48000 - 290s - loss: 0.2311 - acc: 0.9150 - val_loss: 0.2391 - val_acc: 0.9106
Epoch 5/15
48000/48000 - 366s - loss: 0.2085 - acc: 0.9239 - val_loss: 0.2234 - val_acc: 0.9191
Epoch 6/15
48000/48000 - 306s - loss: 0.1901 - acc: 0.9302 - val_loss: 0.2169 - val_acc: 0.9195
Epoch 7/15
48000/48000 - 283s - loss: 0.1733 - acc: 0.9359 - val_loss: 0.2125 - val_acc: 0.9237
Epoch 8/15
48000/48000 - 286s - loss: 0.1622 - acc: 0.9396 - val_loss: 0.2004 - val_acc: 0.9291
Epoch 9/15
48000/48000 - 282s - loss: 0.1523 - acc: 0.9436 - val_loss: 0.2307 - val_acc: 0.9192
Epoch 10/15
48000/48000 - 284s - loss: 0.1366 - acc: 0.9495 - val_loss: 0.2297 - val_acc: 0.9196
Epoch 11/15
48000/48000 - 281s - loss: 0.1296 - acc: 0.9528 - val_loss: 0.2359 - val_acc: 0.9231
Epoch 12/15
48000/48000 - 281s - loss: 0.1189 - acc: 0.9563 - val_loss: 0.2026 - val_acc: 0.9284
Epoch 13/15
48000/48000 - 282s - loss: 0.1110 - acc: 0.9583 - val_loss: 0.2159 - val_acc: 0.9283
Epoch 14/15
48000/48000 - 283s - loss: 0.1065 - acc: 0.9597 - val_loss: 0.2227 - val_acc: 0.9278
Epoch 15/15
48000/48000 - 280s - loss: 0.0987 - acc: 0.9636 - val_loss: 0.1998 - val_acc: 0.9333
Validation accuracy:
0.93325
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 278s - loss: 0.4650 - acc: 0.8321 - val_loss: 1.9973 - val_acc: 0.3484
Epoch 2/15
48000/48000 - 276s - loss: 0.2920 - acc: 0.8921 - val_loss: 0.8837 - val_acc: 0.7999
Epoch 3/15
48000/48000 - 277s - loss: 0.2468 - acc: 0.9088 - val_loss: 0.3259 - val_acc: 0.8830
Epoch 4/15
48000/48000 - 278s - loss: 0.2071 - acc: 0.9238 - val_loss: 0.2365 - val_acc: 0.9143
Epoch 5/15
48000/48000 - 278s - loss: 0.1835 - acc: 0.9330 - val_loss: 0.2595 - val_acc: 0.9102
Epoch 6/15
48000/48000 - 276s - loss: 0.1610 - acc: 0.9410 - val_loss: 0.2627 - val_acc: 0.9115
Epoch 7/15
48000/48000 - 277s - loss: 0.1423 - acc: 0.9474 - val_loss: 0.2954 - val_acc: 0.9007
Epoch 8/15
48000/48000 - 276s - loss: 0.1292 - acc: 0.9525 - val_loss: 0.2957 - val_acc: 0.9041
Epoch 9/15
48000/48000 - 284s - loss: 0.1099 - acc: 0.9596 - val_loss: 0.2312 - val_acc: 0.9238
Epoch 10/15
48000/48000 - 275s - loss: 0.0998 - acc: 0.9643 - val_loss: 0.2263 - val_acc: 0.9273
Epoch 11/15
48000/48000 - 280s - loss: 0.0862 - acc: 0.9678 - val_loss: 0.3831 - val_acc: 0.9023
Epoch 12/15
48000/48000 - 275s - loss: 0.0767 - acc: 0.9713 - val_loss: 0.2874 - val_acc: 0.9170
Epoch 13/15
48000/48000 - 276s - loss: 0.0651 - acc: 0.9761 - val_loss: 0.2880 - val_acc: 0.9187
Epoch 14/15
48000/48000 - 281s - loss: 0.0532 - acc: 0.9804 - val_loss: 0.2569 - val_acc: 0.9249
Epoch 15/15
48000/48000 - 299s - loss: 0.0472 - acc: 0.9824 - val_loss: 0.2553 - val_acc: 0.9253
Validation accuracy:
0.9253333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 504s - loss: 0.4997 - acc: 0.8255 - val_loss: 1.9061 - val_acc: 0.2397
Epoch 2/15
48000/48000 - 521s - loss: 0.3117 - acc: 0.8871 - val_loss: 0.7381 - val_acc: 0.8123
Epoch 3/15
48000/48000 - 374s - loss: 0.2638 - acc: 0.9033 - val_loss: 0.3173 - val_acc: 0.9051
Epoch 4/15
48000/48000 - 307s - loss: 0.2310 - acc: 0.9142 - val_loss: 0.3764 - val_acc: 0.8665
Epoch 5/15
48000/48000 - 284s - loss: 0.2104 - acc: 0.9225 - val_loss: 0.2602 - val_acc: 0.9062
Epoch 6/15
48000/48000 - 286s - loss: 0.1927 - acc: 0.9286 - val_loss: 0.2228 - val_acc: 0.9162
Epoch 7/15
48000/48000 - 281s - loss: 0.1776 - acc: 0.9346 - val_loss: 0.2288 - val_acc: 0.9182
Epoch 8/15
48000/48000 - 278s - loss: 0.1631 - acc: 0.9395 - val_loss: 0.2368 - val_acc: 0.9152
Epoch 9/15
48000/48000 - 302s - loss: 0.1494 - acc: 0.9432 - val_loss: 0.2070 - val_acc: 0.9283
Epoch 10/15
48000/48000 - 294s - loss: 0.1367 - acc: 0.9491 - val_loss: 0.2279 - val_acc: 0.9224
Epoch 11/15
48000/48000 - 299s - loss: 0.1297 - acc: 0.9512 - val_loss: 0.2175 - val_acc: 0.9303
Epoch 12/15
48000/48000 - 281s - loss: 0.1168 - acc: 0.9561 - val_loss: 0.2153 - val_acc: 0.9287
Epoch 13/15
48000/48000 - 298s - loss: 0.1106 - acc: 0.9594 - val_loss: 0.2273 - val_acc: 0.9249
Epoch 14/15
48000/48000 - 285s - loss: 0.0997 - acc: 0.9628 - val_loss: 0.2179 - val_acc: 0.9268
Epoch 15/15
48000/48000 - 293s - loss: 0.0911 - acc: 0.9653 - val_loss: 0.2402 - val_acc: 0.9263
Validation accuracy:
0.9263333
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 290s - loss: 0.5250 - acc: 0.8080 - val_loss: 1.5599 - val_acc: 0.5357
Epoch 2/15
48000/48000 - 285s - loss: 0.3266 - acc: 0.8809 - val_loss: 0.7080 - val_acc: 0.8078
Epoch 3/15
48000/48000 - 280s - loss: 0.2708 - acc: 0.9007 - val_loss: 0.2866 - val_acc: 0.8979
Epoch 4/15
48000/48000 - 316s - loss: 0.2436 - acc: 0.9113 - val_loss: 0.2319 - val_acc: 0.9153
Epoch 5/15
48000/48000 - 282s - loss: 0.2171 - acc: 0.9212 - val_loss: 0.2435 - val_acc: 0.9133
Epoch 6/15
48000/48000 - 280s - loss: 0.2039 - acc: 0.9253 - val_loss: 0.2249 - val_acc: 0.9193
Epoch 7/15
48000/48000 - 279s - loss: 0.1894 - acc: 0.9283 - val_loss: 0.2181 - val_acc: 0.9211
Epoch 8/15
48000/48000 - 277s - loss: 0.1749 - acc: 0.9339 - val_loss: 0.2016 - val_acc: 0.9293
Epoch 9/15
48000/48000 - 292s - loss: 0.1667 - acc: 0.9380 - val_loss: 0.2069 - val_acc: 0.9243
Epoch 10/15
48000/48000 - 319s - loss: 0.1516 - acc: 0.9426 - val_loss: 0.2041 - val_acc: 0.9283
Epoch 11/15
48000/48000 - 324s - loss: 0.1466 - acc: 0.9464 - val_loss: 0.1940 - val_acc: 0.9312
Epoch 12/15
48000/48000 - 336s - loss: 0.1346 - acc: 0.9495 - val_loss: 0.1996 - val_acc: 0.9291
Epoch 13/15
48000/48000 - 325s - loss: 0.1266 - acc: 0.9529 - val_loss: 0.2160 - val_acc: 0.9249
Epoch 14/15
48000/48000 - 318s - loss: 0.1207 - acc: 0.9564 - val_loss: 0.1963 - val_acc: 0.9309
Epoch 15/15
48000/48000 - 316s - loss: 0.1124 - acc: 0.9582 - val_loss: 0.1974 - val_acc: 0.9337
Validation accuracy:
0.93375
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 320s - loss: 0.5347 - acc: 0.8057 - val_loss: 1.7260 - val_acc: 0.4125
Epoch 2/15
48000/48000 - 328s - loss: 0.3244 - acc: 0.8810 - val_loss: 0.8437 - val_acc: 0.7933
Epoch 3/15
48000/48000 - 315s - loss: 0.2721 - acc: 0.9006 - val_loss: 0.2847 - val_acc: 0.9038
Epoch 4/15
48000/48000 - 314s - loss: 0.2407 - acc: 0.9118 - val_loss: 0.2448 - val_acc: 0.9100
Epoch 5/15
48000/48000 - 312s - loss: 0.2145 - acc: 0.9217 - val_loss: 0.2350 - val_acc: 0.9136
Epoch 6/15
48000/48000 - 323s - loss: 0.1996 - acc: 0.9262 - val_loss: 0.2145 - val_acc: 0.9218
Epoch 7/15
48000/48000 - 322s - loss: 0.1820 - acc: 0.9331 - val_loss: 0.2248 - val_acc: 0.9183
Epoch 8/15
48000/48000 - 323s - loss: 0.1688 - acc: 0.9377 - val_loss: 0.2095 - val_acc: 0.9215
Epoch 9/15
48000/48000 - 319s - loss: 0.1593 - acc: 0.9410 - val_loss: 0.2163 - val_acc: 0.9237
Epoch 10/15
48000/48000 - 306s - loss: 0.1468 - acc: 0.9445 - val_loss: 0.2374 - val_acc: 0.9187
Epoch 11/15
48000/48000 - 273s - loss: 0.1365 - acc: 0.9503 - val_loss: 0.2200 - val_acc: 0.9281
Epoch 12/15
48000/48000 - 271s - loss: 0.1291 - acc: 0.9534 - val_loss: 0.2140 - val_acc: 0.9261
Epoch 13/15
48000/48000 - 270s - loss: 0.1191 - acc: 0.9554 - val_loss: 0.2114 - val_acc: 0.9253
Epoch 14/15
48000/48000 - 267s - loss: 0.1090 - acc: 0.9597 - val_loss: 0.2275 - val_acc: 0.9243
Epoch 15/15
48000/48000 - 267s - loss: 0.1039 - acc: 0.9617 - val_loss: 0.1976 - val_acc: 0.9351
Validation accuracy:
0.9350833
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 271s - loss: 0.5436 - acc: 0.8052 - val_loss: 1.7580 - val_acc: 0.5061
Epoch 2/15
48000/48000 - 270s - loss: 0.3276 - acc: 0.8807 - val_loss: 0.9048 - val_acc: 0.7625
Epoch 3/15
48000/48000 - 267s - loss: 0.2742 - acc: 0.8999 - val_loss: 0.3052 - val_acc: 0.8953
Epoch 4/15
48000/48000 - 270s - loss: 0.2452 - acc: 0.9108 - val_loss: 0.2654 - val_acc: 0.8998
Epoch 5/15
48000/48000 - 269s - loss: 0.2240 - acc: 0.9175 - val_loss: 0.2216 - val_acc: 0.9175
Epoch 6/15
48000/48000 - 267s - loss: 0.2058 - acc: 0.9247 - val_loss: 0.2517 - val_acc: 0.9118
Epoch 7/15
48000/48000 - 268s - loss: 0.1946 - acc: 0.9273 - val_loss: 0.2143 - val_acc: 0.9208
Epoch 8/15
48000/48000 - 267s - loss: 0.1816 - acc: 0.9327 - val_loss: 0.2005 - val_acc: 0.9293
Epoch 9/15
48000/48000 - 268s - loss: 0.1691 - acc: 0.9374 - val_loss: 0.2116 - val_acc: 0.9272
Epoch 10/15
48000/48000 - 267s - loss: 0.1574 - acc: 0.9420 - val_loss: 0.2095 - val_acc: 0.9264
Epoch 11/15
48000/48000 - 266s - loss: 0.1493 - acc: 0.9442 - val_loss: 0.2019 - val_acc: 0.9280
Epoch 12/15
48000/48000 - 267s - loss: 0.1422 - acc: 0.9471 - val_loss: 0.1948 - val_acc: 0.9297
Epoch 13/15
48000/48000 - 265s - loss: 0.1331 - acc: 0.9501 - val_loss: 0.1956 - val_acc: 0.9310
Epoch 14/15
48000/48000 - 265s - loss: 0.1276 - acc: 0.9514 - val_loss: 0.2342 - val_acc: 0.9259
Epoch 15/15
48000/48000 - 267s - loss: 0.1205 - acc: 0.9547 - val_loss: 0.2466 - val_acc: 0.9193
Validation accuracy:
0.91933334
Train on 48000 samples, validate on 12000 samples
Epoch 1/15
48000/48000 - 267s - loss: 0.4902 - acc: 0.8236 - val_loss: 1.8765 - val_acc: 0.3355
Epoch 2/15
48000/48000 - 263s - loss: 0.3102 - acc: 0.8851 - val_loss: 0.8477 - val_acc: 0.7654
Epoch 3/15
48000/48000 - 265s - loss: 0.2571 - acc: 0.9067 - val_loss: 0.2894 - val_acc: 0.9018
Epoch 4/15
48000/48000 - 267s - loss: 0.2269 - acc: 0.9173 - val_loss: 0.2842 - val_acc: 0.8911
Epoch 5/15
48000/48000 - 268s - loss: 0.2091 - acc: 0.9223 - val_loss: 0.2350 - val_acc: 0.9167
Epoch 6/15
48000/48000 - 266s - loss: 0.1895 - acc: 0.9300 - val_loss: 0.2086 - val_acc: 0.9258
Epoch 7/15
48000/48000 - 267s - loss: 0.1757 - acc: 0.9358 - val_loss: 0.2041 - val_acc: 0.9287
Epoch 8/15
48000/48000 - 267s - loss: 0.1618 - acc: 0.9409 - val_loss: 0.2555 - val_acc: 0.9115
Epoch 9/15
48000/48000 - 264s - loss: 0.1512 - acc: 0.9439 - val_loss: 0.2151 - val_acc: 0.9217
Epoch 10/15
48000/48000 - 264s - loss: 0.1415 - acc: 0.9475 - val_loss: 0.2338 - val_acc: 0.9172
Epoch 11/15
48000/48000 - 264s - loss: 0.1312 - acc: 0.9507 - val_loss: 0.2118 - val_acc: 0.9276
Epoch 12/15
48000/48000 - 267s - loss: 0.1236 - acc: 0.9538 - val_loss: 0.2052 - val_acc: 0.9286
Epoch 13/15
48000/48000 - 267s - loss: 0.1106 - acc: 0.9577 - val_loss: 0.2138 - val_acc: 0.9262
Epoch 14/15
48000/48000 - 267s - loss: 0.1030 - acc: 0.9612 - val_loss: 0.2249 - val_acc: 0.9245
Epoch 15/15
48000/48000 - 267s - loss: 0.0960 - acc: 0.9640 - val_loss: 0.2342 - val_acc: 0.9224
Validation accuracy:
0.9224167
100%|█████████████████████████████████████████████| 30/30 [30:58:50<00:00, 3717.68s/it, best loss: -0.9350833296775818]
Hyperas best results
In [36]:
# Save entire model to a HDF5 file
best_model.save('best_Fashion_MNIST_model')
In [35]:
best_run
Out[35]:
{'Dense': 0,
'Dense_1': 1,
'Dropout': 0.23436375142305166,
'Dropout_1': 0.5304004974162193,
'Dropout_2': 0.18899070090342202,
'Dropout_3': 0.07653051057116944,
'Dropout_4': 0.22894424857523937,
'Dropout_5': 0.9929202847742498,
'choiceval': 0,
'model_choice': 1}
Predicting ¶
In [6]:
b_model = tf.keras.Sequential()
b_model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu', input_shape=(28, 28, 1)))
b_model.add(tf.keras.layers.Conv2D(32, kernel_size=3, padding='same', activation='relu'))
b_model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
b_model.add(tf.keras.layers.Dropout(0.18899070090342202))
b_model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
b_model.add(tf.keras.layers.Conv2D(64, kernel_size=3, activation='relu'))
b_model.add(tf.keras.layers.BatchNormalization())
b_model.add(tf.keras.layers.MaxPool2D(pool_size=2, strides=2))
b_model.add(tf.keras.layers.Dropout(0.07653051057116944))
b_model.add(tf.keras.layers.Flatten())
b_model.add(tf.keras.layers.Dense(256, activation='relu'))
b_model.add(tf.keras.layers.BatchNormalization())
b_model.add(tf.keras.layers.Dropout(0.22894424857523937))
b_model.add(tf.keras.layers.Dense(10, activation='softmax'))
WARNING:tensorflow:From C:\ProgramData\Anaconda3\lib\site-packages\tensorflow_core\python\ops\resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version. Instructions for updating: If using Keras pass *_constraint arguments to layers.
In [7]:
%%time
optimizer = tf.keras.optimizers.Adam(learning_rate=0.001)
b_model.compile(optimizer=optimizer, loss='categorical_crossentropy', metrics=['accuracy'])
b_model.fit(X_train, y_train, epochs=15, batch_size=256, verbose=1, validation_data=[X_val, y_val])
score, acc = b_model.evaluate(X_test, y_test)
Train on 48000 samples, validate on 12000 samples Epoch 1/15 48000/48000 [==============================] - 347s 7ms/sample - loss: 0.5318 - acc: 0.8113 - val_loss: 1.7158 - val_acc: 0.4259 Epoch 2/15 48000/48000 [==============================] - 361s 8ms/sample - loss: 0.3235 - acc: 0.8838 - val_loss: 0.8675 - val_acc: 0.7772 Epoch 3/15 48000/48000 [==============================] - 357s 7ms/sample - loss: 0.2709 - acc: 0.9019 - val_loss: 0.3098 - val_acc: 0.9008 Epoch 4/15 48000/48000 [==============================] - 452s 9ms/sample - loss: 0.2419 - acc: 0.9120 - val_loss: 0.2428 - val_acc: 0.9076 Epoch 5/15 48000/48000 [==============================] - 506s 11ms/sample - loss: 0.2206 - acc: 0.9193 - val_loss: 0.2402 - val_acc: 0.9137 Epoch 6/15 48000/48000 [==============================] - 445s 9ms/sample - loss: 0.2053 - acc: 0.9250 - val_loss: 0.2242 - val_acc: 0.9183 Epoch 7/15 48000/48000 [==============================] - 549s 11ms/sample - loss: 0.1850 - acc: 0.9313 - val_loss: 0.2217 - val_acc: 0.9225 Epoch 8/15 48000/48000 [==============================] - 424s 9ms/sample - loss: 0.1738 - acc: 0.9355 - val_loss: 0.2486 - val_acc: 0.9081 Epoch 9/15 48000/48000 [==============================] - 371s 8ms/sample - loss: 0.1619 - acc: 0.9394 - val_loss: 0.1968 - val_acc: 0.9276 Epoch 10/15 48000/48000 [==============================] - 308s 6ms/sample - loss: 0.1508 - acc: 0.9440 - val_loss: 0.2144 - val_acc: 0.9249 Epoch 11/15 48000/48000 [==============================] - 313s 7ms/sample - loss: 0.1427 - acc: 0.9461 - val_loss: 0.2131 - val_acc: 0.9261 Epoch 12/15 48000/48000 [==============================] - 353s 7ms/sample - loss: 0.1318 - acc: 0.9513 - val_loss: 0.2076 - val_acc: 0.9288 Epoch 13/15 48000/48000 [==============================] - 453s 9ms/sample - loss: 0.1217 - acc: 0.9542 - val_loss: 0.2002 - val_acc: 0.9308 Epoch 14/15 48000/48000 [==============================] - 463s 10ms/sample - loss: 0.1128 - acc: 0.9581 - val_loss: 0.2003 - val_acc: 0.9323 Epoch 15/15 48000/48000 [==============================] - 329s 7ms/sample - loss: 0.1058 - acc: 0.9610 - val_loss: 0.1977 - val_acc: 0.9348 10000/10000 [==============================] - 17s 2ms/sample - loss: 0.2224 - acc: 0.9280 Wall time: 1h 40min 47s
Improvements
- there's a lot more parameter tuning that could be done
- compare different optimizers and their parameters
- compare different loss functions
- test different weight/bias initializations
- more parameters tuning with either Scikit_Learn GridSearchCV or RandomSearchCV wrapper for keras or hyperas
Note
:
Neural Networks is time and computation consuming, powerful machine is needed to be able to search for results.