Hand-written Digits Recognition with Mnist dataset

The complete example is available from https://github.com/viebboy/PyGOP/tree/master/examples/train_mnist.py . ‘train_mnist.py’ is the only source code we need, beside having PyGOP installed

Since we will use Mnist dataset available from Keras, we will simply create a data function by loading Mnist from keras and create a generator to generate mini-batches of data according to the batch size and decide whether to shuffle the data depending on the train or test set:

def data_func(data_argument):
    """
    Data function of mnist for PyGOP models which should produce a generator and the number
    of steps per epoch

    Args:
        data_argument: a tuple of batch_size and split ('train' or 'test')

    Return:
        generator, steps_per_epoch

    """
    batch_size, split = data_argument

    # load dataset from keras datasets
    (x_train, y_train), (x_test, y_test) = mnist.load_data()

    if split == 'train':
        X = x_train
        Y = y_train
    else:
        X = x_test
        Y = y_test

    # reshape image to vector
    X = np.reshape(X, (-1, 28 * 28))
    # convert to one-hot vector of classes
    Y = to_categorical(Y, 10)
    N = X.shape[0]

    steps_per_epoch = int(np.ceil(N/ float(batch_size)))

    def gen():
        while True:
            indices = np.arange(N)
            # if train set, shuffle data in each epoch
            if split == 'train':
                np.random.shuffle(indices)

            for step in range(steps_per_epoch):
                start_idx = step * batch_size
                stop_idx = min(N, (step + 1) * batch_size)
                idx = indices[start_idx:stop_idx]
                yield X[idx], Y[idx]

    # it's important to return generator object, which is gen() with the bracket
    return gen(), steps_per_epoch

It’s worth noting that PyGOP will pickle data_argument passed to data_func, we should not pass the actual dataset through data_argument because it is computationally expensive. data_argument should ideally contain hyper-parameters of the generator only, such as batch_size and split above, or as you will see from the next example, a path to the actual data. In this way, we can perform the data reading step inside data_func to avoid many write and read operations of the actual data.

The main function then continues with parsing the two arguments: the name of the model and the type of computation (cpu/gpu):

try:
    opts, args = getopt.getopt(argv, "m:c:")
except getopt.GetoptError:
    print('train_mnist.py -m <model> -c <computation option cpu/gpu>')
    sys.exit(2)

for opt, arg in opts:
    if opt == '-m':
        model_name = arg
    if opt == '-c':
        computation = arg

The type of model/algorithm is in model_name variable, and the type of computation is in computation variable. The main function continues with creating the corresponding model instance and setting the model’s parameters:

# input 728 raw pixel values
# output 10 class probability
input_dim = 28 * 28
output_dim = 10

if computation == 'cpu':
    search_computation = ('cpu', 8)
    finetune_computation = ('cpu', )
else:
    search_computation = ('gpu', [0, 1, 2, 3])
    finetune_computation = ('gpu', [0, 1, 2, 3])

if model_name == 'hemlgop':
    Model = models.HeMLGOP
elif model_name == 'homlgop':
    Model = models.HoMLGOP
elif model_name == 'hemlrn':
    Model = models.HeMLRN
elif model_name == 'homlrn':
    Model = models.HoMLRN
elif model_name == 'pop':
    Model = models.POP
elif model_name == 'popfast':
    Model = models.POPfast
elif model_name == 'popmemo':
    Model = models.POPmemO
elif model_name == 'popmemh':
    Model = models.POPmemH
else:
    raise Exception('Unsupported model %s' % model_name)

# create model
model = Model()
model_name += '_mnist'

# get default parameters and assign some specific values
params = model.get_default_parameters()

tmp_dir = os.path.join(os.getcwd(), 'tmp')
if not os.path.exists(tmp_dir):
    os.mkdir(tmp_dir)

params['tmp_dir'] = tmp_dir
params['model_name'] = model_name
params['input_dim'] = input_dim
params['output_dim'] = output_dim
params['metrics'] = ['acc', ]
params['loss'] = 'categorical_crossentropy'
params['output_activation'] = 'softmax'
params['convergence_measure'] = 'acc'
params['direction'] = 'higher'
params['search_computation'] = search_computation
params['finetune_computation'] = finetune_computation
params['output_activation'] = 'softmax'
params['input_dropout'] = 0.2
params['weight_constraint'] = 3.0
params['weight_constraint_finetune'] = 3.0
params['optimizer'] = 'adam'
params['lr_train'] = (1e-3, 1e-4, 1e-5)
params['epoch_train'] = (60, 60, 60)
params['lr_finetune'] = (1e-3, 1e-4, 1e-5)
params['epoch_finetune'] = (60, 60, 60)
params['direct_computation'] = False
params['max_block'] = 5
params['block_size'] = 40
params['max_layer'] = 4
params['max_topology'] = [200, 200, 200, 200]

To train the model instance, we simply call the fit() method from the model instance, using train_func as specified above:

batch_size = 64
start_time = time.time()

performance, _, _ = model.fit(params,
                              train_func=data_func,
                              train_data=[batch_size, 'train'],
                              val_func=None,
                              val_data=None,
                              test_func=data_func,
                              test_data=[batch_size, 'test'],
                              verbose=True)

stop_time = time.time()

In order to run the script using Heterogeneous Multilayer Operational Perceptron (HeMLGOP) algorithm, for example, with cpu, simply run the following command:

python train_mnist.py -m hemlgop -c cpu

This will train the model with 8 parallel threads on cpu. The number of cpu threads or the gpu devices can be set within train_mnist.py