Training Data Expansion and Boosting of Convolutional Neural Networks for Reducing the MNIST Dataset Error Rate

Вадим Васильович Романюк

Abstract


Background. Due to that the preceding approaches for improving the MNIST image dataset error rate do not have a clear structure which could let repeat it in a strengthened manner, the formalization of the performance improvement is considered.

Objective. The goal is to strictly formalize a strategy of reducing the MNIST dataset error rate.

Methods. An algorithm for achieving the better performance by expanding the training data and boosting with ensembles is suggested. The algorithm uses the designed concept of the training data expansion. Coordination of the concept and the algorithm defines a strategy of the error rate reduction.

Results. In relative comparison, the single convolutional neural network performance on the MNIST dataset has been bettered almost by 30 %. With boosting, the performance is 0.21 % error rate meaning that only 21 handwritten digits from 10,000 are not recognized.

Conclusions. The training data expansion is crucial for reducing the MNIST dataset error rate. The boosting is ineffective without it. Application of the stated approach has an impressive impact for reducing the MNIST dataset error rate, using only 5 or 6 convolutional neural networks against those 35 ones in the benchmark work.


Keywords


MNIST; Convolutional neural network; Error rate; Training data expansion; Boosting

Full Text:

PDF

References


Y. LeCun et al., “Gradient-based learning applied to document recognition”, Proc. IEEE, vol. 86, iss. 11, pp. 2278–2324, 1998. doi: 10.1109/5.726791

E. Kussul and T. Baidyk, “Improved method of handwritten digit recognition tested on MNIST database”, Image and Vision Computing, vol. 22, iss. 12, pp. 971–981, 2004. doi: 10.1016/j.imavis.2004.03.008

D. Ciresan et al., “Flexible, high performance convolutional neural networks for image classification”, in Proc. 22nd Int. Joint Conf. Artificial Intelligence, vol. 2, pp. 1237–1242, 2011.

M. Ranzato et al., “Efficient learning of sparse representations with an energy-based model”, Advances in Neural Information Processing Systems, vol. 19, pp. 1137–1144, 2006.

D. Ciresan et al., “Multi-column deep neural networks for image classification”, in Proc. 2012 IEEE Conf. Computer Vision and Pattern Recognition, pp. 3642–3649, 2012. doi: 10.1109/CVPR.2012.6248110

P. Date et al., “Design index for deep neural networks”, Procedia Computer Sci., vol. 88, pp. 131–138, 2016. doi: 10.1016/ j.procs.2016.07.416

D. Ciresan et al., “Deep big simple neural nets excel on handwritten digit recognition”, Neural Computation, vol. 22, no. 12, pp. 3207–3220, 2010. doi: 10.1162/NECO_a_00052

B. Kégl and R. Busa-Fekete, “Boosting products of base classifiers”, in Proc. 26th Annual Int. Conf. Machine Learning, pp. 497–504, 2009. doi: 10.1145/1553374.1553439

V.V. Romanuke, “Boosting ensembles of heavy two-layer perceptrons for increasing classification accuracy in recognizing shifted-turned-scaled flat images with binary features”, J. Inform. Organiz. Sci., vol. 39, no. 1, pp. 75–84, 2015.

N. Srivastava et al., “Dropout: A simple way to prevent neural networks from overfitting”, J. Machine Learning Res., vol. 15, iss. 1, pp. 1929–1958, 2014.

V.V. Romanuke, “Two-layer perceptron for classifying flat scaled-turned-shifted objects by additional feature distortions in training”, J. Uncertain Systems, vol. 9, no. 4, pp. 286–305, 2015.

V.V. Romanuke, “A method of resume-training of discontinuous wear state trackers for composing boosting high-accurate ensembles needed to regard statistical data inaccuracies and shifts”, Problems of Tribology, no. 3, pp. 19–22, 2015.

V.V. Romanuke, “Accuracy improvement in wear state discontinuous tracking model regarding statistical data inaccuracies and shifts with boosting mini-ensemble of two-layer perceptrons”, Problems of Tribology, no. 4, pp. 55–58, 2014.


GOST Style Citations


  1. Gradient-based learning applied to document recognition / Y. LeCun, L. Bottou, Y. Bengio, P. Haffner // Proc. IEEE. – 1998. – 86, iss. 11. – P. 2278–2324. doi: 10.1109/5.726791

  2. Kussul E., Baidyk T. Improved method of handwritten digit recognition tested on MNIST database // Image and Vision Computing. – 2004. – 22, iss. 12. – P. 971–981. doi: 10.1016/j.imavis.2004.03.008

  3. Flexible, high performance convolutional neural networks for image classification / D. Ciresan, U. Meier, J. Masci et al. // Proc. 22nd Int. Joint Conf. Artificial Intelligence. – 2011. – 2. – P. 1237–1242.

  4. Efficient learning of sparse representations with an energy-based model / M. Ranzato, C. Poultney, S. Chopra, Y. LeCun // Advances in Neural Information Processing Systems. – 2006. – 19. – P. 1137–1144.

  5. Ciresan D., Meier U., Schmidhuber J. Multi-column deep neural networks for image classification // 2012 IEEE Conf. Computer Vision and Pattern Recognition. – 2012. – P. 3642–3649. doi: 10.1109/CVPR.2012.6248110

  6. Date P., Hendler J.A., Carothers C.D. Design index for deep neural networks // Procedia Computer Sci. – 2016. – 88. – P. 131–138. doi: 10.1016/j.procs.2016.07.416

  7. Deep big simple neural nets excel on handwritten digit recognition / D. Ciresan, U. Meier, L.M. Gambardella, J. Schmidhuber // Neural Computation. – 2010. – 22, № 12. – P. 3207–3220. doi: 10.1162/NECO_a_00052

  8. Kégl B., Busa-Fekete R. Boosting products of base classifiers // Proc. 26th Annual Int. Conf. Machine Learning. – 2009. –  P. 497–504. doi: 10.1145/1553374.1553439

  9. Romanuke V.V. Boosting ensembles of heavy two-layer perceptrons for increasing classification accuracy in recognizing shifted-turned-scaled flat images with binary features // J. Inform. Organiz. Sci. – 2015. – 39, № 1. – P. 75–84.

  10. Dropout: A simple way to prevent neural networks from overfitting / N. Srivastava, G. Hinton, A. Krizhevsky et al. // J. Machine Learning Res. – 2014. – 15, iss. 1. – P. 1929–1958.

  11. Romanuke V.V. Two-layer perceptron for classifying flat scaled-turned-shifted objects by additional feature distortions in training // J. Uncertain Systems. – 2015. – 9, № 4. – P. 286–305.

  12. Romanuke V.V. A method of resume-training of discontinuous wear state trackers for composing boosting high-accurate ensembles needed to regard statistical data inaccuracies and shifts // Problems of Tribology. – 2015. – № 3. – P. 19–22.

  13. Romanuke V.V. Accuracy improvement in wear state discontinuous tracking model regarding statistical data inaccuracies and shifts with boosting mini-ensemble of two-layer perceptrons // Problems of Tribology. – 2014. – № 4. – P. 55–58.




DOI: https://doi.org/10.20535/1810-0546.2016.6.84115

Refbacks

  • There are currently no refbacks.




Copyright (c) 2017 NTUU KPI