Investigating Artificial Intelligence Systems Through the Use of Constrained Deep Neural Networks

Deep neural networks have significantly advanced the field of image and text categorization, pushing the boundaries of machine learning. However, designing efficient neural network architectures remains a challenge, often requiring complex and costly methods to find optimal configurations. This paper introduces a novel approach to architectural design through a restricted search method, focusing on creating networks that are both cost-effective and fast, suitable for AI systems with strict memory and time limitations, particularly in near-sensor applications. Neural networks now surpass traditional machine learning techniques in various computational perception tasks. Despite their success, deploying these advanced models on mobile and IoT devices is computationally challenging, leading to reliance on cloud-based solutions. Such dependence increases communication costs and potential system inoperability during connectivity outages. Our method addresses these issues, offering a viable solution for efficient, local deployment of advanced neural networks in resource-constrained environments. We propose a conceptual framework that leverages a Deep neural networks (DNN) approach to decide whether data should be processed locally or sent to the cloud, optimizing both computational resources and performance. Our findings suggest that this method requires sending only 52% of test data to the server, achieving an overall system accuracy of 48%. This significantly enhances the efficiency of client-server models and supports the implementation of AI capabilities on local devices. 

By employing a strategic search for computational models based on content extraction, we improve the efficiency and speed of AI operations. Our experiments demonstrate the practicality and effectiveness of this approach, which has also been tested on actual hardware, offering a promising direction for enhancing AI applications in resource-constrained environments.

[1] Chollet, F. (2017). Xception: Deep Learning with Depthwise Separable Convolutions. arXiv preprint arXiv:1610.02357.

[2] Figurnov, M., Collins, M. D., Zhu, Y., Zhang, L., Huang, J., Vetrov, D., & Salakhutdinov, R. (2017). Spatially adaptive computation time for residual networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1039-1048).

[3] Huang, G., Chen, D., Li, T., Wu, F., Van Der Maaten, L., & Weinberger, K. Q. (2017). Multiscale dense networks for resource efficient image classification. arXiv preprint arXiv:1703.09844.

[4] Bolukbasi, T., Wang, J., Dekel, O., & Saligrama, V. (2017, July). Adaptive neural networks for efficient inference. In International Conference on Machine Learning (pp. 527-536). PMLR..

[5] Lechervy, A., & Jurie, F. (2023). Multi-Exit Resource-Efficient Neural Architecture for Image Classification with Optimized Fusion Block. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 1486-1491)..

[6] He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition.

[7] In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR) (pp. 770-778).

[8] LeCun, Y., Bengio, Y., & Hinton, G. (2015). Deep learning. Nature, 521(7553), 436-444.

[9] Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep Learning (Vol. 1). MIT press Cambridge.

[10] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., ... & Hassabis, D. (2016). Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587), 484-489.

[11] Redmon, J., & Farhadi, A. (2018). YOLOv3: An Incremental Improvement. arXiv preprint arXiv:1804.02767.

[12] Wang, Z., & He, B. (2016). A novel deep learning method for imbalanced fault classification of machinery. Mechanical Systems and Signal Processing, 72, 303-315.

[13] 1Xu, B., Wang, N., Chen, T., & Li, M. (2015). Empirical Evaluation of Rectified Activations in Convolutional Network. arXiv preprint arXiv:1505.00853.

[14] Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.

[15] Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2017). ImageNet classification with deep convolutional neural networks. Communications of the ACM, 60(6), 84-90.

[16] Hinton, G., Deng, L., Yu, D., Dahl, G. E., Mohamed, A. R., Jaitly, N., ... & Kingsbury, B. (2012). Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 29(6), 82-97.

[17] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., ... & Petersen, S. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529-533.

[18] Wang, S., Tang, J., Zou, W., & Hou, J. (2017). FANG: A Fast and Scalable Word Embedding Approach. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management (CIKM) (pp. 1857-1860).

[19] Tan, M., & Le, Q. (2019, May). Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning (pp. 6105-6114). PMLR..

[20] Elsken, T., Metzen, J. H., & Hutter, F. (2019). Neural architecture search: A survey. Journal of Machine Learning Research, 20(55), 1-21..

[21] Zoph, B., & Le, Q. V. (2016). Neural architecture search with reinforcement learning. arXivpreprint arXiv:1611.01578..

[22] Liu, H., Simonyan, K., & Yang, Y. (2018). Darts: Differentiable architecture search. arXiv preprint arXiv:1806.09055.

[23] Real, E., Aggarwal, A., Huang, Y., & Le, Q. V. (2019, July). Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence (Vol. 33, No. 01, pp. 4780-4789).

[24] Manishimwe, A., Alexander, H., Kaluuma, H., & Dida, M. (2021). Integrated mobile application based on machine learning for East Africa stock market. Journal of Information Systems Engineering & Management, 6(3), em0143.