PPO-Based Neural Architecture Search for Efficient and Hardware-Aware Windows Malware Detection in Cybersecurity
Article Sidebar
Main Article Content
Abstract
The rapid growth of malicious software targeting Windows operating systems has created significant challenges for modern cybersecurity systems. Traditional malware detection approaches often rely on manually designed models or signature-based techniques, which struggle to identify newly emerging and highly obfuscated malware variants. Although recent deep learning methods have improved detection accuracy, many of these models depend on fixed neural architectures and require extensive manual tuning, limiting their adaptability and efficiency. To address these challenges, this study proposes a reinforcement learning–driven Intelligent Agent framework for automated neural architecture search in Windows malware detection. The proposed system employs a reinforcement learning agent based on Proximal Policy Optimization to explore a simplified architecture search space and automatically construct an effective deep learning model. The approach was evaluated using the EMBER Dataset, a widely used benchmark dataset for malware classification. Experimental results demonstrate that the architecture discovered by the proposed Intelligent Agent achieves strong detection performance, reaching an accuracy of 97.3 while also outperforming several baseline approaches in terms of Precision, Recall, F1-Score, and AUC metrics. These findings indicate that integrating reinforcement learning with neural architecture search can significantly enhance malware detection performance while reducing the need for manual model design. In addition to achieving high detection accuracy, the proposed approach considers computational efficiency by enabling the discovery of hardware-aware neural architectures suitable for real-world deployment. The proposed framework highlights the potential of intelligent agent–based systems in developing adaptive and efficient cybersecurity solutions capable of addressing evolving malware threats.
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Licensed under a CC-BY license: https://creativecommons.org/licenses/by-nc-sa/4.0/
How to Cite
References
Y. M. Mohialden, M. T. Younis, S. A. Salman, and E. A. W. Hachim, "Challenges and advancements in quantum cryptography: Standardization, security risks, and practical implementations," Proceedings of the International Conference on Applied Innovations in IT (ICAIIT), vol. 13, no. 4, pp. 307–314, 2025.
J. Ferdous, R. Islam, A. Mahboubi, and M. Z. Islam, "A survey on ML techniques for multi-platform malware detection: Securing PC, mobile devices, IoT, and cloud environments," Sensors, vol. 25, no. 4, p. 1153, 2025.
M. Tanha and S. Kafaie, "A review of explainable machine learning for Android malware detection and analysis," IEEE Access, 2025.
A. Basim, I. M. Ali, R. A. Lateef, and Z. L. Ali, "Enhanced ransomware traffic detection using a hybrid ResNeSt101 and Isolation Forest framework," in Third International Conference on Emerging Trends in AI and Computational Technologies (ICONEST 2025), Proc. SPIE, vol. 14141, pp. 87–96, 2026.
Y. Dehfouli and A. Habibi Lashkari, "Memory analysis for malware detection: A comprehensive survey using the OSCAR methodology," ACM Computing Surveys, vol. 58, no. 4, pp. 1–58, 2025.
M. Alshoulie and A. Mehmood, "Deep learning approaches for malware detection: A comprehensive review of techniques, challenges, and future directions," IEEE Access, 2025.
I. K. Abbas, M. S. Mahdi, A. Basim, and K. I. Abbas, "An agent-based reinforcement learning framework for dynamic cryptographic security," Dijlah Journal of Engineering Sciences (DJES), vol. 3, no. 1, pp. 377–391, Mar. 2026.
S. Alzahrani, Y. Xiao, S. Asiri, J. Zheng, and T. Li, "A survey of ransomware detection methods," IEEE Access, 2025.
S. F. Ali, M. R. Abdulrazzaq, and M. T. Gaata, "Learning techniques-based malware detection: A comprehensive review," Mesopotamian Journal of CyberSecurity, vol. 5, no. 1, pp. 273–300, 2025.
W. Almobaideen, O. Abu Alghanam, M. Abdullah, S. B. Hussain, and U. Alam, "Comprehensive review on machine learning and deep learning techniques for malware detection in Android and IoT devices," International Journal of Information Security, vol. 24, no. 3, p. 110, 2025.
M. S. Mahdi, "A deep learning–based hybrid neural network model for malware detection," Journal of Techniques, vol. 8, no. 1, pp. 62–70, 2026.
X. Huang, L. Ma, W. Yang, and Y. Zhong, "A method for Windows malware detection based on deep learning," Journal of Signal Processing Systems, vol. 93, no. 2–3, pp. 265–273, 2020.
E. Amer and I. Zelinka, "A dynamic Windows malware detection and prediction method based on contextual understanding of API call sequence," Computers & Security, vol. 92, p. 101760, 2020.
F. Çatak, A. Yazı, O. Elezaj, and J. Ahmed, "Deep learning based sequential model for malware analysis using Windows exe API calls," PeerJ Computer Science, vol. 6, p. e285, 2020.
C. Wang, Z. Zhao, F. Wang, and Q. Li, "A novel malware detection and family classification scheme for IoT based on DEAM and DenseNet," Security and Communication Networks, vol. 2021, pp. 1–16, 2021.
N. Azeez, O. Odufuwa, S. Misra, J. Oluranti, and R. Damaševičius, "Windows PE malware detection using ensemble learning," Informatics, vol. 8, no. 1, p. 10, 2021.
M. Dener, G. Ok, and A. Orman, "Malware detection using memory analysis data in big-data environment," Applied Sciences, vol. 12, no. 17, p. 8604, 2022.
V. Ravi and M. Alazab, "Attention-based convolutional neural network deep learning approach for robust malware classification," Computational Intelligence, vol. 39, no. 1, pp. 145–168, 2022.
F. ALGorain and J. Clark, "Bayesian hyper-parameter optimisation for malware detection," Electronics, vol. 11, no. 10, p. 1640, 2022.
S. Kudrekar and U. Rani, "Classification of malware using multinomial linked latent modular double Q learning," Indonesian Journal of Electrical Engineering and Computer Science, vol. 28, no. 1, p. 577, 2022.
M. Kim, H. Cho, and J. Yi, "Large-scale analysis on anti-analysis techniques in real-world malware," IEEE Access, vol. 10, pp. 75802–75815, 2022.
A. Koçak, E. Söğüt, M. Alkan, and O. Erdem, "Detection of different Windows PE malware using machine learning methods," Politeknik Dergisi, vol. 26, no. 3, pp. 1185–1197, 2023.
I. E. Maulani, T. Herdianto, M. O. Laksana, F. Syawaludin, and K. Komarudin, "Exploring the effectiveness of artificial intelligence in detecting malware and improving cyber security in computer networks," Eduvest: Journal of Universal Studies, vol. 3, no. 4, 2023.
C. Du et al., "Toward detecting malware based on process-aware behaviors," Security and Communication Networks, vol. 2023, pp. 1–16, 2023.
D. Syeda and M. Asghar, "Dynamic malware classification and API categorisation of Windows portable executable files using machine learning," Applied Sciences, vol. 14, no. 3, p. 1015, 2024.
I. Zada et al., "Fine-tuning cyber security defenses: Evaluating supervised machine learning classifiers for Windows malware detection," Computers, Materials & Continua, vol. 80, no. 2, pp. 2917–2939, 2024.
H. Gururaja, N. Khandige, N. Nayak, R. Srivatsan, and N. S., "Ensemble learning for robust malware detection in the Windows 7 environment," International Research Journal of Advanced Engineering Hub, vol. 2, no. 2, pp. 261–270, 2024.
M. Imran, A. Appice, and D. Malerba, "Evaluating realistic adversarial attacks against machine learning models for Windows PE malware detection," Future Internet, vol. 16, no. 5, p. 168, 2024.
S. Khan and M. Nauman, "Interpretable detection of malicious behavior in Windows portable executables using multi-head 2D transformers," Big Data Mining and Analytics, vol. 7, no. 2, pp. 485–499, 2024.
"EMBER: Dataset for training static PE malware machine learning models", Kaggle, 2026. [Online]. Available: https://www.kaggle.com/datasets/dhoogla/ember-2018-v2-features. [Accessed: Mar. 14, 2026].
M. Adnan, M. O. Imam, M. F. Javed, and I. Murtza, "Improving spam email classification accuracy using ensemble techniques: A stacking approach," International Journal of Information Security, vol. 23, no. 1, pp. 505–517, 2024.