Application of representation learning in detecting botne…

Scientific Reports – Application of representation learning in detecting botnet attacks

Botnet detection remains a perennial and critical challenge in cybersecurity. As long as the internet exists, threat actors will devise new ways to create and disguise these malicious networks, making the development of robust detection methods a task that will never be obsolete. Traditional approaches, relying on rigid signatures and manual feature engineering, are often locked in a reactive cycle.

A more critical limitation is their poor generalization; models trained on known botnets frequently fail to detect novel, unseen threats, rendering them vulnerable in real-world scenarios. This paper introduces a robust framework that significantly enhances botnet detection by overcoming these limitations. We propose a novel methodology that combines advanced feature engineering, such as octet splitting for IP addresses, with a sophisticated representation learning technique using the Hilbert space-filling curve to transform network flows into 2D images.

This approach preserves data locality and eliminates the noise introduced by traditional zero-padding. Furthermore, we address the critical issue of class imbalance using a combination of SMOTE, a weighted sampler, and Focal Loss to focus the model on more challenging samples. To rigorously evaluate the model's real-world applicability, we employed a challenging cross-scenario validation strategy, training the model on the Murlo botnet (Scenario 8) and testing it on the completely unseen Rbot botnet (Scenario 10) from the publicly available CTU-13 dataset.

Our proposed model achieved an outstanding accuracy of 98.34% and a weighted F1-score of 98.38%, demonstrating a remarkable ability to generalize to novel botnet attacks. This result validates our approach and highlights the superiority of learned, spatially-aware representations over traditional models, which failed to detect the unseen botnet. Our work presents a significant step towards creating more adaptive and resilient intrusion detection systems capable of handling novel, unseen botnet families.

The dataset analyzed during the current study is the CTU-13 dataset, which is publicly available in the Stratosphere Laboratory repository, [https://www.stratosphereips.org/datasets-ctu13] (https://www.stratosphereips.org/datasets-ctu13). The source code, pre-processing scripts, and model architectures used to generate the results in this study are available in the GitHub repository: https://github.com/occbuu/RepLearningDetectBotnetAttack.

Al-Shurbaji, T. et al. Deep learning-based intrusion detection system for detecting IoT botnet attacks: A review. IEEE Access 13, 11792–11822. https://doi.org/10.1109/access.2025.3526711 (2025). Kundu, P. P., Truong-Huu, T., Chen, L., Zhou, L. & Teo, S. G. Detection and classification of botnet traffic using deep learning with model explanation. IEEE Trans. Dependable Secure Comput. https://doi.org/10.1109/tdsc.2022.3183361 (2024).

Ahmed, A., and Tjortjis, C. Machine learning based IoT-BotNet attack detection using real-time heterogeneous data. In 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET) (2022). Nketia, I. K., Yaokumah, W., & Appati, J. K. A comprehensive review of internet-of-things (IoT) botnet detection techniques. In Smart and Agile Cybersecurity for IoT and IIoT Environments 50–81.

(IGI Global, 2024). Suthar, F., Patel, N. & Khanna, S. V. O. A signature-based botnet (emotet) detection mechanism. Int. J. Eng. Trends Technol. 70(5), 185–193. https://doi.org/10.14445/22315381/ijett-v70i5p220 (2022). Saeed, M. et al. Anomaly detection in 6G networks using machine learning methods. Electronics 12(15), 3300. https://doi.org/10.3390/electronics12153300 (2023). Dasgupta, D., Akhtar, Z.

& Sen, S. Machine learning in cybersecurity: A comprehensive survey. J. Def. Model. Simul. Appl. Methodol. Technol. 19(1), 57–106. https://doi.org/10.1177/1548512920951275 (2022). Xing, Y., Shu, H., Zhao, H., Li, D. & Guo, L. Survey on botnet detection techniques: Classification, methods, and evaluation. Math. Probl. Eng. 2021, 1–24. https://doi.org/10.1155/2021/6640499 (2021). Awad, L., Al-HajBaddar, S., & Sleit, A.

Machine learning and deep learning for botnet detection techniques: A comparative review. In Lecture Notes in Networks and Systems 401–412. (Springer Nature Switzerland, 2023). Alomari, D., Anis, F., Alabdullatif, M., & Aljamaan, H. A survey on botnets attack detection utilizing machine and deep learning models. In Proceedings of the 27th International Conference on Evaluation and Assessment in Software Engineering, 493–498 (2023).

Aversano, L., Bernardi, M. L., Cimitile, M. & Pecori, R. A systematic review on Deep Learning approaches for IoT security. Comput. Sci. Rev. 40, 100389. https://doi.org/10.1016/j.cosrev.2021.100389 (2021). Ali, S. et al. A novel approach of botnet detection using hybrid deep learning for enhancing security in IoT networks. Alex. Eng. J. 103, 88–97. https://doi.org/10.1016/j.aej.2024.05.113 (2024).

Emirmahmutoğlu, E. & Atay, Y. A feature selection-driven machine learning framework for anomaly-based intrusion detection systems. Peer-to-Peer Netw. Appl. https://doi.org/10.1007/s12083-025-01947-4 (2025). Ahmed, U. et al. Signature-based intrusion detection using machine learning and deep learning approaches empowered with fuzzy clustering. Sci. Rep. 15(1), 1726. https://doi.org/10.1038/s41598-025-85866-7 (2025).

Mijwil, M. M., Salem, I. E. & Ismaeel, M. M. The significance of machine learning and deep learning techniques in cybersecurity: A comprehensive review. Iraqi J. Comput. Sci. Math. https://doi.org/10.52866/ijcsm.2023.01.01.008 (2023). Alnajim, A., Habib, S., Islam, M., Thwin, S. & Alotaibi, F. A comprehensive survey of cybersecurity threats, attacks, and effective countermeasures in Industrial Internet of Things.

Technologies 11(6), 161. https://doi.org/10.3390/technologies11060161 (2023). Alshamkhany, M., Alshamkhany, W., Mansour, M., Khan, M., Dhou, S., & Aloul, F. Botnet Attack detection using machine learning. In 2020 14th International Conference on Innovations in Information Technology (IIT) (2020). Pokhrel, S., Abbas, R., & Aryal, B. IoT Security: Botnet detection in IoT using Machine learning. arXiv:2104.02231.

https://doi.org/10.48550/ARXIV.2104.02231. (2021). Buczak, A. L. & Guven, E. A survey of data mining and machine learning methods for cyber security intrusion detection. IEEE Commun. Surv. Tutor. 18(2), 1153–1176. https://doi.org/10.1109/comst.2015.2494502 (2016). Mohale, V. Z. & Obagbuwa, I. C. Evaluating machine learning-based intrusion detection systems with explainable AI: Enhancing transparency and interpretability.

Front. Comput. Sci. https://doi.org/10.3389/fcomp.2025.1520741 (2025). Raghava, N. S., & Sahgal, D. (2012). Botnet: A deadly threat to cyber security. Int. J. Comput. Appl., 44(2). Ahmad, Z., Shahid Khan, A., Wai Shiang, C., Abdullah, J. & Ahmad, F. Network intrusion detection system: A systematic study of machine learning and deep learning approaches. Trans. Emerg. Telecommun. Technol. https://doi.org/10.1002/ett.4150 (2021).

Janabi, A. H., Kanakis, T. & Johnson, M. Survey: Intrusion detection system in software-defined networking. IEEE Access 12, 164097–164120. https://doi.org/10.1109/access.2024.3493384 (2024). Chen, Z. Detection of IoT botnets using decision trees (Doctoral dissertation). ProQuest Dissertations and Theses (2021). https://www.proquest.com/openview/0e58828dd5d3ff63e47cbc0618cd1baf/1?pq-origsite=gscholar&cbl=18750&diss=y.

Akash, N. S., Rouf, S., Jahan, S., Chowdhury, A. & Uddin, J. Botnet detection in IoT devices using Random Forest Classifier with Independent Component Analysis. J. ICT 21(2), 201–232. https://doi.org/10.32890/jict2022.21.2.3 (2022). Fang, W. Real time botnet detection system based on machine learning algorithms. In L. Shen (Ed.), 2022 2nd Conference on High Performance Computing and Communication Engineering (HPCCE 2022) 44 (SPIE, 2023).

Hoang, X. D. & Vu, X. H. An improved model for detecting DGA botnets using random forest algorithm. Inf. Secur. J. Glob. Perspect. 31(4), 441–450. https://doi.org/10.1080/19393555.2021.1934198 (2022). Ren, F., Jiang, Z., Wang, X. & Liu, J. A DGA domain names detection modeling method based on integrating an attention mechanism and deep neural network. Cybersecurity https://doi.org/10.1186/s42400-020-00046-6 (2020).

Aljehane, N. O. et al. Optimizing intrusion detection using intelligent feature selection with machine learning model. Alex. Eng. J. 91, 39–49. https://doi.org/10.1016/j.aej.2024.01.073 (2024). Kasongo, S. M. & Sun, Y. A deep learning method with filter based feature engineering for wireless intrusion detection system. IEEE Access 7, 38597–38607. https://doi.org/10.1109/access.2019.2905633 (2019).

Alkahtani, H. & Aldhyani, T. H. H. Botnet attack detection by using CNN-LSTM model for Internet of Things applications. Secur. Commun. Networks 2021, 1–23. https://doi.org/10.1155/2021/3806459 (2021). Qazi, E. U. H., Almorjan, A. & Zia, T. A one-dimensional convolutional neural network (1D-CNN) based deep learning system for network intrusion detection. Appl. Sci. 12(16), 7986. https://doi.org/10.3390/app12167986 (2022).

Gueriani, A., Kheddar, H., & Mazari, A. C. Enhancing IoT security with CNN and LSTM-based intrusion detection systems. In 2024 6th International Conference on Pattern Analysis and Intelligent Systems (PAIS) (2024) Soofi, A. A., Tahir, M. & Raza, N. Securing the Internet of Things: A comprehensive review of security challenges and artificial intelligence solutions. Found. Univ. J. Eng. Appl. Sci. 4(2), 1–20.

https://doi.org/10.33897/fujeas.v4i2.779 (2024). Alwhbi, I. A., Zou, C. C. & Alharbi, R. N. Encrypted network traffic analysis and classification utilizing machine learning. Sensors 24(11), 3509. https://doi.org/10.3390/s24113509 (2024). Mothukuri, V. et al. Federated-learning-based anomaly detection for IoT security attacks. IEEE Internet Things J. 9(4), 2545–2554. https://doi.org/10.1109/jiot.2021.3077803 (2022).

Nguyen, T. D., Marchal, S., Miettinen, M., Fereidooni, H., Asokan, N., & Sadeghi, A.-R. DÏoT: A federated self-learning anomaly detection system for IoT. In 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). https://doi.org/10.1109/ICDCS.2019.00080 (2019). Vinayakumar, R., Alazab, M., Soman, K. P., Poongodi, P., & Al-Turjman, F. Deep learning for cyber security applications: A comprehensive survey.

https://doi.org/10.36227/techrxiv.16748161.v1. (2021). Wang, W., Jian, S., Tan, Y., Wu, Q. & Huang, C. Representation learning-based network intrusion detection system by capturing explicit and implicit feature interactions. Comput. Secur. 112, 102537. https://doi.org/10.1016/j.cose.2021.102537 (2022). Ho, C. M. K., Yow, K.-C., Zhu, Z. & Aravamuthan, S. Network intrusion detection via flow-to-image conversion and vision transformer classification.

IEEE Access 10, 97780–97793. https://doi.org/10.1109/access.2022.3200034 (2022). Bakour, K. & Ünver, H. M. DeepVisDroid: Android malware detection by hybridizing image-based features with deep learning techniques. Neural Comput. Appl. 33(18), 11499–11516. https://doi.org/10.1007/s00521-021-05816-y (2021). Shen, M. et al. Machine learning-powered encrypted network traffic analysis: A comprehensive survey.

IEEE Commun. Surv. Tutor. 25(1), 791–824. https://doi.org/10.1109/comst.2022.3208196 (2023). Kim, T. & Pak, W. Deep learning-based network intrusion detection using multiple image transformers. Appl. Sci. 13(5), 2754. https://doi.org/10.3390/app13052754 (2023). Rodríguez, E. et al. Transfer-learning-based intrusion detection framework in IoT networks. Sensors 22(15), 5621. https://doi.org/10.3390/s22155621 (2022).

Garcia, S., Grill, M., Stiborek, J. & Zunino, A. An empirical comparison of botnet detection methods. Comput. Secur. 45, 100–123 (2014). He, K., Zhang, X., Ren, S., & Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/CVPR.2016.90 (2016). We extend our gratitude to the creators of the CTU-13 dataset at the Stratosphere Laboratory for making their valuable data publicly available to the research community.

The author received no financial support for the research, authorship, and/or publication of this article. L.N.H. is the sole author of this work and is responsible for the conceptualization, methodology, software, validation, formal analysis, investigation, and writing of the original draft, as well as reviewing and editing the final manuscript. Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material.

You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/. Le Ngoc, H. Application of representation learning in detecting botnet attacks. Sci Rep (2026). https://doi.org/10.1038/s41598-026-40172-8

Summary

This report covers the latest developments in android. The information presented highlights key changes and updates that are relevant to those following this topic.

---