Improving IoT Botnet Anomaly Detection Based on Dynamic Feature Selection and Hybrid Processing
Subject Areas : electrical and computer engineeringBoshra Pishgoo 1 , Ahmad akbari azirani 2 *
1 - Iran University of Science and Technology
2 - Iran University of Science and Technology
Keywords: Dynamic feature selection, botnet anomaly detection, Internet of things, batch/stream hybrid processing,
Abstract :
The complexity of real-world applications, especially in the field of the Internet of Things, has brought with it a variety of security risks. IoT Botnets are known as a type of complex security attacks that can be detected using machine learning tools. Detection of these attacks, on the one hand, requires the discovery of their behavior patterns using batch processing with high accuracy, and on the other hand, must be operated in real time and adaptive like stream processing. This highlights the importance of using batch/stream hybrid processing techniques for botnet detection. Among the important challenges of these processes, we can mention the selection of appropriate features to build basic models and also the intelligent selection of basic models to combine and present the final result. In this paper, we present a solution based on a combination of stream and batch learning methods with the aim of botnet anomaly detection. This approach uses a dynamic feature selection method that is based on a genetic algorithm and is fully compatible with the nature of hybrid processing. The experimental results in a data set consisting of two known types of botnets indicate that on the one hand, the proposed approach increases the speed of hybrid processing and reduces the detection time of the botnets by reducing the number of features and removing inappropriate features, and on the other hand, increases accuracy by selecting appropriate models for combination.
References:
[1] M. Antonakakis, et al., "Understanding the mirai botnet," in Proc. 26th USENIX Security Symp., pp. 1093-1110, Vancouver, Canada, 16-18 Aug. 2017.
[2] A. Marzano, et al., "The evolution of bashlite and mirai IoT botnets," in Proc. IEEE Symp. on Computers and Communications, ISCC'18, pp. 813-818, Natal, Brazil,25-28 Jun. 2018.
[3] S. García, A. Zunino, and M. Campo, "Survey on network‐based botnet detection methods," Security and Communication Networks, vol. 7, no. 5, pp. 878-903, May. 2014.
[4] R. Alhajri, R. Zagrouba, and F. Al-Haidari, "Survey for anomaly detection of IoT botnets using machine learning auto-encoders," Int. J. Appl. Eng. Res, vol. 14, no. 10, pp. 2417-2421, Jul. 2019.
[5] R. Azmi and B. Pishgoo, "STLR: a novel danger theory based structural TLR algorithm," The ISC International J. of Information Security, vol. 5, no. 2, pp. 209-225, Mar. 2014.
[6] R. Azmi and B. Pishgoo, "SHADuDT: secure hypervisor-based anomaly detection using danger theory," Computers & Security, vol. 39, no. 1, pp. 268-288, Nov. 2013.
[7] L. Yin, L. Qin, Z. Jiang, and X. Xu, "A fast parallel attribute reduction algorithm using Apache Spark," Knowledge-Based Systems, vol. 212, Article ID: 106582, Jan. 2021.
[8] Y. Wu and J. Tang, "Research progress of attribute reduction based on rough set in context of big data," Computer Engineering and Applications, vol. 55, no. 6, pp. 31-38, May 2019.
[9] I. A. Gheyas and L. S. Smith, "Feature subset selection in large dimensionality domains," Pattern Recognition, vol. 43, no. 1, pp. 5-13, Jan. 2010.
[10] D. B. Skillicorn and S. M. McConnell, "Distributed prediction from vertically partitioned data," J. of Parallel and Distributed Computing, vol. 68, no. 1, pp. 16-36, Jan. 2008.
[11] M. Riahi-Madvar, A. A. Azirani, B. Nasersharif, and B. Raahemi, "A new density-based subspace selection method using mutual information for high dimensional outlier detection," Knowledge-Based Systems, vol. 216, Article ID: 106733, Mar. 2021.
[12] M. Banerjee and S. Chakravarty, "Privacy preserving feature selection for distributed data using virtual dimension," in Proc. of the 20th ACM Int. Conf. on Information and Knowledge Management, pp. 2281-2284, Glasgow, Scotland, UK, 24-28 Oct. 2011.
[13] M. Bramer, Principles of Data Mining, vol. 180, pp. 231-238, London: Springer, 2007.
[14] J. Qian, P. Lv, X. Yue, C. Liu, and Z. Jing, "Hierarchical attribute reduction algorithms for big data using MapReduce," Knowledge-Based Systems, vol. 73, no. 1, pp. 18-31, Jan. 2015.
[15] H. Chen, T. Li, Y. Cai, C. Luo, and H. Fujita, "Parallel attribute reduction in dominance-based neighborhood rough set," Information Sciences, vol. 373, pp. 351-368, Dec. 2016.
[16] W. Ding, J. Wang, and J. Wang, "Multigranulation consensus fuzzy-rough based attribute reduction," Knowledge-Based Systems, vol. 198, Article IDL. 105945, Jun. 2020.
[17] H. Kalkan and B. Çetisli, "Online feature selection and classification," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, ICASSP’11, pp. 2124-2127, Prague, Czech Republic, 22-27 May 2011.
[18] D. Levi and S. Ullman, "Learning to classify by ongoing feature selection," Image Vision Comput, vol. 28, no. 4, pp. 715-723, Jun. 2010.
[19] N. AlNuaimi, M. M. Masud, M. A. Serhani, and N. Zaki, "Streaming feature selection algorithms for big data: a survey," Applied Computing and Informatics, vol. 18, no. 1-2, pp. 113-135, Jul. 2020.
[20] N. Parveen and M. Ananthi, "Data processing for large database using feature selection," in Proc. 2nd Int. Conf. on Computing and Communications Technologies, ICCCT'17, pp. 321-326, Chennai, India, 23-24 Feb. 2017.
[21] L. Brezočnik, I. Fister, and V. Podgorelec, "Swarm intelligence algorithms for feature selection: a review," Applied Sciences, vol. 8, no. 9, Article ID:. 1521, Sept 2018.
[22] N. Abd-Alsabour, "A review on evolutionary feature selection," in Proc. European Modelling Symp., pp. 20-26, Pisa, Italy, 21-23 Oct. 2014.
[23] N. Heidari, R. Azmi, and B. Pishgoo, "Fabric textile defect detection, by selecting a suitable subset of wavelet coefficients, through genetic algorithm," International J. of Image Processing, vol. 5, no. 1, pp. 25-35, Jan. 2011. [24] R. Azmi, B. Pishgoo, N. Norozi, M. Koohzadi, and F. Baesi, "A hybrid GA and SA algorithms for feature selection in recognition of hand-printed Farsi characters," in Proc. IEEE Int. Conf. on Intelligent Computing and Intelligent Systems, vol. 3, pp. 384-387, Xiamen, China ,29-31Oct. 2010.
[25] Y. Xing, H. Shu, H. Zhao, D. Li, and L. Guo, "Survey on botnet detection techniques: classification, methods, and evaluation," Mathematical Problems in Engineering, vol. 2021, no. 1, pp. 1-24, Jan. 2021.
[26] S. Almutairi, S. Mahfoudh, S. Almutairi, and J. S. Alowibdi, "Hybrid botnet detection based on host and network analysis," J. of Computer Networks and Communications, vol. 2020, no. 1, pp. 1-17, Jan. 2020.
[27] A. Karim, R. B. Salleh, M. Shiraz, et al., "Botnet detection techniques: review, future trends, and issues," J. of Zhejiang University-Science C, vol. 15, no. 11, pp. 943-983, Nov. 2014.
[28] K. Sinha, V. Arun, and B. Julian, "Tracking temporal evolution of network activity for botnet detection," https:// arxiv.org/abs/1908.03443, 2013.
[29] W. T. Strayer, R. Walsh, C. Livadas, and D. Lapsley, "Detecting botnets with tight command and control," in Proc. 31st IEEE Conf. on Local Computer Networks, pp. 195-202, Tampa, FL, USA, 14-16 Nov. 2006.
[30] E. Passerini, R. Paleari, L. Martignoni, and D. Bruschi, "Fluxor: detecting and monitoring fast-flux service networks," in Proc. Int. Conf. on Detection of Intrusions and Malware and Vulnerability Assessment, pp. 186-206, Paris, France, 10-11 Jul. 2008.
[31] T. F. Yen and M. K. Reiter, "Traffic aggregation for malware detection," in Proc. Int. Conf. on Detection of Intrusions and Malware, and Vulnerability Assessment, pp. 207-227, Paris, France, 10-11 Jul. 2008..
[32] S. Kondo and N. Sato, "Botnet traffic detection techniques by C&C session classification using SVM," in Proc. Int. Workshop on Security, pp. 91-104, Nara, Japan, 29-31 Oct. 2007.
[33] J. François, S. Wang, and T. Engel, "BotTrack: tracking botnets using NetFlow and PageRank," in Proc. Int. Conf. on Research in Networking, pp. 1-14, Valencia, Spain, 9-13 May 2011.
[34] G. Gu, R. Perdisci, J. Zhang, and W. Lee, "Botminer: clustering analysis of network traffic for protocol-and structure-independent botnet detection," in Proc. USENIX Security Symp., pp. 139-154, San Jose, CA, USA, 28 Jul.-1 Aug. 2008.
[35] W. Jung, H. Zhao, M. Sun, and G. Zhou, "IoT botnet detection via power consumption modeling," Smart Health, vol. 15, Article ID: 100103, Mar. 2020.
[36] Y. Zhang and Z. O. U. Fu-Tai, "Detection method of malicious domain name based on knowledge map," Communications Technology, vol. 53, no. 1, pp. 168-173, Jan. 2020.
[37] C. Yin, Research on Network Anomaly Detection Technology Based on Deep Learning, University of Information Engineering, Strategic Support Forces, Zhengzhou, China, 2018.
[38] R. Vinayakumar, et al., "A visualized botnet detection system based deep learning for the internet of things networks of smart cities," IEEE Trans. on Industry Applications, vol. 56, no. 4, pp. 4436-4456, Feb. 2020.
[39] S. I. Popoola, et al., "Federated deep learning for zero-day botnet attack detection in IoT edge devices," IEEE Internet of Things J., vol. 9, no. 9, pp. 3930-3944, Jul. 2021.
[40] A. Almomani, "Fast-flux hunter: a system for filtering online fast-flux botnet," Neural Computing and Applications, vol. 29, no. 7, pp. 483-493, Aug. 2018.
[41] M. Alauthman, N. Aslam, M. Alkasassbeh, S. Khan, A. AL-qerem, and K. K. Raymond Choo, "An efficient reinforcement learning-based botnet detection approach," J. of Network and Computer Applications, vol. 52, Article ID: 102479, Jan. 2019.
[42] H. T. Nguyen, Q. D. Ngo, D. H. Nguyen, et al., "PSI-rooted subgraph: a novel feature for iot botnet detection using classifier algorithms," ICT Express, vol. 6, no. 2, pp. 128-138, Jun. 2020.
[43] D. Zhuang and J. M. Chang, "PeerHunter: detecting peerto-peer botnets through community behavior analysis," in Proc. of the IEEE Conf. on Dependable and Secure Computing, pp. 493-500, Taipei, Taiwan, 7-10 Aug.. 2017.
[44] M. Habib, I. Aljarah, H. Faris, and S. Mirjalili, "Multiobjective particle swarm optimization for botnet detection in internet of things," Evolutionary Machine Learning Techniques, pp. 203-209, Berlin: Germany, Springer, 2020.
[45] A. Al Shorman, H. Faris, and I. Aljarah, "Unsupervised intelligent system based on one class support vector machine and Grey Wolf optimization for IoT botnet detection," J. of Ambient Intelligence and Humanized Computing, vol. 11, no. 7, pp. 2809-2825, Jul. 2020.
[46] S. Y. Huang, C. H. Mao, and H. M. Lee, "Fast-flux service network detection based on spatial snapshot mechanism for delay-free detection," in Proc. of the 5th ACM Symposium on Information, Computer and Communications Security, pp. 101-111, ، Beijing China , 13-16 Apr. 2010.
[47] S. Garg, M. Guizani, S. Guo, and C. Verikoukis, "Guest editorial special section on AI-driven developments in 5G-envisioned industrial automation: big data perspective," IEEE Trans. on Industrial Informatics, vol. 16, no. 2, pp. 1291-1295, Nov. 2020.
[48] X. Wang, Q. Yang, and X. Jin, "Periodic communication detection algorithm of botnet based on quantum computing," J. of Quantum Electronics, vol. 33, no. 2, pp. 182-187, Mar. 2016.
[49] M. Albanese, S. Jajodia, and S. Venkatesan, "Defending from stealthy botnets using moving target defenses," IEEE Security & Privacy, vol. 16, no. 1, pp. 92-97, Feb. 2018.
[50] Z. Zha, A. Wang, Y. Guo, D. Montgomery, and S. Chen, "BotSifter: an SDN-based online bot detection framework in data centers," in Proc. of the IEEE Conf. on Communications and Network Security, CNS’19, pp. 142-150, Washington, D.C., USA, 10-12 Jun. 2019.
[51] G. Spathoulas, N. Giachoudis, G. P. Damiris, and G. Theodoridis, "Collaborative blockchain-based detection of distributed denial of service attacks based on internet of things botnets," Future Internet, vol. 11, Article ID: 226, Oct. 2019.
[52] J. Warren and N. Marz, Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Simon and Schuster, 2015.
[53] B. Twardowski and D. Ryzko, "Multi-agent architecture for real-time big data processing," in Proc. IEEE/WIC/ACM Int.Joint Conf. on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 3, pp. 333-337, Warsaw, Poland, 11-13 Aug. 2014.
[54] M. Di Capua, E. Di Nardo, and A. Petrosino, "An architecture for sentiment analysis in twitter," in Proc. of Int. Conf. on E-learning, Germany, 10 pp., Berlin, Germany, 11-12 Sept. 2015.
[55] V. Nair, "Aligning Machine Learning for the Lambda Architecture," 2015.
[56] -, Madrid. Lambdoop. Retrieved from www.lambdoop.com, 2014.
[57] -, MemSQL. The Lambda Architecture Simplified, 2016.
[58] V. Astakhov and M. Chayel, Lambda Architecture for Batch and Real-Time Processing on AWS with Spark Streaming and Spark SQL, Amazon Web Services, p. 12, 2015.
[59] S. P. T. Krishnan and J. L. U. Gonzalez, Building Your Next Big Thing with Google Cloud Platform: A Guide for Developers and Enterprise Architects, Apress, 2015.
[60] W. Fan and A. Bifet, "Mining big data: current status, and forecast to the future," ACM SIGKDD Explorations Newsletter, vol. 14, no. 2, pp. 1-5, Apr. 2013.
[61] S. Landset, T. M. Khoshgoftaar, A. N. Richter, and T. Hasanin, "A survey of open source tools for machine learning with big data in the Hadoop ecosystem," J. of Big Data, vol. 2, no. 1, pp. 1-36, Dec. 2015.
[62] A. Bifet, "Mining big data in real time," Informatica, vol. 37, no. 1, pp. 15-20, Jan. 2013.
[63] A. Mahesh and P. Manimegalai, "An efficient data processing architecture for smart environments using large scale machine learning," IIOAB J., Special Issue, Emerging Technologies in Networking and Security, vol. 7, no. 9, pp. 795-803, Aug. 2016.
[64] C. H. Kumar and A. S. Sangari, "An efficient distributed data processing method for smart environment," Indian J. Sci. Technol., vol. 9, no. 31, pp. 380-384, Aug. 2016.
[65] X. Liu and P. S. Nielsen, "Scalable prediction-based online anomaly detection for smart meter data," Information Systems, vol. 77, no. 3, pp. 34-47, Sept. 2018.
[66] G. Iuhasz, D. Pop, and I. Dragan, "Architecture of a scalable platform for monitoring multiple big data frameworks," Scalable Computing: Practice and Experience, vol. 17, no. 4, pp. 313-321, Oct. 2016.
[67] M. Kiran, et al., "Lambda architecture for cost-effective batch and speed big data processing," in Proc. IEEE Int Conf. on Big Data (Big Data), pp. 2785-2792, Santa Clara, CA, USA, 29 Oct.-1 Nov. 2015.
[68] -, Oryx 1, Retrieved from https://github.com/certxg/oryx-1, 2013.
[69] -,Oryx2, Retrieved from http://oryx.io/, 2014.
[70] R. C. Fernandez, et al., "Liquid: unifying nearline and offline big data integration," in Proc. 7th Biennial Conf. on Innovative Data Systems Research, CIDR’15, 8 pp., Asilomar, CA, USA, 4-7 Jan. 2105.
[71] D. Namiot, "On big data stream processing," International J. of Open Information Technologies, vol. 3, no. 8, pp. 48-51, aUG. 2015.
[72] L. Magnoni, et al., "Monitoring WLCG with lambda-architecture: a new scalable data store and analytics platform for monitoring at petabyte scale," J. of Physics: Conf. Series, vol. 664, no. 5, Article ID:. 052023, Dec. 2015.
[73] F. Yang, et al., The RADStack: Open Source Lambda Architecture for Interactive Analytics, 2017.
[74] B. Pishgoo, A. A. Azirani, and B. Raahemi, "A hybrid distributed batch-stream processing approach for anomaly detection," Information Sciences, vol. 543, pp. 309-327, Jan. 2021.
[75] J. Cai, J. Luo, S. Wang, and S. Yang, "Feature selection in machine learning: a new perspective," Neurocomputing, vol. 300, no. 7, pp. 70-79, Jul. 2018.
[76] N. Y. Almusallam, Z. Tari, P. Bertok, and A. Y. Zomaya, "Dimensionality reduction for intrusion detection systems in multi-data streams-a review and proposal of unsupervised feature selection scheme," Emergent Computation, vol. 2017, pp. 467-487, Jan. 2017.
[77] J. Li and H. Liu, "Challenges of feature selection for big data analytics," IEEE Intelligent Systems, vol. 32, no. 2, pp. 9-15, Mar. 2017.
[78] R. Xu, et al., "Dynamic feature selection algorithm based on Q-learning mechanism," Applied Intelligence, vol. 51, no. 10, pp. 7233-7244, Oct. 2021.
[79] V. Bolón-Canedo, N. Sánchez-Maroño, and A. Alonso-Betanzos, "Recent advances and emerging challenges of feature selection in the context of big data," Knowledge-Based Systems, vol. 86, no. 9, pp. 33-45, Sept. 2015.
[80] J. Li, K. Cheng, S. Wang, F. Morstatter, R. P. Trevino, J. Tang, and H. Liu, "Feature selection: a data perspective," ACM Computing Surveys (CSUR), vol. 50, no. 6, pp. 1-45, Dec. 2017.
[81] C. Fahy and S. Yang, "Dynamic feature selection for clustering high dimensional data streams," IEEE Access, vol. 7, pp. 127128-127140, Jul. 2019.
[82] J. Jesus, A. Canuto, and D. Araújo, "Dynamic feature selection based on pareto front optimization," in International Joint Conf. on Neural Networks, IJCNN’18, 8 pp., Rio de Janeiro, Brazi, 8-13 Jul. 2018.
[83] R. D. O. Nunes, C. A. Dantas, A. M. Canuto, and J. C. Xavier-Júnior, "An unsupervised-based dynamic feature selection for classification tasks," in Proc. Int. Joint Conf. on Neural Networks, IJCNN’16, pp. 4213-4220, Vancouver, Canada, 24-29 Jul. 2016.
[84] J. P. Barddal, H. M. Gomes, F. Enembreck, and B. Pfahringer, "A survey on feature drift adaptation: definition, benchmark, challenges and future directions," J. of Systems and Software, vol. 127, no. 5, pp. 278-294, May 2017.
[85] G. Wei, J. Zhao, Y. Feng, A. He, and J. Yu, "A novel hybrid feature selection method based on dynamic feature importance," Applied Soft Computing, vol. 93, no. 8, Article ID:. 106337, Aug. 2020.
[86] S. Perkins, K. Lacker, and J. Theiler, "Grafting: fast, incremental feature selection by gradient descent in function space," The J. of Machine Learning Research, vol. 3, no. 3, pp. 1333-1356, Mar. 2003.
[87] I. Katakis, G. Tsoumakas, and I. Vlahavas, "Dynamic feature space and incremental feature selection for the classification of textual data streams," in Proc. Int. Workshop on Knowledge Discovery from Data Streams, ECML/PKDD’06, . pp. 107-116, Berlin, Germany, 18-22 Sept. 2006.
[88] J. Zhou, D. Foster, R. Stine, and L. Ungar, "Streaming feature selection using alpha-investing," in Proc. of the 11th ACM SIGKDD International Conf. on Knowledge Discovery in Data Mining, pp. 384-393, Chicago, IL, USA, 21-24 Aug. 2005.
[89] X. Wu, K. Yu, H. Wang, and W. Ding, "Online streaming feature selection," in Proc. 27th Int. Conf. on Machine Learning, ICML’10, . pp. 1159-1166, 21-24, Jun. 2010.
[90] X. Wu, K. Yu, W. Ding, H. Wang, and X. Zhu, "Online feature selection with streaming features," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 35, no. 5, pp. 1178-1192, Sept. 2012.
[91] C. Zhang, J. Ruan, and Y. Tan, "An incremental feature subset selection algorithm based on boolean matrix in decision system," Convergence Information Technology, vol. 6, no. 12, pp. 16-23, Dec. 2011.
[92] M. Masud, J. Gao, L. Khan, J. Han, and B. M. Thuraisingham, "Classification and novel class detection in concept-drifting data streams under time constraints," IEEE Trans. on Knowledge and Data Engineering, vol. 23, no. 6, pp. 859-874, Jun. 2010.
[93] H. L. Nguyen, Y. K. Woon, W. K. Ng, and L. Wan, "Heterogeneous ensemble for feature drifts in data streams," in Proc. Pacific-Asia Conf. on Knowledge Discovery and Data Mining, 12 pp., Kuala Lumpur, Malaysia, 29 May-1 Jun. 2012.
[94] K. Yu, X. Wu, W. Ding, and J. Pei, "Towards scalable and accurate online feature selection for big data," in Proc. IEEE Int. Conf. on Data Mining, pp. 660-669, Shenzhen, China, 14-17 Dec. 2014.
[95] F. Wang, J. Liang, and Y. Qian, "Attribute reduction: a dimension incremental strategy," Knowledge-Based Systems, vol. 39, no. 2, pp. 95-108, Feb. 2013.
[96] S. Eskandari and M. M. Javidi, "Online streaming feature selection using rough sets," International J. of Approximate Reasoning, vol. 69, no. 2, pp. 35-57, Feb. 2016.
[97] M. M. Javidi and S. Eskandari, "Streamwise feature selection: a rough set method," International J. of Machine Learning and Cybernetics, vol. 9, no. 4, pp. 667-676, Apr. 2018.
[98] J. P. Barddal, H. M. Gomes, F. Enembreck, B. Pfahringer, and A. Bifet, "On dynamic feature weighting for feature drifting data streams," in Proc. Joint European Conf. on Machine Learning and Knowledge Discovery in Databases, pp. 129-144, Riva del Garda, Italy, 19-23 Sept. 2016.
[99] J. P. Barddal, F. Enembreck, H. M. Gomes, A. Bifet, and B. Pfahringer, "Merit-guided dynamic feature selection filter for data streams," Expert Systems with Applications, vol. 116, no. 2, pp. 227-242, Feb. 2019.
[100] J. C. Chamby-Diaz, M. Recamonde-Mendoza, and A. L. Bazzan, "Dynamic correlation-based feature selection for feature drifts in data streams," in Proc. 8th Brazilian Conf. on Intelligent Systems, BRACIS’19, pp. 198-203, Salvador, Brazil, 15-18 Oct. 2019.
[101] J. P. Barddal, F. Enembreck, H. M. Gomes, A. Bifet, and B. Pfahringer, "Boosting decision stumps for dynamic feature selection on data streams," Information Systems, vol. 83, no. 7, pp. 13-29, Jul. 2019.
[102] S. Sahmoud and H. R. Topcuoglu, "A general framework based on dynamic multi-objective evolutionary algorithms for handling feature drifts on data streams," Future Generation Computer Systems, vol. 102, no. 1, pp. 42-52, Jan. 2020.
[103] Y. Meidan, et al., "N-baiot-network-based detection of iot botnet attacks using deep autoencoders," IEEE Pervasive Computing, vol. 17, no. 3, pp. 12-22, Jul.-Sept. 2018.
[104] -, N-BaIoT Dataset, Retrieved from https://archive.ics.uci.edu/ml/datasets/detection_of_IoT_botnet_attacks_N_BaIoT#, 2018.
[105] C. Kolias, G. Kambourakis, A. Stavrou, and J. Voas, "DDoS in the IoT: mirai and other botnets," Computer, vol. 50, no. 7, pp. 80-84, Jul. 2017.
[106] R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," in Proc. the 14th int. joint Conf. on Artificial intelligence, IJCAI'95, vol. 2, pp. 1137-1145, Montreal, Canada, 20-25 Aug. 1995.
[107] T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms," Neural Computation, vol. 10, no. 7, pp. 1895-1923, Oct. 1998.
[1] M. Antonakakis, et al., "Understanding the mirai botnet," in Proc. 26th USENIX Security Symp., pp. 1093-1110, Vancouver, Canada, 16-18 Aug. 2017.
[2] A. Marzano, et al., "The evolution of bashlite and mirai IoT botnets," in Proc. IEEE Symp. on Computers and Communications, ISCC'18, pp. 813-818, Natal, Brazil,25-28 Jun. 2018.
[3] S. García, A. Zunino, and M. Campo, "Survey on network‐based botnet detection methods," Security and Communication Networks, vol. 7, no. 5, pp. 878-903, May. 2014.
[4] R. Alhajri, R. Zagrouba, and F. Al-Haidari, "Survey for anomaly detection of IoT botnets using machine learning auto-encoders," Int. J. Appl. Eng. Res, vol. 14, no. 10, pp. 2417-2421, Jul. 2019.
[5] R. Azmi and B. Pishgoo, "STLR: a novel danger theory based structural TLR algorithm," The ISC International J. of Information Security, vol. 5, no. 2, pp. 209-225, Mar. 2014.
[6] R. Azmi and B. Pishgoo, "SHADuDT: secure hypervisor-based anomaly detection using danger theory," Computers & Security, vol. 39, no. 1, pp. 268-288, Nov. 2013.
[7] L. Yin, L. Qin, Z. Jiang, and X. Xu, "A fast parallel attribute reduction algorithm using Apache Spark," Knowledge-Based Systems, vol. 212, Article ID: 106582, Jan. 2021.
[8] Y. Wu and J. Tang, "Research progress of attribute reduction based on rough set in context of big data," Computer Engineering and Applications, vol. 55, no. 6, pp. 31-38, May 2019.
[9] I. A. Gheyas and L. S. Smith, "Feature subset selection in large dimensionality domains," Pattern Recognition, vol. 43, no. 1, pp. 5-13, Jan. 2010.
[10] D. B. Skillicorn and S. M. McConnell, "Distributed prediction from vertically partitioned data," J. of Parallel and Distributed Computing, vol. 68, no. 1, pp. 16-36, Jan. 2008.
[11] M. Riahi-Madvar, A. A. Azirani, B. Nasersharif, and B. Raahemi, "A new density-based subspace selection method using mutual information for high dimensional outlier detection," Knowledge-Based Systems, vol. 216, Article ID: 106733, Mar. 2021.
[12] M. Banerjee and S. Chakravarty, "Privacy preserving feature selection for distributed data using virtual dimension," in Proc. of the 20th ACM Int. Conf. on Information and Knowledge Management, pp. 2281-2284, Glasgow, Scotland, UK, 24-28 Oct. 2011.
[13] M. Bramer, Principles of Data Mining, vol. 180, pp. 231-238, London: Springer, 2007.
[14] J. Qian, P. Lv, X. Yue, C. Liu, and Z. Jing, "Hierarchical attribute reduction algorithms for big data using MapReduce," Knowledge-Based Systems, vol. 73, no. 1, pp. 18-31, Jan. 2015.
[15] H. Chen, T. Li, Y. Cai, C. Luo, and H. Fujita, "Parallel attribute reduction in dominance-based neighborhood rough set," Information Sciences, vol. 373, pp. 351-368, Dec. 2016.
[16] W. Ding, J. Wang, and J. Wang, "Multigranulation consensus fuzzy-rough based attribute reduction," Knowledge-Based Systems, vol. 198, Article IDL. 105945, Jun. 2020.
[17] H. Kalkan and B. Çetisli, "Online feature selection and classification," in Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing, ICASSP’11, pp. 2124-2127, Prague, Czech Republic, 22-27 May 2011.
[18] D. Levi and S. Ullman, "Learning to classify by ongoing feature selection," Image Vision Comput, vol. 28, no. 4, pp. 715-723, Jun. 2010.
[19] N. AlNuaimi, M. M. Masud, M. A. Serhani, and N. Zaki, "Streaming feature selection algorithms for big data: a survey," Applied Computing and Informatics, vol. 18, no. 1-2, pp. 113-135, Jul. 2020.
[20] N. Parveen and M. Ananthi, "Data processing for large database using feature selection," in Proc. 2nd Int. Conf. on Computing and Communications Technologies, ICCCT'17, pp. 321-326, Chennai, India, 23-24 Feb. 2017.
[21] L. Brezočnik, I. Fister, and V. Podgorelec, "Swarm intelligence algorithms for feature selection: a review," Applied Sciences, vol. 8, no. 9, Article ID:. 1521, Sept 2018.
[22] N. Abd-Alsabour, "A review on evolutionary feature selection," in Proc. European Modelling Symp., pp. 20-26, Pisa, Italy, 21-23 Oct. 2014.
[23] N. Heidari, R. Azmi, and B. Pishgoo, "Fabric textile defect detection, by selecting a suitable subset of wavelet coefficients, through genetic algorithm," International J. of Image Processing, vol. 5, no. 1, pp. 25-35, Jan. 2011. [24] R. Azmi, B. Pishgoo, N. Norozi, M. Koohzadi, and F. Baesi, "A hybrid GA and SA algorithms for feature selection in recognition of hand-printed Farsi characters," in Proc. IEEE Int. Conf. on Intelligent Computing and Intelligent Systems, vol. 3, pp. 384-387, Xiamen, China ,29-31Oct. 2010.
[25] Y. Xing, H. Shu, H. Zhao, D. Li, and L. Guo, "Survey on botnet detection techniques: classification, methods, and evaluation," Mathematical Problems in Engineering, vol. 2021, no. 1, pp. 1-24, Jan. 2021.
[26] S. Almutairi, S. Mahfoudh, S. Almutairi, and J. S. Alowibdi, "Hybrid botnet detection based on host and network analysis," J. of Computer Networks and Communications, vol. 2020, no. 1, pp. 1-17, Jan. 2020.
[27] A. Karim, R. B. Salleh, M. Shiraz, et al., "Botnet detection techniques: review, future trends, and issues," J. of Zhejiang University-Science C, vol. 15, no. 11, pp. 943-983, Nov. 2014.
[28] K. Sinha, V. Arun, and B. Julian, "Tracking temporal evolution of network activity for botnet detection," https:// arxiv.org/abs/1908.03443, 2013.
[29] W. T. Strayer, R. Walsh, C. Livadas, and D. Lapsley, "Detecting botnets with tight command and control," in Proc. 31st IEEE Conf. on Local Computer Networks, pp. 195-202, Tampa, FL, USA, 14-16 Nov. 2006.
[30] E. Passerini, R. Paleari, L. Martignoni, and D. Bruschi, "Fluxor: detecting and monitoring fast-flux service networks," in Proc. Int. Conf. on Detection of Intrusions and Malware and Vulnerability Assessment, pp. 186-206, Paris, France, 10-11 Jul. 2008.
[31] T. F. Yen and M. K. Reiter, "Traffic aggregation for malware detection," in Proc. Int. Conf. on Detection of Intrusions and Malware, and Vulnerability Assessment, pp. 207-227, Paris, France, 10-11 Jul. 2008..
[32] S. Kondo and N. Sato, "Botnet traffic detection techniques by C&C session classification using SVM," in Proc. Int. Workshop on Security, pp. 91-104, Nara, Japan, 29-31 Oct. 2007.
[33] J. François, S. Wang, and T. Engel, "BotTrack: tracking botnets using NetFlow and PageRank," in Proc. Int. Conf. on Research in Networking, pp. 1-14, Valencia, Spain, 9-13 May 2011.
[34] G. Gu, R. Perdisci, J. Zhang, and W. Lee, "Botminer: clustering analysis of network traffic for protocol-and structure-independent botnet detection," in Proc. USENIX Security Symp., pp. 139-154, San Jose, CA, USA, 28 Jul.-1 Aug. 2008.
[35] W. Jung, H. Zhao, M. Sun, and G. Zhou, "IoT botnet detection via power consumption modeling," Smart Health, vol. 15, Article ID: 100103, Mar. 2020.
[36] Y. Zhang and Z. O. U. Fu-Tai, "Detection method of malicious domain name based on knowledge map," Communications Technology, vol. 53, no. 1, pp. 168-173, Jan. 2020.
[37] C. Yin, Research on Network Anomaly Detection Technology Based on Deep Learning, University of Information Engineering, Strategic Support Forces, Zhengzhou, China, 2018.
[38] R. Vinayakumar, et al., "A visualized botnet detection system based deep learning for the internet of things networks of smart cities," IEEE Trans. on Industry Applications, vol. 56, no. 4, pp. 4436-4456, Feb. 2020.
[39] S. I. Popoola, et al., "Federated deep learning for zero-day botnet attack detection in IoT edge devices," IEEE Internet of Things J., vol. 9, no. 9, pp. 3930-3944, Jul. 2021.
[40] A. Almomani, "Fast-flux hunter: a system for filtering online fast-flux botnet," Neural Computing and Applications, vol. 29, no. 7, pp. 483-493, Aug. 2018.
[41] M. Alauthman, N. Aslam, M. Alkasassbeh, S. Khan, A. AL-qerem, and K. K. Raymond Choo, "An efficient reinforcement learning-based botnet detection approach," J. of Network and Computer Applications, vol. 52, Article ID: 102479, Jan. 2019.
[42] H. T. Nguyen, Q. D. Ngo, D. H. Nguyen, et al., "PSI-rooted subgraph: a novel feature for iot botnet detection using classifier algorithms," ICT Express, vol. 6, no. 2, pp. 128-138, Jun. 2020.
[43] D. Zhuang and J. M. Chang, "PeerHunter: detecting peerto-peer botnets through community behavior analysis," in Proc. of the IEEE Conf. on Dependable and Secure Computing, pp. 493-500, Taipei, Taiwan, 7-10 Aug.. 2017.
[44] M. Habib, I. Aljarah, H. Faris, and S. Mirjalili, "Multiobjective particle swarm optimization for botnet detection in internet of things," Evolutionary Machine Learning Techniques, pp. 203-209, Berlin: Germany, Springer, 2020.
[45] A. Al Shorman, H. Faris, and I. Aljarah, "Unsupervised intelligent system based on one class support vector machine and Grey Wolf optimization for IoT botnet detection," J. of Ambient Intelligence and Humanized Computing, vol. 11, no. 7, pp. 2809-2825, Jul. 2020.
[46] S. Y. Huang, C. H. Mao, and H. M. Lee, "Fast-flux service network detection based on spatial snapshot mechanism for delay-free detection," in Proc. of the 5th ACM Symposium on Information, Computer and Communications Security, pp. 101-111, ، Beijing China , 13-16 Apr. 2010.
[47] S. Garg, M. Guizani, S. Guo, and C. Verikoukis, "Guest editorial special section on AI-driven developments in 5G-envisioned industrial automation: big data perspective," IEEE Trans. on Industrial Informatics, vol. 16, no. 2, pp. 1291-1295, Nov. 2020.
[48] X. Wang, Q. Yang, and X. Jin, "Periodic communication detection algorithm of botnet based on quantum computing," J. of Quantum Electronics, vol. 33, no. 2, pp. 182-187, Mar. 2016.
[49] M. Albanese, S. Jajodia, and S. Venkatesan, "Defending from stealthy botnets using moving target defenses," IEEE Security & Privacy, vol. 16, no. 1, pp. 92-97, Feb. 2018.
[50] Z. Zha, A. Wang, Y. Guo, D. Montgomery, and S. Chen, "BotSifter: an SDN-based online bot detection framework in data centers," in Proc. of the IEEE Conf. on Communications and Network Security, CNS’19, pp. 142-150, Washington, D.C., USA, 10-12 Jun. 2019.
[51] G. Spathoulas, N. Giachoudis, G. P. Damiris, and G. Theodoridis, "Collaborative blockchain-based detection of distributed denial of service attacks based on internet of things botnets," Future Internet, vol. 11, Article ID: 226, Oct. 2019.
[52] J. Warren and N. Marz, Big Data: Principles and Best Practices of Scalable Realtime Data Systems, Simon and Schuster, 2015.
[53] B. Twardowski and D. Ryzko, "Multi-agent architecture for real-time big data processing," in Proc. IEEE/WIC/ACM Int.Joint Conf. on Web Intelligence (WI) and Intelligent Agent Technologies (IAT), vol. 3, pp. 333-337, Warsaw, Poland, 11-13 Aug. 2014.
[54] M. Di Capua, E. Di Nardo, and A. Petrosino, "An architecture for sentiment analysis in twitter," in Proc. of Int. Conf. on E-learning, Germany, 10 pp., Berlin, Germany, 11-12 Sept. 2015.
[55] V. Nair, "Aligning Machine Learning for the Lambda Architecture," 2015.
[56] -, Madrid. Lambdoop. Retrieved from www.lambdoop.com, 2014.
[57] -, MemSQL. The Lambda Architecture Simplified, 2016.
[58] V. Astakhov and M. Chayel, Lambda Architecture for Batch and Real-Time Processing on AWS with Spark Streaming and Spark SQL, Amazon Web Services, p. 12, 2015.
[59] S. P. T. Krishnan and J. L. U. Gonzalez, Building Your Next Big Thing with Google Cloud Platform: A Guide for Developers and Enterprise Architects, Apress, 2015.
[60] W. Fan and A. Bifet, "Mining big data: current status, and forecast to the future," ACM SIGKDD Explorations Newsletter, vol. 14, no. 2, pp. 1-5, Apr. 2013.
[61] S. Landset, T. M. Khoshgoftaar, A. N. Richter, and T. Hasanin, "A survey of open source tools for machine learning with big data in the Hadoop ecosystem," J. of Big Data, vol. 2, no. 1, pp. 1-36, Dec. 2015.
[62] A. Bifet, "Mining big data in real time," Informatica, vol. 37, no. 1, pp. 15-20, Jan. 2013.
[63] A. Mahesh and P. Manimegalai, "An efficient data processing architecture for smart environments using large scale machine learning," IIOAB J., Special Issue, Emerging Technologies in Networking and Security, vol. 7, no. 9, pp. 795-803, Aug. 2016.
[64] C. H. Kumar and A. S. Sangari, "An efficient distributed data processing method for smart environment," Indian J. Sci. Technol., vol. 9, no. 31, pp. 380-384, Aug. 2016.
[65] X. Liu and P. S. Nielsen, "Scalable prediction-based online anomaly detection for smart meter data," Information Systems, vol. 77, no. 3, pp. 34-47, Sept. 2018.
[66] G. Iuhasz, D. Pop, and I. Dragan, "Architecture of a scalable platform for monitoring multiple big data frameworks," Scalable Computing: Practice and Experience, vol. 17, no. 4, pp. 313-321, Oct. 2016.
[67] M. Kiran, et al., "Lambda architecture for cost-effective batch and speed big data processing," in Proc. IEEE Int Conf. on Big Data (Big Data), pp. 2785-2792, Santa Clara, CA, USA, 29 Oct.-1 Nov. 2015.
[68] -, Oryx 1, Retrieved from https://github.com/certxg/oryx-1, 2013.
[69] -,Oryx2, Retrieved from http://oryx.io/, 2014.
[70] R. C. Fernandez, et al., "Liquid: unifying nearline and offline big data integration," in Proc. 7th Biennial Conf. on Innovative Data Systems Research, CIDR’15, 8 pp., Asilomar, CA, USA, 4-7 Jan. 2105.
[71] D. Namiot, "On big data stream processing," International J. of Open Information Technologies, vol. 3, no. 8, pp. 48-51, aUG. 2015.
[72] L. Magnoni, et al., "Monitoring WLCG with lambda-architecture: a new scalable data store and analytics platform for monitoring at petabyte scale," J. of Physics: Conf. Series, vol. 664, no. 5, Article ID:. 052023, Dec. 2015.
[73] F. Yang, et al., The RADStack: Open Source Lambda Architecture for Interactive Analytics, 2017.
[74] B. Pishgoo, A. A. Azirani, and B. Raahemi, "A hybrid distributed batch-stream processing approach for anomaly detection," Information Sciences, vol. 543, pp. 309-327, Jan. 2021.
[75] J. Cai, J. Luo, S. Wang, and S. Yang, "Feature selection in machine learning: a new perspective," Neurocomputing, vol. 300, no. 7, pp. 70-79, Jul. 2018.
[76] N. Y. Almusallam, Z. Tari, P. Bertok, and A. Y. Zomaya, "Dimensionality reduction for intrusion detection systems in multi-data streams-a review and proposal of unsupervised feature selection scheme," Emergent Computation, vol. 2017, pp. 467-487, Jan. 2017.
[77] J. Li and H. Liu, "Challenges of feature selection for big data analytics," IEEE Intelligent Systems, vol. 32, no. 2, pp. 9-15, Mar. 2017.
[78] R. Xu, et al., "Dynamic feature selection algorithm based on Q-learning mechanism," Applied Intelligence, vol. 51, no. 10, pp. 7233-7244, Oct. 2021.
[79] V. Bolón-Canedo, N. Sánchez-Maroño, and A. Alonso-Betanzos, "Recent advances and emerging challenges of feature selection in the context of big data," Knowledge-Based Systems, vol. 86, no. 9, pp. 33-45, Sept. 2015.
[80] J. Li, K. Cheng, S. Wang, F. Morstatter, R. P. Trevino, J. Tang, and H. Liu, "Feature selection: a data perspective," ACM Computing Surveys (CSUR), vol. 50, no. 6, pp. 1-45, Dec. 2017.
[81] C. Fahy and S. Yang, "Dynamic feature selection for clustering high dimensional data streams," IEEE Access, vol. 7, pp. 127128-127140, Jul. 2019.
[82] J. Jesus, A. Canuto, and D. Araújo, "Dynamic feature selection based on pareto front optimization," in International Joint Conf. on Neural Networks, IJCNN’18, 8 pp., Rio de Janeiro, Brazi, 8-13 Jul. 2018.
[83] R. D. O. Nunes, C. A. Dantas, A. M. Canuto, and J. C. Xavier-Júnior, "An unsupervised-based dynamic feature selection for classification tasks," in Proc. Int. Joint Conf. on Neural Networks, IJCNN’16, pp. 4213-4220, Vancouver, Canada, 24-29 Jul. 2016.
[84] J. P. Barddal, H. M. Gomes, F. Enembreck, and B. Pfahringer, "A survey on feature drift adaptation: definition, benchmark, challenges and future directions," J. of Systems and Software, vol. 127, no. 5, pp. 278-294, May 2017.
[85] G. Wei, J. Zhao, Y. Feng, A. He, and J. Yu, "A novel hybrid feature selection method based on dynamic feature importance," Applied Soft Computing, vol. 93, no. 8, Article ID:. 106337, Aug. 2020.
[86] S. Perkins, K. Lacker, and J. Theiler, "Grafting: fast, incremental feature selection by gradient descent in function space," The J. of Machine Learning Research, vol. 3, no. 3, pp. 1333-1356, Mar. 2003.
[87] I. Katakis, G. Tsoumakas, and I. Vlahavas, "Dynamic feature space and incremental feature selection for the classification of textual data streams," in Proc. Int. Workshop on Knowledge Discovery from Data Streams, ECML/PKDD’06, . pp. 107-116, Berlin, Germany, 18-22 Sept. 2006.
[88] J. Zhou, D. Foster, R. Stine, and L. Ungar, "Streaming feature selection using alpha-investing," in Proc. of the 11th ACM SIGKDD International Conf. on Knowledge Discovery in Data Mining, pp. 384-393, Chicago, IL, USA, 21-24 Aug. 2005.
[89] X. Wu, K. Yu, H. Wang, and W. Ding, "Online streaming feature selection," in Proc. 27th Int. Conf. on Machine Learning, ICML’10, . pp. 1159-1166, 21-24, Jun. 2010.
[90] X. Wu, K. Yu, W. Ding, H. Wang, and X. Zhu, "Online feature selection with streaming features," IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 35, no. 5, pp. 1178-1192, Sept. 2012.
[91] C. Zhang, J. Ruan, and Y. Tan, "An incremental feature subset selection algorithm based on boolean matrix in decision system," Convergence Information Technology, vol. 6, no. 12, pp. 16-23, Dec. 2011.
[92] M. Masud, J. Gao, L. Khan, J. Han, and B. M. Thuraisingham, "Classification and novel class detection in concept-drifting data streams under time constraints," IEEE Trans. on Knowledge and Data Engineering, vol. 23, no. 6, pp. 859-874, Jun. 2010.
[93] H. L. Nguyen, Y. K. Woon, W. K. Ng, and L. Wan, "Heterogeneous ensemble for feature drifts in data streams," in Proc. Pacific-Asia Conf. on Knowledge Discovery and Data Mining, 12 pp., Kuala Lumpur, Malaysia, 29 May-1 Jun. 2012.
[94] K. Yu, X. Wu, W. Ding, and J. Pei, "Towards scalable and accurate online feature selection for big data," in Proc. IEEE Int. Conf. on Data Mining, pp. 660-669, Shenzhen, China, 14-17 Dec. 2014.
[95] F. Wang, J. Liang, and Y. Qian, "Attribute reduction: a dimension incremental strategy," Knowledge-Based Systems, vol. 39, no. 2, pp. 95-108, Feb. 2013.
[96] S. Eskandari and M. M. Javidi, "Online streaming feature selection using rough sets," International J. of Approximate Reasoning, vol. 69, no. 2, pp. 35-57, Feb. 2016.
[97] M. M. Javidi and S. Eskandari, "Streamwise feature selection: a rough set method," International J. of Machine Learning and Cybernetics, vol. 9, no. 4, pp. 667-676, Apr. 2018.
[98] J. P. Barddal, H. M. Gomes, F. Enembreck, B. Pfahringer, and A. Bifet, "On dynamic feature weighting for feature drifting data streams," in Proc. Joint European Conf. on Machine Learning and Knowledge Discovery in Databases, pp. 129-144, Riva del Garda, Italy, 19-23 Sept. 2016.
[99] J. P. Barddal, F. Enembreck, H. M. Gomes, A. Bifet, and B. Pfahringer, "Merit-guided dynamic feature selection filter for data streams," Expert Systems with Applications, vol. 116, no. 2, pp. 227-242, Feb. 2019.
[100] J. C. Chamby-Diaz, M. Recamonde-Mendoza, and A. L. Bazzan, "Dynamic correlation-based feature selection for feature drifts in data streams," in Proc. 8th Brazilian Conf. on Intelligent Systems, BRACIS’19, pp. 198-203, Salvador, Brazil, 15-18 Oct. 2019.
[101] J. P. Barddal, F. Enembreck, H. M. Gomes, A. Bifet, and B. Pfahringer, "Boosting decision stumps for dynamic feature selection on data streams," Information Systems, vol. 83, no. 7, pp. 13-29, Jul. 2019.
[102] S. Sahmoud and H. R. Topcuoglu, "A general framework based on dynamic multi-objective evolutionary algorithms for handling feature drifts on data streams," Future Generation Computer Systems, vol. 102, no. 1, pp. 42-52, Jan. 2020.
[103] Y. Meidan, et al., "N-baiot-network-based detection of iot botnet attacks using deep autoencoders," IEEE Pervasive Computing, vol. 17, no. 3, pp. 12-22, Jul.-Sept. 2018.
[104] -, N-BaIoT Dataset, Retrieved from https://archive.ics.uci.edu/ml/datasets/detection_of_IoT_botnet_attacks_N_BaIoT#, 2018.
[105] C. Kolias, G. Kambourakis, A. Stavrou, and J. Voas, "DDoS in the IoT: mirai and other botnets," Computer, vol. 50, no. 7, pp. 80-84, Jul. 2017.
[106] R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," in Proc. the 14th int. joint Conf. on Artificial intelligence, IJCAI'95, vol. 2, pp. 1137-1145, Montreal, Canada, 20-25 Aug. 1995.
[107] T. G. Dietterich, "Approximate statistical tests for comparing supervised classification learning algorithms," Neural Computation, vol. 10, no. 7, pp. 1895-1923, Oct. 1998.