Multi-Agent Deep Reinforcement Learning-Based Decentralized Computation Offloading in Mobile Edge Computing

Abstract :

It is hardly possible to support latency-sensitive and computational-intensive applications for mobile devices with limited battery capacity and low computing resources. The development of mobile edge computing and wireless power transfer technologies enable mobile devices to offload computing tasks to edge servers and harvest energy to extend their battery lifetime. However, computation offloading faces challenges such as the limited computing resources of the edge server, the quality of the available communication channel, and the limited time for energy harvesting. In this paper, we study the joint problem of decentralized computation offloading and resource allocation in the dynamic environment of mobile edge computing. To this end, we propose a multi-agent deep reinforcement learning-based offloading scheme that considers the cooperation between mobile devices to adjust their strategies. To be specific, we propose an improved version of the multi-agent deep deterministic policy gradient algorithm by employing the features of clipped double Q-learning, delayed policy update, target policy smoothing, and prioritized experience replay. The simulation results reveal that the proposed offloading scheme has better convergence performance than other baseline methods and also reduces the average energy consumption, average processing delay and task failure rate.

References:

[1] N. Abbas, Y. Zhang, A. Taherkordi, and T. Skeie, "Mobile edge computing: a survey," IEEE Internet of Things J., vol. 5, no. 1, pp. 450-465, Feb. 2018.
[2] J. Wang, J. Pan, F. Esposito, P. Calyam, Z. Yang, and P. Mohapatra, "Edge cloud offloading algorithms: issues, methods, and perspectives," ACM Computing Surveys, vol. 52, no. 1, pp. 1-23, Feb. 2019.
[3] Q. H. Nguyen and F. Dressler, "A smartphone perspective on computation offloading-a survey," Computer Communications, vol. 159, pp. 133-154, Jun. 2020.
[4] H. Lin, S. Zeadally, Z. Chen, H. Labiod, and L. Wang, "A survey on computation offloading modeling for edge computing," J. of Network and Computer Applications, vol. 169, Article ID: 102781, Nov. 2020.
[5] P. Mach and Z. Becvar, "Mobile edge computing: a survey on architecture and computation offloading," IEEE Communications Surveys & Tutorials, vol. 19, no. 3, pp. 1628-1656, Mar. 2017.
[6] Y. Mao, C. You, J. Zhang, K. Huang, and K. B. Letaief, "A survey on mobile edge computing: the communication perspective," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2322-2358, Aug. 2017.
[7] X. Wang, et al., "Wireless powered mobile edge computing networks: a survey," ACM Computing Surveys, vol. 55, no. 13s, Article ID: 263, 37 pp., Dec. 2023.
[8] U. M. Malik, M. A. Javed, S. Zeadally, and S. ul Islam, "Energy-efficient fog computing for 6G-enabled massive IoT: recent trends and future opportunities," IEEE Internet of Things J., vol. 9, no. 16, pp. 14572-14594, Aug. 2022.
[9] Q. Luo, S. Hu, C. Li, G. Li, and W. Shi, "Resource scheduling in edge computing: a survey," IEEE Communications Surveys & Tutorials, vol. 23, no. 4, pp. 2131-2165, Aug. 2021.
[10] Y. Fan, J. Ge, S. Zhang, J. Wu, and B. Luo, "Decentralized scheduling for concurrent tasks in mobile edge computing via deep reinforcement learning," IEEE Trans. on Mobile Computing, vol. 23, no. 4, pp. 2765-2779, Apr. 2023.
[11] P. Gazori, D. Rahbari, and M. Nickray, "Saving time and cost on the scheduling of fog-based IoT applications using deep reinforcement learning approach," Future Generation Computer Systems, vol. 110, pp. 1098-1115, Sept. 2020.
[12] H. Djigal, J. Xu, L. Liu, and Y. Zhang, "Machine and deep learning for resource allocation in multi-access edge computing: a survey," IEEE Communications Surveys & Tutorials, vol. 24, no. 4, pp. 2449-2494, Aug. 2022.
[13] A. Feriani and E. Hossain, "Single and multi-agent deep reinforcement learning for AI-enabled wireless networks: a tutorial," IEEE Communications Surveys & Tutorials, vol. 23, no. 2, pp. 1226-1252, Mar. 2021.
[14] T. Li, K. Zhu, N. C. Luong, D. Niyato, Q. Wu, Y. Zhang, and B. Chen, "Applications of multi-agent reinforcement learning in future internet: a comprehensive survey," IEEE Communications Surveys & Tutorials, vol. 24, no. 2, pp. 1240-1279, Mar. 2022.
[15] T. T. Nguyen, N. D. Nguyen, and S. Nahavandi, "Deep reinforcement learning for multiagent systems: a review of challenges, solutions, and applications," IEEE Trans. on Cybernetics, vol. 50, no. 9, pp. 3826-3839, Sept. 2020.
[16] K. Zhang, Z. Yang, and T. Başar, "Multi-agent reinforcement learning: a selective overview of theories and algorithms," In: Vamvoudakis, K.G., Wan, Y., Lewis, F.L., Cansever, D. (eds) Handbook of Reinforcement Learning and Control. Studies in Systems, Decision and Control, vol. 325, pp. 321-384, 2021.
[17] R. Lowe, et al., "Multi-agent actor-critic for mixed cooperative-competitive environments," in Proc. 31st Conf. on Neural Information Processing Systems, NIPS'17, 12 pp., Long Beach, CA, USA, 4-9 Dec. 2017.
[18] T. P. Lillicrap, et al., Continuous Control with Deep Reinforcement Learning, arXiv preprint arXiv: 1509.02971, 2015.
[19] S. Fujimoto, H. Hoof, and D. Meger, "Addressing function approximation error in actor-critic methods," in Proc. of the 35th Int. Conf. on Machine Learning, PMLR'80, pp. 1587-1596, Stockholm Sweden, 10-15 Jul. 2018.
[20] O. K. Shahryari, H. Pedram, V. Khajehvand, and M. D. TakhtFooladi, "Energy and task completion time trade-off for task offloading in fog-enabled IoT networks," Pervasive and Mobile Computing, vol. 74, Article ID: 101395, Jul. 2021.
[21] J. Bi, H. Yuan, S. Duanmu, M. Zhou, and A. Abusorrah, "Energy-optimized partial computation offloading in mobile-edge computing with genetic simulated-annealing-based particle swarm optimization," IEEE Internet of Things J., vol. 8, no. 5, pp. 3774-3785, Sept. 2020.
[22] S. Fu, F. Zhou, and R. Q. Hu, "Resource allocation in a relay-aided mobile edge computing system," IEEE Internet of Things J., vol. 9, no. 23, pp. 23659-23669, Jul. 2022.
[23] G. Yang, L. Hou, X. He, D. He, S. Chan, and M. Guizani, "Offloading time optimization via markov decision process in mobile-edge computing," IEEE Internet of Things J., vol. 8, no. 4, pp. 2483-2493, Oct. 2020.
[24] B. Cao, L. Zhang, Y. Li, D. Feng, and W. Cao, "Intelligent offloading in multi-access edge computing: a state-of-the-art review and framework," IEEE Communications Magazine, vol. 57, no. 3, pp. 56-62, Mar. 2019.
[25] Z. Liu, Y. Yang, K. Wang, Z. Shao, and J. Zhang, "POST: parallel offloading of splittable tasks in heterogeneous fog networks," IEEE Internet of Things J., vol. 7, no. 4, pp. 3170-3183, Jan. 2020.
[26] M. Guo, Q. Li, Z. Peng, X. Liu, and D. Cui, "Energy harvesting computation offloading game towards minimizing delay for mobile edge computing," Computer Networks, vol. 204, Article ID: 108678, Feb. 2022.
[27] T. Zhang and W. Chen, "Computation offloading in heterogeneous mobile edge computing with energy harvesting," IEEE Trans. on Green Communications and Networking, vol. 5, no. 1, pp. 552-565, Jan. 2021.
[28] H. Teng, Z. Li, K. Cao, S. Long, S. Guo, and A. Liu, "Game theoretical task offloading for profit maximization in mobile edge computing," IEEE Trans. on Mobile Computing, vol. 22, no. 9, pp. 5313-5329, May 2022.
[29] H. Wu, Z. Zhang, C. Guan, K. Wolter, and M. Xu, "Collaborate edge and cloud computing with distributed deep learning for smart city Internet of Things," IEEE Internet of Things J., vol. 7, no. 9, pp. 8099-8110, May 2020.
[30] L. Ale, N. Zhang, X. Fang, X. Chen, S. Wu, and L. Li, "Delay-aware and energy-efficient computation offloading in mobile-edge computing using deep reinforcement learning," IEEE Trans. on Cognitive Communications and Networking, vol. 7, no. 3, pp. 881-892, Mar. 2021.
[31] C. Li, J. Xia, F. Liu, D. Li, L. Fan, G. K. Karagiannidis, and A. Nallanathan, "Dynamic offloading for multiuser muti-CAP MEC networks: a deep reinforcement learning approach," IEEE Trans. on Vehicular Technology, vol. 70, no. 3, pp. 2922-2927, Feb. 2021.
[32] L. Wang and G. Zhang, "Deep reinforcement learning based joint partial computation offloading and resource allocation in mobility-aware MEC system," China Communications, vol. 19, no. 8, pp. 85-99, Aug. 2022.
[33] J. Niu, S. Zhang, K. Chi, G. Shen, and W. Gao, "Deep learning for online computation offloading and resource allocation in NOMA," Computer Networks, vol. 216, Article ID: 109238, Oct. 2022.
[34] H. Lu, X. He, M. Du, X. Ruan, Y. Sun, and K. Wang, "Edge QoE: computation offloading with deep reinforcement learning for Internet of Things," IEEE Internet of Things J., vol. 7, no. 10, pp. 9255-9265, Mar. 2020.
[35] V. D. Tuong, T. P. Truong, T. V. Nguyen, W. Noh, and S. Cho, "Partial computation offloading in NOMA-assisted mobile-edge computing systems using deep reinforcement learning," IEEE Internet of Things J., vol. 8, no. 17, pp. 13196-13208, Mar. 2021.
[36] Z. Hu, J. Niu, T. Ren, B. Dai, Q. Li, M. Xu, and S. K. Das, "An efficient online computation offloading approach for large-scale mobile edge computing via deep reinforcement learning," IEEE Trans. on Services Computing, vol. 15, no. 2, pp. 669-683, Sept. 2021.
[37] J. Chen and Z. Wu, "Dynamic computation offloading with energy harvesting devices: a graph-based deep reinforcement learning approach," IEEE Communications Letters, vol. 25, no. 9, pp. 2968-2972, Jul. 2021.
[38] X. He, H. Lu, M. Du, Y. Mao, and K. Wang, "QoE-based task offloading with deep reinforcement learning in edge-enabled Internet of Vehicles," IEEE Trans. on Intelligent Transportation Systems, vol. 22, no. 4, pp. 2252-2261, Aug. 2020,
[39] Z. Chen and X. Wang, "Decentralized computation offloading for multi-user mobile edge computing: a deep reinforcement learning approach," EURASIP J. on Wireless Communications and Networking, vol. 2020, Article ID: 188, 21 pp., 2020.
[40] J. Chen, H. Xing, Z. Xiao, L. Xu, and T. Tao, "A DRL agent for jointly optimizing computation offloading and resource allocation in MEC," IEEE Internet of Things J., vol. 8, no. 24, pp. 17508-17524, May 2021.
[41] Z. Cheng, M. Min, M. Liwang, L. Huang, and Z. Gao, "Multiagent DDPG-based joint task partitioning and power control in fog computing networks," IEEE Internet of Things J., vol. 9, no. 1, pp. 104-116, Jun. 2021.
[42] Z. Chen, L. Zhang, Y. Pei, C. Jiang, and L. Yin, "NOMA-based multi-user mobile edge computation offloading via cooperative multi-agent deep reinforcement learning," IEEE Trans. on Cognitive Communications and Networking, vol. 8, no. 1, pp. 350-364, Jun. 2021.
[43] X. Huang, S. Leng, S. Maharjan, and Y. Zhang, "Multi-agent deep reinforcement learning for computation offloading and interference coordination in small cell networks," IEEE Trans. on Vehicular Technology, vol. 70, no. 9, pp. 9282-9293, Jul. 2021.
[44] N. Zhao, Z. Ye, Y. Pei, Y. C. Liang, and D. Niyato, "Multi-agent deep reinforcement learning for task offloading in UAV-assisted mobile edge computing," IEEE Trans. on Wireless Communications, vol. 21, no. 9, pp. 6949-6960, Mar. 2022.
[45] M. Chen, A. Guo, and C. Song, "Multi-agent deep reinforcement learning for collaborative task offloading in mobile edge computing networks," Digital Signal Processing, vol. 140, Article ID: 104127, Aug. 2023.
[46] Q. Tang, R. Xie, F. R. Yu, T. Huang, and Y. Liu, "Decentralized computation offloading in IoT fog computing system with energy harvesting: a Dec-POMDP approach," IEEE Internet of Things J., vol. 7, no. 6, pp. 4898-4911, Feb. 2020.
[47] S. Zeng, X. Huang, and D. Li, "Joint communication and computation cooperation in wireless-powered mobile-edge computing networks with NOMA," IEEE Internet of Things J., vol. 10, no. 11, pp. 9849-9862, Jan. 2023.
[48] L. Huang, S. Bi, and Y. J. A. Zhang, "Deep reinforcement learning for online computation offloading in wireless powered mobile-edge computing networks," IEEE Trans. on Mobile Computing, vol. 19, no. 11, pp. 2581-2593, Jul. 2019.
[49] S. Bi and Y. J. Zhang, "Computation rate maximization for wireless powered mobile-edge computing with binary computation offloading," IEEE Trans. on Wireless Communications, vol. 17, no. 6, pp. 4177-4190, Apr. 2018.
[50] M. Min, et al., "Learning-based computation offloading for IoT devices with energy harvesting," IEEE Trans. on Vehicular Technology, vol. 68, no. 2, pp. 1930-1941, Jan. 2019.
[51] D. Silver et al., "Deterministic policy gradient algorithms," in Proc. of the 31st Int. Conf. on Machine Learning, PMLR'32, pp. 387-395, Beijing, China, 22-24 Jun. 2014.
[52] F. Zhang, J. Li, and Z. Li, "A TD3-based multi-agent deep reinforcement learning method in mixed cooperation-competition environment," Neurocomputing, vol. 411, pp. 206-215, Oct. 2020.
[53] P. Sun, W. Zhou, and H. Li, "Attentive experience replay," in Proc. of the AAAI Conf. on Artificial Intelligence, vol. 34, no. 04, pp. 5900-5907, Apr. 2020.
[54] Y. Hou, L. Liu, Q. Wei, X. Xu, and C. Chen, "A novel DDPG method with prioritized experience replay," in Proc. IEEE Int. Conf. on Systems, Man, and Cybernetics, SMC'17, pp. 316-321, Banff, Canada, 5-8 Oct. 2017.
[55] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, Prioritized Experience Replay, arXiv preprint arXiv:1511.05952, 2015.
[56] P. Cheridito, H. Kawaguchi, and M. Maejima, "Fractional ornstein-uhlenbeck processes," Electron. J. Probab, vol. 8, Article ID: 3, 14 pp., 2003.
[57] http://www.powercastco.com
[58] D. P. Kingma and J. Ba, Adam: A Method for Stochastic Optimization, arXiv preprint arXiv:1412.6980, 2014.