Home Energy Management System Based on Multi-Agent Reinforcement Learning Considering Simultaneous Participation in Energy and Flexibility Markets
Subject Areas : مهندسی برق و کامپیوتر
Peyman Madehkhaksa
1
,
Hamed Delkhosh
2
*
1 - Power group, Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
2 - Power group, Faculty of Electrical and Computer Engineering, Tarbiat Modares University, Tehran, Iran
Keywords: Energy management, Smart Homes, Electricity Markets, Flexibility Services, Multi-Agent Reinforcement Learning,
Abstract :
Moving toward renewable generation and utilizing the potential of demand response in downstream power networks are considered the key solutions to address the challenges of future power systems. Also, the decentralization and digitalization gigatrends have highlighted the importance of optimal energy management for smart homes, including generation, consumption, and storage devices, based on modern approaches. This paper proposes a home energy management system (HEMS) for a building consisting of different appliances (three types of fixed power, time-shiftable, and power-shiftable) and a renewable distributed generation (solar panel). This HEMS receives the solar generation power and energy market price from a neural network-based forecasting tool to obtain the optimal daily schedule of the smart home using multi-agent reinforcement learning based on the Q-learning solution method. The main innovative aspect of the proposed model is the possibility of simultaneous participation in energy and flexibility markets, which can be aligned or opposite in terms of economic profit. Complicating factors of user satisfaction (comfort concerns) and upstream obligations (required generation curtailment) are also considered in the model. The effectiveness of the proposed method is demonstrated through the simulation for various scenarios and sensitivity analysis to key parameters.
[1] M. Parsa Moghaddam, S. Nasiri, and M. Yousefian, "5D giga trends in future power systems," In M. Parsa Moghaddam, R. Zamani, H. Haes Alhelou, and P. Siani (eds.), Decentralized Frameworks for Future Power Systems, vol. 1, ch. 2, pp. 19-50, 2022.
[2] M. Jäntti, A. Jäntti, and M. Shafie-khah, "Toward customer-centric power grid: Residential EV charging simulator for smart homes," In M. Parsa Moghaddam, R. Zamani, H. Haes Alhelou, and P. Siani (eds.), Decentralized Frameworks for Future Power Systems, vol. 1, ch. 9, pp. 207-226, 2022.
[3] H. Delkhosh and M. Jorjani, "Green approaches in future power systems," In M. Parsa Moghaddam, R. Zamani, H. Haes Alhelou, and P. Siani (eds.), Decentralized Frameworks for Future Power Systems,vol. 1, ch. 5, pp. 99-127, 2022.
[4] Y. M. Rind, M. H. Raza, M. Zubair, M. Q. Mehmood, and Y. Massoud, "Smart energy meters for smart grids, an internet of things perspective," Energies, vol. 16, no. 4, Article ID: 1974, Feb. 2023.
[5] F. F. Alruwaili, et al., "A decentralized approach to smart home security: blockchain with red-tailed hawk-enabled deep learning," IEEE Access, vol. 12, pp.14146-14156, 2024.
[6] A. Saif, S. K. Khadem, M. F. Conlon, and B. Norton, "Impact of distributed energy resources in smart homes and community-based electricity market," IEEE Trans. on Industry Applications, vol. 59, no. 1, pp. 59-69, Aug. 2022.
[7] H. Delkhosh, P. Emami, and M. P. Moghaddam, "Developments toward sustainable energy system operation," In Hosting Capacity Aspects in Distribution Networks Towards Sustainable Energy Systems, vol. 1, ch. 11, pp. 227-250, 2025.
[8] M. A. Khan, et al., "Investigation and analysis of demand response approaches, bottlenecks, and future potential capabilities for IoT-enabled smart grid," IET Renewable Power Generation, vol. 18, no. 15, pp. 3509-3535, May 2024.
[9] M. Maldet, et al., "Trends in local electricity market design: Regulatory barriers and the role of grid tariffs," Journal of Cleaner Production, vol. 358, Article ID: 131805, Feb. 2022.
[10] O. Rebenaque, C, Schmitt, K. Schumann, T. Dronne, and F. Roques, "Success of local flexibility market implementation: A review of current projects," Utilities Policy, vol. 80, Article ID: 101491, Feb. 2023.
[11] H. Tang, S. Wang, and H. Li, "Flexibility categorization, sources, capabilities and technologies for energy-flexible and grid-responsive buildings: State-of-the-art and future perspective," Energy, vol. 219, Article ID: 119598, Mar. 2021.
[12] H. Khajeh H. Firoozi, and H. Laaksonen, "Flexibility potential of a smart home to provide TSO-DSO-level services," Electric Power Systems Research, vol. 205, Article ID: 107767, Apr. 2022.
[13] F. Heider, A. Jahic, M. Plenz, and D. Schulz, "Extended residential power management interface for flexibility communication and uncertainty reduction for flexibility system operators," Energies, vol. 15, no. 4, Article ID: 1257, Feb. 2022.
[14] H. Karimi, G. B. Gharehpetian, R. Ahmadiahangar, and A. Rosin, "Optimal energy management of grid-connected multi-microgrid systems considering demand-side flexibility: A two-stage multi-objective approach," Electric Power Systems Research, vol. 214, pt. A, Article ID: 108902. Jan. 2023.
[15] C. Srithapon and D. Månsson, "Predictive control and coordination for energy community flexibility with electric vehicles, heat pumps and thermal energy storage," Applied Energy, vol. 347, Article ID: 121500, Oct. 2023.
[16] M. Nutakki and S. Mandava, "Review on optimization techniques and role of Artificial Intelligence in home energy management systems," Engineering Applications of Artificial Intelligence, vol. 119, Article ID: 105721, Mar. 2023.
[17] Z. Huang, F, Wang, Y, Lu, X.Chen, and Q. Wu, "Optimization model for home energy management system of rural dwellings," Energy, vol. 283, Article ID: 129039, Nov. 2023.
[18] I. L. R, Gomes, M. G. Ruano, and A. E, Ruano, "MILP-based model predictive control for home energy management systems: A real case study in Algarve, Portugal" Energy and Buildings, vol. 281, Article ID: 112774, Feb. 2023.
[19] S, Bahramara, "Robust optimization of the flexibility-constrained energy management problem for a smart home with rooftop photovoltaic and an energy storage," Journal of Energy Storage, vol. 36, Article ID: 102358, Apr. 2021.
[20] S, Dorahaki, M. Mollahassani-Pour, and M. Rashidinejad, "Optimizing energy payment, user satisfaction, and self-sufficiency in flexibility-constrained smart home energy management: a multi-objective optimization approach," e-Prime-Advances in Electrical Engineering, Electronics and Energy, vol. 6, Article ID: 100385, Dec. 2023.
[21] D, Vamvakas, P, Michailidis, C. Korkas, and E, Kosmatopoulos, "Review and evaluation of reinforcement learning frameworks on smart grid applications," Energies, vol. 16, no. 14, Article ID: 5326, Jul. 2023.
[22] A, Dolatabadi, H. Abdeltawab, and Y. A. R. I, Mohamed, "A novel model-free deep reinforcement learning framework for energy management of a PV integrated energy hub," IEEE Trans. on Power Systems, vol. 38, no. 5, pp.4840-4852, Sept. 2022.
[23] Y, Zhang, et al., "Two-step diffusion policy deep reinforcement learning method for low-carbon multi-energy microgrid energy management," IEEE Transactions on Smart Grid, vol. 15, no. 5, pp.4576-4588, Sept. 2024.
[24] M, Hashemnezhad, H. Delkhosh, and M. P, Moghaddam, " Aggregator pricing strategy for community energy management based on multi-agent reinforcement learning considering customer loss or gain," Sustainable Energy, Grids and Networks, vol. 41, Article ID: 101607, Mar. 2025.
[25] W, Pinthurat, T. Surinkaew, and B, Hredzak, "An overview of reinforcement learning-based approaches for smart home energy management systems with energy storages," Renewable and Sustainable Energy Reviews, vol. 202, Article ID: 114648, Sept. 2024.
[26] X. Xu, et al., "A multi-agent reinforcement learning-based data-driven method for home energy management," IEEE Trans. on Smart Grid, vol. 11, no. 4, pp.3201-3211, Jul. 2020.
[27] K, Ren, et al., "A data-driven DRL-based home energy management system optimization framework considering uncertain household parameters," Applied Energy, vol. 355, Article ID: 122258, Mar. 2024.
[28] L. Langer, and T, Volling, "A reinforcement learning approach to home energy management for modulating heat pumps and photovoltaic systems," Applied energy, vol. 327, Article ID: 120020, Dec. 2022.
[29] A, Shahabi, H. Delkhosh, and M. P, Moghaddam, "Home energy management system based on multi-agent deep reinforcement learning handling the user's thermal preferences," in Proc. 2025 10th Int. Conf. on Technology and Energy Management, 5 pp., Tariz, Iran, Apr. 2025.
[30] R, Lu, et al., "Reward shaping-based actor–critic deep reinforcement learning for residential energy management," IEEE Trans. on Industrial Informatics, vol. 19, no. 3, pp.2662-2673, Nov. 2022.
[31] G, Wei, et al., "Deep reinforcement learning for real-time energy management in smart home," IEEE Systems Journal, vol. 17, no. 2, pp.2489-2499, Jun. 2023.
[32] X, Zhou, S, Xue, H. Du, and Z. Ma, , "Optimization of building demand flexibility using reinforcement learning and rule-based expert systems," Applied Energy, vol. 350, Article ID: 121792, Nov. 2023.
[33] Y. Zheng, et al., "Interior-point policy optimization based multi-agent deep reinforcement learning method for secure home energy management under various uncertainties," Applied Energy, vol. 376, Article ID: 124155, Dec. 2024.
[34] A. A, Amer, K. Shaban, and A. M, Massoud, "DRL-HEMS: Deep reinforcement learning agent for demand response in home energy management systems considering customers and operators perspectives. IEEE Trans. on Smart Grid, vol. 14, no. 1, pp.239-250, Jan. 2022.
[35] S. S. Shuvo, and Y, Yilmaz, "Home energy recommendation system (HERSs): A deep reinforcement learning method based on residents’ feedback and activity," IEEE Trans. on Smart Grid, vol. 13, no. 4, pp. 2812-2821, Jul. 2022.