Optimal Resource Management in Fog-Cloud Environments via A2C Reinforcement Learning: Dynamic Task Scheduling and Task Result Caching

Document Type : Research Article

Authors

Faculty of Computer Engineering, K. N. Toosi University of Technology, Tehran, Iran

Abstract

In order to effectively manage tasks in fog-cloud environments, this paper proposes a two-agent architecture-based framework. In this framework, a task scheduling agent is responsible for selecting the computing execution node and allocating resources, while a separate agent manages the caching of results. In each decision cycle, the resource manager first checks whether a valid, fresh result already exists in the cache; if so, the cached result is immediately returned. Otherwise, the execution agent evaluates current conditions — such as network load, nodes’ computational capacity, and user proximity — and assigns the task to the most appropriate node. After task execution completes, an independent storage agent is selected to store the results, potentially operating on a node distinct from the execution node. Through extensive simulations and comparisons with advanced methods (e.g., A3C-R2N2, DDQN, LR-MMT, and LRR-MMT), we demonstrate significant improvements in response latency, computational efficiency, and inter-node communication management. The proposed framework decouples execution scheduling from result storage through two distinct agents while implementing history-based caching that tracks both task request frequencies and result recency. This design enables effective adaptation to variable workloads and dynamic network conditions. The two-agent architecture and history-based caching serve as core innovations that optimize resource utilization and enhance system responsiveness. The resulting decoupled, history-based strategy delivers scalable, low-latency performance and provides a robust solution for real-time service delivery in fog-cloud environments.

Keywords

Main Subjects

References

[1] B. Huang, X. Liu, Y. Xiang, D. Yu, S. Deng, S. Wang, Reinforcement learning for cost-effective IoT service caching at the edge, Journal of Parallel and Distributed Computing, 168 (2022) 120-136.
[2] O.A. Khan, S.U. Malik, F.M. Baig, S.U. Islam, H. Pervaiz, H. Malik, S.H. Ahmed, A cache‐based approach toward improved scheduling in fog computing, Software: Practice and Experience, 51(12) (2021) 2360-2372.
[3] P. Bellavista, C. Giannelli, D.D.P. Montenero, F. Poltronieri, C. Stefanelli, M. Tortonesi, HOlistic pRocessing and NETworking (HORNET): An Integrated Solution for IoT-Based Fog Computing Services, IEEE Access, 8 (2020) 66707-66721.
[4] C. Mouradian, D. Naboulsi, S. Yangui, R.H. Glitho, M.J. Morrow, P.A. Polakos, A Comprehensive Survey on Fog Computing: State-of-the-Art and Research Challenges, IEEE Communications Surveys & Tutorials, 20(1) (2018) 416-464.
[5] R. Mahmud, S.N. Srirama, K. Ramamohanarao, R. Buyya, Quality of Experience (QoE)-aware placement of applications in Fog computing environments, Journal of Parallel and Distributed Computing, 132 (2019) 190-203.
[6] A. Yousefpour, C. Fung, T. Nguyen, K. Kadiyala, F. Jalali, A. Niakanlahiji, J. Kong, J.P. Jue, All one needs to know about fog computing and related edge computing paradigms: A complete survey, Journal of Systems Architecture, 98 (2019) 289-330.
[7] M. Mukherjee, S. Kumar, Q. Zhang, R. Matam, C.X. Mavromoustakis, Y. Lv, G. Mastorakis, Task data offloading and resource allocation in fog computing with multi-task delay guarantee, IEEE Access, 7 (2019) 152911-152918.
[8] S. Bansal, H. Aggarwal, M. Aggarwal, A systematic review of task scheduling approaches in fog computing, Transactions on Emerging Telecommunications Technologies, 33(9) (2022) e4523.
[9] J. Xu, X. Sun, R. Zhang, H. Liang, Q. Duan, Fog-cloud task scheduling of energy consumption optimisation with deadline consideration, International Journal of Internet Manufacturing and Services, 7(4) (2020) 375-392.
[10] J. Lu, J. Yang, S. Li, Y. Li, W. Jiang, J. Dai, J. Hu, A2C-DRL: Dynamic Scheduling for Stochastic Edge-Cloud Environments Using A2C and Deep Reinforcement Learning, IEEE Internet of Things Journal,  (2024).
[11] G. Dong, J. Wang, M. Wang, T. Su, An improved scheduling with advantage actor-critic for Storm workloads, Cluster Computing, 27(10) (2024) 13421-13433.
[12] S. Radhika, S. Keshari Swain, S. Adinarayana, B. Ramesh Babu, Efficient task scheduling in cloud using double deep QNetwork, International Journal of Computing and Digital Systems, 16(1) (2024) 1-11.
[13] X. Sun, Y. Duan, Y. Deng, F. Guo, G. Cai, Y. Peng, Dynamic operating system scheduling using double DQN: A reinforcement learning approach to task optimization, in:  2025 8th International Conference on Advanced Algorithms and Control Engineering (ICAACE), IEEE, 2025, pp. 1492-1497.
[14] X. Zhang, Z. Hu, Y. Liang, H. Xiao, A. Xu, M. Zheng, C. Sun, A federated deep reinforcement learning-based low-power caching strategy for cloud-edge collaboration, Journal of Grid Computing, 22(1) (2024) 21.
[15] Y. Wang, X. Yang, Intelligent resource allocation optimization for cloud computing via machine learning, arXiv preprint arXiv:2504.03682,  (2025).
[16] A. Avan, A. Azim, Q. Mahmoud, Agile Reinforcement Learning for Real-Time Task Scheduling in Edge Computing, arXiv preprint arXiv:2506.08850,  (2025).
[17] A. Amayuelas, J. Yang, S. Agashe, A. Nagarajan, A. Antoniades, X.E. Wang, W. Wang, Self-resource allocation in multi-agent LLM systems, arXiv preprint arXiv:2504.02051,  (2025).
[18] Y. Yang, F. Ren, M. Zhang, A Decentralized Multiagent-Based Task Scheduling Framework for Handling Uncertain Events in Fog Computing, arXiv preprint arXiv:2401.02219,  (2024).
[19] L. Lu, Y. Jiang, M. Bennis, Z. Ding, F.-C. Zheng, X. You, Distributed edge caching via reinforcement learning in fog radio access networks, in:  2019 IEEE 89th Vehicular Technology Conference (VTC2019-Spring), IEEE, 2019, pp. 1-6.
[20] H. Fabelo, R. Leon, E. Torti, S. Marco, A. Badouh, M. Verbers, C. Vega, J. Santana-Nunez, Y. Falevoz, Y. Ramallo-Fariña, C. Weis, A.M. Wägner, E. Juarez, C. Rial, A. Lagares, G. Burström, F. Leporati, L. Jimenez-Roldan, E. Marenzi, T. Cervero, M. Moreto, G. Danese, S. Zinger, F. Manni, M.L. Alvarez-Male, M.A. García-Bello, L. García, J. Morera, J.F. Piñeiro, C. Bairaktari, B. Noriega-Ortega, B. Clavo, G.M. Callico, STRATUM project: AI-based point of care computing for neurosurgical 3D decision support tools, Microprocessors and Microsystems, 116 (2025) 105157.
[21] S.S. Tripathy, K. Mishra, D.S. Roy, K. Yadav, A. Alferaidi, W. Viriyasitavat, J. Sharmila, G. Dhiman, R.K. Barik, State-of-the-art load balancing algorithms for mist-fog-cloud assisted paradigm: a review and future directions, Archives of Computational Methods in Engineering, 30(4) (2023) 2725-2760.
[22] J. Lim, Versatile cloud resource scheduling based on artificial intelligence in cloud-enabled fog computing environments, Hum.-Centric Comput. Inf. Sci, 13 (2023) 54.
[23] J. Singh, J. Sidhu, Comparative analysis of VM consolidation algorithms for cloud computing, Procedia Computer Science, 167 (2020) 1390-1399.
[24] A. Beloglazov, R. Buyya, Optimal online deterministic algorithms and adaptive heuristics for energy and performance efficient dynamic consolidation of virtual machines in cloud data centers, Concurrency and Computation: Practice and Experience, 24(13) (2012) 1397-1420.
[25] D. Basu, X. Wang, Y. Hong, H. Chen, S. Bressan, Learn-as-you-go with megh: Efficient live migration of virtual machines, IEEE Transactions on Parallel and Distributed Systems, 30(8) (2019) 1786-1801.
[26] H. Mao, M. Alizadeh, I. Menache, S. Kandula, Resource management with deep reinforcement learning, in:  Proceedings of the 15th ACM workshop on hot topics in networks, 2016, pp. 50-56.
[27] D. Pathak, P. Krahenbuhl, T. Darrell, Constrained convolutional neural networks for weakly supervised segmentation, in:  Proceedings of the IEEE international conference on computer vision, 2015, pp. 1796-1804.
[28] S. Tuli, S. Ilager, K. Ramamohanarao, R. Buyya, Dynamic scheduling for stochastic edge-cloud computing environments using a3c learning and residual recurrent neural networks, IEEE Transactions on Mobile Computing, 21(3) (2020) 940-954.
[29] A. Jesson, C. Lu, G. Gupta, N. Beltran-Velez, A. Filos, J.N. Foerster, Y. Gal, Relu to the rescue: Improve your on-policy actor-critic with positive advantages, arXiv preprint arXiv:2306.01460,  (2023).
[30] M. Kölle, M. Hgog, F. Ritz, P. Altmann, M. Zorn, J. Stein, C. Linnhoff-Popien, Quantum advantage actor-critic for reinforcement learning, arXiv preprint arXiv:2401.07043,  (2024).
[31] S. Shen, V. Van Beek, A. Iosup, Statistical characterization of business-critical workloads hosted in cloud datacenters, in:  2015 15th IEEE/ACM international symposium on cluster, cloud and grid computing, IEEE, 2015, pp. 465-474.
[32] H. Gupta, A. Vahid Dastjerdi, S.K. Ghosh, R. Buyya, iFogSim: A toolkit for modeling and simulation of resource management techniques in the Internet of Things, Edge and Fog computing environments, Software: Practice and Experience, 47(9) (2017) 1275-1296.
[33] R.N. Calheiros, R. Ranjan, A. Beloglazov, C.A. De Rose, R. Buyya, CloudSim: a toolkit for modeling and simulation of cloud computing environments and evaluation of resource provisioning algorithms, Software: Practice and experience, 41(1) (2011) 23-50.
[34] M. Cheng, J. Li, S. Nazarian, DRL-cloud: Deep reinforcement learning-based resource provisioning and task scheduling for cloud service providers, in:  2018 23rd Asia and South Pacific design automation conference (ASP-DAC), IEEE, 2018, pp. 129-134.
[35] D. Aksu, S. Üstebay, M.A. Aydin, T. Atmaca, Intrusion detection with comparative analysis of supervised learning techniques and Fisher score feature selection algorithm, in:  International Symposium on Computer and Information Sciences, Springer, 2018, pp. 141-149.
[36] A. Horri, G. Dastghaibyfard, A novel cost-based model for energy consumption in cloud computing, ScientificWorldJournal, 2015 (2015) 724524.
[37] S. Sarmad Shah, A. Ali, Optimizing Resource Allocation and Energy Efficiency in Federated Fog Computing for IoT, arXiv e-prints,  (2025) arXiv: 2504.00791.
AUT Journal of Electrical Engineering
Volume 57, Issue 3
2025
Pages 589-610

Files

Share

How to cite

Statistics