Abstract
In this paper, we propose a deep reinforcement learning (DRL) based mobilityload balancing (MLB) algorithm along with a two-layer architecture to solve thelarge-scale load balancing problem for ultra-dense networks (UDNs). Ourcontribution is three-fold. First, this work proposes a two-layer architectureto solve the large-scale load balancing problem in a self-organized manner. Theproposed architecture can alleviate the global traffic variations bydynamically grouping small cells into self-organized clusters according totheir historical loads, and further adapt to local traffic variations throughintra-cluster load balancing afterwards. Second, for the intra-cluster loadbalancing, this paper proposes an off-policy DRL-based MLB algorithm toautonomously learn the optimal MLB policy under an asynchronous parallellearning framework, without any prior knowledge assumed over the underlying UDNenvironments. Moreover, the algorithm enables joint exploration with multiplebehavior policies, such that the traditional MLB methods can be used to guidethe learning process thereby improving the learning efficiency and stability.Third, this work proposes an offline-evaluation based safeguard mechanism toensure that the online system can always operate with the optimal andwell-trained MLB policy, which not only stabilizes the online performance butalso enables the exploration beyond current policies to make full use ofmachine learning in a safe way. Empirical results verify that the proposedframework outperforms the existing MLB methods in general UDN environmentsfeatured with irregular network topologies, coupled interferences, and randomuser movements, in terms of the load balancing performance.