Information freshness optimization in energy harvesting IoT networks

Information freshness is crucial for time-critical IoT applications, e.g., environment monitoring and control systems. We consider an Internet of things (IoT) network with multiple energy harvesting sensors, users, and an edge node. The users are interested in time-sensitive information about physical quantities, each measured by a sensor. Users make on-demand requests to a cache-enabled edge node where the cache contains the most recently received measurements from each sensor. The edge node serves the users' requests by deciding whether to command the corresponding sensors to send fresh status updates or use the aged data in the cache. We aim to find the best actions of the edge node at each time slot for each sensor, i.e., optimal policies, that minimizes the average age of information (AoI) of the served measurements, i.e., average on-demand AoI.

In the present work, we first study status updating for decoupled sensors, i.e., the sensors have independent communication channels to the edge node. We model this problem as a Markov decision process (MDP) and developed two classes of reinforcement learning (RL) based algorithms: a model-based relative value iteration algorithm (RVIA) relying on dynamic programming, and a model-free Q-learning method. Then, we study status updating in IoT networks under a transmission constraint where the edge node can command only a limited number of sensors at each time slot, i.e., leading to per-slot transmission constraint. We model the problem as an MDP for which an iterative algorithm is proposed to obtain an optimal policy. Note, however, that the computational complexity of finding an optimal policy increases exponentially in the number of sensors. Thus, we develop an asymptotically optimal low-complexity algorithm -- termed relax-then-truncate -- and prove that it is optimal as the number of sensors goes to infinity. Finally, we study status updating under inexact knowledge about the battery levels of the sensors; namely, the edge node is informed about the sensors’ battery levels only via the status update packets, leading to uncertainty about the battery levels for the decision-making. Accounting for the partial battery knowledge, we model the problem as a partially observable MDP (POMDP) for which we develop a novel dynamic programming algorithm that obtains an optimal policy.