Loading...

Model-free LQR design by Q-function learning

Farjadnasab, M ; Sharif University of Technology | 2022

131 Viewed
  1. Type of Document: Article
  2. DOI: 10.1016/j.automatica.2021.110060
  3. Publisher: Elsevier Ltd , 2022
  4. Abstract:
  5. Reinforcement learning methods such as Q-learning have shown promising results in the model-free design of linear quadratic regulator (LQR) controllers for linear time-invariant (LTI) systems. However, challenges such as sample-efficiency, sensitivity to hyper-parameters, and compatibility with classical control paradigms limit the integration of such algorithms in critical control applications. This paper aims to take some steps towards bridging the well-known classical control requirements and learning algorithms by using optimization frameworks and properties of conic constraints. Accordingly, a new off-policy model-free approach is proposed for learning the Q-function and designing the discrete-time LQR controller. The design procedure is based on non-iterative semi-definite programs (SDP) with linear matrix inequality (LMI) constraints. It is sample-efficient, inherently robust to model uncertainties, and does not require an initial stabilizing controller. The proposed model-free approach is extended to distributed control of interconnected systems, as well. The performance of the presented design is evaluated on several stable and unstable synthetic systems. The data-driven control scheme is also implemented on the IEEE 39-bus New England power grid. The results confirm optimality, sample-efficiency, and satisfactory performance of the proposed approach in centralized and distributed design. © 2021 Elsevier Ltd
  6. Keywords:
  7. Convex optimization ; Distributed control ; Linear quadratic regulation (LQR) ; Semi-definite programming (SDP) ; Controllers ; Design ; Distributed parameter control systems ; Efficiency ; Electric power system control ; Electric power transmission networks ; Iterative methods ; Learning algorithms ; Linear control systems ; Linear matrix inequalities ; Reinforcement learning ; Uncertainty analysis ; Classical control ; Convex optimisation ; Distributed-control ; Linear quadratic regulation ; Linear quadratic regulations ; Model free ; Q-functions ; Q-learning ; Semi-definite programming
  8. Source: Automatica ; Volume 137 , 2022 ; 00051098 (ISSN)
  9. URL: https://www.sciencedirect.com/science/article/abs/pii/S0005109821005884