This paper introduces a direct comparative study of Physics-Informed Neural Networks (PINNs) and Long Short-Term Memory (LSTM) networks for adaptive steam temperature control in Heat Recovery Steam Generators (HRSGs), particularly under valve leakage faults. Maintaining precise steam temperature in HRSGs is critical for efficiency and safety, yet traditional control strategies struggle with nonlinear, fault-induced dynamics. Both architectures are designed to adaptively tune the gains of a PI-plus-feedforward control law in real-time. The LSTM controller, a purely data-driven approach, was trained offline on historical operational data, while the PINN controller integrates fundamental thermodynamic laws directly into its online learning process through a physics-based loss function. Their performance was evaluated using a model validated with data from a combined cycle power plant, under normal load changes and a challenging valve leakage fault scenario. Results demonstrate that while the LSTM controller offers significant improvement over conventional methods, its performance degrades under the unseen fault. The PINN controller consistently delivered superior robustness and performance, achieving a 54\% reduction in integral absolute error compared to the LSTM under fault conditions. This study concludes that embedding physical knowledge into data-driven control is essential for developing reliable, fault-tolerant autonomous control systems in complex industrial applications.