Recurrent Neural networks (RNN) have shown promising potential for learningdynamics of sequential data. However, artificial neural networks are known toexhibit poor robustness in presence of input noise, where the sequentialarchitecture of RNNs exacerbates the problem. In this paper, we will use ideasfrom control and estimation theories to propose a tractable robustness analysisfor RNN models that are subject to input noise. The variance of the output ofthe noisy system is adopted as a robustness measure to quantify the impact ofnoise on learning. It is shown that the robustness measure can be estimatedefficiently using linearization techniques. Using these results, we proposed alearning method to enhance robustness of a RNN with respect to exogenousGaussian noise with known statistics. Our extensive simulations on benchmarkproblems reveal that our proposed methodology significantly improves robustnessof recurrent neural networks.