We consider the problem of scaling automated suggested replies for Outlookemail system to multiple languages. Faced with increased compute requirementsand low resources for language expansion, we build a single universal model forimproving the quality and reducing run-time costs of our production system.However, restricted data movement across regional centers prevents jointtraining across languages. To this end, we propose a multi-task continuallearning framework, with auxiliary tasks and language adapters to learnuniversal language representation across regions. The experimental results showpositive cross-lingual transfer across languages while reducing catastrophicforgetting across regions. Our online results on real user traffic showsignificant gains in CTR and characters saved, as well as 65% training costreduction compared with per-language models. As a consequence, we have scaledthe feature in multiple languages including low-resource markets.