Semi-Supervised Learning (SSL) has received extensive attention in the domainof computer vision, leading to development of promising approaches such asFixMatch. In scenarios where training data is decentralized and resides onclient devices, SSL must be integrated with privacy-aware training techniquessuch as Federated Learning. We consider the problem of federated imageclassification and study the performance and privacy challenges with existingfederated SSL (FSSL) approaches. Firstly, we note that even state-of-the-artFSSL algorithms can trivially compromise client privacy and other real-worldconstraints such as client statelessness and communication cost. Secondly, weobserve that it is challenging to integrate EMA (Exponential Moving Average)updates into the federated setting, which comes at a trade-off betweenperformance and communication cost. We propose a novel approach FedSwitch, thatimproves privacy as well as generalization performance through ExponentialMoving Average (EMA) updates. FedSwitch utilizes a federated semi-supervisedteacher-student EMA framework with two features - local teacher adaptation andadaptive switching between teacher and student for pseudo-label generation. Ourproposed approach outperforms the state-of-the-art on federated imageclassification, can be adapted to real-world constraints, and achieves goodgeneralization performance with minimal communication cost overhead.