While dynamic Neural Radiance Fields (NeRF) have shown success inhigh-fidelity 3D modeling of talking portraits, the slow training and inferencespeed severely obstruct their potential usage. In this paper, we propose anefficient NeRF-based framework that enables real-time synthesizing of talkingportraits and faster convergence by leveraging the recent success of grid-basedNeRF. Our key insight is to decompose the inherently high-dimensional talkingportrait representation into three low-dimensional feature grids. Specifically,a Decomposed Audio-spatial Encoding Module models the dynamic head with a 3Dspatial grid and a 2D audio grid. The torso is handled with another 2D grid ina lightweight Pseudo-3D Deformable Module. Both modules focus on efficiencyunder the premise of good rendering quality. Extensive experiments demonstratethat our method can generate realistic and audio-lips synchronized talkingportrait videos, while also being highly efficient compared to previousmethods.