Factoring a matrix into two low rank matrices is at the heart of manyproblems. The problem of matrix completion especially uses it to decompose asparse matrix into two non sparse, low rank matrices which can then be used topredict unknown entries of the original matrix. We present a scalable anddecentralized approach in which instead of learning two factors for theoriginal input matrix, we decompose the original matrix into a grid blocks,each of whose factors can be individually learned just by communicating(gossiping) with neighboring blocks. This eliminates any need for a centralserver. We show that our algorithm performs well on both synthetic and realdatasets.