Deep Reinforcement Learning (DRL) sometimes needs a large amount of data toconverge in the training procedure and in some cases, each action of the agentmay produce regret. This barrier naturally motivates different data sets orenvironment owners to cooperate to share their knowledge and train their agentsmore efficiently. However, it raises privacy concerns if we directly merge theraw data from different owners. To solve this problem, we proposed a new DeepNeural Network (DNN) architecture with both global NN and local NN, and adistributed training framework. We allow the global weights to be updated byall the collaborator agents while the local weights are only updated by theagent they belong to. In this way, we hope the global weighs can share thecommon knowledge among these collaborators while the local NN can keep thespecialized properties and ensure the agent to be compatible with its specificenvironment. Experiments show that the framework can efficiently help agents inthe same or similar environments to collaborate in their training process andgain a higher convergence rate and better performance.