Deep Reinforcement Learning (DRL) has been applied to address a variety ofcooperative multi-agent problems with either discrete action spaces orcontinuous action spaces. However, to the best of our knowledge, no previouswork has ever succeeded in applying DRL to multi-agent problems withdiscrete-continuous hybrid (or parameterized) action spaces which is verycommon in practice. Our work fills this gap by proposing two novel algorithms:Deep Multi-Agent Parameterized Q-Networks (Deep MAPQN) and Deep Multi-AgentHierarchical Hybrid Q-Networks (Deep MAHHQN). We follow the centralizedtraining but decentralized execution paradigm: different levels ofcommunication between different agents are used to facilitate the trainingprocess, while each agent executes its policy independently based on localobservations during execution. Our empirical results on several challengingtasks (simulated RoboCup Soccer and game Ghost Story) show that both Deep MAPQNand Deep MAHHQN are effective and significantly outperform existing independentdeep parameterized Q-learning method.