Abstract
We present ActionDiffusion -- a novel diffusion model for procedure planningin instructional videos that is the first to take temporal inter-dependenciesbetween actions into account in a diffusion model for procedure planning. Thisapproach is in stark contrast to existing methods that fail to exploit the richinformation content available in the particular order in which actions areperformed. Our method unifies the learning of temporal dependencies betweenactions and denoising of the action plan in the diffusion process by projectingthe action information into the noise space. This is achieved 1) by addingaction embeddings in the noise masks in the noise-adding phase and 2) byintroducing an attention mechanism in the noise prediction network to learn thecorrelations between different action steps. We report extensive experiments onthree instructional video benchmark datasets (CrossTask, Coin, and NIV) andshow that our method outperforms previous state-of-the-art methods on allmetrics on CrossTask and NIV and all metrics except accuracy on Coin dataset.We show that by adding action embeddings into the noise mask the diffusionmodel can better learn action temporal dependencies and increase theperformances on procedure planning.