Visual attention serves as a means of feature selection mechanism in theperceptual system. Motivated by Broadbent's leaky filter model of selectiveattention, we evaluate how such mechanism could be implemented and affect thelearning process of deep reinforcement learning. We visualize and analyze thefeature maps of DQN on a toy problem Catch, and propose an approach to combinevisual selective attention with deep reinforcement learning. We experiment withoptical flow-based attention and A2C on Atari games. Experiment results showthat visual selective attention could lead to improvements in terms of sampleefficiency on tested games. An intriguing relation between attention and batchnormalization is also discovered.
Quick Read (beta)
An Initial Attempt of Combining Visual Selective Attention
with Deep Reinforcement Learning
Visual attention serves as a means of feature selection mechanism in the perceptual system. Motivated by Broadbent’s leaky filter model of selective attention, we evaluate how such mechanism could be implemented and affect the learning process of deep reinforcement learning. We visualize and analyze the feature maps of DQN on a toy problem Catch, and propose an approach to combine visual selective attention with deep reinforcement learning. We experiment with optical flow-based attention and A2C on Atari games. Experiment results show that visual selective attention could lead to improvements in terms of sample efficiency on tested games. An intriguing relation between attention and batch normalization is also discovered.
An Initial Attempt of Combining Visual Selective Attention
with Deep Reinforcement Learning
Liu Yuezhang††thanks: Work mostly performed at The University of Texas at Austin. Department of Automation Tsinghua University [email protected] Ruohan Zhang Department of Computer Science The University of Texas at Austin [email protected] Dana H. Ballard Department of Computer Science The University of Texas at Austin [email protected]
In recent years, deep reinforcement learning (RL) has made significant progress in many domains, such as video games (?; ?), the game of Go (?; ?) and continuous control tasks (?; ?). However, despite their impressive super-human end performance, many of the deep reinforcement learning methods require a large amount of samples for training.
Visual attention, especially human gaze attention, has been widely studied in vision science. Broadbent proposed his famous filter model of selective attention in perceptual system (?; ?). Research afterwards focused on modeling human gaze, based on the stimulus salience (bottom-up) (?), task priority (top-down) (?), or a combination of the both (?). In these studies of gaze behaviors, it is conjectured that gaze serves as a feature selection mechanism based on saliency or task demand. Since nowadays feature selection is a core issue in machine learning, it is intuitive to study whether such mechanism could be beneficial for reinforcement learning since deep RL also requires feature extraction. Successful stories of combining attention with machine learning can be found in nature language processing (?), computer vision (?) and imitation learning (?).
In this work, we propose a method to combine visual selective attention with deep reinforcement learning. We first analyze the feature maps of a deep neural network trained by reinforcement learning. Then we propose our method and evaluate on the toy problems as well as Atari games.
Recurrent models for visual attention.
Several works implement visual attention using the glimpse network in combination with recurrent models (?; ?). This approach was later extended to deep reinforcement learning in the Atari games domain, either using recurrent models (?) or glimpse sensory (?). However, these approaches only attempt to integrate visual attention at input level. Therefore, they resembled foveal vision at retinal level rather than visual selective attention, which might also exist in the deeper structure of the network. These pioneered works showed promising results which encourages us to further study how visual attention could benefit deep reinforcement learning.
Understanding deep RL.
Understanding features learned by deep RL is critical for our goal since we target at combining attention with learned features. Some research visualize and analyze the property of the learned policy by deep Q-network (DQN) (?) based on t-SNE and SAMDP (?). Others use perturbation to extract the visual features the agent attended to (?). However, most of these work focused on either the input layer or the last several fully connected layers. Thus, the feature maps in the mid layers and their relation with visual inputs and task reward requires further investigation.
Combining different sources of knowledge in CNN.
Hypothetically, introducing visual attention into deep RL can be thought as integrating a different source of visual information into an existing neural network. Similar problems have been studied extensively by the computer vision community. For example, multiplicative fusion with optical flow in CNN is proposed for action recognition (?). More fusion methods in two-stream network were discussed (?). As for video prediction, action-conditional architecture also involve a multiplication structure to combine action information (?).
The Toy Problem Catch
A toy problem called Catch was designed to study the properties of the CNN feature maps in deep reinforcement learning (see Figure 1). A ball falls down from the top of the screen, and the agent controls the paddle at the bottom to catch the ball. To be more specific:
State space: , with only black and white pixels. The ball is of size and the paddle is .
Initial state: at the beginning of each episode, the ball and the paddle are placed at a random position at the top and the bottom row respectively.
Action space: actions, move the paddle 1 pixel left, stay, or move right.
Dynamics: the ball falls at the speed of 1 pixel per timestep.
Episode length: each episode ends when the ball reaches the bottom, thus it is fixed to 20 timesteps.
Reward: the player only receives reward at the end of episodes, if the paddle catches the ball successfully, otherwise.
Understanding Feature Maps in DQN
Visual attention can be treated as a special visual input that highlights important spatial information. Such additional visual input, or mask, could be applied at input level (?) or at higher level, i.e., deeper in the network (?; ?). We are interested in the effect of introducing attention at the last convolution layer of DQN. Therefore we first need to examine the features learned by the last convolution layer of DQN to ensure that it is sensible to combine attention with these features.
To achieve this goal, we choose the classic DQN as our learning algorithm for Catch (?). We preserve the technical details of DQN as much as possible except for three modifications. Since the observation space reduces from to , we simply remove the first convolution layer of DQN, as the size of the second convolution layer in original DQN-CNN is exactly . We use Adam optimizer here, rather than RMSProp, with a learning rate . As for the hyperparameters, we modify several hyperparameters as the game is simple.
The convolution layers of DQN generally serve as a visual feature extractor. In the Catch environment, according to the modifications we mention above, the size of the input and feature map at each layer are , and respectively. We now visualize the feature maps of the last convolution layer.
A fixed set of visual stimulus is made to evaluate the learning process of DQN, as shown in Figure 1. The stimulus set is feed to the trained DQN. We then record the outputs of feature maps from the last convolution layer. One way to visualize these feature maps is to directly average all the feature maps, as shown in Figure 2. Notice that these averaged feature maps are very similar to the visual inputs shown in Figure 1. They all have a ’ball’ at the upper left corner, and a ’paddle’ at the bottom.
We hypothesize that the blurred paddle-like object in averaged feature maps may represent a distribution related to the position of the paddle, rather than the exact shape of the paddle, where the former is closely related to the task reward. To test this hypothesis, we manipulate reward-related variables (reward and discount factor) to further study the potential relation between feature maps and task reward. The results are shown in Figure 3, where (b) visualizes the averaged feature map learned with the original Catch environment.
Zero Discounted factor. If the discounted factor is set to 0, the intensity of the paddle object significantly decreases as shown in 3 (c).
No reward. Furthermore, we set the reward to 0, which means the agent would not receive any reward from this environment. Then the paddle object completely disappeared as shown in (d).
These findings are consistent with previous results that the features learned by a neural network is modulated by task-related variables (?). We conclude that the feature maps learned by DQN is a joint representation of both visual inputs and task reward. Visual spatial information is preserved through convolution layers, and the intensity of the feature maps is associated with the reward. Therefore, applying visual attention at the last convolution layer is a reasonable choice.
Combining Visual Attention with Deep RL
We now propose a method to combine visual selective attention with deep reinforcement learning, as shown in Figure 4.
The feature maps of size with channels at time are:
where denotes last frames of visual inputs with size .
Let denotes some function generating a visual attention map based on the visual inputs at time :
We first reshape the attention mask to be the same shape as the feature maps . The values in the attention mask are in the range instead of an ordinary mask. Hence no information is discarded but more important features are scaled up. In this paper, we set .
Next, we simply multiply the attention mask with the feature maps of the last convolution layer elementwise:
Moreover, the scale of the feature maps is also critical for deep reinforcement learning. Therefore we add a batch normalization (?) layer after the multiplication operation:
The filtered feature vector is then fed to the fully connected layers in DQN. Such normalization structure could be understood as a naive brightness adaptation mechanism (?), and is also proposed in other attention models (?).
Experiments and Result
Given the techniques proposed, we experiment using DQN with or without attention and batch normalization. We first combine hand-crafted attention with DQN (?) on the Catch environments as a sanity check. After that we extend to Atari Games in which we combine optical flow-based attention with A2C (?). Due to the stochastic nature of the training process and in compliance with the reproducibility standard (?), all experiments are repeated with different random seeds.
Experiments with Catch
The effect of attention may not be very salient given a simple visual task with clean inputs. Inspired by (?), we modify the original Catch environment to include certain level of background noise. A latticed background which has a pixel value of 0.9 at every odd coordinate position is added upon the original image. We name this environment ’Catch with Background’, or simply ’CatchBG’ (see Figure 5).
CatchBG could be a lot more difficult for deep reinforcement learning methods without visual attention, since the agent would observe a screen full of ’balls’ to catch - the agent is forced to learn to tell the difference between 0.9 and 1 in grayscale.
Here, we use hand-crafted features as visual attention. The attention map marks the true position of the ball and the paddle. So it is identical to the visual inputs of the original Catch environments. We downsample the attention map by maxpooling to match the size of the feature maps, rescale it and combine with DQN by multiplication. Results comparing different methods on the Catch environments are shown in Figure 6.
In the Catch environment, all four methods perform similarly. All methods get a more than 0.9 averaged return (which also denotes an over 90% successful rate since we average along past 100 episodes and set the final exploration rate ) after 40000 timesteps, indicating that DQN has learned the task successfully.
However, for the CatchBG environment, only the method with both visual attention and batch normalization can still achieve a similar performance - over 90% success rate. The other three methods can only success in about 50% of the trials – which is exactly the probability that the ball is initialized in the columns with no background, as we only place background at the odd coordinate grids - therefore, fail to learn the difference between the ball and the background in this environment. In addition, attention alone without normalization also results in a poor performance in this test.
Experiments with Atari Games
For more complex visuomotor tasks such as Atari games, a reasonable source of attention is motion, as objects movement are good indicator of important visual features in video games. Hence we use optical flow between two frames (?) to construct an attention map. We combine optical flow-based attention with A2C on Atari games following the same method and test our approach on four popular Atari games - Breakout, Seaquest, MsPacman and Centipede. Results are shown in Figure 7. We use OpenAI baselines (?) with default hyperparameters.
Comparing with the original A2C algorithm, moderate performance improvements are observed at different stages in all four tested games. Improvements at early stage of learning are present in Breakout and Seaquest, since moving objects are critical for both of the games, which indicates that the optical flow might be an appropriate attention source. Also the rules of the games are quite straightforward, therefore improvements in sensory system are more likely to result in improvements in task performance. MsPacman and Centipede are games with many visually identical objects, hence motion-based attention is likely to help to tell apart the objects that matters for the current decision step. Experiments on more games should be conduct to provide a more comprehensive evaluation for the effect of introducing visual attention.
Before drawing the conclusion, we would like to discuss the potential relation between our approach and Broadbent’s leaky filter model of selective attention (?). The model is shown in Figure 8.
In Broadbent’s filter model of selective attention, messages flow through the sensory memory, filter and detector successively (?). In our model, these three components roughly correspond to the convolution layers, integration (multiplication and normalization) and fully connected layers in the network. The sensory memory holds the input messages for a short period, extracts features and passes to the filter. The filter then selects the attended messages based on their physical characteristics, as we choose optical flow for instance, and filters out the others. The attended messages are then passed to the detector for further cognitive processes (?).
We should note that in the original Broadbent’s filter model, only attended messages could pass through the filter. However, a modification was proposed based on several psychology experiments, which suggested that unattended messages are also able to pass through the filter (?). It is known as leaky filter model. Therefore, we multiply our visual attention scaling from 1 rather than 0 to simply keep all the unattended messages.
A significant difference still exists between the sensory memory module and convolution layers, since CNNs do not explicitly model memory. It could be an interesting topic to study how sensory memory contribute to the generation of visual attention function . In addition, unlike the visual scanning that could only exist at the input layer, we presume that the selective attention mechanism might be universal as in the neural gain model (?), and could be utilized hierarchically in any layer.
Conclusion and Future Work
In this paper, we aim at combining visual selective attention with deep reinforcement learning. We first try to understand the feature maps on the toy problem Catch, and then propose our approach for the fusion problem. An interesting interaction effect between attention and normalization is discovered and should be studied more carefully. The new learning method achieves performance improvements on tested games. The simple CatchBG example demonstrates that visual attention could be particularly helpful in an environment with cluttered and noisy visual input.
However, many issues remain to be solved to achieve a potentially better performance. For example, the attention map is approximated by optical flow, which can be replaced with a more comprehensive saliency model (?), or a real human attention model from gaze (?), or generated by the sensory memory as mentioned in the discussion. It’s also possible that the first several convolution layers have stronger correlation with input, while the last several layers correlates more with the value function. Hence at which layer (or probably all layers respectively) attention should be integrated to produce the best performance could also be explored.
The work was supported by NIH Grant EY05729, NIH Grant T32 EY21462-6, NSF Grant CNS-1624378, Google AR/VR Research Award and Tsinghua Initiative Scientific Research Program.
- [Broadbent 1958] Broadbent, D. E. 1958. Perception and Communication. Pergamon Press.
- [Dhariwal et al. 2017] Dhariwal, P.; Hesse, C.; Klimov, O.; Nichol, A.; Plappert, M.; Radford, A.; Schulman, J.; Sidor, S.; Wu, Y.; and Zhokhov, P. 2017. Openai baselines. https://github.com/openai/baselines.
- [Eldar, Cohen, and Niv 2013] Eldar, E.; Cohen, J. D.; and Niv, Y. 2013. The effects of neural gain on attention and learning. Nature Neuroscience 16(8):1146–1153.
- [Farnebäck 2003] Farnebäck, G. 2003. Two-frame motion estimation based on polynomial expansion. In Scandinavian conference on Image analysis, 363–370. Springer.
- [Feichtenhofer, Pinz, and Zisserman 2016] Feichtenhofer, C.; Pinz, A.; and Zisserman, A. 2016. Convolutional two-stream network fusion for video action recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 1933–1941. IEEE.
- [Gluck and Myers 1993] Gluck, M. A., and Myers, C. E. 1993. Hippocampal mediation of stimulus representation: A computational theory. Hippocampus 3(4):491–516.
- [Goldstein 2014] Goldstein, E. B. 2014. Attention. In Cognitive Psychology: Connecting Mind, Research and Everyday Experience. Cengage Learning, 4th edition. 84–117.
- [Gregor et al. 2018] Gregor, M.; Nemec, D.; Janota, A.; and Pirník, R. 2018. A visual attention operator for playing pac-man. In 2018 ELEKTRO, 1–6.
- [Greydanus et al. 2018] Greydanus, S.; Koul, A.; Dodge, J.; and Fern, A. 2018. Visualizing and understanding atari agents. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, 1792–1801. PMLR.
- [Gu et al. 2016] Gu, S.; Lillicrap, T.; Sutskever, I.; and Levine, S. 2016. Continuous deep q-learning with model-based acceleration. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, 2829–2838. PMLR.
- [Hayhoe and Ballard 2005] Hayhoe, M., and Ballard, D. 2005. Eye movements in natural behavior. Trends in Cognitive Sciences 9(4):188–194.
- [Henderson et al. 2018] Henderson, P.; Islam, R.; Bachman, P.; Pineau, J.; Precup, D.; and Meger, D. 2018. Deep reinforcement learning that matters. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, New Orleans, Louisiana, USA, February 2-7, 2018.
- [Hessel et al. 2018] Hessel, M.; Modayil, J.; Hasselt, H. v.; Schaul, T.; Ostrovski, G.; Dabney, W.; Horgan, D.; Piot, B.; Azar, M.; and Silver, D. 2018. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, 3215–3222.
- [Ioffe and Szegedy 2015] Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, 448–456. PMLR.
- [Itti, Koch, and Niebur 1998] Itti, L.; Koch, C.; and Niebur, E. 1998. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on pattern analysis and machine intelligence 20(11):1254–1259.
- [Kanan et al. 2009] Kanan, C.; Tong, M. H.; Zhang, L.; and Cottrell, G. W. 2009. SUN: Top-down saliency using natural statistics. Visual cognition 17(6):979–1003.
- [Larochelle and Hinton 2010] Larochelle, H., and Hinton, G. E. 2010. Learning to combine foveal glimpses with a third-order boltzmann machine. In Advances in Neural Information Processing Systems 23. Curran Associates, Inc. 1243–1251.
- [Lillicrap et al. 2015] Lillicrap, T. P.; Hunt, J. J.; Pritzel, A.; Heess, N.; Erez, T.; Tassa, Y.; Silver, D.; and Wierstra, D. 2015. Continuous control with deep reinforcement learning. arXiv:1509.02971 [cs, stat].
- [Mnih et al. 2014] Mnih, V.; Heess, N.; Graves, A.; and kavukcuoglu, k. 2014. Recurrent models of visual attention. In Advances in Neural Information Processing Systems 27. Curran Associates, Inc. 2204–2212.
- [Mnih et al. 2015] Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015. Human-level control through deep reinforcement learning. Nature 518(7540):529–533.
- [Mnih et al. 2016] Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016. Asynchronous methods for deep reinforcement learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, 1928–1937. PMLR.
- [Oh et al. 2015] Oh, J.; Guo, X.; Lee, H.; Lewis, R. L.; and Singh, S. 2015. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems 28. Curran Associates, Inc. 2863–2871.
- [Palazzi et al. 2018] Palazzi, A.; Abati, D.; Calderara, S.; Solera, F.; and Cucchiara, R. 2018. Predicting the driver’s focus of attention: the DR(eye)VE project. IEEE Transactions on Pattern Analysis and Machine Intelligence 1–1.
- [Park et al. 2016] Park, E.; Han, X.; Berg, T. L.; and Berg, A. C. 2016. Combining multiple sources of knowledge in deep CNNs for action recognition. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), 1–8.
- [Purves et al. 2004] Purves, D.; Augustine, G. J.; Fitzpatrick, D.; Hall, W. C.; LaMantia, A.-S.; McNamara, J. O.; and Williams, S. M. 2004. Vision: The eye. In Neuroscience. Sinauer Associates, Inc., 3rd edition. 229–258.
- [Silver et al. 2016] Silver, D.; Huang, A.; Maddison, C. J.; Guez, A.; Sifre, L.; Driessche, G. v. d.; Schrittwieser, J.; Antonoglou, I.; Panneershelvam, V.; Lanctot, M.; Dieleman, S.; Grewe, D.; Nham, J.; Kalchbrenner, N.; Sutskever, I.; Lillicrap, T.; Leach, M.; Kavukcuoglu, K.; Graepel, T.; and Hassabis, D. 2016. Mastering the game of go with deep neural networks and tree search. Nature 529(7587):484.
- [Silver et al. 2017] Silver, D.; Schrittwieser, J.; Simonyan, K.; Antonoglou, I.; Huang, A.; Guez, A.; Hubert, T.; Baker, L.; Lai, M.; Bolton, A.; Chen, Y.; Lillicrap, T.; Hui, F.; Sifre, L.; Driessche, G. v. d.; Graepel, T.; and Hassabis, D. 2017. Mastering the game of go without human knowledge. Nature 550(7676):354.
- [Sorokin et al. 2015] Sorokin, I.; Seleznev, A.; Pavlov, M.; Fedorov, A.; and Ignateva, A. 2015. Deep attention recurrent q-network. arXiv:1512.01693 [cs].
- [Treisman 1964] Treisman, A. 1964. Monitoring and storage of irrelevant messages in selective attention. Journal of Verbal Learning and Verbal Behavior 3(6):449–459.
- [Vaswani et al. 2017] Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, .; and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc. 5998–6008.
- [Zahavy, Ben-Zrihem, and Mannor 2016] Zahavy, T.; Ben-Zrihem, Z.; and Mannor, S. 2016. Graying the black box: Understanding DQNs. Proceedings of The 33rd International Conference on Machine Learning 48:1899–1908.
- [Zhang et al. 2018] Zhang, R.; Liu, Z.; Zhang, L.; Whritner, J.; Muller, K.; Hayhoe, M.; and Ballard, D. 2018. AGIL: Learning attention from human for visuomotor tasks. The European Conference on Computer Vision (ECCV).