Abstract
Agent-based Transformers have been widely adopted in recent reinforcementlearning advances due to their demonstrated ability to solve complex tasks.However, the high computational complexity of Transformers often results insignificant energy consumption, limiting their deployment in real-worldautonomous systems. Spiking neural networks (SNNs), with their biologicallyinspired structure, offer an energy-efficient alternative for machine learning.In this paper, a novel Spike-Transformer Reinforcement Learning (STRL)algorithm that combines the energy efficiency of SNNs with the powerfuldecision-making capabilities of reinforcement learning is developed.Specifically, an SNN using multi-step Leaky Integrate-and-Fire (LIF) neuronsand attention mechanisms capable of processing spatio-temporal patterns overmultiple time steps is designed. The architecture is further enhanced withstate, action, and reward encodings to create a Transformer-like structureoptimized for reinforcement learning tasks. Comprehensive numerical experimentsconducted on state-of-the-art benchmarks demonstrate that the proposed SNNTransformer achieves significantly improved policy performance compared toconventional agent-based Transformers. With both enhanced energy efficiency andpolicy optimality, this work highlights a promising direction for deployingbio-inspired, low-cost machine learning models in complex real-worlddecision-making scenarios.