Abstract
We introduce a novel positional encoding strategy for Transformer-stylemodels, addressing the shortcomings of existing, often ad hoc, approaches. Ourframework provides a flexible mapping from the algebraic specification of adomain to an interpretation as orthogonal operators. This design preserves thealgebraic characteristics of the source domain, ensuring that the model upholdsits desired structural properties. Our scheme can accommodate variousstructures, ncluding sequences, grids and trees, as well as their compositions.We conduct a series of experiments to demonstrate the practical applicabilityof our approach. Results suggest performance on par with or surpassing thecurrent state-of-the-art, without hyper-parameter optimizations or "tasksearch" of any kind. Code is available athttps://github.com/konstantinosKokos/ape.