Abstract
Multivariate information theory provides a general and principled frameworkfor understanding how the components of a complex system are connected.Existing analyses are coarse in nature -- built up from characterizations ofdiscrete subsystems -- and can be computationally prohibitive. In this work, wepropose to study the continuous space of possible descriptions of a compositesystem as a window into its organizational structure. A description consists ofspecific information conveyed about each of the components, and the space ofpossible descriptions is equivalent to the space of lossy compression schemesof the components. We introduce a machine learning framework to optimizedescriptions that extremize key information theoretic quantities used tocharacterize organization, such as total correlation and O-information. Throughcase studies on spin systems, Sudoku boards, and letter sequences from naturallanguage, we identify extremal descriptions that reveal how system-widevariation emerges from individual components. By integrating machine learninginto a fine-grained information theoretic analysis of composite randomvariables, our framework opens a new avenues for probing the structure ofreal-world complex systems.