Abstract

This work explores entropy analysis as a tool for probing informationdistribution within Transformer-based architectures. By quantifying token-leveluncertainty and examining entropy patterns across different stages ofprocessing, we aim to investigate how information is managed and transformedwithin these models. As a case study, we apply the methodology to a GPT-basedlarge language model, illustrating its potential to reveal insights into modelbehavior and internal representations. This approach may offer insights intomodel behavior and contribute to the development of interpretability andevaluation frameworks for transformer-based models

 

Quick Read (beta)