Abstract
Quantization has been widely studied as an effective technique for reducingthe memory requirement of large language models (LLMs), potentially improvingthe latency time as well. Utilizing the characteristic of rotational invarianceof transformer, we propose the rotation-based saliency-aware weightquantization (ROSAQ), which identifies salient channels in the projectionfeature space, not in the original feature space, where the projected"principal" dimensions are naturally considered as "salient" features. Theproposed ROSAQ consists of 1) PCA-based projection, which first performsprincipal component analysis (PCA) on a calibration set and transforms via thePCA projection, 2) Salient channel dentification, which selects dimensionscorresponding to the K-largest eigenvalues as salient channels, and 3)Saliency-aware quantization with mixed-precision, which uses FP16 for salientdimensions and INT3/4 for other dimensions. Experiment results show that ROSAQshows improvements over the baseline saliency-aware quantization on theoriginal feature space and other existing quantization methods. With kernelfusion, ROSAQ presents about 2.3x speed up over FP16 implementation ingenerating 256 tokens with a batch size of 64.