Abstract
The increasing computational and memory demands in deep learning presentsignificant challenges, especially in resource-constrained environments. Weintroduce a zero-order quantized optimization (ZOQO) method designed fortraining models with quantized parameters and operations. Our approachleverages zero-order approximations of the gradient sign and adapts thelearning process to maintain the parameters' quantization without the need forfull-precision gradient calculations. We demonstrate the effectiveness of ZOQOthrough experiments in fine-tuning of large language models and black-boxadversarial attacks. Despite the limitations of zero-order and quantizedoperations training, our method achieves competitive performance compared tofull-precision methods, highlighting its potential for low-resourceenvironments.