Abstract
Vertical Federated Learning (VFL) attracts increasing attention because itempowers multiple parties to jointly train a privacy-preserving model oververtically partitioned data. Recent research has shown that applyingzeroth-order optimization (ZOO) has many advantages in building a practical VFLalgorithm. However, a vital problem with the ZOO-based VFL is its slowconvergence rate, which limits its application in handling modern large models.To address this problem, we propose a cascaded hybrid optimization method inVFL. In this method, the downstream models (clients) are trained with ZOO toprotect privacy and ensure that no internal information is shared. Meanwhile,the upstream model (server) is updated with first-order optimization (FOO)locally, which significantly improves the convergence rate, making it feasibleto train the large models without compromising privacy and security. Wetheoretically prove that our VFL framework converges faster than the ZOO-basedVFL, as the convergence of our framework is not limited by the size of theserver model, making it effective for training large models with the major parton the server. Extensive experiments demonstrate that our method achievesfaster convergence than the ZOO-based VFL framework, while maintaining anequivalent level of privacy protection. Moreover, we show that the convergenceof our VFL is comparable to the unsafe FOO-based VFL baseline. Additionally, wedemonstrate that our method makes the training of a large model feasible.