Abstract
Machine learning models trained on private datasets have been shown to leaktheir private data. While recent work has found that the average data point israrely leaked, the outlier samples are frequently subject to memorization and,consequently, privacy leakage. We demonstrate and analyse an Onion Effect ofmemorization: removing the "layer" of outlier points that are most vulnerableto a privacy attack exposes a new layer of previously-safe points to the sameattack. We perform several experiments to study this effect, and understand whyit occurs. The existence of this effect has various consequences. For example,it suggests that proposals to defend against memorization without training withrigorous privacy guarantees are unlikely to be effective. Further, it suggeststhat privacy-enhancing technologies such as machine unlearning could actuallyharm the privacy of other users.