Abstract
Pre-trained language models such as BERT have achieved great success in abroad range of natural language processing tasks. However, BERT cannot wellsupport E-commerce related tasks due to the lack of two levels of domainknowledge, i.e., phrase-level and product-level. On one hand, many E-commercetasks require an accurate understanding of domain phrases, whereas suchfine-grained phrase-level knowledge is not explicitly modeled by BERT'straining objective. On the other hand, product-level knowledge like productassociations can enhance the language modeling of E-commerce, but they are notfactual knowledge thus using them indiscriminately may introduce noise. Totackle the problem, we propose a unified pre-training framework, namely,E-BERT. Specifically, to preserve phrase-level knowledge, we introduce AdaptiveHybrid Masking, which allows the model to adaptively switch from learningpreliminary word knowledge to learning complex phrases, based on the fittingprogress of two modes. To utilize product-level knowledge, we introduceNeighbor Product Reconstruction, which trains E-BERT to predict a product'sassociated neighbors with a denoising cross attention layer. Our investigationreveals promising results in four downstream tasks, i.e., review-based questionanswering, aspect extraction, aspect sentiment classification, and productclassification.