Image rasterization is a mature technique in computer graphics, while imagevectorization, the reverse path of rasterization, remains a major challenge.Recent advanced deep learning-based models achieve vectorization and semanticinterpolation of vector graphs and demonstrate a better topology of generatingnew figures. However, deep models cannot be easily generalized to out-of-domaintesting data. The generated SVGs also contain complex and redundant shapes thatare not quite convenient for further editing. Specifically, the cruciallayer-wise topology and fundamental semantics in images are still not wellunderstood and thus not fully explored. In this work, we propose Layer-wiseImage Vectorization, namely LIVE, to convert raster images to SVGs andsimultaneously maintain its image topology. LIVE can generate compact SVG formswith layer-wise structures that are semantically consistent with humanperspective. We progressively add new bezier paths and optimize these pathswith the layer-wise framework, newly designed loss functions, andcomponent-wise path initialization technique. Our experiments demonstrate thatLIVE presents more plausible vectorized forms than prior works and can begeneralized to new images. With the help of this newly learned topology, LIVEinitiates human editable SVGs for both designers and other downstreamapplications. Codes are made available athttps://github.com/Picsart-AI-Research/LIVE-Layerwise-Image-Vectorization.