Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale

  • 2022-01-13 17:53:31
  • Gang Li, Gilles Baechler, Manuel Tragut, Yang Li
  • 0

Abstract

The layout of a mobile screen is a critical data source for UI designresearch and semantic understanding of the screen. However, UI layouts inexisting datasets are often noisy, have mismatches with their visualrepresentation, or consists of generic or app-specific types that are difficultto analyze and model. In this paper, we propose the CLAY pipeline that uses adeep learning approach for denoising UI layouts, allowing us to automaticallyimprove existing mobile UI layout datasets at scale. Our pipeline takes boththe screenshot and the raw UI layout, and annotates the raw layout by removingincorrect nodes and assigning a semantically meaningful type to each node. Toexperiment with our data-cleaning pipeline, we create the CLAY dataset of59,555 human-annotated screen layouts, based on screenshots and raw layoutsfrom Rico, a public mobile UI corpus. Our deep models achieve high accuracywith F1 scores of 82.7% for detecting layout objects that do not have a validvisual representation and 85.9% for recognizing object types, whichsignificantly outperforms a heuristic baseline. Our work lays a foundation forcreating large-scale high quality UI layout datasets for data-driven mobile UIresearch and reduces the need of manual labeling efforts that are prohibitivelyexpensive.

 

Quick Read (beta)

loading the full paper ...