Region anchors are the cornerstone of modern object detection techniques.State-of-the-art detectors mostly rely on a dense anchoring scheme, whereanchors are sampled uniformly over the spatial domain with a predefined set ofscales and aspect ratios. In this paper, we revisit this foundational stage.Our study shows that it can be done much more effectively and efficiently.Specifically, we present an alternative scheme, named Guided Anchoring, whichleverages semantic features to guide the anchoring. The proposed method jointlypredicts the locations where the center of objects of interest are likely toexist as well as the scales and aspect ratios at different locations. On top ofpredicted anchor shapes, we mitigate the feature inconsistency with a featureadaption module. We also study the use of high-quality proposals to improvedetection performance. The anchoring scheme can be seamlessly integrated toproposal methods and detectors. With Guided Anchoring, we achieve $9.1\%$higher recall on MS COCO with $90\%$ fewer anchors than the RPN baseline. Wealso adopt Guided Anchoring in Fast R-CNN, Faster R-CNN and RetinaNet,respectively improving the detection mAP by $2.2\%$, $2.7\%$ and $1.2\%$.