Abstract
Cross-validation (CV) is a common method to tune machine learning methods andcan be used for model selection in regression as well. Because of thestructured nature of small, traditional experimental designs, the literaturehas warned against using CV in their analysis. The striking increase in the useof machine learning, and thus CV, in the analysis of experimental designs, hasled us to empirically study the effectiveness of CV compared to other methodsof selecting models in designed experiments, including the little bootstrap. Weconsider both response surface settings where prediction is of primaryinterest, as well as screening where factor selection is most important.Overall, we provide evidence that the use of leave-one-out cross-validation(LOOCV) in the analysis of small, structured is often useful. More general$k$-fold CV may also be competitive but its performance is uneven.