Abstract
This study centers on overcoming the challenge of selecting the bestannotators for each query in Active Learning (AL), with the objective ofminimizing misclassifications. AL recognizes the challenges related to cost andtime when acquiring labeled data, and decreases the number of labeled dataneeded. Nevertheless, there is still the necessity to reduce annotation errors,aiming to be as efficient as possible, to achieve the expected accuracy faster.Most strategies for query-annotator pairs do not consider internal factors thataffect productivity, such as mood, attention, motivation, and fatigue levels.This work addresses this gap in the existing literature, by not onlyconsidering how the internal factors influence annotators (mood and fatiguelevels) but also presenting a new query-annotator pair strategy, using aKnowledge-Based Recommendation System (RS). The RS ranks the availableannotators, allowing to choose one or more to label the queried instance usingtheir past accuracy values, and their mood and fatigue levels, as well asinformation about the instance queried. This work bases itself on existingliterature on mood and fatigue influence on human performance, simulatingannotators in a realistic manner, and predicting their performance with the RS.The results show that considering past accuracy values, as well as mood andfatigue levels reduces the number of annotation errors made by the annotators,and the uncertainty of the model through its training, when compared to notusing internal factors. Accuracy and F1-score values were also better in theproposed approach, despite not being as substantial as the aforementioned. Themethodologies and findings presented in this study begin to explore the openchallenge of human cognitive factors affecting AL.