Our software package MACE (Multi-Annotator Competence Estimation) is out! It provides competence estimates of the individual annotators and the most likely answer to each item. All you need to provide is a CSV file with one item per line.
In tests, MACE’s trust estimates correlated highly with the annotators’ true competence, and it achieved accuracies of over 0.9 on several data sets. Additionally, MACE can take annotated control items into account, and provides thresholding to exclude low-confidence answers. Feel free to check it out. Comments welcome!