Crowd-Kit: Computational Quality Control for Crowdsourcing
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets. We strive to implement functionality that simplifies working with crowdsourced data.
Currently, Crowd-Kit contains:
- implementations of commonly-used aggregation methods for categorical, pairwise, textual, and segmentation responses
- metrics of uncertainty, consistency, and agreement with aggregate
- loaders for popular crowdsourced datasets
The library is currently in a heavy development state, and interfaces are subject to change.
Installing
Installing Crowd-Kit is as easy as pip install crowd-kit
Getting Started
This example shows how to use Crowd-Kit for categorical aggregation using the classical Dawid-Skene algorithm.
First, let us do all the necessary imports.
from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd
Then, you need to read your annotations into Pandas DataFrame with columns task, performer, label. Alternatively, you can download an example dataset.
df = pd.read_csv('results.csv') # should contain columns: task, performer, label
# df, ground_truth = load_dataset('relevance-2') # or download an example dataset
Then you can aggregate the performer responses as easily as in scikit-learn:
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)
Implemented Aggregation Methods
Below is the list of currently implemented methods, including the already available (
Categorical Responses
| Method | Status |
|---|---|
| Majority Vote | |
| Dawid-Skene | |
| Gold Majority Vote | |
| M-MSR | |
| Wawa | |
| Zero-Based Skill | |
| GLAD | |
| BCC | |
Textual Responses
| Method | Status |
|---|---|
| RASA | |
| HRRASA | |
| ROVER | |
Image Segmentation
| Method | Status |
|---|---|
| Segmentation MV | |
| Segmentation RASA | |
| Segmentation EM | |
Pairwise Comparisons
| Method | Status |
|---|---|
| Bradley-Terry | |
| Noisy Bradley-Terry | |
Citation
- Ustalov D., Pavlichenko N., Losev V., Giliazev I., and Tulin E. A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python. The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track. HCOMP 2021. 2021. arXiv: 2109.08584 [cs.HC].
@inproceedings{HCOMP2021/CrowdKit,
author = {Ustalov, Dmitry and Pavlichenko, Nikita and Losev, Vladimir and Giliazev, Iulian and Tulin, Evgeny},
title = {{A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python}},
year = {2021},
booktitle = {The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track},
series = {HCOMP~2021},
eprint = {2109.08584},
eprinttype = {arxiv},
eprintclass = {cs.HC},
url = {https://www.humancomputation.com/assets/wips_demos/HCOMP_2021_paper_85.pdf},
language = {english},
}
Questions and Bug Reports
- For reporting bugs please use the Toloka/bugreport page.
- Join our English-speaking slack community for both tech and abstract questions.
License
© YANDEX LLC, 2020-2021. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
48 Nov 15, 2022
24 Sep 07, 2022
527 Dec 04, 2022
156 Nov 28, 2022
88 Dec 23, 2022
16 Oct 10, 2022
35 Oct 07, 2022
17 Oct 12, 2022
679 Jan 04, 2023
1.2k Dec 26, 2022
51 Nov 10, 2022
2 Apr 07, 2022
29 Aug 30, 2022
438 Dec 22, 2022
1 Nov 12, 2021
1 Dec 27, 2021
21 Dec 15, 2022
573 Jan 05, 2023
3 Nov 15, 2021
47 Dec 13, 2022