Not logged in.
Quick Search - Contribution
Contribution Details
| Type | Conference or Workshop Paper |
| Scope | Discipline-based scholarship |
| Published in Proceedings | Yes |
| Title | Collect, measure, repeat: Reliability factors for responsible AI data collection |
| Organization Unit | |
| Authors |
|
| Presentation Type | paper |
| Item Subtype | Original Work |
| Refereed | No |
| Status | Published in final form |
| Language |
|
| ISBN | 978-1-57735-884-8 |
| Page Range | 51 - 64 |
| Event Title | Eleventh AAAI Conference on Human Computation and Crowdsourcing (HCOMP 2023) |
| Event Type | conference |
| Event Location | Delft, the Netherlands |
| Event Start Date | November 6 - 2023 |
| Event End Date | November 10 - 2023 |
| Series Name | Proceedings of the AAAI Conference on Human Computation and Crowdsourcing |
| Number | 11 |
| Place of Publication | Delft, the Netherlands |
| Publisher | Association for the Advancement of Artificial Intelligence |
| Abstract Text | The rapid entry of machine learning approaches in our dailyactivities and high-stakes domains demands transparency andscrutiny of their fairness and reliability. To help gauge ma-chine learning models’ robustness, research typically focuseson the massive datasets used for their deployment,e.g., cre-ating and maintaining documentation to understand theirorigin, process of development, and ethical considerations.However, data collection for AI is still typically a one-offpractice, and oftentimes datasets collected for a certain pur-pose or application are reused for a different problem. Addi-tionally, dataset annotations may not be representative overtime, contain ambiguous or erroneous annotations, or be un-able to generalize across domains. Recent research has shownthese practices might lead to unfair, biased, or inaccurate out-comes. We argue that data collection for AI should be per-formed in a responsible manner where the quality of the datais thoroughly scrutinized and measured through a systematicset of appropriate metrics. In this paper, we propose a Re-sponsible AI (RAI) methodology designed to guide the datacollection with a set of metrics for an iterative in-depth analy-sis of thefactors influencing the quality and reliabilityof thegenerated data. We propose a granular set of measurements toinform on theinternal reliabilityof a dataset and itsexternalstabilityover time. We validate our approach across nine ex-isting datasets and annotation tasks and four input modalities.This approach impacts theassessment of data robustnessusedin real world AI applications, where diversity of users andcontent is eminent. Furthermore, it deals with fairness andaccountability aspects in data collection by providing system-atic and transparent quality analysis for data collections. |
| Free access at | Official URL |
| Digital Object Identifier | 10.1609/hcomp.v11i1.27547 |
| Other Identification Number | merlin-id:24183 |
| PDF File |
Download from ZORA
|
| Export |
BibTeX
EP3 XML (ZORA)
|