Proceedings of the 2022 2nd International Conference on Computer Technology and Media Convergence Design (CTMCD 2022)

PNRE: Proactive Learning for Neural Relation Extraction with Multiple Annotators

Authors
Rui Qiu1, 2, Wen Ji1, *, Yundan Liang3, Haini Qu4, Jingce Xu5
1Institute of Computing Technology, Chinese Academy of Sciences, Beijing, China
2University of Chinese Academy of Science, Beijing, China
3State Grid Corporation of China, Beijing, China
4State Grid Shanghai Electric Power Company, Shanghai, China
5State Grid Energy Research Institute, Beijing, China
*Corresponding author. Email: jiwen@ict.ac.cn
Corresponding Author
Wen Ji
Available Online 17 December 2022.
DOI
10.2991/978-94-6463-046-6_42How to use a DOI?
Keywords
Proactive Learning; Relation Extraction; Cost-Effectiveness
Abstract

Relation extraction is one of the essential tasks of information extraction, and it is also a fundamental part of knowledge graph construction. Many works have been proposed for supervised relation extraction, which commonly requires a massive amount of human-annotated data with both time and cost. To reduce the labeling time and cost, active learning has been proposed with the assumption of a single perfect annotator that always furnishes the correct label. However, more generally, the annotator will provide incorrect labels according to their labeling capabilities, and different labeling capabilities correspond to distinct costs. To unleash the power of annotators with diverse expertise level and unlabeled data for better model performance with the lowest cost, we develop PNRE, a novel proactive learning based framework for neural relation extraction that actively select the most suitable sample-annotator pairs to construct high-quality relation extraction corpus. Specifically, PNRE utilizes (1) Expert Performance Estimation module to precompute each annotator’s performance considering class prediction probability; (2) Sample Selection module to select the most informative and representative sample based on a hybrid query strategy; (3) Sample Allocation module to allocate appropriate sample to each annotator under the condition of annotation utility maximization. The framework is evaluated on three corpora and is shown to achieve promising results with a significant reduction in labeling costs.

Copyright
© 2023 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

Volume Title
Proceedings of the 2022 2nd International Conference on Computer Technology and Media Convergence Design (CTMCD 2022)
Series
Advances in Computer Science Research
Publication Date
17 December 2022
ISBN
978-94-6463-046-6
ISSN
2352-538X
DOI
10.2991/978-94-6463-046-6_42How to use a DOI?
Copyright
© 2023 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Cite this article

TY  - CONF
AU  - Rui Qiu
AU  - Wen Ji
AU  - Yundan Liang
AU  - Haini Qu
AU  - Jingce Xu
PY  - 2022
DA  - 2022/12/17
TI  - PNRE: Proactive Learning for Neural Relation Extraction with Multiple Annotators
BT  - Proceedings of the 2022 2nd International Conference on Computer Technology and Media Convergence Design (CTMCD 2022)
PB  - Atlantis Press
SP  - 352
EP  - 364
SN  - 2352-538X
UR  - https://doi.org/10.2991/978-94-6463-046-6_42
DO  - 10.2991/978-94-6463-046-6_42
ID  - Qiu2022
ER  -