RBERT for Relation Extraction task for KLUE
Project Description
- Relation Extraction task is one of the task of Korean Language Understanding Evaluation(KLUE) Benchmark.
- Relation extraction can be defined as multiclass classification task for relationship between subject entity and object entity.
- Classes are such as
no_relation,per:employee_of,org:founded_by... totaling 30 labels. - This repo contains custom fine-tuning method utilizing monologg's R-BERT Implementation.
- Custom punctuations with Pororo NER has been added to the dataset prior to the model's training.
- If you want to refer to the experimentation note such as punctuation method of the entity, please refer to the blog post
Arguments Usage
| Argument | type | Default | Explanation |
|---|---|---|---|
| batch_size | int | 40 | batch size for training and inferece |
| num_folds | int | 5 | number of fold for Stratified KFold |
| num_train_epochs | int | 5 | number of epochs for training |
| loss | str | focalloss | loss function |
| gamma | float | 1.0 | focalloss's gamma value |
| optimizer | str | adamp | optimizer for training |
| scheduler | str | get_cosine_schedule_with_warmup | learning rate scheduler |
| learning_rate | float | 0.00005 | initial learning rate |
| weight_decay | float | 0.01 | Loss function's weight decay, preventing overfit |
| warmup_step | int | 500 | |
| debug | bool | false | debug with CPU device for better error representation |
| dropout_rate | float | 0.1 | |
| save_steps | int | 100 | number of steps for saving the model |
| evaluation_steps | int | 100 | number of step until the evaluation |
| metric_for_best_model | str | eval/loss | the metric for determining which is the best model |
| load_best_model_at_end | bool | True |
References
- monologg's R-BERT Implementation in Pytorch
- Enriching Pre-trained Language Model with Entity Information for Relation Classification
- KLUE: Korean Language Understanding Evaluation
- KLUE RE (Relation Extraction) Dataset / CC BY-SA
- Pororo's NER Documentation
Authorship
Hardware
GPU : Tesla V100 32GB
3 Dec 05, 2022
14 Nov 29, 2022
293 Jan 02, 2023
4 Aug 06, 2021
1 Mar 20, 2022
767 Jan 09, 2023
97 Dec 15, 2022
22 Oct 21, 2022
144 Dec 27, 2022
75 Dec 27, 2022
226 Dec 30, 2022
3 May 23, 2022
1k Dec 30, 2022
45 Sep 06, 2022
24 Dec 29, 2022
45 Jan 05, 2023
91 Nov 29, 2022
89 Dec 18, 2022
1.4k Dec 30, 2022
219 Jan 02, 2023