RBERT for Relation Extraction task for KLUE
Project Description
- Relation Extraction task is one of the task of Korean Language Understanding Evaluation(KLUE) Benchmark.
- Relation extraction can be defined as multiclass classification task for relationship between subject entity and object entity.
- Classes are such as
no_relation,per:employee_of,org:founded_by... totaling 30 labels. - This repo contains custom fine-tuning method utilizing monologg's R-BERT Implementation.
- Custom punctuations with Pororo NER has been added to the dataset prior to the model's training.
- If you want to refer to the experimentation note such as punctuation method of the entity, please refer to the blog post
Arguments Usage
| Argument | type | Default | Explanation |
|---|---|---|---|
| batch_size | int | 40 | batch size for training and inferece |
| num_folds | int | 5 | number of fold for Stratified KFold |
| num_train_epochs | int | 5 | number of epochs for training |
| loss | str | focalloss | loss function |
| gamma | float | 1.0 | focalloss's gamma value |
| optimizer | str | adamp | optimizer for training |
| scheduler | str | get_cosine_schedule_with_warmup | learning rate scheduler |
| learning_rate | float | 0.00005 | initial learning rate |
| weight_decay | float | 0.01 | Loss function's weight decay, preventing overfit |
| warmup_step | int | 500 | |
| debug | bool | false | debug with CPU device for better error representation |
| dropout_rate | float | 0.1 | |
| save_steps | int | 100 | number of steps for saving the model |
| evaluation_steps | int | 100 | number of step until the evaluation |
| metric_for_best_model | str | eval/loss | the metric for determining which is the best model |
| load_best_model_at_end | bool | True |
References
- monologg's R-BERT Implementation in Pytorch
- Enriching Pre-trained Language Model with Entity Information for Relation Classification
- KLUE: Korean Language Understanding Evaluation
- KLUE RE (Relation Extraction) Dataset / CC BY-SA
- Pororo's NER Documentation
Authorship
Hardware
GPU : Tesla V100 32GB
4 Sep 28, 2022
13 Jun 09, 2022
1 Jan 01, 2022
2 Sep 20, 2022
2 Feb 08, 2022
1.8k Dec 21, 2022
1 Jun 14, 2022
32 Dec 18, 2022
5 Aug 15, 2022
11.9k Jan 08, 2023
684 Jan 09, 2023
3k Dec 26, 2022
1 Dec 03, 2021
4 Dec 01, 2021
2 Aug 06, 2022
47 Jul 26, 2022
7 Jul 18, 2022
5.1k Dec 30, 2022
919 Jan 03, 2023
276 Dec 31, 2022