Long text token classification using LongFormer

Overview

Long text token classification using LongFormer

The data comes from: https://www.kaggle.com/c/feedback-prize-2021/

To train the model for 5 folds, you can run:

python train.py --fold 0 --model allenai/longformer-large-4096 --lr 1e-5 --epochs 10 --max_len 1536 --batch_size 4 --valid_batch_size 4
python train.py --fold 1 --model allenai/longformer-large-4096 --lr 1e-5 --epochs 10 --max_len 1536 --batch_size 4 --valid_batch_size 4
python train.py --fold 2 --model allenai/longformer-large-4096 --lr 1e-5 --epochs 10 --max_len 1536 --batch_size 4 --valid_batch_size 4
python train.py --fold 3 --model allenai/longformer-large-4096 --lr 1e-5 --epochs 10 --max_len 1536 --batch_size 4 --valid_batch_size 4
python train.py --fold 4 --model allenai/longformer-large-4096 --lr 1e-5 --epochs 10 --max_len 1536 --batch_size 4 --valid_batch_size 4

Note that you need have kfold column in training data.

Owner
abhishek thakur
Kaggle: www.kaggle.com/abhishek
abhishek thakur
๐Ÿค— Transformers: State-of-the-art Natural Language Processing for Pytorch, TensorFlow, and JAX.

English | ็ฎ€ไฝ“ไธญๆ–‡ | ็น้ซ”ไธญๆ–‡ State-of-the-art Natural Language Processing for Jax, PyTorch and TensorFlow ๐Ÿค— Transformers provides thousands of pretrained mo

Hugging Face 77.2k Jan 03, 2023
A library for finding knowledge neurons in pretrained transformer models.

knowledge-neurons An open source repository replicating the 2021 paper Knowledge Neurons in Pretrained Transformers by Dai et al., and extending the t

EleutherAI 96 Dec 21, 2022
Stuff related to Ben Eater's 8bit breadboard computer

8bit breadboard computer simulator This is an assembler + simulator/emulator of Ben Eater's 8bit breadboard computer. For a version with its RAM upgra

Marijn van Vliet 29 Dec 29, 2022
CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

CPT This repository contains code and checkpoints for CPT. CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Gener

fastNLP 342 Jan 05, 2023
YACLC - Yet Another Chinese Learner Corpus

ๆฑ‰่ฏญๅญฆไน ่€…ๆ–‡ๆœฌๅคš็ปดๆ ‡ๆณจๆ•ฐๆฎ้›†YACLC V1.0 ไธญๆ–‡ | English ๆฑ‰่ฏญๅญฆไน ่€…ๆ–‡ๆœฌๅคš็ปดๆ ‡ๆณจๆ•ฐๆฎ้›†๏ผˆYet Another Chinese Learner

BLCU-ICALL 47 Dec 15, 2022
This is the code for the EMNLP 2021 paper AEDA: An Easier Data Augmentation Technique for Text Classification

The baseline code is for EDA: Easy Data Augmentation techniques for boosting performance on text classification tasks

Akbar Karimi 81 Dec 09, 2022
Gold standard corpus annotated with verb-preverb connections for Hungarian.

Hungarian Preverb Corpus A gold standard corpus manually annotated with verb-preverb connections for Hungarian. corpus The corpus consist of the follo

RIL Lexical Knowledge Representation Research Group 3 Jan 27, 2022
Coreference resolution for English, French, German and Polish, optimised for limited training data and easily extensible for further languages

Coreferee Author: Richard Paul Hudson, Explosion AI 1. Introduction 1.1 The basic idea 1.2 Getting started 1.2.1 English 1.2.2 French 1.2.3 German 1.2

Explosion 70 Dec 12, 2022
The official implementation of "BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies?, ACL 2021 main conference"

BERT is to NLP what AlexNet is to CV This is the official implementation of BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Iden

Asahi Ushio 20 Nov 03, 2022
Official PyTorch implementation of SegFormer

SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers Figure 1: Performance of SegFormer-B0 to SegFormer-B5. Project page

NVIDIA Research Projects 1.4k Dec 29, 2022
official ( API ) for the zAmericanEnglish app in [ Google play ] and [ App store ]

official ( API ) for the zAmericanEnglish app in [ Google play ] and [ App store ]

Plugin 3 Jan 12, 2022
๋‰ด์Šค ๋„๋ฉ”์ธ ์งˆ์˜์‘๋‹ต ์‹œ์Šคํ…œ (21-1ํ•™๊ธฐ ์กธ์—… ํ”„๋กœ์ ํŠธ)

๋‰ด์Šค ๋„๋ฉ”์ธ ์งˆ์˜์‘๋‹ต ์‹œ์Šคํ…œ ๋ณธ ํ”„๋กœ์ ํŠธ๋Š” ๋‰ด์Šค๊ธฐ์‚ฌ์— ๋Œ€ํ•œ ์งˆ์˜์‘๋‹ต ์„œ๋น„์Šค ๋ฅผ ์ œ๊ณตํ•˜๊ธฐ ์œ„ํ•ด์„œ ์ง„ํ–‰ํ•œ ํ”„๋กœ์ ํŠธ์ž…๋‹ˆ๋‹ค. ์•ฝ 3๊ฐœ์›”๊ฐ„ ( 21. 03 ~ 21. 05 ) ์ง„ํ–‰ํ•˜์˜€์œผ๋ฉฐ Transformer ์•„ํ‚คํ…์ณ ๊ธฐ๋ฐ˜์˜ Encoder๋ฅผ ์‚ฌ์šฉํ•˜์—ฌ ํ•œ๊ตญ์–ด ์งˆ์˜์‘๋‹ต ๋ฐ์ดํ„ฐ์…‹์œผ๋กœ

TaegyeongEo 4 Jul 08, 2022
A python framework to transform natural language questions to queries in a database query language.

__ _ _ _ ___ _ __ _ _ / _` | | | |/ _ \ '_ \| | | | | (_| | |_| | __/ |_) | |_| | \__, |\__,_|\___| .__/ \__, | |_| |_| |___/

Machinalis 1.2k Dec 18, 2022
AI-Broad-casting - AI Broad casting with python

Basic Code 1. Use The Code Configuration Environment conda create -n code_base p

In this project, we aim to achieve the task of predicting emojis from tweets. We aim to investigate the relationship between words and emojis.

Making Emojis More Predictable by Karan Abrol, Karanjot Singh and Pritish Wadhwa, Natural Language Processing (CSE546) under the guidance of Dr. Shad

Karanjot Singh 2 Jan 17, 2022
Behavioral Testing of Clinical NLP Models

Behavioral Testing of Clinical NLP Models This repository contains code for testing the behavior of clinical prediction models based on patient letter

Betty van Aken 2 Sep 20, 2022
์ดˆ์„ฑ ํ•ด์„๊ธฐ based on ko-BART

์ดˆ์„ฑ ํ•ด์„๊ธฐ ๊ฐœ์š” ํ•œ๊ตญ์–ด ์ดˆ์„ฑ๋งŒ์œผ๋กœ ์ด๋ฃจ์–ด์ง„ ๋ฌธ์žฅ์„ ์ž…๋ ฅํ•˜๋ฉด, ์™„์„ฑ๋œ ๋ฌธ์žฅ์„ ์˜ˆ์ธกํ•˜๋Š” ์ดˆ์„ฑ ํ•ด์„๊ธฐ์ž…๋‹ˆ๋‹ค. ์ดˆ์„ฑ: ใ„ดใ„ด ใ„ดใ„น ใ…ˆใ…‡ใ…Ž ์˜ˆ์ธก ๋ฌธ์žฅ: ๋‚˜๋Š” ๋„ˆ๋ฅผ ์ข‹์•„ํ•ด ๋ชจ๋ธ ๋ชจ๋ธ์€ SKT-AI์—์„œ ๊ณต๊ฐœํ•œ Ko-BART๋ฅผ ์ด์šฉํ•ฉ๋‹ˆ๋‹ค. ๋ฐ์ดํ„ฐ ๋ฌธ์žฅ ๋‹จ์œ„๋กœ ์ด๋ฃจ์–ด์ง„ ์•„๋ฌด ์ฝ”ํผ์Šค๋‚˜

Dawoon Jung 29 Oct 28, 2022
Knowledge Graph,Question Answering System๏ผŒๅŸบไบŽ็Ÿฅ่ฏ†ๅ›พ่ฐฑๅ’Œๅ‘้‡ๆฃ€็ดข็š„ๅŒป็–—่ฏŠๆ–ญ้—ฎ็ญ”็ณป็ปŸ

Knowledge Graph,Question Answering System๏ผŒๅŸบไบŽ็Ÿฅ่ฏ†ๅ›พ่ฐฑๅ’Œๅ‘้‡ๆฃ€็ดข็š„ๅŒป็–—่ฏŠๆ–ญ้—ฎ็ญ”็ณป็ปŸ

wangle 823 Dec 28, 2022
An open source library for deep learning end-to-end dialog systems and chatbots.

DeepPavlov is an open-source conversational AI library built on TensorFlow, Keras and PyTorch. DeepPavlov is designed for development of production re

Neural Networks and Deep Learning lab, MIPT 6k Dec 31, 2022
Linear programming solver for paper-reviewer matching and mind-matching

Paper-Reviewer Matcher A python package for paper-reviewer matching algorithm based on topic modeling and linear programming. The algorithm is impleme

Titipat Achakulvisut 66 Jul 05, 2022