For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

Last update: Nov 02, 2022

Related tags

Deep Learning SciBERTSUM

Overview

LongScientificFormer

For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

Some codes are borrowed from ONMT(https://github.com/OpenNMT/OpenNMT-py)

Data Preparation

Option 1: download the processed data

Pre-processed data

Put all files into raw_data directory

Step 2. Download Stanford CoreNLP

We will need Stanford CoreNLP to tokenize the data. Download it here and unzip it. Then add the following command to your bash_profile:

export CLASSPATH=/path/to/stanford-corenlp-4.2.2/stanford-corenlp-4.2.2.jar

replacing /path/to/ with the path to where you saved the stanford-corenlp-4.2.2 directory.

step 3. extracting sections from GROBID XML files

python preprocess.py -mode extract_pdf_sections -log_file ../logs/extract_section.log

step 4. extracting text from TIKA XML files

python preprocess.py -mode get_text_clean_tika -log_file ../logs/extract_tika_text.log

step 5. Tokenize texts from papers and slides using stanfordCoreNLP

python preprocess.py -mode tokenize  -save_path ../temp -log_file ../logs/tokenize_by_corenlp.log

Step 6. Extract source, section, and target from tokenized files

python preprocess.py -mode clean_paper_jsons -save_path ../json_data/  -n_cpus 10 -log_file ../logs/build_json.log

Step 7. Generate BERT `.pt` files from source, sections and targets

python preprocess.py -mode format_to_bert -raw_path ../json_data/ -save_path ../bert_data  -lower -n_cpus 40 -log_file ../logs/build_bert_files.log

Model Training

First run: For the first time, you should use single-GPU, so the code can download the BERT model. Use -visible_gpus -1, after downloading, you could kill the process and rerun the code with multi-GPUs.

Train

python train.py  -ext_dropout 0.1 -lr 2e-3  -visible_gpus 1,2,3 -report_every 200 -save_checkpoint_steps 1000 -batch_size 1 -train_steps 100000 -accum_count 2  -log_file ../logs/ext_bert -use_interval true -warmup_steps 10000

To continue training from a checkpoint

python train.py  -ext_dropout 0.1 -lr 2e-3  -train_from ../models/model_step_99000.pt -visible_gpus 1,2,3 -report_every 200 -save_checkpoint_steps 1000 -batch_size 1 -train_steps 100000 -accum_count 2  -log_file ../logs/ext_bert -use_interval true -warmup_steps 10000

Test

python train.py -mode test  -test_batch_size 1 -bert_data_path ../bert_data -log_file ../logs/ext_bert_test -test_from ../models/model_step_99000.pt -model_path ../models -sep_optim true -use_interval true -visible_gpus 1,2,3 -alpha 0.95 -result_path ../results/ext

For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

Related tags

Overview

LongScientificFormer

Data Preparation

Option 1: download the processed data

Step 2. Download Stanford CoreNLP

step 3. extracting sections from GROBID XML files

step 4. extracting text from TIKA XML files

step 5. Tokenize texts from papers and slides using stanfordCoreNLP

Step 6. Extract source, section, and target from tokenized files

Step 7. Generate BERT `.pt` files from source, sections and targets

Model Training

Train

Test

Owner

Athar Sefid

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

This repository provides a basic implementation of our GCPR 2021 paper "Learning Conditional Invariance through Cycle Consistency"

Project page for End-to-end Recovery of Human Shape and Pose

This is a Deep Leaning API for classifying emotions from human face and human audios.

SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

Keras Realtime Multi-Person Pose Estimation - Keras version of Realtime Multi-Person Pose Estimation project

Unofficial Pytorch Implementation of WaveGrad2

Cobalt Strike teamserver detection.

Geneva is an artificial intelligence tool that defeats censorship by exploiting bugs in censors

A minimalist tool to display a network graph.

[WACV21] Code for our paper: Samuel, Atzmon and Chechik, "From Generalized zero-shot learning to long-tail with class descriptors"

EfficientMPC - Efficient Model Predictive Control Implementation

[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation

The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

This is the code repository implementing the paper "TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction".

天勤量化开发包, 期货量化, 实时行情/历史数据/实盘交易

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

The implementation code for "DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction"

Code of Periodic Activation Functions Induce Stationarity

House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent for Professional Architects

For encoding a text longer than 512 tokens, for example 800. Set max_pos to 800 during both preprocessing and training.

Related tags

Overview

LongScientificFormer

Data Preparation

Option 1: download the processed data

Step 2. Download Stanford CoreNLP

step 3. extracting sections from GROBID XML files

step 4. extracting text from TIKA XML files

step 5. Tokenize texts from papers and slides using stanfordCoreNLP

Step 6. Extract source, section, and target from tokenized files

Step 7. Generate BERT .pt files from source, sections and targets

Model Training

Train

Test

Owner

Athar Sefid

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

This repository provides a basic implementation of our GCPR 2021 paper "Learning Conditional Invariance through Cycle Consistency"

Project page for End-to-end Recovery of Human Shape and Pose

This is a Deep Leaning API for classifying emotions from human face and human audios.

SpeechNAS Better Trade off between Latency and Accuracy for Large Scale Speaker Verification

Keras Realtime Multi-Person Pose Estimation - Keras version of Realtime Multi-Person Pose Estimation project

Unofficial Pytorch Implementation of WaveGrad2

Cobalt Strike teamserver detection.

Geneva is an artificial intelligence tool that defeats censorship by exploiting bugs in censors

A minimalist tool to display a network graph.

[WACV21] Code for our paper: Samuel, Atzmon and Chechik, "From Generalized zero-shot learning to long-tail with class descriptors"

EfficientMPC - Efficient Model Predictive Control Implementation

[CVPR2021] Domain Consensus Clustering for Universal Domain Adaptation

The Official Implementation of the ICCV-2021 Paper: Semantically Coherent Out-of-Distribution Detection.

This is the code repository implementing the paper "TreePartNet: Neural Decomposition of Point Clouds for 3D Tree Reconstruction".

天勤量化开发包, 期货量化, 实时行情/历史数据/实盘交易

Learning from Synthetic Data with Fine-grained Attributes for Person Re-Identification

The implementation code for "DAGAN: Deep De-Aliasing Generative Adversarial Networks for Fast Compressed Sensing MRI Reconstruction"

Code of Periodic Activation Functions Induce Stationarity

House-GAN++: Generative Adversarial Layout Refinement Network towards Intelligent Computational Agent for Professional Architects

Step 7. Generate BERT `.pt` files from source, sections and targets