Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Last update: Dec 08, 2022

Related tags

Deep Learning Ancient-Greek-BERT

Overview

Ancient Greek BERT

The first and only available Ancient Greek sub-word BERT model!

State-of-the-art post fine-tuning on Part-of-Speech Tagging and Morphological Analysis.

Pre-trained weights are made available for a standard 12 layer, 768d BERT-base model.

You can also use the model directly on the HuggingFace Model Hub here.

Please refer to our paper titled: "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek". In Proceedings of The 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2021).

How to use

Requirements:

pip install transformers
pip install unicodedata
pip install flair

Can be directly used from the HuggingFace Model Hub with:

from transformers import AutoTokenizer, AutoModel
tokeniser = AutoTokenizer.from_pretrained("pranaydeeps/Ancient-Greek-BERT")
model = AutoModel.from_pretrained("pranaydeeps/Ancient-Greek-BERT")

Fine-tuning for POS/Morphological Analysis

finetune_pos.py can be used to finetune the BERT model for POS tagging on your own data. We provide sample files from the Gold standard treebanks, however the full treebanks can't be made available at this time. Please contact the authors for more details.
Even though the full treebanks aren't made available, we provide a pre-trained POS Tagging model in the directory SuperPeitho-FLAIR-v2, which can directly be used for inference and has an accuracy of ~90 percent on the 3 treebanks available. You can import the pre-trained model in FLAIR with:

from flair.models import SequenceTagger
tagger = SequenceTagger.load('SuperPeitho-FLAIR-v2/final-model.pt')

Training data

The model was initialised from AUEB NLP Group's Greek BERT and subsequently trained on monolingual data from the First1KGreek Project, Perseus Digital Library, PROIEL Treebank and Gorman's Treebank

Training and Eval details

Standard de-accentuating and lower-casing for Greek as suggested in AUEB NLP Group's Greek BERT. The model was trained on 4 NVIDIA Tesla V100 16GB GPUs for 80 epochs, with a max-seq-len of 512 and results in a perplexity of 4.8 on the held out test set. It also gives state-of-the-art results when fine-tuned for PoS Tagging and Morphological Analysis on all 3 treebanks averaging >90% accuracy. Please consult our paper or contact me for further questions!

Cite

If you end up using Ancient-Greek-BERT in your research, please cite the paper:

@inproceedings{ancient-greek-bert,
author = {Singh, Pranaydeep and Rutten, Gorik and Lefever, Els},
title = {A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek},
year = {2021},
booktitle = {The 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature (LaTeCH-CLfL 2021)}
}

Comments

Morphological Analysis Examples
@pranaydeeps Is possible to add simple code example how to do morphological analysis of some short ancient greek sentence? Maybe some examples of:

POS tagging

cosine sentence similarity

Thank You very much for the model. I tried to train smaller BERT model but i didn't have enough GPU resources. I would like to use your model for New Testament analysis.
documentation
opened by PeterPirog 3

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

LightningDOT: Pre-training Visual-Semantic Embeddings for Real-Time Image-Text Retrieval This repository contains source code and pre-trained/fine-tun

65 Dec 26, 2022

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data This repository provides the implementation details for

124 Dec 27, 2022

Chinese clinical named entity recognition using pre-trained BERT model

Chinese clinical named entity recognition (CNER) using pre-trained BERT model Introduction Code for paper Chinese clinical named entity recognition wi

109 Dec 14, 2022

Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

Lottery Jackpots Exist in Pre-trained Models (Paper Link) Requirements Python = 3.7.4 Pytorch = 1.6.1 Torchvision = 0.4.1 Reproduce the Experiment

27 Jun 28, 2022

The implementation of our CIKM 2021 paper titled as: "Cross-Market Product Recommendation"

FOREC: A Cross-Market Recommendation System This repository provides the implementation of our CIKM 2021 paper titled as "Cross-Market Product Recomme

16 Sep 12, 2022

Ever felt tired after preprocessing the dataset, and not wanting to write any code further to train your model? Ever encountered a situation where you wanted to record the hyperparameters of the trained model and able to retrieve it afterward? Models Playground is here to help you do that. Models playground allows you to train your models right from the browser.

Models Playground 🗂️ Upload a Preprocessed Dataset 🌠 Choose whether to perform Classification or Regression 🦹 Enter the Dependent Variable ?

19 Dec 10, 2022

This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Dynamic-Vision-Transformer (Pytorch) This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT). Not All Ima

210 Dec 18, 2022

Code and pre-trained models for MultiMAE: Multi-modal Multi-task Masked Autoencoders

MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann*, David Mizrahi*, Andrei Atanov, Amir Zamir Website | arXiv | BibTeX Official PyTo

Visual Intelligence & Learning Lab, Swiss Federal Institute of Technology (EPFL)

385 Jan 6, 2023

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Image Crop Analysis This is a repo for the code used for reproducing our Image Crop Analysis paper as shared on our blog post. If you plan to use this

239 Jan 2, 2023

Pre-trained BERT Models for Ancient and Medieval Greek, and associated code for LaTeCH 2021 paper titled - "A Pilot Study for BERT Language Modelling and Morphological Analysis for Ancient and Medieval Greek"

Related tags

Overview

Ancient Greek BERT

How to use

Fine-tuning for POS/Morphological Analysis

Training data

Training and Eval details

Cite

You might also like...

source code and pre-trained/fine-tuned checkpoint for NAACL 2021 paper LightningDOT

The source codes for ACL 2021 paper 'BoB: BERT Over BERT for Training Persona-based Dialogue Models from Limited Personalized Data'

Chinese clinical named entity recognition using pre-trained BERT model

Pytorch implementation of our paper under review — Lottery Jackpots Exist in Pre-trained Models

The implementation of our CIKM 2021 paper titled as: "Cross-Market Product Recommendation"

This repo contains the official code and pre-trained models for the Dynamic Vision Transformer (DVT).

Code and pre-trained models for MultiMAE: Multi-modal Multi-task Masked Autoencoders

Code for reproducing our analysis in the paper titled: Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency

Comments

Morphological Analysis Examples

Releases(v1.0.0)

v1.0.0(Sep 24, 2021)

Owner

Pranaydeep Singh

Hierarchical-Bayesian-Defense - Towards Adversarial Robustness of Bayesian Neural Network through Hierarchical Variational Inference (Openreview)

img2pose: Face Alignment and Detection via 6DoF, Face Pose Estimation

Sentinel-1 vessel detection model used in the xView3 challenge

RMNA: A Neighbor Aggregation-Based Knowledge Graph Representation Learning Model Using Rule Mining

Anchor-free Oriented Proposal Generator for Object Detection

Source code and dataset of the paper "Contrastive Adaptive Propagation Graph Neural Networks forEfficient Graph Learning"

PyTorch experiments with the Zalando fashion-mnist dataset

Revisiting, benchmarking, and refining Heterogeneous Graph Neural Networks.

SGPT: Multi-billion parameter models for semantic search

A general and strong 3D object detection codebase that supports more methods, datasets and tools (debugging, recording and analysis).

A really easy-to-use and powerful sudoku solver.

A PyTorch implementation of the Transformer model in "Attention is All You Need".

Code for Learning to Segment The Tail (LST)

2021 Artificial Intelligence Diabetes Datathon

3D ResNets for Action Recognition (CVPR 2018)

Apache Spark - A unified analytics engine for large-scale data processing

PyTorch implementation of PP-LCNet

Official repository for Jia, Raghunathan, Göksel, and Liang, "Certified Robustness to Adversarial Word Substitutions" (EMNLP 2019)

A BaSiC Tool for Background and Shading Correction of Optical Microscopy Images

Reference implementation of code generation projects from Facebook AI Research. General toolkit to apply machine learning to code, from dataset creation to model training and evaluation. Comes with pretrained models.