Hashtag segmentation is the task of automatically adding spaces between the words on a hashtag.

Hashformers is the current state-of-the-art for hashtag segmentation, as demonstrated on this paper accepted at LREC 2022.

Hashformers is also language-agnostic: you can use it to segment hashtags not just with English models, but also using any language model available on the Hugging Face Model Hub.

from hashformers import TransformerWordSegmenter as WordSegmenter

ws = WordSegmenter(
    segmenter_model_name_or_path="gpt2",
    segmenter_model_type="incremental",
    reranker_model_name_or_path="google/flan-t5-base",
    reranker_model_type="seq2seq"
)

segmentations = ws.segment([
    "#weneedanationalpark",
    "#icecold"
])

print(segmentations)

# [ 'we need a national park',
# 'ice cold' ]

It is also possible to use hashformers without a reranker by setting the reranker_model_name_or_path and the reranker_model_type to None.

pip install hashformers

Visit the HuggingFace Model Hub and choose your models for the WordSegmenter class.

You can use any model supported by the minicons library. Currently hashformers supports the following model types as the segmenter_model_type or reranker_model_type:

Auto-regressive models like GPT-2 and XLNet, or any model that can be loaded with AutoModelForCausalLM. This includes large language models (LLMs) such as Alpaca-LoRA ( chainyo/alpaca-lora-7b ) and GPT-J ( EleutherAI/gpt-j-6b ).

ws = WordSegmenter(
    segmenter_model_name_or_path="EleutherAI/gpt-j-6b",
    segmenter_model_type="incremental",
    reranker_model_name_or_path=None,
    reranker_model_type=None
)

Masked language models like BERT, or any model that can be loaded with AutoModelForMaskedLM.

Seq2Seq models like FLAN-T5 ( google/flan-t5-base ), or any model that can be loaded with AutoModelForSeq2SeqLM.

Best results are usually achieved by using an incremental model as the segmenter_model_name_or_path and a masked or seq2seq model as the reranker_model_name_or_path.

A segmenter is always required, however a reranker is optional.

Pull requests are welcome! Read our paper for more details on the inner workings of our framework.

If you want to develop the library, you can install hashformers directly from this repository ( or your fork ):

git clone https://github.com/ruanchaves/hashformers.git
cd hashformers
pip install -e .

This is a collection of papers that have utilized the hashformers library as a tool in their research.

These papers have utilized hashformers version 1.3 or below.

• Zero-shot hashtag segmentation for multilingual sentiment analysis

• HashSet -- A Dataset For Hashtag Segmentation (LREC 2022)

• Generalizability of Abusive Language Detection Models on Homogeneous German Datasets

• The problem of varying annotations to identify abusive language in social media content

• 15 Datasets for Word Segmentation on the Hugging Face Hub

@misc{rodrigues2021zeroshot,
      title={Zero-shot hashtag segmentation for multilingual sentiment analysis}, 
      author={Ruan Chaves Rodrigues and Marcelo Akira Inuzuka and Juliana Resplande Sant'Anna Gomes and Acquila Santos Rocha and Iacer Calixto and Hugo Alexandre Dantas do Nascimento},
      year={2021},
      eprint={2112.03213},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}