Extract knowledge from raw text

Overview

Extract knowledge from raw text

This repository is a nearly copy-paste of "From Text to Knowledge: The Information Extraction Pipeline" with some cosmetic updates. I made an installable version to evaluate it easily. The original code is available @ trinity-ie. To create some value, I added the Luke model to predict relations between entities. Luke is a transformer (same family as Bert), its particularity is that during its pre-training, it trains parameters dedicated to entities within the attention mechanism. Luke is in fact a very efficient model on entity-related tasks. We use here the version of Luke fine-tuned on the dataset TACRED.

In this blog post, Tomaz Bratanic presents a complete pipeline for extracting triples from raw text. The first step of the pipeline is to resolve the coreferences. The second step of the pipeline is to identify entities using the Wikifier API. Finally, Tomaz Bratanic proposes to use the Opennre library to extract relations between entities within the text.

🔧 Installation

pip install git+https://github.com/raphaelsty/textokb --upgrade

You will have to download spacy en model to do coreference resolution:

pip install spacy==2.1.0 && python -m spacy download en

Quick start

>> device = "cpu" # or device = "cuda" if you do own a gpu. >>> pipeline = pipeline.TextToKnowledge(key="jueidnxsctiurpwykpumtsntlschpx", types=types, device=device) >>> text = """Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, ... CEO, CTO, and chief designer of SpaceX. He is also early investor, CEO, and product architect of ... Tesla, Inc. He is also the founder of The Boring Company and the co-founder of Neuralink. A ... centibillionaire, Musk became the richest person in the world in January 2021, with an estimated ... net worth of $185 billion at the time, surpassing Jeff Bezos. Musk was born to a Canadian mother ... and South African father and raised in Pretoria, South Africa. He briefly attended the University ... of Pretoria before moving to Canada aged 17 to attend Queen's University. He transferred to the ... University of Pennsylvania two years later, where he received dual bachelor's degrees in economics ... and physics. He moved to California in 1995 to attend Stanford University, but decided instead to ... pursue a business career. He went on co-founding a web software company Zip2 with his brother ... Kimbal Musk.""" >>> pipeline.process_sentence(text = text) head relation tail score 0 Tesla, Inc. architect Elon Musk 0.803398 1 Tesla, Inc. field of work The Boring Company 0.733903 2 Elon Musk residence University of Pennsylvania 0.648434 3 Elon Musk field of work The Boring Company 0.592007 4 Elon Musk manufacturer Tesla, Inc. 0.553206 5 The Boring Company manufacturer Tesla, Inc. 0.515352 6 Elon Musk developer Kimbal Musk 0.475639 7 University of Pennsylvania subsidiary Elon Musk 0.435384 8 The Boring Company developer Elon Musk 0.387753 9 SpaceX winner Elon Musk 0.374090 10 Kimbal Musk sibling Elon Musk 0.355944 11 Elon Musk manufacturer SpaceX 0.221294 ">
>>> from textokb import pipeline

# A list of types of entities that I search:
>>> types = [
...   "human", 
...   "person", 
...   "company", 
...   "enterprise", 
...   "business", 
...   "geographic region", 
...   "human settlement", 
...   "geographic entity", 
...   "territorial entity type", 
...   "organization",
... ]

>>> device = "cpu" # or device = "cuda" if you do own a gpu.

>>> pipeline = pipeline.TextToKnowledge(key="jueidnxsctiurpwykpumtsntlschpx", types=types, device=device)

>>> text = """Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, 
... CEO, CTO, and chief designer of SpaceX. He is also early investor, CEO, and product architect of 
... Tesla, Inc. He is also the founder of The Boring Company and the co-founder of Neuralink. A 
... centibillionaire, Musk became the richest person in the world in January 2021, with an estimated 
... net worth of $185 billion at the time, surpassing Jeff Bezos. Musk was born to a Canadian mother 
... and South African father and raised in Pretoria, South Africa. He briefly attended the University 
... of Pretoria before moving to Canada aged 17 to attend Queen's University. He transferred to the 
... University of Pennsylvania two years later, where he received dual bachelor's degrees in economics 
... and physics. He moved to California in 1995 to attend Stanford University, but decided instead to 
... pursue a business career. He went on co-founding a web software company Zip2 with his brother 
... Kimbal Musk."""

>>> pipeline.process_sentence(text = text)
                          head       relation                        tail     score
0                  Tesla, Inc.      architect                   Elon Musk  0.803398
1                  Tesla, Inc.  field of work          The Boring Company  0.733903
2                    Elon Musk      residence  University of Pennsylvania  0.648434
3                    Elon Musk  field of work          The Boring Company  0.592007
4                    Elon Musk   manufacturer                 Tesla, Inc.  0.553206
5           The Boring Company   manufacturer                 Tesla, Inc.  0.515352
6                    Elon Musk      developer                 Kimbal Musk  0.475639
7   University of Pennsylvania     subsidiary                   Elon Musk  0.435384
8           The Boring Company      developer                   Elon Musk  0.387753
9                       SpaceX         winner                   Elon Musk  0.374090
10                 Kimbal Musk        sibling                   Elon Musk  0.355944
11                   Elon Musk   manufacturer                      SpaceX  0.221294

By default the model used is wiki80_cnn_softmax. I also added the model Luke (Language Understanding with Knowledge-based Embeddings) which provide a pre-trained models to do relation extraction. The results of the Luke model seem to be of better quality but the number of predicted relationships is smaller.

Here is how to use LUKE

>> device = "cpu" # or device = "cuda" if you do own a gpu. >>> pipeline = pipeline.TextToKnowledge(key="jueidnxsctiurpwykpumtsntlschpx", types=types, device=device, luke=True) >>> text = """Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, ... CEO, CTO, and chief designer of SpaceX. He is also early investor, CEO, and product architect of ... Tesla, Inc. He is also the founder of The Boring Company and the co-founder of Neuralink. A ... centibillionaire, Musk became the richest person in the world in January 2021, with an estimated ... net worth of $185 billion at the time, surpassing Jeff Bezos. Musk was born to a Canadian mother ... and South African father and raised in Pretoria, South Africa. He briefly attended the University ... of Pretoria before moving to Canada aged 17 to attend Queen's University. He transferred to the ... University of Pennsylvania two years later, where he received dual bachelor's degrees in economics ... and physics. He moved to California in 1995 to attend Stanford University, but decided instead to ... pursue a business career. He went on co-founding a web software company Zip2 with his brother ... Kimbal Musk.""" >>> pipeline.process_sentence(text = text) head relation tail score 0 Elon Musk per:siblings Kimbal Musk 10.436224 1 Kimbal Musk per:siblings Elon Musk 10.040980 2 Elon Musk per:schools_attended University of Pennsylvania 9.808870 3 The Boring Company org:founded_by Elon Musk 8.823962 4 Elon Musk per:employee_of Tesla, Inc. 8.245111 5 SpaceX org:founded_by Elon Musk 7.795369 6 Elon Musk per:employee_of SpaceX 7.765485 7 Elon Musk per:employee_of The Boring Company 7.217330 8 Tesla, Inc. org:founded_by Elon Musk 7.002990 ">
>>> from textokb import pipeline

# A list of types of entities that I search:
>>> types = [
...   "human", 
...   "person", 
...   "company", 
...   "enterprise", 
...   "business", 
...   "geographic region", 
...   "human settlement", 
...   "geographic entity", 
...   "territorial entity type", 
...   "organization",
... ]

>>> device = "cpu" # or device = "cuda" if you do own a gpu.

>>> pipeline = pipeline.TextToKnowledge(key="jueidnxsctiurpwykpumtsntlschpx", types=types, device=device, luke=True)

>>> text = """Elon Musk is a business magnate, industrial designer, and engineer. He is the founder, 
... CEO, CTO, and chief designer of SpaceX. He is also early investor, CEO, and product architect of 
... Tesla, Inc. He is also the founder of The Boring Company and the co-founder of Neuralink. A 
... centibillionaire, Musk became the richest person in the world in January 2021, with an estimated 
... net worth of $185 billion at the time, surpassing Jeff Bezos. Musk was born to a Canadian mother 
... and South African father and raised in Pretoria, South Africa. He briefly attended the University 
... of Pretoria before moving to Canada aged 17 to attend Queen's University. He transferred to the 
... University of Pennsylvania two years later, where he received dual bachelor's degrees in economics 
... and physics. He moved to California in 1995 to attend Stanford University, but decided instead to 
... pursue a business career. He went on co-founding a web software company Zip2 with his brother 
... Kimbal Musk."""

>>> pipeline.process_sentence(text = text)
                 head              relation                        tail      score
0           Elon Musk          per:siblings                 Kimbal Musk  10.436224
1         Kimbal Musk          per:siblings                   Elon Musk  10.040980
2           Elon Musk  per:schools_attended  University of Pennsylvania   9.808870
3  The Boring Company        org:founded_by                   Elon Musk   8.823962
4           Elon Musk       per:employee_of                 Tesla, Inc.   8.245111
5              SpaceX        org:founded_by                   Elon Musk   7.795369
6           Elon Musk       per:employee_of                      SpaceX   7.765485
7           Elon Musk       per:employee_of          The Boring Company   7.217330
8         Tesla, Inc.        org:founded_by                   Elon Musk   7.002990

Here is the list of available relations using Luke studio-ousia/luke-large-finetuned-tacred:

[
    'no_relation',
    'org:alternate_names',
    'org:city_of_headquarters',
    'org:country_of_headquarters',
    'org:dissolved',
    'org:founded',
    'org:founded_by',
    'org:member_of',
    'org:members',
    'org:number_of_employees/members',
    'org:parents',
    'org:political/religious_affiliation',
    'org:shareholders',
    'org:stateorprovince_of_headquarters',
    'org:subsidiaries',
    'org:top_members/employees',
    'org:website',
    'per:age',
    'per:alternate_names',
    'per:cause_of_death',
    'per:charges',
    'per:children',
    'per:cities_of_residence',
    'per:city_of_birth',
    'per:city_of_death',
    'per:countries_of_residence',
    'per:country_of_birth',
    'per:country_of_death',
    'per:date_of_birth',
    'per:date_of_death',
    'per:employee_of',
    'per:origin',
    'per:other_family',
    'per:parents',
    'per:religion',
    'per:schools_attended',
    'per:siblings',
    'per:spouse',
    'per:stateorprovince_of_birth',
    'per:stateorprovince_of_death',
    'per:stateorprovinces_of_residence',
    'per:title'
]

Notes

The first time you initialize the model with Opennre or Luke, you may have to wait a few minutes for the model to download. Since we use the Wikifier API to track entities (NEL), it is necessary that your computer is connected to the internet. You can create your own credential for the API here: Wikifier API registration. Tomaz Bratanic mentions the possibility to replace Wikifier with BLINK however this library is very RAM intensive.

♻️ Work in progress

I failed to use the wiki80_bert_softmax model from Opennre due to a pre-trained model loading error (i.e. Tensorflow errors on Mac M1). I used the lighter model wiki80_cnn_softmax when reproducing Tomaz Bratanic's blog post. It would be interesting to be able to easily add different models and especially transformers. The API I used are not optimized for batch predictions. There are a lot of room for improvement by simply updating Opennre and Luke APIs.

You might also like...
Fixes mojibake and other glitches in Unicode text, after the fact.

ftfy: fixes text for you print(fix_encoding("(ง'⌣')ง")) (ง'⌣')ง Full documentation: https://ftfy.readthedocs.org Testimonials “My life is li

Paranoid text spacing in Python

pangu.py Paranoid text spacing for good readability, to automatically insert whitespace between CJK (Chinese, Japanese, Korean) and half-width charact

Markup is an online annotation tool that can be used to transform unstructured documents into structured formats for NLP and ML tasks, such as named-entity recognition. Markup learns as you annotate in order to predict and suggest complex annotations. Markup also provides integrated access to existing and custom ontologies, enabling the prediction and suggestion of ontology mappings based on the text you're annotating.
Markup is an online annotation tool that can be used to transform unstructured documents into structured formats for NLP and ML tasks, such as named-entity recognition. Markup learns as you annotate in order to predict and suggest complex annotations. Markup also provides integrated access to existing and custom ontologies, enabling the prediction and suggestion of ontology mappings based on the text you're annotating.

Markup is an online annotation tool that can be used to transform unstructured documents into structured formats for NLP and ML tasks, such as named-entity recognition. Markup learns as you annotate in order to predict and suggest complex annotations. Markup also provides integrated access to existing and custom ontologies, enabling the prediction and suggestion of ontology mappings based on the text you're annotating.

🐸   Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙‍♀️
🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙‍♀️

🐸 Identify anything. pyWhat easily lets you identify emails, IP addresses, and more. Feed it a .pcap file or some text and it'll tell you what it is! 🧙‍♀️

box is a text-based visual programming language inspired by Unreal Engine Blueprint function graphs.
box is a text-based visual programming language inspired by Unreal Engine Blueprint function graphs.

Box is a text-based visual programming language inspired by Unreal Engine blueprint function graphs. $ cat factorial.box ┌─ƒ(Factorial)───┐

Export solved codewars kata challenges to a text file.

Codewars Kata Exporter Note:this is not totally my work.i've edited the project to make more easier and faster for me.you can find the original work h

AnnIE - Annotation Platform, tool for open information extraction annotations using text files.
AnnIE - Annotation Platform, tool for open information extraction annotations using text files.

AnnIE - Annotation Platform, tool for open information extraction annotations using text files.

py-trans is a Free Python library for translate text into different languages.

Free Python library to translate text into different languages.

A production-ready pipeline for text mining and subject indexing

A production-ready pipeline for text mining and subject indexing

Releases(0.0.1)
Owner
Raphael Sourty
PhD Student @ IRIT and Renault
Raphael Sourty
Extract price amount and currency symbol from a raw text string

price-parser is a small library for extracting price and currency from raw text strings.

Scrapinghub 252 Dec 31, 2022
Getting git-style versioning working on RDFlib

Getting git-style versioning working on RDFlib

Gabe Fierro 1 Feb 01, 2022
一款高性能敏感词(非法词/脏字)检测过滤组件,附带繁体简体互换,支持全角半角互换,汉字转拼音,模糊搜索等功能。

一款高性能非法词(敏感词)检测组件,附带繁体简体互换,支持全角半角互换,获取拼音首字母,获取拼音字母,拼音模糊搜索等功能。

ToolGood 3.6k Jan 07, 2023
Text to ASCII and ASCII to text

Text2ASCII Description This python script (converter.py) contains two functions: encode() is used to return a list of Integer, one item per character

4 Jan 22, 2022
Convert ebooks with few clicks on Telegram!

E-Book Converter Bot A bot that converts e-books to various formats, powered by calibre! It currently supports 34 input formats and 19 output formats.

Youssif Shaaban Alsager 45 Jan 05, 2023
A working (ish) python script to convert text to a gradient.

verticle-horiontal-gradient-script A working (ish) python script to convert text to a gradient. This script is poorly made with the well known python

prmze 1 Feb 20, 2022
A python Tk GUI that creates, writes text and attaches images into a custom spreadsheet file

A python Tk GUI that creates, writes text and attaches images into a custom spreadsheet file

Mirko Simunovic 13 Dec 09, 2022
Wikipedia Extractive Text Summarizer + Keywords Identification (entropy-based)

Wikipedia Extractive Text Summarizer + Keywords Identification (entropy-based)Wikipedia Extractive Text Summarizer + Keywords Identification (entropy-based)

Kevin Lai 1 Nov 08, 2021
Repositori untuk belajar pemrograman Python dalam bahasa Indonesia

Python Repositori ini berisi kumpulan dari berbagai macam contoh struktur data, algoritma dan komputasi matematika yang diimplementasikan dengan mengg

Bellshade 111 Dec 19, 2022
text-to-speach bot - You really do NOT have time for read a newsletter? Now you can listen to it

NewsletterReader You really do NOT have time for read a newsletter? Now you can listen to it The Newsletter of Filipe Deschamps is a great place to re

ItanuRomero 8 Sep 18, 2021
Aml - anti-money laundering

Anti-money laundering Dedect relationship between A and E by tracing through payments with similar amounts and identifying payment chains. For example

3 Nov 21, 2022
A python tool one can extract the "hash" from a WINDOWS HELLO PIN

WINHELLO2hashcat About With this tool one can extract the "hash" from a WINDOWS HELLO PIN. This hash can be cracked with Hashcat, more precisely with

33 Dec 05, 2022
This project is a small tool for processing url-containing texts delivered by HUAWEI Share on Windows.

hwshare_helper This project is a small tool for handling url-containing texts delivered by HUAWEI Share on Windows. config Before use, please install

1 Jan 19, 2022
A Python app which can convert normal text to Handwritten text.

Text to HandWritten Text ✍️ Converter Watch Tutorial for this project Usage:- Clone my repository. Open CMD in working directory. Run following comman

Kushal Bhavsar 5 Dec 11, 2022
PyMultiDictionary is a Dictionary Module for Python 3+ to get meanings, translations, synonyms and antonyms of words in 20 different languages

PyMultiDictionary PyMultiDictionary is a Dictionary Module for Python 3+ to get meanings, translations, synonyms and antonyms of words in 20 different

Pablo Pizarro R. 19 Dec 26, 2022
A program that looks through entered text and replaces certain commands with mathematical symbols

TextToSymbolConverter A program that looks through entered text and replaces certain commands with mathematical symbols Example: Syntax: Enter text in

1 Jan 02, 2022
Simple python program to auto credit your code, text, book, whatever!

Credit Simple python program to auto credit your code, text, book, whatever! Setup First change credit_text to whatever text you would like to credit

Hashm 1 Jan 29, 2022
Little python script + dictionary to help solve Wordle puzzles

Wordle Solver Little python script + dictionary to help solve Wordle puzzles Usage Usage: ./wordlesolver.py [letters in word] [letters not in word] [p

Luke Stephens (hakluke) 4 Jul 24, 2022
This is an AI that is supposed to say you if your text is formal or not

This is an AI that is supposed to say you if your text is formal or not. It's written in Python 3 and has some german examples (because I'm german yk) in the text.json file. This file contains the te

1 Jan 12, 2022
a python package that lets you add custom colors and text formatting to your scripts in a very easy way!

colormate Python script text formatting package What is colormate? colormate is a python library that lets you add text formatting to your scripts, it

Rodrigo 2 Dec 14, 2022